Emotion recognition method and apparatus, electronic device, and storage medium

Abstract

The present disclosure relates to an emotion recognition method and device, electronic equipment and storage medium. The method comprises: segmenting the image to be recognized into a foreground image and a background image, and extracting foreground features of the foreground image and background features of the background image, respectively, wherein the foreground image comprises a target object; extracting text features of text information included in the image to be recognized, and determining first interaction features according to the text features and the foreground features; extracting scene features of scene label information included in the image to be recognized, and determining second interaction features according to the scene features and the background features; and determining the emotion of the target object in the image to be recognized according to the first interaction features, the second interaction features and the text features.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the technical field of emotion recognition, and in particular to an emotion recognition method and device, an electronic device, and a storage medium. BACKGROUND

[0002] With the continuous development and progress of artificial intelligence technology, human emotions can be recognized, and therefore robots or voice assistants are equipped with emotion recognition functions. Under the guidance of emotion recognition results, robots or voice assistants can better communicate and interact with users, improving the user experience. In related technologies, the accuracy of emotion recognition needs to be improved. SUMMARY

[0003] To overcome the problems in related technologies, the present disclosure provides an emotion recognition method, device, electronic device, and storage medium to solve the defects in related technologies.

[0004] According to a first aspect of the present disclosure, an emotion recognition method is provided, comprising:

[0005] segmenting the image to be recognized into a foreground image and a background image, and extracting foreground features of the foreground image and background features of the background image, respectively, wherein the foreground image includes a target object;

[0006] extracting text features of text information included in the image to be recognized, and determining first interaction features according to the text features and the foreground features;

[0007] extracting scene features of scene label information included in the image to be recognized, and determining second interaction features according to the scene features and the background features;

[0008] determining the emotion of the target object in the image to be recognized according to the first interaction features, the second interaction features, and the text features.

[0009] In some embodiments, the segmentation of the image to be recognized into a foreground image and a background image comprises:

[0010] determining an attention image of the image to be recognized, wherein each pixel of the attention image has an attention value;

[0011] determining a pixel region with an attention value higher than or equal to a preset attention value threshold as the foreground image, and determining a pixel region with an attention value lower than the attention value threshold as the background image.

[0012] In some embodiments, the extraction of text features of text information included in the image to be recognized comprises:

[0013] encode the text information to obtain an implicit representation of the text information;

[0014] input the implicit representation of the text information into a self-attention mechanism network, so that the self-attention mechanism network extracts a text feature of the text information;

[0015] The scene feature of the scene label information of the to-be-recognized image includes:

[0016] encode the scene label information to obtain an implicit representation of the scene label information;

[0017] input the implicit representation of the scene label information into a self-attention mechanism network, so that the self-attention mechanism network extracts a scene feature of the scene label information.

[0018] In some embodiments, the determining a first interaction feature according to the text feature and the foreground feature includes:

[0019] input the text feature and the foreground feature into a target attention mechanism network, so that the target attention mechanism network outputs the first interaction feature;

[0020] The second interaction feature is determined according to the scene feature and the background feature, including:

[0021] input the scene feature and the background feature into a target attention mechanism network, so that the target attention mechanism network outputs the second interaction feature.

[0022] In some embodiments, the inputting the text feature and the foreground feature into a target attention mechanism network includes:

[0023] the text feature is taken as a query condition of the target attention mechanism network, and the foreground feature is taken as a key-value pair of the target attention mechanism network;

[0024] The scene feature and the background feature are input into a target attention mechanism network, including:

[0025] the scene feature is taken as a query condition of the target attention mechanism network, and the background feature is taken as a key-value pair of the target attention mechanism network.

[0026] In some embodiments, the determining an emotion of a target object in the to-be-recognized image according to the first interaction feature, the second interaction feature and the text feature includes:

[0027] concatenate the first interaction feature, the second interaction feature and the text feature, and input the concatenated result into a multi-modal attention mechanism network, so that the multi-modal attention mechanism network outputs a multi-modal interaction feature;

[0028] input the multi-modal interaction feature into a first emotion classifier, so that the first emotion classifier outputs an emotion of the target object.

[0029] In some embodiments, the scene label information is obtained in the following manner:

[0030] input the to-be-identified image into a pre-trained second emotion classifier, so that the second emotion classifier outputs scene label information of the to-be-identified image.

[0031] According to a second aspect of the embodiments of the present disclosure, an emotion recognition device is provided, comprising:

[0032] a segmentation module configured to segment the to-be-identified image into a foreground image and a background image, and extract foreground features of the foreground image and background features of the background image respectively, wherein the foreground image comprises a target object;

[0033] a first interaction module configured to extract text features of text information included in the to-be-identified image, and determine first interaction features according to the text features and the foreground features;

[0034] a second interaction module configured to extract scene features of scene label information included in the to-be-identified image, and determine second interaction features according to the scene features and the background features;

[0035] a classification module configured to determine an emotion of a target object in the to-be-identified image according to the first interaction features, the second interaction features and the text features.

[0036] The segmentation module is configured to, when segmenting the to-be-identified image into a foreground image and a background image, specifically configured to:

[0037] determine an attention image of the to-be-identified image, wherein each pixel of the attention image has an attention value;

[0038] determine a pixel region with an attention value higher than or equal to a preset attention value threshold as the foreground image, and determine a pixel region with an attention value lower than the attention value threshold as the background image.

[0039] In some embodiments, the first interaction module is configured to, when extracting text features of text information of the to-be-identified image, specifically configured to:

[0040] encode the text information to obtain an implicit representation of the text information;

[0041] input the implicit representation of the text information into a self-attention mechanism network, so that the self-attention mechanism network extracts text features of the text information;

[0042] The scene feature of the scene label information of the to-be-recognized image includes:

[0043] The scene label information is encoded to obtain an implicit representation of the scene label information;

[0044] input the implicit representation of the scene label information into a self-attention mechanism network, so that the self-attention mechanism network extracts scene features of the scene label information.

[0045] In some embodiments, the first interaction module is configured to determine a first interaction feature according to the text features and the foreground features, specifically configured to:

[0046] input the text features and the foreground features into a target attention mechanism network, so that the target attention mechanism network outputs the first interaction feature;

[0047] The second interaction feature is determined according to the scene features and the background features, including:

[0048] input the scene features and the background features into a target attention mechanism network, so that the target attention mechanism network outputs the second interaction feature.

[0049] In some embodiments, the first interaction module is configured to input the text features and the foreground features into a target attention mechanism network, specifically configured to:

[0050] the text features are used as query conditions of the target attention mechanism network, and the foreground features are used as key-value pairs of the target attention mechanism network;

[0051] The second interaction module is configured to input the scene features and the background features into a target attention mechanism network, specifically configured to:

[0052] the scene features are used as query conditions of the target attention mechanism network, and the background features are used as key-value pairs of the target attention mechanism network.

[0053] In some embodiments, the classification module is specifically configured to:

[0054] concatenate the first interaction feature, the second interaction feature and the text feature, and input the concatenated result into a multi-modal attention mechanism network, so that the multi-modal attention mechanism network outputs a multi-modal interaction feature;

[0055] input the multi-modal interaction feature into a first emotion classifier, so that the first emotion classifier outputs an emotion of the target object.

[0056] In some embodiments, further comprising a scene label module configured to obtain the scene label information by:

[0057] input the to-be-identified image into a pre-trained second emotion classifier, so that the second emotion classifier outputs scene label information of the to-be-identified image.

[0058] According to a third aspect of the embodiments of the present disclosure, an electronic device is provided, which comprises a memory and a processor, the memory is configured to store computer instructions executable on the processor, and the processor is configured to execute the computer instructions to implement the emotion recognition method according to the first aspect.

[0059] According to a fourth aspect of the embodiments of the present disclosure, a computer readable storage medium is provided, which stores a computer program, and the program is executed by a processor to implement the method according to the first aspect.

[0060] The technical solutions provided by the embodiments of the present disclosure can have the following beneficial effects:

[0061] The embodiments of the present disclosure first divide the to-be-identified image into a foreground image and a background image, extract foreground features of the foreground image and background features of the background image respectively, wherein the foreground image comprises a target object; then extract text features of text information included in the to-be-identified image, and determine first interaction features according to the text features and the foreground features; then extract scene features of scene label information included in the to-be-identified image, and determine second interaction features according to the scene features and the background features; finally, determine an emotion of the target object in the to-be-identified image according to the first interaction features, the second interaction features and the text features. Since the first interaction features and the second interaction features are combined, the first interaction features are obtained by combining the text features of the text information and the foreground features of the foreground image, the second interaction features are obtained by combining the scene features of the scene label information and the background features of the background image, that is, in addition to focusing on the foreground features and the text features, scene information is also focused on, so that the accuracy of emotion recognition is improved, and the use experience of products related to emotion recognition is further improved. BRIEF DESCRIPTION OF DRAWINGS

[0062] The accompanying drawings, which are incorporated herein and constitute part of this specification, illustrate implementations of the present disclosure and, together with the description, further serve to explain the principles of the present disclosure.

[0063] Figure 1 is a flowchart of an emotion recognition method according to an example embodiment of the present disclosure;

[0064] Figure 2 is a structural diagram of an emotion recognition model according to an example embodiment of the present disclosure;

[0065] Figure 3 is a structural diagram of an emotion recognition device according to an example embodiment of the present disclosure;

[0066] Figure 4 is a block diagram of an electronic device according to an example embodiment of the present disclosure. DETAILED DESCRIPTION

[0067] The example embodiments are described herein with reference to the accompanying drawings, in which the example embodiments are shown. When the description is made with reference to the drawings, the same numbers in different drawings are intended to represent the same or similar elements. The embodiments described in the example embodiments below do not represent all the implementations in accordance with the present disclosure. Rather, they are merely examples in accordance with some aspects of the present disclosure as detailed in the appended claims.

[0068] The terminology used in the present disclosure is for the purpose of describing particular embodiments only and is not intended to be limiting of the present disclosure. As used in the present disclosure and the appended claims, the singular forms "a," "an" and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will also be understood that the term "and / or" as used herein refers to and encompasses any and all possible combinations of one or more of the associated listed items.

[0069] It will be understood that, although the terms first, second, third, etc. can be used herein to describe various information, these terms are not intended to denote a temporal or chronological order. Rather, these terms are used merely as labels to distinguish between different sets of information. For example, a first information can be termed a second information, and similarly, a second information can be termed a first information, without departing from the scope of the present disclosure. The word "if' as used herein means "when" or "upon" or "in response to the determination" depending on the context.

[0070] With the continuous development and progress of artificial intelligence technology, human emotions can be recognized, so robots or voice assistants are equipped with emotion recognition functions. Under the guidance of emotion recognition results, robots or voice assistants can better communicate with users and improve user experience. In related technologies, the accuracy of emotion recognition needs to be improved.

[0071] For example, in related technologies, when performing emotion recognition, a multi-modal sarcasm detection is performed on a tweet composed of text and images. The global features of the image are used to supervise whether the text describes an ironic situation, and then the text description is calibrated as positive or negative. However, this is from the text, and the overall image or a certain block image is used to determine the emotional attribute of a certain target, so the accuracy of emotion recognition is still low.

[0072] Therefore, in a first aspect, at least one embodiment of the present disclosure provides an emotion recognition method. Please refer to the accompanying Figure 1 which shows the flow of the method, including step S101 and step S104.

[0073] The emotion recognition method can be executed by an electronic device such as a terminal device or a server. The terminal device can be a user equipment (UE), a mobile device, a user terminal, a terminal, a cellular phone, a cordless phone, a personal digital assistant (PDA) handheld device, a computing device, a vehicle-mounted device, a wearable device, etc. The method can be realized by a processor calling computer readable instructions stored in a memory.

[0074] The method can be applied to a scenario where a robot or a voice assistant provides services, for example, a robot in a restaurant performs emotion recognition when a customer enters the restaurant, and plays a more appropriate welcome word according to the customer's emotion.

[0075] In step S101, the image to be recognized is divided into a foreground image and a background image, and foreground features of the foreground image and background features of the background image are extracted, respectively. The foreground image includes a target object.

[0076] The foreground image can be a region where the target object is located, or a region where the face of the target object is located, that is, a region that can show the expression of the emotion of the target object. This is because the expression of the target object can show the emotion. For example, after detecting that a person's mouth is raised, the robot may determine that he is in a good mood, and if the nose is wrinkled, it represents anger or disgust.

[0077] The background image is other regions outside the foreground image, i.e., a scene in the to-be-identified image, such as a scene environment (for example, a wedding scene, a funeral scene, a proposal scene, etc.) in which a target object is located. The scene in the background image has a certain guiding effect on emotion recognition. For example, people in a wedding scene are generally happy, and people in a funeral scene are generally sad. People in both wedding and funeral scenes may cry, but the emotions corresponding to the crying expression are completely different, i.e., crying in a wedding scene is a positive emotion such as happiness and excitement, while crying in a funeral scene is sadness.

[0078] In some embodiments, the to-be-identified image can be segmented into a foreground image and a background image in the following manner: first, determine an attention image of the to-be-identified image, wherein each pixel of the attention image has an attention value; determine a pixel region with an attention value higher than or equal to a preset attention value threshold as the foreground image, and determine a pixel region with an attention value lower than the attention value threshold as the background image. The attention threshold can be preset to distinguish between a high-attention part (such as a human facial expression) and a low-attention part (such as a scene environment). In some possible implementations, an Attention Cropping module can be used to segment the to-be-identified image.

[0079] In addition, a ResNet module can be used to extract foreground features of the foreground image and background features of the background image.

[0080] In step S102, a text feature of text information included in the to-be-identified image is extracted, and a first interaction feature is determined according to the text feature and the foreground feature.

[0081] The text information included in the to-be-identified image is a general description of the to-be-identified image, for example, there is a child playing in the park in the image, and the text information can be "a child playing in the park". The text feature can represent an emotional view expressed by the text, the foreground feature can represent an emotional view obtained from the foreground image, and the first interaction feature can combine the two emotional views to obtain a comprehensive emotional view.

[0082] In some embodiments, the text feature of the text information included in the to-be-identified image can be extracted in the following manner: first, encode the text information using a BERT encoder or other encoder to obtain an implicit representation of the text information; next, input the implicit representation of the text information into a self-attention mechanism network (Self Attention) to enable the self-attention mechanism network to extract the text feature of the text information.

[0083] It can be understood that the self-attention mechanism network is used for feature extraction on the two objects of the implicit representation of the text information and the implicit representation of the text information, so the implicit representation of the text information can be used as both a query condition of the self-attention mechanism network and a key-value pair of the self-attention mechanism network.

[0084] In some embodiments, the first interaction feature can be determined according to the text feature and the foreground feature in the following manner: the text feature and the foreground feature are input into a target attention mechanism network (TargetAttention) so that the target attention mechanism network outputs the first interaction feature.

[0085] It can be understood that the target attention mechanism network is used for feature extraction on the two objects of the text feature and the foreground feature, so the text feature is used as a query condition of the target attention mechanism network and the foreground feature is used as a key-value pair of the target attention mechanism network.

[0086] The target attention mechanism network can be an m-head target attention mechanism network, and the target attention mechanism network can match the target and the image respectively to obtain a target-sensitive visual representation.

[0087] In step S103, scene feature of scene label information included in the to-be-identified image is extracted, and a second interaction feature is determined according to the scene feature and the background feature.

[0088] The scene label information included in the to-be-identified image is a scene description of the to-be-identified image, for example, there is a woman in a wedding dress crying at a wedding in the image, and the scene label information can be "wedding". The scene feature can represent the emotional direction corresponding to the scene, the background feature can represent the emotional direction obtained by the background image, and the second interaction feature can obtain a comprehensive emotional direction by combining the two emotional directions.

[0089] In some embodiments, the scene label information is obtained in the following manner: the to-be-identified image is input into a pre-trained second emotion classifier so that the second emotion classifier outputs the scene label information of the to-be-identified image. The second emotion classifier can be obtained by training using images labeled with scene information.

[0090] In some embodiments, the scene feature of the scene label information included in the to-be-identified image can be extracted in the following manner: first, encoding the scene label information using an encoder such as a BERT encoder to obtain an implicit representation of the scene label information; next, inputting the implicit representation of the scene label information into a self-attention mechanism network (Self Attention) to enable the self-attention mechanism network to extract the scene feature of the scene label information.

[0091] It can be understood that the self-attention mechanism network is used for feature extraction on one object, i.e., the implicit representation of the scene label information, so the implicit representation of the scene label information serves as both the query condition (query) and the key-value pair (key, value) of the self-attention mechanism network.

[0092] In some embodiments, the second interaction feature can be determined according to the scene feature and the background feature in the following manner: inputting the scene feature and the background feature into a target attention mechanism network (Target Attention) to enable the target attention mechanism network to output the second interaction feature.

[0093] It can be understood that the target attention mechanism network is used for feature extraction on two objects, i.e., the scene feature and the background feature, so the scene feature serves as the query condition (query) of the target attention mechanism network and the background feature serves as the key-value pair (key, value) of the target attention mechanism network.

[0094] The target attention mechanism network can be an m-head target attention mechanism network, which can match the target and the image respectively to obtain a target-sensitive visual representation.

[0095] In step S104, the emotion of the target object in the to-be-identified image is determined according to the first interaction feature, the second interaction feature, and the text feature.

[0096] In some embodiments, this step can be performed in the following manner: first, concatenating the first interaction feature, the second interaction feature, and the text feature, and inputting the concatenation result into a multimodal attention mechanism network (Multimodal Attention) to enable the multimodal attention mechanism network to output a multimodal interaction feature; next, inputting the multimodal interaction feature into a first emotion classifier to enable the first emotion classifier to output the emotion of the target object.

[0097] The multi-modal interaction feature is used to combine the emotional view of the first interaction feature and the emotional direction of the second interaction feature to obtain a comprehensive emotional view, i.e., the emotional view is corrected by the emotional direction.

[0098] The multi-modal attention mechanism network is used to extract features for the cascade result as one object, so the cascade result is used as a query condition of the multi-modal attention mechanism network and a key-value pair of the multi-modal attention mechanism network.

[0099] The multi-modal interaction feature combines the local of the image, the scene of the image, the text description, the scene label, the emotional view and the emotional direction, and the emotional view and the emotional direction correct each other, so that the emotional result obtained by the first emotional classifier is more accurate.

[0100] The disclosure first divides the image to be recognized into a foreground image and a background image, extracts foreground features of the foreground image and background features of the background image, respectively, wherein the foreground image includes a target object; then extracts text features of text information included in the image to be recognized, determines first interaction features according to the text features and the foreground features; then extracts scene features of scene label information included in the image to be recognized, determines second interaction features according to the scene features and the background features; finally, determines the emotion of the target object in the image to be recognized according to the first interaction features, the second interaction features and the text features. Since the first interaction features and the second interaction features are combined, the first interaction features are obtained by combining the text features of the text information and the foreground features of the foreground image, the second interaction features are obtained by combining the scene features of the scene label information and the background features of the background image, i.e., in addition to focusing on the foreground features and the text features, the scene information is also focused on, so that the accuracy of emotion recognition is improved, and the use experience of products related to emotion recognition is further improved.

[0101] Please refer to the attached drawings Figure 2An exemplary emotion recognition model for performing the emotion recognition method provided by the present disclosure is shown in FIG. 3, which includes three self-attention mechanism networks 201, two target attention mechanism networks 202, a multi-modal attention mechanism network 203, an Attention Cropping module, a ResNet module, and a first emotion classifier 204. The two self-attention mechanism networks 201 are used to extract text features of text information, one self-attention mechanism network 201 is used to extract scene features of scene label information, the Attention Cropping module is used to segment the image to be recognized into a foreground image and a background image, the ResNet module is used to extract foreground features of the foreground image and background features of the background image, one target attention mechanism network 202 is used to extract first interaction features in combination with the text features and the foreground features, the other target attention mechanism network 202 is used to extract second interaction features in combination with the scene features and the background features, the multi-modal attention mechanism network 203 is used to extract multi-modal interaction features in combination with the first interaction features, the second interaction features, and the text features, and the first emotion classifier 204 is used to determine an emotion recognition result according to the multi-modal interaction features. The emotion recognition model can combine the multi-modal emotion recognition method of scene, local, and language, and accurately recognize emotions in all directions.

[0102] According to a second aspect of the embodiments of the present disclosure, an emotion recognition device is provided, please refer to the accompanying Figure 3 , the device comprises:

[0103] The segmentation module 301 is configured to segment the image to be recognized into a foreground image and a background image, and extract foreground features of the foreground image and background features of the background image, respectively. The foreground image includes a target object.

[0104] The first interaction module 302 is configured to extract text features of text information included in the image to be recognized, and determine first interaction features according to the text features and the foreground features.

[0105] The second interaction module 303 is configured to extract scene features of scene label information included in the image to be recognized, and determine second interaction features according to the scene features and the background features.

[0106] The classification module 304 is configured to determine an emotion of the target object in the image to be recognized according to the first interaction features, the second interaction features, and the text features.

[0107] When the segmentation module 301 segments the image to be recognized into a foreground image and a background image, the segmentation module 301 is specifically configured to:

[0108] determine an attention image of the to-be-identified image, wherein each pixel of the attention image has an attention value;

[0109] determine a pixel region with an attention value higher than or equal to a preset attention value threshold as the foreground image, and determine a pixel region with an attention value lower than the attention value threshold as the background image.

[0110] In some embodiments, when the first interaction module 302 is used to extract the text feature of the text information of the to-be-identified image, specifically for:

[0111] encode the text information to obtain an implicit representation of the text information;

[0112] input the implicit representation of the text information into a self-attention mechanism network, so that the self-attention mechanism network extracts the text feature of the text information;

[0113] The extraction of the scene feature of the scene label information of the to-be-identified image includes:

[0114] The scene label information is encoded to obtain an implicit representation of the scene label information;

[0115] input the implicit representation of the scene label information into a self-attention mechanism network, so that the self-attention mechanism network extracts the scene feature of the scene label information.

[0116] In some embodiments, when the first interaction module 302 is used to determine the first interaction feature according to the text feature and the foreground feature, specifically for:

[0117] input the text feature and the foreground feature into a target attention mechanism network, so that the target attention mechanism network outputs the first interaction feature;

[0118] The determination of the second interaction feature according to the scene feature and the background feature includes:

[0119] input the scene feature and the background feature into a target attention mechanism network, so that the target attention mechanism network outputs the second interaction feature.

[0120] In some embodiments, when the first interaction module 302 is used to input the text feature and the foreground feature into a target attention mechanism network, specifically for:

[0121] the text feature is used as a query condition of the target attention mechanism network, and the foreground feature is used as a key-value pair of the target attention mechanism network;

[0122] The second interaction module 303 is configured to input the scene feature and the background feature into a target attention mechanism network, and specifically configured to:

[0123] The scene feature is taken as a query condition of the target attention mechanism network, and the background feature is taken as a key-value pair of the target attention mechanism network.

[0124] In some embodiments, the classification module 304 is specifically configured to:

[0125] The first interaction feature, the second interaction feature and the text feature are concatenated, and the concatenation result is input into a multi-modal attention mechanism network, so that the multi-modal attention mechanism network outputs a multi-modal interaction feature;

[0126] The multi-modal interaction feature is input into a first emotion classifier, so that the first emotion classifier outputs an emotion of the target object.

[0127] In some embodiments, further comprising a scene label module 305 configured to obtain the scene label information in the following manner:

[0128] The to-be-recognized image is input into a pre-trained second emotion classifier, so that the second emotion classifier outputs scene label information of the to-be-recognized image.

[0129] As to the apparatus in the above-mentioned embodiments, the specific manners in which various modules perform operations have been described in detail in the embodiments of the method in the first aspect, and thus will not be described in detail here.

[0130] According to a third aspect of the embodiments of the present disclosure, an electronic device is provided, please refer to the attached Figure 4 The electronic device includes a memory and a processor, the memory is configured to store computer instructions executable on the processor, and the processor is configured to execute the computer instructions to implement the emotion recognition method according to the first aspect.

[0131] According to a fourth aspect of the embodiments of the present disclosure, a computer readable storage medium is provided, which stores a computer program executable by a processor, and the program is executed by the processor to implement the method according to the first aspect.

[0132] Other embodiments of the present disclosure will be readily apparent to those skilled in the art upon considering the description of the specification and practicing the disclosed disclosure. The present application is intended to cover any variations, uses, or adaptive changes of the present disclosure that follow the general principles of the present disclosure and include common knowledge or conventional technical means in the art that are not disclosed by the present disclosure. The specification and examples are only considered as exemplary, and the true scope and spirit of the present disclosure are indicated by the following claims.

[0133] It should be understood that the present disclosure is not limited to the precise construction that has been described above and shown in the accompanying drawings, and that various modifications and changes can be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims

1. A method of emotion recognition, characterized by, The method comprises the following steps: segmenting the to-be-identified image into a foreground image and a background image, and extracting foreground features of the foreground image and background features of the background image respectively, wherein the foreground image comprises a target object; extracting text features of text information included in the to-be-identified image, and determining first interaction features according to the text features and the foreground features; extracting scene features of scene label information included in the to-be-identified image, and determining second interaction features according to the scene features and the background features; determining the emotion of the target object in the to-be-identified image according to the first interaction features, the second interaction features and the text features. 2.The emotion recognition method of claim 1, wherein, The step of segmenting the to-be-identified image into a foreground image and a background image comprises the following steps: determining an attention image of the to-be-identified image, wherein each pixel of the attention image has an attention value; determining a pixel region with an attention value higher than or equal to a preset attention value threshold as the foreground image, and determining a pixel region with an attention value lower than the attention value threshold as the background image. 3.The emotion recognition method of claim 1, wherein, The step of extracting text features of text information included in the to-be-identified image comprises the following steps: encoding the text information to obtain an implicit representation of the text information; inputting the implicit representation of the text information into a self-attention mechanism network to enable the self-attention mechanism network to extract text features of the text information; The step of extracting scene features of scene label information included in the to-be-identified image comprises the following steps: encoding the scene label information to obtain an implicit representation of the scene label information; inputting the implicit representation of the scene label information into a self-attention mechanism network to enable the self-attention mechanism network to extract scene features of the scene label information.

4. The emotion recognition method of claim 1, wherein, The step of determining first interaction features according to the text features and the foreground features comprises the following steps: inputting the text features and the foreground features into a target attention mechanism network to enable the target attention mechanism network to output the first interaction features; The step of determining second interaction features according to the scene features and the background features comprises the following steps: inputting the scene features and the background features into a target attention mechanism network to enable the target attention mechanism network to output the second interaction features.

5. The emotion recognition method of claim 4, wherein, The step of inputting the text features and the foreground features into a target attention mechanism network comprises the following steps: taking the text features as a query condition of the target attention mechanism network, and taking the foreground features as a key-value pair of the target attention mechanism network; The step of inputting the scene features and the background features into a target attention mechanism network comprises the following steps: taking the scene features as a query condition of the target attention mechanism network, and taking the background features as a key-value pair of the target attention mechanism network.

6. The emotion recognition method of claim 4, wherein, The step of determining the emotion of the target object in the to-be-identified image according to the first interaction features, the second interaction features and the text features comprises the following steps: concatenating the first interaction features, the second interaction features and the text features, and inputting a concatenated result into a multi-modal attention mechanism network to enable the multi-modal attention mechanism network to output multi-modal interaction features; input the multi-modal interaction feature into a first emotion classifier, so that the first emotion classifier outputs an emotion of the target object.

7. The emotion recognition method of claim 1, wherein, The scene label information is obtained in the following manner: input the to-be-identified image into a pre-trained second emotion classifier, so that the second emotion classifier outputs scene label information of the to-be-identified image.

8. An emotion recognition apparatus characterized by comprising: comprise: a segmentation module, configured to segment a to-be-identified image into a foreground image and a background image, and extract foreground features of the foreground image and background features of the background image respectively, wherein the foreground image comprises a target object; a first interaction module, configured to extract text features of text information included in the to-be-identified image, and determine first interaction features according to the text features and the foreground features; a second interaction module, configured to extract scene features of scene label information included in the to-be-identified image, and determine second interaction features according to the scene features and the background features; a classification module, configured to determine an emotion of a target object in the to-be-identified image according to the first interaction features, the second interaction features, and the text features.

9. An electronic device, comprising: The electronic device comprises a memory and a processor, the memory is used to store computer instructions executable on the processor, and the processor is used to execute the computer instructions to implement the emotion recognition method in any one of claims 1 to 7.

10. A computer-readable storage medium having stored thereon a computer program, characterized in that, The program is executed by the processor to implement the method in any one of claims 1 to 7.

Images