Emotion recognition method, device, electronic device and storage medium

Abstract

The embodiment of the present invention discloses an emotion recognition method, device, electronic device and storage medium. The emotion recognition method includes: collecting a facial image of a person to be recognized, and voice information output by the person to be recognized associated with the collection time of the facial image; performing face detection on the facial image, and obtaining an image coding result according to the face detection result; performing text coding and concept coding on the voice information, and obtaining a voice coding result according to the text coding result and the concept coding result; fusing the image coding result and the voice coding result, and identifying the emotion of the person to be recognized according to the fused coding result. The embodiment of the present invention relates to the field of artificial intelligence technology, and improves the accuracy of character emotion recognition by fusing coding results.

Description

Technical Field

[0001] Embodiments of the present invention relate to the field of artificial intelligence technology, and in particular to an emotion recognition method, device, electronic device and storage medium. Background Art

[0002] Emotion recognition is an important part of human-computer interaction and affective computing research. Emotional computing, also known as artificial emotional intelligence or emotional AI, is the research and development of systems and devices to identify, understand and simulate human emotions. Common and important categories of human emotions include happiness, anger, disgust, fear, sadness, surprise, etc. The purpose of emotion recognition research is to simulate and calculate emotions. This technology enables machines to understand the emotional state of humans during human-computer interaction, react to their behavior, and respond appropriately to the recognized emotions, thereby improving the intelligence of human-computer interaction.

[0003] At present, there are four main types of emotion recognition technologies. Based on speech emotion recognition, various changes in the autonomic nervous system can indirectly change a person's speech, and emotion recognition technology can use this information to identify emotions. For example, speech produced in a state of fear, anger or joy will be pronounced quickly, loudly and accurately, with a higher pitch and a wider range, while emotions such as tiredness, boredom, and sadness often produce slow, low and unclear speech. Based on facial expression emotion recognition, the facial expressions of the face are analyzed to identify emotions such as happiness, anger, surprise, fear, etc. Based on body movement emotion recognition, automatic action analysis can be effectively used to detect the user's specific emotional state. For example, when you don't know the answer to a question, you can raise your shoulders, when you use an object, you can point to them with your fingers, etc. These actions can be captured and recognized. Based on body data monitoring emotion recognition, the user's emotional state is detected by detecting and analyzing the user's body data signals, such as heart rate, blood pressure, brain wave data, etc.

[0004] In the existing technology, emotion recognition is mainly based on one of the facial expressions, voice, body movements or body data monitoring; among them, the facial expression, voice or body movement method has the problem of low accuracy and stability; the method based on body data monitoring requires the active cooperation of the person being tested to be completed, and has the problem of low user experience. Summary of the invention

[0005] The embodiments of the present invention provide an emotion recognition method, device, electronic device and storage medium to improve the accuracy of human emotion recognition.

[0006] In a first aspect, an embodiment of the present invention provides an emotion recognition method, comprising:

[0007] Collecting a facial image of a person to be identified, and voice information output by the person to be identified associated with a timing of collecting the facial image;

[0008] Performing face detection on the face image, and obtaining an image encoding result according to the face detection result;

[0009] Performing text coding and concept coding on the speech information, and obtaining a speech coding result according to the text coding result and the concept coding result;

[0010] The image encoding result and the speech encoding result are fused, and the emotion of the person to be identified is identified according to the fused encoding result.

[0011] In a second aspect, an embodiment of the present invention further provides an emotion recognition device, comprising:

[0012] An information collection module, used to collect a facial image of a person to be identified, and voice information output by the person to be identified associated with a collection timing of the facial image;

[0013] An image processing module is used to perform face detection on the face image and obtain an image encoding result according to the face detection result;

[0014] A speech processing module, used for performing text coding and concept coding on the speech information, and obtaining a speech coding result according to the text coding result and the concept coding result;

[0015] The emotion recognition module is used to fuse the image encoding result and the speech encoding result, and recognize the emotion of the person to be recognized according to the fused encoding result.

[0016] In a third aspect, an embodiment of the present invention further provides an electronic device, including:

[0017] one or more processors;

[0018] a storage device for storing one or more programs,

[0019] When the one or more programs are executed by the one or more processors, the one or more processors implement the emotion recognition method as described in any embodiment of the present invention.

[0020] In a fourth aspect, an embodiment of the present invention further provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the emotion recognition method as described in any embodiment of the present invention.

[0021] The embodiment of the present invention is based on encoding the facial image of the person to be identified to obtain an image encoding result, and encoding the voice information output by the person to be identified to obtain a voice encoding result, wherein the voice encoding result is obtained based on text encoding and concept encoding of the voice information to improve the accuracy of the voice encoding result; then, emotion recognition of the person to be identified is performed based on the fusion encoding result of the image encoding result and the voice encoding result to improve the accuracy of emotion recognition of the person. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1 is a flow chart of the emotion recognition method in Embodiment 1 of the present invention;

[0023] Figure 2 is a schematic diagram of the structure of an emotion recognition device in Embodiment 2 of the present invention;

[0024] Figure 3 It is a schematic diagram of the structure of an electronic device in Embodiment 3 of the present invention. DETAILED DESCRIPTION

[0025] The present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It is to be understood that the specific embodiments described herein are only used to explain the present invention, rather than to limit the present invention. It should also be noted that, for ease of description, only the parts related to the present invention, rather than all structures, are shown in the accompanying drawings. And in this application, all actions to obtain signals, information or data are carried out under the premise of complying with the corresponding data protection laws and policies of the country of residence and obtaining authorization from the owner of the corresponding device.

[0026] Embodiment 1

[0027] Figure 1 is a flow chart of the emotion recognition method in the first embodiment of the present invention. This embodiment is applicable to the case of recognizing the real-time emotions of a person. The method can be executed by an emotion recognition device, which can be implemented in software and / or hardware and can be configured in an electronic device, for example, a background server or other device with communication and computing capabilities. Figure 1 As shown, the method specifically includes:

[0028] Step 101: collect a facial image of a person to be identified and voice information output by the person to be identified that is associated with a collection timing of the facial image.

[0029] Among them, the person to be identified refers to an object that needs to be subject to emotion recognition, such as an object that is interacting with a machine. During human-computer interaction, the machine needs to obtain the person's emotions in real time so as to respond promptly according to the person's emotions. The face image refers to the image of the face part collected when emotion recognition is performed, and the voice information output by the person to be identified associated with the timing of collecting the face image refers to the voice output by the person to be identified at the same time when the face image is collected. Exemplarily, during human-computer interaction, the face image of the person in the interaction is obtained in real time, and the voice information output by the person to be identified within a time period of a preset length corresponding to the current time is obtained. For example, a real-time face image of a person is obtained every second, and the voice information output by the person to be identified within this second is obtained.

[0030] Step 102: Perform face detection on the face image, and obtain an image encoding result according to the face detection result.

[0031] The purpose of face detection is to obtain the expression feature information on the face image. Specifically, a single-stage pixel face positioning method is used to perform face detection and face alignment, and finally face feature extraction is performed, and encoding is performed based on the extracted features to obtain an image encoding result.

[0032] In a feasible embodiment, performing face detection on a face image includes:

[0033] Face detection is performed on face images based on a multi-task learning model; wherein the multi-tasks include at least a face classification task, a face frame position regression task, and a face key point regression task.

[0034] The face detection is performed by using a hyper-supervised and self-supervised multi-task approach to obtain multi-task face detection results. The multi-task includes a classification task branch for predicting the face frame, a face frame position regression task branch, and a face key point regression task branch. The face detection result obtained by the multi-task learning model is to output the face prediction score, face frame prediction position, and face key point prediction position for each input face image.

[0035] Exemplarily, the model is trained by hyper-supervision and self-supervision multi-task. After the model training is completed, a face image is input, and it can be predicted whether the face image contains a face, the position of the face frame, and the position of the face key points.

[0036] In a feasible embodiment, the multi-task loss function during multi-task learning model training is:

[0037]

[0038] Among them, L represents the final loss value in the multi-task learning model, represents the loss value of the face classification task for the i-th face image sample, represents the loss value of the face box regression task of the i-th face image sample, represents the loss value of the face key point regression task of the i-th face image sample, λ1 and λ2 are preset parameter values, and p i represents the face prediction probability in the i-th face image sample, represents the true value of the face annotation in the i-th face image sample, t i represents the predicted position information of the face frame in the i-th face image sample, represents the face frame annotation position information in the i-th face image sample, l i represents the predicted position information of facial key points in the i-th face image sample, Represents the location information of facial key points in the i-th face image sample.

[0039] Specifically, the loss value of the face classification task of the i-th face image sample is obtained through the face classification task, the loss value of the face frame regression task of the i-th face image sample is obtained through the face frame position regression task, and the loss value of the face key point regression task of the i-th face image sample is obtained through the face key point regression task. λ1 and λ2 are preset parameter values, which can be determined according to experience and are not limited in the embodiment of the present invention. i Refers to the prediction results obtained through the face classification task, Refers to the data that is pre-labeled to indicate whether there is a face in the sample. For example, if there is a face in the i-th face image sample, then If there is no face, t i It refers to the prediction result obtained through the face frame position regression task. Refers to the pre-annotated location information of the face frame in the sample. For example, t i =(t x ,t y ,t w ,t h ) i , where t x ,t y ,t w and t h They represent the x-coordinate value, y-coordinate value of the upper left corner of the predicted face frame, and the width and height of the face frame respectively. in, and Respectively represent the x-coordinate value, y-coordinate value of the upper left corner of the pre-annotated face frame, and the width and height of the face frame. i It refers to the prediction result obtained through the face key point regression task. It refers to the pre-annotated location information of the facial key points in the sample.

[0040] The multi-task loss function is used to train the multi-task learning model, that is, to find the optimal solution to minimize the loss L. After the model is trained, it can predict whether it is a face, the position information of the face frame, the position information of the face key points, etc.

[0041] Optionally, the multi-task loss function for multi-task learning model training is:

[0042] Among them, L Pixel represents the density regression loss, represents the true value of the face annotation in the i-th face image sample, and λ3 is a preset parameter value, which can be determined according to the empirical value and is not limited in the embodiment of the present invention. By adding the density regression loss value of the face image to the multi-task loss function, the training accuracy of the multi-task learning model is improved.

[0043] In a feasible embodiment, obtaining an image encoding result according to a face detection result includes:

[0044] Perform feature extraction on the face detection result based on the convolutional neural network to obtain the feature extraction result;

[0045] Covariance pooling is used to calculate the feature extraction results to obtain the image encoding results.

[0046] Among them, the face detection results include: the probability of face recognition, the position information of the face frame and the position information of the face key points.

[0047] Specifically, new features are extracted from the detected and aligned faces by using convolutional neural network calculations and then covariance pooling. The key to identifying facial expressions into different categories is to capture the distortion of facial key point areas, and the second-order statistics of covariance can better capture this distortion in facial area features, thereby ensuring that the image encoding results accurately express features related to emotions and improving the accuracy of emotion recognition.

[0048] In a feasible embodiment, the calculation formula of covariance pooling is as follows:

[0049]

[0050] Among them, C represents the calculation result of covariance pooling, n represents the feature dimension in the feature extraction result, and f i ∈(f1,f2,…,f n ), f1,f2,…,f nRepresents the feature data of n dimensions in the feature extraction result,

[0051] Among them, f i is the feature of the i-th dimension in the feature extraction set, (f1,f2,…,f n ) is the feature extraction set obtained by convolutional neural network feature extraction, is the average value of the feature extraction set. Specifically, the calculation result C of the covariance pooling is the image encoding result, which is recorded as e1.

[0052] Step 103: Perform text coding and concept coding on the voice information, and obtain a voice coding result according to the text coding result and the concept coding result.

[0053] Among them, text coding refers to the expression of speech information at the text level; or text coding expresses the literal meaning of speech information; concept coding refers to the expression of speech information at the conceptual level, or concept coding expresses the inherent meaning of speech information. The speech coding results are used to comprehensively express speech information at the text level and the conceptual level, and realize the in-depth mining of the meaning expressed by the speech information.

[0054] Specifically, after receiving the voice information, the voice information is subjected to text encoding and concept encoding to obtain text encoding results and concept encoding results, wherein the text encoding results reflect the characterization of the literal meaning of the voice information; the concept encoding results reflect the hidden meaning obtained according to the knowledge graph. Finally, the voice encoding results are obtained according to the text encoding results and the concept encoding results, thereby improving the accuracy and comprehensiveness of the voice encoding results. Exemplarily, the voice encoding results obtained according to the text encoding results and the concept encoding results can be obtained by fusing the two in a certain preset combination method, and the preset combination method can be set according to the dimensions of the text encoding results and the concept encoding results and other information, such as fusion coding, etc.

[0055] In a feasible embodiment, before text encoding and concept encoding are performed on the voice information, the method further includes:

[0056] Perform speech recognition on the voice information to obtain text recognition results.

[0057] Both text encoding and concept encoding are obtained by processing the text recognition results of voice information. Therefore, before performing text encoding and concept encoding, it is first necessary to perform voice recognition on the voice information output by the person to be identified to obtain the text recognition results in the voice information.

[0058] In a feasible embodiment, text encoding of voice information includes:

[0059] Use Bi-LSTM to process the text recognition results and obtain the intermediate vector representation of the text recognition results;

[0060] Use the scaled dot product attention mechanism to process the intermediate vector representation to obtain the matrix representation;

[0061] The matrix representation is processed using maximum pooling to obtain the sentence representation of the text recognition result as the text encoding result.

[0062] Among them, Bi-LSTM (Bidirectional Long Short Term Memory) combines the information of the input sequence in both the forward and backward directions. For the output at time t, the forward LSTM layer has the information of time t and before in the input sequence, while the backward LSTM layer has the information of time t and after in the input sequence. The output vectors of the forward LSTM layer at time t and the backward LSTM layer at time t can be processed by addition, average or concatenation.

[0063] Specifically, after obtaining the text recognition results, before using self-attention, Bi-LSTM is used to transform the underlying input. The attention mechanism uses weighted sums to generate output vectors, so its representation ability is limited. At the same time, Bi-LSTM is good at capturing the contextual information of the sequence, which can further improve the expression ability of the attention network. After the Bi-LSTM output, it passes through a self-attention mechanism to extract more effective features. The attention mechanism uses a scaled dot-product attention mechanism. The calculation formula of the scaled dot-product attention mechanism is as follows:

[0064]

[0065] Among them, A is the output feature matrix representation of the self-attention mechanism, and then the maximum pooling layer is used to obtain the sentence representation q, which is the text encoding result. The purpose of the maximum pooling layer is to select the maximum value in each dimension of the vector to capture the most important features. Among them, Q, K, and V are the data calculated by multiplying the data calculated by Bi-LSTM with the weight parameter matrix obtained by model training. Q is the query matrix, K is the keyword matrix, V is the target matrix, and u is the preset parameter value.

[0066] In a feasible embodiment, before using Bi-LSTM to process the text recognition result, it also includes:

[0067] Perform character embedding and / or word embedding on the text recognition result to obtain a text embedding vector; wherein, character embedding is performed based on a convolutional neural network, and word embedding is performed based on a pre-trained embedding vector;

[0068] Accordingly, Bi-LSTM is used to process the text recognition results, including:

[0069] Use Bi-LSTM to process text embedding vectors.

[0070] In order to improve the processing efficiency of text recognition results and increase the dimensional information of text features, the text recognition results are processed before using Bi-LSTM to process the text recognition results.

[0071] After speech recognition, the text recognition result is a short text sequence of length n. When performing speech recognition, in order to ensure real-time performance, the input text is a short text, so the text recognition result is a short text sequence. In order to improve the processing efficiency of the text recognition result, the short text sequence is converted into a vector representation. Specifically, the text recognition result is character embedded and / or word embedded to obtain a text embedding vector. Among them, any one or both of character embedding and word embedding can be used to process the short text sequence to obtain a text embedding vector corresponding to the short text sequence.

[0072] Specifically, a short text sequence is represented as a vector x=(x1,x2,…,x n ), calculate x through Bi-LSTM to obtain the intermediate representation data Q, K and V, and then obtain the final representation data q of x through the scaled dot product attention mechanism and maximum pooling, that is, the text encoding result, that is, the features obtained by text extraction.

[0073] In a feasible embodiment, conceptual encoding of voice information includes:

[0074] Perform entity recognition on the text recognition results to obtain a set of entity concepts;

[0075] Determine the initial attention weight of each entity concept in the entity concept set to the text according to the C-ST attention mechanism;

[0076] Determine the feature selection weight of each entity concept in the entity concept set according to the C-CS attention mechanism;

[0077] According to the initial attention weight and feature selection weight, determine the updated attention weight of each entity concept in the entity concept set to the text;

[0078] Based on the updated attention weights and entity concept set, the concept encoding result is determined.

[0079] Among them, the entity concept set is generated through the text recognition results. The specific generation method is: first find the entity set from the short text sequence, and then input the entity information into the pre-set knowledge graph to obtain the entity concept set corresponding to the entity set. For example, in the short text "On the Creation of Chinese Historical Dramas", "China" and "Historical Dramas" are entities, and the generated entity concept set is "country", "history", "ancient country", etc.

[0080] Specifically, the entity concept set obtained according to the text recognition result is a set c of size m, expressed as c = (c1, c2, ..., c m ), where c i is the i-th concept vector. Two attention mechanisms, C-ST (Concept towards Short Text) and C-CS (Concept towards Concept Set), are used to focus more on important concepts and improve the accuracy of feature expression in entity concept sets.

[0081] The C-ST attention mechanism is used to calculate the semantic relevance between text and the corresponding concept set, reducing the adverse effects of incorrect concepts introduced due to entity ambiguity or KB (Knowledge Retrieve) noise. The C-ST attention mechanism calculates the initial attention weight of each entity concept in the entity concept set to the text through the following formula:

[0082]

[0083] Among them, α i Represents the i-th entity concept c i The initial attention weight for the text, α i The larger the value, the more relevant the corresponding entity concept is to the semantics of the short text. f(·) is a nonlinear activation function, such as the hyperbolic tangent transform tanh. The softmax function is used to normalize the initial attention weight of each entity concept. W1 is the weight matrix, w1 is the weight vector, and b1 is the offset, which can be set according to empirical values ​​and is not limited here.

[0084] The C-CS attention mechanism is used to calculate the importance of each entity concept in the entity concept set. The C-CS attention mechanism calculates the feature selection weight of each entity concept in the entity concept set through the following formula:

[0085]

[0086] Among them, β i Represents the i-th entity concept c iThe attention weight in the entire entity concept set c, that is, the feature selection weight, W2 is the weight matrix, w2 is the weight vector, b2 is the offset, which can be set according to the empirical value and is not limited here. The effect of the C-CS attention mechanism is similar to that of feature selection, which realizes assigning larger weights to important entity concepts and smaller weights to ordinary entity concepts.

[0087] The updated attention weight of each entity concept in the entity concept set on the text can be obtained by adding α i and β i Combine them and the combination formula is as follows:

[0088]

[0089] Among them, α i ′ represents the final updated attention weight of the i-th entity concept for the text, and γ∈{0,1} is used to adjust α i and β i The soft switch of the weights of the two, k∈[1,m], m is the length of the entity concept set, and there are two ways to set the value of γ: treat γ as a hyperparameter and achieve the optimal value through empirical adjustment; γ participates in the training of the neural network and automatically adjusts. In the embodiment of the present invention, the calculation method of γ is to participate in the training of the neural network and calculate it through the following formula:

[0090] γ=σ(w T [α; β]+b);

[0091] Among them, w and b are the parameters that need to be learned in the training of the neural network, w is the weight parameter, b is the bias parameter, σ is the sigmoid function, α and β are the m α i , β i The matrix composed of.

[0092] Finally, according to the updated attention weight and the entity concept set, the concept encoding result is determined, that is, the weighted sum of the entity concept set and the updated attention weight is calculated to obtain the semantic vector p representing the concept, that is, the concept encoding result. The calculation formula of the concept encoding result p is as follows:

[0093]

[0094] Among them, c i represents the i-th entity concept in the entity concept set, α i ′ Represents the final updated attention weight of the i-th entity concept for the text.

[0095] In a feasible embodiment, after performing entity recognition on the text recognition result to obtain an entity concept set, the method further includes:

[0096] The concept embedding is performed on the entity concept set to obtain a concept embedding vector; wherein the concept embedding is performed based on the pre-trained embedding vector.

[0097] In order to improve the processing efficiency of entity concept sets and increase the dimensional information of concept features, before performing entity recognition on text recognition results to obtain entity concept sets, the entity concept sets are processed.

[0098] After performing entity recognition on the text recognition results, an entity concept set of size m is obtained. In order to improve the extraction dimension of concept features, the entity concept set is converted into a vector representation. Specifically, concept embedding is performed on the entity concept set to obtain a concept embedding vector. Subsequently, the C-ST attention mechanism and the C-CS attention mechanism are used based on the concept embedding vector to determine the concept encoding result.

[0099] In a feasible embodiment, obtaining a speech encoding result according to a text encoding result and a concept encoding result includes:

[0100] The text encoding result and the concept encoding result are combined, and the speech encoding result is determined according to the combined result.

[0101] Specifically, after obtaining the text encoding result q and the concept encoding result p, the text encoding result and the concept encoding result are merged to generate a new code e2, which is the speech encoding result. Exemplarily, e2 = [p; q], that is, the text encoding result is spliced ​​after the concept encoding result. Optionally, the text encoding result and the concept encoding result can be spliced ​​and merged after dimensional consistency processing.

[0102] Among them, step 102 and step 103 can be executed in parallel, and the execution order of these two steps is not limited. In addition, text coding and concept coding can also be executed in parallel, and the execution order is not limited.

[0103] Step 104: fuse the image coding result and the speech coding result, and identify the emotion of the person to be identified based on the fused coding result.

[0104] Specifically, after obtaining the image coding result e1 and the speech coding result e2, e1 and e2 are merged to generate a new code e, which is the fusion coding result. The fusion coding result includes both the information of the key points of the face in the face image and the characteristic information of the emotion in the speech information. Recognizing emotions through the fusion coding result is conducive to improving the accuracy of emotion recognition.

[0105] Exemplarily, e=[e1; e2], that is, the speech coding result is spliced ​​after the image coding result. Optionally, the image coding result and the speech coding result can be spliced ​​and merged after dimension consistency processing.

[0106] After obtaining the fusion coding result, based on the pre-trained emotion recognition model, the fusion coding result is input into the emotion recognition model to obtain the prediction probability of each emotion classification result, and the emotion of the person to be recognized is determined according to the comparison result of the prediction probability. Exemplarily, the emotion recognition model is obtained by training with pre-collected image and voice samples, and the training method of the model is not limited in the embodiment of the present invention.

[0107] In a feasible embodiment, step 104 includes:

[0108] Combining the image coding result and the speech coding result, and determining a fusion coding result according to the combined result;

[0109] The fusion coding result is processed using a preset softmax regression function to obtain a classification result, and the emotion of the person to be identified is identified based on the classification result.

[0110] The pre-trained emotion recognition model may use a pre-set softmax regression function. Specifically, the pre-set softmax regression function is used to calculate the fusion coding result to obtain the regression value of each emotion classification result, and then identify the emotion of the person to be identified. Exemplarily, the emotion classification result includes at least happiness, calmness, sadness, surprise, fear, disgust and anger.

[0111] The embodiment of the present invention encodes facial expressions and text data, fuses the features of facial expression encoding and text data encoding, and uses a softmax classifier to perform emotion classification. Facial expression encoding includes the processes of face detection, face alignment, and face feature extraction. In the process of face feature extraction, after performing convolutional neural network calculation on the face, covariance pooling is used for calculation to obtain facial expression encoding; text data encoding includes obtaining text recognition results and entity concept sets at the input layer, performing text encoding on the text recognition results, performing concept encoding on the entity concept set, and fusing text encoding results and concept encoding results.

[0112] The embodiment of the present invention is based on encoding the facial image of the person to be identified to obtain an image encoding result, and encoding the voice information output by the person to be identified to obtain a voice encoding result, wherein the voice encoding result is obtained based on text encoding and concept encoding of the voice information to improve the accuracy of the voice encoding result; then, emotion recognition of the person to be identified is performed based on the fusion encoding result of the image encoding result and the voice encoding result to improve the accuracy of emotion recognition of the person.

[0113] Embodiment 2

[0114] Figure 2 is a schematic diagram of the structure of the emotion recognition device in the second embodiment of the present invention. This embodiment is applicable to the situation of recognizing the real-time emotions of a person. Figure 2 As shown, the device comprises:

[0115] The information collection module 210 is used to collect the facial image of the person to be identified and the voice information output by the person to be identified associated with the collection timing of the facial image;

[0116] The image processing module 220 is used to perform face detection on the face image and obtain an image encoding result according to the face detection result;

[0117] A speech processing module 230, configured to perform text coding and concept coding on the speech information, and obtain a speech coding result according to the text coding result and the concept coding result;

[0118] The emotion recognition module 240 is used to fuse the image coding result and the speech coding result, and recognize the emotion of the person to be recognized according to the fused coding result.

[0119] The embodiment of the present invention is based on encoding the facial image of the person to be identified to obtain an image encoding result, and encoding the voice information output by the person to be identified to obtain a voice encoding result, wherein the voice encoding result is obtained based on text encoding and concept encoding of the voice information to improve the accuracy of the voice encoding result; then, emotion recognition of the person to be identified is performed based on the fusion encoding result of the image encoding result and the voice encoding result to improve the accuracy of emotion recognition of the person.

[0120] Optionally, the image processing module includes a face detection unit, which is used to:

[0121] Face detection is performed on face images based on a multi-task learning model; wherein the multi-tasks include at least a face classification task, a face frame position regression task, and a face key point regression task.

[0122] Optionally, the multi-task loss function during training of the multi-task learning model is:

[0123]

[0124] Among them, L represents the final loss value in the multi-task learning model, represents the loss value of the face classification task for the i-th face image sample, represents the loss value of the face box regression task for the i-th face image sample, represents the loss value of the face key point regression task of the i-th face image sample, λ1 and λ2 are preset parameter values, and p i represents the face prediction probability in the i-th face image sample, represents the true value of the face annotation in the i-th face image sample, t i represents the predicted position information of the face frame in the i-th face image sample, represents the face frame annotation position information in the i-th face image sample, l i represents the predicted position information of the facial key points in the i-th face image sample, Represents the location information of facial key points in the i-th face image sample.

[0125] Optionally, the image processing module includes an image encoding unit, which is used to:

[0126] Performing feature extraction on the face detection result based on a convolutional neural network to obtain a feature extraction result;

[0127] Covariance pooling is used to calculate the feature extraction result to obtain an image encoding result.

[0128] Optionally, the calculation formula for covariance pooling is as follows:

[0129]

[0130] Among them, C represents the calculation result of covariance pooling, n represents the feature dimension in the feature extraction result, and f i ∈(f1,f2,…,f n ), f1,f2,…,f n Represents the feature data of n dimensions in the feature extraction result,

[0131]

[0132] Optionally, the device further includes a speech recognition module, which is used to:

[0133] Before performing text encoding and concept encoding on the voice information, voice recognition is performed on the voice information to obtain a text recognition result.

[0134] Optionally, the speech processing module includes a text encoding unit for:

[0135] Processing the text recognition result using Bi-LSTM to obtain an intermediate vector representation of the text recognition result;

[0136] Processing the intermediate vector representation using a scaled dot product attention mechanism to obtain a matrix representation;

[0137] The matrix representation is processed using maximum pooling to obtain a sentence representation of the text recognition result as a text encoding result.

[0138] Optionally, the text encoding unit also includes a text embedding subunit, which is used to:

[0139] Before using Bi-LSTM to process the text recognition result, character embedding and / or word embedding are performed on the text recognition result to obtain a text embedding vector; wherein the character embedding is performed based on a convolutional neural network, and the word embedding is performed based on a pre-trained embedding vector;

[0140] Correspondingly, the text encoding unit includes:

[0141] The text embedding vector is processed using Bi-LSTM.

[0142] Optionally, the speech processing module includes a concept encoding unit for:

[0143] Performing entity recognition on the text recognition result to obtain an entity concept set;

[0144] Determine the initial attention weight of each entity concept in the entity concept set to the text according to the C-ST attention mechanism;

[0145] Determine the feature selection weight of each entity concept in the entity concept set according to the C-CS attention mechanism;

[0146] Determining an updated attention weight of each entity concept in the entity concept set to the text according to the initial attention weight and the feature selection weight;

[0147] A concept encoding result is determined based on the updated attention weight and the entity concept set.

[0148] Optionally, the concept encoding unit includes a concept embedding subunit, which is used to:

[0149] After performing entity recognition on the text recognition result to obtain an entity concept set, concept embedding is performed on the entity concept set to obtain a concept embedding vector; wherein the concept embedding is performed based on a pre-trained embedding vector.

[0150] Optionally, the speech processing module includes a coding merging unit, which is used to:

[0151] The text encoding result and the concept encoding result are combined, and a speech encoding result is determined according to the combined result.

[0152] Optional, emotion recognition module, specifically used for:

[0153] Combining the image encoding result and the speech encoding result, and determining a fusion encoding result according to the combined result;

[0154] The fusion coding result is processed using a preset softmax regression function to obtain a classification result, and the emotion of the person to be identified is identified according to the classification result.

[0155] The emotion recognition device provided in the embodiment of the present invention can execute the emotion recognition method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects for executing the emotion recognition method.

[0156] Embodiment 3

[0157] Figure 3 It is a structural schematic diagram of an electronic device provided in Embodiment 3 of the present invention. Figure 3 A block diagram of an exemplary electronic device 12 suitable for use in implementing embodiments of the present invention is shown. Figure 3 The electronic device 12 shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present invention.

[0158] like Figure 3 As shown, the electronic device 12 is in the form of a general purpose computing device. The components of the electronic device 12 may include, but are not limited to: one or more processors or processing units 16, system memory 28, and a bus 18 connecting various system components (including the system memory 28 and the processing unit 16).

[0159] Bus 18 represents one or more of several types of bus structures, including a storage device bus or storage device controller, a peripheral bus, an accelerated graphics port, a processor or a local bus using any of a variety of bus architectures. By way of example, these architectures include, but are not limited to, an Industry Standard Architecture (ISA) bus, a Micro Channel Architecture (MAC) bus, an Enhanced ISA bus, a Video Electronics Standards Association (VESA) local bus, and a Peripheral Component Interconnect (PCI) bus.

[0160] The electronic device 12 typically includes a variety of computer system readable media. These media can be any available media that can be accessed by the electronic device 12, including volatile and non-volatile media, removable and non-removable media.

[0161] The system storage device 28 may include computer system readable media in the form of volatile storage devices, such as random access memory devices (RAM) 30 and / or cache memory devices 32. The electronic device 12 may further include other removable / non-removable, volatile / non-volatile computer system storage media. By way of example only, the storage system 34 may be used to read and write non-removable, non-volatile magnetic media ( Figure 3 not shown, usually called a "hard drive"). Although Figure 3 Not shown in the figure, a disk drive for reading and writing to a removable non-volatile disk (e.g., a "floppy disk"), and an optical disk drive for reading and writing to a removable non-volatile optical disk (e.g., a CD-ROM, a DVD-ROM, or other optical media) may be provided. In these cases, each drive may be connected to the bus 18 via one or more data medium interfaces. The storage device 28 may include at least one program product having a set (e.g., at least one) of program modules that are configured to perform the functions of the various embodiments of the present invention.

[0162] A program / utility 40 having a set (at least one) of program modules 42 may be stored, for example, in the storage device 28, such program modules 42 including, but not limited to, an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment. The program modules 42 generally perform the functions and / or methods of the embodiments described herein.

[0163] The electronic device 12 may also communicate with one or more external devices 14 (e.g., keyboards, pointing devices, displays 24, etc.), one or more devices that enable a user to interact with the device 12, and / or any device that enables the device 12 to communicate with one or more other computing devices (e.g., network cards, modems, etc.). Such communication may be performed via an input / output (I / O) interface 22. Furthermore, the electronic device 12 may also communicate with one or more networks (e.g., local area networks (LANs), wide area networks (WANs), and / or public networks, such as the Internet) via a network adapter 20. Figure 3 As shown, the network adapter 20 communicates with other modules of the electronic device 12 via the bus 18. It should be understood that although Figure 3 Not shown, other hardware and / or software modules may be used in conjunction with the electronic device 12, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0164] The processing unit 16 executes various functional applications and data processing by running the programs stored in the system storage device 28, for example, implementing the emotion recognition method provided by the embodiment of the present invention, including:

[0165] Collecting a facial image of a person to be identified, and voice information output by the person to be identified associated with a timing of collecting the facial image;

[0166] Performing face detection on the face image, and obtaining an image encoding result according to the face detection result;

[0167] Performing text coding and concept coding on the speech information, and obtaining a speech coding result according to the text coding result and the concept coding result;

[0168] The image encoding result and the speech encoding result are fused, and the emotion of the person to be identified is identified according to the fused encoding result.

[0169] Embodiment 4

[0170] Embodiment 4 of the present invention further provides a computer-readable storage medium, on which a computer program is stored. When the program is executed by a processor, the emotion recognition method provided in the embodiment of the present invention is implemented, including:

[0171] Collecting a facial image of a person to be identified, and voice information output by the person to be identified associated with a timing of collecting the facial image;

[0172] Performing face detection on the face image, and obtaining an image encoding result according to the face detection result;

[0173] Performing text coding and concept coding on the speech information, and obtaining a speech coding result according to the text coding result and the concept coding result;

[0174] The image encoding result and the speech encoding result are fused, and the emotion of the person to be identified is identified according to the fused encoding result.

[0175] The computer storage medium of the embodiment of the present invention can adopt any combination of one or more computer-readable media. The computer-readable medium can be a computer-readable signal medium or a computer-readable storage medium. The computer-readable storage medium can be, for example, but not limited to, a system, device or device of electricity, magnetism, light, electromagnetic, infrared, or semiconductor, or any combination of the above. More specific examples (non-exhaustive list) of computer-readable storage media include: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this document, a computer-readable storage medium can be any tangible medium containing or storing a program, which can be used by an instruction execution system, a device or a device or used in combination with it.

[0176] Computer-readable signal media may include data signals propagated in baseband or as part of a carrier wave, which carry computer-readable program code. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Computer-readable signal media may also be any computer-readable medium other than a computer-readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0177] The program code embodied on the computer readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.

[0178] Computer program code for performing the operation of the present invention may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0179] Note that the above are only preferred embodiments of the present invention and the technical principles used. Those skilled in the art will understand that the present invention is not limited to the specific embodiments described herein, and that various obvious changes, readjustments and substitutions can be made by those skilled in the art without departing from the scope of protection of the present invention. Therefore, although the present invention has been described in more detail through the above embodiments, the present invention is not limited to the above embodiments, and may include more other equivalent embodiments without departing from the concept of the present invention, and the scope of the present invention is determined by the scope of the appended claims.

Claims

1. An emotion recognition method, characterized in that: include: Collecting a facial image of a person to be identified, and voice information output by the person to be identified associated with a timing of collecting the facial image; Performing face detection on the face image, and obtaining an image coding result according to the face detection result, including: performing feature extraction on the face detection result based on a convolutional neural network to obtain a feature extraction result; and calculating the feature extraction result using covariance pooling to obtain an image coding result; Performing speech recognition on the speech information to obtain a text recognition result; The speech information is text-encoded, including: using Bi-LSTM to process the text recognition result to obtain an intermediate vector representation of the text recognition result; using a scaled dot product attention mechanism to process the intermediate vector representation to obtain a matrix representation; using maximum pooling to process the matrix representation to obtain a sentence representation of the text recognition result as a text encoding result; The speech information is conceptually encoded, including: performing entity recognition on the text recognition result to obtain an entity concept set; determining an initial attention weight of each entity concept in the entity concept set to the text according to a C-ST attention mechanism; determining a feature selection weight of each entity concept in the entity concept set according to a C-CS attention mechanism; determining an updated attention weight of each entity concept in the entity concept set to the text according to the initial attention weight and the feature selection weight; and determining a concept encoding result according to the updated attention weight and the entity concept set; Obtaining a speech encoding result according to the text encoding result and the concept encoding result, including: merging the text encoding result and the concept encoding result, and determining the speech encoding result according to the merging result; The image encoding result and the speech encoding result are fused, and the emotion of the person to be identified is identified according to the fused encoding result.

2. The method according to claim 1, characterized in that The performing face detection on the face image comprises: Performing face detection on the face image based on a multi-task learning model; wherein the multi-tasks include at least a face classification task, a face frame position regression task, and a face key point regression task.

3. The method according to claim 2, characterized in that The multi-task loss function during the training of the multi-task learning model is: Among them, L represents the final loss value in the multi-task learning model, represents the loss value of the face classification task for the i-th face image sample, represents the loss value of the face box regression task for the i-th face image sample, represents the loss value of the face key point regression task of the i-th face image sample, λ1 and λ2 are preset parameter values, and p i represents the face prediction probability in the i-th face image sample, represents the true value of the face annotation in the i-th face image sample, t i represents the predicted position information of the face frame in the i-th face image sample, represents the face frame annotation position information in the i-th face image sample, l i represents the predicted position information of the facial key points in the i-th face image sample, Represents the location information of facial key points in the i-th face image sample.

4. The method according to claim 1, characterized in that: The calculation formula of the covariance pooling is as follows: Among them, C represents the calculation result of covariance pooling, n represents the feature dimension in the feature extraction result, and f i ∈(f1,f2,…,f n ), f1,f2,…,f n Represents the feature data of n dimensions in the feature extraction result, 5. The method according to claim 1, characterized in that Before using Bi-LSTM to process the text recognition result, the method further includes: Performing character embedding and / or word embedding on the text recognition result to obtain a text embedding vector; wherein the character embedding is performed based on a convolutional neural network, and the word embedding is performed based on a pre-trained embedding vector; Accordingly, the text recognition result is processed using Bi-LSTM, including: The text embedding vector is processed using the Bi-LSTM.

6. The method according to claim 1, characterized in that After performing entity recognition on the text recognition result to obtain an entity concept set, the method further includes: The entity concept set is conceptually embedded to obtain a concept embedding vector; wherein the concept embedding is performed based on a pre-trained embedding vector.

7. The method according to claim 1, characterized in that The step of fusing the image encoding result and the speech encoding result, and identifying the emotion of the person to be identified according to the fused encoding result, comprises: Combining the image encoding result and the speech encoding result, and determining a fusion encoding result according to the combined result; The fusion coding result is processed using a preset softmax regression function to obtain a classification result, and the emotion of the person to be identified is identified according to the classification result.

8. An emotion recognition device, characterized in that: include: An information collection module, used to collect a facial image of a person to be identified, and voice information output by the person to be identified associated with a collection timing of the facial image; An image processing module is used to perform face detection on the face image and obtain an image encoding result according to the face detection result; The image processing module includes: an image encoding unit, which is used to perform feature extraction on the face detection result based on a convolutional neural network to obtain a feature extraction result; and to calculate the feature extraction result using covariance pooling to obtain an image encoding result; A speech recognition module, used to perform speech recognition on the speech information to obtain a text recognition result; A speech processing module, used for performing text coding and concept coding on the speech information, and obtaining a speech coding result according to the text coding result and the concept coding result; The speech processing module includes: a text encoding unit, a concept encoding unit and a encoding merging unit; The text encoding unit is used to process the text recognition result using Bi-LSTM to obtain an intermediate vector representation of the text recognition result; process the intermediate vector representation using a scaled dot product attention mechanism to obtain a matrix representation; and process the matrix representation using maximum pooling to obtain a sentence representation of the text recognition result as the text encoding result; The concept encoding unit is used to perform entity recognition on the text recognition result to obtain an entity concept set; determine the initial attention weight of each entity concept in the entity concept set to the text according to the C-ST attention mechanism; determine the feature selection weight of each entity concept in the entity concept set according to the C-CS attention mechanism; determine the updated attention weight of each entity concept in the entity concept set to the text according to the initial attention weight and the feature selection weight; determine the concept encoding result according to the updated attention weight and the entity concept set; The coding merging unit is used to merge the text coding result and the concept coding result, and determine the speech coding result according to the merging result; The emotion recognition module is used to fuse the image encoding result and the speech encoding result, and recognize the emotion of the person to be recognized according to the fused encoding result.

9. An electronic device, characterized in that: include: one or more processors; a storage device for storing one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors implement the emotion recognition method as described in any one of claims 1-7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the emotion recognition method as described in any one of claims 1 to 7 is implemented.

Images