Emotion recognition method and device

Abstract

The invention provides an emotion recognition method and device, and relates to the technical field of data processing. The method comprises the following steps: performing feature extraction according to acquired multi-modal data for emotion recognition to obtain various feature vectors; performing feature fusion on the multiple feature vectors to obtain fused features, and performing sentiment classification on the fused features by adopting a first preset classifier to obtain a sentiment recognition result; or adopting a second preset classifier to carry out sentiment classification on the multiple feature vectors to obtain a classification result corresponding to each feature vector, and fusing the classification results corresponding to all the feature vectors to obtain a sentiment recognition result. According to the method, emotion recognition is performed based on the multi-modal data, comprehensive utilization of various types of data is realized by performing feature extraction, fusion and classification on the multi-modal data, the emotion of the corresponding individual can be considered more comprehensively, the limitation of single-modal data is made up, and the accuracy and comprehensiveness of emotion recognition of the corresponding individual are improved.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to an emotion recognition method and device. Background Art

[0002] In the field of computer science, especially in the application of artificial intelligence (AI) technology, emotion recognition and behavior analysis have gradually become an important research direction, especially emotion recognition technology based on facial expressions, which uses computer vision and machine learning algorithms to analyze human expressions and then identify their emotional states.

[0003] However, current emotion recognition technology still has some defects. For example, most existing recognition technologies only rely on a single modality of data input, usually image or video data, and ignore other important information sources that may affect emotion judgment, such as language text. This single modality reliance limits the effectiveness of current recognition technology in complex emotion judgment scenarios and affects the accuracy of facial expression recognition. Summary of the invention

[0004] The problem solved by the present invention is: how to improve the accuracy of emotion recognition.

[0005] To solve the above problems, the present invention provides an emotion recognition method, comprising:

[0006] Perform feature extraction based on the acquired multimodal data for emotion recognition to obtain multiple feature vectors;

[0007] Perform feature fusion on the multiple feature vectors to obtain fused features, and use the first preset classifier to perform emotion classification on the fused features to obtain emotion recognition results; or use the second preset classifier to perform emotion classification on the multiple feature vectors to obtain classification results corresponding to each feature vector, and fuse the classification results corresponding to all the feature vectors to obtain the emotion recognition result.

[0008] Optionally, the multimodal data includes a plurality of unimodal data for emotion recognition;

[0009] The feature extraction is performed based on the acquired multimodal data for emotion recognition to obtain a variety of feature vectors including:

[0010] Feature extraction is performed respectively according to the acquired multiple unimodal data for emotion recognition to obtain the feature vector corresponding to each type of unimodal data.

[0011] Optionally, the multimodal data includes video data having image information and corresponding audio information;

[0012] The feature extraction is performed based on the acquired multimodal data for emotion recognition to obtain a variety of feature vectors including:

[0013] Performing video frame extraction and voice recognition on the video data of the same individual to obtain the image information and the audio information;

[0014] Feature extraction is performed on the image information to obtain a first feature vector; feature extraction is performed on the audio information to obtain a second feature vector.

[0015] Optionally, extracting features from the image information to obtain a first feature vector includes:

[0016] Using a first preset model to extract features from the image information to obtain a first feature vector;

[0017] The extracting features of the audio information to obtain a second feature vector comprises:

[0018] A second preset model is used to extract features from the audio information to obtain a second feature vector.

[0019] Optionally, the performing feature fusion on the multiple feature vectors to obtain fused features, and using a first preset classifier to perform emotion classification on the fused features to obtain an emotion recognition result includes:

[0020] Concatenating and fusing the first feature vector and the second feature vector to obtain the fused feature;

[0021] The first preset classifier is used to perform sentiment classification on the fusion feature to obtain the sentiment recognition result.

[0022] Optionally, the performing feature fusion on the multiple feature vectors to obtain fused features, and using a first preset classifier to perform emotion classification on the fused features to obtain an emotion recognition result includes:

[0023] Fusing the first feature vector and the second feature vector at an attention layer to obtain the fused feature;

[0024] The first preset classifier is used to perform sentiment classification on the fusion feature to obtain the sentiment recognition result.

[0025] Optionally, the second preset classifier includes a second expression feature classifier and a second voice or text classifier;

[0026] The emotion recognition result obtained by using the second preset classifier to perform emotion classification on the plurality of feature vectors to obtain the classification results corresponding to each feature vector, and fusing the classification results corresponding to all the feature vectors includes:

[0027] Using the second expression feature classifier to perform emotion classification on the first feature vector to obtain a second expression emotion classification result; using the second speech or text classifier to perform emotion classification on the second feature vector to obtain a second speech or text emotion classification result;

[0028] The second expression emotion classification result and the second speech or text emotion classification result are fused to obtain the emotion recognition result.

[0029] Optionally, the first feature vector includes an expression feature vector and an image hidden state feature vector, and the second feature vector includes a speech or text feature vector and a speech or text hidden state feature vector;

[0030] The step of fusing the multiple feature vectors to obtain fused features, and using a first preset classifier to perform emotion classification on the fused features to obtain an emotion recognition result includes:

[0031] Fusing the image hidden state feature vector and the speech or text hidden state feature vector to obtain a fused hidden state feature vector;

[0032] The expression feature vector is concatenated and fused with the fused hidden state feature vector to obtain a third feature vector; the speech or text feature vector is concatenated and fused with the fused hidden state feature vector to obtain a fourth feature vector;

[0033] Using the first preset classifier to perform emotion classification on the third feature vector to obtain a first emotion recognition result; using the first preset classifier to perform emotion classification on the fourth feature vector to obtain a second emotion recognition result;

[0034] The first emotion recognition result and the second emotion recognition result are fused to obtain the emotion recognition result.

[0035] Optionally, the first feature vector includes an expression feature vector and an image hidden state feature vector, the second feature vector includes a speech or text feature vector and a speech or text hidden state feature vector; the first preset classifier includes a first expression feature classifier and a first speech or text classifier;

[0036] The step of fusing the multiple feature vectors to obtain fused features, and using a first preset classifier to perform emotion classification on the fused features to obtain an emotion recognition result includes:

[0037] Input the image hidden state feature vector and the speech or text hidden state feature vector into a preset multi-head attention mechanism, and output the speech or text to image attention output and the image to speech or text attention output;

[0038] The expression feature vector is merged with the attention output from the speech or text to the image to obtain a fifth feature vector; the speech or text feature vector is merged with the attention output from the image to the speech or text to obtain a sixth feature vector;

[0039] Using the first expression feature classifier to perform emotion classification on the fifth feature vector to obtain a first expression emotion classification result; using the first speech or text classifier to perform emotion classification on the sixth feature vector to obtain a first speech or text emotion classification result;

[0040] The first expression emotion classification result and the first speech or text emotion classification result are fused to obtain the emotion recognition result.

[0041] In order to solve the above problems, the present invention further provides an emotion recognition device, comprising:

[0042] A feature extraction unit, used for extracting features based on the acquired multimodal data for emotion recognition to obtain multiple feature vectors;

[0043] The emotion recognition unit is used to perform feature fusion on the multiple feature vectors to obtain fused features, and use the first preset classifier to perform emotion classification on the fused features to obtain emotion recognition results; or, used to perform emotion classification on the multiple feature vectors using the second preset classifier to obtain classification results corresponding to each feature vector, and fuse the classification results corresponding to all the feature vectors to obtain the emotion recognition results.

[0044] Compared with the prior art, the present invention has the following beneficial effects: the present invention can be used to realize the recognition of human emotions, provide more accurate emotional health assessment for the corresponding individuals, etc., so as to timely discover the emotional problems that may exist in the corresponding individuals, such as depression, emotional disorders, etc., and can also play an auxiliary role in the prevention and treatment of emotional problems of the corresponding individuals. Specifically, different from the use of a single type of unimodal data for emotion recognition, the present invention is based on multimodal data for emotion recognition, which can more comprehensively consider the emotions of the corresponding individuals and improve the accuracy and comprehensiveness of the emotion recognition of the corresponding individuals. By extracting, fusing and classifying features of multimodal data to achieve the comprehensive utilization of multiple types of data, it is convenient to understand the emotional state of the corresponding individuals from multiple dimensions, thereby improving the comprehensiveness and accuracy of the emotion recognition results. Among them, data of different modalities complement and enhance each other, which can improve the representation ability of the corresponding features, make up for the limitations of single modal data, and improve the comprehensiveness and accuracy of emotion recognition results. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] Figure 1 A schematic diagram of a flow chart of an emotion recognition method in an embodiment of the present invention;

[0046] Figure 2 A processing flow chart of the emotion recognition method in an embodiment of the present invention;

[0047] Figure 3 A processing flow chart of the emotion recognition method in an embodiment of the present invention;

[0048] Figure 4 A processing flow chart of the emotion recognition method in an embodiment of the present invention;

[0049] Figure 5 A processing flow chart of the emotion recognition method in an embodiment of the present invention;

[0050] Figure 6 4 is a structural block diagram of an emotion recognition device in an embodiment of the present invention. DETAILED DESCRIPTION

[0051] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, specific embodiments of the present invention are described in detail below with reference to the accompanying drawings.

[0052] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the numbers used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein.

[0053] Combination Figure 1As shown, an embodiment of the present invention provides an emotion recognition method, comprising the following steps:

[0054] Step 100: extract features based on the acquired multimodal data for emotion recognition to obtain multiple feature vectors.

[0055] Specifically, considering that data that can reflect human emotions are of various types, such as images, audio, text, etc., the method of this embodiment analyzes and identifies the emotions (or moods) of the corresponding individual (person) based on multimodal data for emotion recognition; wherein multimodal data are various types of data that can reflect human emotions (each type corresponds to a unimodal data), such as image data about human expressions, corresponding voice data, etc. In step 100, based on the acquired multimodal data for emotion recognition, feature extraction is performed on these multimodal data to obtain multiple feature vectors (or feature representations), that is, feature vectors corresponding to each unimodal data of the multimodal data are obtained, which are used for subsequent corresponding processing to obtain the emotion recognition results of the corresponding individuals.

[0056] Step 200: perform feature fusion on multiple feature vectors to obtain fused features, and use the first preset classifier to perform sentiment classification on the fused features to obtain sentiment recognition results; or, use the second preset classifier to perform sentiment classification on multiple feature vectors to obtain classification results corresponding to each feature vector, and fuse the classification results corresponding to all feature vectors to obtain sentiment recognition results.

[0057] Specifically, in order to ensure that multimodal data for emotion recognition can be combined to achieve comprehensive and accurate recognition and analysis of the emotions of the corresponding individuals, in step 200, it is possible to first perform feature fusion on multiple feature vectors (such as fusing corresponding feature vectors by weighted summation, splicing, attention mechanism, etc.) to obtain fused features, and then use a preset corresponding classifier (recorded as the first preset classifier) ​​to perform emotion classification on the fused features, such as classifying the fused features according to emotion types (such as), thereby obtaining the emotion recognition results of the corresponding individuals. In this way, by first fusing the features of multiple different modalities, the information of different modalities can be complemented and enhanced with each other, the representation ability of the features can be improved, thereby improving the accuracy and comprehensiveness of emotion recognition, and by performing classification and recognition after feature fusion, the computing resources required for classification and recognition can be reduced, the computing cost can be reduced, and the computing efficiency can be improved. Alternatively, a preset corresponding classifier (recorded as the second preset classifier) ​​may be used to perform sentiment classification on multiple feature vectors to obtain the classification results corresponding to each feature vector, and then the classification results corresponding to all feature vectors may be integrated to obtain the corresponding individual emotion recognition results. In this way, by classifying multiple feature vectors separately, the classification results of different modalities may be obtained, so as to comprehensively consider the contribution of each modality to emotion recognition, etc., make full use of the information of each modality, and improve the accuracy and comprehensiveness of emotion recognition.

[0058] In this way, the method of this embodiment is used to realize the recognition of human emotions, provide more accurate emotional health assessment for the corresponding individuals, etc., so as to timely discover the possible (or potential) emotional problems of the corresponding individuals, such as depression, emotional disorders, etc., and can also play an auxiliary role in the prevention and treatment of emotional problems of the corresponding individuals. Specifically, different from the use of a single type of unimodal data for emotion recognition, the method of this embodiment is based on multimodal data for emotion recognition, which can more comprehensively consider the emotions of the corresponding individuals and improve the accuracy and comprehensiveness of the emotion recognition of the corresponding individuals. By extracting, fusing and classifying features of multimodal data to achieve the comprehensive utilization of multiple types of data, it is convenient to understand the emotional state of the corresponding individuals from multiple dimensions (angles), thereby improving the comprehensiveness and accuracy of the emotion recognition results. Among them, data of different modalities complement and enhance each other, which can improve the representation ability of the corresponding features, make up for the limitations of single modal data, and improve the comprehensiveness and accuracy of emotion recognition results.

[0059] Optionally, the multimodal data includes a plurality of unimodal data for emotion recognition;

[0060] Based on the acquired multimodal data, feature extraction is performed to obtain a variety of feature vectors including:

[0061] Feature extraction is performed on each of the acquired multiple unimodal data for emotion recognition to obtain a feature vector corresponding to each unimodal data.

[0062] Specifically, multimodal data includes a variety of unimodal data (such as images, audio, text and other types of data) used for emotion recognition. Different types of unimodal data have different capabilities and information to represent individual emotions. By extracting features from each type of unimodal data separately, the emotional information unique to each data type is retained and highlighted, so as to facilitate a more comprehensive understanding (analysis, evaluation) of the emotional state of the corresponding individual. Moreover, different unimodal data come from different perception channels (or acquisition methods), and contain multiple aspects and dimensions of individual emotional expression. Extracting the feature vectors of each type of unimodal data separately can make full use of this information, so that the emotion recognition of the method of this embodiment can more comprehensively capture the emotional characteristics of individuals and improve recognition accuracy.

[0063] Optionally, the multimodal data includes video data having image information and corresponding audio information.

[0064] The multimodal data used in the method of this embodiment includes video data having image information and corresponding audio information about the same individual. Specifically, the image information includes images of facial expressions of the corresponding individual, and the audio information includes audio of the voice of the corresponding individual. Exemplarily, the video data may be recorded video data including facial expression images of the corresponding individual and corresponding voice audio, such as a recorded video of the corresponding individual speaking.

[0065] Step 100 includes:

[0066] Video frame extraction and speech recognition are performed on video data about the same individual to obtain image information and audio information.

[0067] Specifically, by performing video frame extraction on the video data about the same individual, the corresponding video data is decomposed into a series of image frames, and the image information containing the facial expression information of the individual is (obtained). In this way, by decomposing the video stream into multiple (frame) images, the feature extraction of the video data can be replaced by the feature extraction of the image information, which can reduce the difficulty of the corresponding feature extraction and the required computing resources, and the image information can be performed frame by frame, reducing the need to process a large amount of data at the same time. Moreover, through video frame extraction, redundant information can be reduced, and useless image information can be conveniently removed, such as conveniently removing images without facial expression information of the corresponding individual in the video data, retaining images containing facial expression information of the corresponding individual, reducing the amount of computation of feature extraction, etc.

[0068] By performing speech recognition on video data about the same individual, that is, identifying the speech information of the corresponding individual in the video data, audio information containing the speech information of the corresponding individual is obtained, so as to analyze the emotion of the individual in combination with the speech information of the corresponding individual. The audio information is easy to convert into text or sound features for processing, thereby improving the efficiency of data analysis and processing.

[0069] Feature extraction is performed on the image information to obtain a first feature vector; feature extraction is performed on the audio information to obtain a second feature vector.

[0070] Specifically, feature extraction is performed on the extracted image information and audio information to convert the original data into feature vectors (first feature vectors and second feature vectors) with high abstraction and representation capabilities, which is convenient for subsequent emotion recognition. Feature extraction can convert complex image and audio data into feature vectors in a high-dimensional feature space, thereby reducing the dimension and complexity of the data and improving subsequent computational efficiency and the accuracy of emotion recognition.

[0071] Optionally, performing feature extraction on the image information to obtain a first feature vector includes:

[0072] A first preset model is used to extract features from the image information to obtain a first feature vector.

[0073] Specifically, a preset image processing model (referred to as the first preset model) is used to extract features from the image information to obtain a corresponding feature vector (referred to as the first feature vector). The first preset model is used to process image information, including a convolutional neural network (CNN), a deep convolutional neural network (such as VGG, ResNet) or other image processing models. In some embodiments, the first preset model is trained based on a large-scale facial expression image dataset of different individuals (people), and can extract high-level features in the image, such as texture, shape, edge and other information, thereby reflecting the emotions of the corresponding individual.

[0074] The audio information is subjected to feature extraction, and the second feature vector obtained includes:

[0075] A second preset model is used to extract features from the audio information to obtain a second feature vector.

[0076] Specifically, a preset audio processing model (referred to as a second preset model) is used to extract features from the audio information to obtain a corresponding feature vector (referred to as a second feature vector).

[0077] The second preset model includes an audio feature extraction model, which processes the audio information through the audio feature extraction model to extract key features (to capture important information in the audio data, such as the rhythm, pitch, and voice emotion of the voice), such as spectrum information, voice features, etc., to obtain a second feature vector for reflecting the emotion of the corresponding individual. Alternatively, the second preset model includes a speech recognition model and a semantic processing model (such as a large language model, including BERT, GPT, RoBERTa, XLNet and other models), the speech of the corresponding individual in the audio information is recognized by the speech recognition model, and converted into text information, and the text information is processed by the semantic processing model to obtain a second feature vector for reflecting the emotion of the corresponding individual.

[0078] Optionally, combined Figure 1 , Figure 2 As shown, multiple feature vectors are fused to obtain fused features, and the fused features are sentimentally classified using a first preset classifier, and the sentiment recognition results obtained include:

[0079] The first feature vector and the second feature vector are concatenated and fused to obtain a fused feature.

[0080] Specifically, the first feature vector and the second feature vector are concatenated and fused, such as connecting the two feature vectors in a certain way, which can be achieved by simply connecting the two vectors, or by other more complex ways, such as weighted summation or attention mechanism. In this way, information from different modalities can be integrated into one vector (i.e., fused features) to provide more comprehensive information to support the sentiment classification task.

[0081] The first preset classifier is used to perform sentiment classification on the fused features to obtain the sentiment recognition result.

[0082] Specifically, the fused features are sentimentally classified using a first preset classifier. The first preset classifier may be a pre-trained model specifically used for sentiment classification tasks. By classifying the fused features, the result of sentiment recognition may be obtained, that is, the specific sentiment state of the corresponding individual is classified. In some embodiments, the fused features are sentimentally classified using a first preset classifier to obtain a probability distribution of different recognized emotions, that is, to obtain different recognized emotions and the probabilities of each emotion, and the emotion corresponding to the maximum probability is taken as the sentiment recognition result.

[0083] Exemplarily, the NaiveCat model based on feature splicing (which is a simple model based on weighted fusion) is used for feature fusion. The extracted first feature vector and the second feature vector are fused into a feature fusion vector by splicing, and then the corresponding probability vector is calculated by a fully connected classifier (as the first preset classifier). The fusion process of this method is simple, fusion first and then classification, and the running speed and memory usage are low.

[0084] Optionally, multiple feature vectors are subjected to feature fusion to obtain fused features, and the fused features are subjected to emotion classification using a first preset classifier to obtain an emotion recognition result including:

[0085] The first eigenvector and the second eigenvector are fused at the attention layer to obtain the fused feature.

[0086] The first preset classifier is used to perform sentiment classification on the fused features to obtain the sentiment recognition result.

[0087] Specifically, the first feature vector and the second feature vector are fused at the attention layer, and then the first preset classifier is used to perform sentiment classification on the fused features, thereby obtaining the sentiment recognition result. Among them, by fusing at the attention layer, the first feature vector and the second feature vector can influence each other according to different attention weights, thereby generating a more informative fusion feature, and the fused feature vector can better represent the sentiment information of the corresponding individual. By using the first preset classifier to perform sentiment classification on the fused features, these fused features can be mapped to the sentiment space, and a corresponding probability is assigned to each sentiment category, which helps to determine the sentiment category (i.e., the emotion with the highest probability) that the input data most likely corresponds to, thereby achieving sentiment recognition.

[0088] Exemplarily, the OTE (Output Transformer Encoder) model is used. The first eigenvector and the second eigenvector are fused in the attention layer. The attention layer uses Transformer Encoder to perform self-attention calculation on the input sequence, and assigns different weights to each position according to the calculation results. After fusion, the fused feature vector is sent to the fully connected classifier (as the first preset classifier), and after multiple layers of linear transformation and nonlinear activation, the final classification probability vector is obtained. In this way, the efficient fusion of image and speech or text features is achieved by utilizing a simple attention mechanism; the attention mechanism can adaptively adjust the weights of different modal features, so that the model can more accurately capture key information during classification, thereby improving classification performance and improving emotion recognition effects.

[0089] Optionally, combined Figure 1 , Figure 3As shown, the second preset classifier includes a second expression feature classifier and a second voice or text classifier;

[0090] The second preset classifier is used to perform sentiment classification on multiple feature vectors to obtain the classification results corresponding to each feature vector, and the classification results corresponding to all feature vectors are fused to obtain the sentiment recognition results including:

[0091] Using a second expression feature classifier to perform emotion classification on the first feature vector to obtain a second expression emotion classification result; using a second speech or text classifier to perform emotion classification on the second feature vector to obtain a second speech or text emotion classification result;

[0092] Specifically, for the first feature vector and the second feature vector corresponding to different types of data, different classifiers are used to classify them, and the second expression feature classifier and the second speech or text classifier are respectively adapted to the first feature vector and the second feature vector to ensure the accuracy of the classification (emotion recognition) results, and obtain the corresponding emotion classification results independently on each modality.

[0093] The second expression emotion classification result and the second speech or text emotion classification result are fused to obtain an emotion recognition result.

[0094] Specifically, the second expression emotion classification result and the second speech or text emotion classification result are fused, that is, the corresponding emotion classification results obtained on each modality are fused, and the emotion recognition result is determined based on the fusion result, so as to comprehensively and fully consider the emotion classification results of different modalities and improve the final recognition accuracy and robustness.

[0095] Exemplarily, a NaiveCombine model based on probability vector addition is used. The extracted first eigenvector and the second eigenvector are respectively passed through a fully connected classifier (i.e., the second expression feature classifier and the second speech or text classifier both use a fully connected classifier) ​​to calculate the emotion category probability vector, and then the two probability vectors corresponding to the first eigenvector and the second eigenvector are added and normalized by a SoftMax function (normalized exponential function, or activation function) to obtain the final probability vector. Among them, the dual-path classifier structure and probability fusion strategy of the model enhance the expression ability and classification accuracy. At the same time, the weighted loss function also makes the model more robust when processing unbalanced data sets.

[0096] Optionally, combined Figure 1 , Figure 4 As shown, the first feature vector includes an expression feature vector and an image hidden state feature vector, and the second feature vector includes a speech or text feature vector and a speech or text hidden state feature vector.

[0097] Specifically, the expression feature vector is extracted from image or video data, and mainly captures the facial expression, posture and other emotion-related information, which may include micro-expressions of facial expressions, eye contact, facial muscle activity, etc. The image hidden state feature vector is obtained by processing the image data through a certain deep learning model (such as convolutional neural network, etc.), which may contain abstract information such as texture, shape, object position, etc. in the image.

[0098] The speech or text feature vector is extracted from speech data or text data (which can be obtained by recognizing speech data), and mainly captures the emotional information contained in the speech content. In speech, these features may include acoustic features related to emotion, such as speech rate, intonation, and pitch; in text, these features may include the semantics of words, emotional color, and the frequency of use of emotional vocabulary. The speech or text hidden state feature vector is obtained by processing speech data or text data through a certain deep learning model (such as a recurrent neural network or a Transformer model, etc.), which may contain information such as intonation, speech rate, and stress in speech, or information such as semantics, grammar, and emotional tendency in text.

[0099] In this way, by performing feature extraction in multiple dimensions and comprehensively utilizing multimodal data, the comprehensiveness of emotion recognition can be improved.

[0100] The multiple feature vectors are subjected to feature fusion to obtain fusion features, and the fusion features are subjected to emotion classification using the first preset classifier, and the emotion recognition results obtained include:

[0101] The image hidden state feature vector and the speech or text hidden state feature vector are fused to obtain a fused hidden state feature vector.

[0102] Specifically, the image hidden state feature vector and the speech or text hidden state feature vector capture the emotional information in the image and speech or text data respectively. By fusing them together to obtain a fused hidden state feature vector, we can comprehensively utilize information from different modalities to more comprehensively describe the emotional expression in the input data (i.e., multimodal data). Moreover, the fused hidden state feature vector can enrich the representation ability of the features and improve the model's ability to understand and extract emotion-related information.

[0103] The expression feature vector is concatenated and fused with the fused hidden state feature vector to obtain a third feature vector; the speech or text feature vector is concatenated and fused with the fused hidden state feature vector to obtain a fourth feature vector.

[0104] Specifically, the expression feature vector contains visual information related to emotion, such as changes in facial expressions and eye contact, etc. The speech or text feature vector contains semantic information related to emotion, such as emotional vocabulary and intonation in the language, etc. The fused hidden state feature vector contains emotional features that comprehensively consider image and speech or text information. By performing corresponding splicing and fusion, the third and fourth feature vectors with more comprehensive and rich emotional expression features are obtained, which increases the amount of information in the third and fourth feature vectors to provide more comprehensive emotional expression information.

[0105] The first preset classifier is used to perform emotion classification on the third feature vector to obtain a first emotion recognition result; the first preset classifier is used to perform emotion classification on the fourth feature vector to obtain a second emotion recognition result.

[0106] Specifically, the third feature vector is formed by concatenating the expression feature vector and the fused hidden state feature vector, and the fourth feature vector is formed by concatenating the speech or text feature vector and the fused hidden state feature vector, both of which have comprehensive emotion expression characteristics. The third feature vector and the fourth feature vector are respectively classified by the first preset classifier to obtain the first emotion recognition result and the second emotion recognition result, respectively, so as to take advantage of the comprehensive features, improve the accuracy and comprehensiveness of emotion recognition, and make full use of multimodal information to make emotion classification more accurate and comprehensive.

[0107] The first emotion recognition result and the second emotion recognition result are fused to obtain an emotion recognition result.

[0108] Specifically, the emotion recognition result is obtained by fusing the first emotion recognition result and the second emotion recognition result, performing a simple weighted average, or adopting a more complex integrated learning method, such as voting method, stacking method, etc., so as to comprehensively consider the emotion recognition results of multiple modalities or perspectives, enhance the comprehensive understanding of emotions, and make the final emotion recognition result more comprehensive and accurate.

[0109] Exemplarily, based on the HiddenStateTransformerEncoder model. After extracting the expression feature vector, the image hidden state feature vector, the speech or text feature vector, and the speech or text hidden state feature vector, the two hidden states are fused through the attention layer to obtain a fused hidden state, and the fused hidden state is spliced ​​with the expression feature vector and the speech or text feature vector respectively, and then input into the fully connected network (as the first preset classifier) ​​to obtain two category probability vectors, and the final probability vector is obtained by simple addition to realize emotion recognition.

[0110] Optionally, combined Figure 1 , Figure 5As shown, the first feature vector includes an expression feature vector and an image hidden state feature vector, the second feature vector includes a speech or text feature vector and a speech or text hidden state feature vector; the first preset classifier includes a first expression feature classifier and a first speech or text classifier;

[0111] The multiple feature vectors are subjected to feature fusion to obtain fusion features, and the fusion features are subjected to emotion classification using the first preset classifier, and the emotion recognition results obtained include:

[0112] The image hidden state feature vector and the speech or text hidden state feature vector are input into a preset multi-head attention mechanism, and the speech or text to image attention output and the image to speech or text attention output are output.

[0113] Specifically, the preset multi-head attention mechanism is used to process the image hidden state feature vector and the speech or text hidden state feature vector, so as to obtain the speech or text to image attention output and the image to speech or text attention output. Among them, the multi-head attention mechanism is used to capture the dependency and importance between different positions in the input sequence. The input vector is divided into multiple heads (which can also be understood as subspaces), each head learns different attention weights, and finally the outputs of these heads are spliced ​​together to form the final attention output. The speech or text to image attention output indicates the degree of attention or importance of the corresponding image to the corresponding speech or text; the image to speech or text attention output indicates the degree of attention or importance of the corresponding speech or text to the corresponding image.

[0114] The expression feature vector is fused with the attention output from speech or text to image to obtain a fifth feature vector; the speech or text feature vector is fused with the attention output from image to speech or text to obtain a sixth feature vector.

[0115] Specifically, the expression feature vector and the attention output from speech or text to image are fused to obtain the fifth feature vector; the speech or text feature vector and the attention output from image to speech or text are fused to obtain the sixth feature vector. In this way, the expression feature vector and the attention output from speech or text to image are fused, and the speech or text feature vector and the attention output from image to speech or text are fused to combine the interactive information between expression and speech or text, comprehensively consider the multimodal characteristics of individual emotions, and improve the accuracy and comprehensiveness of emotion recognition.

[0116] The first expression feature classifier is used to perform emotion classification on the fifth feature vector to obtain a first expression emotion classification result; the first speech or text classifier is used to perform emotion classification on the sixth feature vector to obtain a first speech or text emotion classification result.

[0117] Specifically, for the fifth eigenvector and the sixth eigenvector corresponding to different types of data, different classifiers are used to classify them, and the first expression feature classifier and the first speech or text classifier are respectively combined with the fifth eigenvector and the sixth eigenvector to ensure the accuracy of the classification (emotion recognition) results, and obtain the corresponding emotion classification results independently on each modality.

[0118] The first expression emotion classification result and the first speech or text emotion classification result are fused to obtain an emotion recognition result.

[0119] Specifically, the first expression emotion classification result and the first speech or text emotion classification result are fused, that is, the corresponding emotion classification results obtained on each modality are fused, and the emotion recognition result is determined based on the fusion result, so as to comprehensively and fully consider the emotion classification results of different modalities and improve the final recognition accuracy and robustness.

[0120] Exemplarily, based on the CrossModalityAttentionCombine model. The hidden states of the two modalities are input into the multi-head attention mechanism to calculate the attention outputs from text to image and image to text, the expression feature vector and the attention output from speech or text to image are spliced, and then input into the first expression feature classifier (such as an image classifier, etc.) to obtain the probability vector of the emotion shown in the corresponding image; the speech or text feature vector and the attention output from image to speech or text are spliced, and then input into the first speech or text classifier (such as a speech or text classifier) ​​to obtain the probability vector of the emotion shown in the corresponding speech or text. By merging the two probability vectors and applying the SoftMax function, we can derive the final predicted probability vector to achieve emotion recognition.

[0121] Combination Figure 6 As shown, another embodiment of the present invention provides an emotion recognition device, comprising:

[0122] The feature extraction unit is used to extract features based on the acquired multimodal data for emotion recognition to obtain multiple feature vectors.

[0123] The emotion recognition unit is used to perform feature fusion on multiple feature vectors to obtain fused features, and use a first preset classifier to perform emotion classification on the fused features to obtain emotion recognition results; or, to use a second preset classifier to perform emotion classification on multiple feature vectors to obtain classification results corresponding to each feature vector, and fuse the classification results corresponding to all feature vectors to obtain emotion recognition results.

[0124] Although the present invention is disclosed as above, the protection scope of the present invention is not limited thereto. Those skilled in the art may make various changes and modifications without departing from the spirit and scope of the present invention, and these changes and modifications will fall within the protection scope of the present invention.

Claims

1. An emotion recognition method, characterized in that: include: Perform feature extraction based on the acquired multimodal data for emotion recognition to obtain multiple feature vectors; Perform feature fusion on the multiple feature vectors to obtain fused features, and use the first preset classifier to perform emotion classification on the fused features to obtain emotion recognition results; or use the second preset classifier to perform emotion classification on the multiple feature vectors to obtain classification results corresponding to each feature vector, and fuse the classification results corresponding to all the feature vectors to obtain the emotion recognition result.

2. The emotion recognition method according to claim 1, characterized in that: The multimodal data includes a plurality of single-modal data for emotion recognition; The feature extraction is performed based on the acquired multimodal data for emotion recognition to obtain a variety of feature vectors including: Feature extraction is performed respectively according to the acquired multiple unimodal data for emotion recognition to obtain the feature vector corresponding to each type of unimodal data.

3. The emotion recognition method according to claim 1 or 2, characterized in that: The multimodal data includes video data having image information and corresponding audio information; The feature extraction is performed based on the acquired multimodal data for emotion recognition to obtain a variety of feature vectors including: Performing video frame extraction and voice recognition on the video data of the same individual to obtain the image information and the audio information; Feature extraction is performed on the image information to obtain a first feature vector; feature extraction is performed on the audio information to obtain a second feature vector.

4. The emotion recognition method according to claim 3, characterized in that: The extracting features of the image information to obtain a first feature vector comprises: Using a first preset model to extract features from the image information to obtain a first feature vector; The extracting features of the audio information to obtain a second feature vector comprises: A second preset model is used to extract features from the audio information to obtain a second feature vector.

5. The emotion recognition method according to claim 3, characterized in that: The step of fusing the multiple feature vectors to obtain fused features, and using a first preset classifier to perform emotion classification on the fused features to obtain an emotion recognition result includes: Concatenating and fusing the first feature vector and the second feature vector to obtain the fused feature; The first preset classifier is used to perform sentiment classification on the fusion feature to obtain the sentiment recognition result.

6. The emotion recognition method according to claim 3, characterized in that: The step of fusing the multiple feature vectors to obtain fused features, and using a first preset classifier to perform emotion classification on the fused features to obtain an emotion recognition result includes: Fusing the first feature vector and the second feature vector at an attention layer to obtain the fused feature; The first preset classifier is used to perform sentiment classification on the fusion feature to obtain the sentiment recognition result.

7. The emotion recognition method according to claim 3, characterized in that: The second preset classifier includes a second expression feature classifier and a second voice or text classifier; The emotion recognition result obtained by using the second preset classifier to perform emotion classification on the multiple feature vectors to obtain the classification results corresponding to each feature vector, and fusing the classification results corresponding to all the feature vectors to obtain the emotion recognition result includes: Using the second expression feature classifier to perform emotion classification on the first feature vector to obtain a second expression emotion classification result; using the second speech or text classifier to perform emotion classification on the second feature vector to obtain a second speech or text emotion classification result; The second expression emotion classification result and the second speech or text emotion classification result are fused to obtain the emotion recognition result.

8. The emotion recognition method according to claim 3, characterized in that: The first feature vector includes an expression feature vector and an image hidden state feature vector, and the second feature vector includes a speech or text feature vector and a speech or text hidden state feature vector; The step of fusing the multiple feature vectors to obtain fused features, and using a first preset classifier to perform emotion classification on the fused features to obtain an emotion recognition result includes: Fusing the image hidden state feature vector and the speech or text hidden state feature vector to obtain a fused hidden state feature vector; The expression feature vector is concatenated and fused with the fused hidden state feature vector to obtain a third feature vector; the speech or text feature vector is concatenated and fused with the fused hidden state feature vector to obtain a fourth feature vector; Using the first preset classifier to perform emotion classification on the third feature vector to obtain a first emotion recognition result; using the first preset classifier to perform emotion classification on the fourth feature vector to obtain a second emotion recognition result; The first emotion recognition result and the second emotion recognition result are fused to obtain the emotion recognition result.

9. The emotion recognition method according to claim 3, characterized in that: The first feature vector includes an expression feature vector and an image hidden state feature vector, the second feature vector includes a speech or text feature vector and a speech or text hidden state feature vector; the first preset classifier includes a first expression feature classifier and a first speech or text classifier; The step of fusing the multiple feature vectors to obtain fused features, and using a first preset classifier to perform emotion classification on the fused features to obtain an emotion recognition result includes: Input the image hidden state feature vector and the speech or text hidden state feature vector into a preset multi-head attention mechanism, and output the speech or text to image attention output and the image to speech or text attention output; The expression feature vector is merged with the attention output from the speech or text to the image to obtain a fifth feature vector; the speech or text feature vector is merged with the attention output from the image to the speech or text to obtain a sixth feature vector; Using the first expression feature classifier to perform emotion classification on the fifth feature vector to obtain a first expression emotion classification result; using the first speech or text classifier to perform emotion classification on the sixth feature vector to obtain a first speech or text emotion classification result; The first expression emotion classification result and the first speech or text emotion classification result are fused to obtain the emotion recognition result.

10. An emotion recognition device, characterized in that: include: A feature extraction unit, used for extracting features based on the acquired multimodal data for emotion recognition to obtain multiple feature vectors; The emotion recognition unit is used to perform feature fusion on the multiple feature vectors to obtain fused features, and use the first preset classifier to perform emotion classification on the fused features to obtain emotion recognition results; or, used to perform emotion classification on the multiple feature vectors using the second preset classifier to obtain classification results corresponding to each feature vector, and fuse the classification results corresponding to all the feature vectors to obtain the emotion recognition results.

Images