Multi-modal emotion recognition method and system, electronic equipment and storage medium

Abstract

The embodiment of the invention relates to the technical field of emotion recognition, in particular to a multi-modal emotion recognition method and system, electronic equipment and a storage medium, and the method comprises the steps: obtaining multi-modal sample data composed of voice sample data, image sample data and text sample data through a millimeter wave radar, and labeling an emotion label; constructing a multi-modal emotion recognition model composed of an image feature extraction branch network, a voice feature extraction branch network, a text feature extraction branch network, a fusion network, a residual network and a discrimination network; performing iterative training on the model until convergence based on an emotion category recognition result of the multi-modal sample data output by the model and an emotion label marked on the multi-modal sample data to obtain a trained model; and obtaining multi-modal data of a to-be-identified object through the millimeter wave radar, inputting the multi-modal data into the trained model, and obtaining an emotion category identification result of the to-be-identified object. According to the method, the emotion recognition accuracy is effectively improved.

Description

Technical Field

[0001] Embodiments of the present application relate to the field of emotion recognition technology, and in particular, to a multimodal emotion recognition method, system, electronic device, and storage medium. Background Art

[0002] Emotion recognition technology is a research direction with wide applications and great significance in the field of artificial intelligence technology. With the continuous development of artificial intelligence technology, emotion recognition technology has gradually expanded from simple text, voice, image and other media recognition to the fusion recognition of multimodal information. The main application scenarios of emotion recognition technology include but are not limited to human-computer interaction, social networks, health care, education, entertainment, etc. The core task of emotion recognition technology is to identify human emotional states (such as happiness, displeasure, anxiety, etc.) and convert them into a form that computers can understand.

[0003] Millimeter wave radar is a radar that works in the millimeter wave band. Its frequency domain is between 30GHz and 300GHz. The wavelength of millimeter wave is between microwave and centimeter wave, so millimeter wave radar has some advantages of microwave radar and photoelectric radar. Millimeter wave radar transmits millimeter wave signals to the target object and receives the echo signal reflected by the target object. After processing, it can realize imaging of the target object. Millimeter wave radar can also accurately capture the vibration of the speaker. After high-precision detection and processing of the captured vibration, voice data can be obtained.

[0004] It can be seen that millimeter-wave radar can simultaneously obtain image data and voice data of the target object. After voice-to-text recognition, the voice data can be converted into text data. Therefore, the basic data for multimodal emotion recognition can be obtained through only one device, the millimeter-wave radar.

[0005] However, millimeter-wave radar imaging technology and voice acquisition technology are still in the development stage. The multimodal emotion recognition models currently proposed in the industry are designed for voice data and image data obtained by microphones, cameras and other devices. The effect of multimodal emotion recognition using data obtained by millimeter-wave radar on these models is not good, and the accuracy of emotion recognition is relatively low. Summary of the invention

[0006] In view of this, the embodiments of the present application propose a multimodal emotion recognition method, system, electronic device and storage medium, and design a suitable multimodal emotion recognition model for the scenario where millimeter-wave radar acquires data. When fusing image data, voice data and text data, their respective complementary characteristics are retained, thereby effectively improving the accuracy of emotion recognition.

[0007] In a first aspect, an embodiment of the present application proposes a multimodal emotion recognition method, comprising the following steps: acquiring multimodal sample data through a millimeter wave radar, and marking emotion labels for the multimodal sample data; wherein the multimodal sample data includes speech sample data and facial image sample data when the sample object speaks with different emotions, and text sample data obtained by converting the speech sample data through a preset speech-to-text recognition model; constructing a multimodal emotion recognition model; wherein the multimodal emotion recognition model is composed of an image feature extraction branch network, a speech feature extraction branch network, a text feature extraction branch network, a fusion network, a residual network and a discriminant network, the image feature extraction branch network is used to extract image features from the image sample data and obtain a first hidden state vector corresponding to the image features, the speech feature extraction branch network is used to extract speech features from the speech sample data and obtain a second hidden state vector corresponding to the speech features, and the text feature extraction branch network is used to extract speech features from the text sample data. In this sample data, text features are extracted and the third hidden state vector corresponding to the text features is obtained. The fusion network is used to fuse image features, speech features and text features. The residual network is used to perform residual connection on the first hidden state vector, the second hidden state vector, the third hidden state vector and the output of the fusion network to obtain the final features. The discriminant network is used to classify and discriminate the final features to obtain the emotion category recognition results of the multimodal sample data; based on the emotion category recognition results of the multimodal sample data output by the multimodal emotion recognition model and the emotion labels marked on the multimodal sample data, the multimodal emotion recognition model is iteratively trained until convergence to obtain a trained multimodal emotion recognition model; the multimodal data of the object to be identified is obtained through the millimeter wave radar, and the multimodal data of the object to be identified is input into the trained multimodal emotion recognition model to obtain the emotion category recognition results of the object to be identified output by the trained multimodal emotion recognition model.

[0008] Optionally, the image feature extraction branch network is composed of a Zernike image feature extractor and an AlexNet image classification network connected in sequence, the Zernike image feature extractor is used to extract image features from image sample data, and the AlexNet image classification network is used to process the image features to obtain a first hidden state vector;

[0009] The speech feature extraction branch network is composed of a wav2vec speech feature extractor and a Bi-LSTM-A speech classification network connected in sequence. The wav2vec speech feature extractor is used to extract speech features from speech sample data, and the Bi-LSTM-A speech classification network is used to process the speech features to obtain a second hidden state vector.

[0010] The text feature extraction branch network consists of a sequentially connected word2vec text feature extractor and a TextCNN text classification network. The word2vec text feature extractor is used to extract text features from text sample data, and the TextCNN text classification network is used to process the text features to obtain a third hidden state vector.

[0011] Optionally, the Bi-LSTM-A speech classification network consists of a three-layer structure, namely a word embedding layer, a LSTM+attention mechanism layer, and a fully connected layer;

[0012] The input of the word embedding layer is the speech features of each frame. The speech features of each frame are embedded into words and the output after processing is the context features.

[0013] The LSTM+attention mechanism layer is used to process context features, add the attention mechanism to obtain attention weights, and output speech features with attention weights.

[0014] The fully connected layer is used to perform speech classification based on the speech features assigned with attention weights, and finally outputs the speech classification result as the second hidden state vector.

[0015] Optionally, the fusion network consists of a first fusion unit, a second fusion unit, a third fusion unit and a fourth fusion unit; the first fusion unit is used to fuse image features and speech features to obtain first mixed features; the second fusion unit is used to fuse image features and text features to obtain second mixed features; the third fusion unit is used to fuse speech features and text features to obtain third mixed features; the fourth fusion unit is used to fuse image features, speech features and text features to obtain fourth mixed features.

[0016] Optionally, the residual network is used to merge the first hidden state vector, the second hidden state vector and the third hidden state vector to obtain a fifth mixed feature, and to perform residual connection on the first mixed feature, the second mixed feature, the third mixed feature, the fourth mixed feature and the fifth mixed feature to obtain a final feature.

[0017] Optionally, after converting the speech sample data through a preset speech-to-text recognition model to obtain text sample data, the method further includes: using regular expressions to remove http and https web page links in the text sample data, using the value range of an ASCII string to determine whether the text sample data is an English string, performing word segmentation, recognizing English words, and removing punctuation marks on the cleaned text sample data; performing English word segmentation using nltk's TweetTokenizer, segmenting the string into a list, using enchant to check whether the string is an English word, and then using the isalpha(·) function to filter, filtering out strings containing punctuation marks, and obtaining a pure English word segmentation list; performing part-of-speech tagging, part-of-speech restoration, and stop word filtering operations on the word segmentation list, using the postag(·) function to perform part-of-speech tagging, using the lemmatize(·) function to restore the words after the part-of-speech tagging to their original form, and then using the nltk English stop word dictionary to perform stop word filtering to obtain preprocessed text sample data.

[0018] Optionally, when the multimodal emotion recognition model is iteratively trained to convergence, the loss value is calculated based on the cross entropy loss function, the initial learning rate is set to 0.0001, and the Adam optimizer in pytorch is used for forward propagation to update the model parameters in the multimodal emotion recognition model;

[0019] The expression of the cross entropy loss function is:

[0020] loss=CrossEntropyLoss(pred_label, real_label);

[0021] Among them, pred_label represents the emotion category recognition result of the multimodal sample data output by the multimodal emotion recognition model, real_label represents the emotion label annotated on the multimodal sample data, CrossEntropyLoss(·) represents the cross entropy loss function, and loss represents the calculated loss value.

[0022] Through the above method, the present application constructs and trains a multimodal emotion recognition model for multimodal emotion recognition, and designs image feature extraction branch networks, speech feature extraction branch networks, and text feature extraction branch networks to process image data, speech data, and text data. The features extracted by the three feature extraction branch networks are fused by the fusion network, and then the residual network performs residual connection on the hidden state vectors obtained by processing the three feature extraction branch networks and the features fused by the fusion network to obtain the final features, which are finally handed over to the discriminant network for emotion category recognition. In the training samples used to train the multimodal emotion recognition model, both the speech sample data and the image sample data are obtained by millimeter wave radar, so that the multimodal emotion recognition model trained can be well adapted to the scene where the millimeter wave radar obtains data. The design of the fusion network and the residual network retains the complementary characteristics of the image data, speech data, and text data. The residual network can also prevent the model from being too deep and causing the gradient to disappear. Finally, the image features, speech features, and text features can be fully and effectively utilized for multimodal emotion analysis, greatly improving the accuracy of emotion recognition.

[0023] In the second aspect, an embodiment of the present application proposes a multimodal emotion recognition system, including: an acquisition module, used to acquire multimodal sample data through a millimeter wave radar, and annotate emotion labels for the multimodal sample data, wherein the multimodal sample data includes speech sample data and facial image sample data when the sample object speaks with different emotions, and text sample data obtained by converting the speech sample data through a preset speech-to-text recognition model; a construction module, used to construct a multimodal emotion recognition model, wherein the multimodal emotion recognition model is composed of an image feature extraction branch network, a speech feature extraction branch network, a text feature extraction branch network, a fusion network, a residual network and a discriminant network, the image feature extraction branch network is used to extract image features from the image sample data and obtain a first hidden state vector corresponding to the image features, the speech feature extraction branch network is used to extract speech features from the speech sample data and obtain a second hidden state vector corresponding to the speech features, and the text feature extraction branch network is used to extract speech features from the text sample data and obtain a second hidden state vector corresponding to the speech features. Text features are extracted from this data and the third hidden state vector corresponding to the text features is obtained. The fusion network is used to fuse image features, speech features and text features. The residual network is used to perform residual connection on the first hidden state vector, the second hidden state vector, the third hidden state vector and the output of the fusion network to obtain the final features. The discriminant network is used to classify and discriminate the final features to obtain the emotion category recognition results of the multimodal sample data; the training module is used to iteratively train the multimodal emotion recognition model until convergence based on the emotion category recognition results of the multimodal sample data output by the multimodal emotion recognition model and the emotion labels marked on the multimodal sample data to obtain the trained multimodal emotion recognition model; the use module is used to obtain the multimodal data of the object to be identified through the millimeter wave radar, input the multimodal data of the object to be identified into the trained multimodal emotion recognition model, and obtain the emotion category recognition results of the object to be identified output by the trained multimodal emotion recognition model.

[0024] In a third aspect, an embodiment of the present application proposes an electronic device, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute a multimodal emotion recognition method as described in the first aspect above.

[0025] In a fourth aspect, an embodiment of the present application proposes a computer-readable storage medium storing a computer program, which, when executed by a processor, can implement a multimodal emotion recognition method as described in the first aspect above.

[0026] It can be understood that the beneficial effects of the second to fourth aspects mentioned above can be found in the relevant description of the first aspect mentioned above, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the related technologies, the drawings required for use in the embodiments of the present application or the related technical descriptions will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0028] Figure 1 is a flowchart of a multimodal emotion recognition method provided in one embodiment of the present application;

[0029] Figure 2 is a schematic diagram of a model structure of a multimodal emotion recognition model provided in an embodiment of the present application;

[0030] Figure 3 It is a schematic diagram of the structure of a Bi-LSTM-A speech classification network provided in one embodiment of the present application;

[0031] Figure 4 It is a schematic diagram of the principle of an LSTM+attention mechanism provided in one embodiment of the present application;

[0032] Figure 5 is a structural schematic diagram of a multimodal emotion recognition system provided in another embodiment of the present application;

[0033] Figure 6 It is a structural schematic diagram of an electronic device provided in another embodiment of the present application. DETAILED DESCRIPTION

[0034] In order to make the purpose, technical scheme and advantages of the embodiments of the present application clearer, the embodiments of the present application will be described in detail below in conjunction with the accompanying drawings. In the various embodiments of the present application, many technical details are proposed in order to make the reader better understand the present application. However, even without these technical details and various changes and modifications based on the following embodiments, the technical scheme claimed in the present application can also be implemented. The division of the following embodiments is only for the convenience of description, and the specific implementation mode of the present application should not constitute any limitation. The various embodiments can be combined with each other and referenced to each other under the premise of no contradiction.

[0035] An embodiment of the present application proposes a multimodal emotion recognition method, which is applied to an electronic device, wherein the electronic device is a terminal or a server. In this embodiment and the following embodiments, the electronic device is described using the server as an example. The implementation details of the multimodal emotion recognition method proposed in this embodiment are specifically described below. The following content is only the implementation details provided for the convenience of understanding and is not necessary for the implementation of this solution.

[0036] The specific process of the multimodal emotion recognition method proposed in this embodiment can be as follows: Figure 1 As shown, including:

[0037] Step 101, acquiring multimodal sample data through a millimeter wave radar, and labeling the multimodal sample data with emotion labels, wherein the multimodal sample data includes speech sample data and facial image sample data when the sample object speaks with different emotions, and text sample data obtained by converting the speech sample data through a preset speech-to-text recognition model.

[0038] In the specific implementation, the server must first collect training sample data. The multimodal emotion recognition method proposed in this application is designed for the scenario where millimeter-wave radar obtains data. Therefore, the server needs to obtain multimodal sample data through the millimeter-wave radar and annotate the multimodal sample data with emotion labels. The multimodal sample data obtained by the millimeter-wave radar includes speech sample data and facial image sample data when the sample object speaks with different emotions, as well as text sample data obtained by converting the speech sample data through a preset speech-to-text recognition model. The preset speech-to-text recognition model can be a Whisper model, a DeepSpeech model, a SenseVoice model, etc.

[0039] In one example, there are three types of sentiment labels annotated for multimodal sample data, namely positive label, negative label and neutral label, to represent three different emotions: positive, negative and neutral.

[0040] In one example, for image sample data, data preprocessing including size normalization, data cleaning and data enhancement is required.

[0041] In one example, for speech sample data, it is necessary to resample at a sampling rate of 16000 Hz, and then perform data preprocessing including speech enhancement and noise processing.

[0042] In one example, for text sample data, after the server converts the voice sample data through a preset speech-to-text recognition model to obtain text sample data, it is also necessary to use regular expressions to remove http and https web page links in the text sample data, use the value range of the ASCII string to determine whether the text sample data is an English string, and perform word segmentation, English word recognition, and punctuation removal operations on the cleaned text sample data. Then, use nltk's TweetTokenizer to perform English word segmentation, segment the string into a list, use enchant to check whether the string is an English word, and then use the isalpha(·) function to filter, filter out the string containing punctuation, and obtain a pure English word segmentation list. Finally, perform part-of-speech tagging, part-of-speech restoration, and stop word filtering operations on the word segmentation list, use the postag(·) function to perform part-of-speech tagging, use the lemmatize(·) function to restore the words after part-of-speech tagging to their original form, and then use the nltk English stop word dictionary to perform stop word filtering to obtain preprocessed text sample data.

[0043] It is understandable that appropriate preprocessing of image sample data, speech sample data, and text sample data can improve the accuracy and efficiency when processing downstream tasks.

[0044] Step 102, constructing a multimodal emotion recognition model, wherein the multimodal emotion recognition model is composed of an image feature extraction branch network, a speech feature extraction branch network, a text feature extraction branch network, a fusion network, a residual network and a discriminant network.

[0045] In the specific implementation, after the server completes the collection of training sample data, it can build a multimodal emotion recognition model, where the multimodal emotion recognition model is composed of an image feature extraction branch network, a speech feature extraction branch network, a text feature extraction branch network, a fusion network, a residual network and a discriminant network.

[0046] The image feature extraction branch network is used to extract image features from the image sample data and obtain a first hidden state vector corresponding to the image features. The speech feature extraction branch network is used to extract speech features from the speech sample data and obtain a second hidden state vector corresponding to the speech features. The text feature extraction branch network is used to extract text features from the text sample data and obtain a third hidden state vector corresponding to the text features.

[0047] The fusion network is used to fuse image features, speech features and text features.

[0048] The residual network is used to perform residual connection on the first hidden state vector, the second hidden state vector, the third hidden state vector and the output of the fusion network to obtain the final feature.

[0049] The discriminant network is used to classify and discriminate the final features to obtain the emotion category recognition results of the multimodal sample data input into the multimodal emotion recognition model.

[0050] In one example, the model structure of a multimodal emotion recognition model is as follows: Figure 2 shown.

[0051] The image feature extraction branch network consists of a Zernike image feature extractor and an AlexNet image classification network connected in sequence. The Zernike image feature extractor is used to extract image features from image sample data, and input the extracted image features into the AlexNet image classification network and the fusion network. The AlexNet image classification network is used to process the image features to obtain the first hidden state vector, and input it into the residual network.

[0052] The speech feature extraction branch network consists of a wav2vec speech feature extractor and a Bi-LSTM-A speech classification network connected in sequence. The wav2vec speech feature extractor is used to extract speech features from speech sample data, and input the extracted speech features into the Bi-LSTM-A speech classification network and the fusion network. The Bi-LSTM-A speech classification network is used to process the speech features to obtain the second hidden state vector, and input it into the residual network.

[0053] The text feature extraction branch network consists of a sequentially connected word2vec text feature extractor and a TextCNN text classification network. The word2vec text feature extractor is used to extract text features from text sample data and input the extracted text features into the TextCNN text classification network and the fusion network. The TextCNN text classification network is used to process the text features to obtain the third hidden state vector and input it into the residual network.

[0054] The fusion network is specifically composed of a first fusion unit, a second fusion unit, a third fusion unit and a fourth fusion unit, and different fusion units are used to fuse different features.

[0055] The first fusion unit is used to fuse the image feature and the speech feature to obtain a first mixed feature.

[0056] The second fusion unit is used to fuse the image feature and the text feature to obtain a second mixed feature.

[0057] The third fusion unit is used to fuse the speech feature and the text feature to obtain a third mixed feature.

[0058] The fourth fusion unit is used to fuse the image feature, the voice feature and the text feature to obtain a fourth mixed feature.

[0059] The mixed features output by the first fusion unit, the second fusion unit, the third fusion unit and the fourth fusion unit are all input into the residual network for the next step of processing.

[0060] The residual network is specifically used to merge the first hidden state vector, the second hidden state vector and the third hidden state vector to obtain the fifth mixed feature, and then perform residual connection on the first mixed feature, the second mixed feature, the third mixed feature, the fourth mixed feature and the fifth mixed feature to obtain the final feature.

[0061] In an example, the image features extracted from the image sample data by the Zernike image feature extractor can be expressed as img_feature[B, W×H, D], the speech features extracted from the speech sample data by the wav2vec speech feature extractor can be expressed as wav_feature[B, T, D], and the text features extracted from the text sample data by the word2vec text feature extractor can be expressed as text_feature[s, U, D], where B represents the batch size, W×H represents the size of the image sample data, D represents the feature dimension, T represents the frame length of the speech sample data, and U represents the length of the text sample data.

[0062] Taking the third fusion unit of the fusion network as an example, when fusing speech features and text features, it is first necessary to expand wav_feature[B, T, D] from three dimensions to four dimensions of wav_feature[B, T, 1, D]. Similarly, it is also necessary to expand text_feature[B, U, D] from three dimensions to four dimensions of text_feature[B, 1, U, D].

[0063] Then repeat wav_feature[B, T, 1, D] U times and text_feature[B, 1, U, D] T times, so that the feature vector sizes of both become [B, T, U, D]. Merge (concatenate) wav_feature[B, T, U, D] and text_feature[B, T, U, D] to obtain hybrid_feature[B, T, U, 2D], and then after two layers of fully connected layers, the third hybrid feature hybrid_feature[B, T, U, D] can be obtained.

[0064] In an example, Figure 3As shown in the figure, the Bi-LSTM-A speech classification network consists of three layers, namely, the word embedding layer, the LSTM+attention mechanism layer, and the fully connected layer. The input of the word embedding layer is the speech features of each frame, and the speech features of each frame are word embedded. After processing, the output is the context feature. The LSTM+attention mechanism layer is used to process the context features, add the attention mechanism to obtain the attention weight, and output the speech features with the attention weight. The fully connected layer is used to perform speech classification based on the speech features with the attention weight, and finally output the speech classification result as the second hidden state vector.

[0065] In an example, the calculation process of LSTM+attention mechanism is as follows Figure 4 shown.

[0066] Step 103, based on the emotion category recognition results of the multimodal sample data output by the multimodal emotion recognition model and the emotion labels marked on the multimodal sample data, the multimodal emotion recognition model is iteratively trained until convergence to obtain a trained multimodal emotion recognition model.

[0067] In the specific implementation, after the server completes the construction of the multimodal emotion recognition model, it can enter the model training stage. Based on the emotion category recognition results of the multimodal sample data output by the multimodal emotion recognition model and the emotion labels marked on the multimodal sample data, the multimodal emotion recognition model is iteratively trained until convergence to obtain a trained multimodal emotion recognition model.

[0068] In an example, when the server iteratively trains the multimodal emotion recognition model until convergence, the loss value can be calculated based on the cross entropy loss function, the initial learning rate is set to 0.0001, and the Adam optimizer in pytorch is used for forward propagation to update the model parameters in the multimodal emotion recognition model.

[0069] The expression of the cross entropy loss function is:

[0070] loss=CrossEntropyLoss(pred_label, real-label);

[0071] Among them, pred_label represents the emotion category recognition result of the multimodal sample data output by the multimodal emotion recognition model, real_label represents the emotion label annotated on the multimodal sample data, CrossEntropyLoss(·) represents the cross entropy loss function, and loss represents the calculated loss value.

[0072] Step 104, obtaining multimodal data of the object to be identified through the millimeter wave radar, inputting the multimodal data of the object to be identified into the trained multimodal emotion recognition model, and obtaining the emotion category recognition result of the object to be identified output by the trained multimodal emotion recognition model.

[0073] In a specific implementation, after the server obtains the trained multimodal emotion recognition model, it can deploy the trained multimodal emotion recognition model in the required scenario, obtain the multimodal data of the object to be identified through the millimeter wave radar, input the multimodal data of the object to be identified into the trained multimodal emotion recognition model, and obtain the emotion category recognition result of the object to be identified output by the trained multimodal emotion recognition model.

[0074] In this embodiment, a multimodal emotion recognition model is constructed and trained for multimodal emotion recognition. An image feature extraction branch network, a speech feature extraction branch network, and a text feature extraction branch network are designed to process image data, speech data, and text data. The features extracted by the three feature extraction branch networks are fused by a fusion network, and then the residual network performs residual connection on the hidden state vectors obtained by processing the three feature extraction branch networks and the features fused by the fusion network to obtain the final features, which are finally handed over to the discriminant network for emotion category recognition. In the training samples used to train the multimodal emotion recognition model, both the speech sample data and the image sample data are obtained by millimeter wave radars. In this way, the multimodal emotion recognition model trained can be well adapted to the scene where the millimeter wave radar obtains data. The design of the fusion network and the residual network retains the complementary characteristics of the image data, speech data, and text data. The residual network can also prevent the model from being too deep and causing the gradient to disappear. Finally, the image features, speech features, and text features can be effectively used for multimodal emotion analysis, which greatly improves the accuracy of emotion recognition.

[0075] The step division of the above methods is only for clear description. When implemented, they can be combined into one step or some steps can be split and decomposed into multiple steps. As long as they include the same logical relationship, they are all within the scope of protection of this application; adding insignificant modifications to the algorithm or process or introducing insignificant designs without changing the core design of the algorithm and process are all within the scope of protection of this application.

[0076] Another embodiment of the present application proposes a multimodal emotion recognition system. The implementation details of the multimodal emotion recognition system proposed in this embodiment are described in detail below. The following content is only for the convenience of understanding the implementation details provided, and is not necessary for implementing this example.

[0077] Figure 520 is a schematic diagram of the structure of the multimodal emotion recognition system proposed in this embodiment, including: an acquisition module 201, a construction module 202, a training module 203 and a use module 204.

[0078] The acquisition module 201 is used to acquire multimodal sample data through millimeter wave radar and annotate the multimodal sample data with emotion labels. The multimodal sample data includes speech sample data and facial image sample data when the sample object speaks with different emotions, as well as text sample data obtained by converting the speech sample data through a preset speech-to-text recognition model.

[0079] The construction module 202 is used to construct a multimodal emotion recognition model, wherein the multimodal emotion recognition model is composed of an image feature extraction branch network, a speech feature extraction branch network, a text feature extraction branch network, a fusion network, a residual network and a discriminant network. The image feature extraction branch network is used to extract image features from image sample data and obtain a first hidden state vector corresponding to the image features. The speech feature extraction branch network is used to extract speech features from speech sample data and obtain a second hidden state vector corresponding to the speech features. The text feature extraction branch network is used to extract text features from text sample data and obtain a third hidden state vector corresponding to the text features. The fusion network is used to fuse image features, speech features and text features. The residual network is used to perform residual connection on the first hidden state vector, the second hidden state vector, the third hidden state vector and the output of the fusion network to obtain the final features. The discriminant network is used to classify and discriminate the final features to obtain the emotion category recognition result of the multimodal sample data.

[0080] The training module 203 is used to iteratively train the multimodal emotion recognition model until convergence based on the emotion category recognition results of the multimodal sample data output by the multimodal emotion recognition model and the emotion labels marked on the multimodal sample data, so as to obtain a trained multimodal emotion recognition model.

[0081] Module 204 is used to obtain multimodal data of the object to be identified through the millimeter wave radar, input the multimodal data of the object to be identified into the trained multimodal emotion recognition model, and obtain the emotion category recognition result of the object to be identified output by the trained multimodal emotion recognition model.

[0082] It is worth mentioning that all modules involved in this embodiment are logic modules. In practical applications, a logic unit can be a physical unit, a part of a physical unit, or a combination of multiple physical units. In addition, in order to highlight the innovative part of this application, this embodiment does not introduce units that are not closely related to solving the technical problems proposed by this application, but this does not mean that there are no other units in this embodiment.

[0083] It is not difficult to find that this embodiment is a system embodiment corresponding to the above method embodiment, and this embodiment can be implemented in conjunction with the above method embodiment. The relevant technical details and technical effects mentioned in the above embodiments are still valid in this embodiment, and in order to reduce repetition, they are not repeated here. Accordingly, the relevant technical details mentioned in this embodiment can also be applied in the above embodiments.

[0084] Another embodiment of the present application provides an electronic device, the structure of which is as follows: Figure 6 As shown, it includes: at least one processor 301; and a memory 302 that is communicatively connected to the at least one processor 301; wherein the memory 302 stores instructions that can be executed by the at least one processor 301, and the instructions are executed by the at least one processor 301 so that the at least one processor 301 can execute a multimodal emotion recognition method described in the above-mentioned method embodiments.

[0085] Among them, the memory and the processor are connected in a bus manner, and the bus may include any number of interconnected buses and bridges, and the bus connects various circuits of one or more processors and memories together. The bus can also connect various other circuits such as peripheral devices, voltage regulators, and power management circuits, which are well known in the art and will not be further described in this article. The bus interface is responsible for providing an interface between the bus and the transceiver. The transceiver can be one component or multiple components, such as multiple receivers and transmitters, providing a unit for communicating with various other devices on a transmission medium. The data processed by the processor is transmitted on a wireless medium through an antenna, and further, the antenna also receives data and transmits the data to the processor.

[0086] The processor is responsible for managing the bus and general processing, and can also provide various functions, including timing, peripheral interfaces, voltage regulation, power management, and other control functions. Memory can be used to store data used by the processor when performing operations.

[0087] Another embodiment of the present application proposes a computer-readable storage medium storing a computer program. When the computer program is executed by a processor, it can implement a multimodal emotion recognition method described in the above method embodiments.

[0088] That is, those skilled in the art can understand that all or part of the steps in the above-mentioned embodiment method can be completed by instructing the relevant hardware through a program, and the program is stored in a storage medium, including a number of instructions to enable a device (which can be a single-chip microcomputer, chip, etc.) or a processor to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, ROM (Read-Only Memory), RAM (Random Access Memory), disk or optical disk and other media that can store program codes.

[0089] Those skilled in the art will appreciate that the above embodiments are specific embodiments for implementing the present application, and in actual applications, various changes may be made thereto in form and detail without departing from the spirit and scope of the present application.

Claims

1. A multimodal emotion recognition method, characterized in that: include: Acquire multimodal sample data through millimeter wave radar, and annotate the multimodal sample data with emotion labels; wherein the multimodal sample data includes speech sample data and facial image sample data when the sample object speaks with different emotions, and text sample data obtained by converting the speech sample data through a preset speech-to-text recognition model; Construct a multimodal emotion recognition model; wherein the multimodal emotion recognition model is composed of an image feature extraction branch network, a speech feature extraction branch network, a text feature extraction branch network, a fusion network, a residual network and a discriminant network, the image feature extraction branch network is used to extract image features from image sample data and obtain a first hidden state vector corresponding to the image features, the speech feature extraction branch network is used to extract speech features from speech sample data and obtain a second hidden state vector corresponding to the speech features, the text feature extraction branch network is used to extract text features from text sample data and obtain a third hidden state vector corresponding to the text features, the fusion network is used to fuse image features, speech features and text features, the residual network is used to perform residual connection on the first hidden state vector, the second hidden state vector, the third hidden state vector and the output of the fusion network to obtain the final features, and the discriminant network is used to classify and discriminate the final features to obtain the emotion category recognition result of the multimodal sample data; Based on the emotion category recognition results of the multimodal sample data output by the multimodal emotion recognition model and the emotion labels marked on the multimodal sample data, the multimodal emotion recognition model is iteratively trained until convergence to obtain a trained multimodal emotion recognition model; The multimodal data of the object to be identified is obtained through the millimeter wave radar, and the multimodal data of the object to be identified is input into the trained multimodal emotion recognition model to obtain the emotion category recognition result of the object to be identified output by the trained multimodal emotion recognition model.

2. A multimodal emotion recognition method according to claim 1, characterized in that: The image feature extraction branch network is composed of a Zernike image feature extractor and an AlexNet image classification network connected in sequence, the Zernike image feature extractor is used to extract image features from image sample data, and the AlexNet image classification network is used to process the image features to obtain a first hidden state vector; The speech feature extraction branch network is composed of a wav2vec speech feature extractor and a Bi-LSTM-A speech classification network connected in sequence. The wav2vec speech feature extractor is used to extract speech features from speech sample data, and the Bi-LSTM-A speech classification network is used to process the speech features to obtain a second hidden state vector. The text feature extraction branch network consists of a sequentially connected word2vec text feature extractor and a TextCNN text classification network. The word2vec text feature extractor is used to extract text features from text sample data, and the TextCNN text classification network is used to process the text features to obtain a third hidden state vector.

3. A multimodal emotion recognition method according to claim 2, characterized in that: The Bi-LSTM-A speech classification network consists of three layers: word embedding layer, LSTM+attention mechanism layer, and fully connected layer; The input of the word embedding layer is the speech features of each frame. The speech features of each frame are embedded into words and the output after processing is the context features. The LSTM+attention mechanism layer is used to process context features, add the attention mechanism to obtain attention weights, and output speech features with attention weights. The fully connected layer is used to perform speech classification based on the speech features assigned with attention weights, and finally outputs the speech classification result as the second hidden state vector.

4. A multimodal emotion recognition method according to claim 2, characterized in that: The fusion network consists of a first fusion unit, a second fusion unit, a third fusion unit and a fourth fusion unit; The first fusion unit is used to fuse the image feature and the speech feature to obtain a first mixed feature; The second fusion unit is used to fuse the image feature and the text feature to obtain a second mixed feature; The third fusion unit is used to fuse the speech feature and the text feature to obtain a third mixed feature; The fourth fusion unit is used to fuse the image feature, the voice feature and the text feature to obtain a fourth mixed feature.

5. A multimodal emotion recognition method according to claim 4, characterized in that: The residual network is used to merge the first hidden state vector, the second hidden state vector and the third hidden state vector to obtain a fifth mixed feature, and to perform residual connection on the first mixed feature, the second mixed feature, the third mixed feature, the fourth mixed feature and the fifth mixed feature to obtain a final feature.

6. A multimodal emotion recognition method according to any one of claims 1 to 5, characterized in that: After converting the speech sample data into text sample data through a preset speech-to-text recognition model, the method further includes: Use regular expressions to remove http and https web page links in the text sample data, use the value range of ASCII strings to determine whether the text sample data is an English string, and perform word segmentation, English word recognition, and punctuation removal operations on the cleaned text sample data; Use nltk's TweetTokenizer to segment English words, segment the string into a list, use enchant to check whether the string is an English word, and then use the isalpha(·) function to filter out the string containing punctuation marks to get a pure English word list; Perform part-of-speech tagging, part-of-speech restoration, and stop word filtering operations on the word segmentation list. Use the postag(·) function to perform part-of-speech tagging, use the lemmatize(·) function to restore the tagged words to their original form, and then use the nltk English stop word dictionary to perform stop word filtering to obtain the preprocessed text sample data.

7. A multimodal emotion recognition method according to any one of claims 1 to 5, characterized in that: When the multimodal emotion recognition model is iteratively trained to convergence, the loss value is calculated based on the cross entropy loss function, the initial learning rate is set to 0.0001, and the Adam optimizer in pytorch is used for forward propagation to update the model parameters in the multimodal emotion recognition model; The expression of the cross entropy loss function is: loss=CrossEntropyLoss(pred_label, real_label); Among them, pred_label represents the emotion category recognition result of the multimodal sample data output by the multimodal emotion recognition model, real_label represents the emotion label annotated on the multimodal sample data, CrossEntropyLoss(·) represents the cross entropy loss function, and loss represents the calculated loss value.

8. A multimodal emotion recognition system, characterized in that: include: An acquisition module is used to acquire multimodal sample data through a millimeter wave radar and annotate the multimodal sample data with emotion labels, wherein the multimodal sample data includes speech sample data and facial image sample data when the sample object speaks with different emotions, and text sample data obtained by converting the speech sample data through a preset speech-to-text recognition model; A construction module is used to construct a multimodal emotion recognition model, wherein the multimodal emotion recognition model is composed of an image feature extraction branch network, a speech feature extraction branch network, a text feature extraction branch network, a fusion network, a residual network and a discriminant network, the image feature extraction branch network is used to extract image features from image sample data and obtain a first hidden state vector corresponding to the image features, the speech feature extraction branch network is used to extract speech features from speech sample data and obtain a second hidden state vector corresponding to the speech features, the text feature extraction branch network is used to extract text features from text sample data and obtain a third hidden state vector corresponding to the text features, the fusion network is used to fuse image features, speech features and text features, the residual network is used to perform residual connection on the first hidden state vector, the second hidden state vector, the third hidden state vector and the output of the fusion network to obtain the final features, and the discriminant network is used to classify and discriminate the final features to obtain the emotion category recognition result of the multimodal sample data; A training module is used to iteratively train the multimodal emotion recognition model until convergence based on the emotion category recognition results of the multimodal sample data output by the multimodal emotion recognition model and the emotion labels annotated on the multimodal sample data, so as to obtain a trained multimodal emotion recognition model; A module is used to obtain multimodal data of an object to be identified through a millimeter wave radar, input the multimodal data of the object to be identified into a trained multimodal emotion recognition model, and obtain an emotion category recognition result of the object to be identified output by the trained multimodal emotion recognition model.

9. An electronic device, characterized in that: include: at least one processor; and, a memory communicatively coupled to the at least one processor; Wherein, the memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute a multimodal emotion recognition method as described in any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, it can implement a multimodal emotion recognition method as described in any one of claims 1 to 7.

Images