Noise enhancement multi-mode emotion recognition method based on auxiliary network

Abstract

The invention belongs to the field of image signal processing, and relates to a noise enhancement multi-mode emotion recognition method based on an auxiliary network, which comprises the following steps of: inputting a video into a feature extraction and random noise enhancement module to obtain an enhanced audio feature Xa and an enhanced image feature Xv; inputting the features Xa and Xv into a feature interaction and fusion network to obtain a fusion feature; inputting the features Xa and Xv into an audio auxiliary network and an image auxiliary network to obtain an audio auxiliary feature # imgabs0 # and an image auxiliary feature # imgabs1 #, combining the fusion feature, the feature # imgabs2 # and the feature # imgabs3 #, and inputting the combined features, the feature # imgabs2 # and the feature # imgabs3 # into a classifier to obtain an emotion recognition result; updating model parameters according to an emotion recognition result until a trained multi-modal emotion recognition model is obtained; according to the invention, the feature interaction and fusion network shares parameters with the Transform of the auxiliary audio network and the auxiliary image network, so that the focus of the feature interaction and fusion network can be balanced; the feature interaction and fusion network disclosed by the invention realizes deep interaction between audio features and image features.

Description

Technical Field

[0001] The present invention belongs to the field of image signal processing and relates to a noise-enhanced multimodal emotion recognition method based on an auxiliary network. Background Art

[0002] With the rapid development of artificial intelligence, the interaction between artificial intelligence products and humans is becoming more and more common. If we can accurately identify and understand the user's emotional state, we can adjust the interaction strategy in time and improve the user's satisfaction with artificial intelligence products. In addition, multimodal emotion recognition technology not only helps to improve the user's interactive experience, but also has important application value in many fields such as medical care, education, and entertainment.

[0003] The task of detecting a person's emotional state can be regarded as a type of multimodal emotion recognition. From the perspective of the model, it is to quickly classify a person's emotional state through a classification task model. Traditional emotion recognition methods mainly rely on people's intuitive judgment, which is not only time-consuming and labor-intensive, but also cannot guarantee accuracy. Therefore, how to use artificial intelligence technology, especially deep learning technology, to perform multimodal emotion recognition analysis has become an urgent problem to be solved. However, existing multimodal emotion recognition methods have problems such as low utilization of multimodal feature interaction, low recognition accuracy, and insufficient robustness.

[0004] Most of the existing multimodal emotion recognition methods are papers, such as "Exploring Complementary Features in Multi-Modal Speech Emotion Recognition" published in ICASSP in 2023. The paper first extracts audio feature information and text feature information through the audio feature extraction module and the text feature extraction module respectively. After that, the audio feature information and the text feature information are simply spliced ​​together to obtain the spliced ​​features, and the spliced ​​features are sent to the emotion prediction model to obtain the predicted emotion information; this type of method simply splices the extracted audio features with the facial image features, and then sends the spliced ​​features to the model for emotion prediction. This simple splicing method not only ignores the deep emotional information inside the audio features and the image features, but also ignores the potential complex emotional interaction information between the audio features and the image features, and these emotional information have a significant impact on the final emotion prediction results. The paper "AuxFormer Robust Approach to Audiovisual Emotion Recognition" published in ICASSP in 2022 first separates audio information and image information from video files, and then uses a pre-trained model to obtain audio features and facial image features. The model is then divided into three branches. The first branch is the audio emotion prediction model, which only uses audio features for emotion prediction. The second branch is the fusion network, which uses the attention mechanism to interactively fuse audio features and image features, and predicts emotion on the fused features. The third branch is the image emotion prediction model, which only uses image features for emotion prediction. Finally, the predicted emotion results of the three branches are averaged to obtain the output emotion. This type of method usually adds a fusion network and an auxiliary network, but in the fusion network, it simply uses the Attention mechanism to interact between audio features and image features. This interaction method does not take into account the deep interaction information between different modal features. And because there is a large heterogeneity between audio features and image features, feature information will be lost during the interactive fusion process. In addition, the two auxiliary networks (the unimodal audio emotion prediction network and the image prediction network) are only used for emotion prediction and do not interact with the fusion network. Finally, the predicted emotion states output by the three networks are averaged to obtain the final predicted emotion. This method is somewhat subjective because the emotion states predicted by the audio emotion prediction network, the image emotion prediction network and the fusion network all have certain deviations. Using the average method to calculate the emotion state will result in large errors.

[0005] In existing methods, noise-free, loss-free audio and image data in the data set are used for model training and prediction. However, in actual scenarios, the image and audio data collected by cameras and microphones may contain certain noise data or lost data.

[0006] In summary, the existing technology fails to take into account the deep interactive information between different modal features and the loss of feature information during the interactive fusion of modal features, and has poor robustness and practicality. Summary of the invention

[0007] In order to solve the above-mentioned prior art problems, the present invention adopts a noise-enhanced multimodal emotion recognition method based on an auxiliary network, comprising:

[0008] Obtain a video file, input the video file into a trained multimodal emotion recognition model, and obtain an emotion recognition result; the multimodal emotion recognition model includes: a feature extraction and random noise enhancement module, a feature interaction and fusion network, an auxiliary audio network, an auxiliary image network, and a classifier;

[0009] The training process of the multimodal emotion recognition model includes:

[0010] S1. Obtain a video file, input the video file into a feature extraction and random noise enhancement module, and obtain enhanced audio features and enhanced image features;

[0011] S2, inputting the enhanced audio features and the enhanced image features into the feature interaction and fusion network to obtain fusion features;

[0012] S3, inputting the enhanced audio features and the enhanced image features into the audio auxiliary network and the image auxiliary network respectively to obtain audio auxiliary features and image auxiliary features;

[0013] S4, combining the fusion features, the audio auxiliary features and the image auxiliary features, and inputting the combined features into the classifier to obtain the emotion recognition result;

[0014] S5. Calculate the loss function value according to the emotion recognition result, and update the model parameters according to the loss function value. When the loss function value is minimum, a trained multimodal emotion recognition model is obtained.

[0015] The feature extraction and random noise enhancement module includes: a feature extraction module and a random noise enhancement module; the feature extraction and random noise enhancement module processes the video file including: inputting the video file into the feature extraction module to obtain initial audio features and initial image features; inputting the initial audio features and initial image features into the random noise enhancement module respectively to obtain enhanced audio features and enhanced image features.

[0016] The video file includes a video frame sequence and a speech signal; the feature extraction module includes: MTCNN, EfficientFace pre-trained model and audio feature extraction module; the feature extraction module processes the video file including: selecting video frames from the video frame sequence, inputting the selected video frames into MTCNN, obtaining video frames containing face bounding boxes, cropping the video frames containing face bounding boxes to obtain face images, inputting the face images into the EfficientFace pre-trained model, and obtaining initial image features; inputting the speech signal into the audio feature extraction module to obtain initial audio features; wherein MTCNN is a multi-task cascade convolutional neural network and EfficientFace is efficient face.

[0017] The random noise enhancement module processes the initial audio features and the initial image features respectively, including: adding random noise data to the initial audio features and the initial image features respectively to obtain the first audio features and the first image features; obtaining empty features, combining the empty features with the initial audio features and the initial image features respectively to obtain the second audio features and the second image features; splicing the first audio features, the second audio features and the initial audio features to obtain enhanced audio features; splicing the first image features, the second image features and the initial image features to obtain enhanced image features.

[0018] The feature interaction and fusion network includes: an audio branch, an image branch, a cross-attention module and a global average pooling layer; the feature interaction and fusion network processes enhanced audio features and enhanced image features including: inputting the enhanced audio features and enhanced image features into the audio branch and the image branch to obtain audio interaction features and image interaction features; inputting the audio interaction features and the image interaction features into the cross-attention module, and inputting the output of the cross-attention module into the global average pooling layer to obtain fusion features.

[0019] The audio branch includes: an audio multi-head attention module and an audio Transformer encoding module; the image branch includes: an image multi-head attention module and an image Transformer encoding module; the audio branch and the image branch process the enhanced audio features and the enhanced image features, including: inputting the enhanced audio features and the enhanced image features into the audio multi-head attention module and the image multi-head attention module, and inputting the outputs of the audio multi-head attention module and the image multi-head attention module into the audio Transformer encoding module and the image Transformer encoding module to obtain audio interaction features and image interaction features.

[0020] The audio multi-head attention module processes the enhanced audio features and enhanced image features by: a As the key vector and value vector, the image features X will be enhanced vAs the query vector, we get the output X of the audio multi-head attention module based on the key vector, value vector, and query vector. a ′; The image multi-head attention module processes the enhanced audio features and enhanced image features by: a As the query vector, the image feature X v As the key vector and value vector, the output X of the image multi-head attention module is obtained according to the key vector, value vector and query vector. v ′.

[0021] The audio Transformer encoding module and the image Transformer encoding module both include: multiple encoding layers connected in series; each encoding layer includes: a multi-head self-attention layer and a feedforward neural network;

[0022] The first encoding layer of the audio Transformer encoding module processes the outputs of the audio multi-head attention module and the image multi-head attention module, including: using the output of the audio multi-head attention module as the key vector and value vector of the first encoding layer, using the output of the image multi-head attention module as the query vector of the first encoding layer, obtaining the output of the multi-head self-attention layer of the first encoding layer according to the key vector, value vector and query vector, and inputting the output of the multi-head self-attention layer into the feedforward neural network of the first encoding layer to obtain the output of the first encoding layer.

[0023] The first encoding layer of the image Transformer encoding module processes the outputs of the audio multi-head attention module and the image multi-head attention module, including: using the output of the image multi-head attention module as the key vector and value vector of the first encoding layer, using the output of the audio multi-head attention module as the query vector of the first encoding layer, obtaining the output of the multi-head self-attention layer of the first encoding layer according to the key vector, value vector and query vector, and inputting the output of the multi-head self-attention layer into the feedforward neural network of the first encoding layer to obtain the output of the first encoding layer.

[0024] Both the audio-assisted network and the image-assisted network include: a fully connected layer, a Transformer encoding module, and a global average pooling layer;

[0025] The audio auxiliary network processes the enhanced audio features by: inputting the enhanced audio features into a fully connected layer, inputting the output of the fully connected layer into a Transformer encoding module, and inputting the output of the Transformer layer into a global average pooling layer to obtain audio auxiliary features;

[0026] The image-assisted network processes the enhanced image features by: inputting the enhanced image features into the fully connected layer, inputting the output of the fully connected layer into the Transformer encoding module, and inputting the output of the Transformer layer into the global average pooling layer to obtain the image-assisted features.

[0027] The Transformer encoding module of the audio-assisted network shares parameters with the audio Transformer encoding module, and the Transformer encoding module of the image-assisted network shares parameters with the image Transformer encoding module.

[0028] Beneficial effects:

[0029] 1. The present invention performs noise injection and feature discarding in the training stage to simulate the noise interference and modal loss that may occur in the real environment, so as to improve the robustness and practicality of the model; 2. The present invention proposes ShareTransformer, which can utilize the unimodal feature information in the auxiliary network to balance the focus of the Transformer in the feature interaction network, so as to avoid the model paying too much attention to the interaction information and ignoring other important emotional information, thereby obtaining feature representations containing rich emotional information; 3. The present invention proposes a new cross-modal interaction fusion method, which uses Multi-Head Attention and Transformer to realize the deep interaction between audio features and image features, and uses Cross-Attention to realize the fusion of audio features and image features; this method can fully explore and utilize the complex interaction information between different modalities to obtain fusion features containing rich emotional information. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1 An architecture diagram of a noise-enhanced multimodal emotion recognition method based on an auxiliary network provided in an embodiment of the present invention;

[0031] Figure 2 A comparison result diagram of the RAVDESS dataset provided by an embodiment of the present invention;

[0032] Figure 3 This is a diagram of the ablation experiment results of the present invention on the RAVDESS dataset provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0033] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0034] like Figure 1 As shown, the embodiment of the present invention adopts a noise-enhanced multimodal emotion recognition method based on an auxiliary network, including: obtaining a video, inputting the video into a trained multimodal emotion recognition model, and obtaining an emotion recognition result; the multimodal emotion recognition model includes: a feature extraction and random noise enhancement module, a feature interaction and fusion network, an audio auxiliary network, an image auxiliary network, and a classifier;

[0035] The training process of the multimodal emotion recognition model includes:

[0036] S1. Obtain a video file, input the video file into a feature extraction and random noise enhancement module, and obtain enhanced audio features and enhanced image features;

[0037] The feature extraction and random noise enhancement module includes: a feature extraction module and a random noise enhancement module. The video file includes a video frame sequence and a speech signal.

[0038] The feature extraction module extracts image features by: evenly selecting a certain number of video frames from the video frame sequence, using MTCNN to perform face recognition on each selected video frame, and once a face is detected, cropping the frame according to the detected face bounding box to obtain the face image in the video frame; using the EfficientFace pre-trained model to convert each face image into a corresponding feature; stacking all face features to obtain the initial image features; where MTCNN is a multi-task cascade convolutional neural network and EfficientFace is an efficient face.

[0039] The feature extraction module extracts audio features including: using a Hamming window to divide the speech signal into multiple 46ms short frames, and the step size of the Hamming window is 12ms; the Hamming window is a commonly used window function, which can effectively reduce the edge effect of the signal; performing a short-time Fourier transform (STFT) on each short frame to obtain the spectrum of each short frame; performing a Mel filter on the spectrum of each short frame, taking the logarithm of the result after Mel filtering of each short frame, and performing a discrete cosine transform on the result after taking the logarithm of each short frame to obtain the MFCC feature of each short frame; stacking the MFCC features of each short frame to obtain the current initial audio feature; wherein MFCC is the Mel frequency cepstrum coefficient.

[0040] In the process of collecting audio and image data, noise interference and data loss are inevitable phenomena. However, most of the current methods do not fully consider these situations, which leads to a problem: when the model encounters data containing noise interference or modality loss, its emotion recognition effect is usually poor.

[0041] Therefore, in the training strategy, the present invention uses a random noise enhancement module to inject noise into the modal features and discard features to simulate the noise interference and data loss that may occur in the actual environment. This strategy allows the model to gradually adapt to these uncertainties during the training process, thereby improving its robustness in the face of noise interference and modal loss. The specific steps are as follows:

[0042] Adding random noise data to the initial audio features and image features to obtain first audio features and first image features, so as to simulate the situation where noise interference exists in both the audio and image data;

[0043] The empty feature is combined with the initial image feature to obtain the second image feature to simulate the situation when the audio data is lost and only the image data is available; the empty feature is combined with the initial audio feature to obtain the second audio to simulate the situation when the image data is lost and the audio data exists;

[0044] The first audio feature, the second audio feature and the initial audio feature are concatenated to obtain an enhanced audio feature; the first image feature, the second image feature and the initial image feature are concatenated to obtain an enhanced image feature.

[0045] S2, inputting the enhanced audio features and the enhanced image features into the feature interaction and fusion network to obtain fusion features;

[0046] The feature interaction and fusion network is the core component of the model. Its main task is to achieve deep interaction and fusion between audio features and image features to obtain fusion features containing rich emotional information. In order to achieve this goal, the present invention adopts a multi-head attention module (Multi-Head Attention), a Transformer encoding module and a cross-attention module (Cross-Attention) to achieve deep interaction and fusion between audio features and image features.

[0047] The feature interaction and fusion network includes: audio branch, image branch, cross attention module and global average pooling layer; the audio branch includes: audio multi-head attention module and audio Transformer encoding module; the image branch includes: image multi-head attention module and image Transformer encoding module; the feature interaction and fusion network processes enhanced audio features and enhanced image features including:

[0048] The feature interaction and fusion network processes the enhanced audio features and enhanced image features including:

[0049] S21, inputting the enhanced audio features and the enhanced image features into the audio branch and the image branch to obtain audio interaction features and image interaction features;

[0050] The audio branch and the image branch process the enhanced audio features and enhanced image features including:

[0051] S211, inputting the enhanced audio features and the enhanced image features into the audio multi-head attention module and the image multi-head attention module;

[0052] The audio multi-head attention module processes the enhanced audio features and enhanced image features by: a As the key vector and value vector, the image features X will be enhanced v As the query vector, the output X of the audio multi-head attention module is obtained according to the key vector, value vector and query vector a ′;

[0053] Similarly, the image multi-head attention module processes the enhanced audio features and enhanced image features by: a As a query, the enhanced image feature X v As the key and value, the output X of the image multi-head attention module is obtained according to the key vector, value vector and query vector v ′.

[0054] The principle of Multi-Head Attention is that it calculates a similarity matrix by comparing the similarity between the features of another modality and the features of the current modality. This similarity matrix reveals the parts of the current modality features that are highly correlated with the features of another modality. Then, by calculating this similarity matrix with the current modality features, the model can be guided to focus on these highly correlated features, thereby achieving effective interaction between audio features and image features. And this interaction method will not cause strong mutual dependence between audio features and image features, ensuring that during the interaction process, the features of each modality can retain their original information to avoid the loss of feature information. The formula for Multi-Head Attention is as follows:

[0055] MHA(Q,K,V)=[head1,...,head h ]W0

[0056]

[0057] Where h represents the number of headers, h=4, i represents the index of the header, Q, K and V represent the query vector, key vector and value vector respectively, and W i Q , W i K , W iV is the weight matrix of the attention head, W0 is the output weight matrix, d k Represents the dimension of K.

[0058] In one embodiment, the audio multi-head attention module and the image multi-head attention module that implement feature interaction between audio features and image features can be replaced by non-attention mechanisms and gating mechanisms.

[0059] S212. Input the outputs of the audio multi-head attention module and the image multi-head attention module into the audio Transformer encoding module and the image Transformer encoding module to obtain audio interaction features and image interaction features.

[0060] After the processing of Multi-Head Attention, those feature information with strong correlation has been highlighted in the audio features and image features. However, these interactions are not deep enough and only reveal the correlation between audio features and image features at the surface level. In order to further explore and understand these interactive features, this embodiment introduces a Transformer encoding module, which includes multiple encoding layers; each encoding layer includes: a multi-head self-attention layer and a feedforward neural network; the Transformer encoding module can capture the long-distance dependencies in the input features through the multi-head attention mechanism, thereby obtaining richer and deeper emotional information. The feedforward neural network (FFNN) can be regarded as a nonlinear transformation, which can further extract and integrate the feature information in the attention layer to obtain a deeper feature representation. Therefore, by combining the attention mechanism and FFNN, the Transformer encoding module can achieve deep feature interaction, thereby extracting richer and deeper emotional features; wherein, FFNN is composed of multiple fully connected layers, and in the Transformer encoding module, each self-attention layer is followed by a FFNN.

[0061] Specifically, in the audio Transformer encoding module, X′ a As the key and value of the first encoding layer, X′ v As the query of the first encoding layer, we get the deep audio interaction feature X″ a Similarly, in the image Transformer encoding module, X′ v As the key and value of the first encoding layer, X′ a As the query of the first encoding layer, we get the deep image interaction feature X″ v . This process can be briefly expressed as follows:

[0062] X″ a =FFNN(MHA(X′ v ,X′ a ,X′ a ))

[0063] X″ v =FFNN(MHA(X′ a ,X′ v ,X′ v )

[0064] Among them, FFNN stands for feedforward neural network and MHA stands for Multi-Head Attention.

[0065] Although the core component of the Transformer encoding module is Multi-Head Attention, it has different goals and functions from the audio multi-head attention module and the image multi-head attention module. Due to the significant heterogeneity between audio features and image features, if they are directly sent to the Transformer, the feature information that the model focuses on may not meet expectations. For these unexpected feature information, if they are deeply understood by the feedforward neural network, they may have a negative impact on the prediction effect of the model. Therefore, an additional Multi-Head Attention mechanism is introduced before the Transformer. This mechanism can perform preliminary feature interactions on the original audio features and image features from multiple angles, highlight the similar parts of the features, and thus alleviate the heterogeneity between audio features and image features. Then, sending these preliminarily interacted features into the Transformer can guide the Transformer to focus on the feature information we expect and deepen the understanding of this important information, thereby improving the model effect.

[0066] S22. Input the audio interaction features and the image interaction features into the cross-attention module, and input the output of the cross-attention module into the global average pooling layer to obtain the fusion features.

[0067] After the feature interaction of Multi-Head Attention and Transformer, the audio interaction features and image interaction features contain rich interaction information. These interaction information often contain a lot of emotional information. If these two features are simply spliced ​​together directly, these important interaction information may be ignored. In order to make full use of this interaction information, this embodiment adopts the cross-attention mechanism to realize the fusion of audio interaction features and image interaction features. In the fusion process, the audio interaction features are selected as the key and value, and the image interaction features are selected as the query, because the audio interaction features contain more emotional information, such as intonation, speaking speed, etc. In this way, fusion features containing rich emotional information can be obtained. The specific process is simplified as follows:

[0068]

[0069] Among them, W q , W v represents the parameter matrix, X″ v Represents the image interaction feature, X″ a represents the audio interaction feature, X f Represents the output of the criss-cross attention module.

[0070] S3: Input the enhanced audio features and enhanced image features into the audio auxiliary network and the image auxiliary network respectively to obtain the audio auxiliary features. and image-assisted features

[0071] In the process of feature fusion and interaction, some feature information may be lost. However, since the auxiliary network does not perform feature interaction and fusion, it can retain complete single-modal information and make up for the feature information lost in the process of feature interaction and fusion.

[0072] The auxiliary network has a similar structure, consisting of a fully connected layer, a Transformer encoding module, and a global average pooling layer. The fully connected layer is mainly responsible for feature extraction, feature conversion, and feature dimension adjustment. It can map the input features to a new feature space to adapt to the processing of the subsequent Transformer encoding module. In addition, the fully connected layer can also enhance the nonlinearity of the model, allowing the model to learn more complex feature representations. The main function of the Transformer encoding module is to obtain higher-level feature representations.

[0073] In the feature interaction and fusion network, audio features and image features interact deeply through Transformer, which causes the model to pay too much attention to the interaction information between modalities and ignore the important emotional information contained in the current modal features. To solve this problem, Shared Transformer is proposed. The Transformer encoding module of the audio auxiliary network shares parameters with the audio Transformer encoding module, and the Transformer encoding module of the image auxiliary network shares parameters with the image Transformer encoding module. Shared Transformer utilizes the single-modal features in the auxiliary network to balance the focus of the Transformer in the feature interaction and fusion network, thereby obtaining feature representations containing more emotional information.

[0074] In one embodiment, the audio auxiliary network (Aux network) and the audio branch in the feature interaction and fusion network use the same Transformer encoding module, and the image auxiliary network and the image branch in the feature interaction and fusion network use the same Transformer encoding module, so as to achieve the effect of Shared Transformer.

[0075] S4, combining the fusion features, the audio auxiliary features and the image auxiliary features, and inputting the combined features into the classifier to obtain the emotion recognition result;

[0076] S5. Calculate the loss function value according to the emotion recognition result, and update the model parameters according to the loss function value. When the loss function value is minimum, a trained multimodal emotion recognition model is obtained.

[0077] This embodiment uses the cross entropy loss function, and the formula is as follows:

[0078]

[0079] in, Indicates that the i-th video file belongs to the true category y i The probability of, N represents the total number of video files, L represents the cross entropy loss, x ic represents the normalized prediction score of the i-th video for the c-th category, x ij represents the normalized prediction score of the i-th video file for the j-th category, p ic represents the probability that the i-th video file belongs to the c-th category, and for the i-th video file, all p ic The sum is 1, and C is the number of categories.

[0080] like Figure 2As shown in the figure, the comparative experimental results of the model proposed in the present invention and other methods on the RAVDESS dataset (emotional speech and facial expression dataset), among which [1] is the method proposed in "A multimodal facial emotion recognition framework through the fusion of speech with visible and infrared images, multimodal technol" in 2021, [2] is the method proposed in "A proposal for multimodal motion recognition using aural transformers and action units on ravdess dataset" in 2021, [3] is the method proposed in "Deep learning based multimodal emotion recognition using model-level fusion of audio-visual modalities" in 2022, [4] is the method proposed in "Emotion Recognition from Audio-Visual Information based on Convolutional Neural Network" in 2023, [5] is the method proposed in "Multimodal emotion recognition using cross modal audio-video fusion with attention and deepmetric learning" in 2023, and [6] is the method proposed in "AttA-NET Attention" in 2024. Aggregation Network for Audio-Visual Emotion Recognition". According to the experimental results, the multimodal emotion recognition model proposed in the embodiment of the present invention achieves an accuracy of 89.78% (89.78% is the average of 5 cross-validation experimental results, and the highest result is 92.22%), achieving excellent experimental results.

[0081] like Figure 3As shown, the results of the ablation experiments on various modules of the multimodal emotion recognition model adopted in the embodiment of the present invention are shown; after removing (w / o) random noise Random Nosie, the model accuracy (Accuracy) drops from 92.22% to 88.33%; after removing Share Transformer, the model accuracy drops from 92.22% to 83.89%, a significant drop; the proposed method * indicates that in the cross-attention mechanism, audio interaction features are used as query vectors, and image interaction features are used as key vectors and value vectors. This method pays more attention to image interaction features, but the image interaction features contain less emotional information than audio features, so the accuracy is slightly lower. The method proposed in the present invention indicates that in the cross-attention mechanism, audio features are used as key vectors and value vectors, and image interaction features are used as query vectors, so as to pay more attention to audio features and obtain better results.

[0082] The above embodiments further illustrate the purpose, technical solutions and advantages of the present invention in detail. It should be understood that the above embodiments are only preferred implementation modes of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made to the present invention within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A noise-enhanced multimodal emotion recognition method based on an auxiliary network, characterized in that: include: Obtain a video file, input the video file into a trained multimodal emotion recognition model, and obtain an emotion recognition result; The multimodal emotion recognition model includes: feature extraction and random noise enhancement module, feature interaction and fusion network, auxiliary audio network, auxiliary image network and classifier; The training process of the multimodal emotion recognition model includes: S1. Obtain a video file, input the video file into a feature extraction and random noise enhancement module, and obtain enhanced audio features and enhanced image features; S2, inputting the enhanced audio features and the enhanced image features into the feature interaction and fusion network to obtain fusion features; S3, inputting the enhanced audio features and the enhanced image features into the audio auxiliary network and the image auxiliary network respectively to obtain audio auxiliary features and image auxiliary features; S4, combining the fusion features, the audio auxiliary features and the image auxiliary features, and inputting the combined features into the classifier to obtain the emotion recognition result; S5. Calculate the loss function value according to the emotion recognition result, and update the model parameters according to the loss function value. When the loss function value is minimum, a trained multimodal emotion recognition model is obtained.

2. According to claim 1, a noise-enhanced multimodal emotion recognition method based on auxiliary network is characterized in that: The feature extraction and random noise enhancement module includes: a feature extraction module and a random noise enhancement module; the feature extraction and random noise enhancement module processes the video file including: inputting the video file into the feature extraction module to obtain initial audio features and initial image features; inputting the initial audio features and initial image features into the random noise enhancement module respectively to obtain enhanced audio features and enhanced image features.

3. According to claim 2, a noise-enhanced multimodal emotion recognition method based on auxiliary network is characterized in that: The video file includes a video frame sequence and a speech signal; the feature extraction module includes: MTCNN, EfficientFace pre-trained model and audio feature extraction module; the feature extraction module processes the video file including: selecting video frames from the video frame sequence, inputting the selected video frames into MTCNN, obtaining video frames containing face bounding boxes, cropping the video frames containing face bounding boxes to obtain face images, inputting the face images into the EfficientFace pre-trained model, and obtaining initial image features; inputting the speech signal into the audio feature extraction module to obtain initial audio features; wherein MTCNN is a multi-task cascade convolutional neural network and EfficientFace is efficient face.

4. The noise-enhanced multimodal emotion recognition method based on auxiliary network according to claim 3 is characterized in that: The random noise enhancement module processes the initial audio features and the initial image features respectively, including: adding random noise data to the initial audio features and the initial image features respectively to obtain the first audio features and the first image features; obtaining empty features, combining the empty features with the initial audio features and the initial image features respectively to obtain the second audio features and the second image features; splicing the first audio features, the second audio features and the initial audio features to obtain enhanced audio features; splicing the first image features, the second image features and the initial image features to obtain enhanced image features.

5. The noise-enhanced multimodal emotion recognition method based on auxiliary network according to claim 1, characterized in that: The feature interaction and fusion network includes: an audio branch, an image branch, a cross-attention module and a global average pooling layer; the feature interaction and fusion network processes enhanced audio features and enhanced image features including: inputting the enhanced audio features and enhanced image features into the audio branch and the image branch to obtain audio interaction features and image interaction features; inputting the audio interaction features and the image interaction features into the cross-attention module, and inputting the output of the cross-attention module into the global average pooling layer to obtain fusion features.

6. The noise-enhanced multimodal emotion recognition method based on auxiliary network according to claim 5, characterized in that: The audio branch includes: an audio multi-head attention module and an audio Transformer encoding module; the image branch includes: an image multi-head attention module and an image Transformer encoding module; the audio branch and the image branch process the enhanced audio features and the enhanced image features, including: inputting the enhanced audio features and the enhanced image features into the audio multi-head attention module and the image multi-head attention module, and inputting the outputs of the audio multi-head attention module and the image multi-head attention module into the audio Transformer encoding module and the image Transformer encoding module to obtain audio interaction features and image interaction features.

7. The noise-enhanced multimodal emotion recognition method based on auxiliary network according to claim 6, characterized in that: The audio multi-head attention module processes the enhanced audio features and enhanced image features by: a As the key vector and value vector, the image features X will be enhanced v As the query vector, we get the output X of the audio multi-head attention module based on the key vector, value vector, and query vector. a ′; The image multi-head attention module processes the enhanced audio features and enhanced image features by: a As the query vector, the image feature X v As the key vector and value vector, the output X of the image multi-head attention module is obtained according to the key vector, value vector and query vector. v ′.

8. The noise-enhanced multimodal emotion recognition method based on auxiliary network according to claim 6, characterized in that: The audio Transformer encoding module and the image Transformer encoding module both include: multiple encoding layers connected in series; each encoding layer includes: a multi-head self-attention layer and a feedforward neural network; The first encoding layer of the audio Transformer encoding module processes the outputs of the audio multi-head attention module and the image multi-head attention module, including: using the output of the audio multi-head attention module as the key vector and value vector of the first encoding layer, using the output of the image multi-head attention module as the query vector of the first encoding layer, obtaining the output of the multi-head self-attention layer of the first encoding layer according to the key vector, value vector and query vector, and inputting the output of the multi-head self-attention layer into the feedforward neural network of the first encoding layer to obtain the output of the first encoding layer. The first encoding layer of the image Transformer encoding module processes the outputs of the audio multi-head attention module and the image multi-head attention module, including: using the output of the image multi-head attention module as the key vector and value vector of the first encoding layer, using the output of the audio multi-head attention module as the query vector of the first encoding layer, obtaining the output of the multi-head self-attention layer of the first encoding layer according to the key vector, value vector and query vector, and inputting the output of the multi-head self-attention layer into the feedforward neural network of the first encoding layer to obtain the output of the first encoding layer.

9. The noise-enhanced multimodal emotion recognition method based on auxiliary network according to claim 6, characterized in that: Both the audio-assisted network and the image-assisted network include: a fully connected layer, a Transformer encoding module, and a global average pooling layer; The audio auxiliary network processes the enhanced audio features by: inputting the enhanced audio features into a fully connected layer, inputting the output of the fully connected layer into a Transformer encoding module, and inputting the output of the Transformer layer into a global average pooling layer to obtain audio auxiliary features; The image-assisted network processes the enhanced image features by: inputting the enhanced image features into the fully connected layer, inputting the output of the fully connected layer into the Transformer encoding module, and inputting the output of the Transformer layer into the global average pooling layer to obtain the image-assisted features.

10. The noise-enhanced multimodal emotion recognition method based on auxiliary network according to claim 9, characterized in that: The Transformer encoding module of the audio-assisted network shares parameters with the audio Transformer encoding module, and the Transformer encoding module of the image-assisted network shares parameters with the image Transformer encoding module.

Images