Emotion recognition method, system and equipment based on multi-modal adaptive fusion and storage medium

Abstract

The invention discloses an emotion recognition method, system and device based on multi-modal adaptive fusion and a storage medium, and relates to the technical field of artificial intelligence, and the method comprises the steps: selecting a pre-training model, respectively extracting the original features of an audio and a video, carrying out the preliminary extraction of the audio through a convolution layer, carrying out the multi-module processing of the video, and keeping the time sequence information. Constructing an attention module to generate an attention matrix and interaction features, and adjusting the original features by using the matrix; and inputting the weighted and fused features into a convolutional network to extract advanced time sequence features, performing pooling compression on the advanced time sequence features in a time dimension, splicing audio and video features, and finally sending the spliced audio and video features into a full-connection layer classifier to obtain an emotion classification result. According to the method, the weights of different features can be dynamically adjusted, so that the audio and visual features are effectively fused, the accuracy and robustness of emotion recognition are improved, the weighted recall rate and the unweighted recall rate are remarkably improved, and the method has high calculation efficiency and expandability.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to an emotion recognition method, system, device and storage medium based on multimodal adaptive fusion. Background Art

[0002] With the continuous development of artificial intelligence technology, emotion recognition, as a key research area, has demonstrated tremendous application value in a wide range of scenarios, including human-computer interaction, intelligent customer service, and psychological analysis. Multimodal emotion recognition methods combine information from multiple modalities, such as audio and vision, and offer greater accuracy and reliability than single-modality emotion recognition.

[0003] Most existing multimodal fusion methods rely on fixed rules or simple weighted averages to combine information from different modalities. While these methods are effective in certain scenarios, they lack sufficient flexibility and adaptability. Specifically, fixed rules and weighted averages fail to account for the varying importance of each modality in a given context. Consequently, they are unable to dynamically adjust the weights between modalities, resulting in decreased accuracy in emotion recognition in complex or changing environments. Summary of the Invention

[0004] In view of the above-mentioned problems, the present invention is proposed.

[0005] Therefore, the technical problem solved by the present invention is that the existing multimodal fusion emotion recognition method cannot dynamically adjust the modal weights because it does not fully consider the differences in the importance of each modality in different situations, cannot flexibly adjust the audio and video feature weights when complex emotional expressions are interfered with by noise or when the situation changes, and fails to effectively suppress redundant information, resulting in reduced accuracy, adaptability and robustness of emotion recognition.

[0006] In order to solve the above technical problems, the present invention provides the following technical solutions:

[0007] In a first aspect, an embodiment of the present invention provides an emotion recognition method based on multimodal adaptive fusion, comprising:

[0008] As a preferred solution for emotion recognition method based on multimodal adaptive fusion,

[0009] The extracting of audio and video features from the data set to obtain original audio features and original video features includes:

[0010] Select appropriate pre-trained models and use them to extract features from audio and video respectively;

[0011] For audio feature extraction, the convolutional layer in the selected pre-trained model is used to perform preliminary feature extraction on the audio data to obtain preliminary features of specific shapes.

[0012] As a preferred solution for emotion recognition method based on multimodal adaptive fusion,

[0013] Extracting features from the audio and video data sets to obtain original audio features and original video features further includes:

[0014] For video feature extraction, each frame of video is processed step by step through the convolutional layer, maximum pooling and feature extraction modules in the selected pre-trained model to extract local features of the image, downsample the features, and compress the spatial features of each frame into a global feature vector representing the overall information of the frame; the input frame sequence is adjusted to a specific shape through the reshaping operation, and the specific tags of each frame are extracted and connected in chronological order to form a time series with a specific shape containing video timing information as the original video feature.

[0015] As a preferred solution for emotion recognition method based on multimodal adaptive fusion,

[0016] The method of constructing an attention module based on the extracted audio features and visual features, and generating an attention matrix and interaction features through the attention module includes:

[0017] The audio and visual features are normalized and introduced into the multi-head attention mechanism. The similarity between the query vector and the key vector is calculated by dot product and scaled. The calculated similarity is normalized so that the sum of the attention weights of each query is 1, thereby obtaining the attention matrix.

[0018] The beneficial effects of this preferred technical solution are: normalization processing makes the feature data have a unified scale, which is conducive to the stability of subsequent calculations. The multi-head attention mechanism can focus on the relationship between features from different subspaces. By calculating similarity and generating an attention matrix, it can highlight the correlation between different features and capture the interactive information between multimodal data, providing a basis for more accurate fusion of audio and video features.

[0019] As a preferred solution for emotion recognition method based on multimodal adaptive fusion,

[0020] The original audio features and the original video features are adjusted by the generated attention matrix, and the adjusted original features include:

[0021] The multi-head attention output matrix is averaged across all heads, compressing the matrix dimension from a high-dimensional form containing multi-head information to a lower-dimensional form, fusing the multi-head attention information into an average value.

[0022] The original features of the current modality are element-wise multiplied using the attention weights from another modality to adjust the features of the current modality.

[0023] The beneficial effects of this preferred technical solution are: averaging the multi-head attention output matrix reduces data complexity while integrating multi-head attention information, enabling the model to more comprehensively capture cross-modal dependencies. Adjusting features through element-by-element multiplication strengthens important features and suppresses irrelevant ones, enabling the model to more flexibly and accurately learn multimodal information and improve emotion recognition performance.

[0024] As a preferred solution for emotion recognition method based on multimodal adaptive fusion,

[0025] The extracting of advanced temporal features from the weighted fused audio and video features includes:

[0026] The weighted audio features and visual features are respectively input into the convolutional network to further extract high-level time series features.

[0027] As a preferred solution for emotion recognition method based on multimodal adaptive fusion,

[0028] The process of compressing high-level temporal features and combining them for sentiment classification includes:

[0029] The high-level time series features are pooled in the time dimension, the time series features are aggregated into a single representation, the audio features and visual features after feature compression are spliced in the last dimension, the spliced features are sent to a classifier composed of fully connected layers for processing, and finally the classification results are returned.

[0030] The beneficial effects of this preferred technical solution are as follows: pooling operations are performed on the time dimension, aggregating time series features into a single representation, reducing the amount of data while retaining key information. Concatenating audio and video features fully integrates multimodal information, enabling the classifier to comprehensively consider audio and video features for sentiment classification. The classifier composed of fully connected layers has strong nonlinear mapping capabilities and can accurately classify sentiment based on the concatenated features, improving the reliability of sentiment recognition.

[0031] In a second aspect, an embodiment of the present invention provides an emotion recognition system based on multimodal adaptive fusion, comprising:

[0032] The original feature generation module is used to collect audio and visual data containing emotional expression information to form a data set, and extract features from the audio and video from the data set to obtain original audio features and original video features;

[0033] Attention matrix generation module, which is used to build an attention module based on the original audio features and the original video features, and generate attention matrix and interaction features through the attention module;

[0034] An adjustment module is used to adjust the original audio features and the original video features through the generated attention matrix to obtain adjusted original features;

[0035] The weighted fusion module is used to calculate the learnable weights and adaptively weight the interaction features with the adjusted original features through the learnable weights;

[0036] The temporal feature extraction module is used to extract high-level temporal features from the weighted fused audio and video features to obtain high-level temporal features;

[0037] The classification module is used to compress and concatenate high-level temporal features for sentiment classification.

[0038] In a third aspect, an embodiment of the present invention provides an electronic device, including:

[0039] memory and processor;

[0040] The memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions. When the one or more programs are executed by the one or more processors, the one or more processors implement the emotion recognition method based on multimodal adaptive fusion as described in any embodiment of the present invention.

[0041] In a fourth aspect, an embodiment of the present invention provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, implement the emotion recognition method based on multimodal adaptive fusion.

[0042] The present invention utilizes a multi-head attention mechanism and adaptive weighted fusion technology to dynamically adjust the weights of different features, effectively integrating audio and visual features to enhance the accuracy and robustness of emotion recognition. This method demonstrates significant improvements in both weighted and unweighted recall, overcoming the shortcomings of existing methods in feature fusion and emotion capture. It also reduces redundant information and noise, while also offering high computational efficiency and scalability. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0044] Figure 1 This is an overall flow chart of the emotion recognition method based on multimodal adaptive fusion provided by the present invention;

[0045] Figure 2 This is a model framework diagram of the emotion recognition method based on multimodal adaptive fusion provided by the present invention;

[0046] Figure 3 This is an adaptive weighted fusion module diagram of the emotion recognition method based on multimodal adaptive fusion provided by the present invention. DETAILED DESCRIPTION

[0047] To make the above-mentioned objects, features, and advantages of the present invention more clearly understood, the following detailed description of the specific embodiments of the present invention is given in conjunction with the accompanying drawings. It is obvious that the described embodiments are only part of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, all other embodiments obtained by ordinary persons in this field without creative work should fall within the scope of protection of the present invention.

[0048] Example 1, with reference to Figure 1 , which is the first embodiment of the present invention, provides an emotion recognition method based on multimodal adaptive fusion, including:

[0049] S1: Collect audio and visual data containing emotional expression information to form a dataset, extract features from the audio and video from the dataset, and obtain original audio features and original video features;

[0050] S2: Based on the original audio features and original video features, an attention module is constructed, and the attention matrix and interaction features are generated through the attention module;

[0051] S3: Adjust the original audio features and original video features through the generated attention matrix to obtain the adjusted original features;

[0052] S4: Calculate the learnable weights and adaptively weight the interaction features with the adjusted original features through the learnable weights;

[0053] S5: extracting high-level temporal features from the weighted fused audio and video features to obtain high-level temporal features;

[0054] S6: Compress and concatenate high-level temporal features for sentiment classification.

[0055] It should be noted that through steps S1-S6, features can be systematically extracted from the collected audio and visual data, interactive features can be generated using the attention module, and then adaptively weighted fusion can be performed in combination with learnable weights. Finally, a series of operations such as high-level temporal feature extraction, feature compression and splicing, and emotion classification are completed, forming a complete and efficient emotion recognition process. This method fully utilizes the data information of both audio and visual modalities, effectively solving the problem of insufficient information in single-modal emotion recognition methods, and improving the accuracy and robustness of emotion recognition. The introduction of the attention module can strengthen important features and suppress irrelevant features, helping the model to learn multimodal information more flexibly and accurately. The adaptive weighted fusion mechanism enables the model to dynamically adjust the weights of original features and interactive features according to actual conditions, enhancing the model's ability to cope with different scenarios and data.

[0056] Example 2, reference Figure 1-Figure 3 , which is an embodiment of the present invention, provides an emotion recognition method based on multimodal adaptive fusion based on the previous embodiment, including:

[0057] In this embodiment, the audio and visual data containing emotional expression information collected in step S1 to form a data set includes:

[0058] The collected data can come from public datasets or through custom collection, and the collected data should cover a variety of emotions.

[0059] After collecting the data of the two modalities, the data is preprocessed as follows:

[0060] Each audio file is trimmed or padded according to the specified target duration, so that the duration of each audio and video file is fixed and saved as a new audio file.

[0061] Extract specific frames from the video file, perform face detection and cropping, and then save the processed results as a video in .avi format and a NumPy group in .npy format.

[0062] Load the processed video and audio files, extract Mel-Frequency Cepstral Coefficient (MFCC) features from the audio signal, and apply spatial transformations (such as data augmentation) to the video data while performing feature transformations on the audio features to prepare the data for the final audio-visual emotion recognition task.

[0063] The input data is passed into Compose, where a series of transformations (such as cropping, rotation, flipping, etc.) are applied and noise is introduced to enhance the generalization ability of the model.

[0064] In this embodiment, in step S1, the features of the audio and video are extracted from the data set to obtain the original audio features and original video features, including:

[0065] Select appropriate pre-trained models and use them to extract features from audio and video respectively.

[0066] In a preferred embodiment, two publicly available pre-trained models based on VisionTransformer are selected: AudioMAE and MAE-Face, both of which follow the ViT-base architecture and therefore have the same depth.

[0067] The two convolutional layers in the AudioMAE model are used to preliminarily extract preliminary features from the audio data, whose feature shape is B×T×D; B is the batch size, T is the time length, and D is the dimension.

[0068] The spatial features of each frame are gradually extracted through the convolutional layer, maximum pooling and feature extraction modules in the MAE-Face model: in the spatial feature extraction process of each frame, the convolutional layer gradually extracts the local features of the image, and the maximum pooling operation downsamples the features to reduce the spatial dimension while retaining key information.

[0069] Global average pooling is used to compress the spatial features of each frame into a global feature vector, which represents the overall information of the frame.

[0070] The input frame sequence is reshaped into a shape of B×T×D through the view operation, that is, the [CLS] feature tag of each frame will become the global feature representation of each frame; then the [CLS] tag of each frame is extracted, and these tags are concatenated in chronological order to form a time series with a shape of B×T×D. This time series contains the global feature representation of each frame and retains the temporal information of the video.

[0071] In another possible implementation, in addition to using pre-trained models to extract features, manual features can also be combined, such as audio pitch, loudness, and timbre, and video color histograms and texture features. These manual features can be combined with features extracted by the deep learning model to enrich the feature information.

[0072] In addition to AudioMAE and MAE-Face, you can also choose other pre-trained models suitable for audio and video feature extraction, such as ResNet, VGG, etc. for video feature extraction, and Wav2Vec, HuBERT, etc. for audio feature extraction.

[0073] In this embodiment, in step S2, an attention module is constructed based on the extracted audio features and visual features, and the attention matrix and interaction features are generated by the attention module.

[0074] Normalize the audio feature A and the visual feature V to obtain A1 and V1, and then introduce the features of the two modalities as query, key, and value respectively into the multi-head attention mechanism to calculate the correlation between the inputs:

[0075] In the multi-head attention mechanism, the similarity between the query vector and the key vector is calculated by dot product, which is expressed as:

[0076]

[0077] Where q is the query vector, v is the key vector, and d is the dimension of the query vector and key vector.

[0078] It should be noted that this operation calculates the similarity between each query and all keys, and Scaling to avoid exploding gradients due to large values.

[0079] The calculated similarity attn(q,v) is normalized by the softmax function so that the sum of the attention weights of each query is 1, and the attention matrix a is finally obtained, which is expressed as:

[0080] a=softmax(attn(q,k))

[0081] The weighted output x is obtained by adding the weighted value vector v, which is expressed as:

[0082] x=a·v

[0083] Where v is the value vector associated with the key, and the attention weight determines the importance of each value to the output. The output x is projected into the output space for processing, and then regularized (DropOut) is performed to obtain the output x1.

[0084] The output x after attention calculation is regularized by the DropPath operation to improve the robustness of the model. The formula of the DropPath operation is as follows:

[0085]

[0086] Where p is the probability of dropping, and r is a binary random tensor indicating whether to keep the current path (1 for keeping, 0 for dropping). By multiplying the output with r and dividing by 1-p, we can ensure that the output strength of the model remains unchanged when dropping a path.

[0087] After the regularization operation, the output is further processed by layer normalization and a feed-forward neural network (MLP), which is expressed as:

[0088] x mlp=MLP(Norm2(x2))

[0089] It should be noted that Norm2 performs a second normalization on the input x, and MLP is a feedforward neural network consisting of multiple linear layers and activation function GELU.

[0090] Attention output x out and the MLP output x mlp By adding residual connections, information can be retained and transmitted, which can be expressed as:

[0091]

[0092] Among them, x out is the interactive feature output by the attention module.

[0093] It should be noted that this connection method enables the network to better capture the global dependency of the input, while further enhancing the expressive power of the model through MLP.

[0094] Apply the attention module to the original audio features and original video features to generate the attention matrix and interaction features: a v , v a , x av , x va .

[0095] Among them, a v and x av is the attention matrix and interaction features generated by audio based on vision, v a and x va It is the attention matrix and interaction features generated by vision based on audio.

[0096] In this embodiment, the original audio features and the original video features are adjusted using the attention matrix generated in step S3, and the adjusted original features include:

[0097] The output matrix a of the multi-head attention v and v a Average the output of all heads and change its dimension from (B, num_heads, N q ,N k ) is compressed to (B,1,N q ,N k ), which fuses the attention information of multiple heads into an average value to better capture cross-modal dependencies.

[0098] Among them, B represents the batch size, num_heads represents the number of attention heads, and N q and N kRepresent the feature dimension of the query feature and the feature dimension of the key respectively. q Compressed into a single feature representation, the resulting dimension becomes (B, 1, N k ). This step reduces the information of the time series dimension and retains the global attention weight.

[0099] Through cross-modal feature adjustment, the features of the current modality are adjusted by element-wise multiplication of the attention weights from another modality with the features of the current modality:

[0100] Use audio-to-visual attention weights v a Multiply the original audio features element-wise to adjust the features of the current modality:

[0101] Using vision-to-audio attention weights a v Multiply the original video features element-wise to adjust the features of the current modality:

[0102] It should be noted that by adjusting the original audio features and original video features through the generated attention matrix, important features can be further strengthened, irrelevant features can be suppressed, and the model can learn multimodal information more flexibly and accurately.

[0103] In this embodiment, calculating the learnable weight in step S4 includes:

[0104] like Figure 3 As shown, by the attention matrix a v and v a After averaging, the interaction features x are calculated using a learnable fully connected layer av and x va The weighted weights of the audio features are normalized by the sigmoid activation function to ensure that the weights are in the range of [0, 1]. Finally, the weighted weights w1 of the audio features and w2 of the visual features are obtained.

[0105] Expand the obtained weights from [B, 1] to [B, C, T], ensuring that they are consistent with the interaction features x av and x va have the same dimensions, enabling element-wise multiplication.

[0106] In this embodiment, the adaptive weighted fusion of the interactive features and the adjusted original features using the learnable weights in step S4 includes:

[0107] The visual features and audio features are weightedly fused and expressed as:

[0108] h av =A*w1+xav *(1-w1)

[0109] h va =V*w2+x va *(1-w2)

[0110] Among them, h av , h va The features after weighted fusion, w1, w2 are learnable weights with a value range of 0≤w1, w2≤1, and their sizes will be adjusted during training.

[0111] It should be noted that through weighted fusion, the weights of original features and interactive features can be dynamically adjusted according to actual needs, making the model more flexible and thus better able to cope with downstream tasks.

[0112] In another possible implementation, in addition to using a learnable fully connected layer to calculate weights, dynamic models such as recurrent neural networks (RNNs) and long short-term memory networks (LSTMs) can be used to dynamically generate weights based on input data. These models can consider the temporal information of the data and adjust weights more flexibly. The adaptive weighted fusion process can be divided into multiple stages, each using different weights for fusion. For example, coarse-grained weighted fusion can be performed first, followed by fine-grained weighted fusion, gradually improving the fusion effect.

[0113] In this embodiment, the extraction of high-level temporal features from the weighted fused audio and video features in step S5 includes:

[0114] The weighted audio features and visual features are input into the convolutional networks of the audio model and visual model respectively to further extract their high-level time series features.

[0115] It should be noted that through this process, the model can extract more abstract and semantic features from audio and visual inputs, providing more useful representations for subsequent tasks.

[0116] In another possible implementation, in addition to using convolutional networks to extract high-level temporal features, temporal models such as RNN, LSTM, and Gated Recurrent Unit (GRU) can also be used. These models can better process sequence data and capture the temporal dependencies of the data.

[0117] Multi-scale convolutional networks or time series models can be used to simultaneously extract high-level time series features at different scales. For example, convolution kernels of different sizes or RNNs with different time steps can be used, and then these features can be fused to improve the expressive power of the features.

[0118] In this embodiment, the step S6 of compressing the high-level temporal features and concatenating them for sentiment classification includes:

[0119] The obtained high-level audio and visual temporal features are pooled in the time dimension, and the time series features are aggregated into a single representation, thereby reducing the impact of the time dimension and retaining global information.

[0120] The further obtained audio features and visual features are concatenated in the last dimension to obtain a new tensor y, whose shape is (B, C a +C v ), which is the concatenated feature representation, contains audio and visual information. a and C v The number of channels for audio and vision respectively.

[0121] The concatenated feature y is sent to a classifier composed of a fully connected layer for processing, and the classification result y1 is finally returned.

[0122] In another possible implementation, training parameters may be set to train an emotion recognition model based on multimodal adaptive fusion and the model may be evaluated.

[0123] Specifically, the training parameters mainly include the number of expression categories, initial learning rate, learning rate decay rounds, learning rate decay ratio, learning rate automatic decay conditions, batch size, training rounds, optimizer and optimizer momentum size.

[0124] After setting the parameters, start training the model and perform a validity test on the trained model. Evaluate the model based on the loss rate and accuracy in the test results, and then adjust parameters such as training parameters and learning rate until the model achieves the optimal effect.

[0125] It should be noted that if Figure 2 As shown in the figure, through steps S1-S6, a complete emotion recognition model based on multimodal adaptive fusion is constructed. The training parameter settings cover all trainable parts of the ER-MAF model, including:

[0126] Attention weights and MLP layer parameters in the attention module (S2).

[0127] Learnable weights (w1, w2) in adaptive weighted fusion (S4).

[0128] Convolutional network parameters in Advanced Temporal Feature Extraction (S5).

[0129] Parameters of the fully connected layers in the classifier (S6).

[0130] Example 3. The above is a schematic scheme of the emotion recognition method based on multimodal adaptive fusion in this embodiment. It should be noted that the technical solution of the emotion recognition system based on multimodal adaptive fusion and the technical solution of the emotion recognition method based on multimodal adaptive fusion described above are based on the same concept. For details not described in detail in the technical solution of the emotion recognition system based on multimodal adaptive fusion in this embodiment, please refer to the description of the technical solution of the emotion recognition method based on multimodal adaptive fusion described above.

[0131] This embodiment also provides an emotion recognition system based on multimodal adaptive fusion, including:

[0132] The original feature generation module is used to collect audio and visual data containing emotional expression information to form a data set, and extract features from the audio and video from the data set to obtain original audio features and original video features;

[0133] Attention matrix generation module, which is used to build an attention module based on the original audio features and the original video features, and generate attention matrix and interaction features through the attention module;

[0134] An adjustment module is used to adjust the original audio features and the original video features through the generated attention matrix to obtain adjusted original features;

[0135] The weighted fusion module is used to calculate the learnable weights and adaptively weight the interaction features with the adjusted original features through the learnable weights;

[0136] The temporal feature extraction module is used to extract high-level temporal features from the weighted fused audio and video features to obtain high-level temporal features;

[0137] The classification module is used to compress and concatenate high-level temporal features for sentiment classification.

[0138] This embodiment further provides an electronic device applicable to the emotion recognition method based on multimodal adaptive fusion, including:

[0139] Memory and processor; the memory is used to store computer-executable instructions, and the processor is used to execute computer-executable instructions to implement the emotion recognition method based on multimodal adaptive fusion proposed in the above embodiment.

[0140] This embodiment also provides a storage medium on which a computer program is stored. When the program is executed by a processor, the emotion recognition method based on multimodal adaptive fusion proposed in the above embodiment is implemented.

[0141] The storage medium proposed in this embodiment and the emotion recognition method based on multimodal adaptive fusion proposed in the above embodiment belong to the same inventive concept. For technical details not described in detail in this embodiment, please refer to the above embodiment, and this embodiment has the same beneficial effects as the above embodiment.

[0142] Example 4, referring to Tables 1-3, is an embodiment of the present invention, which provides an emotion recognition method based on multimodal adaptive fusion. In order to verify the beneficial effects of the present invention, scientific demonstration is carried out through simulation experiments.

[0143] This example provides an emotion recognition method based on multimodal adaptive fusion (ER-MAF). Experimental comparisons are performed on the DFEW and MAFW databases using the unweighted average recall (UAR) and weighted average recall (WAR) evaluation metrics with other multimodal fusion methods for emotion classification. The results are shown in Table 1.

[0144] Table 1 Comparative experimental results

[0145]

[0146] As shown in Table 1, the ER-MAF provided in this embodiment achieves superior results in terms of UAR and WAR evaluation indicators on both the DFEW and MAFW databases compared to other methods. This demonstrates that the ER-MAF provided in this embodiment improves the accuracy and robustness of the model's emotion recognition through the use of the attention module and the adaptive weighted fusion method.

[0147] Verification of the effectiveness of multimodal fusion:

[0148] The experimental comparison results of the ER-MAF model with other unimodal sentiment classification methods on the DFEW and MAFW databases using the unweighted average recall rate (UAR) and weighted average recall rate (WAR) evaluation indicators are shown in Table 2:

[0149] Table 2 Effectiveness of multimodal fusion methods

[0150]

[0151]

[0152] As shown in Table 2, the ER-MAF method provided by the present embodiment achieves superior results in terms of UAR and WAR evaluation metrics on both the DFEW and MAFW databases compared to other single-modality emotion recognition methods. This demonstrates that multimodal fusion improves the accuracy of emotion recognition while providing greater interpretability.

[0153] Ablation experiment verification:

[0154] The rationality of each component of the ER-MAF model was verified. The unweighted average recall (UAR) and weighted average recall (WAR) were used as evaluation metrics for the ablation study on the DFEW database. The experimental verification results are shown in Table 3.

[0155] Table 3 Ablation study table

[0156] ATT ATT+ WF Drop UAR WAR - - - - 57.82 68.23 + - - - 61.23 72.36 + + - - 64.76 75.87 + + + - 68.12 78.82 + + + + 69.56 80.42

[0157] In the table, 'ATT' indicates that the attention module is used to generate interactive features, its '+' indicates that the module is used, and '-' indicates that it is not used; 'ATT+' indicates that the attention module is used to generate interactive features and attention weights, and the original features are adjusted with the attention weights, its '+' indicates that the original features are adjusted with the attention weights, and '-' indicates that the original features are not adjusted with the attention weights; 'WF' indicates that the learnable weights are calculated and the interactive features and the original features are weightedly fused, its '+' indicates that this operation is performed, and '-' indicates that only a simple splicing is performed; 'Drop' indicates a regularization operation, its '+' indicates that a regularization operation is performed, and '-' indicates that no regularization operation is performed.

[0158] The results in Table 3 show that the performance of the pre-trained model alone is quite poor, while each component gradually improves the performance. This indicates that each component helps the model improve the accuracy and robustness of emotion recognition.

[0159] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit the present invention. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present invention may be modified or replaced by equivalents without departing from the spirit and scope of the technical solutions of the present invention, which should all be included in the scope of the claims of the present invention.

Claims

1. An emotion recognition method based on multimodal adaptive fusion, characterized in that: include: Collect audio and visual data containing emotional expression information to form a data set, extract features from the audio and video from the data set, and obtain original audio features and original video features; Based on the original audio features and original video features, an attention module is constructed, and the attention matrix and interaction features are generated through the attention module; The original audio features and the original video features are adjusted through the generated attention matrix to obtain the adjusted original features; Calculate the learnable weights and adaptively weight the interaction features with the adjusted original features through the learnable weights; Performing high-level temporal feature extraction on the weighted fused audio and video features to obtain high-level temporal features; High-level temporal features are compressed and concatenated for sentiment classification.

2. The emotion recognition method based on multimodal adaptive fusion according to claim 1, characterized in that: The extracting of audio and video features from the data set to obtain original audio features and original video features includes: Select appropriate pre-trained models and use them to extract features from audio and video respectively; For audio feature extraction, the convolutional layer in the selected pre-trained model is used to perform preliminary feature extraction on the audio data to obtain preliminary features of specific shapes.

3. The emotion recognition method based on multimodal adaptive fusion according to claim 2, characterized in that: Extracting features from the audio and video data sets to obtain original audio features and original video features further includes: For video feature extraction, each frame of video is processed step by step through the convolutional layer, maximum pooling and feature extraction modules in the selected pre-trained model to extract local features of the image, downsample the features, and compress the spatial features of each frame into a global feature vector representing the overall information of the frame; the input frame sequence is adjusted to a specific shape through the reshaping operation, and the specific tags of each frame are extracted and connected in chronological order to form a time series with a specific shape containing video timing information as the original video feature.

4. The emotion recognition method based on multimodal adaptive fusion according to claim 3, characterized in that: The method of constructing an attention module based on the extracted audio features and visual features, and generating an attention matrix and interaction features through the attention module includes: The audio and visual features are normalized and introduced into the multi-head attention mechanism. The similarity between the query vector and the key vector is calculated by dot product and scaled. The calculated similarity is normalized so that the sum of the attention weights of each query is 1, thereby obtaining the attention matrix.

5. The emotion recognition method based on multimodal adaptive fusion according to claim 4, characterized in that: The original audio features and the original video features are adjusted by the generated attention matrix, and the adjusted original features include: The multi-head attention output matrix is averaged across all heads, compressing the matrix dimension from a high-dimensional form containing multi-head information to a lower-dimensional form, fusing the multi-head attention information into an average value. The original features of the current modality are element-wise multiplied using the attention weights from another modality to adjust the features of the current modality.

6. The emotion recognition method based on multimodal adaptive fusion according to claim 5, characterized in that: The extracting of advanced temporal features from the weighted fused audio and video features includes: The weighted audio features and visual features are respectively input into the convolutional network to further extract high-level time series features.

7. The emotion recognition method based on multimodal adaptive fusion according to claim 6, characterized in that: The process of compressing high-level temporal features and combining them for sentiment classification includes: The high-level time series features are pooled in the time dimension, the time series features are aggregated into a single representation, the audio features and visual features after feature compression are spliced in the last dimension, the spliced features are sent to a classifier composed of fully connected layers for processing, and finally the classification results are returned.

8. An emotion recognition system based on multimodal adaptive fusion, applying the method according to any one of claims 1 to 7, characterized in that: include: The original feature generation module is used to collect audio and visual data containing emotional expression information to form a data set, and extract features from the audio and video from the data set to obtain original audio features and original video features; Attention matrix generation module, which is used to build an attention module based on the original audio features and the original video features, and generate attention matrix and interaction features through the attention module; An adjustment module is used to adjust the original audio features and the original video features through the generated attention matrix to obtain adjusted original features; The weighted fusion module is used to calculate the learnable weights and adaptively weight the interaction features with the adjusted original features through the learnable weights; The temporal feature extraction module is used to extract high-level temporal features from the weighted fused audio and video features to obtain high-level temporal features; The classification module is used to compress and concatenate high-level temporal features for sentiment classification.

9. An electronic device comprising: memory and processor; The memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions. When the computer-executable instructions are executed by the processor, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium storing computer-executable instructions, wherein the computer-executable instructions, when executed by a processor, implement the steps of the method according to any one of claims 1 to 7.

Images