A multimodal emotion recognition method based on modality adaptive mask fusion network
Through the modal adaptive mask fusion network, the audio-guided text encoder and self-mask mechanism are used to solve the problems of information density differences and information homogeneity in multimodal speech emotion recognition, and more accurate feature alignment and Information capture significantly improves emotional recognition performance.
Patent Information
- Application Number
- CN202411280779.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-13
- Publication Date
- 2024-12-03
- Estimated Expiration
- 2044-09-13
AI Technical Summary
Existing multimodal speech emotion recognition technology can easily lead to information loss caused by information density differences when the speech and text features are fused, and information homogeneity may occur during the cross-modal alignment process, resulting in poor fusion effect.
Using modal adaptive mask fusion network (MAMF-Net), the text semantic representation is enhanced through an audio-guided text encoder, and partially obstructs the migration information through the self-mask mechanism to capture complementary information between modals and ensure that Accurate alignment of feature dimensions and information density.
It effectively solved the problems of information loss and cross-modal alignment, improved the fusion effect and system stability of multimodal emotion recognition, and significantly improved the emotional recognition performance. The experimental results were found in CMU-MOSI, CMU-MOSEI and CH-SIMS datasets. Better than existing methods.
Smart Images

Figure CN118800218B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to speech emotion recognition technology, and in particular to a multimodal emotion recognition method based on a modality adaptive mask fusion network. Background Art
[0002] With the rapid development of artificial intelligence, speech emotion recognition (SER) has become an important research direction. Especially in human-computer interaction, advanced emotion recognition technology can not only improve the intelligence of the system, but also improve the user experience. In addition, speech emotion recognition has shown great potential in many application scenarios such as mental health analysis and customer service.
[0003] The transmission and understanding of speech emotion is a complex process of nonlinear dynamic multi-level interaction. The subtle differences in speech emotion cannot be fully displayed by speech or text alone. In the process of communication, humans accurately convey their emotional state through multi-dimensional information, such as intonation and speaking speed in speech and vocabulary selection and sentence structure in text. At present, speech emotion recognition research is gradually turning to multimodal models. By integrating multiple perceptual modalities such as speech and text, it captures the multi-dimensional attributes of human speech emotion at multiple levels and angles, overcoming the limitations of single-modal speech emotion recognition technology, which is limited by one-sided information sources and is easily affected by external factors and differences in individual expressions.
[0004] Cross-modal Transformers have shown great potential in the fusion of multimodal information of audio and text. Through the self-attention mechanism, cross-modal Transformers can effectively capture and model the complex relationships and interdependencies between audio and text. For example, models such as AudioBERT and SpeechBERT have made significant progress in audio and text fusion tasks. These models use the self-attention mechanism of Transformer to process audio and text data separately, and fuse the information of the two modalities together through a cross-attention layer to generate a joint representation. These joint representations enable the model to better understand and process multimodal data, and improve performance in tasks such as speech recognition, speech translation, and sentiment analysis. In addition, the VL-BERT model proposed by Li et al. further improves the performance of multimodal tasks by aligning the representations of multiple modalities such as language and audio through pre-training on large-scale multimodal data.
[0005] Masked autoencoders (MAEs) have played an important role in self-supervised learning in the text and audio fields. The core idea is to mask part of the input data and train the model to reconstruct these missing parts to learn effective representations. In the text field, BERT masks part of the input vocabulary and trains the model to predict these masked vocabulary to obtain robust language representations, significantly improving the performance of various natural language processing tasks such as question answering, text classification, and named entity recognition. In the audio field, masked autoencoders are also used to process speech data. For example, the TERA model learns rich audio representations by masking audio frames and training the model to reconstruct these frames. This method performs well in tasks such as speech recognition. Masked autoencoders not only perform well in single-modality, but can also be combined with other techniques to further enhance the quality of representation learning. The reconstruction ability of masked autoencoders and the discriminative ability of contrastive learning are utilized to achieve better performance in downstream tasks.
[0006] At present, the main challenge facing multimodal speech emotion recognition is that the time series characteristics and expressions of different modalities (speech and text) are different, which makes data fusion highly complex. Therefore, it is necessary to design and optimize data fusion strategies so that these heterogeneous information flows can be organically combined and work together to ensure the accuracy and sensitivity of the system in speech emotion analysis. Existing research content includes: Mustaqeem Khan et al. designed a novel multimodal speech emotion recognition model, which uses a multi-head cross-attention mechanism to complete the feature fusion of speech and text encoders, realize cross-modal interaction between different layers and branches, and improve the generalization ability of the overall model. Guan-Nan Dong et al. proposed a temporal relational reasoning network TRIN for multimodal speech emotion recognition. In the multimodal fusion process, the network combines the temporal structure global information generated by the temporal reasoning of each modality to complete the temporal alignment of cross-modal features, and adopts an adaptive mechanism to select implicit complementary information between modalities to enhance the dependency between subspaces and achieve more detailed modal fusion. Zheng Lian[6] et al. designed a multimodal speech emotion analysis network SMIN based on semi-supervision. Under the framework of semi-supervised learning, the model extracts key information within a single modality through intra-modal interaction and cross-modal interaction modules, and completes information processing and integration between different modalities.
[0007] However, the current method has the following problems: When performing multimodal fusion of speech and text, two types of feature extractors are usually used to extract features, and feature alignment is performed before feature interaction or mixing to keep the feature dimensions of different modalities consistent. However, due to the differences in feature dimensions and information density between text and speech modalities, speech features have higher temporal resolution and rich acoustic information, while text features have higher semantic density. During the alignment process, the above operations may lead to information loss in the high information density modality.
[0008] It should be noted that the information disclosed in the above background technology section is only used for understanding the background of the present application, and therefore may include information that does not constitute prior art known to ordinary technicians in the field. Summary of the invention
[0009] The main purpose of the present invention is to solve the problems existing in the above-mentioned background technology and provide a multimodal emotion recognition method based on a modality adaptive mask fusion network.
[0010] To achieve the above object, the present invention adopts the following technical solutions:
[0011] A multimodal emotion recognition method based on a modality adaptive mask fusion network comprises the following steps:
[0012] S1. Data acquisition: obtain audio data and text data as input;
[0013] S2. Text feature extraction: Use the text encoding model to convert text data into word vectors, and use the self-attention mechanism to map the vocabulary-level features of the text to extract the semantic structure features of the text;
[0014] S3. Audio feature extraction: Use the audio context feature extraction model to extract audio context features from the original audio waveform, convert the audio signal into a Mel-spectrogram, and use the audio time-dependent feature extraction model to extract time-dependent features;
[0015] S4. Cross-modal feature alignment and fusion: Combining audio features and text features, cross-modal feature fusion is achieved through an audio-guided text encoder, and the transfer information is partially masked through a self-masking mechanism to capture complementary information between modalities;
[0016] S5. Multimodal feature fusion processing: Use the multi-head attention mechanism to comprehensively process the fused multimodal features, and convert the comprehensively processed features into the probability distribution of emotion categories through a linear mapping layer;
[0017] S6. Generation of sentiment classification results: Determine the final sentiment classification result based on probability distribution calculation and output the result.
[0018] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the multimodal emotion recognition method based on a modality adaptive mask fusion network.
[0019] A computer program product comprises a computer program, wherein when the computer program is executed by a processor, the multimodal emotion recognition method based on a modality adaptive mask fusion network is implemented.
[0020] The present invention has the following beneficial effects:
[0021] The present invention proposes an innovative multimodal emotion recognition method, which is based on the designed modality adaptive mask fusion network MAMF-Net, and effectively solves the information loss and cross-modal alignment problems existing in traditional multimodal fusion. MAMF-Net uses the temporal resolution and contextual information of the speech signal through an audio-guided text encoder to enhance the semantic representation of the text and ensure a more accurate alignment of different modal features in terms of dimension and information density. In addition, the model adopts the occlusion strategy of the self-masking mechanism. Aiming at the problem of information homogeneity that may occur in the process of cross-modal knowledge transfer, the model promotes the capture of complementary information between modalities by partially occluding the transfer information, thereby improving the fusion effect and system stability. The significant performance improvement of the MAMF-Net of the present invention in the emotion recognition task is due to its efficient integration ability of inter-modal information. The experimental results on the CMU-MOSI, CMU-MOSEI and CH-SIMS datasets all show that MAMF-Net is superior to the existing methods, verifying its efficient integration ability and wide application potential in multimodal emotion analysis, and providing an accurate and robust solution for emotion recognition tasks.
[0022] Other beneficial effects of the embodiments of the present invention will be further described below. BRIEF DESCRIPTION OF THE DRAWINGS
[0023] Figure 1 The figure is an algorithm architecture diagram of a modality adaptive mask fusion network (MAMF-Net) according to an embodiment of the present invention.
[0024] Figure 2 t-SNE visualization of feature distributions under different conditions: text and audio unimodal, cross-modal, and MAMFNet.
[0025] Figure 3 The present invention is a flowchart of a multimodal emotion recognition method based on a modality adaptive mask fusion network. DETAILED DESCRIPTION
[0026] The following is a detailed description of the embodiments of the present invention. It should be emphasized that the following description is only exemplary and is not intended to limit the scope and application of the present invention.
[0027] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the features. In the description of the embodiments of the present invention, the meaning of "plurality" is two or more, unless otherwise clearly and specifically defined.
[0028] The present invention proposes a multimodal emotion recognition method and designs a Modality-Adaptive Masked Fusion Network for Speech Emotion Recognition (MAMF-Net). Its important innovations are: 1. Through an audio-guided text encoder, the high temporal resolution and contextual information in the speech are used to enhance the semantic structure in the text. While maintaining the original modal feature dimensions, the feature dimensions are more accurately aligned, thereby improving the information integrity and system performance after fusion. 2. A self-masking mechanism masking strategy based on modal migration: A self-masking mechanism is used to mask part of the migrated information to solve the problem of losing complementary information due to the high degree of similarity between modalities during cross-modal knowledge transfer. In this way, the model's ability to capture key information can be enhanced, the effect and stability of modal fusion can be improved, thereby enhancing the complementarity between modalities and effectively improving modal fusion.
[0029] See also Figure 3 The embodiment of the present invention provides a multimodal emotion recognition method based on a modality adaptive mask fusion network, comprising the following steps:
[0030] Step S1. Data acquisition: Acquire audio data and text data as input.
[0031] Step S2. Text feature extraction: Use a text encoding model such as the Text-Roberta model to convert text data into word vectors, and map the vocabulary-level features of the text through a self-attention mechanism to extract the semantic structure features of the text.
[0032] In some embodiments, step S2 specifically includes:
[0033] S2.1 Use the Text-Roberta model to encode the input text data and convert the text segments into word vectors through the encoding function;
[0034] S2.2 Apply the sub-attention mechanism to process the word vector to extract the key information in the text;
[0035] S2.3 Perform nonlinear transformation on the processed word vector through activation function and bias term to obtain the semantic structure feature representation of the text data.
[0036] Step S3. Audio feature extraction: Use an audio context feature extraction model such as the wav2vec model to extract audio context features from the original audio waveform, convert the audio signal into a Mel-spectrogram (MFCC), and use an audio time-dependent feature extraction model such as the BiLSTM model to extract time-dependent features.
[0037] In some embodiments, step S3 specifically includes:
[0038] S3.1 Use the wav2vec model to perform self-supervised learning on the input raw audio signal and extract the feature vector related to the context;
[0039] S3.2 uses the extracted feature vector as the context feature representation of the audio data;
[0040] S3.3 converts the original audio signal into a Mel-spectrogram;
[0041] S3.4 Use the BiLSTM model to process the Mel-spectrogram and extract the dependencies of audio features in the time series;
[0042] S3.5 Concatenate the hidden states of the forward and backward passes in the BiLSTM model to obtain an audio feature representation that takes time dependency into account.
[0043] Step S4. Cross-modal feature alignment and fusion: Combining audio features and text features, cross-modal feature fusion is achieved through an audio-guided text encoder, and the migration information is partially masked through a self-masking mechanism to capture complementary information between modalities.
[0044] In some embodiments, step S4 specifically includes:
[0045] S4.2 uses the audio feature guided attention layer to concatenate audio context features and time-dependent features as the key vector and value vector of the attention mechanism to guide the cross-modal feature fusion process;
[0046] S4.3 uses text features as query vectors and implements audio-based cross-modal fusion through a cross-attention mechanism layer to obtain high-dimensional text features that incorporate audio information;
[0047] S4.4 uses a self-masking mechanism module to randomly mask the input features of the audio modality and the text modality, prompting the model to learn complementary information between the modalities;
[0048] S4.5 constructs a loss function based on the fused features processed by the self-masking mechanism to make the sentiment distribution of the generated text data consistent with the actual sentiment data distribution, thereby enhancing the stability and consistency of the model in multimodal processing.
[0049] In some embodiments, in step S4, the processing of the cross-attention mechanism layer specifically includes: receiving text features as query vectors, and performing dot product operations with key vectors guided by audio features; applying a scaling operation to the obtained dot product results to adjust the calculation scale and prevent the values from being too large; using a softmax function to normalize the scaled dot product results to generate an attention score; and based on the generated attention scores, performing weighted summation on the numerical vectors guided by audio features to obtain a high-dimensional text feature representation that incorporates audio information.
[0050] In some embodiments, in step S4, the processing of the self-masking mechanism module specifically includes: randomly masking the input features of the two modes of audio and text, and selecting a set proportion of feature elements for masking; wherein, the audio context features and time-dependent features of the audio modality, and the features of the text modality are randomly masked respectively; for the masked feature elements, the training model predicts or reproduces the missing features to prompt the model to learn the complementary information between different modalities, strengthen the interaction and information integration between different modalities, enhance the model's sensitivity to emotion-related features, and improve the accuracy and robustness of emotion recognition.
[0051] In some embodiments, in step S4, the processing of the loss function specifically includes: for features of different modalities, respectively calculating the difference between the masked target features and the predicted features; summing the feature differences of each modality to obtain the sum of differences within the modality; dividing the sum of differences within the modality by the dimension of the modal features to obtain the average difference within the modality; assigning a weight coefficient to each modality according to the importance of the modality; adding the weighted average differences to obtain the final loss function value, which is used to guide feature optimization during model training.
[0052] Step S5. Multimodal feature fusion processing: Use a multi-head attention mechanism to comprehensively process the fused multimodal features, and convert the comprehensively processed features into probability distribution of emotion categories through a linear mapping layer;
[0053] In some embodiments, step S5 specifically includes:
[0054] S5.1 uses a multi-head attention mechanism to process text and audio features in parallel to capture information in different subspaces;
[0055] S5.2 Comprehensive feature representation generated by multi-head attention mechanism, integrating feature information from different modalities;
[0056] S5.3 inputs the fused features into a linear mapping layer to convert the multimodal features into target sentiment categories for sentiment classification tasks;
[0057] S5.4 uses the output of the linear mapping layer to convert it into the probability distribution of emotion categories as the basis for emotion classification.
[0058] Step S6. Generate sentiment classification results: Determine the final sentiment classification result based on probability distribution calculation and output the result.
[0059] The multimodal emotion recognition method based on the modality adaptive mask fusion network MAMF-Net proposed in the embodiment of the present invention effectively solves the problems of information loss and cross-modal alignment in the process of speech and text modal fusion. As a multimodal emotion recognition model, MAMF-Net uses an audio-guided text encoder to enhance the semantic representation of text through the temporal resolution and contextual information of speech, ensuring the accuracy of modal feature alignment. In addition, the model adopts a self-masking mechanism occlusion strategy based on modality migration, which effectively captures the complementary information between modalities by partially occluding the migration information, thereby improving the fusion effect and system stability. Experimental results show that MAMF-Net outperforms existing methods on the CMU-MOSI, CMU-MOSEI and CH-SIMS datasets, demonstrating its potential for wide application in multimodal emotion analysis.
[0060] The specific embodiments of the present invention and algorithm examples thereof are further described below.
[0061] Figure 1 The overall algorithm architecture of the modality-adaptive mask fusion network MAMF-Net of an embodiment of the present invention is demonstrated, which mainly includes four key steps: 1. Feature extraction is performed on the input modality of text data, and Text-Roberta is used in combination with text segmentation to convert it into word vectors, which are mapped to word-level features through the self-attention mechanism to obtain semantic structure. 2. Feature extraction is performed on the input modality of audio data, and wav2vec and biLSTM are used to extract the contextual relationship and time dependency of the input audio respectively. 3. Audio features and their temporal relationship information are used to guide and explore the implicit structure of text across modalities. By injecting additional semantic clues, the quality of text representation is enhanced, the accuracy of feature alignment is improved, and the collaborative fusion of subsequent tasks is promoted. 4. A multi-head attention mechanism is used to comprehensively process multimodal fusion information, and the final emotion category is generated after linear mapping.
[0062] 1. Text feature extraction
[0063] For the input mode of text data, the Text-Roberta model is used to transform the original text data into word vectors. As an advanced Transformer-based model, Text-Roberta has shown significant capabilities in the field of natural language processing, especially in text understanding and feature extraction. The specific transformation formula is as follows:
[0064]
[0065] in, Represents a text segment, is the encoding function provided by Text-Roberta, It is a trainable sub-attention mechanism. and Represent the activation function and the initial distribution bias term respectively, represents dot product (the same below). Finally, the semantic structure feature representation inside the text data is obtained .
[0066] 2. Audio feature extraction
[0067] According to the input modality of audio data, wav2vec and BiLSTM models are used to extract features based on the contextual relationship and time-dependent characteristics of audio data, respectively, so as to achieve the purpose of more comprehensive analysis of audio content.
[0068] As a self-supervised learning model, wav2vec is mainly used to extract feature vectors from raw audio waveforms. Its basic formula can be expressed as:
[0069]
[0070] in, is the context-related features extracted, Represents the input audio signal.
[0071] In addition, considering the rich time dependencies in audio data, the BiLSTM model is selected to parse the audio data. First, the original audio file is converted into a Mel-spectrogram and processed using a bidirectional long short-term memory network (BiLSTM). The dependency of audio features in the time series can be obtained. The processing formula is:
[0072]
[0073] It represents Audio feature representation after considering time dependence, Represents input audio Mel spectrum graph, and Represents the concatenation of the hidden states of the forward and backward passes in a bidirectional LSTM network.
[0074] In summary, using the wav2vec model to extract the contextual information of the audio and combining it with the BiLSTM model to process the time-dependent features can enable a more comprehensive and in-depth analysis of the audio data from multiple dimensions.
[0075] 3. Cross-modal feature alignment and fusion
[0076] For cross-modal knowledge transfer, combine the acoustic features generated by the audio encoder Wav2vec2 and BiLSTM and MFCC features , to further refine the text features This method aims to narrow the gap in information density between different modalities, so that the model can complete the multimodal alignment task more accurately. Figure 1 As shown in Figure 2, this process is mainly achieved through a guided attention layer for audio features. and contextual information Perform feature concatenation and use it as the key vector of the attention mechanism and a numeric vector At the same time, text features with low information density is considered as a query vector Therefore, audio-based cross-modal attention is implemented through the following cross-attention mechanism layers:
[0077]
[0078] in, In order to obtain high-dimensional text features based on cross-modal knowledge transfer, is the scaling factor parameter, Represents the key vector The transpose of .
[0079] In addition, in order to deal with the problem that the information is highly homogeneous and lacks complementary information during the process of cross-state knowledge transfer, , and A self-masking mechanism module is designed. Different from the MAE commonly used for image representation (which randomly masks the input image and then encourages the model to reproduce the missing pixels), the present invention randomly masks the two types of modalities, so that the model can learn the complementary information between the two types of modalities, further improving the accuracy of the final model prediction. Figure 1As shown in the figure, the masked data and position embedding information are connected and input into the self-attention mechanism. This process is trained simultaneously with the sentiment analysis, rather than as a separate training task. By fusing features, a corresponding loss function is designed to measure the consistency between the sentiment distribution of the generated text data and the distribution of the sentiment data of the event, maintaining the stability and consistency of the model when processing multimodal data. The loss function is shown below:
[0080]
[0081] in, Indicates different modes, and Respectively represent the target features and predicted features after masking, is the dimension of the feature, is the modal weight coefficient.
[0082] 4. Multimodal feature fusion
[0083] At the end of the multimodal fusion framework MAMF-Net, a multi-head attention mechanism is introduced to comprehensively process text and audio information to achieve the ultimate goal of sentiment classification. The key to this step is to use the multi-head attention mechanism to simultaneously focus on information in different subspaces, so that it can be more efficient and comprehensive when fusing multimodal features, and generate sentiment categories through a linear mapping layer.
[0084] Through the above process, MAMF-Net can effectively integrate the information of text and audio modalities and use their complementarity to improve the accuracy of sentiment classification tasks. The final output of the model is the probability distribution of each sentiment category, and the final sentiment classification result can be directly obtained based on this distribution.
[0085] experiment
[0086] Datasets: Three datasets, CMU-MOSI, CMU-MOSEI, and CH-SIMS, are used to provide diverse experimental results for multimodal sentiment analysis. The CMU-MOSI dataset was developed by Zadeh et al. (2016) and contains 2,199 annotated English video clips, focusing on sentiment analysis through multiple modalities such as audio and text. CMU-MOSEI, as an extension of CMU-MOSI (2018), has been expanded to cover 23,453 annotated video clips and covers a wider range of topics. CH-SIMS was developed by Yu et al. (2020) and focuses on multimodal sentiment analysis of Chinese TV programs and movie content. It contains 2,281 annotated video clips, and the Chinese background gives it unique cultural characteristics. In addition, CH-SIMS also increases data complexity through multiple labels. The diversity and richness of these three datasets provide an experimental basis for multimodal sentiment analysis research in different scenarios.
[0087] Evaluation indicators: Three evaluation indicators, F1 Score, Pearson correlation coefficient and accuracy, are used for a comprehensive investigation to more comprehensively and scientifically evaluate the classification and regression capabilities of the model on an imbalanced data set. The F1 score (F1 Score) comprehensively measures the accuracy and recall rate, which can better reflect the model's ability to recognize minority class emotions; the Pearson correlation coefficient (Corr) is used to measure the linear correlation between the predicted value and the true emotional intensity; for sentiment analysis tasks, the accuracy rate (Acc) is used as a basic and intuitive indicator to evaluate the overall performance of the model in multi-category sentiment classification tasks. The calculation formula is as follows:
[0088]
[0089]
[0090] in Refers to instances that the model correctly predicts as positive; It refers to the instances that the model correctly predicts as negative classes; This refers to the situation where the model mistakenly predicts negative instances as positive. It refers to the situation where the model mistakenly predicts positive instances as negative. and Respectively The predicted value and true value of samples, and is the mean of the predicted value and the true value.
[0091] Examples
[0092] The PyTorch 1.13.0 framework and CUDA Toolkit 11.6 environment were used to build the proposed MAMF-Net model. For the processing of acoustic data, the Librosa toolbox was used to extract 39-dimensional MFCC (Mel Frequency Cepstral Coefficients). For the training part, AdamW was used as the optimizer, the learning rate was set to 1e-5, and the batch size was 16. The early stopping strategy was set to take effect at 8 epochs, freezing the parameters of the Text-Roberta model, while the Wav2vec model was fine-tuned after loading the pre-trained model. (Text-Roberta has good generalization ability for text processing, while the Wav2vec model needs to be optimized for the acoustic features specific to the data set.) The training was completed on a server equipped with an NVIDIA GEFORCE RTX 4090 GPU, an Intel Core i9-11900k CPU, and 32GB DDR4 4000 memory. To ensure a fair comparison with the current state-of-the-art, the results shown in all tables are averages of data from three independent runs with different random seeds.
[0093] Ablation experiment
[0094] The ablation experiment of the model was carried out, including four cases: text mode, audio mode, cross-modal fusion and MAMF-Net, and the features extracted in each case were analyzed by t-SNE visualization, as shown in the figure. Figure 2 As shown. By comparing different experimental settings, the following conclusions can be drawn: The comparison between (a) and (b) shows that the audio modality provides richer temporal information and contextual information than the text modality, while the feature distribution is more concentrated in the absence of such information in the text modality, resulting in a decrease in the ability to distinguish between different emotion categories. (c) After the introduction of cross-modal fusion, the emotional classification ability between features is significantly improved compared to the case of using only the text modality (b). Specifically, the distinction between the two types of emotions, Happy and Neutral, is clearer, which shows that cross-modal fusion effectively improves the discriminability of feature representation. (d) The self-masking mechanism occlusion strategy based on modality transfer is introduced in the MAMF-Net model to enhance the robustness of the model. The features enhanced by the self-masking mechanism occlusion strategy show a clearer clustering effect, and the boundaries between the various features are more obvious, further improving the performance of the model in the emotion classification task.
[0095] Figure 2 t-SNE visualization showing the feature distribution under different conditions: unimodal, cross-modal, and MAMF Net. The measurements in brackets are (F1 / Corr / Acc).
[0096] Experimental results and analysis:
[0097] Comparison with existing methods:
[0098] In order to comprehensively evaluate MAMF-Net, several key benchmark models in the field of multimodal sentiment analysis were selected for comparison. These models can be roughly divided into four categories: 1. Multi-task and self-supervised learning models: Self-MM (Yu et al., 2021) proposed multi-task learning through single-modal representation; TEASEL (Arjmand et al., 2021) and UniMSE (Hu et al., 2022) emphasized the importance of self-supervised learning and modal fusion, and unified multimodal sentiment analysis and sentiment recognition tasks through a knowledge sharing framework. 2. Mutual information maximization and feature fusion models: MMIM (Han et al., 2021) focuses on hierarchically maximizing mutual information, while MMML (Wu et al., 2023) and ALMT (Wu et al., 2023) optimize feature selection and fusion strategies through multimodal multi-loss fusion and adaptive super-modal learning modules, respectively. 3. Modality independence and feature fusion methods: EMT (Sun et al., 2023) focuses on enhancing modality independence through independent and multimodal transformer modules; MISA (Hazarika et al., 2020) optimizes multimodal data fusion through projection of modality-specific and modality-invariant subspaces. 4. Knowledge distillation and plug-in assistance: SeMuL-PCD (Anand et al., 2023) uses a multimodal transformer network combined with modality-specific peer network feedback for knowledge distillation; MELTR (Ko et al., 2020) is a plug-in module that assists in fine-tuning the video base model, automatically nonlinearly combining various loss functions to assist in learning the target task. The above models each propose different innovative techniques and methods to solve the challenges in multimodal sentiment analysis, such as multi-task learning, mutual information maximization, self-supervised learning and modality fusion, feature selection and fusion strategies, modality independence, data fusion optimization, and knowledge distillation and modality integration. Each category of models proposes innovative solutions and methods in different aspects of multimodal sentiment analysis, aiming to improve the accuracy and efficiency of sentiment recognition. Through the comparative analysis of these representative models, we can more clearly understand the innovations and performance of MAMF-Net in multimodal sentiment analysis.
[0099] Table 1. Overall results of comparison with the existing state-of-the-art (SOTA) on three sentiment recognition (SER) benchmark corpora. Evaluation metrics are F1, Corr, Acc. “-” indicates the lack of this metric, and the best results are highlighted in bold.
[0100]
[0101] Table 1
[0102]
[0103] In the experiment of this invention, the performance of MAMF-Net on three emotion recognition benchmark corpora, CMU-MOSI, CMU-MOSEI and CH-SIMS, was comprehensively evaluated and compared with multiple existing multimodal sentiment analysis models. The experimental results show that after introducing text information, the three indicators of MAMF-Net, F1, Corr and Acc, have been significantly improved. Compared with the SOTA method MMML, the average improvement is 1.61%, 2.48% and 0.78% respectively.
[0104] Compared with other models, the significant advantage of the present invention lies in its innovative introduction of the modal adaptive mask fusion network MAMF-Net. The network effectively improves the accuracy of modal interaction in multimodal speech sentiment analysis through two key components: an audio-guided text encoder and a self-masking mechanism occlusion strategy based on modal migration. The audio-guided text encoder enhances the text semantics and ensures the precise alignment of modal features by utilizing the temporal resolution and contextual information of the speech, thereby improving the information integrity of feature fusion; in addition, the occlusion strategy of the self-masking mechanism solves the problem of complementary information loss caused by the similarity between modalities by partially occluding the migration information. The above strategies not only enhance the model's ability to capture key information, but also significantly improve the effect and stability of modal fusion, so that MAMF-Net performs well in multimodal emotion recognition tasks. Specifically, when MAMF-Net combines text information, its performance is significantly improved: on the CMU-MOSI corpus, the F1 value is improved by 1.33%, the Corr is improved by 2.56%, and the Acc is improved by 0.79%; on the CMU-MOSEI corpus, the F1 is improved by 1.70%, the Corr is improved by 2.58%, and the Acc is improved by 0.72%; on the CH-SIMS corpus, the F1 is improved by 1.79%, the Corr is improved by 2.30%, and the Acc is improved by 0.83%. Overall, the average performance of MAMF-Net is improved by 1.73%, showing that its overall effect in emotion recognition tasks is significantly enhanced.
[0105] In summary, thanks to the efficient integration of inter-modal information, the multimodal emotion recognition method based on MAMF-Net of the present invention has significantly improved the performance in emotion recognition tasks. Experiments have shown that the MAMF-Net of the present invention performs well on current benchmark datasets, has broad application potential, and provides a more accurate and robust solution for future emotion recognition tasks.
[0106] An embodiment of the present invention further provides a storage medium for storing a computer program, which at least performs the above method when executed.
[0107] An embodiment of the present invention further provides a control device, comprising a processor and a storage medium for storing a computer program; wherein the processor is configured to execute at least the method described above when executing the computer program.
[0108] An embodiment of the present invention further provides a processor, wherein the processor executes a computer program and at least executes the method described above.
[0109] The storage medium can be implemented by any type of non-volatile storage device, or a combination thereof. Among them, the non-volatile memory can be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), a magnetic random access memory (FRAM), a ferromagnetic random access memory (Flash Memory), a magnetic surface memory, an optical disc, or a compact disc read-only memory (CD-ROM); the magnetic surface memory can be a disk memory or a tape memory. The storage medium described in the embodiments of the present invention is intended to include, but is not limited to, these and any other suitable types of memory.
[0110] In the several embodiments provided by the present invention, it should be understood that the disclosed systems and methods can be implemented in other ways. The device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as: multiple units or components can be combined, or can be integrated into another system, or some features can be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the components shown or discussed can be through some interfaces, and the indirect coupling or communication connection of the devices or units can be electrical, mechanical or other forms.
[0111] The units described above as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units; some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.
[0112] In addition, all functional units in the embodiments of the present invention may be integrated into one processing unit, or each unit may be separately used as a unit, or two or more units may be integrated into one unit; the above-mentioned integrated units may be implemented in the form of hardware or in the form of hardware plus software functional units.
[0113] A person skilled in the art can understand that: all or part of the steps of implementing the above method embodiment can be completed by hardware related to program instructions, and the aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it executes the steps of the above method embodiment; and the aforementioned storage medium includes: mobile storage devices, read-only memories (ROM, Read-Only Memory), random access memories (RAM, Random Access Memory), disks or optical disks, etc. Various media that can store program codes.
[0114] Alternatively, if the above-mentioned integrated unit of the present invention is implemented in the form of a software function module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the embodiment of the present invention can be essentially or partly reflected in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the methods described in each embodiment of the present invention. The aforementioned storage medium includes: various media that can store program codes, such as mobile storage devices, ROM, RAM, magnetic disks or optical disks.
[0115] The methods disclosed in the several method embodiments provided by the present invention can be arbitrarily combined without conflict to obtain new method embodiments.
[0116] The features disclosed in several product embodiments provided by the present invention can be arbitrarily combined without conflict to obtain new product embodiments.
[0117] The features disclosed in several method or device embodiments provided by the present invention can be arbitrarily combined without conflict to obtain new method embodiments or device embodiments.
[0118] The above contents are further detailed descriptions of the present invention in combination with specific preferred embodiments, and it cannot be determined that the specific implementation of the present invention is limited to these descriptions. For those skilled in the art of the present invention, several equivalent substitutions or obvious variations can be made without departing from the concept of the present invention, and the performance or use is the same, which should be regarded as belonging to the protection scope of the present invention.
Claims
1. A multimodal emotion recognition method based on modality adaptive mask fusion network, characterized in that: The following steps are involved: S1. Data acquisition: obtain audio data and text data as input; S2. Text feature extraction: Use the text encoding model to convert text data into word vectors, and use the self-attention mechanism to map the vocabulary-level features of the text to extract the semantic structure features of the text; S3. Audio feature extraction: Use the audio context feature extraction model to extract audio context features from the audio waveform, convert the audio signal into a Mel-spectrogram, and use the audio time-dependent feature extraction model to extract time-dependent features; S4. Cross-modal feature alignment and fusion: Combining audio features and text features, cross-modal feature fusion is achieved through an audio-guided text encoder, and the transfer information is partially masked through a self-masking mechanism to capture complementary information between modalities; S5. Multimodal feature fusion processing: Use the multi-head attention mechanism to comprehensively process the fused multimodal features, and convert the comprehensively processed features into the probability distribution of emotion categories through a linear mapping layer; S6. Sentiment classification result generation: determining the final sentiment classification result based on the probability distribution calculation and outputting the result; Step S4 specifically includes: S4.2 uses the audio feature guided attention layer to concatenate audio context features and time-dependent features as the key vector and value vector of the attention mechanism to guide the cross-modal feature fusion process; S4.3 uses text features as query vectors and implements audio-based cross-modal fusion through a cross-attention mechanism layer to obtain high-dimensional text features that incorporate audio information; S4.4 uses a self-masking mechanism module to randomly mask the input features of the audio modality and the text modality, prompting the model to learn complementary information between the modalities; S4.5 constructs a loss function based on the fused features processed by the self-masking mechanism to make the sentiment distribution of the generated text data consistent with the actual sentiment data distribution; In step S4, the processing of the self-masking mechanism module specifically includes: The input features of the two modes of audio and text are randomly masked, and feature elements of a set ratio are selected for masking; the audio context features and time-dependent features of the audio mode, and the features of the text mode are randomly masked respectively; For masked feature elements, the model is trained to predict or reproduce the missing features, so as to enable the model to learn complementary information between different modalities and strengthen the interaction and information integration between different modalities. In step S4, the processing of the loss function specifically includes: For features of different modalities, the difference between the masked target features and the predicted features is calculated respectively; The characteristic differences of each mode are summed to obtain the sum of differences within the mode; The sum of the differences within the modality was divided by the dimension of the modality feature to obtain the mean difference within the modality; Assign a weight coefficient to each mode according to its importance; The weighted average differences are added together to obtain the final loss function value, which is used to guide feature optimization during model training.
2. The multimodal emotion recognition method based on modality adaptive mask fusion network as claimed in claim 1, characterized in that: Step S2 specifically includes: S2.1 Use the Text-Roberta model to encode the input text data and convert the text segments into word vectors through the encoding function; S2.2 Apply the sub-attention mechanism to process the word vector to extract the key information in the text; S2.3 Perform nonlinear transformation on the processed word vector through activation function and bias term to obtain the semantic structure feature representation of the text data.
3. The multimodal emotion recognition method based on modality adaptive mask fusion network as claimed in claim 1, characterized in that: Step S3 specifically includes: S3.1 Use the wav2vec model to perform self-supervised learning on the input raw audio signal and extract the feature vector related to the context; S3.2 uses the extracted feature vector as the context feature representation of the audio data; S3.3 converts the original audio signal into a Mel-spectrogram; S3.4 Use the BiLSTM model to process the Mel-spectrogram and extract the dependencies of audio features in the time series; S3.5 Concatenate the hidden states of the forward and backward passes in the BiLSTM model to obtain an audio feature representation that takes time dependency into account.
4. The multimodal emotion recognition method based on modality adaptive mask fusion network according to any one of claims 1 to 3, characterized in that: In step S4, the processing of the cross attention mechanism layer specifically includes: Receive the text feature as the query vector and perform a dot product operation with the key vector guided by the audio feature; Apply a scaling operation to the resulting dot product result to adjust the calculation scale and prevent the value from being too large; Use the softmax function to normalize the scaled dot product results to generate attention scores; According to the generated attention scores, the numerical vectors guided by the audio features are weighted summed to obtain a high-dimensional text feature representation that incorporates audio information.
5. The multimodal emotion recognition method based on modality adaptive mask fusion network according to any one of claims 1 to 3, characterized in that: Step S5 specifically includes: S5.1 uses a multi-head attention mechanism to process text and audio features in parallel to capture information in different subspaces; S5.2 Comprehensive feature representation generated by multi-head attention mechanism, integrating feature information from different modalities; S5.3 inputs the fused features into a linear mapping layer to convert the multimodal features into target sentiment categories for sentiment classification tasks; S5.4 uses the output of the linear mapping layer to convert it into the probability distribution of emotion categories as the basis for emotion classification.
6. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the multimodal emotion recognition method based on a modality adaptive mask fusion network as described in any one of claims 1 to 5 is implemented.
7. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the multimodal emotion recognition method based on a modality adaptive mask fusion network as described in any one of claims 1 to 5 is implemented.
