A multi-modal sentiment recognition method based on intra-modal perception and inter-modal cross fusion of a Transformer mode

Abstract

The application discloses a multi-modal emotion recognition method based on intra-modal perception and inter-modal cross fusion of a Transformer mode, and steps include the following: firstly, encoding and extracting deep features of speech and text; then, based on the intra-modal perception module of the Transformer proposed by the application, long-distance dependency relationships in each mode are captured, local perception learning of emotion features is realized, and redundant information in the deep features is reduced; secondly, in order to fuse unaligned multi-modal sequence information and fully utilize the complementarity of different modal information, the inter-modal interactive fusion module based on the Transformer is proposed to capture information dependency relationships between different modes and obtain fused multi-modal global information; finally, an ablation experiment is conducted to verify the effectiveness of the method. The application realizes effective parallel computing for multi-modal emotion recognition, and further improves the recognition performance and generalization ability of a multi-modal emotion recognition system.

Description

Technical Field

[0001] This invention belongs to the field of multimodal emotion recognition technology, specifically involving a multimodal emotion recognition method based on Transformer intramodal perception and intermodal cross-fusion. Background Technology

[0002] Emotion recognition is a crucial component of artificial intelligence. It primarily explores how to leverage mathematical processing methods to deeply analyze input data, enabling computers to accurately capture human emotional states. By building such emotion recognition systems, we can hope to create a more natural and seamless human-computer interaction environment. Emotion recognition can generally be categorized into direct and indirect emotion recognition. Direct emotion recognition mainly involves the use of multiple modalities of information, including text, speech, images, and video. Indirect emotion recognition relies on monitoring implicit emotional responses, including eye movements, body movements, electroencephalogram (EEG) signals, and electrocardiogram (ECG) signals. When a human's emotional state changes, it is often accompanied by changes in multiple pieces of information. Information from different modalities is often highly correlated, thus comprehensively influencing human emotion recognition. Therefore, research on multimodal emotion recognition that integrates emotional information from multiple modalities for affective computing is of great significance.

[0003] In the early stages of emotion recognition research, researchers mostly employed unimodal emotion recognition techniques to identify human emotions. Unimodal emotion recognition refers to the process of identifying and understanding emotions using only a single type of data source. Typically, this single data source can be speech, text, images, etc. Since emotions are expressed in multiple ways, humans can perceive the emotions or intentions of others by integrating facial expressions, speech, or other information. Therefore, the field of emotion recognition based on multimodal information has received increasing attention. In the early stages of multimodal emotion recognition research, researchers mostly used traditional machine learning models for feature extraction, such as Hidden Markov Models (HMMs) and Gaussian Mixture Models (GMMs). However, these models can only study limited emotional contextual information and cannot fully utilize the characteristics of slow changes in human emotions and their strong dependence on contextual information.

[0004] In recent years, with the rapid development of deep learning algorithms, deep learning-based emotion recognition technology has shown new vitality. Deep learning technology enables researchers to extract complex modal information and subtle nuances from multimodal data, thereby promoting a deeper understanding of complex emotional expressions. In 2021, Cao et al. constructed a multimodal emotion recognition system using the stacked network HNSD to better capture high-sensitivity emotional features. In 2023, Xie et al. proposed a multimodal emotion recognition method based on multi-task learning and attention mechanisms, achieving emotion recognition rates of 85.36% and 84.61% on the CMU-MOSI and CMU-MOSEI databases, respectively. In 2024, Li et al. proposed a multimodal shared network with cross-modal constraints to achieve continuous emotion recognition tasks.

[0005] Speech and text, as important forms of emotional information in daily human life, provide crucial judgment information for multimodal emotion recognition systems. Considering the heterogeneity of speech and text information in real-world situations, the performance of multimodal emotion recognition systems will be significantly affected by whether the technology can effectively acquire key emotional and complementary information from speech and text data, and whether it can effectively fuse the extracted speech and text information. Summary of the Invention

[0006] The purpose of this invention is to address the shortcomings and deficiencies of existing technologies by proposing a multimodal emotion recognition method based on Transformer-based intramodal perception and intermodal cross-fusion. By introducing a Transformer-based intramodal perception module, long-distance dependencies within each modality are captured, enabling local perceptual learning of emotion features and reducing redundant information in deep features. Furthermore, a Transformer-based intermodal interaction fusion module is introduced to capture information dependencies between different modalities, obtaining fused multimodal global information and fully utilizing the complementarity of information from different modalities. This invention achieves efficient parallel computation for multimodal emotion recognition, improving training speed and recognition performance, and exhibiting better generalization ability and practicality.

[0007] The technical solution adopted by this invention to solve its technical problem is: a multimodal emotion recognition method based on Transformer intramodal perception and intermodal cross-fusion, the method comprising the following steps:

[0008] Step 1: Perform shallow feature extraction on the original input speech and text modalities respectively, and introduce the Mel spectrogram-AlexNet branch to make up for the lack of spatial information in the speech features;

[0009] Step 2: Input the extracted shallow features into a deep coding network to obtain deep features. For the text modality, construct a GRUs network for deep coding. For the speech modality, construct a CNNs network for deep coding. For the Mel spectrogram, construct a Mel spectrogram-AlexNet branch.

[0010] Step 3: Input the deep encoded text features, multi-scale speech features, and Mel spectrogram features into the Transformer-based intramodal perception module to obtain perceptual learning features of key emotional information within the modality;

[0011] Step 4: Input the perceptual learning text features, perceptual learning multi-scale speech features, and perceptual learning Mel spectrogram features into the Transformer-based intermodal cross-fusion module. Use Mel spectrograms as reused data information for interactive fusion. Information from different modalities can complement each other. Input the results into the sentiment classifier for sentiment discrimination and classification.

[0012] Step 5: Evaluate the performance of the proposed multimodal emotion recognition method based on Transformer intramodal perception and intermodal cross-fusion.

[0013] Furthermore, the specific steps of step 1 include:

[0014] Step 1-1: Pass each piece of text information through the Word2vec sub-network to obtain a 300-dimensional discourse-level text feature vector;

[0015] Steps 1-2: Preprocess each speech signal with different frame lengths, with frame lengths of 256 and 512 respectively, and use librosa for feature extraction to fuse speech features at different scales to obtain 750-dimensional multi-scale speech features.

[0016] Steps 1-3: Use librosa to generate a Mel spectrogram with a window size of 25ms and a step size of 10ms, and crop it to a size of 224×224 to obtain a uniform input size and acquire the time-frequency spatial emotional information in the speech signal.

[0017] Furthermore, the specific steps of step 2 include:

[0018] Step 2-1: Input 300-dimensional Word2vec features into the GRUs network, and output the deep features extracted after passing through 1 Masking layer, 2 GRU layers, and 1 Dropout layer;

[0019] Step 2-2: Input the 750-dimensional multi-scale speech features into a CNN network consisting of two one-dimensional convolutional blocks, where each convolutional block contains one one-dimensional convolutional layer, one ReLU activation layer and one max pooling layer.

[0020] Steps 2-3: Input the 224×224 Mel spectrogram into the pre-trained AlexNet network for deep encoding. Set the number of filters in the last fully connected layer of the AlexNet network to 300 to obtain 300-dimensional Mel spectrogram information.

[0021] Furthermore, step 3 specifically includes the following steps:

[0022] Step 3-1: Input the deep-encoded text features, multi-scale speech features, and Mel spectrogram features into the Transformer module for computation to improve the model's perceptual learning effect on intramodal information;

[0023] Step 3-2: The feature information obtained after the operation by the Transformer module is transformed nonlinearly using a tanh layer to further enhance the perception module's ability to capture data. At the same time, the data information from the Transformer module is operated by the sigmoid function so that the probability of the feature data is mapped to the interval (0, 1).

[0024] Step 3-3: Perform Hadamard product operation on the two outputs of Step 3-2 to achieve element-wise multiplication of feature data of the same dimension, and obtain the output of intramodal perceptual learning;

[0025] Steps 3-4: Considering the dimensional information of each modal data, add a Flatten layer to each modal branch to flatten the modal data.

[0026] Furthermore, in step 3-1, the specific method for the Transformer module to perform the operation includes the following steps:

[0027] Step 3-1-1: Input the deep-encoded text features and the deep-encoded multi-scale speech features into the Transformer_1 module for computation. The Transformer_1 module has 4 layers, the hidden unit dimension is set to 128, the hidden layer dimension in the feedforward neural network is 512, the number of heads in the multi-head attention mechanism is 8, the input shape is 300×256, and the maximum length of the position encoding is 1000.

[0028] Step 3-1-2: Input the depth-encoded Mel spectrogram features into the Transformer_2 module. Its parameters are basically the same as those of Transformer_1, except that the input shape of Transformer_2 is 300×512.

[0029] Furthermore, step 4 specifically includes the following steps:

[0030] Step 4-1: Input information from different modalities into a multimodal gating fusion mechanism, and use Mel spectrograms as multiplexed data information for interactive fusion to obtain text-Mel spectrogram information and multi-scale speech-Mel spectrogram information;

[0031] Step 4-2: Input the output text-Mel spectrogram information and multi-scale speech-Mel spectrogram information into the Transformer module for feature calculation. The Transformer module has 4 layers, the hidden unit dimension is set to 128, the hidden layer dimension in the feedforward neural network is 512, the number of heads in the multi-head attention mechanism is 8, the input shape is 300×256, and the maximum length of the positional encoding is 1000.

[0032] Step 4-3: Calculate the weighted values ​​using the sigmoid function to obtain the activation values ​​of the Transformer feature information, and effectively filter the key sentiment features in the sample data;

[0033] Step 4-4: Perform a Hadamard product operation on the activation values ​​of the above feature information and the Transformer feature information, and obtain the final multimodal sentiment feature representation through the Concat operation.

[0034] Furthermore, in step 5, the specific method for performance evaluation of the proposed multimodal emotion recognition method based on Transformer intramodal perception and intermodal cross-fusion includes the following steps:

[0035] Step 5-1: Compare the multimodal emotion recognition method based on Transformer intramodal perception and intermodal gating interaction with the multimodal emotion recognition method based on intramodal perception and intermodal gating interaction to verify the effectiveness of the Transformer intramodal perception module;

[0036] Step 5-2: Compare the multimodal emotion recognition method based on Transformer intramodal perception and intermodal cross-fusion with the multimodal emotion recognition method based on Transformer intramodal perception and intermodal gating interaction to verify the effectiveness of the Transformer intermodal cross-fusion module;

[0037] Beneficial effects:

[0038] 1. This invention proposes an intramodal perception module based on Transformer, which utilizes the self-attention mechanism of Transformer to achieve local learning of emotional features, reduce redundant information in deep features, and achieve effective feature selection.

[0039] 2. In order to fuse unaligned multimodal sequence information and make full use of the complementarity of different modal information, this invention introduces a Transformer-based intermodal interaction fusion module. This module is used to explore the information dependencies between different modalities and obtain fused multimodal global information.

[0040] 3. This invention can model the information dependency relationship within a sequence through the Transformer mechanism, achieve fully parallel computing, improve training speed and recognition performance, and has better generalization ability and practicality. Attached Figure Description

[0041] Figure 1 This is a schematic diagram of the operation process of the present invention.

[0042] Figure 2 This is a block diagram of the Transformer-based intramodal sensing module of the present invention.

[0043] Figure 3 This is a block diagram of the Transformer-based intermodal cross-fusion module of the present invention.

[0044] Figure 4 The figure shows the ablation experiment results based on the Transformer intramodal sensing module and the Transformer intermodal interaction fusion module of this invention. Detailed Implementation

[0045] The invention will now be described in further detail with reference to the accompanying drawings.

[0046] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be described in detail below with reference to the accompanying drawings and specific embodiments. It should be noted that, to avoid obscuring the invention with unnecessary details, only structures and / or processing steps closely related to the solutions of this invention are shown in the drawings, while other details not closely related to this invention are omitted.

[0047] Additionally, it should be noted that the terms “comprising,” “including,” or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus.

[0048] like Figures 1 to 3 As shown, this invention proposes a multimodal emotion recognition method based on Transformer intramodal perception and intermodal cross-fusion. It applies deep neural networks to multimodal emotion recognition to extract deep features, enabling the construction of a higher-performance multimodal emotion recognition system. The specific steps include:

[0049] Step 1: Shallow feature extraction is performed on the speech and text modalities in the original input database, and a Mel spectrogram-AlexNet branch is introduced to compensate for the lack of spatial information in the speech features. The dataset used in this invention is the IEMOCAP sentiment database, which consists of approximately 12 hours of audio and video content recorded by 10 professional performers (5 men and 5 women) in a professional studio. The corpus is divided into 5 parts, each including two performers, one male and one female.

[0050] Step 1 specifically includes:

[0051] Step 1-1: Pass each piece of text information through the Word2vec sub-network to obtain a 300-dimensional discourse-level text feature vector;

[0052] Steps 1-2: Preprocess each speech signal with different frame lengths, with frame lengths of 256 and 512 respectively, and use librosa for feature extraction to fuse speech features at different scales to obtain 750-dimensional multi-scale speech features.

[0053] Steps 1-3: Use librosa to generate a Mel spectrogram with a window size of 25ms and a step size of 10ms, and crop it to a size of 224×224 to obtain a uniform input size and acquire the time-frequency spatial emotional information in the speech signal.

[0054] Step 2: Input the extracted shallow features into a deep encoding network to obtain deep features. For the text modality, construct a GRUs network for deep encoding. For the speech modality, construct a CNNs network for deep encoding. For the Mel spectrogram, construct a Mel spectrogram-AlexNet branch.

[0055] Step 2 specifically includes:

[0056] Step 2-1: Input 300-dimensional Word2vec features into the GRUs network, and output the deep features extracted after passing through 1 Masking layer, 2 GRU layers, and 1 Dropout layer;

[0057] Step 2-2: Input the 750-dimensional multi-scale speech features into a CNN network consisting of two one-dimensional convolutional blocks, where each convolutional block contains one one-dimensional convolutional layer, one ReLU activation layer and one max pooling layer.

[0058] Steps 2-3: Input the 224×224 Mel spectrogram into the pre-trained AlexNet network for deep encoding. Set the number of filters in the last fully connected layer of the AlexNet network to 300 to obtain 300-dimensional Mel spectrogram information.

[0059] Step 3: Input the deep encoded text features, multi-scale speech features, and Mel spectrogram features into the Transformer-based intramodal perception module to obtain perceptual learning features of key emotional information within the modality.

[0060] Step 3 specifically includes:

[0061] Step 3-1: Input the deep-encoded text features, multi-scale speech features, and Mel spectrogram features into the Transformer module for computation to improve the model's perceptual learning effect on intramodal information;

[0062] Step 3-2: The feature information obtained after the operation by the Transformer module is transformed nonlinearly using a tanh layer to further enhance the perception module's ability to capture data. At the same time, the data information from the Transformer module is operated by the sigmoid function so that the probability of the feature data is mapped to the interval (0, 1).

[0063] Step 3-3: Perform Hadamard product operation on the two outputs of Step 3-2 to achieve element-wise multiplication of feature data of the same dimension, and obtain the output of intramodal perceptual learning;

[0064] Steps 3-4: Considering the dimensional information of each modal data, add a Flatten layer to each modal branch to flatten the modal data.

[0065] Step 3-1 specifically includes:

[0066] Step 3-1-1: Input the deep-encoded text features and the deep-encoded multi-scale speech features into the Transformer_1 module for computation. The Transformer_1 module has 4 layers, the hidden unit dimension is set to 128, the hidden layer dimension in the feedforward neural network is 512, the number of heads in the multi-head attention mechanism is 8, the input shape is 300×256, and the maximum length of the position encoding is 1000.

[0067] Step 3-1-2: Input the depth-encoded Mel spectrogram features into the Transformer_2 module. Its parameters are basically the same as those of Transformer_1, except that the input shape of Transformer_2 is 300×512.

[0068] Step 4: Input the perceptual learning text features, perceptual learning multi-scale speech features, and perceptual learning Mel spectrogram features into the Transformer-based intermodal cross-fusion module. Use the Mel spectrogram as reused data information for interactive fusion to achieve complementarity of information from different modalities. Input the results into the sentiment classifier for sentiment discrimination and classification.

[0069] Step 4 specifically includes:

[0070] Step 4-1: Input information from different modalities into a multimodal gating fusion mechanism, and use Mel spectrograms as multiplexed data information for interactive fusion to obtain text-Mel spectrogram information and multi-scale speech-Mel spectrogram information;

[0071] Step 4-2: Input the output text-Mel spectrogram information and multi-scale speech-Mel spectrogram information into the Transformer module for feature calculation. The Transformer module has 4 layers, the hidden unit dimension is set to 128, the hidden layer dimension in the feedforward neural network is 512, the number of heads in the multi-head attention mechanism is 8, the input shape is 300×256, and the maximum length of the positional encoding is 1000.

[0072] Step 4-3: Calculate the weighted values ​​using the sigmoid function to obtain the activation values ​​of the Transformer feature information, thereby effectively filtering key sentiment features in the sample data;

[0073] Step 4-4: Perform a Hadamard product operation on the activation values ​​of the above feature information and the Transformer feature information, and obtain the final multimodal sentiment feature representation through the Concat operation.

[0074] Step 5: Evaluate the performance of the proposed multimodal emotion recognition method based on Transformer intramodal perception and intermodal cross-fusion.

[0075] Step 5 specifically includes:

[0076] Step 5-1: Compare the multimodal emotion recognition method based on Transformer intramodal perception and intermodal gating interaction with the multimodal emotion recognition method based on intramodal perception and intermodal gating interaction to verify the effectiveness of the Transformer intramodal perception module;

[0077] Step 5-2: Compare the multimodal emotion recognition method based on Transformer intramodal perception and intermodal cross-fusion with the multimodal emotion recognition method based on Transformer intramodal perception and intermodal gating interaction to verify the effectiveness of the Transformer intermodal cross-fusion module.

[0078] In this embodiment, experimental performance was evaluated using weighted recognition rate (WA), unweighted recognition rate (UA), F1 score, and precision. The IEMOCAP database contains various emotions, including angry, excited, happy, sad, neutral, and frustrated. These six emotion categories were selected for a six-class multimodal emotion recognition task. Five-fold cross-validation was used to test the performance of the IEMOCAP dataset. During the experiment, a Lenovo XiaoXinPro 16 computer with an AMD Ryzen 7 6800H CPU, Tensorflow version 2.13.0, Keras version 2.13.1, and the Librosa 0.8.1 speech emotion feature extraction library were used. The network model optimizer was Adam, with an initial learning rate of 0.01, a batch size of 32, 100 iterations, and a patience value of 10.

[0079] Ablation experiments were conducted on Transformer-based intra-modal sensing modules and Transformer-based inter-modal interaction fusion modules. Frame A represents the use of intra-modal sensing modules and inter-modal gating interaction modules; Frame B represents the use of Transformer-based intra-modal sensing modules and inter-modal gating interaction modules; and Frame C represents the use of Transformer-based intra-modal sensing modules and Transformer-based inter-modal interaction fusion modules. The results are as follows: Figure 4 As shown.

[0080] We can see that when using frame B for the emotion recognition task, the weighted emotion recognition rate for 6 categories is 73.45%, an improvement of 1.99% compared to the baseline model A; the unweighted recognition rate for 6 categories is 74.65%, 1.5% higher than the unweighted recognition rate of frame A; meanwhile, frame B's F1 score is 73.25%, and its accuracy is 74.17%, which are improvements of 3.07% and 3.01% respectively compared to frame A, which uses only nonlinear computation in its intramodal perception module. It can be seen that the constructed Transformer-based intramodal perception model has achieved certain improvements in all metrics of the 6-category multimodal emotion classification task. When using frame C for testing various metrics, the model's weighted recognition rate is 73.89%, and the unweighted recognition rate is 75.02%, improvements of 2.43% and 1.87% respectively compared to the baseline model frame A; the F1 score of frame C is 73.74%, an improvement of 3.56% compared to frame A, and the accuracy is an improvement of 3.84% compared to frame A. Compared to frame B, frame C improved the weighted and unweighted recognition rates by 0.44% and 0.37%, respectively. Meanwhile, compared to frame B, frame C improved the F1 score and accuracy by 0.49% and 0.83%, respectively. This demonstrates that using the Transformer-based intermodal interaction fusion module resulted in improvements across all model parameters.

[0081] In summary, this embodiment proposes a multimodal emotion recognition method based on Transformer intra-modal perception and inter-modal cross-fusion. It presents two modules—a Transformer intra-modal perception module and a Transformer inter-modal interaction fusion module—that improve the performance of multimodal emotion recognition systems. Experimental results validate the rationality and effectiveness of these two modules. Compared with other methods, the method in this embodiment achieves improvements in various measurement indicators and has significant reference value in practical applications.

[0082] The above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention.

Claims

1. A multimodal emotion recognition method based on trimodal Mamba interaction and cascaded hierarchical fusion, characterized in that, The method includes the following steps: Step 1: Perform shallow feature extraction on the original input speech, text, and video modalities respectively; Step 1-1: Pass each text data through the BERT sub-network to obtain a 1024-dimensional discourse-level text feature vector; Steps 1-2: Extract features from each speech data using the OpenSMILE toolkit with the "IS13 Compare" configuration to obtain a 130-dimensional speech feature vector at the speech level; Steps 1-3: Pass each video data point through the DenseFace sub-network to obtain a 342-dimensional speech-level video feature vector; Step 2: Input the extracted shallow features into a deep coding network to extract deep features; Step 3: Input the deep-encoded speech features, text features, and video features into the trimodal Mamba interaction module. Through the interaction and fusion of information from different modalities, obtain the main modality enhanced features after the first stage of complementary enhancement. Step 4: Input the enhanced main modality features after the first stage of enhancement into the trimodal Mamba interaction module. Through the interaction and fusion of information from different modalities, obtain the enhanced main modality features after the second stage of complementary enhancement. Step 5: Input the enhanced master modality features after the second stage of enhancement into the trimodal Mamba interaction module. Through the interaction and fusion between different modal information, obtain the enhanced master modality features after the third stage of complementary enhancement. Step 6: Combine the deeply encoded speech features, text features, and video features with the enhanced main modality features after the first, second, and third stages of enhancement and obtain the final multimodal sentiment features through a cascaded hierarchical fusion mechanism. Then, input the final multimodal sentiment features into the sentiment classifier for sentiment prediction. Step 7: Evaluate the performance of the proposed multimodal emotion recognition method based on trimodal Mamba interaction and cascaded hierarchical fusion.

2. The multimodal emotion recognition method based on trimodal Mamba interaction and cascaded hierarchical fusion according to claim 1, characterized in that, Step 2 specifically includes: Step 2-1: Construct a unidirectional LSTM network with an input layer dimension of 1024 and a hidden layer dimension of 128. The network contains two LSTM layer units with a dropout rate of 0.

3. Input the 1024-dimensional utterance-level text feature vector into this unidirectional LSTM network to extract high-level text features, obtaining 128-dimensional deep text features. ; Step 2-2: Construct a bidirectional LSTM network with an input layer dimension of 130 and a hidden layer dimension of 64. The network contains 3 LSTM layer units with a dropout rate of 0.

3. Then, input the 130-dimensional speech-level feature vector into this bidirectional LSTM network to extract high-level speech features, obtaining 128-dimensional deep speech features. ; Steps 2-3: Construct a unidirectional LSTM network with an input layer dimension of 342 and a hidden layer dimension of 128. The network contains 3 LSTM layer units with a dropout rate of 0.

3. Input the 342-dimensional speech-level video feature vector into this unidirectional LSTM network to extract high-level video features, obtaining 128-dimensional deep video features. .

3. The multimodal emotion recognition method based on trimodal Mamba interaction and cascaded hierarchical fusion according to claim 1, characterized in that, Step 3 specifically includes: Step 3-1: Using speech as the primary modality and text and video as auxiliary modalities, input the deeply encoded speech features, text features, and video features into the trimodal Mamba interaction module to obtain the speech enhancement features after the first stage of complementary enhancement. : in, Represents a multilayer perceptron. Represents a linear layer. , Represents the gate vector, Represents long-distance dependency features of speech modalities. Represents long-range dependency features of video modalities. Represents long-distance dependency features of text modalities. This represents the video-audio interaction features when the dominant modality is speech. Text-to-audio interaction features representing the dominant modality of speech. Represents element-wise multiplication. This represents the current stage of complementary enhancement; Step 3-2: Using text as the primary modality and video and speech as auxiliary modalities, input the deeply encoded speech features, text features, and video features into the trimodal Mamba interaction module to obtain the text-enhanced features after the first stage of complementary enhancement. : in, This represents the audio-text interaction features when the dominant modality is text. Video-text interaction features representing the dominant modality of text; Step 3-3: Using video as the primary modality and text and speech as auxiliary modalities, input the deeply encoded speech features, text features, and video features into the trimodal Mamba interaction module to obtain the video enhancement features after the first stage of complementary enhancement. : in, This represents the audio-video interaction features when the dominant modality is video. This represents the text-video interaction features when the dominant modality is video.

4. The multimodal emotion recognition method based on trimodal Mamba interaction and cascaded hierarchical fusion according to claim 1, characterized in that, Step 4 specifically includes: Step 4-1: Using speech as the primary modality and text and video as auxiliary modalities, input the speech features, text features, and video features obtained from the first stage of complementary enhancement into the trimodal Mamba interaction module to obtain the speech enhancement features obtained from the second stage of complementary enhancement. ; Step 4-2: Using text as the primary modality and video and speech as auxiliary modalities, input the speech features, text features, and video features obtained from the first stage of complementary enhancement into the trimodal Mamba interaction module to obtain the text-enhanced features obtained from the second stage of complementary enhancement. ; Step 4-3: Using video as the primary modality and text and speech as auxiliary modalities, input the speech features, text features, and video features obtained from the first stage of complementary enhancement into the trimodal Mamba interaction module to obtain the video enhancement features obtained from the second stage of complementary enhancement. .

5. The multimodal emotion recognition method based on trimodal Mamba interaction and cascaded hierarchical fusion according to claim 1, characterized in that, Step 5 specifically includes: Step 5-1: Using speech as the primary modality and text and video as auxiliary modalities, input the speech features, text features, and video features obtained from the second stage of complementary enhancement into the trimodal Mamba interaction module to obtain the speech enhancement features obtained from the third stage of complementary enhancement. ; Step 5-2: Using text as the primary modality and video and speech as auxiliary modalities, input the speech features, text features, and video features after the second stage of complementary enhancement into the trimodal Mamba interaction module to obtain the text-enhanced features after the third stage of complementary enhancement. ; Step 5-3: Using video as the primary modality and text and speech as auxiliary modalities, input the speech features, text features, and video features after the second stage of complementary enhancement into the trimodal Mamba interaction module to obtain the video enhancement features after the third stage of complementary enhancement. .

6. The multimodal emotion recognition method based on trimodal Mamba interaction and cascaded hierarchical fusion according to claim 1, characterized in that, Step 6 specifically includes: Step 6-1: Convert the deep-encoded speech features Text features and video features Fusion to obtain multimodal initial features : Step 6-2: Aggregate the text enhancement features from the first, second, and third stages together, and obtain the aggregated text features through cascaded fusion. : in, Represents the enhancement stage. Representing the Enhanced text features after stage; Step 6-3: Aggregate the speech enhancement features from the first, second, and third stages together, and obtain aggregated speech features through cascaded fusion. : in, Representing the Enhanced speech features after stage; Step 6-4: Aggregate the video enhancement features from the first, second, and third stages together, and obtain the aggregated video features through cascaded fusion. : in, Representing the Enhanced text features after stage; Step 6-5: Fuse text aggregation features, speech aggregation features, and video aggregation features to obtain multimodal aggregation features. : Step 6-6: Fuse the initial multimodal features and the aggregated multimodal features to obtain the final multimodal joint features. : The final multimodal joint features are then input into the sentiment classifier for sentiment prediction.

7. The multimodal emotion recognition method based on trimodal Mamba interaction and cascaded hierarchical fusion according to claim 1, characterized in that, In step 7, the specific method for performance evaluation of the proposed multimodal emotion recognition method based on trimodal Mamba interaction and cascaded hierarchical fusion includes the following steps: Step 7-1: Compare the current mainstream multimodal emotion recognition methods based on trimodal Mamba interaction and cascaded hierarchical fusion to verify the performance and efficiency of the present invention; Step 7-2: Compare and analyze the role of each module in the multimodal emotion recognition method based on trimodal Mamba interaction and cascaded hierarchical fusion.

Images