A multimodal emotion recognition method based on trimodal Mamba interaction and cascaded hierarchical fusion

Abstract

This invention discloses a multimodal emotion recognition method based on trimodal Mamba interaction and cascaded hierarchical fusion. The steps include: first, encoding and extracting deep features from speech, text, and video; capturing long-range dependencies between different modalities using the proposed trimodal Mamba interaction module; and utilizing the complementarity between different modalities to obtain an enhanced representation of the multimodal features; then, dynamically integrating semantic features from different stages using the proposed cascaded hierarchical fusion mechanism to obtain a semantically rich multimodal emotion representation; finally, ablation experiments were conducted to verify the effectiveness of the method. This invention achieves efficient parallel computation for multimodal emotion recognition, improving the recognition performance and generalization ability of the multimodal emotion recognition system.

Description

Technical Field

[0001] This invention belongs to the field of multimodal emotion recognition technology, specifically relating to a multimodal emotion recognition method based on trimodal Mamba interaction and cascaded hierarchical fusion. Background Technology

[0002] With the continuous advancement of information technology, human-computer interaction has become a rapidly developing research area in computer science. In recent years, increasing research has focused on "affective computing," which involves developing systems capable of recognizing, understanding, and managing human emotional responses. The goal of affective computing is to enable machines to understand and respond to human emotions, thereby improving user experience in various application scenarios, such as enhancing customer experience in voice call centers and aiding in the diagnosis of depression through voice analysis. Emotion recognition is a key component of affective computing, involving the perception and interpretation of human emotional states through multiple channels. Common emotion recognition methods utilize multimodal information such as speech, facial expressions, body posture, and text to identify a user's emotional state. Each modality has its unique advantages and limitations. For example, speech signals can convey rich emotional information but are less effective in noisy environments; facial expressions can reflect immediate emotional states but may be inaccurate in occluded or low-light conditions. Multimodal emotion recognition methods attempt to improve the accuracy and robustness of recognition by fusing information from multiple modalities. However, the heterogeneity of multimodal data presents a significant challenge to information fusion. Data from different modalities differ significantly in terms of time, space, and feature dimensions, and how to effectively integrate these data has become a major challenge in the field of emotion recognition.

[0003] Traditional multimodal fusion methods typically include early fusion, late fusion, and mid-fusion. Early fusion directly concatenates the raw data or primary features from different modalities and then inputs them into the emotion recognition model. Late fusion processes the data from each modality independently and then combines the recognition results. However, these methods suffer from information loss and fail to fully capture the complex relationships between modalities. In recent years, deep learning technology has attracted considerable attention due to its significant capabilities in feature learning and nonlinear fitting. The most common fusion method based on deep neural networks extracts features from multiple modalities and then concatenates these features into an extended feature vector. However, this simple concatenation method often fails to capture complex intra- and inter-modal dynamics, resulting in poor performance when recognizing complex emotional states.

[0004] To address these limitations, Tensor Fusion Networks (TFNs) were introduced, which utilize triple Cartesian products to model interactions between unimodal, bimodal, and trimodal modes. While TFNs are effective to some extent, their high computational complexity and large number of parameters can lead to increased resource demands and potential overfitting. Furthermore, human emotions are often hidden in the long-term expression of multimodal signals; failure to capture the long-range dependencies between these modalities may prevent the identification of key emotion-related interactions.

[0005] To further improve the performance of sentiment recognition, the Transformer model was introduced, which excels at capturing long-range dependencies and modeling complex interactions between modalities. The multi-head attention mechanism in the Transformer enables powerful global feature extraction when modeling multimodal sentiment data. However, the computational complexity of the Transformer increases quadratically with the length of the input token, limiting its application in long sequence processing.

[0006] Recent advances in State Space Models (SSMs) have significantly addressed this problem; for example, Mamba achieves global perception with linear complexity. SSMs have performed admirably in various computer vision applications, offering significantly higher computational efficiency compared to Transformers. However, the potential of SSMs in multimodal information fusion, particularly in multimodal emotion recognition tasks, remains largely unexplored. Summary of the Invention

[0007] The purpose of this invention is to address the shortcomings and deficiencies of existing technologies by proposing a multimodal emotion recognition method based on trimodal Mamba interaction and cascaded hierarchical fusion. First, a trimodal Mamba interaction module is introduced to capture long-distance dependencies between different modalities, fully utilizing the complementarity of information from different modalities to achieve complex interactions and seamless feature integration among the three modalities. Subsequently, a cascaded hierarchical fusion strategy is proposed to dynamically integrate enhanced multimodal features from different stages, thereby obtaining a semantically rich multimodal emotion representation. This invention achieves efficient parallel computation for multimodal emotion recognition, improving training speed and recognition performance, and exhibiting better generalization ability and practicality.

[0008] The technical solution adopted by this invention to solve its technical problem is: a multimodal emotion recognition method based on trimodal Mamba interaction and cascaded hierarchical fusion, the method comprising the following steps:

[0009] Step 1: Perform shallow feature extraction on the original input speech, text, and video modalities respectively;

[0010] Step 2: Input the extracted shallow features into a deep coding network to obtain deep features;

[0011] Step 3: Input the deep-encoded speech features, text features, and video features into the trimodal Mamba interaction module. Through the interaction and fusion of information from different modalities, obtain the main modality enhanced features after the first stage of complementary enhancement.

[0012] Step 4: Input the enhanced main modality features after the first stage of enhancement into the trimodal Mamba interaction module. Through the interaction and fusion of information from different modalities, obtain the enhanced main modality features after the second stage of complementary enhancement.

[0013] Step 5: Input the enhanced master modality features after the second stage of enhancement into the trimodal Mamba interaction module. Through the interaction and fusion between different modal information, obtain the enhanced master modality features after the third stage of complementary enhancement.

[0014] Step 6: Combine the deeply encoded speech features, text features, and video features with the enhanced main modality features after the first, second, and third stages of enhancement and obtain the final multimodal sentiment features through a cascaded hierarchical fusion mechanism. Then, input the final multimodal sentiment features into the sentiment classifier for sentiment prediction.

[0015] Step 7: Evaluate the performance of the proposed multimodal emotion recognition method based on trimodal Mamba interaction and cascaded hierarchical fusion.

[0016] Furthermore, step 1 specifically includes:

[0017] Step 1-1: Pass each text data through the BERT sub-network to obtain a 1024-dimensional discourse-level text feature vector;

[0018] Steps 1-2: Extract features from each speech data using the OpenSMILE toolkit with the "IS13 Compare" configuration to obtain a 130-dimensional speech feature vector at the speech level;

[0019] Steps 1-3: Pass each video data point through the DenseFace sub-network to obtain a 342-dimensional speech-level video feature vector.

[0020] Furthermore, step 2 specifically includes:

[0021] Step 2-1: Construct a unidirectional LSTM network with an input layer dimension of 1024 and a hidden layer dimension of 128. The network contains 2 LSTM layer units with a dropout rate of 0.3. Input the 1024-dimensional discourse-level text feature vector into the unidirectional LSTM network to extract the high-level features of the text and obtain the 128-dimensional deep text features T.

[0022] Step 2-2: Construct a bidirectional LSTM network with an input layer dimension of 130 and a hidden layer dimension of 64. The network contains 3 LSTM layer units with a dropout rate of 0.3. Then, input the 130-dimensional speech-level feature vector into the bidirectional LSTM network to extract high-level speech features and obtain 128-dimensional speech deep features A.

[0023] Steps 2-3: Construct a unidirectional LSTM network with an input layer dimension of 342 and a hidden layer dimension of 128. The network contains 3 LSTM layer units with a dropout rate of 0.3. Input the 342-dimensional speech-level video feature vector into the unidirectional LSTM network to extract the high-level features of the video and obtain the 128-dimensional deep video features V.

[0024] Furthermore, step 3 specifically includes:

[0025] Step 3-1: Using speech as the primary modality and video and text as auxiliary modalities, input the deeply encoded speech features A, text features T, and video features V into the trimodal Mamba interaction module to obtain the speech enhancement features aug-A after the first stage of complementary enhancement. (1) :

[0026] X v→a =MLP(G1⊙Y a +G1⊙Y v )

[0027] X t→a =MLP(G2⊙Y a +G2⊙Y t )

[0028] aug-A (i) =Linear(X) v→a +X t→a )

[0029] Where MLP represents a multilayer perceptron, Linear represents a linear layer, G1 and G2 represent gating vectors, and Y represents a gate vector. a Y represents the long-distance dependency feature of the speech modality. v Y represents the long-range dependency feature of the video modality. t X represents the long-distance dependency feature of the text modality. t→aX represents the video-audio interaction features when the dominant modality is speech. t→a The text-to-audio interaction features represent the dominant modality as speech, ⊙ represents element-wise multiplication, and i represents the current stage of complementary enhancement.

[0030] Step 3-2: Using text as the primary modality and speech and video as auxiliary modalities, input the deeply encoded speech features A, text features T, and video features V into the trimodal Mamba interaction module to obtain the first-stage complementary enhancement text enhancement feature aug-T. (1) :

[0031] X a→t =MLP(G1⊙Y t +G1⊙Y a )

[0032] X v→t =MLP(G2⊙Y t +G2⊙Y v )

[0033] aug-T (i) =Linear(X) a→t +X v→t )

[0034] Where X a→t X represents the audio-text interaction features when the dominant modality is text. v→t Video-text interaction features representing the dominant modality of text;

[0035] Step 3-3: Using video as the primary modality and text and speech as auxiliary modalities, input the deeply encoded speech features A, text features T, and video features V into the trimodal Mamba interaction module to obtain the first-stage complementary enhancement video enhancement feature aug-V. (1) :

[0036] X a→v =MLP(G1⊙Y v +G1⊙Y a )

[0037] X t→v =MLP(G2⊙Y v +G2⊙Y t )

[0038] aug-T (i) =Linear(X) a→v +X t→v )

[0039] Where X a→v X represents the audio-video interaction features when the dominant modality is video. t→vThis represents the text-video interaction features when the dominant modality is video.

[0040] Furthermore, step 4 specifically includes:

[0041] Step 4-1: Using speech as the primary modality and video and text as auxiliary modalities, input the speech features, text features, and video features obtained from the first stage of complementary enhancement into the trimodal Mamba interaction module to obtain the speech enhancement features aug-A obtained from the second stage of complementary enhancement. (2) ;

[0042] Step 4-2: Using text as the primary modality and speech and video as auxiliary modalities, input the speech features, text features, and video features obtained from the first stage of complementary enhancement into the trimodal Mamba interaction module to obtain the text-enhanced features aug-T obtained from the second stage of complementary enhancement. (2) ;

[0043] Step 4-3: Using video as the primary modality and text and speech as auxiliary modalities, input the speech features, text features, and video features obtained from the first stage of complementary enhancement into the trimodal Mamba interaction module to obtain the video enhancement features aug-V obtained from the second stage of complementary enhancement. (2) .

[0044] Furthermore, step 5 specifically includes:

[0045] Step 5-1: Using speech as the primary modality and video and text as auxiliary modalities, input the speech features, text features, and video features obtained from the second stage of complementary enhancement into the trimodal Mamba interaction module to obtain the speech enhancement features aug-A obtained from the third stage of complementary enhancement. (3) ;

[0046] Step 5-2: Using text as the primary modality and text and video as auxiliary modalities, input the speech features, text features, and video features after the second stage of complementary enhancement into the trimodal Mamba interaction module to obtain the text-enhanced features aug-T after the third stage of complementary enhancement. (3) ;

[0047] Step 5-3: Using video as the primary modality and text and speech as auxiliary modalities, input the speech features, text features, and video features after the second stage of complementary enhancement into the trimodal Mamba interaction module to obtain the video enhancement features aug-V after the third stage of complementary enhancement. (3) .

[0048] Furthermore, step 6 specifically includes:

[0049] Step 6-1: Fuse the deep-encoded speech features A, text features T, and video features V to obtain the initial multimodal features H. init :

[0050] H init =Linear(Concat(A, V, T))

[0051] Step 6-2: Aggregate the text enhancement features from the first, second, and third stages together, and obtain the aggregated text feature T through cascade fusion. sum :

[0052]

[0053] Where i represents the enhancement stage, aug-T (i) This represents the text features enhanced in stage i.

[0054] Step 6-3: Aggregate the speech enhancement features from the first, second, and third stages together, and obtain the aggregated speech feature A through cascaded fusion. sum :

[0055]

[0056] Among them, aug-A (i) This represents the speech features enhanced in stage i.

[0057] Step 6-4: Aggregate the video enhancement features from the first, second, and third stages together, and obtain the video aggregation feature V through cascaded fusion. sum :

[0058]

[0059] Among them, aug-V (i) This represents the video features enhanced in stage i.

[0060] Step 6-5: Fuse text aggregation features, speech aggregation features, and video aggregation features to obtain multimodal aggregation feature H. sum :

[0061] H sum =Linear(Concat(A sum V sum T sum ))

[0062] Step 6-6: Fuse the initial multimodal features and the aggregated multimodal features to obtain the final multimodal joint feature H. final :

[0063] H final =Concat(H init H sum )

[0064] The final multimodal joint features are then input into the sentiment classifier for sentiment prediction.

[0065] Furthermore, in step 7, the specific method for performance evaluation of the proposed multimodal emotion recognition method based on trimodal Mamba interaction and cascaded hierarchical fusion includes the following steps:

[0066] Step 7-1: Compare the multimodal emotion recognition method based on trimodal Mamba interaction and cascaded hierarchical fusion with mainstream multimodal emotion recognition methods to verify the performance and efficiency of the present invention;

[0067] Step 7-2: Compare and analyze the role of each module in the trimodal interaction and cascaded hierarchical fusion multimodal emotion recognition methods based on Mamba.

[0068] Beneficial effects:

[0069] 1. This invention proposes a trimodal Mamba interaction module to capture long-distance dependencies between different modalities, and at the same time utilizes the complementarity of auxiliary modal information to the main modal information to obtain an enhanced representation of the main modal features.

[0070] 2. This invention introduces a cascaded hierarchical fusion mechanism to dynamically integrate enhanced multimodal features from different stages, thereby achieving effective fusion of multimodal information and obtaining semantically rich multimodal sentiment representations. Attached Figure Description

[0071] Figure 1 This is a schematic diagram of the operation process of the present invention.

[0072] Figure 2 This is a block diagram of the three-modal Mamba interaction module of the present invention. Detailed Implementation

[0073] The invention will now be described in further detail with reference to the accompanying drawings.

[0074] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be described in detail below with reference to the accompanying drawings and specific embodiments. It should be noted that, to avoid obscuring the invention with unnecessary details, only structures and / or processing steps closely related to the solutions of this invention are shown in the drawings, while other details not closely related to this invention are omitted.

[0075] Additionally, it should be noted that the terms “comprising,” “including,” or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus.

[0076] like Figures 1 to 2 As shown, this invention proposes a multimodal emotion recognition method based on trimodal Mamba interaction and cascaded hierarchical fusion. By applying the Mamba network to multimodal emotion recognition, it fully captures the long-distance dependencies between different modalities, enabling the construction of a higher-performance multimodal emotion recognition system. The specific steps include:

[0077] Step 1: Perform shallow feature extraction on the speech, video, and text modalities in the original input database. The dataset used in this invention is the IEMOCAP sentiment database, which consists of approximately 12 hours of audio and video content recorded by 10 professional performers (5 men and 5 women) in a professional studio. The corpus is divided into 5 parts, each containing two performers, one male and one female.

[0078] Step 1 of the present invention specifically includes:

[0079] Step 1-1: Pass each text data through the BERT sub-network to obtain a 1024-dimensional discourse-level text feature vector;

[0080] Steps 1-2: Extract features from each speech data using the OpenSMILE toolkit with the "IS13 Compare" configuration to obtain a 130-dimensional speech feature vector at the speech level;

[0081] Steps 1-3: Pass each video data point through the DenseFace sub-network to obtain a 342-dimensional speech-level video feature vector.

[0082] Step 2: Input the extracted shallow features into a deep coding network to obtain deep features. For the speech modality, construct a bidirectional LSTM network for deep coding. For the text and video modalities, construct a unidirectional LSTM network for deep coding.

[0083] Step 2 of the present invention specifically includes:

[0084] Step 2-1: Construct a unidirectional LSTM network with an input layer dimension of 1024 and a hidden layer dimension of 128. The network contains 2 LSTM layer units with a dropout rate of 0.3. Input the 1024-dimensional discourse-level text feature vector into the unidirectional LSTM network to extract the high-level features of the text and obtain the 128-dimensional deep text features T.

[0085] Step 2-2: Construct a bidirectional LSTM network with an input layer dimension of 130 and a hidden layer dimension of 64. The network contains 3 LSTM layer units with a dropout rate of 0.3. Then, input the 130-dimensional speech-level feature vector into the bidirectional LSTM network to extract high-level speech features and obtain 128-dimensional speech deep features A.

[0086] Steps 2-3: Construct a unidirectional LSTM network with an input layer dimension of 342 and a hidden layer dimension of 128. The network contains 3 LSTM layer units with a dropout rate of 0.3. Input the 342-dimensional speech-level video feature vector into the unidirectional LSTM network to extract the high-level features of the video and obtain the 128-dimensional deep video features V.

[0087] Step 3: Input the deep-encoded speech features A, text features T, and video features V into the trimodal Mamba interaction module. Through the interaction and fusion of information from different modalities, obtain the main modality enhancement features after the first stage of complementary enhancement.

[0088] Step 3 of the present invention specifically includes:

[0089] Step 3-1: Using speech as the primary modality and text and video as auxiliary modalities, input the deeply encoded speech features, text features, and video features into the trimodal Mamba interaction module to obtain the speech enhancement features after the first stage of complementary enhancement.

[0090] Step 3-2: Using text as the primary modality and video and speech as auxiliary modalities, input the deeply encoded speech features, text features, and video features into the trimodal Mamba interaction module to obtain the text-enhanced features after the first stage of complementary enhancement;

[0091] Step 3-3: Using video as the primary modality and text and speech as secondary modalities, input the deeply encoded speech features, text features, and video features into the trimodal Mamba interaction module to obtain the video enhancement features after the first stage of complementary enhancement.

[0092] Step 3-1 specifically includes:

[0093] Step 3-1-1: Receive the deep-encoded features from the main modality (speech modality) and two auxiliary modalities (video modality and text modality), perform layer normalization on the deep-encoded features of each modality to standardize the feature data, and obtain the standardized features of the main modality and the standardized features of the two auxiliary modalities respectively;

[0094] Step 3-1-2: The standardized features of the primary and auxiliary modalities obtained in Step 3-1-1 are passed through three parallel multilayer perceptrons and convolutional layers to obtain the deep features X. m :

[0095]

[0096] Where Conv1D represents one-dimensional convolution, MLP represents multilayer perceptron, and LN represents layer normalization. The input consists of three modal features (one primary modal feature and two secondary modal features), where 'a' represents the speech modality, 'v' represents the video modality, and 't' represents the text modality.

[0097] Step 3-1-3: Input the deep features obtained in Step 3-1-2 into SSM to model long-range dependencies and obtain long-range dependency features Y. m :

[0098] Y m =SSM(X m ), m∈{a, v, t}

[0099] SSM represents the state-space model;

[0100] Step 3-1-4: Input the normalized features of the main modality into two independent MLP1 and MLP2, map the main modality to the complementary spaces of the two auxiliary modalities respectively, and then generate two gating vectors G1 and G2 through activation functions:

[0101]

[0102] When speech is the dominant modality, the input is... for When text is the primary modality input for Input when video is the dominant modality for At this point, the dominant mode is the speech mode. That is

[0103] Step 3-1-5: Use the two gating vectors obtained in Step 3-1-4 to modulate the features of the long-range dependent modality obtained in Step 3-1-3, to obtain the video-audio interaction features X when the dominant modality is speech. v→a Text-to-audio interaction features X t→a :

[0104] X v→a =MLP(G1⊙Y a +G1⊙Y v )

[0105] X t→a =MLP(G2⊙Y a +G2⊙Y t )

[0106] Where ⊙ represents element-wise multiplication;

[0107] Step 3-1-6: Fuse the video-audio interaction features and text-audio interaction features, and pass them through a linear layer to obtain the first-stage complementary enhanced speech feature aug-A. (1) :

[0108] aug-A (1) =Linear(X) v→a +X t→a )

[0109] Step 3-2 is similar to step 3-1, except that the input in step 3-2 is text-based as the primary modality, with video and audio as secondary modalities. That is The output is the text enhancement feature aug-T after the first stage of complementary enhancement. (1) :

[0110] X v→t =MLP(G1⊙Y t +G1⊙Y v )

[0111] X a→t =MLP(G2⊙Y t +G2⊙Y a )

[0112] aug-T (1) =Linear(X) v→t +X a→t )

[0113] Where X v→t X represents the video-text interaction features when the dominant modality is text. a→t Audio-text interaction features representing the dominant modality is text;

[0114] Step 3-3 is similar to step 3-1, except that the input for step 3-3 is video as the primary modality and text and speech as secondary modalities. That is The output yields the first-stage complementary enhancement video enhancement feature aug-V. (1) :

[0115] X t→v =MLP(G1⊙Y v +G1⊙Y t )

[0116] X a→v =MLP(G2⊙Y v +G2⊙Y a )

[0117] aug-V (1) =Linear(X) t→v +X a→v )

[0118] Where X t→v X represents the text-video interaction features when the dominant modality is video. a→v This represents the audio-video interaction characteristics when the dominant modality is video.

[0119] Step 4: Input the enhanced master modality features after the first stage of enhancement into the trimodal Mamba interaction module. Through the interaction and fusion of information from different modalities, obtain the enhanced master modality features after the second stage of complementary enhancement.

[0120] Step 4 of the present invention specifically includes:

[0121] Step 4-1: Using speech as the primary modality and video and text as auxiliary modalities, input the speech features, text features, and video features obtained from the first stage of complementary enhancement into the trimodal Mamba interaction module to obtain the speech enhancement features aug-A obtained from the second stage of complementary enhancement. (2) ;

[0122] Step 4-2: Using text as the primary modality and speech and video as auxiliary modalities, input the speech features, text features, and video features obtained from the first stage of complementary enhancement into the trimodal Mamba interaction module to obtain the text-enhanced features aug-T obtained from the second stage of complementary enhancement. (2) ;

[0123] Step 4-3: Using video as the primary modality and text and speech as auxiliary modalities, input the speech features, text features, and video features obtained from the first stage of complementary enhancement into the trimodal Mamba interaction module to obtain the video enhancement features aug-V obtained from the second stage of complementary enhancement. (2) .

[0124] Step 5: Input the enhanced master modality features after the second stage of enhancement into the trimodal Mamba interaction module. Through the interaction and fusion of information from different modalities, obtain the enhanced master modality features after the third stage of complementary enhancement.

[0125] Step 5 of the present invention specifically includes:

[0126] Step 5-1: Using speech as the primary modality and video and text as auxiliary modalities, input the speech features, text features, and video features obtained from the second stage of complementary enhancement into the trimodal Mamba interaction module to obtain the speech enhancement features aug-A obtained from the third stage of complementary enhancement. (3) ;

[0127] Step 5-2: Using text as the primary modality and text and video as auxiliary modalities, input the speech features, text features, and video features after the second stage of complementary enhancement into the trimodal Mamba interaction module to obtain the text-enhanced features aug-T after the third stage of complementary enhancement. (3) ;

[0128] Step 5-3: Using video as the primary modality and text and speech as auxiliary modalities, input the speech features, text features, and video features after the second stage of complementary enhancement into the trimodal Mamba interaction module to obtain the video enhancement features aug-V after the third stage of complementary enhancement. (3) .

[0129] Step 6: Combine the deeply encoded speech features, text features, and video features with the enhanced main modality features after the first, second, and third stages of enhancement, and obtain the final multimodal emotion features through a cascaded hierarchical fusion mechanism.

[0130] Step 6 of the present invention specifically includes:

[0131] Step 6-1: Fuse the deep-encoded speech features A, text features T, and video features V to obtain the initial multimodal features H. init :

[0132] H init =Linear(Concat(A, V, T))

[0133] Step 6-2: Aggregate the text enhancement features from the first, second, and third stages together, and obtain the aggregated text feature T through cascade fusion. sum :

[0134]

[0135] Where i represents the enhancement stage, aug-T (i) This represents the text features enhanced in stage i.

[0136] Step 6-3: Aggregate the speech enhancement features from the first, second, and third stages together, and obtain the aggregated speech feature A through cascaded fusion. sum :

[0137]

[0138] Among them, aug-A (i) This represents the speech features enhanced in stage i.

[0139] Step 6-4: Aggregate the video enhancement features from the first, second, and third stages together, and obtain the video aggregation feature V through cascaded fusion. sum :

[0140]

[0141] Among them, aug-V (i) This represents the video features enhanced in stage i.

[0142] Step 6-5: Fuse text aggregation features, speech aggregation features, and video aggregation features to obtain multimodal aggregation feature H. sum :

[0143] H sum =Linear(Concat(A sum V sum T sum ))

[0144] Step 6-6: Fuse the initial multimodal features and the aggregated multimodal features to obtain the final multimodal joint feature H. final :

[0145] H final =Concat(H init H sum )

[0146] The final multimodal joint features are then input into the sentiment classifier for sentiment prediction.

[0147] Step 6-1 specifically includes:

[0148] Step 6-1-1: Connect the deep encoded speech features, text features, and video features together using Concat, and then perform dimensionality reduction using pooling;

[0149] Step 6-1-2: Pass the pooled features from Step 6-1-1 through a fully connected layer and a ReLU activation function to obtain the initial multimodal features.

[0150] Step 6-5 specifically includes:

[0151] Step 6-5-1: Connect the text aggregation features obtained in Step 6-2, the speech aggregation features obtained in Step 6-3, and the video aggregation features obtained in Step 6-4 together using Concat, and then reduce the dimensionality using pooling.

[0152] Step 6-5-2: Pass the pooled features from Step 6-5-1 through a fully connected layer and a ReLU activation function to obtain multimodal aggregated features.

[0153] Step 6-6 specifically includes:

[0154] Step 6-6-1: Connect the initial multimodal features obtained in Step 6-1 and the aggregated multimodal features obtained in Step 6-5 together using Concat to obtain the final joint multimodal feature H. final ;

[0155] Step 6-6-2: Combine the final multimodal joint features H from step 6-6-1 final The data is input into a sentiment classifier for sentiment discrimination and classification.

[0156] Step 7: Evaluate the performance of the proposed multimodal emotion recognition method based on trimodal Mamba interaction and cascaded hierarchical fusion.

[0157] Step 7 of the present invention specifically includes:

[0158] Step 7-1: Compare the multimodal emotion recognition method based on trimodal Mamba interaction and cascaded hierarchical fusion with mainstream multimodal emotion recognition methods to verify the performance and efficiency of the present invention.

[0159] Step 7-2: Compare and analyze the role of each module in the trimodal interaction and cascaded hierarchical fusion multimodal emotion recognition methods based on Mamba.

[0160] This invention utilizes the IEMOCAP dataset for multimodal emotion recognition. This dataset contains 10,039 audiovisual data points, including audio, video, and contextual information, recording dialogues between ten professional actors in hypothetical scenarios and their corresponding emotion categories. The excitement category is merged into the happiness category. We primarily focus on four main emotions: anger, happiness, sadness, and neutrality, totaling 5,531 data points used in the experiments. Five-fold leave-one-out cross-validation is used to evaluate model performance. One session is selected as the test set for each fold, and the remaining four sessions are used for training. Evaluation metrics include unweighted accuracy (UA) and weighted accuracy (WA). UA is the average accuracy across all emotion categories, while WA is weighted by the number of samples per category, providing a balanced performance measure. Experiments are implemented using the PyTorch framework with the Adam optimizer, a learning rate of 2e-4, a batch size of 32, and a dropout rate between 0.3 and 0.4 to prevent overfitting. Training was conducted on a high-performance computing system with 24GB of NVIDIA GeForce RTX 3090 graphics cards, lasting 50 epochs, and L2 regularization with parameter 1e-5 was used during training.

[0161] The simulation experiments and data of this invention are as follows:

[0162] First, to verify the effectiveness of the multimodal emotion recognition method based on trimodal Mamba interaction and cascaded hierarchical fusion, the proposed method is compared with some existing mainstream multimodal emotion recognition methods, including MSSF, MAFN, MEMOBERT, SSMM, MCEPM, and MMIN. MSSF integrates a modality-specific self-supervised pre-training architecture and combines non-verbal cues from motion capture, achieving efficient emotion recognition through feature fusion at the decision layer. MAFN utilizes multi-head attention mechanisms and fusion network technology to improve the accuracy of emotion recognition by learning the intrinsic correlations between cross-modal features. MEMOBERT employs self-supervised learning and cue-based techniques, utilizing a cross-modal Transformer to model cross-modal context. SSMM uses deep autoencoders for latent representation learning and improves emotion recognition performance through cross-modal distribution matching. MCEPM combines facial expressions, audio signals, text, and motion data, enhancing responsiveness to relevant stimuli through product fusion. MMIN achieves robust multimodal joint representation through cascaded residual autoencoders and recurrent consistency learning. Experimental results are shown in Table 1.

[0163] Table 1 shows the performance comparison with other mainstream methods on the IEMOCAP dataset.

[0164] Model WA UA SSMM 75.6 74.5 MAFN 75.6 71.4 MSSF - 77.58 MMIN 76.51 77.79 MCEPM - 78.2 MEMoBERT 80.01 81.09 Ours 81.22 82.06

[0165] In comparative experiments, the trimodal Mamba interaction and cascaded hierarchical fusion model of this invention outperforms existing methods on the IEMOCAP dataset. Specifically, the method of this invention improves accuracy by 5.62% on WA and 7.56% on UA. Compared with MSSF and MCEPM, the method of this invention improves accuracy by 4.48% and 3.86% on UA, respectively. Furthermore, compared with MAFN, the method of this invention improves accuracy by 5.62% on WA and 10.66% on UA. Compared with MMIN, the method of this invention improves accuracy by 4.71% and 4.27% on WA and UA, respectively. Compared with MEMoBERT, the method of this invention improves accuracy by 1.21% and 0.97% on WA and UA, respectively. These comparative results demonstrate that the multimodal emotion recognition method of this invention based on trimodal Mamba interaction and cascaded hierarchical fusion has significant advantages in multimodal emotion recognition tasks, especially in handling complex emotion recognition tasks, showing significant improvements in multimodal data fusion and emotion recognition accuracy.

[0166] To analyze the contributions of each component in the proposed multimodal emotion recognition method based on trimodal Mamba interaction and cascaded hierarchical fusion, an ablation experiment was designed to evaluate the performance of various configurations of the trimodal interaction Mamba module and the cascaded hierarchical fusion mechanism. The ablation experiment results are shown in Table 2.

[0167] Table 2 Ablation Experiment Results

[0168] Model WA UA Baseline 78.10 79.09 TI-Mamba (S=1) 80.16 81.24 TI-Mamba (S=2) 80.38 81.43 TI-Mamba (S=3) 80.61 81.69 TI-Mamba (S = 2 cascaded) 80.86 81.67 TI-Mamba (S = 3 cascaded) 81.22 82.06

[0169] In the ablation experiments on the IEMOCAP dataset, TI-Mamba represents the trimodal interactive Mamba module, S represents the number of stages used by TI-Mamba, and cascaded represents the cascaded hierarchical fusion mechanism.

[0170] The ablation experiments focused on the impact of each model on WA and UA. The baseline model used independent LSTM encoders to perform deep encoding of audio, video, and text modalities separately, and then integrated these modalities through late-stage fusion. The baseline model achieved a WA of 78.10% and an UA of 79.09%. Integrating a one-stage trimodal interaction Mamba module on top of the baseline model significantly improved performance, increasing WA to 80.16% and UA to 81.24%. As the number of stages in the trimodal interaction Mamba module increased to two, model performance continued to improve, reaching a WA of 80.38% and an UA of 81.43%. When the number of stages in the trimodal interaction Mamba module increased to three, model performance further improved, reaching a WA of 80.61% and an UA of 81.69%. Combining a cascaded hierarchical fusion mechanism further enhanced model performance. The model configured for two-stage cascaded fusion achieved a WA of 80.86% and an UA of 81.67%. The model configured with three-stage cascaded fusion performed best, achieving a WA of 81.22% and an UA of 82.06%. These experimental results demonstrate that the three-modal interaction Mamba module is significantly effective in enhancing multimodal emotion recognition, especially when combined with the cascaded hierarchical fusion mechanism. Each additional stage and the introduction of the cascaded hierarchical fusion mechanism contributes to progressively improved performance, indicating the value of deeper feature interactions and more complex integration strategies.

[0171] In summary, this embodiment proposes a multimodal emotion recognition method based on trimodal Mamba interaction and cascaded hierarchical fusion. It presents a method to improve the performance of multimodal emotion recognition systems by focusing on the trimodal Mamba interaction module and the cascaded hierarchical fusion mechanism. Experimental results verify the rationality and effectiveness of these two modules. Compared with other methods, the method in this embodiment achieves improvements in various measurement indicators and has significant reference value in practical applications.

[0172] The above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention.

Claims

1. A multimodal emotion recognition method based on trimodal Mamba interaction and cascaded hierarchical fusion, characterized in that, The method includes the following steps: Step 1: Perform shallow feature extraction on the original input speech, text, and video modalities respectively; Step 2: Input the extracted shallow features into a deep coding network to extract deep features; Step 3: Input the deep-encoded speech features, text features, and video features into the trimodal Mamba interaction module. Through the interaction and fusion of information from different modalities, obtain the main modality enhanced features after the first stage of complementary enhancement. Step 4: Input the enhanced main modality features after the first stage of enhancement into the trimodal Mamba interaction module. Through the interaction and fusion of information from different modalities, obtain the enhanced main modality features after the second stage of complementary enhancement. Step 5: Input the enhanced master modality features after the second stage of enhancement into the trimodal Mamba interaction module. Through the interaction and fusion between different modal information, obtain the enhanced master modality features after the third stage of complementary enhancement. Step 6: Combine the deeply encoded speech features, text features, and video features with the enhanced main modality features after the first, second, and third stages of enhancement and obtain the final multimodal sentiment features through a cascaded hierarchical fusion mechanism. Then, input the final multimodal sentiment features into the sentiment classifier for sentiment prediction. Step 7: Evaluate the performance of the proposed multimodal emotion recognition method based on trimodal Mamba interaction and cascaded hierarchical fusion; Step 3 specifically includes: Step 3-1-1: Receive the features from the main modality and the two auxiliary modalities after deep encoding, and perform layer normalization on the features after deep encoding of each modality to standardize the feature data, so as to obtain the standardized features of the main modality and the standardized features of the two auxiliary modalities respectively. Step 3-1-2: The standardized features of the primary and auxiliary modalities obtained in Step 3-1-1 are passed through three parallel multilayer perceptrons and convolutional layers to obtain the deep features X. m : Where Conv1D represents one-dimensional convolution, MLP represents multilayer perceptron, and LN represents layer normalization. The input consists of three modal features: one primary modal feature and two secondary modal features. 'a' represents the speech modality, 'v' represents the video modality, and 't' represents the text modality. Step 3-1-3: Input the deep features obtained in Step 3-1-2 into SSM to model long-range dependencies and obtain long-range dependency features Y. m : Y m =SSM(X m ),m∈{a,v,t} SSM represents the state-space model; Step 3-1-4: Input the normalized features of the main modality into two independent MLP1 and MLP2, map the main modality to the complementary spaces of the two auxiliary modalities respectively, and then generate two gating vectors G1 and G2 through activation functions: When speech is the dominant modality, the input is... for When text is the primary modality input for Input when video is the dominant modality for At this point, the dominant mode is the speech mode. That is Step 3-1-5: Use the two gating vectors obtained in Step 3-1-4 to modulate the features of the long-range dependent modality obtained in Step 3-1-3, to obtain the video-audio interaction features X when the dominant modality is speech. v→a Text-to-audio interaction features X t→a : X v→a =MLP(G1⊙Y a +G1⊙Y v ) X t→a =MLP(G2⊙Y a +G2⊙Y t ) Where ⊙ represents element-wise multiplication; Step 3-1-6: Fuse the video-audio interaction features and text-audio interaction features, and pass them through a linear layer to obtain the first-stage complementary enhanced speech feature aug-A. (1) : aug-A (1) =Linear(X v→a +X t→a ) Step 3-2: Using text as the primary modality and video and speech as auxiliary modalities, input the deeply encoded speech features, text features, and video features into the trimodal Mamba interaction module to obtain the first-stage complementary enhancement text-enhanced features aug-T. (1) : X a→t =MLP(G1⊙Y t +G1⊙Y a ) X v→t =MLP(G2⊙Y t +G2⊙Y v ) aug-T (i) =Linear(X a→t +X v→t ) Where, x a→t X represents the audio-text interaction features when the dominant modality is text. v→t Video-text interaction features representing the dominant modality of text; Step 3-3: Using video as the primary modality and text and speech as auxiliary modalities, input the deeply encoded speech features, text features, and video features into the trimodal Mamba interaction module to obtain the first-stage complementary enhancement video enhancement feature aug-V. (1) : X a→v =MLP(G1⊙Y v +G1⊙Y a ) X t→v =MLP(G2⊙Y v +G2⊙Y t ) aug-T (i) =Linear(X a→v +X t→v ) Among them, X a→v X represents the audio-video interaction features when the dominant modality is video. t→v Text-video interaction features representing the dominant modality is video; Step 4 specifically includes: Step 4-1: Using speech as the primary modality and text and video as auxiliary modalities, input the speech features, text features, and video features obtained from the first stage of complementary enhancement into the trimodal Mamba interaction module to obtain the speech enhancement features aug-A obtained from the second stage of complementary enhancement. (2) ; Step 4-2: Using text as the primary modality and video and speech as auxiliary modalities, input the speech features, text features, and video features obtained from the first stage of complementary enhancement into the trimodal Mamba interaction module to obtain the text-enhanced features aug-T obtained from the second stage of complementary enhancement. (2) ; Step 4-3: Using video as the primary modality and text and speech as auxiliary modalities, input the speech features, text features, and video features obtained from the first stage of complementary enhancement into the trimodal Mamba interaction module to obtain the video enhancement features aug-V obtained from the second stage of complementary enhancement. (2) ; Step 5 specifically includes: Step 5-1: Using speech as the primary modality and text and video as auxiliary modalities, input the speech features, text features, and video features after the second stage of complementary enhancement into the trimodal Mamba interaction module to obtain the speech enhancement feature aug-A after the third stage of complementary enhancement. (3) ; Step 5-2: Using text as the primary modality and video and speech as auxiliary modalities, input the speech features, text features, and video features after the second stage of complementary enhancement into the trimodal Mamba interaction module to obtain the text-enhanced features aug-T after the third stage of complementary enhancement. (3) ; Step 5-3: Using video as the primary modality and text and speech as auxiliary modalities, input the speech features, text features, and video features after the second stage of complementary enhancement into the trimodal Mamba interaction module to obtain the video enhancement features aug-V after the third stage of complementary enhancement. (3) .

2. The multimodal emotion recognition method based on trimodal Mamba interaction and cascaded hierarchical fusion according to claim 1, characterized in that, Step 1 specifically includes: Step 1-1: Pass each text data through the BERT sub-network to obtain a 1024-dimensional discourse-level text feature vector; Steps 1-2: Extract features from each speech data using the OpenSMILE toolkit with the "IS13 Compare" configuration to obtain a 130-dimensional speech feature vector at the speech level; Steps 1-3: Pass each video data point through the DenseFace sub-network to obtain a 342-dimensional speech-level video feature vector.

3. The multimodal emotion recognition method based on trimodal Mamba interaction and cascaded hierarchical fusion according to claim 1, characterized in that, Step 2 specifically includes: Step 2-1: Construct a unidirectional LSTM network with an input layer dimension of 1024 and a hidden layer dimension of 128. The network contains 2 LSTM layer units with a dropout rate of 0.

3. Input the 1024-dimensional discourse-level text feature vector into the unidirectional LSTM network to extract the high-level features of the text and obtain the 128-dimensional deep text features T. Step 2-2: Construct a bidirectional LSTM network with an input layer dimension of 130 and a hidden layer dimension of 64. The network contains 3 LSTM layer units with a dropout rate of 0.

3. Then, input the 130-dimensional speech-level feature vector into the bidirectional LSTM network to extract high-level speech features and obtain 128-dimensional speech deep features A. Steps 2-3: Construct a unidirectional LSTM network with an input layer dimension of 342 and a hidden layer dimension of 128. The network contains 3 LSTM layer units with a dropout rate of 0.

3. Input the 342-dimensional speech-level video feature vector into the unidirectional LSTM network to extract the high-level features of the video and obtain the 128-dimensional deep video features V.

4. The multimodal emotion recognition method based on trimodal Mamba interaction and cascaded hierarchical fusion according to claim 1, characterized in that, Step 6 specifically includes: Step 6-1: Fuse the deep-encoded speech features A, text features T, and video features V to obtain the initial multimodal features H. init : H init =Linear(Concat(A,V,T)) Step 6-2: Aggregate the text enhancement features from the first, second, and third stages together, and obtain the aggregated text feature T through cascaded fusion. sum : Where i represents the enhancement stage, aug-T (i) This represents the text features enhanced in stage i. Step 6-3: Aggregate the speech enhancement features from the first, second, and third stages together, and obtain the aggregated speech feature A through cascaded fusion. sum : Among them, aug-A (i) This represents the speech features enhanced in stage i. Step 6-4: Aggregate the video enhancement features from the first, second, and third stages together, and obtain the video aggregation feature V through cascaded fusion. sum : Among them, aug-V (i) This represents the text features enhanced in stage i. Step 6-5: Fuse text aggregation features, speech aggregation features, and video aggregation features to obtain multimodal aggregation feature H. sum : H sum =Linear(Concat(A sum ,V sum ,T sum )) Step 6-6: Fuse the initial multimodal features and the aggregated multimodal features to obtain the final multimodal joint feature H. final : H final =Concat(H init， H sum ) The final multimodal joint features are then input into the sentiment classifier for sentiment prediction.

5. The multimodal emotion recognition method based on trimodal Mamba interaction and cascaded hierarchical fusion according to claim 1, characterized in that, In step 7, the specific method for performance evaluation of the proposed multimodal emotion recognition method based on trimodal Mamba interaction and cascaded hierarchical fusion includes the following steps: Step 7-1: Compare the current mainstream multimodal emotion recognition methods based on trimodal Mamba interaction and cascaded hierarchical fusion to verify the performance and efficiency of the present invention; Step 7-2: Compare and analyze the role of each module in the multimodal emotion recognition method based on trimodal Mamba interaction and cascaded hierarchical fusion.

Images