Multi-modal emotion recognition method based on three-modal Mama interaction and cascade hierarchical fusion

Abstract

The invention discloses a multi-modal emotion recognition method based on three-modal Mama interaction and cascade hierarchical fusion, and the method comprises the steps: firstly, carrying out the coding of voice, text and video, extracting depth features, capturing the long-distance dependence relation of different modal information through a three-modal Mama interaction module, and carrying out the recognition of the long-distance dependence relation of different modal information; the complementarity among different modal information is utilized to obtain enhanced representation of the multi-modal features; then, semantic features of different stages are dynamically integrated by utilizing a proposed cascade hierarchical fusion mechanism, and multi-modal emotion representation with rich semantics is obtained; finally, an ablation experiment is carried out, and the effectiveness of the method is verified. According to the method, effective parallel computing of multi-modal emotion recognition is realized, and the recognition performance and generalization ability of a multi-modal emotion recognition system are improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of multimodal emotion recognition, and in particular relates to a multimodal emotion recognition method based on trimodal Mamba interaction and cascaded hierarchical fusion. Background Art

[0002] With the continuous advancement of information technology, the field of human-computer interaction has become a booming research direction in computer science. In recent years, more and more research has focused on "affective computing", that is, developing systems that can recognize, understand and manage human emotional responses. The goal of affective computing is to enable machines to understand and respond to human emotions, thereby improving user experience in multiple application scenarios, such as enhancing customer experience in voice call centers and helping to diagnose depression through voice analysis. Emotion recognition is a key component in affective computing, which involves sensing and interpreting human emotional states through multiple channels. Common emotion recognition methods use multiple modal information such as speech, facial expressions, body postures, and text to identify the user's emotional state. Each modality has its own unique advantages and limitations. For example, speech signals can convey rich emotional information, but the effect is poor in noisy environments; facial expressions can reflect immediate emotional states, but may be inaccurate in occlusion or low light conditions. Multimodal emotion recognition methods attempt to improve the accuracy and robustness of recognition by fusing information from multiple modalities. However, the heterogeneity of multimodal data poses a huge challenge to information fusion. Data of different modalities have significant differences in time, space and feature dimensions. How to effectively integrate these data has become a major challenge in the field of emotion recognition.

[0003] Traditional multimodal fusion methods usually include early fusion, late fusion, and mid-term fusion. Early fusion directly splices the raw data or primary features of different modalities together and then inputs them into the emotion recognition model. Late fusion processes the data of each modality independently and finally combines their respective recognition results. However, these methods suffer from information loss and the inability to fully capture the complex relationship between modalities. In recent years, deep learning technology has attracted a lot of attention due to its significant capabilities in feature learning and nonlinear fitting. The most common fusion method based on deep neural networks is to extract features from multiple modalities and then splice these features into an extended feature vector. However, this simple splicing method often fails to capture complex intra-modal and inter-modal dynamics, resulting in poor performance in recognizing complex emotional states.

[0004] To address these limitations, the Tensor Fusion Network (TFN) was introduced, which uses triple Cartesian products to model unimodal, bimodal, and trimodal interactions. Although TFN is effective to a certain extent, its high computational complexity and large number of parameters may lead to increased resource requirements and potential overfitting problems. In addition, human emotions are often hidden in the long-term expression of multimodal signals. If the long-range dependencies between these modalities are not captured, key emotion-related interactions may not be identified.

[0005] In order to further improve the performance of emotion recognition, the Transformer model was introduced, which is good at capturing long-range dependencies and modeling complex interactions between modalities. The multi-head attention mechanism in the Transformer can perform powerful global feature extraction when modeling multimodal emotion data. However, the computational complexity of the Transformer grows with the square of the input token length, which limits its application in long sequence processing.

[0006] Recently, the progress of state space models (SSMs) has significantly solved this problem, such as Mamba, which achieves global perception with linear complexity. SSMs have performed well in various computer vision applications and provide significant computational efficiency compared to Transformers. However, the potential of SSMs in multimodal information fusion has not been fully explored, especially in multimodal emotion recognition tasks. Summary of the invention

[0007] The purpose of the present invention is to address the defects and shortcomings of the above-mentioned prior art, and propose a multimodal emotion recognition method based on trimodal Mamba interaction and cascaded hierarchical fusion, by first introducing a trimodal Mamba interaction module, capturing the long-distance dependency of information between different modalities, making full use of the complementarity of different modal information, and realizing complex interactions and seamless feature integration between three different modalities, and then proposing a cascaded hierarchical fusion strategy for dynamically integrating enhanced multimodal features from different stages, thereby obtaining semantically rich multimodal emotion representation. The present invention realizes effective parallel computing for multimodal emotion recognition, improves training speed and recognition performance, and has better generalization ability and practicality.

[0008] The technical solution adopted by the present invention to solve the technical problem is: a multimodal emotion recognition method based on trimodal Mamba interaction and cascade hierarchical fusion, the method comprising the following steps:

[0009] Step 1: Perform shallow feature extraction on the original input speech, text and video modalities respectively;

[0010] Step 2: Input the extracted shallow features into the deep coding network to obtain deep features;

[0011] Step 3: Input the deeply encoded speech features, text features, and video features into the trimodal Mamba interaction module, and obtain the main modality enhanced features after the first stage of complementary enhancement through interactive fusion between different modal information;

[0012] Step 4: Input the main modality enhancement features after the first stage enhancement into the trimodal Mamba interaction module, and obtain the main modality enhancement features after the second stage complementary enhancement through interactive fusion between different modal information;

[0013] Step 5: Input the main modality enhancement features after the second stage enhancement into the trimodal Mamba interaction module, and obtain the main modality enhancement features after the third stage complementary enhancement through interactive fusion between different modal information;

[0014] Step 6: The speech features, text features and video features after deep encoding and the main modality enhancement features after the first, second and third stages of enhancement are combined and the final multimodal emotion features are obtained through a cascade hierarchical fusion mechanism. The final multimodal emotion features are then input into the emotion classifier for emotion prediction.

[0015] Step 7: Perform a performance evaluation on the proposed multimodal emotion recognition method based on trimodal Mamba interaction and cascaded hierarchical fusion.

[0016] Furthermore, the step 1 specifically includes:

[0017] Step 1-1: Pass each text data through the Bert sub-network to obtain a 1024-dimensional discourse-level text feature vector;

[0018] Step 1-2: Each speech data is passed through the OpenSMILE toolkit and the “IS13 ComParE” configuration for feature extraction to obtain a 130-dimensional utterance-level speech feature vector;

[0019] Step 1-3: Pass each video data through the DenseFace sub-network to obtain a 342-dimensional utterance-level video feature vector.

[0020] Furthermore, the step 2 specifically includes:

[0021] Step 2-1: Build a unidirectional LSTM network with an input layer dimension of 1024 and a hidden layer dimension of 128. The network contains 2 LSTM layer units with a dropout rate of 0.3. Input a 1024-dimensional utterance-level text feature vector into the unidirectional LSTM network to extract high-level features of the text and obtain a 128-dimensional text deep feature T.

[0022] Step 2-2: Build a bidirectional LSTM network with an input layer dimension of 130 and a hidden layer dimension of 64. The network contains 3 LSTM layer units and a dropout rate of 0.3. Then input the 130-dimensional utterance-level speech feature vector into the bidirectional LSTM network to extract the high-level features of the speech and obtain the 128-dimensional deep-layer speech feature A.

[0023] Step 2-3: Build a unidirectional LSTM network with an input layer dimension of 342 and a hidden layer dimension of 128. The network contains 3 LSTM layer units with a dropout rate of 0.3. Input the 342-dimensional speech-level video feature vector into the unidirectional LSTM network to extract the high-level features of the video and obtain the 128-dimensional video deep feature V.

[0024] Furthermore, the step 3 specifically includes:

[0025] Step 3-1: With speech as the main modality and video and text as auxiliary modalities, the deeply encoded speech feature A, text feature T and video feature V are input into the trimodal Mamba interaction module to obtain the speech enhancement feature aug-A after the complementary enhancement in the first stage. (1) :

[0026] X v→a =MLP(G 1 ⊙Y a +G 1 ⊙Y v )

[0027] X t→a =MLP(G 2 ⊙Y a +G 2 ⊙Y t )

[0028] aug-A (i) =Linear(X v→a +X t→a )

[0029] Where MLP stands for multi-layer perceptron, Linear stands for linear layer, G 1 , G 2 represents the gate vector, Y a represents the long-distance dependency feature of speech modality, Y vrepresents the long-distance dependency feature of the video modality, Y t represents the long-distance dependency feature of text modality, X t→a represents the video-audio interaction feature when the main modality is voice, X t→a represents the text-audio interaction feature when the main modality is speech, ⊙ represents element-wise multiplication, and i represents the current stage of complementary enhancement;

[0030] Step 3-2: With text as the main modality and voice and video as auxiliary modalities, the deeply encoded voice features A, text features T and video features V are input into the trimodal Mamba interaction module to obtain the text enhancement feature aug-T after the complementary enhancement in the first stage. (1) :

[0031] X a→t =MLP(G 1 ⊙Y t +G 1 ⊙Y a )

[0032] X v→t =MLP(G 2 ⊙Y t +G 2 ⊙Y v )

[0033] aug-T (i) =Linear(X a→t +X v→t )

[0034] Where X a→t represents the audio-text interaction feature when the main modality is text, X v→t Represents the video-text interaction features when the main modality is text;

[0035] Step 3-3: With video as the main modality and text and speech as auxiliary modalities, the deeply encoded speech feature A, text feature T and video feature V are input into the trimodal Mamba interaction module to obtain the video enhancement feature aug-V after the first stage of complementary enhancement (1) :

[0036] X a→v =MLP(G 1 ⊙Y v +G 1 ⊙Y a )

[0037] X t→v =MLP(G 2 ⊙Y v +G 2 ⊙Y t )

[0038] aug-T (i) =Linear(X a→v +X t→v )

[0039] Where X a→v represents the audio-video interaction feature when the main modality is video, X t→v Represents the text-video interaction features when the main modality is video.

[0040] Furthermore, the step 4 specifically includes:

[0041] Step 4-1: With voice as the main modality and video and text as auxiliary modalities, the voice features, text features and video features after complementary enhancement in the first stage are input into the trimodal Mamba interaction module to obtain the voice enhancement features aug-A after complementary enhancement in the second stage. (2) ;

[0042] Step 4-2: With text as the main modality and voice and video as auxiliary modalities, the voice features, text features and video features after complementary enhancement in the first stage are input into the trimodal Mamba interaction module to obtain the text enhancement features after complementary enhancement in the second stage, aug-T (2) ;

[0043] Step 4-3: With video as the main modality and text and voice as auxiliary modalities, the voice features, text features and video features after complementary enhancement in the first stage are input into the trimodal Mamba interaction module to obtain the video enhancement features aug-V after complementary enhancement in the second stage. (2) .

[0044] Furthermore, the step 5 specifically includes:

[0045] Step 5-1: With voice as the main modality and video and text as auxiliary modalities, the voice features, text features and video features after the second stage of complementary enhancement are input into the trimodal Mamba interaction module to obtain the voice enhancement features aug-A after the third stage of complementary enhancement (3) ;

[0046] Step 5-2: With text as the main modality and text and video as auxiliary modalities, the speech features, text features and video features after complementary enhancement in the second stage are input into the trimodal Mamba interaction module to obtain the text enhancement features after complementary enhancement in the third stage, aug-T (3) ;

[0047] Step 5-3: With video as the main modality and text and voice as auxiliary modalities, the voice features, text features and video features after the second stage of complementary enhancement are input into the trimodal Mamba interaction module to obtain the video enhancement features aug-V after the third stage of complementary enhancement.(3) .

[0048] Furthermore, the step 6 specifically includes:

[0049] Step 6-1: Fuse the deeply encoded speech feature A, text feature T and video feature V to obtain the multimodal initial feature H init :

[0050] H init =Linear(Concat(A, V, T))

[0051] Step 6-2: Aggregate the text enhancement features after the first, second, and third stages of enhancement, and obtain the text aggregation feature T through cascade fusion sum :

[0052]

[0053] Among them, i represents the stage of enhancement, aug-T (i) Represents the text features after enhancement in the i-th stage;

[0054] Step 6-3: Aggregate the speech enhancement features after the first, second, and third stages of enhancement, and obtain the speech aggregation feature A through cascade fusion sum :

[0055]

[0056] Among them, aug-A (i) represents the speech features after enhancement in the i-th stage;

[0057] Step 6-4: Aggregate the video enhancement features after the first, second, and third stages of enhancement, and obtain the video aggregation feature V through cascade fusion sum :

[0058]

[0059] Among them, aug-V (i) Represents the enhanced video features of the i-th stage;

[0060] Step 6-5: Fuse text aggregation features, voice aggregation features, and video aggregation features to obtain multimodal aggregation features H sum :

[0061] H sum =Linear(Concat(A sum , V sum , T sum ))

[0062] Step 6-6: Fuse the multimodal initial features and the multimodal aggregation features to obtain the final multimodal joint features H final :

[0063] H final =Concat(H init , H sum )

[0064] The final multimodal joint features are then input into the sentiment classifier for sentiment prediction.

[0065] Furthermore, in step 7, a specific method for evaluating the performance of the proposed multimodal emotion recognition method based on trimodal Mamba interaction and cascaded hierarchical fusion includes the following steps:

[0066] Step 7-1: Conduct a comparative experiment between the multimodal emotion recognition method based on trimodal Mamba interaction and cascaded hierarchical fusion and the mainstream multimodal emotion recognition method to verify the performance and efficiency of the present invention;

[0067] Step 7-2: Compare and analyze the role of each module in Mamba's trimodal interaction and cascaded hierarchical fusion multimodal emotion recognition methods.

[0068] Beneficial effects:

[0069] 1. The present invention proposes a tri-modal Mamba interaction module to capture the long-distance dependency of information between different modalities, and at the same time utilizes the complementarity of auxiliary modal information to primary modal information to obtain an enhanced representation of primary modal features.

[0070] 2. The present invention introduces a cascaded hierarchical fusion mechanism to dynamically integrate enhanced multimodal features from different stages to achieve effective fusion of multimodal information, thereby obtaining semantically rich multimodal emotion representation. BRIEF DESCRIPTION OF THE DRAWINGS

[0071] Figure 1 It is a schematic diagram of the operation flow of the present invention.

[0072] Figure 2 This is a block diagram of the tri-modal Mamba interaction module of the present invention. DETAILED DESCRIPTION

[0073] The invention will be further described in detail below in conjunction with the accompanying drawings.

[0074] In order to make the purpose, technical solution and advantages of the present invention clearer, the present invention is described in detail below in conjunction with the accompanying drawings and specific embodiments. It should be noted that, in order to avoid blurring the present invention due to unnecessary details, only structures and / or processing steps closely related to the solutions of the present invention are shown in the accompanying drawings, while other details that are not closely related to the present invention are omitted.

[0075] In addition, it should be noted that the terms "comprises", "includes" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or apparatus that includes a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or apparatus.

[0076] like Figure 1 to Figure 2 As shown, the present invention proposes a multimodal emotion recognition method based on trimodal Mamba interaction and cascade hierarchical fusion, applies the Mamba network to multimodal emotion recognition, fully captures the long-distance dependency relationship between information of different modalities, and can build a higher performance multimodal emotion recognition system, which specifically includes the following steps:

[0077] Step 1: Perform shallow feature extraction on the speech, video and text modalities in the original input database. The dataset used in the present invention is the IEMOCAP emotion database, which consists of about 12 hours of audio and video content recorded by 10 professional performers (5 males and 5 females) in a professional studio. The corpus is divided into 5 parts, each of which includes two actors, one male and one female.

[0078] The above step 1 of the present invention specifically comprises:

[0079] Step 1-1: Pass each text data through the Bert sub-network to obtain a 1024-dimensional discourse-level text feature vector;

[0080] Step 1-2: Each speech data is passed through the OpenSMILE toolkit and the “IS13 ComParE” configuration for feature extraction to obtain a 130-dimensional utterance-level speech feature vector;

[0081] Step 1-3: Pass each video data through the DenseFace sub-network to obtain a 342-dimensional utterance-level video feature vector.

[0082] Step 2: Input the extracted shallow features into the deep coding network to obtain deep features. For the speech modality, build a bidirectional LSTM network for deep coding. For the text and video modalities, build a unidirectional LSTM network for deep coding.

[0083] The above step 2 of the present invention specifically includes:

[0084] Step 2-1: Build a unidirectional LSTM network with an input layer dimension of 1024 and a hidden layer dimension of 128. The network contains 2 LSTM layer units with a dropout rate of 0.3. Input a 1024-dimensional utterance-level text feature vector into the unidirectional LSTM network to extract high-level features of the text and obtain a 128-dimensional text deep feature T.

[0085] Step 2-2: Build a bidirectional LSTM network with an input layer dimension of 130 and a hidden layer dimension of 64. The network contains 3 LSTM layer units and a dropout rate of 0.3. Then input the 130-dimensional utterance-level speech feature vector into the bidirectional LSTM network to extract the high-level features of the speech and obtain the 128-dimensional deep-layer speech feature A.

[0086] Step 2-3: Build a unidirectional LSTM network with an input layer dimension of 342 and a hidden layer dimension of 128. The network contains 3 LSTM layer units with a dropout rate of 0.3. Input the 342-dimensional speech-level video feature vector into the unidirectional LSTM network to extract the high-level features of the video and obtain the 128-dimensional video deep feature V.

[0087] Step 3: Input the deeply encoded speech features A, text features T and video features V into the trimodal Mamba interaction module. Through the interactive fusion of different modal information, the main modality enhanced features after the first stage of complementary enhancement are obtained.

[0088] The above step 3 of the present invention specifically includes:

[0089] Step 3-1: With voice as the main modality and text and video as auxiliary modalities, the voice features, text features and video features after deep encoding are input into the trimodal Mamba interaction module to obtain the voice enhancement features after complementary enhancement in the first stage;

[0090] Step 3-2: With text as the main modality and video and voice as auxiliary modalities, the voice features, text features and video features after deep encoding are input into the trimodal Mamba interaction module to obtain the text enhancement features after complementary enhancement in the first stage;

[0091] Step 3-3: With video as the main modality and text and voice as auxiliary modalities, the deeply encoded voice features, text features, and video features are input into the tri-modal Mamba interaction module to obtain the video enhancement features after the complementary enhancement in the first stage.

[0092] Among them, step 3-1 specifically includes:

[0093] Step 3-1-1: Receive the deeply encoded features from the main modality (speech modality) and two auxiliary modalities (video modality and text modality), perform layer normalization on the deeply encoded features of each modality to standardize the feature data, and obtain the standardized features of the main modality and the standardized features of the two auxiliary modalities respectively;

[0094] Step 3-1-2: The standardized features of the main modality and auxiliary modality obtained in step 3-1-1 above are passed through three parallel multi-layer perceptrons and convolutional layers to obtain the deep feature X m :

[0095]

[0096] Among them, Conv1D represents one-dimensional convolution, MLP represents multi-layer perceptron, and LN represents layer normalization. Represents the three modal features of the input (one main modal feature and two auxiliary modal features), a represents the voice modality, v represents the video modality, and t represents the text modality;

[0097] Step 3-1-3: Input the deep features obtained in step 3-1-2 above into SSM to model the long-distance dependency relationship and obtain the long-distance dependency feature Y m :

[0098] Y m =SSM(X m ), m∈{a,v,t}

[0099] Where SSM stands for state space model;

[0100] Step 3-1-4: Input the main modality normalized features into two independent MLPs 1 、MLP 2 , the main modality is mapped to the complementary space of the two auxiliary modalities, and then two gating vectors G are generated through the activation function 1 , G 2 :

[0101]

[0102] Among them, when voice is the main mode, the input for When the text is in the main modal for When the video is the main mode, the input for At this time, the main mode is voice mode. That is

[0103] Step 3-1-5: The two gated vectors obtained in step 3-1-4 are used to modulate the features of the long-distance dependent modality obtained in step 3-1-3 to obtain the video-audio interaction feature X when the main modality is speech. v→a and text-audio interaction feature X t→a :

[0104] X v→a =MLP(G 1 ⊙Y a +G 1 ⊙Y v )

[0105] X t→a =MLP(G 2 ⊙Y a +G 2 ⊙Y t )

[0106] Where ⊙ represents element-wise multiplication;

[0107] Step 3-1-6: Fuse the video-audio interaction features and the text-audio interaction features and pass them through a linear layer to obtain the speech enhancement feature aug-A after complementary enhancement in the first stage. (1) :

[0108] aug-A (1) =Linear(X v→a +X t→a )

[0109] The specific steps of step 3-2 are similar to those of step 3-1, except that the input of step 3-2 is text-based, and video and voice-based as auxiliary modes. That is The output is the text enhancement feature aug-T after the complementary enhancement in the first stage (1) :

[0110] X v→t =MLP(G 1 ⊙Y t +G 1 ⊙Y v )

[0111] X a→t =MLP(G 2 ⊙Y t +G 2 ⊙Y a )

[0112] aug-T (1) =Linear(X v→t +X a→t )

[0113] Where X v→t represents the video-text interaction feature when the main modality is text, X a→t Represents the audio-text interaction features when the main modality is text;

[0114] The specific steps of step 3-3 are similar to those of step 3-1, except that the input of step 3-3 is video as the main mode, and text and voice as auxiliary modes. That is The output is the video enhancement feature aug-V after the first stage complementary enhancement (1) :

[0115] X t→v =MLP(G 1 ⊙Y v +G 1 ⊙Y t )

[0116] X a→v =MLP(G 2 ⊙Y v +G 2 ⊙Y a )

[0117] aug-V (1) =Linear(X t→v +X a→v )

[0118] Where X t→v represents the text-video interaction feature when the main mode is video, X a→v Represents the audio-video interaction features when the main modality is video;

[0119] Step 4: Input the main modality enhancement features after the first stage enhancement into the tri-modal Mamba interaction module, and obtain the main modality enhancement features after the second stage complementary enhancement through interactive fusion between different modal information.

[0120] The above step 4 of the present invention specifically includes:

[0121] Step 4-1: With voice as the main modality and video and text as auxiliary modalities, the voice features, text features and video features after complementary enhancement in the first stage are input into the trimodal Mamba interaction module to obtain the voice enhancement features aug-A after complementary enhancement in the second stage. (2) ;

[0122] Step 4-2: With text as the main modality and voice and video as auxiliary modalities, the voice features, text features and video features after complementary enhancement in the first stage are input into the trimodal Mamba interaction module to obtain the text enhancement features after complementary enhancement in the second stage, aug-T(2) ;

[0123] Step 4-3: With video as the main modality and text and voice as auxiliary modalities, the voice features, text features and video features after complementary enhancement in the first stage are input into the trimodal Mamba interaction module to obtain the video enhancement features aug-V after complementary enhancement in the second stage. (2) .

[0124] Step 5: Input the main modality enhancement features after the second stage enhancement into the trimodal Mamba interaction module, and obtain the main modality enhancement features after the third stage complementary enhancement through interactive fusion between different modal information.

[0125] The above step 5 of the present invention specifically includes:

[0126] Step 5-1: With voice as the main modality and video and text as auxiliary modalities, the voice features, text features and video features after the second stage of complementary enhancement are input into the trimodal Mamba interaction module to obtain the voice enhancement features aug-A after the third stage of complementary enhancement (3) ;

[0127] Step 5-2: With text as the main modality and text and video as auxiliary modalities, the speech features, text features and video features after complementary enhancement in the second stage are input into the trimodal Mamba interaction module to obtain the text enhancement features after complementary enhancement in the third stage, aug-T (3) ;

[0128] Step 5-3: With video as the main modality and text and voice as auxiliary modalities, the voice features, text features and video features after the second stage of complementary enhancement are input into the trimodal Mamba interaction module to obtain the video enhancement features aug-V after the third stage of complementary enhancement. (3) .

[0129] Step 6: Combine the deeply encoded speech features, text features, and video features as well as the main modality enhancement features after the first, second, and third stages of enhancement and obtain the final multimodal emotion features through a cascaded hierarchical fusion mechanism.

[0130] The above step 6 of the present invention specifically includes:

[0131] Step 6-1: Fuse the deeply encoded speech feature A, text feature T and video feature V to obtain the multimodal initial feature H init :

[0132] H init =Linear(Concat(A, V, T))

[0133] Step 6-2: Aggregate the text enhancement features after the first, second, and third stages of enhancement, and obtain the text aggregation feature T through cascade fusion sum :

[0134]

[0135] Among them, i represents the stage of enhancement, aug-T (i) Represents the text features after enhancement in the i-th stage;

[0136] Step 6-3: Aggregate the speech enhancement features after the first, second, and third stages of enhancement, and obtain the speech aggregation feature A through cascade fusion sum :

[0137]

[0138] Among them, aug-A (i) represents the speech features after enhancement in the i-th stage;

[0139] Step 6-4: Aggregate the video enhancement features after the first, second, and third stages of enhancement, and obtain the video aggregation feature V through cascade fusion sum :

[0140]

[0141] Among them, aug-V (i) Represents the enhanced video features of the i-th stage;

[0142] Step 6-5: Fuse text aggregation features, voice aggregation features, and video aggregation features to obtain multimodal aggregation features H sum :

[0143] H sum =Linear(Concat(A sum , V sum , T sum ))

[0144] Step 6-6: Fuse the multimodal initial features and the multimodal aggregation features to obtain the final multimodal joint features H final :

[0145] H final =Concat(H init , H sum )

[0146] The final multimodal joint features are then input into the sentiment classifier for sentiment prediction.

[0147] Among them, step 6-1 specifically includes:

[0148] Step 6-1-1: Connect the deeply encoded speech features, text features, and video features together through Concat, and then reduce the dimension through pooling;

[0149] Step 6-1-2: Pass the pooled features in step 6-1-1 through the fully connected layer and Relu activation function in turn to obtain the multimodal initial features.

[0150] Among them, step 6-5 specifically includes:

[0151] Step 6-5-1: Connect the text aggregation features obtained in step 6-2, the voice aggregation features obtained in step 6-3, and the video aggregation features obtained in step 6-4 through Concat, and then reduce the dimension through pooling;

[0152] Step 6-5-2: Pass the pooled features in step 6-5-1 through the fully connected layer and Relu activation function in turn to obtain multimodal aggregation features.

[0153] Wherein, step 6-6 specifically includes:

[0154] Step 6-6-1: Concatenate the multimodal initial features obtained in step 6-1 and the multimodal aggregate features obtained in step 6-5 through Concat to obtain the final multimodal joint feature H final ;

[0155] Step 6-6-2: The final multimodal joint feature H in step 6-6-1 final Input into the sentiment classifier for sentiment discrimination and classification.

[0156] Step 7: Perform a performance evaluation on the proposed multimodal emotion recognition method based on trimodal Mamba interaction and cascaded hierarchical fusion.

[0157] The above step 7 of the present invention specifically includes:

[0158] Step 7-1: Conduct a comparative experiment between the multimodal emotion recognition method based on trimodal Mamba interaction and cascaded hierarchical fusion and the mainstream multimodal emotion recognition method to verify the performance and efficiency of the present invention.

[0159] Step 7-2: Compare and analyze the role of each module in Mamba's trimodal interaction and cascaded hierarchical fusion multimodal emotion recognition methods.

[0160] The present invention uses the IEMOCAP dataset for multimodal emotion recognition. The dataset contains 10,039 audio-visual data, including audio, video and contextual information, recording the dialogues of ten professional actors in hypothetical scenes and the corresponding emotion categories. The excitement category is merged into the happiness category. We focus on four main emotions: anger, happiness, sadness and neutrality, with a total of 5,531 data for the experiment. The model performance is evaluated using a five-fold leave-one-session cross-validation, with one session selected from each fold as the test set and the remaining four sessions used for training. The evaluation indicators include unweighted accuracy (UA) and weighted accuracy (WA). UA is the average accuracy of all emotion categories, and WA is weighted by the number of category samples. UA provides a balanced performance measure. The experiment is implemented using the PyTorch framework, using the Adam optimizer, a learning rate of 2e-4, a batch size of 32, and a dropout rate between 0.3 and 0.4 to prevent overfitting. The training was performed on a high-performance computing system with a 24GB NVIDIA GeForce RTX 3090 for 50 epochs, and L2 regularization with a parameter of 1e-5 was used during training.

[0161] The simulation experiment and data of the present invention are as follows:

[0162] First, in order to verify the effectiveness of the multimodal emotion recognition method based on trimodal Mamba interaction and cascaded hierarchical fusion, the proposed method is compared with some existing mainstream multimodal emotion recognition methods, including: MSSF, MAFN, MEMoBERT, SSMM, MCEPM and MMIN. MSSF integrates a modality-specific self-supervised pre-training architecture and combines non-linguistic cues captured by motion to achieve efficient emotion recognition through feature fusion at the decision layer. MAFN uses multi-head attention mechanism and fusion network technology to improve the accuracy of emotion recognition by learning the intrinsic correlation between cross-modal features. MEMoBERT uses self-supervised learning and prompt-based technology and uses cross-modal Transformer to model cross-modal context. SSMM uses deep autoencoders for potential representation learning and improves emotion recognition performance through cross-modal distribution matching technology. MCEPM combines facial expressions, audio signals, text and motion data to enhance the responsiveness to relevant stimuli through product fusion. MMIN achieves robust multimodal joint representation through cascaded residual autoencoders and cycle consistency learning. The experimental results are shown in Table 1:

[0163] Table 1 Performance comparison with other mainstream methods on the IEMOCAP dataset

[0164] Model WA UA SSMM 75.6 74.5 MAFN 75.6 71.4 MSSF - 77.58 MMIN 76.51 77.79 MCEPM - 78.2 MEMoBERT 80.01 81.09 Ours 81.22 82.06

[0165] In comparative experiments, the trimodal Mamba interaction and cascaded hierarchical fusion model of the present invention outperforms the above existing methods on the IEMOCAP dataset. Specifically, the method of the present invention improves 5.62% on WA and 7.56% on UA ​​over SSMM. Compared with MSSF and MCEPM, the method of the present invention improves 4.48% and 3.86% on UA, respectively. In addition, compared with MAFN, the method of the present invention improves 5.62% on WA and 10.66% on UA. Compared with MMIN, the method of the present invention improves 4.71% and 4.27% on WA and UA, respectively. Compared with MEMoBERT, the method of the present invention improves 1.21% and 0.97% on WA and UA, respectively. These comparative results show that the multimodal emotion recognition method based on trimodal Mamba interaction and cascaded hierarchical fusion of the present invention has significant advantages in multimodal emotion recognition tasks, especially when dealing with complex emotion recognition tasks, showing significant improvements in multimodal data fusion and emotion recognition accuracy.

[0166] In order to analyze the contribution of each component in the multimodal emotion recognition method based on trimodal Mamba interaction and cascaded hierarchical fusion proposed in this paper, an ablation experiment was designed to evaluate the performance of trimodal interaction Mamba modules and cascaded hierarchical fusion mechanisms with various configurations. The ablation experiment results are shown in Table 2:

[0167] Table 2 Ablation experiment results

[0168] Model WA UA Baseline 78.10 79.09 TI-Mamba(S=1) 80.16 81.24 TI-Mamba(S=2) 80.38 81.43 TI-Mamba (S=3) 80.61 81.69 TI-Mamba (S = 2 cascaded) 80.86 81.67 TI-Mamba (S = 3 cascaded) 81.22 82.06

[0169] In the ablation experiment on the IEMOCAP dataset, TI-Mamba represents the tri-modal interactive Mamba module, S represents the number of stages used by TI-Mamba, and cascaded represents the cascaded hierarchical fusion mechanism.

[0170] The ablation experiment focuses on the impact of each model on WA and UA. The baseline model uses independent LSTM encoders to deeply encode the audio, video, and text modalities respectively, and integrates these modalities through late fusion. The baseline model has a WA of 78.10% and a UA of 79.09%. On the basis of the baseline model, after integrating a one-stage trimodal interaction Mamba module, the performance is significantly improved, with WA increasing to 80.16% and UA increasing to 81.24%. As the number of stages of the trimodal interaction Mamba module increases to two, the model performance continues to improve, reaching WA 80.38% and UA 81.43%. When the number of stages of the trimodal interaction Mamba module increases to three, the model performance is further improved, reaching WA 80.61% and UA 81.69%. After combining the cascade layered fusion mechanism, the model performance is further enhanced. The model configured as a two-stage cascade fusion achieves WA 80.86% and UA81.67%. The model configured as a three-stage cascade fusion performs best, achieving WA 81.22% and UA 82.06%. These experimental results show that the trimodal interaction Mamba module has a significant effect in enhancing multimodal emotion recognition, especially when combined with the cascaded hierarchical fusion mechanism. Each additional stage and the introduction of the cascaded hierarchical fusion mechanism help to gradually improve the performance, indicating the value of deeper feature interactions and more complex integration strategies.

[0171] In summary, the present embodiment proposes a multimodal emotion recognition method based on trimodal Mamba interaction and cascaded hierarchical fusion, and proposes a method that can improve the performance of the multimodal emotion recognition system from the trimodal Mamba interaction module and cascaded hierarchical fusion mechanism. The experimental results verify the rationality and effectiveness of these two modules. Compared with other methods, the method of this embodiment is improved in multiple measurement indicators and has a good reference significance in practical applications.

[0172] The above embodiments are only used to illustrate the technical solution of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solution of the present invention may be modified or replaced by equivalents without departing from the spirit and scope of the technical solution of the present invention.

Claims

1. A multimodal emotion recognition method based on trimodal Mamba interaction and cascaded hierarchical fusion, characterized in that: The method comprises the following steps: Step 1: Perform shallow feature extraction on the original input speech, text and video modalities respectively; Step 2: Input the extracted shallow features into the deep encoding network to extract deep features; Step 3: Input the deeply encoded speech features, text features, and video features into the trimodal Mamba interaction module, and obtain the main modality enhanced features after the first stage of complementary enhancement through interactive fusion between different modal information; Step 4: Input the main modality enhancement features after the first stage enhancement into the trimodal Mamba interaction module, and obtain the main modality enhancement features after the second stage complementary enhancement through interactive fusion between different modal information; Step 5: Input the main modality enhancement features after the second stage enhancement into the trimodal Mamba interaction module, and obtain the main modality enhancement features after the third stage complementary enhancement through interactive fusion between different modal information; Step 6: The speech features, text features and video features after deep encoding and the main modality enhancement features after the first, second and third stages of enhancement are combined and the final multimodal emotion features are obtained through a cascade hierarchical fusion mechanism. The final multimodal emotion features are then input into the emotion classifier for emotion prediction. Step 7: Perform a performance evaluation on the proposed multimodal emotion recognition method based on trimodal Mamba interaction and cascaded hierarchical fusion.

2. According to claim 1, a multimodal emotion recognition method based on trimodal Mamba interaction and cascaded hierarchical fusion is characterized in that: The step 1 specifically includes: Step 1-1: Pass each text data through the Bert sub-network to obtain a 1024-dimensional discourse-level text feature vector; Step 1-2: Each speech data is passed through the OpenSMILE toolkit and the "IS13 ComParE" configuration for feature extraction to obtain a 130-dimensional utterance-level speech feature vector; Step 1-3: Pass each video data through the DenseFace sub-network to obtain a 342-dimensional utterance-level video feature vector.

3. The multimodal emotion recognition method based on trimodal Mamba interaction and cascaded hierarchical fusion according to claim 1 is characterized in that: The step 2 specifically includes: Step 2-1: Build a unidirectional LSTM network with an input layer dimension of 1024 and a hidden layer dimension of 128. The network contains 2 LSTM layer units with a dropout rate of 0.

3. Input a 1024-dimensional utterance-level text feature vector into the unidirectional LSTM network to extract high-level features of the text and obtain a 128-dimensional text deep feature T. Step 2-2: Build a bidirectional LSTM network with an input layer dimension of 130 and a hidden layer dimension of 64. The network contains 3 LSTM layer units and a dropout rate of 0.

3. Then input the 130-dimensional utterance-level speech feature vector into the bidirectional LSTM network to extract the high-level features of the speech and obtain the 128-dimensional deep-layer speech feature A. Step 2-3: Build a unidirectional LSTM network with an input layer dimension of 342 and a hidden layer dimension of 128. The network contains 3 LSTM layer units with a dropout rate of 0.

3. Input the 342-dimensional speech-level video feature vector into the unidirectional LSTM network to extract the high-level features of the video and obtain the 128-dimensional video deep feature V.

4. The multimodal emotion recognition method based on trimodal Mamba interaction and cascaded hierarchical fusion according to claim 1, characterized in that: The step 3 specifically includes: Step 3-1: With voice as the main modality and text and video as auxiliary modalities, the deeply encoded voice features, text features and video features are input into the trimodal Mamba interaction module to obtain the voice enhancement feature aug-A after complementary enhancement in the first stage. (1) : X v→a =MLP(G1⊙Y a +G1⊙Y v ) X t→a =MLP(G2⊙Y a +G2⊙Y t ) aug-A (i) =linear(X v→a +X t→a ) Among them, MLP stands for multi-layer perceptron, linear stands for linear layer, G1 and G2 stand for gate vectors, and Y a represents the long-distance dependency feature of speech modality, Y v represents the long-distance dependency feature of the video modality, Yt represents the long-distance dependency feature of the text modality, and X v→a represents the video-audio interaction feature when the main modality is voice, X t→a represents the text-audio interaction feature when the main modality is speech, ⊙ represents element-wise multiplication, and i represents the current stage of complementary enhancement; Step 3-2: With text as the main modality and video and voice as auxiliary modalities, the voice features, text features and video features after deep encoding are input into the trimodal Mamba interaction module to obtain the text enhancement feature aug-T after the complementary enhancement in the first stage. (1) : X a→t =MLP(G1⊙Y t +G1⊙Y a ) X v→t =MLP(G2⊙Y t +G2⊙Y v ) aug-T (i) =linear(X a→t +X v→t ) Among them, X a→t represents the audio-text interaction feature when the main modality is text, X v→t Represents the video-text interaction features when the main modality is text; Step 3-3: With video as the main modality and text and voice as auxiliary modalities, the voice features, text features and video features after deep encoding are input into the tri-modal Mamba interaction module to obtain the video enhancement feature aug-V after the complementary enhancement in the first stage. (1) : X a→v =MLP(G1⊙Y v +G1⊙Y a ) X t→v =MLP(G2⊙Y v +G2⊙Y t ) aug-T (i) =linear(X a→v +X t→v ) Among them, X a→v represents the audio-video interaction feature when the main modality is video, X t→v Represents the text-video interaction features when the main modality is video.

5. The multimodal emotion recognition method based on trimodal Mamba interaction and cascaded hierarchical fusion according to claim 1, characterized in that: The step 4 specifically includes: Step 4-1: With voice as the main modality and text and video as auxiliary modalities, the voice features, text features and video features after complementary enhancement in the first stage are input into the trimodal Mamba interaction module to obtain the voice enhancement features aug-A after complementary enhancement in the second stage. (2) ; Step 4-2: With text as the main modality and video and voice as auxiliary modalities, the voice features, text features and video features after complementary enhancement in the first stage are input into the trimodal Mamba interaction module to obtain the text enhancement features after complementary enhancement in the second stage, aug-T (2) ; Step 4-3: With video as the main modality and text and voice as auxiliary modalities, the voice features, text features and video features after complementary enhancement in the first stage are input into the trimodal Mamba interaction module to obtain the video enhancement features aug-V after complementary enhancement in the second stage. (2) .

6. A multimodal emotion recognition method based on trimodal Mamba interaction and cascaded hierarchical fusion according to claim 1, characterized in that: The step 5 specifically includes: Step 5-1: With voice as the main modality and text and video as auxiliary modalities, the voice features, text features and video features after the second stage of complementary enhancement are input into the trimodal Mamba interaction module to obtain the voice enhancement features aug-A after the third stage of complementary enhancement (3) ; Step 5-2: With text as the main modality and video and voice as auxiliary modalities, the voice features, text features and video features after complementary enhancement in the second stage are input into the trimodal Mamba interaction module to obtain the text enhancement features after complementary enhancement in the third stage, aug-T (3) ; Step 5-3: With video as the main modality and text and voice as auxiliary modalities, the voice features, text features and video features after the second stage of complementary enhancement are input into the trimodal Mamba interaction module to obtain the video enhancement features aug-V after the third stage of complementary enhancement. (3) .

7. The multimodal emotion recognition method based on trimodal Mamba interaction and cascaded hierarchical fusion according to claim 1, characterized in that: The step 6 specifically includes: Step 6-1: Fuse the deeply encoded speech feature A, text feature T and video feature V to obtain the multimodal initial feature H init : H init =Linear(Concat(A,V,T)) Step 6-2: Aggregate the text enhancement features after the first, second, and third stages of enhancement, and obtain the text aggregation feature T through cascade fusion sum : Among them, i represents the stage of enhancement, aug-T (i) Represents the text features after enhancement in the i-th stage; Step 6-3: Aggregate the speech enhancement features after the first, second, and third stages of enhancement, and obtain the speech aggregation feature A through cascade fusion sum : Among them, aug-A (i) represents the speech features after enhancement in the i-th stage; Step 6-4: Aggregate the video enhancement features after the first, second, and third stages of enhancement, and obtain the video aggregation feature V through cascade fusion sum : Among them, aug-V (i) Represents the text features after enhancement in the i-th stage; Step 6-5: Fuse text aggregation features, voice aggregation features, and video aggregation features to obtain multimodal aggregation features H sum : H sum =linear(Concat(A sum ,V sum ,T sum )) Step 6-6: Fuse the multimodal initial features and the multimodal aggregation features to obtain the final multimodal joint features H final : H final =Concat(H init ,H sum ) The final multimodal joint features are then input into the sentiment classifier for sentiment prediction.

8. The multimodal emotion recognition method based on trimodal Mamba interaction and cascaded hierarchical fusion according to claim 1, characterized in that: In step 7, a specific method for evaluating the performance of the proposed multimodal emotion recognition method based on trimodal Mamba interaction and cascaded hierarchical fusion includes the following steps: Step 7-1: Conduct a comparative experiment on the multimodal emotion recognition method based on trimodal Mamba interaction and cascade hierarchical fusion and the current mainstream multimodal emotion recognition method to verify the performance and efficiency of the present invention; Step 7-2: Compare and analyze the role of each module in the multimodal emotion recognition method based on trimodal Mamba interaction and cascaded hierarchical fusion.

Images