A multimodal dialogue sentiment recognition method, system, device and medium

Abstract

The application provides a multi-modal dialogue emotion recognition method, system, device and medium, and belongs to the technical field of emotion recognition. The method comprises the following steps: obtaining dialogue data, and extracting modal features of text, video and audio; inputting the modal features of the text, video and audio into a pre-trained multi-modal emotion recognition network, and utilizing a feature learning module for alignment, position information coding and speaker information embedding; utilizing a local region bidirectional Mamba module for feature enhancement; adopting a multi-source consistency attention mechanism to construct a shared semantic space, and performing overall deep fusion on the modal features; performing emotion prediction through an emotion classifier to obtain emotion labels of each utterance in the dialogue. The learning target is aligned across modalities through collaborative learning, the modal gap is reduced, and the feature distinguishability is enhanced. The method can effectively solve the emotion inertia and the computational efficiency bottleneck in long dialogue, realize deep multi-modal fusion beyond pairwise interaction, and significantly improve the accuracy of dialogue emotion recognition.

Description

Technical Field

[0001] This invention belongs to the field of emotion recognition technology, specifically relating to a multimodal dialogue emotion recognition method, system, device, and medium. Background Technology

[0002] Emotion recognition aims to identify and understand the emotional dynamics in dialogue, and it is widely used in intelligent human-computer interaction and intelligent services. Early research mainly focused on text-based methods, but these quickly revealed the inherent limitations of text: the expression of emotion is inherently multi-dimensional, and the meaning expressed by a speaker often dynamically changes with non-textual information (such as tone of voice and facial expressions), leading to insufficient expression of single-modal information in complex emotional contexts. To address this limitation, research on multimodal dialogue emotion recognition integrates and aligns information from multiple modalities, including text, speech, and vision, to predict the emotional state associated with each utterance across different modalities in a dialogue, thereby achieving more accurate emotion recognition.

[0003] With the advancement of deep learning technology, methods based on Transformer and graph neural networks have gradually become mainstream for sentiment recognition. These include: Transformer-based self-distillation models for contextual modeling of multimodal information; and graph neural network-based models for cross-modal modeling. In addition, cross-modal attention mechanisms are widely used in multimodal dialogue sentiment recognition (MERC), integrating cross-modal attention mechanisms into multimodal fusion to utilize available information from different modalities to obtain more comprehensive features. While existing methods have achieved success, models often fall into the trap of "historical sentiment inertia," i.e., over-reliance on previously established global sentiment tone, resulting in insufficient confidence when faced with a shift in sentiment, leading to misjudgments. Furthermore, existing models focus more on paired modal interactions (e.g., text-audio or text-visual fusion), neglecting the limited depth of overall multimodal information fusion. This shallow or pairwise fusion strategy struggles to capture complex and highly nonlinear dependencies between modalities, resulting in poor model performance when understanding complex emotions. Summary of the Invention

[0004] To address the shortcomings of existing technologies in processing dialogue emotion recognition, this invention provides a multimodal dialogue emotion recognition method, system, device, and medium.

[0005] To achieve the above objectives, the present invention provides the following technical solution: A multimodal dialogue emotion recognition method includes the following steps: Acquire dialogue data, and extract text modal features, video modal features, and audio modal features based on the dialogue data; The text modal features, video modal features, and audio modal features are input into a pre-trained multimodal emotion recognition network to obtain the emotion labels for each utterance in the dialogue. The multimodal emotion recognition network includes a feature learning module, a local region bidirectional Mamba module, a convolutional feedforward network, and a multi-source consistency attention mechanism. In the pre-trained multimodal emotion recognition network, the feature learning module converts text modality features, video modality features, and audio modality features into sequence representations of each modality. The local region bidirectional Mamba module processes the sequence representations of each modality to capture contextual information. The captured contextual information is then input into a convolutional feedforward network for local information enhancement to obtain context-aware representations of each modality. The context-aware representations of each modality are fused based on a multi-source consistency attention mechanism to obtain fused features. The emotion tags of each utterance in the dialogue are obtained based on the fused features.

[0006] Preferably, in the local bidirectional Mamba module, the sequence representation of a single modality is simultaneously fed into two parallel convolutional paths for processing. In the first path, the sequence is convolved with a forward convolutional kernel to capture historical context information, resulting in... In the second path, the sequence is flipped in the time dimension and then convolved with an inverse convolutional kernel to capture future contextual information, resulting in... ,Will and The inputs are summed and combined with the sequence representations of each modality through a residual connection to capture contextual information.

[0007] Preferably, in the feature learning module, the text modal features, video modal features, and audio modal features are projected onto a unified dimension using one-dimensional convolution, positional encoding embedding is performed using sine and cosine positional encoding, speaker information embedding is performed using a trainable matrix, and the modal data after dimension unification are added element-wise with the positional encoding embedding and speaker information embedding to obtain the sequence representation of each modality.

[0008] Preferably, in the multi-source consistent attention mechanism, the sequence representations of each modality are concatenated along the feature dimension, and the sequence representations of each modality are projected back to the original hidden dimension through a linear layer to obtain a shared representation that integrates information from all modalities, which serves as the context source for subsequent attention mechanisms. Using the context-aware representation of each modality as a query and the shared representation as a key and value, a cross-modal attention operation is performed to obtain a new representation of each modality after being weighted by multi-source information. The new representations of each modality are then added element-wise to obtain the fused features.

[0009] Preferably, the method further includes training the multimodal emotion recognition network using a collaborative learning strategy based on knowledge distillation. Specifically, this involves calculating the collaborative learning loss using the difference in prediction distribution between the unimodal branch and the multimodal fusion branch. The Logits vectors output by each single-modal branch and the multimodal fusion branch are transformed into probability distributions using the Softmax function with a temperature coefficient. Each unimodal branch is treated as a student branch. The KL divergence between the probability distribution of the student branch and the probability distributions of all other unimodal branches and multimodal fusion branches is calculated. The average of all KL divergences is then used as the collaborative learning loss for that student branch. The total loss is calculated by weighting the task loss, the cross-entropy loss of each single-modal branch, and the collaborative learning loss of the single-modal branch.

[0010] Preferably, text modal features, video modal features, and audio modal features are extracted based on the dialogue data, specifically: text modal features are extracted from the dialogue data using a pre-trained Roberta-Large model; audio modal features are extracted from the dialogue data using the OpenSmile tool; and video modal features are extracted from the dialogue data using a pre-trained DenseNet network.

[0011] This invention also provides a multimodal dialogue emotion recognition system, specifically comprising: The data module is used to acquire dialogue data and extract text modal features, video modal features, and audio modal features based on the dialogue data.

[0012] The emotion recognition module is used to input the text modal features, video modal features, and audio modal features into a pre-trained multimodal emotion recognition network to obtain the emotion labels of each utterance in the dialogue; the multimodal emotion recognition network includes a feature learning module, a local region bidirectional Mamba module, a convolutional feedforward network, and a multi-source consistency attention mechanism; In the pre-trained multimodal emotion recognition network, the feature learning module converts text modality features, video modality features, and audio modality features into sequence representations of each modality. The local region bidirectional Mamba module processes the sequence representations of each modality to capture contextual information. The captured contextual information is then input into a convolutional feedforward network for local information enhancement to obtain context-aware representations of each modality. The context-aware representations of each modality are fused based on a multi-source consistency attention mechanism to obtain fused features. The emotion tags of each utterance in the dialogue are obtained based on the fused features.

[0013] The present invention also provides a computer device, including a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps described in the multimodal dialogue emotion recognition method.

[0014] The present invention also provides a computer-readable storage medium storing a computer program that, when loaded by a processor, can execute the steps described in the multimodal dialogue emotion recognition method.

[0015] The multimodal dialogue emotion recognition method provided by this invention has the following beneficial effects: This invention constructs a multimodal emotion recognition BiSSM-CL network to process multimodal dialogue data. It combines a linearly complex bidirectional Mamba2 network with a convolutional network using a bidirectional state-space modeling module to dynamically balance global emotional tone and local transition features, and dynamically balance global emotional context and local transition cues. This achieves accurate capture of emotional dynamics while maintaining low computational cost. The multi-source consistent attention mechanism aims to share and align learning objectives across different modalities within the feature space and perform deep fusion, overcoming the limitations of shallow pairwise interactions. It exhibits superior performance in capturing key emotional shifts, deep multimodal fusion, and collaborative representation learning. Attached Figure Description

[0016] To more clearly illustrate the embodiments and design schemes of the present invention, the accompanying drawings required for this embodiment will be briefly described below. The drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0017] Figure 1 This is a diagram of the BiSSM-CL network framework according to an embodiment of the present invention.

[0018] Figure 2 This is a t-SNE visualization of the multimodal representations of different sentiment categories on the IEMOCAP dataset in this embodiment of the invention. Figure 2 (a) is the original feature; Figure 2 (b) is for removing collaborative learning distillation; Figure 2 (c) represents collaborative learning distillation.

[0019] Figure 3 This is a t-SNE visualization of the multimodal representations of different sentiment categories on the MELD dataset in this embodiment of the invention. Figure 3 (a) is the original feature; Figure 3 (b) is for removing collaborative learning distillation; Figure 3 (c) represents collaborative learning distillation.

[0020] Figure 4 This is a comparison of cases on the IEMOCAP dataset in the embodiments of the present invention.

[0021] Figure 5 To compare the confusion matrix of the AGF-IB model on the test set of the IEMOCAP dataset.

[0022] Figure 6 This is the confusion matrix of the BiSSM-CL of this invention on the IEMOCAP dataset test set. Detailed Implementation

[0023] To enable those skilled in the art to better understand and implement the technical solutions of the present invention, the present invention will be described in detail below with reference to the accompanying drawings and specific embodiments. The following embodiments are only used to more clearly illustrate the technical solutions of the present invention and should not be construed as limiting the scope of protection of the present invention.

[0024] Example This invention provides a multimodal dialogue emotion recognition method, specifically including the following steps: Step 1: Obtain dialogue data based on public datasets, and extract text modal features, video modal features, and audio modal features of each utterance in the dialogue data.

[0025] Step 2: Construct the enhanced multimodal emotion recognition network BiSSM-CL. The BiSSM-CL network architecture is as follows: Figure 1 As shown. After extracting discourse-level single-modal features, it mainly includes three core components: (a) Modality representation learning, based on Mamba and attention mechanisms, to effectively fuse local sequence information and global cross-modal information. (b) Sentiment classifier, which predicts the final sentiment label through a multimodal sentiment classifier. (c) Collaborative learning, in order to enhance the knowledge transfer and representation learning capabilities between modalities, a collaborative learning loss function based on knowledge distillation is designed, enabling each modality to learn from each other and obtain better feature representations.

[0026] (1) Feature extraction: Acoustic and visual features were extracted using MMGCN. Visual features were extracted using DenseNet pre-trained on the Facial Expression Recognition Plus corpus, and acoustic features were extracted using the OpenSmile toolkit with IS10 configuration. Text features were processed using the COSMIC method, that is, using the Roberta-Large model for pre-training and fine-tuning to extract text features.

[0027] (2) Multimodal feature learning involves processing the initial features of each modality after feature extraction to construct a unified and information-rich input representation. This process mainly includes three steps: dimension alignment, location information encoding, and speaker information embedding.

[0028] Dimension alignment: To ensure that the initial feature representations of each modality are in the same space, one-dimensional convolution is used to project them onto a uniform dimension for processing. (1); in, These represent the feature representations of text, audio, and visual data after dimension alignment, respectively. Represents the sequence length. Indicates the output feature dimension.

[0029] Location information encoding: To help the model capture dependencies between long-distance locations, sine and cosine location encoding embeddings are introduced: (2); (3); in, Indicates the position in the sequence. The dimension index representing the location encoding. , Indicates the dimension of the embedding.

[0030] Speaker information embedding: To capture speaker information from the sequence, speaker embedding was also designed to enhance the sequence. Specifically, for each speaker... , The speakers are mapped as vectors:

[0031] (4); in, It is the total number of speakers. It is a trainable embedding matrix. The speaker The one-hot vector. Therefore, the speaker embedding corresponding to the dialogue can be represented as .

[0032] The positional encoding embedding and speaker embedding are fused with the sequence processed by one-dimensional convolution by element-wise addition: (5); in, This represents the dimension-aligned single-modal feature sequence calculated by formula (1). yes The low-level position of the modality and the speaker's perceived speech sequence representation.

[0033] (3) Feature enhancement: Local Region Bi-Mamba (LRBM) modules are used to capture deep temporal dependencies and key local features for each independent modality (text, audio, vision). For each unimodal sequence representation... Its processing flow within the LRBM module is as follows:

[0034] Bidirectional Mamba2 context encoding. To comprehensively capture the dependencies between each utterance, a bidirectional Mamba2 sequence model based on a structured state space (S4) is employed. The S4 model represents the dynamic characteristics of the system through implicit state propagation across time steps.

[0035] Assume the input is The output is The basic form of the S4 model is shown below: (6); in, , as well as It is a learnable matrix. This represents the hidden state of the system, which changes with the time step. Change and update, specifically in the form of:

[0036] (7); Discrete parameters Discretization rules can be used to extract continuous parameters We obtain, for example, the zero-order preservation (ZOH) rule: , After discretization, the model can be computed in two ways: one is linear recursion for inference as shown in Equation (6), and the other is global convolution for training as shown in Equation (8), where It is a convolution kernel. For sequence length:

[0037] (8); Based on the above theory, the input sequence is first... Simultaneously, the data is fed into two parallel convolutional paths for processing: the sequence and the forward convolution kernel. Convolution is performed to capture historical context information. The sequence is flipped along the time dimension and then convolved with a deflected convolution kernel. Convolution is performed to capture future contextual information. Among these, and Obtained by equation (8), generated from independent SSM parameters.

[0038] Finally, output from both directions and The inputs are summed and combined with the original input through a residual connection to obtain a sequence representation that contains the global context. Finally, the output is connected to the original input via a residual connection. Combined, we obtain a context representation. : (9); in, For sequence length, This represents the index of the current utterance in the sequence. This represents element-wise multiplication.

[0039] Convolutional Feedforward Network (ConvFFN). To enhance the model's ability to perceive key local features, a convolutional feedforward network is used to process the sequence. This network consists of fully connected layers, Gelu activation functions, and depthwise separable convolutional (DwConv) modules, focusing on extracting local features without significantly increasing computational cost. The sequence is obtained from bidirectional Mamba2 context encoding. After level normalization, we obtain The normalized sequence is then fed into a feedforward convolutional network (ConvFFN) for processing.

[0040] (11); Finally, the output is connected to the original input via a residual connection. By combining these, we obtain the final context representation. : (12); (4) Feature fusion to obtain context-aware representations for each modality. , , Subsequently, to achieve deep and holistic information fusion among them, a multi-source consistent attention mechanism is introduced. Its core idea is to guide each modality to draw information from a joint semantic space composed of all modalities, thereby achieving holistic fusion that transcends "pairwise" interactions. The specific process is as follows:

[0041] Constructing a shared semantic space: representing the sequence of three modalities , , The feature dimensions are concatenated, and then projected back to the original hidden dimensions through a linear layer. This process generates a shared representation that integrates information from all modalities. It will serve as the context source (Key and Value) for subsequent attention mechanisms.

[0042] (13); in, and These are the learnable weight matrix and bias of the linear layer, respectively.

[0043] Multi-source consistent attention: Perform cross-modal attention operations on text, audio, and visual modalities respectively to obtain a new representation after weighting the multi-source information. , , : (14); Finally, the individual unimodal representations are summed element-wise to form the overall multimodal representation used for final classification. : (15); (5) Emotion classification: The fused multimodal discourse and single-modal feature representations are fed into the classification layer. The classification layer consists of a fully connected layer, a GELU activation function, and a Softmax function. It is responsible for projecting the high-dimensional feature vectors onto the emotion category space to obtain the emotion recognition results of the dialogue.

[0044] (16); Define the predicted label distribution and the true label distribution The cross-entropy loss between them is as follows: (17); in It is the first A statement Predicted labels, It's a real label.

[0045] (6) Collaborative learning: To effectively address modal differences and enable single-modal branches to learn from fused representations, a collaborative learning distillation strategy was designed. First, for each modal representation... Set up a separate classifier to obtain fine-grained prediction results. :

[0046] (18); These classifiers are trained using cross-entropy loss, which is defined as follows: (19); in Based on fine-grained representation For the A statement The predicted label, and It is the first The true label of a statement.

[0047] Then, the collaborative learning distillation strategy avoids the traditional rigid teacher-student model by constructing a dynamic learning framework in which each branch acts as a teacher to the other. Specifically, it first utilizes a temperature coefficient... The Softmax function combines the branches of each modality feature and the fused feature ( ,in, (Representing the fused multimodal branch) Output logits vector Transformed into a smoother probability distribution :

[0048] (20); in, It is the first The sentence is in the branch The corresponding emotional category The logits output, This represents the total number of sentiment categories. The core of collaborative learning lies in using Kullback-Leibler (KL) divergence to quantify the differences in prediction distributions between different branches. For the student branch... probability distribution and teacher branches probability distribution The KL divergence loss is defined as:

[0049] (twenty one); This loss function, by minimizing the difference between the two distributions, encourages the student branch's predictions to mimic the teacher branch. Within the framework, each unimodal branch learns from all other branches (including other unimodal branches and the final fused branch). Taking the text modality as an example, the collaborative learning loss for the text branch as the student is... Defined as the average of the sum of the KL divergences of its distribution with the prediction distributions of all teacher branches (audio, visual, fusion):

[0050] (twenty two); Similarly, the audio and visual branches serve as the collaborative learning loss for students. and They are respectively: (twenty three); (twenty four); Finally, the model's total loss function for: (25); in, , and It is a hyperparameter that balances different loss terms.

[0051] The proposed model was evaluated using the IEMOCAP and MELD datasets. Statistical data for these two datasets are shown in Table 1.

[0052] IEMOCAP: This dataset contains two-way dialogues between ten speakers, totaling 153 dialogues and 7,433 sentences. The dataset is divided into five sessions, with the first four sessions used for training and the last for testing. Each sentence is labeled with one of six emotions: happy, sad, neutral, angry, excited, and frustrated.

[0053] MELD: This is a multi-speaker dialogue dataset collected from the TV series Friends, containing 1,433 dialogues and 13,708 sentences. Each sentence is labeled with one of seven emotions: neutral, surprise, fear, sadness, joy, disgust, and anger.

[0054] Table 1 Dataset Partitioning To achieve more refined model training, a multi-optimizer strategy was employed. Specifically, a dedicated Adam optimizer was configured for each of the three independent modal processing paths: text, audio, and vision. A fourth independent optimizer was also established for shared features across modal fusion. Furthermore, to achieve adaptive learning rate adjustment, each optimizer was paired with its own learning rate scheduler. The initial learning rate for the IEMOCAP dataset was 1.0e-4, and the initial learning rate for the MELD dataset was 5.0e-6. This design allows each single-modal branch to dynamically adjust its learning rate independently based on its own loss function, while the shared components are optimized based on the overall training loss. The batch sizes for IEMOCAP and MELD were 16 and 8, respectively. The temperature coefficient τ for the IEMOCAP dataset was set to 10, and the temperature coefficient τ for MELD was set to 10.

[0055] For one-dimensional convolutional layers, on the IEMOCAP dataset, the number of input channels (i.e., their respective feature dimensions) for text, audio, and visual modalities were set to 1024, 1582, and 342, respectively. On the MELD dataset, these parameters were set to 1024, 300, and 342, respectively. For the core part of the model, the internal hidden dimension of the bidirectional Mamba module used for intramodal encoding was set to 512; while in the intermodal fusion stage, the number of heads for the multi-head attention mechanism was set to 8. Furthermore, on both datasets, the number of output channels and the kernel size for all three modalities were set to 1024 and 1, respectively. To prevent overfitting, the L2 weight decay was set to 1.0e−5, and a dropout of 0.5 was used on IEMOCAP and a dropout of 0.4 was used on MELD.

[0056] The proposed BISSM-CL model was compared with existing baseline methods on the IEMOCAP and MELD datasets. These baseline methods include: BC-LSTM: which utilizes a Bi-LSTM model to model contextual dependencies from past and future utterances; DialogueRNN: which employs a GRU network to model dialogues, capturing global context, local context, and speaker information to address multimodal sentiment analysis; MMGCN: which combines graph neural networks with multimodal fusion to encode inherent structural dependencies in speaker-utterance interactions; and MM-DFN: which designs a graph-based dynamic fusion module that aggregates intramodal and intermodal context using graph convolutions and captures sequence patterns using gating mechanisms, thereby reducing... Redundancy and enhanced complementarity; GA2MIF: It uses a graph neural architecture for modal feature modeling and combines multi-head attention layers to establish interaction relationships between different modalities; M3NET: It proposes a GNN-based model that enhances multimodal and contextual modeling by emphasizing multi-frequency signals to explore diverse relationships and capture the different importance of emotional differences and commonalities; CFN-ESA: It proposes a cross-modal fusion network that aggregates complementary multimodal cues using a text-centric interaction mechanism and uses an emotion transfer module to capture emotion transitions, thereby improving recognition accuracy in emotion transfer scenarios; SACCMA: It proposes... A speaker-aware cognitive network employs a cross-modal attention mechanism to fuse multimodal features and explicitly integrates speaker information through cognitive modules, thereby enhancing the perception of emotional context in dialogues; AGF-IB: It proposes a dual-flow graph learning architecture, establishing parallel flows to independently model intra-modal and inter-modal information, thus mitigating data heterogeneity and fusion conflicts; BiGMF: This framework implements adversarial alignment to eliminate modal differences and achieves precise feature boundary refinement through graph fusion with information bottleneck regularization; HiMul-LGG: It proposes an attention-based structure to generate unique contexts for text, audio, and visual modalities. Textual features; ConxGNN: This model integrates multi-scale heterogeneous graphs and hypergraphs to model complex contexts and multivariate dependencies, and uses a loss reweighting scheme to solve the class imbalance problem and improve recognition performance; Frame-SCN: It applies a wavelet-based stochastically configured network to learn a unified semantic embedding by extracting multi-frequency components, and uses a modality-aware extraction module to capture general and sensitive information to mitigate noise; LECM: It proposes a multi-task learning framework that utilizes emotional causes through an auxiliary causal implication task, and uses a cause-oriented tendency network to fuse rich semantic features with cause vectors to enhance real-time recognition capabilities.

[0057] The results for all models on the IEMOCAP dataset are shown in Table 2. Evaluation metrics include the F1 score (%) for each emotion and the average Acc and w-F1 (%) scores. Bold text represents the best results.

[0058] Table 2 Results for all models on the IEMOCAP dataset The results for all models on the MELD dataset are shown in Table 3. Evaluation metrics include the F1 (%) score for each emotion and the average Acc and w-F1 (%) scores. Bold text indicates the best results.

[0059] Table 3 Results for all models on the MELD dataset Tables 2 and 3 show the performance of BISSM-CL on the IEMOCAP and MELD datasets, respectively. The F1 score, mean accuracy, and weighted average F1 score for each emotion are reported in both tables. Based on results reported in existing literature, experimental results were tested on the IEMOCAP dataset for happy, sad, neutral, angry, excited, and frustrated emotions, and on the MELD dataset for neutral, surprised, sad, happy, and angry emotions.

[0060] On the IEMOCAP dataset, the proposed BISSM-CL model achieved outstanding performance, with an overall accuracy of 74.00% and a weighted F1 score of 74.07%, surpassing all listed baseline methods. The overall trend indicates that graph-based and fusion-based methods generally outperform earlier temporal modeling methods. Despite the strong competitiveness of current baseline models, BiSSM-CL achieves state-of-the-art performance thanks to its bidirectional Mamba module's global-local context modeling capabilities and the robustness of its collaborative learning mechanism. Compared to these state-of-the-art baselines, BiSSM-CL demonstrates comprehensive performance improvements across key sentiment categories, further confirming its superiority in multimodal emotion recognition tasks.

[0061] On the MELD dataset, BiSSM-CL continues to lead in performance across most metrics. While baseline models such as CFN-ESA and AGF-IB remain competitive, particularly in categories with large datasets (e.g., "neutral" and "joyful"), BiSSM-CL maintains a significant advantage in overall classification accuracy and weighted F1 score. BiSSM-CL consistently leads in fine-grained sentiment discrimination, especially in negative sentiment recognition, demonstrating its robustness and effectiveness in handling complex dialogue dynamics.

[0062] To systematically evaluate the independent contributions and interactions of the core components in the proposed BiSSM-CL framework, a series of ablation experiments were conducted on the IEMOCAP and MELD datasets, and the results are shown in Table 4.

[0063] Table 4 shows the results of ablation experiments performed on the IEMOCAP and MELD datasets.

[0064] When the collaborative learning loss is removed ( and When the F1 score of the two datasets dropped significantly to 71.08% (IEMOCAP) and 65.21% (MELD), this highlights the effectiveness of the co-learning distillation strategy.

[0065] When the MCAF module was removed, the F1 score dropped sharply to 66.87% (IEMOCAP) and 65.41% (MELD). This result highlights the extreme importance of multi-source consistent attention mechanisms for deep multimodal fusion.

[0066] When the LRBM module was replaced with a traditional Bi-LSTM, the F1 score dropped to 71.15% and 65.17%, which clearly demonstrates that Mamba outperforms Bi-LSTM in capturing long-range dependencies.

[0067] Furthermore, the sequential rationality of the model architecture was tested by reversing the processing order of the LRBM and MCAF modules. After the order was reversed, the F1 score on the IEMOCAP dataset decreased by 2.52%, and the MELD dataset by 0.50%, confirming the superiority of the Bi-LSTM design, which sequentially performs deep intramodal encoding via LRBM and global cross-modal fusion via MCAF. This workflow effectively aligns with the cognitive process of refining local modality-specific features before integrating global multimodal information. The model used in the ablation experiments for the loss function incorporates two key losses: an auxiliary cross-entropy loss for single-modal supervision (…). ) and collaborative learning loss for collaborative knowledge transfer ( To verify the importance of these losses, a leave-one-out strategy was used to evaluate the contribution of each loss component. Table 4 shows that... and They are complementary, and the performance is highest when the model is trained using the full objective function. The results show that... This ensures the discriminative ability of each independent mode, and through Bidirectional transfer of soft-label knowledge further improves model performance by promoting the co-evolution of features.

[0068] The impact of the MCAF module was evaluated by comparing it with traditional fusion techniques: (1) element-wise addition, which involves summing feature vectors; and (2) concatenation, which merges features along the channel dimension. The addition strategy treats all representations equally, while the concatenation strategy relies on subsequent layers to implicitly filter important information. However, both rely on shallow interactions. To ensure a fair comparison, the MCAF module in the model was replaced with element-wise addition and concatenation operations, respectively, to implement the addition and concatenation strategies. Experimental results show that the MCAF mechanism significantly outperforms these baseline schemes, demonstrating that deep, holistic cross-modal alignment in a shared space is crucial for capturing complex emotional semantics.

[0069] Based on the results of the ablation experiments, it can be concluded that the three core components—the LRBM module, the MCAF mechanism, and the collaborative learning strategy—are complementary and indispensable for the superior performance of the model. This specific combination and execution order together constitute the optimal solution. To explore the impact of different modalities on model performance, a systematic ablation study was conducted by removing one or two modalities at a time. As observed in Table 5, in the single-modal setting, the text modality clearly dominates, achieving an F1 score of 67.53% on the IEMOCAP dataset, far superior to the audio and visual modalities. Furthermore, all bimodal combinations outperformed their respective single-modal results, confirming the effectiveness of modality fusion. Notably, combinations including the text modality (e.g., Text+Audio achieving an F1 score of 72.09% on the IEMOCAP dataset) significantly outperformed combinations that only fused the audio and visual modalities, further emphasizing the crucial role of text features. Ultimately, using all three modalities yielded the best overall performance (an F1 score of 74.07% on the IEMOCAP dataset), providing strong evidence that effective emotion recognition requires the collaborative use of linguistic, acoustic, and visual information.

[0070] Table 5 shows the results of modal ablation experiments on the IEMOCAP and MELD datasets.

[0071] Figure 2 and Figure 3 This section presents t-SNE visualizations of multimodal representations for different emotion categories on the IEMOCAP and MELD datasets. Compared to the original multimodal representations (…),… Figure 2 (a) Compared to, even without collaborative learning distillation ( Figure 2 In case (b), the feature representations learned by the proposed model also exhibit stronger clustering ability. However, without co-learning distillation, the model still faces difficulties in distinguishing emotionally similar categories (such as "happy" and "excited", "angry" and "frustrated" on IEMOCAP).

[0072] By comparison Figure 3 (b) and Figure 3 (c) It is clearly observed that after introducing co-learning distillation, the model produces better class separation performance, and the mixing degree of different emotion representations is significantly reduced. This strongly demonstrates that co-learning distillation training can effectively learn more discriminative multimodal representations.

[0073] To demonstrate the effectiveness of the proposed model, a case study is presented. For example... Figure 4 As shown, this is a dialogue scene from the IEMOCAP dataset. After a speaker utters a series of utterances with genuine "neutral" emotions, most baseline models (such as M3NET) tend to predict that the emotion of subsequent utterances will still be "neutral." This is mainly because these models overemphasize historical context modeling, thus creating emotional inertia and ignoring crucial local cross-modal information. In contrast, the proposed BiSSM-CL framework effectively balances context modeling with its own information capture through its LRBM module and MCAF mechanism. For example, when a male speaker utters utterances genuinely labeled as "frustrated," the baseline model incorrectly predicts "neutral," while BiSSM-CL successfully identifies the correct "frustrated" emotion, demonstrating BiSSM-CL's superior ability to mitigate historical emotional inertia.

[0074] To evaluate the effectiveness of BiSSM-CL, such as Figure 5 and Figure 6 The confusion matrices of the baseline models AGF-IB and BiSSM-CL on the IEMOCAP dataset are shown. To ensure fairness in the comparison, the original sample counts reported in the original AGF-IB were converted to percentages. These comparisons reveal several observations: (1) While both models show a clear diagonal (indicating basic effectiveness), BiSSM-CL achieves significant performance improvements in specific categories (particularly “sadness,” “frustration,” and “excitement”); (2) Notably, BiSSM-CL demonstrates a stronger ability to distinguish subtle negative emotions. For example, the confusion rate of misclassifying “sadness” as “neutral” is significantly reduced, and the misclassification of “frustration” as “anger” is also significantly reduced, indicating that the model can capture subtle emotional features more accurately.

[0075] This invention also provides a multimodal dialogue emotion recognition system, comprising: The data module is used to acquire dialogue data and extract text modality features, video modality features, and audio modality features based on the dialogue data.

[0076] The emotion recognition module is used to input text modal features, video modal features, and audio modal features into a pre-trained multimodal emotion recognition network to obtain the emotion labels of each utterance in the dialogue. The multimodal emotion recognition network includes a feature learning module, a local region bidirectional Mamba module, a convolutional feedforward network, and a multi-source consistency attention mechanism. In the pre-trained multimodal emotion recognition network, the feature learning module converts text modality features, video modality features, and audio modality features into sequence representations of each modality. The local region bidirectional Mamba module processes the sequence representations of each modality separately to capture contextual information. The captured contextual information is then input into a convolutional feedforward network for local information enhancement to obtain context-aware representations of each modality. The context-aware representations of each modality are fused based on a multi-source consistency attention mechanism to obtain fused features. The emotion tags of each utterance in the dialogue are obtained based on the fused features.

[0077] The modules in the aforementioned multimodal dialogue emotion recognition system can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device as software, so that the processor can call and execute the corresponding operations of each module.

[0078] The present invention also provides a computer device, including a memory, a processor, and a computer program stored in the memory. The processor executes the computer program to implement the steps in an embodiment of a multimodal dialogue emotion recognition method. Specific implementation methods can be found in the method embodiments, and will not be repeated here.

[0079] Furthermore, the present invention also provides a non-transitory computer-readable storage medium containing instructions on which a computer program is stored. For example, a memory containing instructions that can be executed by a processor of a computer device to perform the above-described method. For example, the non-transitory computer-readable storage medium may be a ROM, random access memory (RAM), CD-ROM, magnetic tape, floppy disk, and optical data storage device, etc. When the computer program is executed by the processor, it can implement the steps in an embodiment of a multimodal dialogue emotion recognition method. Specific implementation methods can be found in the method embodiments, which will not be repeated here.

[0080] Those skilled in the art will understand that embodiments of the present invention can provide methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0081] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, as well as combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0082] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0083] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0084] It should be noted that the specific embodiments described above enable those skilled in the art to more fully understand the present invention, but do not limit the present invention in any way. Therefore, although the present invention has been described in detail in this specification and embodiments, those skilled in the art should understand that modifications or equivalent substitutions can still be made to the present invention; and all technical solutions and improvements that do not depart from the spirit and scope of the present invention are covered within the protection scope of the present invention patent. No reference numerals in the claims should be construed as limiting the scope of the claims. Any simple variations or equivalent substitutions of technical solutions that can be readily obtained by those skilled in the art within the scope of the technology disclosed in the present invention are within the protection scope of the present invention.

Claims

1. A multimodal dialogue emotion recognition method, characterized in that, Includes the following steps: Acquire dialogue data, and extract text modal features, video modal features, and audio modal features based on the dialogue data; The text modal features, video modal features, and audio modal features are input into a pre-trained multimodal emotion recognition network to obtain the emotion labels for each utterance in the dialogue. The multimodal emotion recognition network includes a feature learning module, a local region bidirectional Mamba module, a convolutional feedforward network, and a multi-source consistency attention mechanism. In the pre-trained multimodal emotion recognition network, the feature learning module is used to convert text modal features, video modal features and audio modal features into sequence representations of each modality. The local region bidirectional Mamba module is used to process the sequence representations of each modality, capture contextual information, and input the captured contextual information into a convolutional feedforward network for local information enhancement to obtain context-aware representations of each modality. The context-aware representations of the various modalities are fused based on a multi-source consistent attention mechanism to obtain fused features; The emotional labels of each utterance in the dialogue are obtained based on the fusion features.

2. The multimodal dialogue emotion recognition method according to claim 1, characterized in that, In the local bidirectional Mamba module, the sequence representation of a single modality is simultaneously fed into two parallel convolutional paths for processing. In the first path, the sequence is convolved with a forward convolutional kernel to capture historical context information, resulting in... In the second path, the sequence is flipped in the time dimension and then convolved with an inverse convolutional kernel to capture future contextual information, resulting in... ,Will and The inputs are summed and combined with the sequence representations of each modality through a residual connection to capture contextual information.

3. The multimodal dialogue emotion recognition method according to claim 1, characterized in that, In the feature learning module, one-dimensional convolution is used to project the text modality features, video modality features and audio modality features to a unified dimension. Sine and cosine positional encoding is used for positional encoding embedding, and a trainable matrix is ​​used for speaker information embedding. The unified modality data is then added element-wise with the positional encoding embedding and speaker information embedding to obtain the sequence representation of each modality.

4. The multimodal dialogue emotion recognition method according to claim 1, characterized in that, In the multi-source consistent attention mechanism, the sequence representations of each modality are concatenated along the feature dimension. A linear layer projects the sequence representations of each modality back to the original hidden dimension, resulting in a shared representation that integrates information from all modalities. This shared representation serves as the context source for subsequent attention mechanisms. Using the context-aware representation of each modality as a query and the shared representation as a key and value, a cross-modal attention operation is performed to obtain a new representation of each modality after weighting by multi-source information. The new representations of each modality are then summed element-wise to obtain the fused features.

5. The multimodal dialogue emotion recognition method according to claim 1, characterized in that, It also includes a knowledge distillation-based collaborative learning strategy for training the multimodal emotion recognition network. Specifically, it calculates the collaborative learning loss by utilizing the difference in prediction distribution between the unimodal branch and the multimodal fusion branch. The Logits vectors output by each single-modal branch and the multimodal fusion branch are transformed into probability distributions using the Softmax function with a temperature coefficient. Each unimodal branch is treated as a student branch. The KL divergence between the probability distribution of the student branch and the probability distributions of all other unimodal branches and multimodal fusion branches is calculated. The average of all KL divergences is then used as the collaborative learning loss for that student branch. The total loss is calculated by weighting the task loss, the cross-entropy loss of each single-modal branch, and the collaborative learning loss of the single-modal branch.

6. The multimodal dialogue emotion recognition method according to claim 1, characterized in that, Based on the dialogue data, text modal features, video modal features, and audio modal features are extracted. Specifically, a pre-trained Roberta-Large model is used to extract text modal features from the dialogue data. Audio modal features were extracted from the dialogue data using the OpenSmile tool; video modal features were extracted from the dialogue data using a pre-trained DenseNet network.

7. A multimodal dialogue emotion recognition system, characterized in that, include: The data module is used to acquire dialogue data and extract text modal features, video modal features, and audio modal features based on the dialogue data. The emotion recognition module is used to input the text modal features, video modal features, and audio modal features into a pre-trained multimodal emotion recognition network to obtain the emotion labels of each utterance in the dialogue; the multimodal emotion recognition network includes a feature learning module, a local region bidirectional Mamba module, a convolutional feedforward network, and a multi-source consistency attention mechanism; In the pre-trained multimodal emotion recognition network, the feature learning module is used to convert text modal features, video modal features and audio modal features into sequence representations of each modality. The local region bidirectional Mamba module is used to process the sequence representations of each modality, capture contextual information, and input the captured contextual information into a convolutional feedforward network for local information enhancement to obtain context-aware representations of each modality. The context-aware representations of the various modalities are fused based on a multi-source consistent attention mechanism to obtain fused features; The emotional labels of each utterance in the dialogue are obtained based on the fusion features.

8. A computer device, comprising a memory, a processor, and a computer program stored in the memory, characterized in that, The processor executes the computer program to implement the steps of the method according to any one of claims 1 to 6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is loaded by the processor, it is able to perform the steps of the method according to any one of claims 1 to 6.

Images