A multimodal dialogue sentiment recognition method and system

Abstract

This invention provides a multimodal dialogue emotion recognition method and system, belonging to the field of multimodal emotion recognition technology. The method includes: extracting initial features from the dialogue history; generating dynamic modality weights based on the contribution of each modality in the current speaking round, obtaining fusion features corresponding to each speaking round; inputting the sequence of fusion features from multiple speaking rounds into a state space model to obtain context-enhanced representations; reconstructing emotion recognition into a structured generation task; applying emotion consistency constraints and using a composite objective function to perform end-to-end joint optimization of the parameters involved in feature extraction, temporal modeling, and hierarchical generation; and, for real-time input multimodal dialogue streams, sequentially outputting the emotion polarity and fine-grained emotion labels corresponding to the current speaking round through state updates of the state space model and the structured generation task. This invention can alleviate problems such as cross-channel noise, semantic inconsistency, and inefficient long-range dependency modeling.

Description

Technical Field

[0001] This invention relates to the field of multimodal emotion recognition technology, and in particular to a multimodal dialogue emotion recognition method and system. Background Technology

[0002] With the rapid growth of social media, online platforms, and short video applications, massive amounts of multimodal data are generated daily, containing rich clues about users' emotional states. Accurately understanding these emotional signals is becoming increasingly important for a wide range of applications, including personalized recommendations, human-computer interaction, and intelligent assistant systems, where recognizing user emotions can directly improve service quality and user experience. Multimodal Emotion Recognition in Conversations (MERC) addresses this problem by jointly modeling the textual, acoustic, and visual channels within the conversational context to identify the emotion of each utterance. By leveraging complementary information across channels, MERC captures the semantic content and emotional nuances of the conversation, enabling more accurate and context-aware emotion prediction.

[0003] A key challenge of MERC lies in multi-channel fusion, where the efficient allocation of channel importance is crucial for accurate sentiment recognition. Different patterns often contribute differently to sentiment understanding, and inappropriate weights can introduce noise or obscure key cues. Some studies utilize attention mechanisms to adaptively learn channel weights during the fusion process, enabling models to capture cross-channel interactions and dynamically emphasize channels that provide more information. Other studies employ contrastive learning to enforce cross-channel alignment, encouraging consistency in representations from different channels while preserving complementary information. While effective, these methods are largely data-driven and lack explicit guidance on channel importance, making it difficult to consistently control channel contributions. Furthermore, attention-based fusion mechanisms can incur considerable computational overhead when modeling long-term dependencies in multi-channel sequences. This has spurred a need for fusion paradigms that incorporate channel priors and produce representations more suitable for efficient sequence modeling.

[0004] Beyond multimodal fusion, modeling the relationship between emotions and sentiments remains a key but unexplored challenge. Emotions capture coarse-grained extremes, while sentiments reflect fine-grained categories, and the two are inherently correlated (e.g., happiness is often associated with positive emotions). Most existing work treats emotions and sentiments as independent tasks, optimizing them separately. Some research has attempted to jointly model them to leverage their shared emotional information, either by designing separate classifiers within a multi-task learning framework or by employing a sequential generation paradigm for hierarchical prediction. However, these approaches rarely maintain consistency between emotions and sentiments, which can lead to semantically conflicting predictions. Some models can correctly predict negative emotions but misclassify them as joy, resulting in inconsistent predictions. Since emotions are generally more reliable as coarse-grained emotional signals, strengthening emotion-sentiment consistency can mitigate this conflict and improve overall prediction accuracy.

[0005] Therefore, there is an urgent need for an emotion generation framework based on prior guidance and consistency perception. By combining multi-channel fusion of prior guidance with sequence modeling based on State Space Model (SSM), MERC is described as a structured generative problem, achieving robust and efficient long-range dependency learning. Summary of the Invention

[0006] In view of this, the purpose of this invention is to provide a multimodal dialogue emotion recognition method and system to alleviate problems such as cross-channel noise, semantic inconsistency and inefficient remote dependency modeling.

[0007] In a first aspect, embodiments of the present invention provide a multimodal dialogue emotion recognition method, comprising the following steps: Initial features of textual, visual, and audio modalities are extracted from the dialogue history, which contains multiple speaking turns arranged in chronological order. Dynamic modality weights are generated based on the contribution of each modality in the current speaking round. The features of each modality are then weighted and aggregated using these weights to obtain the fusion features corresponding to each speaking round. The sequence of features from multiple speaking rounds is input into the state space model to obtain a context-enhanced representation. Emotion recognition is reconstructed into a structured generation task, which includes: a first generation stage, generating emotion polarity; and a second generation stage, generating fine-grained emotion labels based on emotion polarity. We impose sentiment consistency constraints and use a composite objective function to perform end-to-end joint optimization of the parameters involved in feature extraction, temporal modeling, and hierarchy generation. For real-time input multimodal dialogue streams, the state space model is used to update the state and generate a structured data, which sequentially outputs the emotional polarity and fine-grained sentiment labels corresponding to the current speaking turn.

[0008] Optionally, dynamic modality weights are generated based on the contribution of each modality in the current speaking round, including: The modality weights are calculated based on the initial characteristics of the modality and the preset confidence parameters.

[0009] Optionally, the weighted aggregation employs a cross-attention mechanism, where the features of the text modality are used as the query term, the features of the audio modality are concatenated with the features of the visual modality, and the joint features obtained after concatenation are used as the key and value terms.

[0010] Optionally, the state space model adopts a state space model based on a selective scanning mechanism. The discretized state evolution equation of the state space model realizes long-distance dependency modeling under linear complexity. Furthermore, the same state space model is used as a shared decoder to perform structured generation tasks.

[0011] Optionally, an emotional consistency constraint may be imposed, including: during training, dividing the emotional prediction scores output by the model into a consistent subset and a conflict subset based on the real emotional labels, and applying a marginal penalty to the scores in the conflict subset.

[0012] Optionally, the marginal penalty is: using the average score of the consistent subset as a reference, a preset margin value is applied as a penalty to the difference between each score in the conflicting subset and the average score.

[0013] Optionally, the composite objective function is a weighted sum of the negative log-likelihood loss and the sentiment consistency constraint loss.

[0014] Optionally, output the emotional polarity and fine-grained sentiment tags corresponding to the current speaking round, including: Decoding is performed using a constrained bundle search. At each decoding step, the conditional probability of the candidate word list is calculated, and the multiple paths with the highest probabilities are retained. The sentiment tag sequence with the highest global log-likelihood score is output.

[0015] Optionally, output the emotional polarity and fine-grained sentiment tags corresponding to the current speaking round, including: The predicted scores of emotional polarity and fine-grained sentiment labels are combined to form all valid sentiment pairs. The joint score is calculated according to the preset mapping rules, and the sentiment pair with the highest score is selected as the output.

[0016] Secondly, embodiments of the present invention also provide a multimodal dialogue emotion recognition system, comprising: The extraction unit is used to extract the initial features of the text modality, visual modality, and audio modality in the dialogue history, which contains multiple speaking turns arranged in chronological order. The weighting unit is used to generate dynamic modality weights based on the contribution of each modality in the current speaking round. The weights are then used to aggregate the features of each modality to obtain the fusion features corresponding to each speaking round. The representation acquisition unit inputs a sequence of state space models consisting of employees and the fused features corresponding to multiple speaking rounds to obtain context-enhanced representations. The task generation unit is used to reconstruct emotion recognition into a structured generation task, which includes: a first generation stage, generating emotion polarity; and a second generation stage, generating fine-grained emotion labels based on emotion polarity. The optimization unit is used to apply sentiment consistency constraints and to perform end-to-end joint optimization of the parameters involved in feature extraction, temporal modeling and hierarchy generation using a composite objective function. The output unit is used to output the emotional polarity and fine-grained sentiment labels corresponding to the current speaking turn in a real-time input multimodal dialogue stream through state space model state update and structured generation tasks.

[0017] The embodiments of the present invention bring the following beneficial effects: The multimodal dialogue situation recognition method and system provided in this application transforms the multi-channel emotion recognition problem into a structured emotion generation problem; it obtains a more reliable fusion representation through multi-channel fusion guided by channel importance priors; it achieves efficient and robust long-term dependency modeling by combining sequence modeling based on state space models; it adopts a hierarchical two-stage generation strategy for emotion decoding to capture the inherent hierarchical dependencies between different emotion levels; and it strengthens the consistency between emotions by combining boundary-based training constraints and restricted inference through joint probability reordering. While maintaining competitive performance, this invention significantly improves the consistency of emotion prediction through effective inference, effectively alleviating problems such as cross-channel noise, semantic inconsistency, and inefficient long-term dependency modeling. It can be widely applied in fields such as multimodal emotion recognition and intelligent human-computer interaction.

[0018] Other features and advantages of the invention will be set forth in the description which follows, and will be apparent in part from the description, or may be learned by practicing the invention. The objects and other advantages of the invention are realized and obtained in accordance with the structures particularly pointed out in the description, claims and drawings.

[0019] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, preferred embodiments are described below in detail with reference to the accompanying drawings. Attached Figure Description

[0020] To more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0021] Figure 1 A flowchart of the multimodal dialogue emotion recognition method provided in the embodiments of this application; Figure 2 This is a structural block diagram of a multimodal dialogue emotion recognition system provided in an embodiment of this application. Detailed Implementation

[0022] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0023] A key challenge of MERC lies in multi-channel fusion, where the efficient allocation of channel importance is crucial for accurate sentiment recognition. Different patterns often contribute differently to sentiment understanding, and inappropriate weights can introduce noise or obscure key cues. Some studies utilize attention mechanisms to adaptively learn channel weights during the fusion process, enabling models to capture cross-channel interactions and dynamically emphasize channels that provide more information. Other studies employ contrastive learning to enforce cross-channel alignment, encouraging consistency in representations from different channels while preserving complementary information. While effective, these methods are largely data-driven and lack explicit guidance on channel importance, making it difficult to consistently control channel contributions. Furthermore, attention-based fusion mechanisms can incur considerable computational overhead when modeling long-term dependencies in multi-channel sequences. This has spurred a need for fusion paradigms that incorporate channel priors and produce representations more suitable for efficient sequence modeling.

[0024] Beyond multimodal fusion, modeling the relationship between emotions and sentiments remains a key but unexplored challenge. Emotions capture coarse-grained extremes, while sentiments reflect fine-grained categories, and the two are inherently correlated (e.g., happiness is often associated with positive emotions). Most existing work treats emotions and sentiments as independent tasks, optimizing them separately. Some research has attempted to jointly model them to leverage their shared emotional information, either by designing separate classifiers within a multi-task learning framework or by employing a sequential generation paradigm for hierarchical prediction. However, these approaches rarely maintain consistency between emotions and sentiments, which can lead to semantically conflicting predictions. Some models can correctly predict negative emotions but misclassify them as joy, resulting in inconsistent predictions. Since emotions are generally more reliable as coarse-grained emotional signals, strengthening emotion-sentiment consistency can mitigate this conflict and improve overall prediction accuracy.

[0025] Therefore, there is an urgent need for an emotion generation framework based on prior guidance and consistency perception. By combining prior-guided multi-channel fusion with SSM-based sequence modeling, MERC is described as a structured generative problem, achieving robust and efficient remote dependency learning.

[0026] Based on this, embodiments of the present invention provide a multimodal dialogue emotion recognition method and system to alleviate problems such as cross-channel noise, semantic inconsistency, and inefficient remote dependency modeling.

[0027] To facilitate understanding of this embodiment, a multimodal dialogue emotion recognition method disclosed in this embodiment of the invention will first be described in detail.

[0028] Figure 1 A flowchart illustrating the multimodal dialogue emotion recognition method provided in this application embodiment. Figure 1 As shown, the method may include: S102: Extract initial features from the textual, visual, and audio modalities in the dialogue history.

[0029] In this embodiment, multimodal dialogue data requiring emotion recognition is first acquired. The dialogue history can include dialogues involving multiple modalities such as text, speech, and vision, as well as contextual data. Each dialogue sequence contains dialogues from multiple users, and each dialogue contains information from text, speech, and visual modalities. The dialogue history contains multiple speaking turns arranged chronologically.

[0030] For each speaking round, initial feature vectors are extracted from the text modality, visual modality, and audio modality, as follows: (1) Text modality feature extraction The text content of each speaking turn is obtained from the dialogue history (e.g., text transcription obtained through automatic speech recognition or manually annotated text). The text is then input into a pre-trained text encoder to obtain a deep semantic feature representation.

[0031] In a preferred embodiment, the text encoder can be a pre-trained Mamba encoder. Specifically, after segmenting the text sequence into words, it is input into the pre-trained Mamba language model. The mean pooling or the output vector corresponding to the first token of the last hidden state of the model is taken as the text feature vector for that speaking round, denoted as... .

[0032] (2) Visual modality feature extraction The video clips (usually videos of the speaker's facial area) corresponding to each speaking turn are obtained from the dialogue history. A visual feature extraction network is used to extract visual motion features such as action units and facial expression changes for each frame.

[0033] In a preferred embodiment, densely connected convolutional networks (DenseNet) or three-dimensional convolutional neural networks (3D-CNN) are used to extract visual features. The specific steps are as follows: Uniform sampling is performed on the video segments of each speaking round to obtain a fixed number of video frames (e.g., 16 or 32 frames); each frame image is scaled to a uniform size (e.g., 224×224 pixels) and input into the DenseNet-121 network to extract frame-level features; the frame-level features are then subjected to temporal averaging or 3D-CNN is used to simultaneously capture spatiotemporal information to obtain the visual feature vector of that speaking round.

[0034] To align the visual features with the text and audio features in subsequent fusion, the visual features are mapped to a unified feature dimension through a learnable linear projection layer. The projected features are denoted as... .

[0035] (3) Audio modal feature extraction Obtain the raw audio waveforms for each speaking turn from the dialogue history. Use acoustic feature extraction tools to extract low-level acoustic features.

[0036] In a preferred embodiment, the Open Source Speech and Music Interpretation by Large Space Extraction (OpenSmile) toolkit is used to extract a standard acoustic feature set, obtaining the original audio feature vector for each speech round. This original feature vector is typically high-dimensional (e.g., 88-dimensional, 6373-dimensional, etc.). To reduce the dimensionality and facilitate fusion, it is also mapped to a uniform feature dimension through a learnable linear projection layer. The projected audio features are recorded... .

[0037] It should be noted that the parameters of all encoders and projection layers used in the above feature extraction process are optimized end-to-end in the subsequent joint training phase, rather than remaining fixed. This design allows the feature representation to adaptively adjust to the emotion recognition task, thereby improving recognition accuracy.

[0038] S104: Generate dynamic modality weights based on the contribution of each modality in the current speaking round, and use the weights to aggregate the features of each modality to obtain the fusion features corresponding to each speaking round.

[0039] After completing the initial feature extraction of each modality in step S102, this step first dynamically generates modality importance weights based on the instantaneous contribution of the text modality, visual modality, and audio modality in each speaking round; then, the generated weights are used to perform weighted aggregation of the three modal features, and at the same time, the cross-attention mechanism is combined to achieve deep fusion dominated by text, and finally the fusion feature vector corresponding to each speaking round is output.

[0040] To suppress cross-modal noise and highlight key emotional cues, this embodiment introduces a prior knowledge mechanism to dynamically calculate the contribution of each modality in the current dialogue context and generate modality importance weights. Unlike traditional pure data-driven fusion (such as simple concatenation or linear weighting), this embodiment adopts a prior-guided semi-adaptive fusion mechanism, combining manual priors with learnable parameters, so that the weights conform to domain common sense and can be adaptively adjusted according to the data.

[0041] Based on domain knowledge in dialogue sentiment recognition, the text modality typically carries the richest and most direct emotional information, while audio and visual modalities are significantly affected by environmental noise and individual differences, resulting in relatively lower reliability. Therefore, a prior bias is set for each modality. For modal importance prior bias, for text modality Initialize to positive values, for audio modalities and visual modality Initialize to 0.

[0042] Modal importance prior weights The calculation formula is:

[0043] in, For learnable transformation matrices, This is the original representation of the corresponding mode.

[0044] For the text modality, the prior biases are initialized to positive values ​​to reflect the dominant role of the text modality in multimodal fusion; for the audio modality, the prior biases are initialized to 0; and for the visual modality, the prior biases are initialized to 0. All of these prior biases are learnable parameters that are automatically updated during model training via backpropagation. This design achieves a semi-adaptive mechanism of "prior guidance and data fine-tuning."

[0045] Furthermore, Modal importance scores are obtained by average pooling. The dynamic modal weights are then obtained by normalizing the exponential function softmax.

[0046] Clearly, the dynamic weights of the text modality, visual modality, and audio modality are all 1. These dynamic weights reflect the relative contribution of each modality to emotion recognition in the current speaking round.

[0047] After obtaining the dynamic modal weights, this step further integrates the three modal features. In order to fully utilize the semantic dominance of the text modality and finely characterize the interaction between modalities, this embodiment adopts a text-dominant cross-attention mechanism for early fusion, and then combines it with modal weights for weighted aggregation.

[0048] Specifically, a cross-attention mechanism is employed for early fusion, where text modality features serve as the query, and the concatenation of audio and visual modality features serves as the key and value. This approach demonstrates the dominant role of text modality in multimodal fusion: the text actively queries audio and visual information to extract acoustic and visual cues most relevant to the current text's sentiment.

[0049] The specific formula is as follows:

[0050]

[0051] in, This is the output feature vector after text and visual modalities are fused through cross-attention, where CS-Attention is the cross-attention function. For learnable parameter matrix, For feature dimensions.

[0052] The output feature vector is the result of cross-attention fusion of text and audio modalities. Softmax is a normalized exponential function that converts the attention score into a probability distribution.

[0053] The dynamic modality weights determined in step S102 are combined with the cross-attention fusion results to obtain the final fusion feature vector.

[0054] For the current speaking round, the fused features retain both the fine-grained cross-modal alignment information under the text-driven approach and the macro-adjustment of the global modal importance by dynamic prior weights, effectively suppressing cross-modal noise and strengthening key sentiment cues.

[0055] S106: Input the sequence of features corresponding to multiple speaking rounds into the state space model to obtain a context-enhanced representation.

[0056] After completing step S104 to extract multimodal fusion features for each speaking round, a fusion feature sequence of the entire dialogue history is obtained. In step S106, this sequence is input into a State Space Model (SSM). By utilizing the model's ability to model long-distance dependencies with linear complexity, the dynamic evolution of sentiment with each dialogue round is efficiently tracked temporally, and context-enhanced representations are output as input conditions for subsequent hierarchical sentiment generation.

[0057] The state space model adopts a state space model based on a selective scanning mechanism. The discretized state evolution equation of the state space model realizes long-distance dependency modeling with linear complexity. Furthermore, the same state space model is used as a shared decoder to execute the structured generation task.

[0058] In a preferred embodiment of the present invention, the Mamba model is used as the state space model. After the fused feature sequence is input into the Mamba model, the model updates the hidden states round by round and outputs the corresponding output vector. It should be noted that the Mamba model outputs a context-enhanced representation sequence of the same length as the input. , The context-enhanced representation for the t-th round of speech encodes the trajectory of emotional evolution from the start of the dialogue to the current round, as well as the multimodal fusion information of the current round.

[0059] In a further preferred embodiment of the present invention, the same Mamba model is used not only for the above-mentioned encoding process (from fusion feature sequence to context representation), but also as a shared decoder to perform the two-stage hierarchical sentiment generation task in subsequent steps.

[0060] Because the Mamba model employs a selective scanning mechanism and a parallel algorithm with linear complexity, the computation time in this step increases linearly with the sequence length when processing long dialogues, while its memory usage is significantly lower than that of the Transformer architecture. This enables the invention to be deployed in real-time human-computer interaction systems (such as intelligent customer service and emotional companion robots), meeting the requirements of low latency and low resource consumption.

[0061] S108: Reconstruct emotion recognition into a structured generation task. The structured generation task includes: a first generation stage, generating emotion polarity; and a second generation stage, generating fine-grained emotion labels based on emotion polarity.

[0062] Traditional sentiment recognition methods typically treat sentiment classification as an independent discriminative task, directly predicting the sentiment category for the current round from a fixed set of sentiment labels (e.g., {happiness, sadness, anger, surprise, fear, disgust}). These methods suffer from two main drawbacks: first, they ignore the inherent hierarchical relationships between sentiment labels—there are clear subordinate constraints between emotional polarities (e.g., positive, negative, neutral) and fine-grained sentiments (e.g., "anger" necessarily belongs to the "negative" polarity); second, discriminative classification cannot generate structured output sequences, making it difficult to leverage the dependencies between sentiment levels to improve predictive consistency and accuracy.

[0063] To address the aforementioned problems, this invention reconstructs the traditional discriminative sentiment classification task into a structured generation task. Specifically, the sentiment recognition result of each round of dialogue is represented as a structured output sequence Y=[S,E], where: S represents the emotional polarity, taken from the predefined polarity vocabulary S={positive, negative, neutral}; E represents the fine-grained sentiment label, taken from the predefined sentiment vocabulary E={happiness, sadness, anger, surprise, fear, disgust}. It should be noted that different datasets may contain more or fewer categories; this is just an example.

[0064] In the first generation stage, the decoder takes the context-enhanced representation Z output in step S106 as input to generate an emotion polarity label S. The polarity S output in this stage not only serves as a high-level summary of the current emotion recognition result, but more importantly, it will be passed as an explicit conditional signal to the second generation stage, providing macro-emotional guidance for the subsequent generation of fine-grained emotions.

[0065] The second generation stage, guided by the polarity label S generated in the first stage, further generates fine-grained sentiment labels E. The conditional probability P(E|S,Z) is calculated as follows: Input sequence construction: The polarity token S generated in the first stage is concatenated after the start symbol to form the input prefix of the decoder: [<bos>The hidden state of the decoder continues the final state of the first phase.

[0066] Autoregressive generation: The decoder continues to generate the next token, i.e., the fine-grained sentiment tag E, based on the current hidden state and the input prefix. Since the sentiment vocabulary usually has more than 3 words, this step can generate a complete sentiment word in just one step (if word-level generation is used), or it can generate a more fine-grained sequence of sub-words (but usually the sentiment tag is a single word).

[0067] Conditional dependency: By using S as a prefix input, the decoder can learn the semantic consistency constraints between S and E. For example, if S = "positive", the model tends to assign high probabilities to words like "happy" and "surprised" and low probabilities to words like "sad" and "angry", and vice versa. This explicit polarity guidance mechanism fundamentally avoids the conflict between polarity and emotion.

[0068] This application's embodiments complete fine-grained sentiment tag inference under polarity constraints, ensuring strict semantic consistency between the output sentiment and polarity, eliminating sentiment conflicts at the model structure level, and achieving accurate and compliant hierarchical sentiment generation.

[0069] S110: Apply sentiment consistency constraints and use a composite objective function to perform end-to-end joint optimization of the parameters involved in feature extraction, temporal modeling and hierarchy generation; To ensure that the model not only performs well in classification accuracy, but also that its output sentiment prediction results are semantically consistent, i.e. there is no logical conflict between sentiment polarity and fine-grained sentiment, and the internal representation of the same sentiment is compact, this embodiment of the invention introduces sentiment consistency constraints during the training phase, and combines these constraints with the standard generation loss to form a composite objective function, performing end-to-end joint optimization of all learnable parameters in the model.

[0070] In traditional generative sentiment recognition training, only negative log-likelihood (NLL) loss is used to maximize the generation probability of true label sequences. While this loss can encourage the model to output the correct polarity and sentiment, it does not explicitly penalize predictions that are "semantically conflicting"—for example, the model may output illogical sequences such as "positive-sad" with a high probability, or the score difference between "positive" and "negative" in the model's internal representation may not be significant enough, leading to a decrease in generalization performance.

[0071] To address this issue, this invention proposes a boundary-based marginal loss, which directly affects the sentiment prediction score output by the model. It mandates that for any training sample, the average score of all categories that are semantically consistent with the true sentiment label must be at least one preset margin higher than the scores of all categories that are semantically conflicting.

[0072] In this way, the model is encouraged to learn the prior hierarchical structure between sentiment categories, making the sentiment representation in the output space semantically separable, thereby improving the consistency and robustness of the prediction results.

[0073] S112: For real-time input multimodal dialogue streams, through state space model state update and structured generation tasks, the emotional polarity and fine-grained sentiment labels corresponding to the current speaking round are output sequentially.

[0074] Following training processes S102 to S110, the model has obtained optimized parameters. In practical applications, the system receives real-time input multimodal dialogue streams (e.g., user voice, video, and real-time transcribed text from an intelligent customer service system). It needs to perform real-time emotion recognition on each new statement and output the corresponding emotion polarity and fine-grained emotion label for that round. This step details the execution process of the inference phase.

[0075] In real-world human-computer interaction or sentiment analysis tasks, simply outputting two labels is often insufficient to support an empathetic response. Therefore, this invention further defines a structured empathetic sentiment output mechanism, which packages the model's internal state updates and two-stage decoding results into structured data that contains both a clear sentiment category and implies dialogue context and sentiment intensity. This empowers downstream systems (such as intelligent customer service and emotional companion robots) to generate more natural and empathetic responses.

[0076] "Empathy" is not merely a list of labels, but emphasizes that the model's emotional output helps the machine understand the user's emotional state and respond appropriately. To this end, the structured output implicitly or explicitly provides the following empathy elements: Clear Emotional Hierarchy: By explicitly distinguishing between emotional polarity (general direction) and fine-grained emotion (specific feelings), the downstream system can respond to different granularities. For example, when the polarity is "negative," the system can first express concern ("I understand you are feeling down right now"), and then adjust the soothing strategy according to the fine-grained emotion ("sadness" or "anger").

[0077] Contextual continuity: The hidden state updates of the state-space model ensure that the current output sentiment is generated based on the entire dialogue history. Although the hidden state is not usually output directly, historical information is automatically integrated within the model, so the output sentiment label naturally has contextual coherence, avoiding abrupt changes in sentiment caused by independent predictions in each round.

[0078] Confidence indicator: The confidence field in the output can be calculated from the maximum predicted probability of the model or the maximum logit value after Softmax. This confidence level can help downstream systems determine whether further clarification is needed or whether to fall back to the default response.

[0079] This invention innovatively introduces modality importance priors for dynamic weighted fusion, which differs from the traditional pure data-driven fusion method. It can effectively suppress cross-modal redundant information and noise interference, automatically strengthen key clues that are highly related to emotion in text, audio, and vision, significantly improve the robustness and reliability of multimodal feature aggregation, and maintain stable output even in modality missing or noisy scenarios.

[0080] This method achieves long-range dependency modeling with O(L) linear complexity based on the SSM / Mamba selective scanning mechanism. Compared to the traditional Transformer quadratic complexity scheme, it significantly improves inference speed and reduces training memory usage, efficiently handling ultra-long multi-turn dialogue contexts and perfectly adapting to low-latency scenarios such as real-time human-computer interaction. Employing a hierarchical two-stage generation strategy, combined with consistency loss and constrained inference mechanisms, it explicitly models the hierarchical dependency from emotion polarity to fine-grained sentiment, fundamentally solving the problem of polarity and category semantic conflict in sentiment prediction. This makes the output results more consistent with human emotional logic, achieving a 0% conflict rate on the Interactive Emotional Dyadic Motion Capture (IEMOCAP) and Multimodal EmotionLines Dataset (MELD) datasets. IEMOCAP and MELD are two of the most commonly used benchmark datasets in the field of multimodal dialogue sentiment recognition (MERC). Both are multimodal, multi-speaker dialogue sentiment databases.

[0081] It surpasses current state-of-the-art models on two authoritative multimodal emotion recognition datasets, IEMOCAP and MELD, achieving new highs in accuracy and weighted F1 scores, and significantly improving its ability to identify rare categories such as sadness, anger, and fear. It supports flexible combinations of text, audio, and visual modalities, and can be trained and deployed end-to-end. It is widely applicable to practical systems such as intelligent customer service, emotional support, public opinion analysis, and mental health monitoring, possessing strong engineering and scalability capabilities.

[0082] Table 1 shows the performance comparison of the present invention with mainstream methods in sentiment classification on the IEMOCAP dataset, including the accuracy of each sentiment category, the overall accuracy, and the weighted F1 score.

[0083] Table 2 shows the performance comparison of the present invention with mainstream methods in sentiment classification on the MELD dataset, including the accuracy of each sentiment category, the overall accuracy, and the weighted F1 score.

[0084] Table 3 shows the statistics of the number of dialogues and sentences, and the number of sentiment categories in the training, validation, and test sets of the IEMOCAP and MELD datasets.

[0085] Table 4 shows the main hyperparameter settings used by the Prior-guided Consistency-aware Affective Generation (PCAG) framework of this invention on two datasets.

[0086] Table 5 shows the ablation experiment results of the PCAG framework of this invention, verifying the impact of each core module and different modal combinations on performance.

[0087] Table 1. Experimental results on the IEMOCAP dataset.

[0088] Table 2 Experimental results on the MELD dataset

[0089] Table 3. Dataset statistics for IEMOCAP and MELD

[0090] Table 4. Main hyperparameters of PCAG

[0091] Table 5 Performance of PCAG in ablation experiments

[0092] Experimental results show that the present invention achieves state-of-the-art performance on the two authoritative multimodal dialogue sentiment datasets IEMOCAP and MELD. The accuracy and weighted F1 score significantly outperform the existing best methods, sentiment prediction consistency is greatly improved, and the conflict rate is reduced to 0%.

[0093] Ablation experiments show that prior-guided fusion, multimodal semantic alignment, prior guidance of sentiment latent states, hierarchical two-stage generation, and consistency constraints are all core key modules of the framework. The absence of any one of these components will lead to a significant performance degradation. At the same time, the effectiveness of text-audio-visual trimodal fusion is verified.

[0094] In an optional embodiment, dynamic modality weights are generated based on the contribution of each modality in the current speaking round, including: The modality weights are calculated based on the initial characteristics of the modality and the preset confidence parameters.

[0095] After extracting the initial features for each modality, dynamic fusion weights need to be generated based on the instantaneous contribution of each modality in the current speaking round. Traditional multimodal fusion methods typically employ fixed weights (such as simple averaging) or purely data-driven adaptive weights (such as directly outputting weights through an attention network). However, fixed weights cannot adapt to the dynamic changes in modal information content during dialogue, while purely data-driven methods are prone to overfitting in small sample scenarios and lack prior guidance.

[0096] To address the aforementioned issues, this embodiment proposes a prior-guided semi-adaptive fusion mechanism: a learnable prior trust parameter is introduced for each modality, which reflects the model's prior trust in the modality's emotion recognition; then, combined with the initial features of each modality in the current round, the contribution score of each modality is dynamically calculated through a lightweight transformation network, and finally, the modality weights are obtained through Softmax normalization.

[0097] The formula for calculating the weights of modal importance priors is:

[0098] in, For modality The original representation, This indicates the model's level of trust in that modality, aiming to dynamically adjust the influence of each modality on the fused features through prior knowledge.

[0099] In an optional embodiment, the weighted aggregation employs a cross-attention mechanism, wherein the features of the text modality are used as query terms, the features of the audio modality are concatenated with the features of the visual modality, and the joint features obtained after concatenation are used as key terms and value terms.

[0100] A cross-attention mechanism is used for early fusion, where text features are used as the query, and audio and visual features are used as the key and value, respectively. The specific formula is as follows:

[0101]

[0102] in For learnable parameter matrix, This is a feature dimension. The design reflects the dominant role of textual modality in multimodal fusion.

[0103] In an optional embodiment, the state space model adopts a state space model based on a selective scanning mechanism, and the discretized state evolution equation of the state space model realizes long-distance dependency modeling under linear complexity; and the same state space model is used as a shared decoder to perform structured generation tasks.

[0104] The state-space model uses the Mamba selective scan mechanism for sequence modeling, and its discretized state evolution equation is as follows:

[0105]

[0106]

[0107]

[0108] in, The mechanism achieves the fusion feature obtained in step S104. Modeling long-range dependencies in dialogue with linear complexity. The fused feature sequence obtained in step S104... The Mamba state-space model is input and encoded to obtain the context vector:

[0109] in This is the last time step of the sequence. The same Mambabackbone is then used as a shared decoder for subsequent two-stage hierarchical generation, achieving unified and efficient modeling of encoding and decoding.

[0110] In an optional embodiment, imposing an emotion consistency constraint includes: during training, dividing the emotion prediction scores output by the model into a consistent subset and a conflict subset based on the real emotion labels, and applying a marginal penalty to the scores in the conflict subset.

[0111] During training, to ensure that the sentiment prediction results output by the model remain semantically consistent (i.e., to avoid logical contradictions such as "positive-sad"), this embodiment of the invention introduces a boundary-based marginal loss. This loss does not directly apply to the generated sequence, but rather to the sentiment prediction scores (logits) output by the model. By forcing the category scores that are semantically consistent with the true labels to be significantly higher than the category scores that are semantically conflicting, the model's understanding of the sentiment hierarchy is enhanced.

[0112] Based on the sentiment labeling system used in the task, the semantic relationships between the labels are predefined. Typically, sentiment labels can be organized into a hierarchical tree structure: the top layer represents emotional polarity (e.g., positive, negative, neutral), and the bottom layer represents finer-grained emotions (e.g., happiness, sadness, anger). Based on this hierarchical structure, it can be determined whether any two sentiment labels are consistent (semantically compatible) or conflicting (semantically contradictory).

[0113] Consistent subset A: The true sentiment label y for the current training sample true , and all of them with y true Categories with the same polarity and non-conflicting fine-grained emotions are labeled as a consistent subset. Typically, this set contains at least the true label itself and the polarity label to which it belongs. For example, if the true emotion is "anger," the consistent subset could include "anger" (fine-grained) and "negative" (polarity).

[0114] Conflict Subset I: Categories that contradict the semantics of the true label are marked as conflict subsets. For example, if the true emotion is "anger," then all emotions under the positive polarity (such as "happiness," "surprise") and the "positive" polarity itself belong to the conflict subset. Neutral polarities usually do not conflict with positive or negative polarities, but under certain strict definitions, if the true emotion is positive or negative, then neutrality can also be considered a conflict.

[0115] In a simple yet effective implementation, only the true labels themselves and their corresponding polarities are considered a consistent subset, while all other fine-grained sentiment categories are considered conflicting subsets. This definition, though somewhat coarse, is computationally simple and effective. For the MELD or IEMOCAP datasets, a predefined polarity-sentiment mapping table can be used for partitioning.

[0116] Employing boundary-based training constraints, sentiment logits are generated using real sentiment labels during training. Divided into emotionally consistent subsets and conflict subset The average logit of the consistent subset is calculated as a reference, and a margin penalty is applied to the logits of the conflicting subset. The specific constraint formula is as follows:

[0117] in, The loss is defined by the emotion consistency constraint, where I is the index set of the conflict subset. Let A be the size of the conflicting subset, and A be the set of indices of the consistent subset. The size of the consistent subset; The output for the i-th sentiment category is logit. The j-th sentiment category outputs logit. These are the preset margin hyperparameters.

[0118] In an optional embodiment, the marginal penalty is: calculating the average score of the consistent subset as a reference, and applying a preset margin value as a penalty to the difference between each score in the conflict subset and the average score.

[0119] During training, for each training sample, the sentiment prediction scores output by the model are first divided into a consistent subset A and a conflicting subset I based on the true sentiment labels. The average score of the consistent subset is calculated as a benchmark. For each category in the conflicting subset, the difference between its score and the average score is calculated, and a hinge penalty with a preset margin value m (e.g., m=1 or 3) is applied: a penalty is applied when the difference is less than m; otherwise, the penalty is zero. This marginal penalty is included in the total loss function as a consistency constraint loss term. The applied marginal penalty can force the model to make the scores of semantically consistent sentiment categories significantly higher than those of semantically conflicting categories, thereby eliminating illogical sentiment predictions such as "positive-sad," improving the sentiment consistency and robustness of the model's output, and enhancing its ability to discriminate rare sentiment categories.

[0120] In an optional embodiment, the composite objective function is a weighted sum of the negative log-likelihood loss and the sentiment consistency constraint loss.

[0121] End-to-end training is performed using a combination of generation loss and consistency loss, with the hyperparameter γ set to 1.0. The overall training loss function of the method is... Includes generated terms and constraint terms:

[0122] During training, the AdamW optimizer is used to synchronously update the feature extractor, SSM temporal modeling module, and hierarchical generator decoder to achieve optimal alignment of all module parameters.

[0123] In an optional embodiment, the emotional polarity and fine-grained sentiment label corresponding to the current speaking turn are output sequentially, including: Decoding is performed using a constrained bundle search. At each decoding step, the conditional probability of the candidate word list is calculated, and the multiple paths with the highest probabilities are retained. The sentiment tag sequence with the highest global log-likelihood score is output.

[0124] Furthermore, the inference process employs a constrained bundle search at each step. The model calculates the conditional probability of the candidate word list and, by retaining the Top-K probability paths, finally outputs the sentiment tag sequence that meets the sequence definition and has the highest global log-likelihood score.

[0125] Optionally, output the emotional polarity and fine-grained sentiment tags corresponding to the current speaking round, including: The predicted scores of emotional polarity and fine-grained sentiment labels are combined to form all valid sentiment pairs. The joint score is calculated according to the preset mapping rules, and the sentiment pair with the highest score is selected as the output.

[0126] In a preferred embodiment of the inference phase, the model does not use autoregressive decoding to generate sequences. Instead, it outputs the predicted score vectors for emotion polarity and fine-grained sentiment labels, respectively. First, all legal (polarity, sentiment) combinations (e.g., "positive-happy", "negative-sad") are predefined based on sentiment domain knowledge, excluding illegal combinations such as "positive-sad". Then, for each legal pair, a joint score is calculated according to a preset mapping rule (e.g., multiplying the predicted probabilities of polarity and sentiment, or calculating their weighted sum). Finally, the joint scores of all legal pairs are sorted in descending order, and the sentiment pair with the highest score is selected as the final output for the current speaking round. The technical advantages of this implementation are: no word-by-word autoregressive decoding is required, inference can be completed in a single forward propagation, significantly reducing computational latency and meeting real-time requirements; at the same time, by selecting results only from legal pairs, semantic conflicts between polarity and fine-grained sentiment (e.g., "positive-sad") are fundamentally eliminated, ensuring the self-consistency and interpretability of the output sentiment, making it particularly suitable for edge computing and large-scale concurrent scenarios.

[0127] Figure 2 A structural block diagram of a multimodal dialogue emotion recognition system provided in this application embodiment is shown below. Figure 2 As shown, the system includes: Extraction unit 201 is used to extract initial features of text modality, visual modality and audio modality in the dialogue history, which contains multiple speaking rounds arranged in chronological order; Weighting unit 203 is used to generate dynamic modality weights based on the contribution of each modality in the current speaking round, and to use the weights to aggregate the features of each modality to obtain the fusion features corresponding to each speaking round. The representation acquisition unit 205 inputs a sequence input state space model consisting of the employee and the fused features corresponding to multiple speaking rounds to obtain a context-enhanced representation. The task generation unit 207 is used to reconstruct emotion recognition into a structured generation task, which includes: a first generation stage, generating emotion polarity; and a second generation stage, generating fine-grained emotion labels based on emotion polarity. The optimization unit 209 is used to apply sentiment consistency constraints and to perform end-to-end joint optimization of the parameters involved in feature extraction, temporal modeling and hierarchy generation using a composite objective function. Output unit 211 is used to output the emotional polarity and fine-grained sentiment label corresponding to the current speaking turn in turn through state update and structured generation tasks of state space model for real-time input multimodal dialogue stream.

[0128] The device provided in this application embodiment has the same implementation principle and technical effect as the aforementioned method embodiment. For the sake of brevity, any parts not mentioned in the device embodiment can be referred to the corresponding content in the aforementioned method embodiment.

[0129] It should be noted that similar labels and letters in the following figures indicate similar items. Therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures.

[0130] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions marked in the blocks may occur in a different order than those marked in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0131] The foregoing has described specific embodiments of this application. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps described in the claims may be performed in a different order than that shown in the embodiments and may still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require the specific or sequential order shown to achieve the desired result. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0132] In the description of the embodiments of this application, the terms "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., refer to specific features, structures, materials, or characteristics described in connection with that embodiment or example, which are included in at least one embodiment or example of the embodiments of this application. In the embodiments of this application, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials, or characteristics described may be combined in a suitable manner in any one or more embodiments or examples. Furthermore, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in the embodiments of this application, as well as the features of different embodiments or examples.

[0133] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of embodiments of this application, "multiple" means at least two, such as two, three, etc., unless otherwise explicitly specified.

[0134] Any process or method description in the flowchart or otherwise herein can be understood as representing a module, segment, or portion of code comprising one or more executable instructions for implementing custom logic functions or processes, and the scope of preferred embodiments of this application includes additional implementations in which functions may be performed not in the order shown or discussed, including substantially simultaneously or in reverse order according to the functions involved, as should be understood by those skilled in the art to which embodiments of this application pertain.

[0135] Depending on the context, the word "if" as used here can be interpreted as "when" or "when" or "in response to determination" or "in response to detection." Similarly, depending on the context, the phrase "if determination" or "if detection (of the stated condition or event)" can be interpreted as "when determination" or "in response to determination" or "when detection (of the stated condition or event)" or "in response to detection (of the stated condition or event)."

[0136] It should be noted that the terminals involved in the embodiments of this application may include, but are not limited to, personal computers (PCs), personal digital assistants (PDAs), wireless handheld devices, tablet computers, mobile phones, MP3 players, MP4 players, etc.

[0137] In the embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the displayed or discussed mutual couplings, direct couplings, or communication connections may be through some interfaces; indirect couplings or communication connections between devices or units may be electrical, mechanical, or other forms.

[0138] Furthermore, the functional units in the various embodiments of this application can be integrated into a single processor, or each unit can exist physically separately, or two or more units can be integrated into a single unit. The integrated units described above can be implemented in hardware or in a combination of hardware and software functional units.

[0139] The integrated units implemented as software functional units described above can be stored in a computer-readable storage medium. These software functional units, stored in a storage medium, include several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor to execute some steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0140] The above description is only a preferred embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present application should be included within the scope of protection of the present application.< / bos>

Claims

1. A multimodal dialogue emotion recognition method, characterized in that, Includes the following steps: Initial features of textual, visual, and audio modalities are extracted from the dialogue history, which contains multiple speaking rounds arranged in chronological order. Dynamic modality weights are generated based on the contribution of each modality in the current speaking round. The features of each modality are then weighted and aggregated using these weights to obtain the fusion features corresponding to each speaking round. The sequence formed by the fusion features corresponding to the multiple speaking rounds is input into the state space model to obtain a context-enhanced representation; The emotion recognition is reconstructed into a structured generation task, which includes: a first generation stage, generating emotion polarity; and a second generation stage, generating fine-grained emotion labels based on the emotion polarity. We impose sentiment consistency constraints and use a composite objective function to perform end-to-end joint optimization of the parameters involved in feature extraction, temporal modeling, and hierarchy generation. For real-time input multimodal dialogue streams, the state space model is updated and the structured generation task is used to sequentially output the emotional polarity and fine-grained sentiment labels corresponding to the current speaking round.

2. The multimodal dialogue emotion recognition method according to claim 1, characterized in that, The step of generating dynamic modality weights based on the contribution of each modality in the current speaking round includes: The weight of the modality is calculated based on the initial characteristics of the modality and the preset trust level parameter.

3. The multimodal dialogue emotion recognition method according to claim 1, characterized in that, The weighted aggregation employs a cross-attention mechanism, wherein the features of the text modality are used as query terms, the features of the audio modality are concatenated with the features of the visual modality, and the joint features obtained after concatenation are used as key and value terms.

4. The multimodal dialogue emotion recognition method according to claim 1, characterized in that, The state space model adopts a state space model based on a selective scanning mechanism, and the discretized state evolution equation of the state space model realizes long-distance dependency modeling with linear complexity. Furthermore, the same state-space model is used as a shared decoder to perform the structured generation task.

5. The multimodal dialogue emotion recognition method according to claim 1, characterized in that, The imposition of the sentiment consistency constraint includes: during the training process, dividing the sentiment prediction scores output by the model into a consistent subset and a conflict subset based on the real sentiment labels, and imposing a marginal penalty on the scores in the conflict subset.

6. The multimodal dialogue emotion recognition method according to claim 5, characterized in that, The marginal penalty is as follows: the average score of the consistent subset is calculated as a reference, and a preset margin value is applied as a penalty to the difference between each score in the conflict subset and the average score.

7. The multimodal dialogue emotion recognition method according to claim 1, characterized in that, The composite objective function is a weighted sum of the negative log-likelihood loss and the sentiment consistency constraint loss.

8. The multimodal dialogue emotion recognition method according to claim 1, characterized in that, The sequential output of the emotional polarity and fine-grained sentiment tags corresponding to the current speaking round includes: Decoding is performed using a constrained bundle search. At each decoding step, the conditional probability of the candidate word list is calculated, and the multiple paths with the highest probabilities are retained. The sentiment tag sequence with the highest global log-likelihood score is output.

9. The multimodal dialogue emotion recognition method according to claim 1, characterized in that, The sequential output of the emotional polarity and fine-grained sentiment tags corresponding to the current speaking round includes: The predicted scores of emotional polarity and fine-grained sentiment labels are combined to form all valid sentiment pairs. The joint score is calculated according to the preset mapping rules, and the sentiment pair with the highest score is selected as the output.

10. A multimodal dialogue emotion recognition system, characterized in that, include: An extraction unit is used to extract initial features of text modality, visual modality, and audio modality from the dialogue history, which includes multiple speaking rounds arranged in chronological order. The weighting unit is used to generate dynamic modality weights based on the contribution of each modality in the current speaking round, and to use the weights to perform weighted aggregation of the features of each modality to obtain the fusion features corresponding to each speaking round. The representation acquisition unit inputs the sequence of employees and the fused features corresponding to the multiple speaking rounds into the state space model to obtain a context-enhanced representation. The task generation unit is used to reconstruct emotion recognition into a structured generation task, which includes: a first generation stage, generating emotion polarity; and a second generation stage, generating fine-grained emotion tags based on the emotion polarity. The optimization unit is used to apply sentiment consistency constraints and to perform end-to-end joint optimization of the parameters involved in feature extraction, temporal modeling and hierarchy generation using a composite objective function. The output unit is used to output the emotional polarity and fine-grained sentiment label corresponding to the current speaking turn in turn, in turn, based on the state update of the state space model and the structured generation task, for the real-time input multimodal dialogue stream.

Images