Multi-modal emotion recognition method for dialogue scene

Abstract

According to the multi-modal emotion recognition method for the dialogue scene, emotion transfer auxiliary detection is introduced into multi-modal emotion recognition, optimization is carried out in combination with sample weighted focus contrast loss, and the recognition capacity of minority class and fuzzy class emotions is effectively improved. According to the technical scheme of the invention, the method comprises the following steps: 1, constructing a multi-modal emotion recognition model, and carrying out the deep extraction of multi-modal features; the multi-modal emotion recognition model comprises a feature extraction module, a modal interaction fusion module, an emotion transfer auxiliary module and an emotion classification module. Step 2, training the multi-modal sentiment analysis model; a composite optimization target is adopted in training, the target is formed by weighting and combining four core loss items, and regularization constraint is supplemented; and step 3, carrying out multi-modal sentiment analysis and recognition through the trained model.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to a multimodal emotion recognition method for dialogue scenarios, which is mainly applied to dialogue scenarios that require understanding human emotional states, such as intelligent customer service, online education, mental health monitoring, or public opinion analysis. Background Technology

[0002] Emotion Recognition in Conversation (ERC) is an important task in the field of natural language processing, aiming to automatically identify the emotional state expressed by speakers in multi-turn dialogues. The applications of multimodal emotion recognition have gradually expanded to areas such as dialogue generation, intelligent recommendation, opinion analysis, and clinical diagnostic assistance. Although existing research has achieved some success in this field, in real-world conversational scenarios, there are often complex couplings between different modalities, and emotional expressions are often more implicit and subtle. These characteristics make multimodal emotion recognition still face significant challenges.

[0003] The goal of multimodal dialogue emotion recognition is to accurately determine the emotional state of participants in different utterances by comprehensively utilizing multi-source information such as text, speech, and vision. The complexity of this task lies not only in the need for a deep understanding of the semantic and emotional features within a single modality, but also in revealing the potential dependencies and interaction mechanisms between different modalities to achieve comprehensive and reliable emotion recognition. In this process, inter-utterance dependencies, intramodal structural connections, and intermodal interactions play crucial roles. Inter-utterance relationships ensure the coherence of dialogue at the semantic and emotional levels; for example, a speaker's emotions are often influenced by the preceding context, which may manifest through different channels such as textual expression, vocal prosody, or facial movements. Intramodal relationships reflect the potential connections between multiple utterances within the same modality, while intermodal relationships reveal the complementarity and coupling between different modalities. For example, tonal variations in speech often correspond to emotional vocabulary in text, and facial expressions in the visual modality may correspond to the emotional intensity of speech. These intermodal interactions are important bases for improving the accuracy of emotion recognition. However, due to the heterogeneity of feature spaces and differences in expression methods among different modalities, simple direct fusion can easily lead to the loss of effective information or the introduction of noise. Therefore, how to achieve accurate alignment and efficient fusion of cross-modal information, so that each modality can form a synergistic effect, has become the core issue in improving the performance of dialogue emotion recognition.

[0004] Currently, multimodal dialogue emotion recognition algorithms mainly revolve around Transformer-based attention mechanisms and graph-based interaction modeling. Existing research tends to rely on the text modality, using pre-trained language models (such as BERT, RoBERTa) or recurrent neural networks (such as LSTM, GRU) to semantically represent the dialogue text, and combining context windows or graph neural networks to model speaker and contextual dependencies, thus achieving some success in recognizing most emotion categories. However, because the text modality struggles to cover non-verbal cues such as tone of voice, speech rate variations, and facial expressions, its performance is limited when dealing with dialogues with ambiguous or unclear emotional expressions. To improve recognition performance, some studies have attempted to introduce multimodal fusion methods, combining audio and visual features in addition to text, and integrating information through feature concatenation, gating mechanisms, or attention mechanisms. Audio features are often extracted using Mel-frequency cepstral coefficients (MFCC), spectrogram convolution, or self-supervised speech representation models, while visual features rely on convolutional neural networks or video frame-level expression encoders. While these methods outperform single-modal methods in overall accuracy, most employ equal weighting of information from different modalities, failing to adequately differentiate the emotional information content across modalities. This introduces redundant information and negatively impacts fusion performance. Furthermore, they lag behind in modeling complex intermodal interactions, struggling to capture deep mappings between text, audio, and visual signals. In addition, contextual modeling methods for dialogue scenarios are gaining attention. Researchers utilize recurrent neural networks, Transformers, or graph neural networks to model dialogue structures, unifying the dependencies of multi-turn dialogues and improving temporal consistency in emotion recognition to some extent. However, these methods largely rely on static contextual relationships, neglecting the dynamic shifts in emotion during dialogue. When a speaker's emotion rapidly changes from positive to negative, the model often struggles to capture this change promptly, resulting in significantly lower accuracy in scenarios with rapid emotional fluctuations or shifts.

[0005] In summary, while existing dialogue emotion recognition technologies have made some progress in text modeling, multimodal fusion, and contextual dependence, they still have the following shortcomings: Existing dialogue emotion recognition technologies still have several limitations in multimodal information processing. Existing methods typically employ simple feature concatenation or equal-weighting in multimodal fusion, failing to fully leverage the complementary advantages of each modality and struggling to capture deep interaction relationships between modalities, resulting in insufficient quality of the fused representation. Furthermore, the recognition accuracy for a few emotion categories and semantically similar emotions is low, easily leading to classification confusion. In addition, existing technologies are insufficient in modeling the dynamic transfer of emotions, failing to effectively adapt to scenarios where emotions change rapidly or evolve complexly during dialogue. These problems limit the promotion and performance improvement of existing dialogue emotion recognition technologies in practical applications. Summary of the Invention

[0006] The purpose of this invention is to address the shortcomings of existing dialogue emotion recognition technologies in multimodal information fusion, minority class and semantically similar class recognition, and emotion dynamic transfer modeling. This invention proposes a multimodal emotion recognition method for dialogue scenarios, which introduces emotion transfer-assisted detection into multimodal emotion recognition and optimizes it by combining sample weighted focus contrast loss, effectively improving the recognition ability of minority class and ambiguous class emotions.

[0007] To achieve the above-mentioned objectives, the present invention adopts the following technical solution:

[0008] A multimodal emotion recognition method for dialogue scenarios, characterized by the following steps:

[0009] Step 1: Construct a multimodal emotion recognition model and perform deep extraction of multimodal features; the multimodal emotion recognition model includes a feature extraction module, a modal interaction fusion module, an emotion transfer assistance module, and an emotion classification module;

[0010] Step 2: Train the multimodal sentiment analysis model; a composite optimization objective is used during training. The objective is a weighted combination of four core loss terms, supplemented by regularization constraints;

[0011] Step 3: Perform multimodal sentiment analysis and recognition using the trained model.

[0012] The feature extraction module described in this invention processes input text, audio, and video data in multiple emotional modalities to obtain corresponding text features, audio features, and video features respectively.

[0013] The feature extraction module of this invention includes the following steps:

[0014] S1.1 Text Feature Extraction: The initial semantic representation of the discourse is obtained through a pre-trained Transformer model, and the context dependency is modeled by a bidirectional gated recurrent unit to obtain context-related text features.

[0015] S1.2 Audio Feature Extraction: Acoustic features are learned from the original audio using a self-supervised model, and then average pooling is used to form utterance-level vectors. These vectors are then input into the Transformer model to capture the temporal dependencies of the dialogue and generate context-related audio features.

[0016] S1.3 Video Feature Extraction: Keyframe sampling is performed on video clips, and a pre-trained face recognition network is used to extract frame-level visual features. Then, a temporal convolutional network is used to model the dynamic changes in facial expressions, ultimately obtaining speech-level video features.

[0017] The modal interaction fusion module described in this invention has a core parallel multimodal fusion unit. This parallel multimodal fusion unit is responsible for receiving three basic features—text, audio, and video—output by S1, and generating fusion features that can characterize complementary information between modalities through deep interaction. This parallel multimodal fusion unit consists of three pathways with shared parameters. Each pathway selects one modality as the primary query modality and the other two as auxiliary modalities. In the specific fusion process, the primary modal feature is subjected to multi-head cross-modal attention calculation with the two auxiliary modal features to simultaneously capture the correlation between modalities. Subsequently, the obtained interactive features are concatenated and subjected to dimensionality reduction and information integration through a linear projection layer. Then, they are subjected to nonlinear mapping by a feedforward network and combined with residual connections and layer normalization operations to finally form the deeply fused primary modal feature.

[0018] The modal interaction fusion module processing procedure described in this invention includes the following steps:

[0019] S2.1 Constructing parallel processing paths: The multimodal fusion unit consists of three processing paths that share parameters but have different inputs. Each path selects one of the following modalities—text, audio, and video—as the main query modal, while the other two are used as auxiliary information modalities.

[0020] S2.2 Perform parallel fusion operations: Within each processing path, information is integrated through parallel attention computation and subsequent aggregation.

[0021] The emotion transfer module described in this invention models the potential emotion transfer relationships between sentences as an auxiliary discrimination task, thereby enhancing the emotion recognition performance of the main task. Specifically, the sentence-level features obtained from the pre-encoder are first mapped to two new feature spaces for interaction calculations between sentences. The features of any two sentences are combined to generate a transfer representation, thus constructing a two-dimensional matrix reflecting the intensity of potential emotional transitions between sentences. Based on this, a corresponding supervision label matrix is ​​designed: if the true emotion labels of two sentences are different, it is considered that emotion transfer exists and is marked as transfer; if the labels are the same, the emotion is considered stable and is marked as no transfer.

[0022] The emotion transfer module processing procedure described in this invention includes the following steps:

[0023] S3.1 Construction of Emotional Transfer Features;

[0024] First, the set of statement-level feature vectors extracted by the preorder encoder is represented as follows:

[0025] F = {f1, f2, ..., f |U|},

[0026] Where |U| is the number of statements contained in the current dialogue;

[0027] Next, using two nonlinear mapping units with shared weights, the original features are mapped to two new feature spaces, forming F. (q) With F (k) ;

[0028] Based on this, the feature vectors of any two statements i and j are fed into the interaction function for fusion to obtain the transition representation:

[0029]

[0030] This operation generates a two-dimensional transition matrix:

[0031]

[0032] Each element reflects the intensity of the potential emotional transition from statement i to statement j;

[0033] S3.2: Emotional Transfer Tagging Design;

[0034] If the actual sentiment labels of two statements i and j are different, then it is assumed that there is a sentiment transfer between them, and the corresponding label is y. ij =1; if both labels are the same, the emotion is considered to remain stable and is labeled as y. ij =0; thus we obtain a supervision matrix Y with the same dimensions as matrix Z.

[0035] The sentiment classification module of this invention integrates and discriminates the multi-path fusion features output by the preceding module, and finally outputs the predicted sentiment category. First, it aggregates the three-path fusion features centered on text, audio, and video to obtain a unified vector that can comprehensively represent the current discourse information. Then, this vector is input into a multilayer perceptron for nonlinear feature transformation, and the activation function in the hidden layer is used to improve the model's expressive ability. Next, after feedforward calculation, the result is fed into a Softmax layer to obtain the probability distribution of each sentiment category. Finally, the category with the highest probability value is selected as the final predicted sentiment label for the discourse.

[0036] The emotion classification module processing procedure described in this invention includes the following steps:

[0037] S4.1 Feature Aggregation: The three-way multimodal fusion features f output by the modal interaction fusion module, centered on text, audio, and video respectively, are aggregated. ti f ai f vi Perform aggregation operations to form a unified aggregated feature vector f that can comprehensively represent the current discourse information. i ;

[0038] S4.2, Nonlinear mapping: The aggregated feature vector f i The input is fed into a multilayer perceptron for deep feature transformation;

[0039] S4.3 Category Probability Calculation: The output of the feedforward network is fed into a Softmax layer to calculate the probability distribution p of the utterance belonging to each preset sentiment category. i ;

[0040] S4.4 Sentiment Tag Prediction: Selecting a probability distribution p i The category with the highest probability value is used as the final predicted sentiment label for the i-th utterance.

[0041] The four core loss terms described in this invention are:

[0042] (1) Sample-focal contrast loss Its formula is defined as:

[0043]

[0044] in:

[0045] z i It is the feature vector output by the model for the i-th sample;

[0046] P(i) is the ratio of the sample z within the batch. i A set of positive samples belonging to the same category;

[0047] sim(z i ,z p ) is sample z i Its positive sample z p The similarity score between samples is calculated using a Softmax function with a temperature coefficient temp over all samples;

[0048] This is the sample weighting term. ci It is category c within the batch. i The number of samples, β s It is a hyperparameter for adjusting the weights;

[0049] (1-sim(z i ,z p )) βf It is the focal term, β f It is a focusing hyperparameter;

[0050] (2) Multimodal correlation loss Its formula is defined as:

[0051]

[0052] in, and cov(·) represent the expected value and covariance, respectively. In actual calculations, they are approximated by the sample mean and sample covariance within the batch.

[0053] (3) Loss of main category task Its formula is defined as:

[0054]

[0055] Where, p i y is the probability distribution of sentiment predicted by the model for the i-th sample. i It is the real label in its corresponding one-hot encoded form;

[0056] (4) Loss of emotional transfer Its formula is defined as:

[0057]

[0058] Where: y ij Indicates whether there is a true label for emotional transfer from statement i to statement j.

[0059] Compared with existing technologies, this invention has the following advantages and effects: It can independently assist in detecting and modeling emotion transfer patterns to enhance the temporal understanding of emotions in dialogue; simultaneously, through improved feature modeling and optimization strategies, it improves the model's recognition performance in minority and fuzzy categories. Furthermore, this invention can achieve higher recognition accuracy in real-world dialogue and multimodal interaction scenarios, thereby solving the accuracy and robustness limitations of existing technologies in application. Attached Figure Description

[0060] Figure 1 This is a flowchart of an embodiment of the present invention. Detailed Implementation

[0061] The present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be noted that the following embodiments are only used to explain the present invention and are not intended to limit the scope of protection of the present invention.

[0062] I. A multimodal emotion recognition method for dialogue scenarios according to an embodiment of the present invention includes the following steps:

[0063] Step 1: Construct a multimodal emotion recognition model and perform deep multimodal feature extraction. This multimodal emotion recognition model includes, in sequence, a feature extraction module, a modal interaction fusion module, an emotion transfer assistance module, and an emotion classification module.

[0064] S1. Feature Extraction Module: This module processes input emotional modality data, including text, audio, and video, to obtain corresponding text features, audio features, and video features. The processing includes the following steps:

[0065] S1.1 Text Feature Extraction: Initial semantic representations of the utterance are obtained through a pre-trained Transformer model, and contextual dependencies are modeled using a Bidirectional Gated Recurrent Unit (Bi-GRU) to obtain context-related text features. The processing includes the following steps:

[0066] First, each text utterance T in the dialogue i It is independently input into a pre-trained Transformer model, and an initial semantic representation vector e is generated by taking the output of a specific label in its final hidden layer. i Subsequently, the semantic representation sequence of the entire dialogue {e1,e2,…,e...} is... N The input is fed into a Bidirectional Gated Recurrent Unit (Bi-GRU) network to model temporal contextual information. Finally, the hidden state output of the Bi-GRU at the i-th time step is used as the context-relevant text feature of the utterance. This process can be expressed by the following formula:

[0067] e i =ROBERTa(T) i ),

[0068]

[0069] S1.2 Audio Feature Extraction: Acoustic features are learned from the original audio using self-supervised models such as Wav2Vec 2.0. These features are then averaged to form utterance-level vectors, which are then input into a one-way Transformer model to capture dialogue temporal dependencies, generating context-dependent audio features. The processing includes the following steps:

[0070] Effective acoustic features are learned directly from the original audio waveform. The original audio segment A for each utterance... i The signal is fed into a pre-trained self-supervised learning model, such as Wav2Vec 2.0, which maps the original audio signal to a high-dimensional acoustic feature sequence. This sequence is then subjected to time-dimension mean pooling to obtain a fixed-dimensional vector 'a' representing the overall acoustic information of the utterance. i To further integrate into the dialogue context, the entire acoustic vector sequence {a1, a2, ..., a...} of the dialogue is used. NA separate unidirectional Transformer model layer is fed in, and its temporal dependencies are captured through a self-attention mechanism. The output of this encoder layer at position i is the final context-dependent audio feature. This process can be expressed by the following formula:

[0071] a i =MeanPool(Wav2Vec(A i )),

[0072]

[0073] S1.3 Video Feature Extraction: Keyframe sampling is performed on video clips, and a pre-trained face recognition network is used to extract frame-level visual features. Furthermore, a temporal convolutional network (TCN) is used to model dynamic changes in facial expressions, ultimately obtaining speech-level video features. The processing includes the following steps:

[0074] Capture dynamic changes in facial expressions. First, for each utterance, analyze the video clip V... i Keyframe sampling is performed to obtain K representative images. For each image, a pre-trained face recognition and feature extraction network is used to extract the visual features of the face region. The visual features of the K frames are concatenated along the time dimension to obtain a frame sequence feature matrix M. i Finally, the matrix is ​​input into a lightweight Temporal Convolutional Network (TCN), which uses multiple layers of causal convolution to capture the dynamic trends of facial expressions. The final output of the TCN is then subjected to global average pooling to obtain the context-relevant video features of the utterance. This process can be expressed by the following formula:

[0075]

[0076] S2, the modal interaction fusion module, is centered on a parallel multimodal fusion unit. This unit receives the three basic features (text, audio, and video) output from S1 and generates fusion features that represent complementary information between modalities through deep interaction. This fusion unit consists of three pathways sharing parameters. Each pathway selects one modality as the primary query modality and the other two as auxiliary modalities. During the fusion process, the primary modal feature undergoes multi-head cross-modal attention calculation with the two auxiliary modal features to simultaneously capture the correlation between modalities. Subsequently, the resulting interactive features are concatenated and subjected to dimensionality reduction and information integration through a linear projection layer. Then, they undergo nonlinear mapping via a feedforward network, combined with residual connections and layer normalization operations, ultimately forming the deeply fused primary modal feature. The processing includes the following steps:

[0077] S2.1 Constructing Parallel Processing Paths: The multimodal fusion unit consists of three processing paths that share parameters but have different inputs. Each path selects one of the following modalities—text, audio, or video—as the primary query modality, while the other two serve as auxiliary information modalities.

[0078] S2.2 Perform parallel fusion operations: Within each processing path, information is integrated through parallel attention computation and subsequent aggregation. Taking the path with text as the main query modality as an example, the fusion process includes:

[0079] Parallel cross-modal interaction: combining the main modal features of the previous (or initial) layer. (For example or or As a query, it is used in conjunction with two auxiliary modal features X. a1 and X a2 Parallel multi-head cross-modal attention computations are performed to simultaneously capture the correlation information between the main modality and each auxiliary modality, resulting in two independent interaction features H1 and H2.

[0080] Feature aggregation: The features H1 and H2 obtained from two parallel interactions are concatenated, and then dimensionality reduction and information integration are performed through a linear projection layer to generate aggregated features H. agg .

[0081] Nonlinear transformation: converting aggregated features H agg The input is fed into a feed-forward network for nonlinear mapping, and combined with residual connections and layer normalization operations, finally yielding the deeply fused main modality features output by the current layer.

[0082] The specific mathematical implementation of this process can be expressed by the following formula:

[0083] Parallel attention computation:

[0084]

[0085] Aggregation and Feedforward Networks:

[0086]

[0087] in:

[0088] l represents the current computing level;

[0089] It is the dominant modal feature output from the previous layer;

[0090] Xa1 ,X a2 These represent the features of the two auxiliary modes, respectively.

[0091] CrossModalAttn(Q, K, V) represents the standard multi-head attention function;

[0092] M agg (·) represents a standard feedforward network that includes a nonlinear activation function.

[0093] S3, the emotion transfer module, models the potential emotion transfer relationships between sentences as an auxiliary discrimination task, thereby enhancing the emotion recognition performance of the main task. Specifically, the sentence-level features obtained from the pre-encoder are first mapped to two new feature spaces for interaction calculations between sentences. The features of any two sentences are combined to generate a transfer representation, thus constructing a two-dimensional matrix reflecting the intensity of potential emotional transitions between sentences. Based on this, a corresponding supervision label matrix needs to be designed: if the true emotion labels of two sentences are different, it is considered that emotion transfer exists and is marked as transfer; if the labels are the same, it is considered that the emotion remains stable and is marked as no transfer. The resulting supervision signal provides a reference for model training, thereby better capturing the emotional change patterns in dialogue. Its processing includes the following steps:

[0094] S3.1 Construction of Emotional Transfer Features.

[0095] In a dialogue sequence, emotional states may shift between different statements. To capture this dynamic change, the set of statement-level feature vectors extracted by the pre-encoder is first represented as:

[0096] F = {f1, f2, ..., f |U|},

[0097] Where |U| represents the number of statements in the current dialogue. Then, using two non-linear mapping units with shared weights, the original features are mapped to two new feature spaces, forming F. (q) With F (k) This is to facilitate subsequent statement interactions.

[0098] Based on this, the feature vectors of any two statements i and j are fed into an interactive function for fusion, such as concatenation or element-wise operation, to obtain the transition representation:

[0099]

[0100] This operation generates a two-dimensional transition matrix:

[0101]

[0102] Each element reflects the intensity of the potential emotional transition from statement i to statement j.

[0103] S3.2: Emotional transfer label design.

[0104] To supervise the above transfer representation, a transfer label matrix needs to be constructed. Specifically, if the true sentiment labels of two statements i and j are different, then a sentiment transfer is considered to exist between them, and is labeled as y. ij =1; if both labels are the same, the emotion is considered to remain stable and is labeled as y. ij =0. This yields a supervision matrix Y with the same dimensions as matrix Z, providing a reliable reference for subsequent training.

[0105] S4, the sentiment classification module, integrates and discriminates the multi-path fusion features output from the preceding modules, ultimately outputting the predicted sentiment category. First, it aggregates the three-path fusion features centered on text, audio, and video to obtain a unified vector that comprehensively represents the current discourse information. Then, this vector is input into a multilayer perceptron for nonlinear feature transformation, utilizing activation functions in the hidden layers to enhance the model's expressive power. Next, after feedforward computation, the result is fed into a softmax layer to obtain the probability distribution of each sentiment category. Finally, the category with the highest probability value is selected as the final predicted sentiment label for the discourse. Its processing includes the following steps:

[0106] S4.1 Feature Aggregation: The three-way multimodal fusion features f output by the modal interaction fusion module, centered on text, audio, and video respectively, are aggregated. ti f ai f vi Perform aggregation operations to form a unified aggregated feature vector f that can comprehensively represent the current discourse information. i .

[0107] S4.2, Nonlinear mapping: The aggregated feature vector f i The input is fed into a multilayer perceptron for deep feature transformation. This network contains at least one hidden layer that uses ReLU as the activation function to enhance the model's non-linear expressive power.

[0108] S4.3 Category Probability Calculation: The output of the feedforward network is fed into a Softmax layer to calculate the probability distribution p of the utterance belonging to each preset sentiment category. i .

[0109] S4.4 Sentiment Tag Prediction: Selecting a probability distribution p i The category with the highest probability value is used as the final predicted sentiment label for the i-th utterance.

[0110] The specific mathematical calculation process can be expressed by the following formula:

[0111] f i =[f ti +f ai +f vi ];

[0112] z i =W (1) f i +b (1) ;

[0113] h i =σ(z) i );

[0114] o i =W (2) h i +b (2) ;

[0115]

[0116]

[0117] Among them, W (1) Let b be the weight matrix. (1) σ(·) represents the bias term, σ(·) represents the ReLU activation operation, and C represents the number of sentiment categories.

[0118] Step 2: Train the multimodal sentiment analysis model. This invention employs a composite optimization objective. The objective is a weighted combination of four core loss terms, supplemented by regularization constraints, aiming to achieve end-to-end optimization of the model. The mathematical expression of this overall optimization objective is as follows:

[0119]

[0120] Where ω1, ω2, ω trans It is a weighting coefficient used to balance different optimization tasks, ω reg Θ is the L2 regularization hyperparameter, B represents the batch size, and Θ refers to the set of all learnable parameters in the model.

[0121] (1) Sample-focal contrast loss This improves the model's ability to distinguish between minority class samples and semantically confusing class samples. It achieves this by introducing a sample weighting factor and a dynamic focusing factor into the contrastive learning framework. The sample weighting factor assigns higher optimization weights to minority class samples based on class frequency; the dynamic focusing factor guides the model to focus more on positive sample pairs with low similarity and difficulty in distinguishing them. Its formula is defined as:

[0122]

[0123] in:

[0124] z i It is the feature vector output by the model for the i-th sample;

[0125] P(i) is the ratio of the sample z within the batch. i A set of positive samples belonging to the same category;

[0126] sim(z i ,z p ) is sample z i Its positive sample z p The similarity score between samples is calculated using a Softmax function with a temperature coefficient temp over all samples;

[0127] It is a sample weighting term. It is category c within the batch. i The number of samples, β s It is a hyperparameter for adjusting the weights;

[0128] It is the focal term, β f It is a focusing hyperparameter;

[0129] (2) Multimodal correlation loss Maximizing different modal representations (text E) t Audio E a Visual E v The correlation between modal representations is used to facilitate the model's learning of complementary and consistent information between modalities. This loss is constructed based on the Hirschfeld-Gebelein-Rényi (HGR) correlation coefficient, which measures the dependency between modal representations and is defined as follows:

[0130]

[0131] in, cov(·) and cov(·) represent the expected value and covariance, respectively. In actual calculations, they are approximated by the sample mean and sample covariance within the batch.

[0132] (3) Loss of main category task Supervised sentiment classification task: Measure the difference between the predicted probability distribution and the true labels to ensure the model's performance on the core task. Its definition is as follows:

[0133]

[0134] Where, p iy is the probability distribution of sentiment predicted by the model for the i-th sample. i It is the actual label in its corresponding one-hot encoded form.

[0135] (4) Loss of emotional transfer This approach models the emotion transfer relationship between statements as a binary classification problem, predicting whether emotion transfer exists between any two statements. The loss-constrained model learns the patterns of emotion evolution in dialogue.

[0136] During training, each element in the two-dimensional transfer matrix is ​​input into a feedforward classification layer, which outputs the corresponding transfer probability. The sentiment transfer task is modeled as a binary classification problem, predicting whether a transfer exists between any two statements. The loss function uses the cross-entropy form:

[0137]

[0138] This loss effectively constrains the model to learn the transition relationships between sentences, enabling it to better understand the emotional evolution process in dialogue.

[0139] Where: y ij Indicates whether there is a true label for emotional transfer from statement i to statement j.

[0140] By combining multiple loss functions with weighted summaries and regularization constraints, end-to-end optimization is achieved, enabling the model to capture modal complementarity relationships and dynamic changes in dialogue emotions while recognizing emotions.

[0141] Step 3: When the multimodal sentiment analysis model converges or reaches the preset number of iterations, the training of the multimodal sentiment analysis model is completed, and multimodal sentiment analysis and recognition can be performed using the trained model.

[0142] II. Experiment.

[0143] The proposed method was implemented on two benchmark multimodal emotion recognition datasets, IEMOCAP and MELD, using the PyTorch framework. Table 1 shows the size and class distribution of the two datasets. During model training, batch sizes of 64 (IEMOCAP) and 100 (MELD) were used, with 100 training epochs. The optimizer was Adamax, with an initial learning rate of 0.0001, which gradually decreased during training. To improve the model's generalization ability, Dropout (set to 0.1) was introduced into each network layer. The stacking number of layers in the multimodal interaction module was set to 6 to ensure sufficient learning of cross-modal attention. In the loss function design, classification loss, sample-weighted focus contrast loss, and modality relevance loss were used in combination. By setting reasonable temperature coefficients, sample weight coefficients, and combined weight coefficients, attention to hard-to-classify samples and constraints on modality consistency were achieved. These settings ensured stable convergence of the model on both datasets and effectively avoided overfitting.

[0144] Table 1. Size and category distribution of the IEMOCAP and MELD datasets.

[0145] Dataset training set test set category IEMOCAP 5810 1623 6 MELD 11098 2610 7

[0146] Table 2 shows the experimental results on the IEMOCAP dataset.

[0147]

[0148] Table 3 shows the experimental results on the MELD dataset.

[0149]

[0150] Tables 2 and 3 present the experimental results of this invention. On the IEMOCAP dataset, the method of this invention achieves excellent performance in the recognition of all six emotion categories, significantly outperforming existing models in categories such as Happiness, Sadness, and Frustration, with an overall Weighted-F1 score of 73.52%, an improvement of approximately 2.86 percentage points compared to the best comparable method. On the MELD dataset, this invention also leads in the recognition of seven emotion categories, with particularly significant improvements in the recognition performance of difficult categories such as Surprise, Fear, and Disgust, ultimately achieving a Weighted-F1 score of 66.96%, surpassing the existing best method.

[0151] The above results demonstrate that the emotion recognition method proposed in this invention can effectively improve the overall performance of multimodal dialogue emotion recognition, and exhibits stronger discriminative ability in minority and easily confused categories. Its performance improvement is mainly attributed to:

[0152] (1) The designed modal interaction fusion mechanism can make full use of the complementary information between different modalities;

[0153] (2) Introducing emotion transfer modeling helps to capture the pattern of emotion evolution in dialogue;

[0154] (3) Joint optimization of multiple loss functions ensures classification accuracy and model generalization ability.

[0155] Furthermore, it should be noted that all equivalent or simple variations made based on the structure, features, and principles described in this patent concept are included within the protection scope of this patent. Those skilled in the art can make various modifications or additions to the described specific embodiments or use similar methods to replace them, as long as they do not deviate from the structure of this invention or exceed the scope defined in these claims, all of which should fall within the protection scope of this invention.

Claims

1. A multimodal emotion recognition method for dialogue scenarios, characterized in that: Includes the following steps: Step 1: Construct a multimodal emotion recognition model and perform deep extraction of multimodal features; This multimodal emotion recognition model includes a feature extraction module, a modal interaction fusion module, an emotion transfer assistance module, and an emotion classification module; Step 2: Train the multimodal sentiment analysis model; The training employs a composite optimization objective, which is a weighted combination of four core loss terms, supplemented by regularization constraints. Step 3: Perform multimodal sentiment analysis and recognition using the trained model.

2. The multimodal emotion recognition method for dialogue scenarios according to claim 1, characterized in that: The feature extraction module processes the input text, audio, and video data in various emotional modalities to obtain corresponding text features, audio features, and video features.

3. The multimodal emotion recognition method for dialogue scenarios according to claim 2, characterized in that: The feature extraction module process includes the following steps: S1.1 Text Feature Extraction: The initial semantic representation of the discourse is obtained through a pre-trained Transformer model, and the context dependency is modeled by a bidirectional gated recurrent unit to obtain context-related text features. S1.2 Audio Feature Extraction: Acoustic features are learned from the original audio using a self-supervised model, and then average pooling is used to form utterance-level vectors. These vectors are then input into the Transformer model to capture the temporal dependencies of the dialogue and generate context-related audio features. S1.3 Video Feature Extraction: Keyframe sampling is performed on video clips, and a pre-trained face recognition network is used to extract frame-level visual features. Then, a temporal convolutional network is used to model the dynamic changes in facial expressions, ultimately obtaining speech-level video features.

4. The multimodal emotion recognition method for dialogue scenarios according to claim 1, characterized in that: The modal interaction fusion module is centered on a parallel multimodal fusion unit. This parallel multimodal fusion unit is responsible for receiving the three basic features of text, audio and video output by S1, and generating fusion features that can characterize complementary information between modalities through deep interaction. This parallel multimodal fusion unit consists of three pathways with shared parameters. Each pathway selects one modality as the primary query modality and the other two as auxiliary modalities. During the fusion process, the primary modality features are subjected to multi-head cross-modal attention calculations with the two auxiliary modality features to simultaneously capture the correlation between modalities. Subsequently, the obtained interactive features are spliced ​​together and then dimensionality reduced and information integrated through a linear projection layer. After nonlinear mapping by a feedforward network, combined with residual connections and layer normalization operations, the deep fusion master modality features are finally formed.

5. The multimodal emotion recognition method for dialogue scenarios according to claim 4, characterized in that: The modal interaction fusion module processing procedure includes the following steps: S2.1 Constructing parallel processing paths: The multimodal fusion unit consists of three processing paths that share parameters but have different inputs. Each path selects one of the following modalities—text, audio, and video—as the main query modal, while the other two are used as auxiliary information modalities. S2.2 Perform parallel fusion operations: Within each processing path, information is integrated through parallel attention computation and subsequent aggregation.

6. The multimodal emotion recognition method for dialogue scenarios according to claim 1, characterized in that: The aforementioned emotion transfer module models the possible emotion transfer relationships between sentences as an auxiliary discrimination task, thereby enhancing the emotion recognition effect of the main task. In specific implementation, the sentence-level features obtained by the pre-encoder are first mapped to two new feature spaces to perform interactive calculations between sentences. The features of any two sentences are combined to generate a transfer representation, thereby constructing a two-dimensional matrix that reflects the intensity of potential emotional transitions between sentences.

7. The multimodal emotion recognition method for dialogue scenarios according to claim 6, characterized in that: The emotion transfer module process includes the following steps: S3.1 Construction of Emotional Transfer Features; First, the set of statement-level feature vectors extracted by the preorder encoder is represented as follows: F={f1,f2,…,f |U| }, Where |U| is the number of statements contained in the current dialogue; Next, using two nonlinear mapping units with shared weights, the original features are mapped to two new feature spaces, forming F. (q) With F (k) ; Based on this, the feature vectors of any two statements i and j are fed into the interaction function for fusion to obtain the transition representation: This operation generates a two-dimensional transition matrix: Each element reflects the intensity of the potential emotional transition from statement i to statement j; S3.2: Emotional Transfer Tagging Design; If the actual sentiment labels of two statements i and j are different, then it is assumed that there is a sentiment transfer between them, and the corresponding label is y. ij =1; if both labels are the same, the emotion is considered to remain stable and is labeled as y. ij =0; thus we obtain a supervision matrix Y with the same dimensions as matrix Z.

8. The multimodal emotion recognition method for dialogue scenarios according to claim 1, characterized in that: The sentiment classification module integrates and discriminates the multi-path fusion features output by the preceding module, ultimately outputting the predicted sentiment category. First, it aggregates the three-path fusion features centered on text, audio, and video to obtain a unified vector that comprehensively represents the current discourse information. Subsequently, this vector is input into a multilayer perceptron for nonlinear feature transformation, utilizing activation functions in the hidden layer to enhance the model's expressive power. Next, after feedforward calculation, the result is fed into a Softmax layer to obtain the probability distribution of each sentiment category. Finally, the category with the highest probability value is selected as the final predicted sentiment label for the discourse.

9. The multimodal emotion recognition method for dialogue scenarios according to claim 8, characterized in that: The emotion classification module processing procedure includes the following steps: S4.1 Feature Aggregation: The three-way multimodal fusion features f output by the modal interaction fusion module, centered on text, audio, and video respectively, are aggregated. ti f ai f vi Perform aggregation operations to form a unified aggregated feature vector f that can comprehensively represent the current discourse information. i ; S4.2, Nonlinear mapping: The aggregated feature vector f i The input is fed into a multilayer perceptron for deep feature transformation; S4.3 Category Probability Calculation: The output of the feedforward network is fed into a Softmax layer to calculate the probability distribution p of the utterance belonging to each preset sentiment category. i ; S4.4 Sentiment Tag Prediction: Selecting a probability distribution p i The category with the highest probability value is used as the final predicted sentiment label for the i-th utterance.

10. The multimodal emotion recognition method for dialogue scenarios according to claim 1, characterized in that: The four core loss terms are: (1) Sample-focal contrast loss Its formula is defined as: in: z i It is the feature vector output by the model for the i-th sample; P(i) is the ratio of the sample z within the batch. i A set of positive samples belonging to the same category; sim(z i ,z p ) is sample z i Its positive sample z p The similarity score between samples is calculated using a Softmax function with a temperature coefficient temp over all samples; It is a sample weighting term. It is category c within the batch. i The number of samples, β s It is a hyperparameter for adjusting the weights; It is the focal term, β f It is a focusing hyperparameter; (2) Multimodal correlation loss Its formula is defined as: in, and cov(·) represent the expected value and covariance, respectively. In actual calculations, they are approximated by the sample mean and sample covariance within the batch. (3) Loss of main category task Its formula is defined as: Where, p i y is the probability distribution of sentiment predicted by the model for the i-th sample. i It is the real label in its corresponding one-hot encoded form; (4) Loss of emotional transfer Its formula is defined as: Where: y ij Indicates whether there is a true label for emotional transfer from statement i to statement j.

Images