A Multimodal Emotion Recognition Method for Intelligent Interaction

Abstract

This invention discloses a multimodal emotion recognition method for intelligent interaction. It extracts unimodal features from raw text, audio, and visual data in input dialogue utterances, and inputs these features into a speaker-perception recurrent neural network module. This module models the evolution of the speaker's internal emotional state and the emotional interactions between speakers in parallel. The resulting enhanced features, rich in speaker relational context information, are then input into a cross-modal convolutional attention fusion module. Through stacked bidirectional multi-head cross-attention layers and convolutional layers, multiple rounds of cross-modal interaction are performed to uncover local and global correlations between modalities, generating deeply fused multimodal features for predicting the probability distribution of emotion categories. This invention achieves more efficient cross-modal interaction, more comprehensive dialogue context understanding, and stronger discriminative ability against imbalanced data and semantically similar emotions. It can be applied to emotion understanding tasks in intelligent customer service, online education, meeting analysis, and human-computer interaction systems.

Description

Technical Field

[0001] This invention belongs to the fields of artificial intelligence, affective computing and multimodal information processing technology, and specifically relates to a multimodal emotion recognition method for intelligent interaction, which is applicable to application scenarios such as human-computer dialogue, intelligent customer service and online interaction. Background Technology

[0002] With the rapid development of information technology, especially the widespread adoption of interactive platforms such as social media, online conferencing, and virtual reality, human communication is increasingly characterized by multimodality, real-time interaction, and emotional engagement. Accurately understanding the emotions within a conversation is crucial for building empathetic artificial intelligence systems.

[0003] In dialogue contexts, the expression and evolution of emotions are driven by complex factors, primarily including: self-dependency: referring to the inherent continuity of a speaker's own emotional state, where current emotions are often influenced by previous emotional states; and interpersonal dependence: referring to the mutual influence of emotions among dialogue participants, such as contagion, imitation, or antagonism. An effective multimodal emotion recognition model for conversation (ERC) must be able to capture the contextual information of the dialogue, including local and global dependencies of the utterance sequence, as well as the interaction relationships between different speakers. Early ERC research mainly focused on the text modality, using sequence models such as recurrent neural networks and long short-term memory networks to model the context. In recent years, Transformer-based models have also been introduced into ERC tasks due to their powerful long-range dependency modeling capabilities. However, relying solely on the text modality ignores important cues rich in emotional information, such as intonation, rhythm, pitch in speech, and facial expressions and body language in vision. Despite the progress made in multimodal ERC research, the following key challenges remain, which constitute the technical problems that this invention aims to solve: (1) Insufficient fusion of multimodal information: Text, speech, and visual modal data exist in heterogeneous semantic spaces, with different statistical characteristics and temporal dynamics. Simple early or late fusion is insufficient to uncover deep-seated complementary and synergistic relationships between modalities. Existing methods, such as attention-based or Transformer-based fusion mechanisms, can model global interactions, but they are insufficient in capturing fine-grained, locally related cross-modal associations. Conversely, methods that rely solely on convolutional neural networks are good at extracting local features, but their ability to model global context is weak.

[0004] (2) Insufficient characterization of speaker relationships in dialogue context modeling: Many models treat dialogue as a continuous sequence, failing to explicitly distinguish and model the emotional evolution patterns within different speakers and the mutual influence of emotions between speakers. Although some studies have assigned independent recurrent neural networks (RNNs) to each speaker or used global RNNs for indirect interaction, this approach often fails to directly capture the complex and dynamic binary interaction relationships between speakers, resulting in incomplete capture of key interpersonal emotional cues in the dialogue.

[0005] (3) Class imbalance and difficulty in distinguishing semantically similar emotions: In reality, there is a common problem of highly uneven distribution of class sample numbers. At the same time, some emotion categories are highly similar in semantics, such as "anger" and "disgust". During training, the model tends to favor the category with more samples and learn insufficiently the minority categories; for semantically similar categories, their features are easily confused in the embedding space, resulting in low discriminative power.

[0006] In summary, existing technologies have limitations in terms of the depth and efficiency of multimodal fusion, the fine-grained modeling of speaker relationships in the context of dialogue, and robust optimization for data imbalance and semantic similarity. Summary of the Invention

[0007] To overcome the shortcomings of the prior art, the present invention aims to provide a multimodal emotion recognition method for intelligent interaction. This method, through innovative modular design, aims to achieve more efficient cross-modal interaction, more comprehensive understanding of dialogue context, and stronger ability to distinguish between imbalanced data and semantically similar emotions. It can be applied to practical scenarios such as intelligent customer service, online education, meeting analysis, and mental health assistance.

[0008] To achieve the above objectives, the technical solution adopted by the present invention is as follows: A multimodal emotion recognition method for intelligent interaction includes the following steps: Step 1: Extract unimodal features from the raw text, audio, and visual data of the input dialogue utterances; Step 2: Input the extracted modal features into a speaker-perception recurrent neural network module. This module models the evolution of the speaker's internal emotional state and the emotional interaction between speakers in parallel, and outputs enhanced features rich in speaker relational context information. Step 3: Input the enhanced features of each modality into a cross-modal convolutional attention fusion module. This module performs multiple rounds of cross-modal interaction through stacked bidirectional multi-head cross-attention layers and convolutional layers to mine the local and global correlations between modalities and generate deeply fused multimodal features. Step 4: Input the multimodal features into the classifier to predict the probability distribution of sentiment categories.

[0009] Compared with the prior art, the present invention has the following beneficial effects: (1) This invention exhibits higher recognition accuracy and robustness in complex real-world scenarios, such as online meetings and intelligent customer service in intelligent interactive systems. In practical applications (such as online meetings and customer service dialogues), issues such as speech noise, facial occlusion, and text ambiguity often exist. Traditional methods show significant performance degradation when single-modal signals are unreliable. This invention, through a speaker-aware recurrent neural network module, explicitly and in parallel models the evolution of the speaker's internal state and the interaction between speakers, better aligning with the psychological mechanisms of emotion generation in dialogue and capturing richer contextual emotional cues. This invention combines multi-head cross-attention with convolutional operations through a cross-modal convolutional attention fusion module. Attention mechanisms excel at establishing long-range, global intermodal associations, while convolutional layers effectively capture local, fine-grained cross-modal correspondence patterns. The two complement each other, enabling modal fusion to both view the global context and focus on local details, significantly improving the depth and effectiveness of multimodal interaction. Specifically, this invention achieves deep cross-modal complementary fusion through the CMCA module. When information in one modality is weakened, other modalities can be used for effective compensation and correction. At the same time, the refined dialogue structure modeling provided by the Speaker-RNN module enables the system to combine historical context and speaker roles for comprehensive reasoning, effectively combating local noise interference. This allows the system to maintain stable and accurate emotion recognition capabilities in complex and ever-changing real-world environments, significantly improving the system's practical reliability.

[0010] (2) This invention significantly improves fine-grained emotion recognition capabilities and enhances the human-computer interaction experience. In scenarios such as emotional support, high-end customer service, and mental health screening, distinguishing between semantically similar emotions such as "anger" and "disgust" is crucial. Traditional methods are limited by class imbalance and feature confusion, resulting in limited discriminative power. This invention optimizes the Assured Loss Function (ASWFC) specifically for minority classes and semantically similar classes, enabling the model to learn more discriminative representations in the feature space, thereby more accurately identifying subtle emotional differences. This allows systems using this invention to provide more empathetic and personalized feedback, significantly improving the naturalness of human-computer interaction and user satisfaction. Specifically, the adaptive sample weighted focus contrast loss widens the inter-class distance and narrows the intra-class distance through contrastive learning. It also intelligently adjusts the intensity of attention to minority and similar class samples during training through a dynamically decaying adaptive weight mechanism. This design avoids overfitting or underfitting that may result from fixed weights, improving the minority class recognition rate while maintaining the overall generalization performance of the model, effectively solving the challenges of class imbalance and semantic similarity in ERC.

[0011] (3) The end-to-end architecture of this invention balances performance and deployment efficiency. Traditional multimodal systems often employ complex multi-stage pipelines, resulting in high latency and difficult deployment. This invention adopts an end-to-end deep learning architecture, completing the process from raw data to sentiment prediction in one integrated manner, without the need for manually designing complex rules. Although it contains multiple innovative modules, core computations such as attention calculation and convolution operations can be efficiently parallelized. After optimization, the system can achieve high-speed inference, meeting the needs of real-time interactive scenarios such as real-time meeting sentiment perception and live stream sentiment analysis, achieving a good balance between accuracy and efficiency while maintaining excellent recognition performance.

[0012] (4) This invention has good domain adaptability and scalability. The distribution of sentiment data varies across different application domains. Traditional fixed models require a large amount of re-labeled data for full training. The powerful cross-modal fusion capability and robustness to imbalanced learning of this invention enable it to learn efficiently from limited domain data. The modular design allows for easy integration of new modalities or replacement of more advanced underlying encoders, and can be quickly adapted to new tasks through fine-tuning, significantly reducing the cost and difficulty of cross-domain migration and long-term iterative upgrades of the system. Attached Figure Description

[0013] Figure 1 This invention provides the framework for a multimodal emotion recognition method based on convolutional enhanced attention fusion and focus contrast learning.

[0014] Figure 2 This is a schematic diagram of the speaker perception recurrent neural network module of the present invention.

[0015] Figure 3 This is a schematic diagram of the cross-modal convolutional attention fusion module of the present invention.

[0016] Figure 4 This is a confusion matrix diagram of the MELD dataset in an embodiment of the present invention.

[0017] Figure 5 This is a schematic diagram illustrating the influence of hyperparameters (number of CMCA layers) in an embodiment of the present invention.

[0018] Figure 6 This is a schematic diagram illustrating the influence of hyperparameters (number of inter-speaker RNN layers) in an embodiment of the present invention.

[0019] Figure 7 This is a visualization result of the test set category t-SNE embedding space (10 rounds) in an embodiment of the present invention.

[0020] Figure 8 This is a visualization result of the test set category t-SNE embedding space (50 rounds) in an embodiment of the present invention. Detailed Implementation

[0021] The embodiments of the present invention will now be described in detail with reference to the accompanying drawings and examples.

[0022] To achieve more accurate and fine-grained multimodal dialogue sentiment recognition, this invention provides a multimodal sentiment recognition method for intelligent interaction. It integrates information from text, speech, and vision modalities, and through innovative context modeling and cross-modal fusion mechanisms, achieves accurate and fine-grained recognition of speaker emotions in dialogue. Furthermore, it incorporates a loss function designed to address class imbalance and semantic similarity issues, improving model training performance. This invention can be widely applied to scenarios such as intelligent customer service, social media sentiment analysis, online education sentiment perception, mental health auxiliary diagnosis, and immersive human-computer interaction. It can also be deployed on servers or terminal devices as a sentiment analysis module in intelligent interaction systems.

[0023] The specific implementation framework of this invention is as follows: Figure 1 As shown, it includes: 1. Single-modal feature extraction.

[0024] The raw text, audio, and visual data from the input dialogue are processed separately to extract high-level discourse-level feature representations for each modality.

[0025] In this invention, a module for multimodal information feature extraction is set up to extract and embed text, audio and visual features into a low-dimensional space to obtain the embedded features of the three modalities.

[0026] In embodiments of the present invention, the feature extraction module is described as follows. For the three modalities of text, audio, and visual data in the dialogue, dedicated feature extraction techniques are employed to transform the raw data into low-dimensional, efficient, and dense feature representations, providing high-quality basic features for subsequent modeling and fusion.

[0027] Text modality feature extraction: The pre-trained language model RoBERTa is used as the text encoder. This model has been pre-trained on a large-scale corpus and possesses powerful semantic understanding capabilities. Each utterance in the dialogue is input into the RoBERTa model, and the pre-trained model is fine-tuned using the ERC dataset to adapt to the specific needs of the dialogue sentiment recognition task, extracting a 768-dimensional high-dimensional text feature vector. This feature vector is then input into a fully connected layer, where dimensionality compression and feature optimization are performed through linear transformation to obtain a 256-dimensional text feature representation.

[0028] Audio modal feature extraction: Using the IS13 profile from the OpenSmile toolkit, acoustic features are extracted from each speech audio data. This profile can comprehensively capture and extract 6373-dimensional acoustic features, including fundamental frequency, energy, spectral features, and formants, fully representing the speaker's tone, volume, speech rate, and other emotion-related information. The extracted high-dimensional acoustic features are input into a fully connected layer, where feature filtering and compression are performed through nonlinear transformations of the neural network, ultimately yielding a 512-dimensional audio feature representation.

[0029] Visual modality feature extraction: First, algorithms such as MTCNN (Multi-Task Cascaded Convolutional Neural Network) are used to accurately detect and crop the speaker's facial region from video frames, effectively removing interference from irrelevant background information and focusing on facial expression regions that reflect emotional state. The cropped facial image is then input into a ResNet-101 model pre-trained on the VGGFace2 dataset. This model has powerful facial feature extraction capabilities, capturing subtle changes in facial expressions, and ultimately outputting a 1000-dimensional visual feature representation.

[0030] 2. Context-enhanced modeling.

[0031] To address the contextual dependencies within and between speakers, the extracted modal features are input into a speaker-perceptual recurrent neural network module to obtain contextual information. This module models the evolution of emotional states within speakers and the emotional interactions between speakers in parallel, outputting enhanced features rich in contextual information about speaker relationships.

[0032] In embodiments of the present invention, the speaker-perception recurrent neural network module is as follows: Figure 2 As shown, this module mainly consists of two sub-modules: inter-speaker RNN and intra-speaker RNN. These sub-modules model the emotional interactions between speakers and the evolution of emotional states within a speaker, respectively, reflecting the inter-speaker interdependence and the emotional coherence within a speaker. Furthermore, enhanced contextual feature representations, or augmented features, are obtained through feature fusion. This allows for a comprehensive capture of the emotional dynamics and changes in the dialogue, represented as follows: in Enhanced features representing text, audio, and visual modalities.

[0033] Furthermore, the inter-speaker RNN submodule extracts contextual information features, including inter-speaker and intra-speaker cues, using LSTM and DialogueRNN networks. Its modeling of inter-speaker context is defined as follows: in, For contextual features between speakers, This indicates a deep BiLSTM network. Indicates the number of floors. In a conversation The modality of discourse Preprocessing feature representation, Representing text, audio, and visual modalities. It is the preprocessing feature dimension of the modality. This represents the hidden state of the 0th utterance at level 0. Indicates the first Layer The hidden state of a discourse.

[0034] The intra-speaker RNN submodule is implemented through a bidirectional DialogueRNN network, using different GRU networks to model the speaker's internal environment. Specifically, given a dialogue... Each speaker can obtain a mask for each speaker. ,in Indicates whether the utterance was made by the speaker. Therefore, the modeling definition of the intra-speaker RNN submodule is as follows: in, For the contextual features within the speaker's speech, This represents a bidirectional DialogueRNN network. For the number of speakers, For mask data, i.e. .

[0035] Finally, the learned contextual features between speakers are connected with the contextual features within speakers to obtain more comprehensive contextual clue information.

[0036] in, Indicates a connection operation. .

[0037] 3. Cross-modal deep interaction and fusion.

[0038] To achieve interactive fusion of multimodal information, the enhanced features of each modality obtained through context enhancement are input into a cross-modal convolutional attention fusion module. This module performs multiple rounds of cross-modal interaction through stacked bidirectional multi-head cross-attention layers and convolutional layers to deeply explore the local and global correlations between modalities and generate deep fused multimodal features containing information from all modalities.

[0039] In embodiments of the present invention, the Cross-Modal Convolutional Attention (CMCA) module is as follows: Figure 3 As shown, this module combines the advantages of bidirectional multi-head cross-attention mechanisms and convolutional networks, simultaneously capturing global correlations and local feature correlations between modalities. Through multi-layer stacking, it achieves deep fusion of multimodal information, effectively solving the challenge of heterogeneous modal fusion. It mainly comprises three sub-modules: CMCAtext, CMCAaudio, and CMCAvisual, which respectively implement cross-modal fusion of text and audiovisual modalities, audio and text-visual modalities, and visual and text-audio modalities. The three sub-modules have identical structures, ultimately yielding standardized multimodal fusion features. , , ,Will , , The connection yields the deep fusion multimodal features F described in this invention.

[0040] The following is a detailed explanation of the fusion process using the CMCAtext (text and audio-visual modal fusion) submodule as an example: In the first stage, we learn the cross-modal associations and mappings between text modalities and audio modalities, and perform bidirectional multi-head cross-attention operations using text modalities as queries and audio modalities as keys and values.

[0041] In the second stage, the output of the first stage is used as a new query, the visual modality is used as the key and value of another cross-attention layer, the textual modality is fused with visual cues, and the fused information is input into a convolutional module to capture local features. The function is standardized to obtain standardized multimodal fusion features. .

[0042] Furthermore, for the CMCAtext submodule, the implementation method of its first stage is expressed by the formula: The method for achieving the second-stage fusion is expressed by the following formula: in, Textual features representing all given utterances. Indicates audio features, Representing video features, Represents the query matrix. Represents the key value matrix. Represents the Value matrix. Represents text modality, Indicates audio modality, Indicates video modality, This represents the features after text-audio fusion. The attention score represents the result of modality fusion. This represents a multi-head splicing matrix representing three modes. express , The merged Query, Key, and Value matrix express , , The merged Query, Key, and Value matrix express , The projection matrix of Query, Key, and Value after fusion. express , , The projection matrix of Query, Key, and Value after fusion. This indicates that attention should be paid to the number of heads. Indicates a connection operation. Indicates fusion , The weight matrix of the output transformation in the multi-head attention layer of the feature. It indicates that it has been integrated , , The weight matrix of the output transformation in the multi-head attention layer of the feature. This represents the features resulting from the fusion of three modalities using an attention mechanism.

[0043] Furthermore, the second-stage convolutional module consists of gated, pointwise convolutional, and gated linear units, and is connected to a depthwise convolutional layer, as shown in the following formula: in, This represents the convolution operation. This represents the features resulting from the fusion of text and audio-visual modalities via the CMCAtext module. The features obtained after convolution are then processed. Standardization is performed to stabilize gradients and facilitate efficient training of deep models, ultimately... The input consists of a feedforward network comprising two fully connected layers and a ReLU activation function. This invention enhances local cues by adding a deep convolutional layer after the attention layer and using a gating mechanism to point-to-convolutional network.

[0044] Furthermore, after multimodal fusion, the learned multimodal fusion text, audio, and visual feature representations are concatenated to obtain the deep fusion multimodal feature F, expressed by the formula: in, For the first in F i One characteristic, for The first in i One characteristic, for The first in i One characteristic, for The first in i One characteristic.

[0045] 4. Emotional Classification.

[0046] The fused multimodal features are input into a classifier to predict the probability distribution of sentiment categories.

[0047] In this embodiment of the invention, the features of each utterance after fusion The input classifier can consist of a fully connected layer and two multilayer perceptrons with ReLU activation. Finally, a softmax layer is used to calculate the probability distribution of the sentiment category to obtain the final prediction result, as shown in the following formula: in, Indicates the first The result after linear transformation of each feature Indicates the first The result after nonlinear transformation of each feature The sentiment label represents the predicted sentiment. For activation function, This is the weight matrix. This is the bias value. This represents the probability distribution for predicting sentiment.

[0048] 5. Model optimization.

[0049] In the model training phase, to address the issues of semantic similarity and a limited number of categories, a composite loss function is employed for optimization. The core of this loss function is an adaptive sample-weighted dynamic focus contrast loss, which dynamically adjusts the training focus, increases attention to minority class samples and semantically similar samples that are difficult to distinguish, and widens the distance between samples of different categories in the feature space through a contrastive learning mechanism. It is also supplemented by modality correlation loss and traditional classification loss to jointly guide model training.

[0050] In an embodiment of the present invention, the adaptive sample weighted focus contrast loss method is as follows: by introducing an adaptive sample weight adjustment term, higher weights are given to minority class samples that are difficult to classify during the training phase. As training progresses, the focus on minority classes is gradually reduced, thereby improving the generalization ability of the model. Simultaneously, the loss function makes sample pairs with different sentiment labels mutually exclusive, maximizing the inter-class distance and better distinguishing semantically similar sentiments.

[0051] In embodiments of the present invention, a Soft-HGR loss function is incorporated to enhance the correlation between multimodal information. This function aims to maximize the complementary relationships between different modalities, enhance their interdependence during training, and thereby improve the performance of the emotion recognition task.

[0052] satisfy. in, , , , , Represents the mathematical expectation. This represents the Soft-HGR loss function. The expectation and covariance are approximated by the sample mean and sample covariance.

[0053] In embodiments of the present invention, the cross-entropy loss function is used to optimize the performance of the multimodal emotion recognition model. It effectively measures the difference between the model's predicted emotion category distribution and the true labels, and exhibits good convergence and accuracy in multi-classification tasks.

[0054] Among them, It is a dialogue Chinese discourse The probability distribution in the sentiment category, It is a dialogue Chinese discourse The truth value label.

[0055] In embodiments of the present invention, the total loss function is composed of an adaptive sample-weighted focus contrast loss function. Soft-HGR loss function and standard cross-entropy loss function Composition, represented as: in, Representative by The number of samples in a batch consisting of 10 dialogues. Represents the number of sessions Indicates the first The number of conversations in a session Indicates the L2 regularization weights. Represents the set of trainable parameters. and These represent the weighting coefficients of the loss function. For batch labels The number of It is the first In the first session The truth value label of a statement, Is with A set of positive feature samples with the same sample label. For the first In the first session Output features of a fully connected layer for each discourse; Except A set of features other than those in the batch. This represents the adaptive sample weighting function. These are sample weight parameters that control the degree of attention given to the minority class. To adjust the adjustment factor for sample weights, This is the lower bound of the sample weight parameter. To control the temperature coefficient of the distance between samples, It forces the model to focus on the focusing parameters of samples that are difficult to classify. It is the first In the first session The probability distribution of each utterance in terms of sentiment.

[0056] This invention proposes an adaptive focus-based contrastive learning method based on a contrastive learning loss function. First, it uses softmax normalization to transform the cosine similarity between features into a probabilistic similarity score, which measures the similarity. and The similarity is measured, with values ​​ranging from 0 to 1. Secondly, the weight parameters in subsequent loss calculations are dynamically controlled using p to balance model stability and adaptability. For the entire loss: This represents the category balancing weight, which assigns higher weights to clusters with smaller sample sizes to balance the contribution of clusters of different sizes to the loss. This means that assigning higher weights to samples with low similarity scores allows the model to focus more on optimizing semantically similar samples.

[0057] To verify the effectiveness and practicality of the proposed method in real-world dialogue scenarios, the classic multimodal emotion recognition public dataset MELD was selected for experimental testing. This dataset contains text, audio, and visual multimodal information with fine-grained emotion annotations, fully simulating the complex emotional dynamics of multi-person interactions, making it suitable for quantitative evaluation and comparative analysis of the technical solution of this invention. The specific implementation process of this invention is as follows: 1. Experimental parameter design This experiment was conducted using the PyTorch framework and computed using an NVIDIA RTX 3090 GPU. In the adaptive sample weight focus contrast loss function, the temperature parameter, sample weight parameter, lower bound of the sample weight parameter, and focus parameter were set to 0.8, 0.8, 0.5, and 2, respectively. The adjustment factor was set to decay by 0.9 every 10 epochs, and the loss function coefficients were adjusted to 0.4 and 0.3, respectively. Dropout was set to 0.1, and the batch size was designed to be 100 on MELD. The number of training epochs was 100. The learning rate was initialized to 0.0001, and the Adam optimizer was used. β1 = 0.9, β2 = 0.99, and the L2 regularization weight was 0.00001.

[0058] 2. Introduction to the Dataset This invention evaluates the effectiveness of the proposed model on a multimodal benchmark dataset, the MultimodalEmotion Lines Dataset (MELD). This dataset contains multi-party dialogue data collected from television dramas, including 13,708 utterances and 1,433 dialogues. MELD includes three or more speakers in a single dialogue. Each utterance in the dialogue is labeled with seven emotion tags: anger, disgust, fear, joy, neutrality, sadness, and surprise.

[0059] 3. Introduction to the Baseline Model BC-LSTM: It encodes contextual information using bidirectional LSTM, but does not take into account different speaker information.

[0060] DialogueRNN: It simulates speakers and contextual information in a dialogue by using three different GRUs: Global GRU, Speaker GRU, and Emotional GRU.

[0061] MVN: It effectively captures the sentiment representation of a query from both word-level and discourse-level views.

[0062] UniMSE: It integrates acoustic and visual modal features with multi-level text features and performs inter-modal contrastive learning to obtain discriminative multimodal representations.

[0063] HiMul-LGG: A model that employs hierarchical decision fusion and local-global networks to capture intramodal and intermodal speaker dependencies.

[0064] 4. Comparative Experiment Table 1 shows a comparison between the present invention and existing baseline models on MELD. The confusion matrix is ​​as follows: Figure 4 As shown. Analyzing the experimental results, the following points can be observed: This invention achieves state-of-the-art performance on the Friends dataset, improving the overall w-F1 score by 1.21% compared to UniMSE, and outperforming all baseline models across all sentiment categories. Significant improvements are also observed in minority classes and semantically similar categories.

[0065] Table 1 Experimental results on the MELD dataset Baseline model neutral surprise fear sad joy disgust anger w-F1 BC-LSTM 76.31 47.10 5.10 18.95 53.04 5.20 39.55 56.70 DialogueRNN 76.13 45.93 1.23 26.23 54.55 2.82 43.10 59.21 MVN 76.35 53.17 11.15 21.86 53.62 21.86 42.53 59.06 UniMSE - - - - - - - 65.51 HiMul-LGG - - - - - - - 65.18 This invention 79.62 59.11 30.95 43.75 64.35 30.91 55.97 66.82 5. Ablation test To evaluate the effectiveness of each multimodal feature enhancement module in the model, ablation experiments were conducted. Table 2 shows the ablation experiment results for different modules. The results indicate that the combination of various components has a significant impact on the performance improvement of the model.

[0066] Removing the Speaker-RNN module: It can be observed that removing the Speaker-RNN module significantly degrades the model's emotion classification performance on the dataset. This is because contextual modeling captures historical information in the dialogue, including the speaker's emotional state, topic shifts, and the overall atmosphere of the conversation. This information is crucial. Furthermore, to analyze the impact of the components of the Speaker-RNN module on model performance, this paper also conducted ablation experiments on inter-RNN and intra-RNN. Removing each module leads to a decrease in model performance.

[0067] The CMCA module was removed, and a feature connection method was used to replace it to fuse multimodal features. Experimental analysis showed that removing the CMCA module significantly reduced model performance in the MELD sentiment category. This is because cross-modal interaction helps bridge the semantic gap between different modalities. Allowing cross-modal interaction between text, audio, and visual modalities not only enhances the correlation and dependency between features of the same dimension but also improves the accuracy of sentiment prediction.

[0068] Removing the ASWFC loss function: Experimental results show that the proposed ASWFC strategy significantly improves the classification performance of minority class sentiment while having a relatively small impact on the performance of majority class samples. This is clearly validated on the dataset, and the overall performance of the dataset is also improved. Notably, the ASWFC loss function has a significant impact on the MELD dataset. This is because MELD has more severe class imbalance and its label quality is higher, thus feature optimization based on label information is more effective.

[0069] Table 2 Ablation experimental results on the MELD dataset category neutral surprise fear sad joy disgust anger w-F1 Remove speaker-RNN 78.63 57.45 27.91 40.51 63.85 28.32 53.99 65.56 Remove inter-RNN 79.41 57.72 17.14 39.88 64.33 27.08 54.87 65.81 Remove intra-RNN 79.04 57.24 26.51 41.62 63.71 29.31 53.47 65.66 Remove CMCA 79.58 58.09 16.34 38.20 63.11 22.90 49.56 63.94 Remove ASWFC 79.08 58.14 27.16 39.39 64.10 25.49 55.07 65.78 CAFN-FCL 79.62 59.11 30.95 43.75 64.35 30.91 55.97 66.82 6. Hyperparameter Analysis For the selection of the number of CMCA layers and the number of inter-speaker RNN layers, this paper uses grid search based on test performance. Figure 5 The curves showing the weighted average F1 score variation for different numbers of CMCA layers on the MELD dataset are presented. On the MELD dataset, the weighted average F1 score exhibits a trend of first increasing and then decreasing, with optimal performance achieved at three CMCA layers. Figure 6 The results show the weighted average F1 score variation for different numbers of inter-speaker RNN layers on two datasets. In the MELD dataset, due to the shorter conversations and features of multiple speakers, the performance initially increases, then decreases, and eventually plateaus as the number of layers increases. This is because the dependencies between multiple speakers are strong, thus the three-layer inter-speaker RNN performs best.

[0070] 7. Embedded Spatial Visualization Analysis To visualize the model, t-SNE visualization results were analyzed on the MELD test set, demonstrating the model's performance. Figure 7 and Figure 8 The t-SNE visualization results for the 10th and 50th epochs are given respectively, where dark blue represents sadness, cyan represents neutrality, cyan represents anger, yellow-green represents excitement, orange-yellow represents joy, dark red represents frustration, and brown represents fear.

[0071] As can be observed from the figure, with the increase of iterations, all categories can be roughly separated, and different emotion samples form clear clusters. Positive emotions and negative emotions are relatively separated spatially, while sadness and neutral emotions are distributed in the lower left and lower right regions. This indicates that the proposed method can effectively classify different emotions.

Claims

1. A multimodal emotion recognition method for intelligent interaction, characterized in that, Includes the following steps: Step 1: Extract unimodal features from the raw text, audio, and visual data of the input dialogue utterances; Step 2: Input the extracted modal features into a speaker-perception recurrent neural network module. This module models the evolution of the speaker's internal emotional state and the emotional interaction between speakers in parallel, and outputs enhanced features rich in speaker relational context information. Step 3: Input the enhanced features of each modality into a cross-modal convolutional attention fusion module. This module performs multiple rounds of cross-modal interaction through stacked bidirectional multi-head cross-attention layers and convolutional layers to mine the local and global correlations between modalities and generate deeply fused multimodal features. Step 4: Input the multimodal features into the classifier to predict the probability distribution of sentiment categories.

2. The multimodal emotion recognition method for intelligent interaction according to claim 1, characterized in that, In step 1, a pre-trained language model RoBERTa is used as a text encoder, and fine-tuned using the ERC dataset to extract a 768-dimensional high-dimensional text feature vector. Then, dimensional compression and feature optimization are performed through linear transformation to obtain a 256-dimensional text feature representation. The IS13 configuration file of the OpenSmile toolkit is used to extract 6373-dimensional acoustic features, and then nonlinear transformation is used for feature filtering and compression to obtain a 512-dimensional audio feature representation. The MTCNN algorithm was used to detect and crop the speaker's facial region from video frames. The cropped facial image was then input into a ResNet-101 model pre-trained on the VGGFace2 dataset to obtain a 1000-dimensional visual feature representation.

3. The multimodal emotion recognition method for intelligent interaction according to claim 1, characterized in that, Step 2, the speaker perception recurrent neural network module includes an inter-speaker RNN submodule and an intra-speaker RNN submodule, which respectively model the emotional interaction between speakers and the evolution of emotional states within a speaker, and obtain enhanced features through feature fusion. , , Enhanced features representing text, audio, and visual modalities.

4. The multimodal emotion recognition method for intelligent interaction according to claim 3, characterized in that, The inter-speaker RNN submodule extracts contextual information features, including inter-speaker and intra-speaker cues, through LSTM and DialogueRNN networks; the intra-speaker RNN submodule is implemented through a bidirectional DialogueRNN network, using different GRU networks to model the speaker's internal environment. The modeling definition of the inter-speaker RNN submodule is as follows: in, For contextual features between speakers, This indicates a deep BiLSTM network. Indicates the number of floors. In a conversation The modality of discourse Preprocessing feature representation, Representing text, audio, and visual modalities. It is the preprocessing feature dimension of the modality. This represents the hidden state of the 0th utterance at level 0. Indicates the first Layer The hidden state of a discourse; The modeling definition of the intra-speaker RNN submodule is as follows: in, For the contextual features within the speaker's speech, This represents a bidirectional DialogueRNN network. For the number of speakers, For mask data, This indicates whether the utterance was made by the speaker. As stated.

5. The multimodal emotion recognition method for intelligent interaction according to claim 1, characterized in that, Step 3, the cross-modal convolutional attention fusion module includes a CMCAtext submodule, a CMCAaudio submodule, and a CMCAvisual submodule. These three submodules have identical structures and respectively implement cross-modal fusion of text and audiovisual modalities, audio and text-visual modalities, and visual and text-audio modalities, to obtain standardized multimodal fusion features. , , ,Will , , The deep fusion of multimodal features is obtained by connecting the components; the fusion process for the CMCAtext submodule is as follows: In the first stage, we learn the cross-modal associations and mappings between text modalities and audio modalities, and perform bidirectional multi-head cross-attention operations using text modalities as queries and audio modalities as keys and values. In the second stage, the output of the first stage is used as a new query, the visual modality is used as the key and value of another cross-attention layer, the textual modality is fused with visual cues, and the fused information is input into a convolutional module to capture local features. The function is standardized to obtain standardized multimodal fusion features. .

6. The multimodal emotion recognition method for intelligent interaction according to claim 5, characterized in that, For the CMCAtext submodule, the first stage is implemented as follows: In the second stage, the method for achieving fusion is expressed by the following formula: in, Textual features representing all given utterances. Indicates audio features, Representing video features, Represents the query matrix. Represents the key value matrix. Represents the Value matrix. Represents text modality, Indicates audio modality, Indicates video modality, This represents the features after text-audio fusion. The attention score represents the result of modality fusion. This represents a multi-head splicing matrix representing three modes. express , The merged Query, Key, and Value matrix express , , The merged Query, Key, and Value matrix express , The projection matrix of Query, Key, and Value after fusion. express , , The projection matrix of Query, Key, and Value after fusion. This indicates that attention should be paid to the number of heads. Indicates a connection operation. Indicates fusion , The weight matrix of the output transformation in the multi-head attention layer of the feature. It indicates that it has been integrated , , The weight matrix of the output transformation in the multi-head attention layer of the feature. This represents the features resulting from the fusion of three modalities using an attention mechanism.

7. The multimodal emotion recognition method for intelligent interaction according to claim 6, characterized in that, In the second stage, the convolutional module consists of gated, pointwise convolutional, and gated linear units, and is connected to a depthwise convolutional layer, as shown in the following formula: in, This represents the convolution operation. This represents the features obtained after convolution operations on the features of text and audio-visual modalities fused by the CMCAtext module.

8. The multimodal emotion recognition method for intelligent interaction according to claim 7, characterized in that, After multimodal fusion, the learned multimodal fusion text, audio, and visual feature representations are concatenated to obtain the deep fusion multimodal feature F, expressed by the formula: in, For the first in F i One characteristic, for The first in i One characteristic, for The first in i One characteristic, for The first in i One characteristic.

9. The multimodal emotion recognition method for intelligent interaction according to claim 8, characterized in that, Step 4 involves fusing the features of each discourse. The input classifier calculates the probability distribution of sentiment categories through a Softmax layer, yielding the final prediction result, expressed by the following formula: in, Indicates the first The result after linear transformation of each feature Indicates the first The result after nonlinear transformation of each feature The sentiment label represents the predicted sentiment. For activation function, This is the weight matrix. This is the bias value. This represents the probability distribution for predicting sentiment.

10. The multimodal emotion recognition method for intelligent interaction according to claim 1, characterized in that, During the model training phase, a composite loss function is used for optimization. This loss function is composed of an adaptive sample-weighted focus contrast loss function. Soft-HGR loss function and standard cross-entropy loss function Composition, represented as: in, Representative by The number of samples in a batch consisting of 10 dialogues. Represents the number of sessions Indicates the first The number of conversations in a session Indicates the L2 regularization weights. Represents the set of trainable parameters. and These represent the weighting coefficients of the loss function. For batch labels The number of It is the first In the first session The truth value label of a statement, Is with A set of positive feature samples with the same sample label. For the first In the first session Output features of a fully connected layer for each discourse; , Except A set of features other than those in the batch. This represents the adaptive sample weighting function. , These are sample weight parameters that control the degree of attention given to the minority class. To adjust the adjustment factor for sample weights, This is the lower bound of the sample weight parameter. To control the temperature coefficient of the distance between samples, It forces the model to focus on the focusing parameters of samples that are difficult to classify. satisfy , , , , , Represents the mathematical expectation. It is the first In the first session The probability distribution of each utterance in terms of sentiment.

Images