A conversational emotion recognition method based on multi-scale temporal features and cross-modal fusion

Abstract

This invention discloses a conversational emotion recognition method based on multi-scale temporal features and cross-modal fusion. Preliminary modal features obtained through processing with RoBERTa, 3D-CNN, and OpenSMILE models are input into a multi-scale temporal feature processing module. This module uses Bi-LSTM and Bi-GRU with different intermediate layers to process single-modal information. Next, a multi-modal interaction fusion module is used to interactively fuse the obtained features. This module mainly uses a cross-attention mechanism, taking one modality as the primary modality and fusing information from other modalities to obtain three fused modalities. The three modalities are then used in a hierarchical manner for interactive fusion, rotating among them as the primary modality. The resulting three feature information are then used for modal fusion using a graph constructed based on speaker identity for emotion classification. Furthermore, by concatenating the three preliminary modal features and speaker features, and processing them with MTFP and CIF to obtain fused features, multi-modal emotion recognition is assisted.

Description

Technical Field

[0001] This invention relates to the interdisciplinary field of multimodal fusion and emotion recognition, specifically to a conversational emotion recognition method based on multi-scale temporal features and cross-modal fusion, applicable to the recognition of emotional states in dialogue processes in scenarios such as healthcare and human-computer interaction. Background Technology

[0002] Multimodal emotion recognition (MERC) is a core task at the intersection of artificial intelligence, natural language processing, computer vision, and speech signal processing. It aims to identify the emotional state of speakers in a conversation by fusing information from text, audio, and visual modalities, and has wide applications in healthcare, human-computer interaction, and other scenarios. The core challenges of this task include: noise interference within the modalities (such as ambient sounds in audio and irrelevant background information in visuals); data heterogeneity between modalities (significant differences in feature dimensions and semantic expressions between different modalities); and the difficulty in modeling the dynamic interaction of speaker emotions (a speaker's emotions in a dialogue are influenced by both their own state and the other party's feedback, exhibiting explicit relationships and potential semantic dependencies).

[0003] The technology mainly revolves around three core aspects: modal feature extraction, cross-modal fusion, and speaker interaction modeling. In modal feature extraction, pre-trained models (such as language models and convolutional neural networks) are often used to extract single-modal features. For cross-modal fusion, strategies such as early fusion, late fusion, and attention mechanism fusion have been developed. In speaker interaction modeling, methods are mainly divided into two categories: sequence modeling based on recurrent units (such as LSTM and GRU) and relation modeling based on graph neural networks (GCN). Existing modal fusion methods have many shortcomings, specifically in the following aspects:

[0004] (1) The existing unified modal processing method lacks specificity and has poor intramodal noise filtering effect, thus reducing the quality of modal features; (2) When processing noise information at a single scale, it is difficult to capture the emotional dependence between discourses, thus reducing the intelligence of recognition; (3) The existing cross-modal fusion mechanism is imperfect and it is difficult to eliminate the problem of modal heterogeneity, thus reducing the synergy of the mechanism; (4) The existing speaker interaction modeling is one-sided or overemphasizes explicit relationships or is difficult to capture long-term potential dependence; (5) The existing model has poor adaptability in class imbalance scenarios, thus reducing the accuracy and robustness of multimodal conversation emotion recognition. Summary of the Invention

[0005] The purpose of this invention is to provide a conversational emotion recognition method based on multi-scale temporal features and cross-modal fusion, in order to solve the following problems mentioned in the background technology: (1) the existing unified modal processing method lacks specificity and has poor intra-modal noise filtering effect, thereby reducing the quality of modal features; (2) when processing noise information at a single scale, it is difficult to capture the emotional dependence between discourses, thereby reducing the intelligence of recognition; (3) the existing cross-modal fusion mechanism is imperfect and it is difficult to eliminate the problem of modal heterogeneity, thereby reducing the synergy of the mechanism; (4) the existing speaker interaction modeling is one-sided or overemphasizes explicit relationships or is difficult to capture long-term potential dependence; (5) the existing model has poor adaptability in class imbalance scenarios, thereby reducing the accuracy and robustness of multimodal conversational emotion recognition.

[0006] To achieve the above objectives, the present invention provides the following technical solution: a conversational emotion recognition method based on multi-scale temporal features and cross-modal fusion, the method comprising the following steps:

[0007] Step 1: Collect multimodal sentiment data, including video, audio, text, and speaker information from conversations;

[0008] Step 2: Preprocess the multimodal sentiment data;

[0009] Step 3: Construct a fusion modality that integrates multimodal and speaker information. Based on the text, audio, visual, and speaker information obtained in Step 2, obtain the fusion modality features. Then, fuse the four types of information using a linear layer to obtain the desired fusion features. ,in This represents the fused modal features of the i-th dialogue, which are concatenated with the text features of the corresponding dialogue. Audio features Visual features and Corresponding speaker information features The fused modal features are then obtained through processing by a linear layer.

[0010] Step 4: Using the multi-scale temporal feature processing module MTFP, BiGRU and BiLSTM are selected to capture multi-scale temporal dynamic information for different modal characteristics.

[0011] Step 5: Through the cross-modal interaction fusion module CIF, fine-grained semantic alignment and information complementarity between modalities are achieved based on a hierarchical cross-attention mechanism;

[0012] Step 6: Combine speaker relationship graph modeling to fuse the three modalities to obtain fused features, and use MLP for dynamic learning;

[0013] Step 7: Utilize the fused features for sentiment classification and fusion modality-assisted classification to further enhance sentiment classification.

[0014] Preferably, step two includes:

[0015] Audio features are obtained by processing audio data using OpenSmile. 3D-CNN is used to process video data to obtain video features. ;

[0016] RoBERTa was used to process the text data to obtain text features. ;

[0017] Encode the speaker's identity to obtain speaker identity information. Corresponding session sequence There is a mapping relationship Let N represent the number of dialogues, M represent the number of speakers, and N represents the number of dialogues. In multimodal tasks, dialogues mainly include three modalities: text, audio, and visual, which can be formally represented as: , where a, t, and v represent audio, text, and visual features, respectively.

[0018] Preferably, step four includes:

[0019] By selecting BiLSTM as the temporal feature processor to process visual and audio features, irrelevant emotional information in the visual domain, including the speaker's facial features and environmental information, is removed. In the audio domain, timbre and ambient noise interference are also removed. BiLSTM effectively reduces noise information by reducing the dimensionality of the intermediate layers, extracting the core emotional information, which can be formalized as follows:

[0020]

[0021]

[0022] BiGRU is used as the temporal feature processor to process text features and fused features, in order to preserve more contextual details and improve the efficiency of long sequence modeling. This can be formalized as follows:

[0023]

[0024]

[0025] in, , where m represents the intermediate layer dimension of the temporal feature processor, and the corresponding processors capture temporal dynamic features of different complexities. This means that the features of the corresponding n intermediate layers are transformed into feature vectors with a unified dimension of 512.

[0026] Preferably, step five includes:

[0027] Construct a cross-attention module for queries ,key Sum Applying linear projection, we perform n different linear transformations, where n represents the number of long positions. These represent the sequence lengths of the query, key, and value, respectively, i.e., the number of elements in each sequence. The feature dimensions of queries, keys, and values, i.e., the number of dimensions for each element, are then defined. Represents one of the modes. To represent another mode, first process Modal data, using different linear variations of n to generate queries Then for another mode By applying n different linear transformations, generate and Formalized as:

[0028] ,

[0029] Then calculate the cross-fusion results:

[0030] ,

[0031] The process of multi-head cross-attention can be summarized as follows:

[0032]

[0033] Received for Modality fusion via multi-head cross-attention The output after modality is processed, and then multimodal interaction and fusion are performed.

[0034] Preferably, the hierarchical energy of the multimodal interaction fusion can be formalized as follows:

[0035] ,

[0036] in, This represents the feature representation after fusion. This represents the modal features obtained through initial step modal processing before fusion;

[0037] By using a multimodal interaction modality, the fused modality is used as the query in a cross-attention mechanism. Visual, audio, and text modalities are used as keys and values ​​for interaction, achieving cross-modal feature alignment and information supplementation, and generating fused features. This process can be represented as follows:

[0038]

[0039] in, This represents the feature representation after fusion. This represents the fusion modal features obtained through initial step modal processing before interactive fusion.

[0040] Preferably, step six includes:

[0041] By constructing a graph structure of speaker relationships in dialogue, semantic information in the dialogue is captured. A directed graph of speaker relationships is constructed using visual, audio, and text modal features. ,in Each node Features containing three modalities ,in To embed speaker information into text feature information, i.e. , Construct bidirectional edges for the speaker information corresponding to the dialogue. Represents a node With nodes The dialogue relationship between nodes, and the varying degrees of influence of different node sentiment on subsequent dialogues, are addressed using edge weights. Representing an edge The direct influence relationship between the two connected nodes.

[0042] Preferably, MLP is used to dynamically learn nodes. and nodes The correlation between them can be formalized as: ,in , These are the network parameters that are being learned. This represents a vector concatenation operation. Next, the softmax function is used to normalize the correlation between nodes, yielding the attention score for each edge. The formula is:

[0043]

[0044] in Represents a node First-order neighbor nodes, Point and nodes The correlation strength between nodes is determined, and finally, GCN encoding and the GELU activation function are used to update the node information representation, as shown in the formula:

[0045]

[0046] in the formula Represents first-order neighbor nodes In edge relations The following set, yes The model, It is a feature vector encoded by GCN.

[0047] Preferably, step seven includes:

[0048] Sentiment classification and auxiliary classification are performed on the fused features obtained from the speaker relationship graph. The fused features obtained from the speaker relationship graph are used to obtain the model's prediction results by applying the Softmax function.

[0049] Compared with the prior art, the beneficial effects of the present invention are:

[0050] 1. This invention provides a multi-scale temporal feature processing scheme that adapts to different modal characteristics. After BiLSTM multi-scale processing of visual and audio modalities, ambient noise and irrelevant background noise are effectively suppressed, and core emotional features are highlighted. As can be seen from the ablation experiment, after introducing only the MTFP module, the model's WAA on the IEMOCAP dataset increased from 70.32% to 71.08%, and WAF1 increased from 70.23% to 71.11%, verifying the effectiveness of noise filtering. While effectively filtering ambient noise and irrelevant background noise in audio and visual modalities, it retains the unique and effective emotional information of each modality and improves the quality of single-modal features.

[0051] 2. This invention designs a feature processing module that uses BiLSTM and BiGRU with different dimensional intermediate layers to process modal information, thereby capturing long-term emotional dependencies between discourses using single-modal information of varying complexity. Multi-scale processing effectively reduces noise information within a modality while retaining unique emotional information specific to certain modalities that is effective for emotion recognition.

[0052] 3. This invention constructs a fine-grained semantic alignment and closed-loop interaction fusion mode through a hierarchical cross-attention mechanism, making full use of the complementary information of text, audio and visual modalities, avoiding the introduction of redundant features during the fusion process, and achieving efficient collaboration of cross-modal information.

[0053] 4. This invention constructs a dual modeling mechanism of "speaker relationship graph + fusion modality assistance", which not only accurately captures the explicit emotional interaction relationship between speakers, but also effectively explores the potential semantic dependence and long-term emotional dynamic changes between discourses, so as to achieve comprehensive modeling of emotional interaction.

[0054] 5. This invention improves the feature representation ability of a few emotion categories by combining multimodal complementary enhancement and fusion modality-assisted classification, ensuring that the model has stable and excellent recognition performance in different emotion categories, especially for categories with few samples, thereby improving the overall accuracy and robustness of multimodal conversation emotion recognition. Attached Figure Description

[0055] Figure 1 This is a structural diagram of the model of the present invention;

[0056] Figure 2 This is a WAF1 diagram showing the training process of different modalities on IEMOCAP according to the present invention;

[0057] Figure 3 This is a WAA diagram of the different modal training processes on IEMOCAP according to the present invention;

[0058] Figure 4 This diagram illustrates the impact of different batch sizes on the model's performance in this invention. Detailed Implementation

[0059] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0060] Please see Figures 1-4 A conversational emotion recognition method based on multi-scale temporal features and cross-modal fusion includes the following steps:

[0061] Step 1: Collect multimodal sentiment data, including video, audio, text, and speaker information from conversations;

[0062] Step 2: Preprocess the multimodal sentiment data;

[0063] Step 3: Construct a fusion modality that integrates multimodal and speaker information. Based on the text, audio, visual, and speaker information obtained in Step 2, obtain the fusion modality features. Then, fuse the four types of information using a linear layer to obtain the desired fusion features. ,in This represents the fused modal features of the i-th dialogue, which are concatenated with the text features of the corresponding dialogue. Audio features Visual features and Corresponding speaker information features The fused modal features are then obtained through processing by a linear layer.

[0064] Step 4: Using the multi-scale temporal feature processing module MTFP, BiGRU and BiLSTM are selected to capture multi-scale temporal dynamic information for different modal characteristics.

[0065] Step 5: Through the cross-modal interaction fusion module CIF, fine-grained semantic alignment and information complementarity between modalities are achieved based on a hierarchical cross-attention mechanism;

[0066] Step 6: Combine speaker relationship graph modeling to fuse the three modalities to obtain fused features, and use MLP for dynamic learning;

[0067] Step 7: Utilize the fused features for sentiment classification and fusion modality-assisted classification to further enhance sentiment classification.

[0068] In this embodiment, step two includes:

[0069] Audio features are obtained by processing audio data using OpenSmile. 3D-CNN is used to process video data to obtain video features. ;

[0070] RoBERTa was used to process the text data to obtain text features. ;

[0071] Encode the speaker's identity to obtain speaker identity information. Corresponding session sequence There is a mapping relationship Let N represent the number of dialogues, M represent the number of speakers, and N represents the number of dialogues. In multimodal tasks, dialogues mainly include three modalities: text, audio, and visual, which can be formally represented as: , where a, t, and v represent audio, text, and visual features, respectively.

[0072] In this embodiment, step four includes:

[0073] By selecting BiLSTM as the temporal feature processor to process visual and audio features, irrelevant emotional information in the visual domain, including the speaker's facial features and environmental information, is removed. In the audio domain, timbre and ambient noise interference are also removed. BiLSTM effectively reduces noise information by reducing the dimensionality of the intermediate layers, extracting the core emotional information, which can be formalized as follows:

[0074]

[0075]

[0076] BiGRU is used as the temporal feature processor to process text features and fused features, in order to preserve more contextual details and improve the efficiency of long sequence modeling. This can be formalized as follows:

[0077]

[0078]

[0079] in, , where m represents the intermediate layer dimension of the temporal feature processor, and the corresponding processors capture temporal dynamic features of different complexities. This means that the features of the corresponding n intermediate layers are transformed into feature vectors with a unified dimension of 512.

[0080] In this embodiment, step five includes:

[0081] Construct a cross-attention module for queries ,key Sum Applying linear projection, we perform n different linear transformations, where n represents the number of long positions. These represent the sequence lengths of the query, key, and value, respectively, i.e., the number of elements in each sequence. The feature dimensions of queries, keys, and values, i.e., the number of dimensions for each element, are then defined. Represents one of the modes. To represent another mode, first process Modal data, using different linear variations of n to generate queries Then for another mode By applying n different linear transformations, generate and Formalized as:

[0082] ,

[0083] Then calculate the cross-fusion results:

[0084] ,

[0085] The process of multi-head cross-attention can be summarized as follows:

[0086]

[0087] Received for Modality fusion via multi-head cross-attention The output after modality is processed, and then multimodal interaction and fusion are performed.

[0088] In this embodiment, the hierarchical energy of multimodal interaction fusion can be formalized as follows:

[0089] ,

[0090] in, This represents the feature representation after fusion. This represents the modal features obtained through initial step modal processing before fusion;

[0091] By using a multimodal interaction modality, the fused modality is used as the query in a cross-attention mechanism. Visual, audio, and text modalities are used as keys and values ​​for interaction, achieving cross-modal feature alignment and information supplementation, and generating fused features. This process can be represented as follows:

[0092]

[0093] in, This represents the feature representation after fusion. This represents the fusion modal features obtained through initial step modal processing before interactive fusion.

[0094] In this embodiment, step six includes:

[0095] By constructing a graph structure of speaker relationships in dialogue, semantic information in the dialogue is captured. A directed graph of speaker relationships is constructed using visual, audio, and text modal features. ,in Each node Features containing three modalities ,in To embed speaker information into text feature information, i.e. , Construct bidirectional edges for the speaker information corresponding to the dialogue. Represents a node With nodes The dialogue relationship between nodes, and the varying degrees of influence of different node sentiment on subsequent dialogues, are addressed using edge weights. Representing an edge The direct influence relationship between the two connected nodes.

[0096] In this embodiment: MLP is used to dynamically learn nodes. and nodes The correlation between them can be formalized as: ,in , These are the network parameters that are being learned. This represents a vector concatenation operation. Next, the softmax function is used to normalize the correlation between nodes, yielding the attention score for each edge. The formula is:

[0097]

[0098] in Represents a node First-order neighbor nodes, Point and nodes The correlation strength between nodes is determined, and finally, GCN encoding and the GELU activation function are used to update the node information representation, as shown in the formula:

[0099]

[0100] in the formula Represents first-order neighbor nodes In edge relations The following set, yes The model, It is a feature vector encoded by GCN.

[0101] In this embodiment, step seven includes:

[0102] Sentiment classification and auxiliary classification are performed on the fused features obtained from the speaker relationship graph. The fused features obtained from the speaker relationship graph are used to obtain the model's prediction results by applying the Softmax function.

[0103] To eliminate intramodal noise and intermodal data heterogeneity, and effectively capture long-term potential emotional dependencies of irrelevant speakers and explicit complex emotional interactions related to speaker identity, this method proposes a multi-scale temporal feature processing and intermodal interaction fusion approach, namely MTFP-CIF. First, the preliminary modal features obtained through processing with RoBERTa, 3D-CNN, and OpenSMILE models are input into the multi-scale temporal feature processing module (MTFP). This module uses Bi-LSTM or Bi-GRU with different intermediate layers to process single-modal information. Second, the obtained features are interactively fused using the multimodal interaction fusion module (CIF). This module mainly uses a cross-attention mechanism, using one modality as the master modality and fusing information from other modalities to obtain three fused modalities. The three modalities are then used in a hierarchical manner, rotating among them as the master modality. Finally, the obtained three feature information are used to perform modal fusion using a graph constructed based on speaker identity for sentiment classification. In addition, by splicing the three initially obtained modal features and speaker features, and processing them through MTFP and CIF to obtain fused features, multimodal emotion recognition is assisted. The model structure is as follows: Figure 1 As shown.

[0104] This method addresses the core shortcomings of existing technologies through an innovative architecture of "multi-scale temporal feature processing + cross-modal interaction fusion + speaker relationship graph modeling + fusion modality-assisted classification". It has achieved significant technical results in multimodal conversational emotion recognition tasks. All results have been experimentally verified on the two benchmark datasets MELD and IEMOCAP, and the data are real and reliable.

[0105] ablation experiment

[0106] The impact of each module of the analysis model on the experimental results will be analyzed, mainly on the IEMOCAP dataset, as shown in Table 1.

[0107] Table 1. Ablation Experiment

[0108]

[0109] (1) Effective filtering of intramodal noise and significant improvement in single-modal feature quality.

[0110] By adapting the BiLSTM / BiGRU processor to different modal characteristics through the MTFP module and combining it with a multi-scale intermediate layer design, a precise balance between noise filtering and effective information preservation is achieved, significantly improving the quality of single-modal features.

[0111] After BiLSTM multi-scale processing of visual and audio modalities, ambient noise and irrelevant background noise are effectively suppressed, and core emotional features are highlighted. As can be seen from the ablation experiment, after introducing only the MTFP module, the model's WAA on the IEMOCAP dataset increased from 70.32% to 71.08%, and WAF1 increased from 70.23% to 71.11%, verifying the effectiveness of noise filtering.

[0112] After BiGRU multi-scale processing, the text and fusion modalities retain the semantic details of the context and avoid the loss of effective information. This results in a WAA of 67.10% for the single-modal recognition of the text modality, making it the core supporting modality for emotion recognition and laying a high-quality foundation for subsequent cross-modal fusion.

[0113] (2) Cross-modal heterogeneity is completely eliminated, and semantic alignment and information complementarity are efficiently achieved.

[0114] The method's CIF module overcomes the challenge of intermodal data heterogeneity through a hierarchical cross-attention mechanism and a closed-loop interaction mode, achieving fine-grained semantic alignment and deep collaboration of multimodal information.

[0115] The hierarchical cross-attention mechanism enables each modality to accurately capture complementary emotional cues from other modalities, avoiding the introduction of redundant features. Ablation experiments show that after adding CIF to the MTFP module, the model's WAA improved by 1.31% (from 71.08% to 72.39%), and WAF1 improved by 1.01% (from 71.11% to 72.12%), demonstrating the significant effect of heterogeneity elimination.

[0116] (3) The emotional interaction modeling is comprehensive and accurate, taking into account both explicit relationships and potential dependencies.

[0117] By employing a dual modeling mechanism of "speaker relationship diagram + fusion modality-assisted classification," the limitations of existing single-modeling technologies are overcome, enabling comprehensive capture of emotional interaction.

[0118] The speaker relationship graph accurately models the explicit emotional interactions between speakers, and GCN encoding effectively conveys the emotional influence between nodes. The fusion modality, after being processed by MTFP and CIF, supplements the potential semantic dependencies between utterances. The combination of the two makes the model perform well in capturing long-term emotional dynamics. For example, in the IEMOCAP dataset, the F1 score for the "excitement" category is 80.94%, and the F1 score for the "depression" category is 69.23%, both of which are better than all baseline models.

[0119] Ablation experiments verified the auxiliary value of the fusion modality. After adding the fusion modality, the model WAA improved from 72.39% to 73.10%, and WAF1 improved from 72.12% to 73.06%, demonstrating the strengthening effect of the dual modeling mechanism on capturing emotional interaction.

[0120] Modal importance

[0121] To investigate the contribution of different modalities to emotion recognition, modal ablation experiments were conducted on the IEMOCAP dataset (e.g., Figure 2 , Figure 3 As shown in the table, experiments were conducted on three single-modal models (text, vision, and audio) and a bimodal model combining text-visual, text-audio, and visual-audio modes, comparing the results with the comprehensive trimodal experimental model of this invention. The experimental results show that the text modality dominates emotion recognition, with its single-modal performance significantly outperforming the visual and audio modalities. In the fused bimodal model, the text-visual combination outperforms other bimodal combinations in most metrics, but still falls short of the single-modal text modality results, indicating insufficient ability of this model to fuse bimodal modes. However, the trimodal fusion model performs best overall, validating the necessity of multimodal complementary information. As shown in the table, the model's accuracy and F1 score exhibit a highly consistent trend, validating the reliability of the model evaluation.

[0122] The trimodal fusion performance far surpasses that of the combination of single-modal and dual-modal approaches. On the IEMOCAP dataset, the WAA of trimodal (avl) fusion reached 73.10% and WAF1 reached 73.06%, which is 6.00% and 6.38% higher than the best single-modal (text l) fusion, respectively, and 8.28% and 8.96% higher than the best dual-modal (lv) fusion, fully verifying the efficient utilization of multimodal complementary information.

[0123] It exhibits strong adaptability to imbalanced scenarios and excellent performance in recognizing a few emotion categories.

[0124] By leveraging multimodal complementary enhancements and optimizing the joint loss function, the model's adaptability in imbalanced scenarios is significantly improved, effectively addressing the pain point of low accuracy in identifying a few sentiment categories.

[0125] Table 2. Baseline model comparison experiment on MELD

[0126]

[0127] On the MELD dataset (which includes a few categories such as "fear" and "disgust"), our model achieved an F1 score of 23.81% for the "fear" category, far exceeding the 5.06% of the best baseline model AGF-IB; and an F1 score of 28.00% for the "disgust" category, while most baseline models had an F1 score close to 0 in this category, demonstrating a strong ability to identify a few categories, as shown in Table 2.

[0128] The model achieved an overall WAA of 66.39% and a WAF1 of 66.21% on the MELD dataset, which is 2.25 and 2.20 percentage points higher than the best baseline AGF-IB (WAA 64.14% and WAF1 64.01%), respectively, demonstrating stronger practicality in real-world scenarios with uneven class distribution.

[0129] Overall recognition performance is leading, with outstanding robustness and generalization ability.

[0130] Its overall performance on two authoritative benchmark datasets surpasses that of existing mainstream baseline models, demonstrating superior recognition accuracy, robustness, and generalization ability.

[0131] Table 3. Baseline model comparison experiment on IEMOCAP

[0132]

[0133] On the IEMOCAP dataset, the model achieved a WAA of 73.10% and a WAF1 of 73.06%, which are 2.64 and 2.70 percentage points higher than the current best baseline AGF-IB (WAA 70.46% and WAF1 70.36%), respectively. The model also achieved the best recognition performance in core categories such as "neutral", "excited", and "frustrated", as shown in Table 3.

[0134] On the IEMOCAP dataset, the model outperformed all baselines overall, and its performance fluctuation was less than 1.5% across experiments with different batch sizes (2-128), demonstrating the model's training stability and generalization ability, and its adaptability to application scenarios with varying data scales. Figure 4 As shown.

[0135] In summary, this invention, through targeted technical design, comprehensively solves the core defects of existing multimodal emotion recognition technologies, achieving breakthrough improvements in feature quality, cross-modal fusion, emotion interaction modeling, and class imbalance adaptation, providing a high-precision and robust emotion recognition solution for practical application scenarios such as healthcare and human-computer interaction.

[0136] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus.

[0137] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A conversational emotion recognition method based on multi-scale temporal features and cross-modal fusion, characterized in that: The method includes the following steps: Step 1: Collect multimodal sentiment data, including video, audio, text, and speaker information from conversations; Step 2: Preprocess the multimodal sentiment data; Step 3: Construct a fusion modality that integrates multimodal and speaker information. Based on the text, audio, visual, and speaker information obtained in Step 2, obtain the fusion modality features. Then, fuse the four types of information using a linear layer to obtain the desired fusion features. ,in This represents the fused modal features of the i-th dialogue, which are concatenated with the text features of the corresponding dialogue. Audio features Visual features and Corresponding speaker information features The fused modal features are then obtained through processing by a linear layer. Step 4: Using the multi-scale temporal feature processing module MTFP, BiGRU and BiLSTM are selected to capture multi-scale temporal dynamic information for different modal characteristics. Step 5: Through the cross-modal interaction fusion module CIF, fine-grained semantic alignment and information complementarity between modalities are achieved based on a hierarchical cross-attention mechanism; Step 6: Combine speaker relationship graph modeling to fuse the three modalities to obtain fused features, and use MLP for dynamic learning; Step 7: Utilize the fused features for sentiment classification and fusion modality-assisted classification to further enhance sentiment classification.

2. The conversational emotion recognition method based on multi-scale temporal features and cross-modal fusion according to claim 1, characterized in that: Step two includes: Audio features are obtained by processing audio data using OpenSmile. 3D-CNN is used to process video data to obtain video features. ; RoBERTa was used to process the text data to obtain text features. ; Encode the speaker's identity to obtain speaker identity information. Corresponding session sequence There is a mapping relationship Let N represent the number of dialogues, M represent the number of speakers, and N represents the number of dialogues. In multimodal tasks, dialogues mainly include three modalities: text, audio, and visual, which can be formally represented as: , where a, t, and v represent audio, text, and visual features, respectively.

3. The conversational emotion recognition method based on multi-scale temporal features and cross-modal fusion according to claim 1, characterized in that: Step four includes: By selecting BiLSTM as the temporal feature processor to process visual and audio features, irrelevant emotional information in the visual domain, including the speaker's facial features and environmental information, is removed. In the audio domain, timbre and ambient noise interference are also removed. BiLSTM effectively reduces noise information by reducing the dimensionality of the intermediate layers, extracting the core emotional information, which can be formalized as follows: BiGRU is used as the temporal feature processor to process text features and fused features, in order to preserve more contextual details and improve the efficiency of long sequence modeling. This can be formalized as follows: in, , where m represents the intermediate layer dimension of the temporal feature processor, and the corresponding processors capture temporal dynamic features of different complexities. This means that the features of the corresponding n intermediate layers are transformed into feature vectors with a unified dimension of 512.

4. The conversational emotion recognition method based on multi-scale temporal features and cross-modal fusion according to claim 1, characterized in that: Step five includes: Construct a cross-attention module for queries ,key Sum Applying linear projection, we perform n different linear transformations, where n represents the number of long positions. These represent the sequence lengths of the query, key, and value, respectively, i.e., the number of elements in each sequence. The feature dimensions of queries, keys, and values, i.e., the number of dimensions for each element, are then defined. Represents one of the modes. To represent another mode, first process Modal data, using different linear variations of n to generate queries Then for another mode By applying n different linear transformations, generate and Formalized as: ， Then calculate the cross-fusion results: ， The process of multi-head cross-attention can be summarized as follows: Received for Modality fusion via multi-head cross-attention The output after modality is processed, and then multimodal interaction and fusion are performed.

5. The conversational emotion recognition method based on multi-scale temporal features and cross-modal fusion according to claim 4, characterized in that: The hierarchical energy of the multimodal interaction fusion can be formalized as follows: ， in, This represents the feature representation after fusion. This represents the modal features obtained through initial step modal processing before fusion; By using a multimodal interaction modality, the fused modality is used as the query in a cross-attention mechanism. Visual, audio, and text modalities are used as keys and values ​​for interaction, achieving cross-modal feature alignment and information supplementation, and generating fused features. This process can be represented as follows: in, This represents the feature representation after fusion. This represents the fusion modal features obtained through initial step modal processing before interactive fusion.

6. The conversational emotion recognition method based on multi-scale temporal features and cross-modal fusion according to claim 1, characterized in that: Step six includes: By constructing a graph structure of speaker relationships in dialogue, semantic information in the dialogue is captured. A directed graph of speaker relationships is constructed using visual, audio, and text modal features. ,in Each node Features containing three modalities ,in To embed speaker information into text feature information, i.e. , Construct bidirectional edges for the speaker information corresponding to the dialogue. Represents a node With nodes The dialogue relationship between nodes, and the varying degrees of influence of different node sentiment on subsequent dialogues, are addressed using edge weights. Representing an edge The direct influence relationship between the two connected nodes.

7. The conversational emotion recognition method based on multi-scale temporal features and cross-modal fusion according to claim 1, characterized in that: Use MLP to dynamically learn nodes and nodes The correlation between them can be formalized as: ,in , These are the network parameters that are being learned. This represents a vector concatenation operation. Next, the softmax function is used to normalize the correlation between nodes, yielding the attention score for each edge. The formula is: in Represents a node First-order neighbor nodes, Point and nodes The correlation strength between nodes is determined, and finally, GCN encoding and the GELU activation function are used to update the node information representation, as shown in the formula: in the formula Represents first-order neighbor nodes In edge relations The following set, yes The model, It is a feature vector encoded by GCN.

8. The conversational emotion recognition method based on multi-scale temporal features and cross-modal fusion according to claim 1, characterized in that: Step seven includes: Sentiment classification and auxiliary classification are performed on the fused features obtained from the speaker relationship graph. The fused features obtained from the speaker relationship graph are used to obtain the model's prediction results by applying the Softmax function.

Images