Emotion recognition method and system based on multi-modal dialogue

Abstract

The invention provides an emotion recognition method and system based on a multi-modal dialogue, and aims to solve the problems of unreasonable information flow, insufficient high-order dependency capture among modals, weak feature emotion correlation and the like in the prior art. The method comprises the steps of multi-modal feature extraction, VA-driven contrast learning, self-adaptive directional hypergraph construction, dynamic hypergraph information processing and sentiment classification. A sample pair optimization feature space is constructed through VA scores, a directed hypergraph is constructed to define the information flow direction, hyperedge weights are dynamically adjusted through multi-head attention and position coding, and efficient multi-modal information fusion is achieved. The system comprises a multi-mode encoder, VA contrast learning, a self-adaptive directional hypergraph and a classifier module. Experiments show that the accuracy and the weighted F1 score of the method on data sets such as IEMOCAP and MELD are superior to those in the prior art, and the method is suitable for scenes such as intelligent customer service and social media analysis.

Description

Technical Field

[0001] This invention belongs to the field of multimodal emotion recognition technology, specifically relating to an emotion recognition method and system based on multimodal dialogue. Background Technology

[0002] In multi-turn dialogue scenarios, emotion recognition enhances the naturalness and effectiveness of human-computer interaction, and has wide applications in areas such as social media analytics, intelligent customer service, and mental health monitoring. However, due to the diversity of information sources, the diversity of emotions, and the strong dependence on context and speaker state, emotion recognition remains a complex task.

[0003] Although deep learning-based methods have made significant progress in emotion recognition tasks in recent years, multimodal dialogue emotion recognition faces numerous challenges. On the one hand, emotional information in dialogue not only encompasses multiple modalities such as text, audio, and vision, but also involves complex interactions and high-order dependencies among these modalities, requiring accurate capture and modeling. On the other hand, dialogue exhibits a clear temporal sequence; the emotion of the current utterance is often influenced by previous utterances, and existing technologies struggle to ensure the rationality of information flow when handling this temporal dependency, easily leading to problems such as information contamination. Furthermore, how to extract common features between modalities while preserving the specific information of each modality during the fusion of multimodal information is also a pressing issue that needs to be addressed.

[0004] Therefore, it is of great significance to propose an emotion recognition method and system based on multimodal dialogue. Summary of the Invention

[0005] The purpose of this invention is to provide a method and system for emotion recognition based on multimodal dialogue, in order to solve the problems of unreasonable information flow, insufficient capture of high-order dependencies between modalities, and weak correlation of feature emotions in the existing technology, thereby improving the accuracy and robustness of multimodal dialogue emotion recognition.

[0006] In a first aspect, the present invention proposes an emotion recognition method based on multimodal dialogue, which includes the following steps:

[0007] Multimodal feature extraction steps: The input raw multimodal data are encoded separately, and the speaker information and the context information of the dialogue are combined to generate the single modal feature representations corresponding to each modality. The raw multimodal data includes text modality, audio modality and visual modality.

[0008] VA-driven contrastive learning steps: For the features of each modality, positive and negative sample pairs are constructed based on valence and arousal scores. The single-modality feature space is optimized by the contrastive loss function so that samples with similar emotions are closer in the feature space.

[0009] Adaptive directional hypergraph construction steps: Construct two types of directed hypergraphs, including temporal hyperedges and multimodal hyperedges. The temporal hyperedges connect different utterance nodes of the same modality in a unidirectional manner to ensure the unidirectionality of temporal dependencies. The multimodal hyperedges connect different modal nodes of the same utterance in a bidirectional manner to integrate cross-modal synchronization information.

[0010] Dynamic hypergraph information processing steps: The weights of the temporal and multimodal hyperedges are dynamically adjusted using a multi-head attention mechanism and positional encoding, combined with residual connections and broadcasting mechanisms, to achieve directional or bidirectional information propagation;

[0011] Emotion classification steps: Input the multimodal features fused from the above steps into the classifier and output the emotion recognition results of the dialogue.

[0012] Preferably, in the VA-driven contrastive learning step, the positive sample pair and the negative sample pair are determined as follows: the Euclidean distance of the VA scores between the sample pairs is calculated. If the distance is less than a preset threshold δ, the sample pair is determined as a positive sample pair; if the distance is greater than or equal to the preset threshold δ, the sample pair is determined as a negative sample pair.

[0013] Preferably, the adaptive directional hypergraph construction step includes: given a speech sequence containing N rounds of dialogue, constructing a directed hypergraph G = (U, E), where each node u ∈ U represents a unimodal speech and text. audio or visual Each directed hyperedge h∈E captures the dependencies between the mode and the context, specifically including:

[0014] The time hyperedge represents each node Where x∈{t,a,v} is connected to all other nodes of the same modality in the dialogue. It only connects past and current discourse nodes within the same modality, and information is transmitted strictly in the direction from the past to the present;

[0015] The multimodal hyperedge represents each node. Connect to the corresponding modal node from the same speech. It connects nodes of different modalities corresponding to the same sentence, supporting bidirectional information exchange between nodes.

[0016] Preferably, the dynamic hypergraph information processing step further includes:

[0017] Hyperedge aggregation sub-step: Using a multi-head attention mechanism, the weights between nodes and hyperedges are dynamically calculated, and positional encoding is introduced to capture the relative positional information between nodes;

[0018] Information broadcasting sub-step: For each node i in the hypergraph, distinguish the contribution of the hyperedges that appear from i as the head (receiving information) and the tail (transmitting information). Merge the information of the head hyperedges and the tail hyperedges through residual connections. Support information propagation using either directional propagation mode or bidirectional propagation mode. The directional propagation mode only uses the information of the head hyperedges, while the bidirectional propagation mode merges the information of the head hyperedges and the tail hyperedges.

[0019] More preferably, in the dynamic hypergraph information processing step, the attention coefficient between node i and hyperedge h in every C attention heads is calculated using the following formula:

[0020]

[0021] Among them, b c and Q c Let X be the trainable parameters of the c-th head. i Let y be the feature vector of node i. h Let h be the feature vector of the hyperedge. MLP represents a multilayer perceptron, LeakyReLU is the activation function, and the concatenation operator || combines the node and hyperedge features together.

[0022] Secondly, embodiments of the present invention provide an emotion recognition system based on multimodal dialogue, the system comprising:

[0023] Multimodal encoder module: configured to encode the input raw multimodal data separately, and combine speaker information and dialogue context information to generate single-modal feature representations corresponding to each modality. The raw multimodal data includes text, audio and visual effects.

[0024] VA Contrastive Learning Module: Configured to construct positive and negative sample pairs based on Valence and Arousal scores for features of each modality. The unimodal feature space is optimized by a contrastive loss function, so that samples with similar emotions are closer together in the feature space.

[0025] Adaptive Directed Hypergraph Module: Configures the construction of two types of directed hypergraphs: those containing temporal hyperedges and those containing multimodal hyperedges. This module includes:

[0026] The dynamic weight adjustment unit dynamically adjusts the weights of the temporal and multimodal hyperedges based on a multi-head attention mechanism and positional encoding.

[0027] The information dissemination unit supports information dissemination in either a directional or bidirectional manner, integrating information from both the head and tail super-edges.

[0028] Classifier module: Configured to process the fused multimodal features and output the emotion recognition results of the dialogue.

[0029] Preferably, the VA contrastive learning module includes:

[0030] Valence (VA) score calculation unit: Obtains the Valence and Arousal scores for each modality of data through a pre-trained model;

[0031] Similarity measurement unit: positive and negative sample pairs are determined based on the distance of VA scores;

[0032] Contrastive loss optimization unit: minimizes the feature distance between positive sample pairs while maximizing the feature distance between negative sample pairs.

[0033] Preferably, the adaptive orientation hypergraph module further includes:

[0034] Temporal hyperedge building unit: connects past and current discourse nodes of the same modality in a unidirectional manner;

[0035] Multimodal hyperedge building blocks: connect nodes corresponding to different modalities of the same sentence in a bidirectional manner;

[0036] Residual fusion unit: Preserves original features and integrates information from the hypergraph propagation process through residual connections.

[0037] Thirdly, embodiments of the present invention provide an electronic device, including: one or more processors; and a storage device for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement the method described in any implementation of the first aspect.

[0038] Fourthly, embodiments of the present invention provide a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method as described in any of the implementations of the first aspect.

[0039] Compared with the prior art, the beneficial results of the present invention are as follows:

[0040] To address the shortcomings of existing methods, such as insufficient information fusion and inadequate capture of complex intermodal dependencies, this invention proposes an adaptive directional hypergraph framework, ADH-VA, which solves the problems of insufficient information fusion and complex dependencies in multimodal dialogue emotion recognition. It integrates VA-driven contrastive learning (which optimizes the discriminability of single-modal features) and ADHConv (which captures complex intermodal relationships), improving classification accuracy and robustness. Experiments on IEMOCAP and MELD demonstrate that this method significantly outperforms state-of-the-art methods, validating its effectiveness in complex dialogue environments. Future work may explore applications in other modalities and the real world to enhance the comprehensiveness of emotion recognition. Attached Figure Description

[0041] The accompanying drawings are included to provide a further understanding of the embodiments and are incorporated in and constitute a part of this specification. The drawings illustrate embodiments and, together with the description, serve to explain the principles of the invention. Other embodiments and many anticipated advantages of the embodiments will be readily recognized as they become better understood through reference to the following detailed description. Elements in the drawings are not necessarily to scale. The same reference numerals refer to corresponding similar parts.

[0042] Figure 1 This is an example diagram illustrating multimodal dialogue emotion recognition implemented using an adaptive directed hypergraph framework, according to an embodiment of the present invention.

[0043] Figure 2 This is a flowchart illustrating the emotion recognition method based on multimodal dialogue, as described in an embodiment of the present invention.

[0044] Figure 3 This is a schematic diagram of the architecture of ADH-VA according to a specific embodiment of the present invention;

[0045] Figure 4 This is a schematic diagram illustrating the sensitivity analysis of IEMOCAP and MELD according to an embodiment of the present invention;

[0046] Figure 5 This is a schematic diagram of the architecture of an emotion recognition system based on multimodal dialogue, according to an embodiment of the present invention.

[0047] Figure 6 This is a schematic diagram of the structure of a computer device suitable for implementing electronic devices according to embodiments of the present invention. Detailed Implementation

[0048] The present invention will now be described in further detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and not intended to limit it. Furthermore, it should be noted that, for ease of description, only the parts relevant to the invention are shown in the accompanying drawings.

[0049] It should be noted that, unless otherwise specified, the embodiments and features described in the present invention can be combined with each other. The present invention will now be described in detail with reference to the accompanying drawings and embodiments.

[0050] In multi-turn dialogue scenarios, emotion recognition enhances the naturalness and effectiveness of human-computer interaction, and has wide applications in areas such as social media analytics, intelligent customer service, and mental health monitoring. However, due to the diversity of information sources, the diversity of emotions, and the strong dependence on context and speaker state, emotion recognition remains a complex task.

[0051] In recent years, deep learning-based methods have made significant progress in emotion recognition tasks. For example, BC-LSTM, DialogueRNN, and DialogueCRN use independent bidirectional LSTM networks to capture contextual information from different modalities, effectively integrate speaker modeling, context modeling, and emotion modeling using gated recurrent units, and introduce inference modules to simulate human emotional reasoning processes, further enhancing the model's emotion understanding capabilities. Furthermore, in emotion computing, multidimensional valence and arousal have been widely used, compared to discrete categories, for a more nuanced understanding of emotions. Multiple datasets, such as IEMOCAP and MER2023, enable research to explore the connections between discrete and dimensional emotions through multi-task learning. UniVA incorporates contrastive valence (VA) into contrastive learning to capture individual modal emotional states and emotional correlations between samples.

[0052] While the aforementioned methods perform well in unimodal or simple multimodal emotion recognition tasks, they still face challenges in more complex MERC tasks, such as insufficient information fusion and difficulty in capturing intermodal dependencies. Graph-based multimodal methods, such as DialogueGCN and MMGCN, capture intermodal contextual relationships by constructing graph structures, but they are still insufficient in handling higher-order relationships and complex dependencies. To address these issues, recent research methods such as DIMMN, MM-DFN, GraphMFT, and COGMEN introduce multi-view networks and graph attention mechanisms, further enhancing the fusion of multimodal information and the discriminability of emotion features. M3NET introduces hypergraphs into MERC, achieving significant improvements in prediction accuracy and time efficiency through multiple hypergraph convolutions using a simple fully connected structure and randomly initialized edge weights. HAUCL further reduces information redundancy by dynamically adjusting hypergraph connections through a variational hypergraph autoencoder. However, due to the diverse information sources, complex emotion types, and strong dependence on context and speaker state, it still faces many challenges.

[0053] To address the aforementioned issues in existing hypergraph-based methods, this invention proposes an ensemble framework called ADH-VA, which combines VA-driven contrastive learning with Adaptive Oriented Hypergraph Convolution (ADHConv) to achieve more efficient and accurate MERC. Specifically, we optimize single-modal representations through VA-driven contrastive learning, making sentiment-similar samples closer in the feature space. Meanwhile, as... Figure 1 As shown, ADHConv transitions from a traditional hypergraph to a directed hypergraph, effectively preventing future utterances from influencing current utterances and ensuring a more accurate temporal flow of information. The introduction of an adaptive module further optimizes the framework by preventing redundancy and oversmoothing in the graph network, thus maintaining the integrity of sentiment features. Therefore, ADHConv is able to capture complex relationships and high-order dependencies between modalities, while addressing the challenge of balancing intra-speaker and inter-speaker contextual dependencies during modal interactions. Experimental results demonstrate that the proposed method outperforms state-of-the-art methods on two widely used MERC datasets, IEMOCAP and MELD, validating its effectiveness and superiority in multimodal emotion recognition tasks.

[0054] It should be noted that in the VA-driven approach involved in the embodiments of this invention, VA typically refers to valence and arousal, two important dimensions in psychology for describing emotions. Valence represents the positive or negative nature of an emotion, such as pleasure or sadness; arousal represents the degree of activation of an emotion, such as calm or excitement. "VA-driven" means that in related technologies or systems, information from these two dimensions, valence and arousal, plays a crucial guiding and driving role. ADHConv, or Adaptive Directed Hypergraph Convolution, is the core graph neural network module for multimodal dialogue emotion recognition in the ADH-VA framework proposed in this invention. It combines directed hypergraphs and adaptive weighting mechanisms, enabling effective modeling of cross-modal, higher-order dependencies, and temporal information in dialogue.

[0055] In this embodiment, VA-driven contrastive learning is combined with Adaptive Directed Hypergraph Convolution (ADHConv). VA-driven contrastive learning optimizes unimodal representations, while ADHConv captures complex relationships and higher-order dependencies between modalities. Working together, they improve the fusion of multimodal information and the performance of emotion recognition. Furthermore, it can be combined with various feature extraction techniques and neural network architectures. During feature extraction, features from different modalities are weighted or filtered based on VA scores to highlight emotion-related features. During neural network training, VA information is incorporated into the calculation of the loss function or network layers, guiding the network to learn representations that better reflect VA emotion characteristics.

[0056] Firstly, Figure 2 The embodiment of the present invention discloses an emotion recognition method based on multimodal dialogue, such as... Figure 2 As shown, the method includes the following steps:

[0057] S1. Multimodal feature extraction steps: Encode the input raw multimodal data separately, and combine the speaker information and the context information of the dialogue to generate the single-modal feature representation corresponding to each modality. The raw multimodal data includes text modality, audio modality and visual modality.

[0058] S2, VA-driven contrastive learning steps: For the features of each modality, positive and negative sample pairs are constructed based on valence and arousal scores. The single-modality feature space is optimized by the contrastive loss function so that samples with similar emotions are closer in the feature space.

[0059] Specifically, in this VA-driven contrastive learning step, the positive and negative sample pairs are determined as follows:

[0060] Calculate the Euclidean distance of the VA scores between sample pairs. If the distance is less than a preset threshold δ, the sample pair is identified as a positive sample pair; if the distance is greater than or equal to the preset threshold δ, the sample pair is identified as a negative sample pair.

[0061] S3. Adaptive Directed Hypergraph Construction Steps: Construct two types of directed hypergraphs, including temporal hyperedges and multimodal hyperedges. The temporal hyperedges connect different utterance nodes of the same modality in a unidirectional manner to ensure the unidirectionality of temporal dependencies. The multimodal hyperedges connect different modal nodes of the same utterance in a bidirectional manner to integrate cross-modal synchronization information.

[0062] Specifically, the adaptive oriented hypergraph construction step includes:

[0063] Given a speech sequence containing N rounds of dialogue, construct a directed hypergraph G = (U, E), where each node u ∈ U represents a unimodal speech and text. audio or visual Each directed hyperedge h∈E captures the dependencies between the mode and the context, specifically including:

[0064] Time superedge represents each node Where X∈{T,a,v} is connected to all other nodes of the same modality in the dialogue. It only connects past and current discourse nodes within the same modality, and information is transmitted strictly in the direction from the past to the present;

[0065] Multimodal hyperedge representation of each node Connect to the corresponding modal node from the same speech. It connects nodes of different modalities corresponding to the same sentence, supporting bidirectional information exchange between nodes.

[0066] S4. Dynamic Hypergraph Information Processing Steps: The weights of the temporal and multimodal hyperedges are dynamically adjusted using a multi-head attention mechanism and positional encoding, combined with residual connections and broadcasting mechanisms, to achieve directional or bidirectional information propagation.

[0067] Specifically, the dynamic hypergraph information processing steps also include:

[0068] Hyperedge aggregation sub-step: Using a multi-head attention mechanism, the weights between nodes and hyperedges are dynamically calculated, and positional encoding is introduced to capture the relative positional information between nodes;

[0069] Information broadcasting sub-step: For each node i in the hypergraph, distinguish the contribution of the hyperedges that appear from i as the head (receiving information) and the tail (transmitting information). Merge the information of the head hyperedges and the tail hyperedges through residual connections. Support information propagation using either directional propagation mode or bidirectional propagation mode. The directional propagation mode only uses the information of the head hyperedges, while the bidirectional propagation mode merges the information of the head hyperedges and the tail hyperedges.

[0070] More preferably, in this dynamic hypergraph information processing step, the attention coefficient between node i and hyperedge h in each C attention heads is calculated using the following formula:

[0071]

[0072] Among them, b c and Q c Let x be the trainable parameters of the c-th head. i Let y be the feature vector of node i. h Let h be the feature vector of the hyperedge. MLP represents a multilayer perceptron, LeakyReLU is the activation function, and the concatenation operator || combines the node and hyperedge features together.

[0073] S5. Sentiment Classification Step: Input the multimodal features fused from the above steps into the classifier and output the sentiment recognition results of the dialogue.

[0074] As a specific embodiment, the ADH-VA framework proposed in this invention combines VA-driven contrastive learning with ADHConv for use in MERC, such as... Figure 3 As shown.

[0075] The method proposed in this invention first extracts features from raw multimodal data (text, audio, and visual effects), and combines speaker and contextual information to generate a context-aware unimodal representation. Then, VA-driven contrastive learning aligns these features in the embedding space. Next, ADHConv involves two steps—directional hyperedge aggregation and broadcasting—both supporting unidirectional and bidirectional modes. Aggregation uses multi-head attention, nonlinear transformations, and positional encoding, while broadcasting employs a direction-based weighting strategy to guide the information flow. Finally, the integrated features are subjected to additional convolutions and fed into a classifier for emotion recognition. Detailed components and processes will be explained in the following sections.

[0076] Data preprocessing: In the unimodal coding process, the raw input features of each modality are first processed, and then combined with speaker and context information to obtain a unimodal representation that can simultaneously perceive the speaker and context. For each utterance (u i ) and its multi-label annotations (y i We obtain the VA score through the following steps:

[0077] Text Modality: We used a fine-tuned RoBERTa base model to perform inference on the EmoBank dataset to obtain the VA score of the text, with values ​​ranging from (-1, 1).

[0078] Visual modality: We use the EmoFAN model to extract VA scores from facial expressions, and average the scores across the entire facial sequence to obtain the VA score for the visual modality. and

[0079] Audio modality: We used the Wav2Vec2-Large-Robust model, fine-tuned on the MSP-Podcast dataset, to obtain the VA score for the audio modality. and

[0080] VA-driven contrastive learning: This step discloses a VA-driven contrastive learning method aimed at improving the quality of unimodal representations using contrastive learning.

[0081] For the text modality, we use fully connected layers (FC) combined with an additional attention mechanism (AddAtt) to obtain sentence-level representations. For the visual and audio modalities, we use self-attention layers (SAT) to process input features, and then use an additional attention mechanism to capture modality-specific sentence-level representations. Specifically, for each modality (m∈{t,v,a}), we obtain unimodal representations (Ht, H ... m ),in and It is a coding feature.

[0082] To construct positive and negative sample pairs, we utilize the VA score for each modality as a supervision signal. These VA scores enable us to compute the sentiment proximity between samples, thus addressing the challenge of multi-label sentiment recognition in dialogue-based multimodal emotion recognition (MERC) tasks. For each sample pair... and We calculate their Euclidean distance in VA space:

[0083]

[0084] Based on this distance, we set a threshold (δ) to determine whether two samples are semantically similar (positive pair) or different (negative pair). If the distance (d < δ), the samples form a positive pair; otherwise, they form a negative pair. To avoid an imbalance where all pairs are negative, we replicate the unimodal representation to create a multi-view perspective, thus generating multiple unimodal representations for each sample.

[0085] Then, for each anchor sample (x) i We compute a VA-driven contrastive loss, aiming to maximize the similarity between the anchor and its positive samples while minimizing the similarity with negative samples. For each modality (m∈{t,v,a}), we obtain the encoded feature vector of sample (i). For clarity, we define: That is, the L2 normalized feature vector of sample i in mode m; That is, the anchor point x contains both positive and negative samples. i The candidate set.

[0086] The contrastive loss function is defined as follows:

[0087]

[0088] in Indicates the relationship with anchor point x i This forms a set of all corresponding samples. The similarity function is calculated as follows:

[0089]

[0090] This method determines positive samples based on sentiment proximity in the VA space, rather than solely relying on identical sentiment labels, thereby improving the model's ability to capture subtle sentiment relationships. Negative samples are selected based on significant differences in VA scores, thus enhancing the discriminative and generalization abilities of the unimodal representation.

[0091] Adaptive Directed Hypergraph Convolution: The ADHConv framework encodes high-order information of directed hypergraphs, flexibly handles the directional information of directed hyperedges, and enhances the performance and generalization ability of the model by improving the expressive power of the attention mechanism, normalization and regularization methods, and the design of the information ensemble encoder.

[0092] Directed Hypergraph Construction: Given a speech sequence containing N rounds of dialogue, we construct a directed hypergraph G = (U, E), where each node u ∈ U represents a single-modal speech (text) segment. Audio or visual Each directed hyperedge h∈E captures the dependencies between modalities and context. The hyperedge design assumes that sentiment in a dialogue is influenced by multimodal and contextual factors, with complex relationships between these dimensions. To model these dependencies, we define two types of directed hyperedges:

[0093] Time superedge: Each node (where x∈{t,a,v}) connects to all other nodes of the same modality in the dialogue. These directed connections represent time dependencies, ensuring that information flows from the past to the future and preventing any backward effects.

[0094] Multimodal hyperedge: Each node Connect to the corresponding modal node from the same speech. This allows the model to integrate cross-modal information at each time step, thereby improving emotion recognition capabilities.

[0095] These directed hyperedges capture multimodal interactions and temporal dependencies, with temporal hyperedges forcing the correct flow of information. This directed hypergraph structure facilitates the realization of rich, high-order feature representations, which are then processed through hypergraph convolutions for further analysis.

[0096] Directed Hyperedge Information Aggregation: In previous studies of hypergraph convolution, the process of aggregating node information through hyperedges did not explicitly consider the individual weights between each node and the hyperedge. We propose an attention-based hyperedge information aggregation method to dynamically adjust the weights between nodes and hyperedges, thereby capturing more complex relationships.

[0097] To enhance the expressive power of the attention mechanism, we introduce multi-head attention, nonlinear transformation, and positional encoding. Multi-head attention allows the model to capture various relationships in parallel across different subspaces, thus significantly improving its expressive power. Specifically, we extend the attention mechanism to C attention heads, each with independent trainable parameters b. c and Q c .

[0098] Furthermore, to enhance the model's ability to model complex relationships, we introduce a multilayer perceptron (MLP) into the attention computation to increase the depth and complexity of the nonlinear transformation. Position encoding is used to capture the relative position and orientation of nodes within directed hyperedges, further enhancing the attention mechanism's understanding of directed structures.

[0099] In each of the C attention heads, the attention coefficient between node i and hyperedge h is calculated as follows:

[0100]

[0101] Where b c and Q c The trainable parameters for the c-th head are x. i It is a feature of node i, and y h This is a feature related to the hyperedge h. Here, the join operator || combines the node and hyperedge features together.

[0102] Then, the standardized attention coefficient is defined as:

[0103]

[0104] in SE(i,h) represents the set of nodes associated with the hyperedge h, while SE(i,h) represents the relative position encoding between node i and hyperedge h.

[0105] Then use the obtained attention coefficient Calculate the weighted sum of node features for each hyperedge:

[0106]

[0107] These aggregated hyperedge representations (obtained for each attention head) are concatenated and used as input to subsequent convolutional layers to update node representations.

[0108] Directed Hyperedge Information Broadcasting: To address the potential information loss issue in traditional directed graph propagation methods, we introduce a novel approach that assigns weights to information propagation based on the direction of the directed edges. This method ensures that a controlled proportion of information is delivered to the tail node during propagation along the directed edges, thereby enhancing the stability and robustness of the propagation process. To further stabilize propagation, batch normalization and Dropout regularization are applied in the integration module.

[0109] Specifically, for each node i in the hypergraph, we distinguish the contributions from the hyperedges that appear at i as the head (i.e., receiving information) and the tail (i.e., transmitting information). Formally, the aggregate features of the head and tail hyperedges are calculated as follows:

[0110]

[0111] in and Let i represent the head and tail superedges of the set associated with node i.

[0112] Then, the node embedding at layer t+1 is updated using residual connections, as shown below:

[0113]

[0114] The function g(·) serves as the information integration encoder, fusing contributions from both the head and tail components. This integration is adjusted according to the desired propagation strategy.

[0115] Directed propagation: In situations where a unidirectional (e.g., past to future) information flow must be enforced to avoid future information contaminating the present, g(·) aims to utilize only the header information. In this case, we define: Here, φ(·) can be implemented as a multilayer perceptron (MLP) or an attention-based module, and can be regularly regularized using batch normalization and Dropout.

[0116] Two-way propagation: In other cases, utilizing incoming (head) and outgoing (tail) information helps capture complementary cues. Here, g(·) integrates two information sources:

[0117]

[0118] Here, || denotes feature connectivity, and φ(·) represents a directed hypergraph information integration encoder that learns to adaptively weight and fuse the two components. Similarly, normalization and dropout layers are added to φ(·) to enhance stability.

[0119] Residual connection The inclusion of this feature ensures that the original node features are preserved and combined with the newly integrated information, which helps to capture local and aggregate contextual cues across layers. Overall, this directed hyperedge information broadcasting mechanism dynamically controls the information flow according to edge directionality, thereby improving the robustness of multimodal feature fusion in complex dialogue emotion recognition tasks.

[0120] Information integration and convolution: In the framework of this embodiment, after aggregating hyperedge information through broadcasting, we further refine the node embedding by combining the aggregated features with learnable transformations and dynamic fusion weights.

[0121] First, we define an intermediate aggregation result: Used for directional (one-way) propagation. Used for two-way communication.

[0122] To further integrate and refine the information in the bidirectional case, we introduce a learnable transformation matrix G. head and G tail The fusion weights are dynamically adjusted using parameters α and β. We define the normalized weights as:

[0123]

[0124] Then, the final node feature update of the bidirectional aggregation is given by the following formula:

[0125] F (t+1) (B)=η·G ha ·H (t+1) (B)+(1-η)·G ti ·H (t+1) (B)+s,

[0126] Where s is the integral term that captures long-term context dependencies. This term is recursively updated as follows:

[0127] s←γs+(1-γ)·(η·G ha ·H (t+1) (B)+(1-η)·G ti ·H (t+1) (B)),

[0128] Here, γ is a decay factor that controls the rate at which s accumulates new information.

[0129] Finally, the fused node embeddings obtained from text, audio, and visual modalities are connected and subjected to sentiment classification using a multilayer perceptron (MLP) to produce the final predicted label.

[0130] The embodiments of the present invention will be further illustrated below with reference to specific experiments.

[0131] In this embodiment, we use two widely used multimodal datasets in MERC to evaluate ADH-VA: IEMOCAP and MELD. We referenced the previous study HAUCL for data partitioning and augmentation.

[0132] The IEMOCAP database contains 151 binary dialogues (7433 utterances) recorded by the USC SAIL lab. It includes audiovisual data (video, speech, facial motion capture, text transcription, and torso / head movements) and labels for six emotions (happy, sad, neutral, angry, excited, frustrated), as well as dimensional annotations. We used 120 dialogues (5810 utterances) for training and validation, and 31 dialogues (1623 utterances) for testing, focusing on three modalities: verbal, video, and audio.

[0133] The MELD dataset is an extension of EmotionLines, containing 1433 multi-party dialogues (13708 sentences) from the TV show *Friends*. Each sentence is labeled with one of seven emotions and sentiment labels (positive, negative, or neutral). The sentiment distribution in this dataset is unbalanced, with neutral labels making up a large proportion. For experiments, we used 1153 dialogues (11098 sentences) for training and validation, and 280 dialogues (2610 sentences) for testing, employing three modes: verbal, video, and audio.

[0134] During training, we used four NVIDIA RTX 4090 GPUs to train the model. Experiments were conducted on the IEMOCAP and MELD datasets, using a batch size of 16, the Adam optimizer (learning rate 1e-4), a hidden dimension Dh = 512, dropout rates of 0.5 and 0.4, and δ = 0.1 and λ, respectively. cl =0.1 and 1.

[0135] Regarding the loss function, we use a combination of classification loss and contrastive loss to optimize model performance. Furthermore, to prevent overfitting, we introduce an L2 regularization term. The specific loss function is defined as follows:

[0136]

[0137] in, The total number of utterances is represented by S; the number of dialogues is represented by c(s); the number of utterances in dialogue s is represented by P. s,i,j It is the predicted probability of the i-th utterance in dialogue s for the emotion category j; y s,i,j λ is the true label of the j-th emotion in the i-th utterance of dialogue s; λ is the regularization weight. It is the comparison in equation 2 The weight.

[0138] To evaluate the performance of the proposed method in MERC tasks, we compared the performance of the proposed model (OURS) with several state-of-the-art (SOTA) methods on two widely used MERC datasets (IEMOCAP and MELD). The comparisons included: (1) non-graph learning: BC-LSTM, DialogueRNN, and DialogueCRN; (2) standard graph learning: DialogueGCN, MMGCN, DIMMN, MMDFN, COGMEN, and GraphMFT; and (3) hypergraph learning: M3NET and HAUCL. For evaluation, we used the most popular evaluation metrics in the field: accuracy (Acc.) and weighted F1 score (WF1).

[0139] As shown in Table 1, the ADH-VA model outperforms several state-of-the-art methods on both the IEMOCAP and MELD datasets, demonstrating its advantages in multimodal emotion recognition tasks. On the IEMOCAP dataset, the ADH-VA model achieves state-of-the-art performance across multiple emotion categories and overall evaluation metrics (1.58%, 1.19%), proving its effectiveness in handling high-dimensional multimodal data and complex emotion categories. On the MELD dataset, despite the imbalanced emotion distribution, the ADH-VA model maintains high accuracy and weighted F1 score, showcasing its potential for emotion recognition in real-world multi-party dialogues. Compared to other methods, the ADH-VA model effectively enhances multimodal information fusion and emotion feature discrimination by introducing ADHConv and VA-driven contrastive learning, further advancing research in the MERC field. These results fully validate the effectiveness and superiority of our proposed model architecture in multimodal dialogue emotion recognition, laying a solid foundation for future research and applications.

[0140] Table 1. Performance of various methods (bold indicates best performance)

[0141]

[0142] Sensitivity analysis: such as Figure 4 As shown, we investigated four key hyperparameters in the framework: (1) learning rate (LR), (2) contrastive loss weights. (3) Determine the threshold δ and (4) the dropout rate in the single-modal encoder. For each parameter, we fix all other parameters at their optimal values.

[0143] The learning rate (LR) controls the step size of parameter updates. An excessively large LR can lead to unstable training, while an excessively small LR will slow down the convergence speed.

[0144] In the top-left sub-image (IEMOCAP), change LR from (5×10) -5 Increase to (1×10) -4 This can significantly improve accuracy, but further increases (e.g., 2×10) are necessary. -4 This will lead to a decrease in performance.

[0145] Similarly, in the bottom left subplot (MELD), the best performance is also in medium LR (1×10). -4 The results were achieved with higher values, while lower or higher values ​​performed poorly, indicating that a balanced LR is crucial for stable training on both datasets.

[0146] Comparison of loss weights This weight balances the classification loss and the contrast loss.

[0147] Looking at the previous second subplot (IEMOCAP), a medium value (e.g., 0.1) yields the highest accuracy, while a larger weight (e.g., 1.0) overemphasizes the target being compared.

[0148] In contrast, the next second subplot (MELD) shows that the higher The optimal value yields the best results, suggesting that the imbalanced nature of MELD may benefit from a stronger contrast signal.

[0149] A threshold (δ) is used to determine whether two samples form a pair based on the VA distance. A smaller δ may produce too few pairs, while a larger δ may introduce noisy pairs.

[0150] The top third of the subplot (IEMOCAP) shows that δ = 0.1 effectively balances the quantity and quality of pairs, achieving optimal accuracy.

[0151] A similar trend was observed in the lower third of the subgraph (MELD), where the same threshold also led to optimal performance.

[0152] Finally, the dropout rate regularizes the unimodal encoder. Too much dropout will discard too many features, while insufficient dropout risks overfitting.

[0153] As can be seen from the top right subplot (IEMOCAP), a medium dropout rate (e.g., 0.5) achieves the highest accuracy. Lower or higher dropout rates will degrade performance.

[0154] In contrast, the bottom right subplot (MELD) shows a slightly lower optimal dropout (e.g., 0.4), reflecting dataset-specific differences.

[0155] Furthermore, to verify the contribution of VA-driven contrastive learning and ADHConv to the overall model performance, we conducted extensive ablation experiments, and the results are shown in Table 2.

[0156] Table 2. Performance of various methods on the IEMOCAP and MELD datasets

[0157]

[0158] Impact of Directed Hypergraphs: Model 1: Remove the function with the adaptive hypergraph structure. Model 2: Replace the directed hypergraph structure with an undirected hypergraph. Model 3: Replace the directed hypergraph structure in the ADHConv module with a standard directed graph. Model 4: Remove the hyperedge weight parameter and set all hyperedge weights to the same value. Model 5: Remove the node weight parameter and set all node weights to the same value. Model 6: Remove both hyperedge and node weight parameters and set all weights to the same value.

[0159] Experimental results show that ADHConv, with its directional, adaptive, and optimized hypergraph structure, significantly improves the performance of multimodal emotion recognition. Removing any key components (such as the adaptive mechanism, orientation information, hyperedge weights, or node weights) leads to performance degradation, thus validating the importance of these components in the model.

[0160] Compared to standard directed graphs, directed hypergraphs can more effectively capture higher-order relationships and complex dependencies, thereby further improving the accuracy and robustness of the model's emotion recognition.

[0161] Impact of VA contrastive learning: Model 7: VA loss function removed from video modality. Model 8: VA loss function removed from audio modality. Model 9: VA loss function removed from text modality. Model 10: VA loss function removed from both video and audio modality. Model 11: VA loss function removed from both video and text modality. Model 12: VA loss function removed from both text and audio modality. Model 13: VA loss function removed from video, text, and audio modality.

[0162] The results of the single-modal ablation experiments show that removing the text modality has the most significant impact on VALoss, followed by the acoustic modality, while the visual modality has the least impact. This is consistent with expectations, as the text modality provides the highest information entropy in multi-turn dialogue emotion recognition tasks, potentially including some signals present in the visual and acoustic modalities. However, the results of the combined ablation experiments show that simultaneously removing both the text and audio modalities leads to greater information loss. This is because these two modalities capture most of the emotion signals, while the visual modality mainly serves as supplementary information to these two modalities. Nevertheless, the presence of visual signals still significantly contributes to the model's performance, as can be seen from the experimental results of Model 7.

[0163] The experimental results show that VA-driven contrastive learning optimizes unimodal feature representations using VA scores, significantly improving the overall performance of multimodal emotion recognition. Removing any modality's VA loss function leads to a performance degradation, indicating that these components play a crucial role in enhancing feature discriminability and model generalization.

[0164] Comprehensive ablation experiments demonstrate that the information provided by all modalities proposed in this embodiment of the invention is crucial for improving model performance.

[0165] In summary, this invention proposes ADH-VA, a framework to address insufficient information fusion and complex dependencies in multimodal dialogue emotion recognition. It integrates VA-driven contrastive learning (optimizing single-modal feature discriminability) and ADHConv (capturing complex intermodal relationships), improving classification accuracy and robustness. Experiments on IEMOCAP and MELD demonstrate that this method significantly outperforms state-of-the-art methods, validating its effectiveness in complex dialogue environments. Future work may explore applications in other modalities and the real world to enhance the comprehensiveness of emotion recognition.

[0166] Further reference Figure 5 As an implementation of the methods shown in the above figures, this application provides an embodiment of an emotion recognition system based on multimodal dialogue, which is similar to... Figure 2 Corresponding to the method embodiments shown, the system can be specifically applied to various electronic devices.

[0167] Secondly, embodiments of the present invention also provide an emotion recognition system based on multimodal dialogue, such as... Figure 5 As shown, the system includes: a multimodal encoder module 51, a VA contrastive learning module 52, an adaptive orientation hypergraph module 53, and a classifier module 54.

[0168] In one specific embodiment, the multimodal encoder module 51 is configured to encode the input raw multimodal data separately, and combine speaker information and dialogue context information to generate unimodal feature representations corresponding to each modality. The raw multimodal data includes text, audio, and visual effects. The VA contrastive learning module 52 is configured to construct positive and negative sample pairs based on valence and arousal scores for the features of each modality, and optimize the unimodal feature space through a contrastive loss function so that samples with similar emotions are closer in the feature space.

[0169] Adaptive Directed Hypergraph Module 53: Configured for constructing two types of directed hypergraphs, including temporal hyperedges and multimodal hyperedges. This module includes: a dynamic weight adjustment unit 531, which dynamically adjusts the weights of the temporal hyperedges and multimodal hyperedges based on a multi-head attention mechanism and position encoding; and an information propagation unit 532, which supports information propagation using directional or bidirectional propagation methods and integrates information from head and tail hyperedges.

[0170] Classifier module 54: Configured to process the fused multimodal features and output the emotion recognition results of the dialogue.

[0171] Preferably, the VA contrastive learning module 52 includes: a VA score calculation unit 521: obtaining the valence and arousal scores of each modality data through a pre-trained model; a similarity measurement unit 522: determining positive and negative sample pairs based on the distance of the VA scores; and a contrastive loss optimization unit 523: minimizing the feature distance between positive sample pairs while maximizing the feature distance between negative sample pairs.

[0172] Preferably, the adaptive orientation hypergraph module 53 further includes: a temporal hyperedge construction unit 533, which connects past utterance nodes and current utterance nodes of the same modality in a unidirectional manner; a multimodal hyperedge construction unit 534, which connects nodes of different modalities corresponding to the same utterance in a bidirectional manner; and a residual fusion unit 535, which preserves the original features and integrates information in the hypergraph propagation process through residual connections.

[0173] The functions and methods of the above modules correspond to each other, and will not be repeated here.

[0174] The main contributions of the embodiments of the present invention are as follows:

[0175] 1. An ADHConv module was designed to effectively encode complex relationships and high-order dependencies between modalities, improving the model's ability to capture sentiment features and its classification performance.

[0176] 2. An integrated multimodal emotion recognition framework is proposed, which combines VA-driven contrastive learning and ADHConv to effectively enhance the fusion of multimodal information and the discriminability of emotion features.

[0177] 3. Comprehensive experimental evaluations on two challenging MERC datasets demonstrate the advantages of the proposed method in handling complex dialogue scenarios and multimodal information fusion.

[0178] In practical applications, in intelligent customer service scenarios, the models of the methods and systems disclosed in the embodiments of this invention are integrated into the customer service system to identify the emotions in user conversations in real time, so that customer service personnel can adjust service strategies in a timely manner and improve customer satisfaction. In social media analysis, emotion recognition is performed on the text, videos and other content posted by users to help companies understand users' attitudes toward products or services and provide a basis for market decisions. In the field of mental health monitoring, by analyzing the emotions in users' conversations, abnormal user emotions can be detected in a timely manner, providing support for mental health intervention.

[0179] The following is for reference. Figure 6 It shows a schematic diagram of the structure of a computer device 600 suitable for implementing an electronic device according to embodiments of the present invention. Figure 6The electronic device shown is merely an example and should not be construed as limiting the functionality and scope of the embodiments of the present invention.

[0180] like Figure 6 As shown, the computer device 600 includes a central processing unit (CPU) 601 and a graphics processing unit (GPU) 602, which can perform various appropriate actions and processes according to programs stored in read-only memory (ROM) 603 or programs loaded from storage section 609 into random access memory (RAM) 604. The RAM 604 also stores various programs and data required for the operation of the device 600. The CPU 601, GPU 602, ROM 603, and RAM 604 are interconnected via a bus 605. An input / output (I / O) interface 606 is also connected to the bus 605.

[0181] The following components are connected to I / O interface 606: an input section 607 including a keyboard, mouse, etc.; an output section 608 including an LCD, speakers, etc.; a storage section 609 including a hard disk, etc.; and a communication section 610 including a network interface card, such as a LAN card or modem. The communication section 610 performs communication processing via a network such as the Internet. A drive 611 may also be connected to I / O interface 606 as needed. A removable medium 612, such as a hard disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on drive 611 as needed so that computer programs read from it can be installed into storage section 609 as needed.

[0182] In particular, according to embodiments disclosed in this invention, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments disclosed in this invention include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication section 610, and / or installed from removable medium 612. When the computer program is executed by central processing unit (CPU) 601 and graphics processing unit (GPU) 602, the functions defined in the methods of this invention are performed.

[0183] It should be noted that the computer-readable medium described in this invention can be a computer-readable signal medium, a computer-readable medium, or any combination thereof. A computer-readable medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor device, apparatus, or any combination thereof. More specific examples of a computer-readable medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this invention, a computer-readable medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution apparatus, device, or device. In this invention, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. Computer-readable signal media can also be any computer-readable medium other than a computer-readable medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution device, apparatus, or apparatus. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wireless, wire, optical fiber, RF, etc., or any suitable combination thereof.

[0184] Computer program code for performing the operations of this invention can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, as well as conventional procedural programming languages ​​such as "C" or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0185] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of apparatus, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using dedicated hardware-based devices that perform the specified functions or operations, or using a combination of dedicated hardware and computer instructions.

[0186] The modules described in the embodiments of the present invention can be implemented in software or hardware. The described modules can also be located in a processor.

[0187] In another aspect, the present invention also provides a computer-readable medium, which may be included in the electronic device described in the above embodiments; or it may exist independently and not assembled into the electronic device. The computer-readable medium carries one or more programs that, when executed by the electronic device, cause the electronic device to perform the methods and steps described in the first aspect.

[0188] The above description is merely a preferred embodiment of the present invention and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of the invention is not limited to the specific combination of the above-described technical features, but also includes other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the inventive concept. For example, technical solutions formed by substituting the above-described features with (but not limited to) technical features with similar functions disclosed in this invention.

Claims

1. An emotion recognition method based on multimodal dialogue, characterized in that, The method includes the following steps: Multimodal feature extraction steps: The input raw multimodal data are encoded separately, and the speaker information and the context information of the dialogue are combined to generate the single modal feature representations corresponding to each modality. The raw multimodal data includes text modality, audio modality and visual modality. VA-driven contrastive learning steps: For the features of each modality, positive and negative sample pairs are constructed based on valence and arousal scores. The single-modality feature space is optimized by the contrastive loss function so that samples with similar emotions are closer in the feature space. Adaptive directional hypergraph construction steps: Construct two types of directed hypergraphs, including temporal hyperedges and multimodal hyperedges. The temporal hyperedges connect different utterance nodes of the same modality in a unidirectional manner to ensure the unidirectionality of temporal dependencies. The multimodal hyperedges connect different modal nodes of the same utterance in a bidirectional manner to integrate cross-modal synchronization information. Dynamic hypergraph information processing steps: The weights of the temporal and multimodal hyperedges are dynamically adjusted using a multi-head attention mechanism and positional encoding, combined with residual connections and broadcasting mechanisms, to achieve directional or bidirectional information propagation; Emotion classification steps: Input the multimodal features fused from the above steps into the classifier and output the emotion recognition results of the dialogue.

2. The emotion recognition method based on multimodal dialogue according to claim 1, characterized in that, In the VA-driven contrastive learning step, the positive and negative sample pairs are determined as follows: Calculate the Euclidean distance of the VA scores between sample pairs. If the distance is less than a preset threshold δ, then the sample pair is determined as a positive sample pair. If the distance is greater than or equal to the preset threshold δ, then the sample pair is determined as a negative sample pair.

3. The emotion recognition method based on multimodal dialogue according to claim 1, characterized in that, The adaptive directional hypergraph construction step includes: given a speech sequence containing N rounds of dialogue, constructing a directed hypergraph G = (U, E), where each node u ∈ U represents a unimodal speech and text. audio or visual Each directed hyperedge h∈E captures the dependencies between the mode and the context, specifically including: The time hyperedge represents each node Where x∈{t,a,v} is connected to all other nodes of the same modality in the dialogue. It only connects past and current discourse nodes within the same modality, and information is transmitted strictly in the direction from the past to the present; The multimodal hyperedge represents each node. Connect to the corresponding modal node from the same speech. It connects nodes of different modalities corresponding to the same sentence, supporting bidirectional information exchange between nodes.

4. The emotion recognition method based on multimodal dialogue according to claim 1, characterized in that, The dynamic hypergraph information processing steps specifically also include: Hyperedge aggregation sub-step: Using a multi-head attention mechanism, the weights between nodes and hyperedges are dynamically calculated, and positional encoding is introduced to capture the relative positional information between nodes; Information broadcasting sub-step: For each node i in the hypergraph, distinguish the contribution of the hyperedges that appear from i as the head (receiving information) and the tail (transmitting information). Merge the information of the head hyperedges and the tail hyperedges through residual connections. Support information propagation using either directional propagation mode or bidirectional propagation mode. The directional propagation mode only uses the information of the head hyperedges, while the bidirectional propagation mode merges the information of the head hyperedges and the tail hyperedges.

5. The emotion recognition method based on multimodal dialogue according to claim 4, characterized in that, In the dynamic hypergraph information processing step, the attention coefficient between node i and hyperedge h in each C attention heads is calculated using the following formula: Among them, b c and Q c Let x be the trainable parameters of the c-th head. i Let y be the feature vector of node i. h Let h be the feature vector of the hyperedge. MLP represents a multilayer perceptron, LeakyReLU is the activation function, and the concatenation operator || combines the node and hyperedge features together.

6. An emotion recognition system based on multimodal dialogue, characterized in that, The system includes: Multimodal encoder module: configured to encode the input raw multimodal data separately, and combine speaker information and dialogue context information to generate single-modal feature representations corresponding to each modality. The raw multimodal data includes text, audio and visual effects. VA Contrastive Learning Module: Configured to construct positive and negative sample pairs based on Valence and Arousal scores for features of each modality. The unimodal feature space is optimized by a contrastive loss function, so that samples with similar emotions are closer together in the feature space. Adaptive Directed Hypergraph Module: Configures the construction of two types of directed hypergraphs: those containing temporal hyperedges and those containing multimodal hyperedges. This module includes: The dynamic weight adjustment unit dynamically adjusts the weights of the temporal and multimodal hyperedges based on a multi-head attention mechanism and positional encoding. The information dissemination unit supports information dissemination in either a directional or bidirectional manner, integrating information from both the head and tail super-edges. Classifier module: Configured to process the fused multimodal features and output the emotion recognition results of the dialogue.

7. The emotion recognition system based on multimodal dialogue according to claim 6, characterized in that, The VA contrastive learning module includes: Valence (VA) score calculation unit: Obtains the Valence and Arousal scores for each modality of data through a pre-trained model; Similarity measurement unit: positive and negative sample pairs are determined based on the distance of VA scores; Contrastive loss optimization unit: minimizes the feature distance between positive sample pairs while maximizing the feature distance between negative sample pairs.

8. The emotion recognition system based on multimodal dialogue according to claim 6, characterized in that, The adaptive orientation hypergraph module also includes: Temporal hyperedge building unit: connects past and current discourse nodes of the same modality in a unidirectional manner; Multimodal hyperedge building blocks: connect nodes corresponding to different modalities of the same sentence in a bidirectional manner; Residual fusion unit: Preserves original features and integrates information from the hypergraph propagation process through residual connections.

9. An electronic device, comprising: One or more processors; Storage device for storing one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the emotion recognition method based on multimodal dialogue as described in any one of claims 1 to 5.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the program implements the emotion recognition method based on multimodal dialogue as described in any one of claims 1 to 5.

Images