Dialogue emotion reason analysis method and system based on multi-modal emotion calculation

Abstract

The invention relates to a dialogue emotion reason analysis method and system based on multi-modal emotion calculation. The method comprises the following steps: firstly, extracting modal features from multi-modal data containing a text modal, an audio modal and a video modal, extracting local context features, and outputting multi-modal local context dialogue features; mapping the multi-modal local context dialogue features into nodes and edges in a heterogeneous graph, and constructing a multi-modal heterogeneous graph; then performing multi-scale semantic fusion based on the multi-modal heterogeneous graph and an attention mechanism, and outputting an utterance vector containing emotion context embedding and reason context embedding; and finally, matching different utterance vectors to generate a triple analysis result containing an emotion category, an emotion utterance index and a reason utterance index. Compared with the prior art, the method has the advantages of high accuracy and the like.

Description

Technical Field

[0001] This invention relates to the field of information mining technology, and in particular to a method and system for analyzing the causes of emotions in dialogues based on multimodal emotion computing. Background Technology

[0002] Emotion is hailed as the grammar of human social life, serving as a crucial vehicle for information exchange, relationship maintenance, and ideological communication between people. Understanding emotion is essential for human survival, communication, and decision-making, and is of great significance to the progress of human society. In the field of affective computing, in addition to recognizing and understanding human emotions, accurately analyzing the reasons behind emotions is a key step in enhancing the empathic capabilities of affective computing models. By deeply understanding the triggers of emotions, affective computing systems can more accurately capture the complexity of human emotions, thereby achieving a more empathetic interactive experience. Emotion cause localization technology has wide applications in the fields of science and technology media, especially in affective data analysis, user behavior analysis, online public opinion analysis, and the recently emerging short video content analysis.

[0003] With the widespread adoption of social media and the extensive use of multimodal data (such as text, speech, and video) in dialogues, multimodal affective computing (MAC) can enhance emotional understanding capabilities compared to unimodal computing by effectively utilizing the synergistic complementarity of information from different modalities. In real-world dialogues, emotional expression is often multimodal, involving multiple aspects such as language, speech, and vision. However, current research on MAC largely focuses on emotion recognition and understanding, while research on the localization of emotion causes in multimodal scenarios is relatively scarce. This limitation means that unimodal models cannot deeply analyze the triggering mechanisms of emotions in complex interaction scenarios, restricting the breadth and depth of the application of affective computing systems. Therefore, how to integrate multimodal semantic information for accurate understanding of multi-scale emotion causes requires further in-depth research. Furthermore, due to the limitations of affective information capture technology and the difficulty of labeling, establishing a high-quality multimodal emotion cause localization dataset is one of the main challenges. However, existing multimodal emotion cause localization datasets are relatively scarce and suffer from the problem of limited data scenarios. This limitation not only makes it difficult to fully reflect the rich and diverse dialogue environments in internet social media but also restricts research on the generalization and robustness of models in complex scenarios. At present, there is an urgent need for a multimodal dialogue emotion causation analysis dataset that can cover various types of dialogue scenarios, in order to better support the development and practical application of related technologies.

[0004] While techniques for locating emotional causes in dialogue have yielded some results, most research is limited to unimodal text data and fails to fully consider the rich emotional expressions and complex interactions in multimodal dialogues. In reality, emotional expression is often not limited to language; nonverbal modalities such as tone of voice and facial expressions also play a crucial role in emotion transmission.

[0005] Therefore, current unimodal emotion cause localization technology cannot fully capture the interaction between emotion and cause in a conversation, especially in long conversations and complex situations, where the accuracy of emotion transmission and cause detection is insufficient. Summary of the Invention

[0006] The purpose of this invention is to overcome the shortcomings of the existing technology and provide a method and system for analyzing the causes of emotions in dialogue based on multimodal emotion computing.

[0007] The objective of this invention can be achieved through the following technical solutions:

[0008] According to one aspect of the present invention, a method for analyzing the causes of emotions in dialogue based on multimodal emotion computing is provided, the method comprising the following steps:

[0009] S1. Extract modal features and local context features from multimodal data containing text, audio, and video modalities, and output multimodal local context dialogue features;

[0010] S2. Map the multimodal local context dialogue features to nodes and edges in a heterogeneous graph to construct a multimodal heterogeneous graph;

[0011] S3. Based on multimodal heterogeneous graphs and attention mechanisms, multi-scale semantic fusion is performed to output discourse vectors containing emotional context embeddings and causal context embeddings.

[0012] S4. Match different discourse vectors to generate triplet analysis results containing emotion category, emotion discourse index, and cause discourse index.

[0013] As a preferred technical solution, the local contextual features of the text modality are obtained through masked multi-head self-attention encoding, the specific process of which is as follows:

[0014] H t =MHSA(E t M),

[0015] Among them, H t The text modality represents the local contextual features after masked multi-head self-attention encoding. MHSA is a masked multi-head self-attention mechanism. t M is the sequence representation of each discourse text modality, and M is the mask.

[0016] The local context features of the audio and video modalities are obtained based on the implicit feature layer alignment method of the recurrent encoder. The specific process is as follows:

[0017] E ′m =LN(E m +GRU(Em )),

[0018] H m =LN(E m +E ′m +FFN(E ′m )),

[0019] Among them, E ′m For intermediate features of a single modality after GRU encoding and LN, global information and temporal dependencies are fused, e m This is a sequence representation of a single mode. For a single modality, the local context features are obtained after residual fusion and LN, where m∈{a,v} is the modality identifier, LN represents layer normalization, and FFN represents the feedforward neural network.

[0020] As a preferred technical solution, the specific process of constructing a multimodal heterogeneous graph in S2 includes:

[0021] S21. Construct nodes for a multimodal heterogeneous graph. The nodes include four types of nodes: discourse supernodes, dialogue supernodes, emotion context nodes, and cause context nodes. Each discourse supernode contains multimodal local context dialogue features, and the dialogue supernode is initialized with the average pooling result of the multimodal local context dialogue features.

[0022] S22. Determine the edge types and the relationships between the nodes corresponding to each edge. Edge types include edges with the same speaker, edges with different speakers, global connection edges, emotional context connection edges, and causal context connection edges.

[0023] The relationships between the corresponding nodes include:

[0024] Same speaker edge: A one-way connection between two speech supernodes of the same speaker;

[0025] Different speaker edge: A one-way connection between two speech supernodes of different speakers;

[0026] Global connection edges: bidirectional connections between discourse supernodes and dialogue supernodes;

[0027] Emotional context connection edge: a one-way connection between a discourse supernode and its corresponding emotional context node;

[0028] Cause-Context Connection Edge: A one-way connection between a discourse supernode and its corresponding cause-context node;

[0029] S23. Establish constraints and construct a multimodal heterogeneous graph based on nodes, edges, and relationships between nodes.

[0030] As a preferred technical solution, the constraints established in S23 include temporal unidirectional constraints, context window constraints, correlation constraints, and global constraints, specifically:

[0031] Temporal unidirectional constraint: Only past time-time discourse supernodes are allowed to establish pointing connections to future time-time discourse supernodes; future time-time discourse supernodes are not allowed to construct pointing connections to past time-time discourse supernodes in reverse.

[0032] Context window constraint: Introducing a direct context window parameter K for a discourse. For any discourse, its direct context is the set of the first K discourses that have the same speaker attribute as that discourse. Only when other discourses belong to the direct context of that discourse can an edge connection relationship with that discourse be established with the same speaker edge or a different speaker edge. Among them, the same speaker edge is used to connect discourse supernodes with the same speaker attribute, and the different speaker edge is used to connect discourse supernodes with different speaker attributes, thereby limiting the redundant construction of invalid edges.

[0033] Association constraint: Each utterance is only connected to its corresponding emotion node and cause node by edge type of emotion context edge connection and cause context connection, and has no edge connection with the emotion nodes and cause nodes of other utterances;

[0034] Global constraint: All speech nodes are connected to dialogue nodes by bidirectional edges with global connections.

[0035] As a preferred technical solution, S3 uses meta-path level multimodal fusion when performing multi-scale semantic fusion based on multimodal heterogeneous graphs and attention mechanisms. The specific steps include:

[0036] S31. Constructing meta-paths based on multimodal heterogeneous graphs;

[0037] S32. Select the target node and use the graph attention mechanism to propagate features between nodes in the metapath to obtain the attention weights of neighboring nodes to the target node.

[0038] S33. Based on attention weights, aggregate the features of the target node based on meta-paths to obtain the emotion context embedding and the cause context embedding.

[0039] S34. Repeat S31-S33 until all nodes have been selected once, and output the discourse vector containing the emotion context embedding and the cause context embedding.

[0040] As a preferred technical solution, the meta-path constructed in S31 is specifically represented as follows:

[0041]

[0042] Where φ is the set of metapaths. For different node types v i and v j The corresponding edge type, Φ(v i v j ) represents v i Connected to v via edge type r′ j All paths. By using different meta-paths Φ, the semantic information of different modalities and scales in a multimodal heterogeneous graph can be revealed.

[0043] As a preferred technical solution, the calculation formulas for emotion context embedding and cause context embedding are as follows:

[0044]

[0045] in, For the neighbor v j For target node v i Attention weights, with larger values ​​indicating stronger semantic associations, H i H j and H k denoted as the node context features of different modalities under the metapath Φ, and σ is the activation function. Represents node v i (Including itself) neighbors based on meta-path Φ, a Φ It is the node-level attention vector of the meta-path Φ. For node v i Based on the characteristics of the metapath Φ; since Φ∈{Φ um Φ cg Φ fg The calculation results include and Embed the emotional context for the i-th utterance. Embedding the causal context for the i-th utterance, Φ um Φ cg and Φ fg These are the global meta-path, the emotion meta-path, and the causal meta-path, respectively.

[0046] As a preferred technical solution, the specific steps for matching different utterance vectors in S4 include:

[0047] S41. Input the emotion context embedding and the cause context embedding into the multilayer perceptron to obtain the emotion category prediction result and the cause clause prediction result;

[0048] S42. Select the emotional context embedding of the i-th utterance and the causal context embedding of the j-th utterance, and calculate the relative position encoding of the two utterances ji.

[0049] S43. Input the emotional context embedding of the i-th utterance, the causal context embedding of the j-th utterance, and the relative position encoding of the two utterances ji into the multilayer perceptron, and output the emotion-cause pair prediction result of the emotion of the i-th utterance and the cause of the j-th utterance. If the result is greater than the preset value, then output i as the emotional utterance index, j as the causal utterance index, and the emotion category prediction result of the i-th utterance as the analysis result.

[0050] As a preferred technical solution, the relative position encoding is constructed based on the Gaussian kernel function, and its specific formula is as follows:

[0051]

[0052] Where, r′ p For the dynamic embedding of the current relative position p; r j Based on position embedding, K p (j) is the Gaussian kernel function, describing the influence strength of relative position p on relative position j, where j∈{-P, ..., P} is one of all possible relative positions traversed, p is the currently focused relative position, and σ k These are the shape control parameters for the kernel function.

[0053] According to another aspect of the present invention, a dialogue emotion cause analysis system based on multimodal emotion computing is provided, the system comprising a multimodal dialogue feature extraction and alignment module, a multi-scale semantic fusion module, and an emotion cause matching module;

[0054] The multimodal dialogue feature extraction and alignment module includes a feature extraction submodule and a feature alignment submodule. The feature extraction submodule is used to extract features of each modality from multimodal data containing text modality, audio modality, and video modality, and to extract local contextual dialogue features of the text modality from the text modality features. The feature alignment submodule is used to align the audio modality features and video modality features, and to extract local contextual dialogue features of the audio modality and video modality from them.

[0055] The multi-scale semantic fusion module includes a multimodal heterogeneous graph construction submodule and a multi-scale semantic fusion submodule. The multimodal heterogeneous graph construction submodule maps multimodal local context dialogue features to nodes and edges in a heterogeneous graph to construct a multimodal heterogeneous graph. The multi-scale semantic fusion submodule performs multi-scale semantic fusion based on the multimodal heterogeneous graph and attention mechanism, and outputs a discourse vector containing emotion context embedding and causal context embedding.

[0056] The emotion cause matching module includes an emotion category prediction submodule and an emotion cause prediction submodule. The emotion category prediction submodule obtains emotion category prediction and cause clause prediction results through the MLP module based on emotion context embedding and cause context embedding. The emotion cause prediction submodule matches different discourse vectors to generate triple analysis results containing emotion category, emotion discourse index and cause discourse index.

[0057] Compared with the prior art, the present invention has the following beneficial effects:

[0058] 1. This invention fills a gap in current methods for analyzing the causes of emotions in multimodal dialogues in Chinese scenarios by extracting multimodal features, constructing a multimodal heterogeneous graph, performing multi-scale semantic fusion, and finally using utterance pairing to complete the emotion cause analysis. It explicitly models the emotional and causal context information in dialogues using a multimodal heterogeneous graph approach, making the extraction of emotion causes more accurate and complete. This addresses the problems of "complex emotion-cause relationships and difficulty in cross-modal semantic fusion" in multimodal dialogues. It achieves end-to-end analysis from raw multimodal data to emotion cause triples, ensuring the completeness and accuracy of emotion category recognition and causal pair mining, and providing a standardized technical path for sentiment computing in complex dialogue scenarios.

[0059] 2. In this invention, the characteristics of text, audio and video modalities are fully considered. For text modalities, masked multi-head self-attention (MHSA) is used to accurately capture local semantic associations within the discourse. For audio or video modalities, GRU and layer normalization are used to effectively align temporal dependent features.

[0060] 3. In this invention, during the multimodal fusion process, a heterogeneous graph attention network is employed to effectively fuse feature information from different modalities at both inter- and intra-discourse scales, thereby enhancing the model's performance in extracting emotional causes. By constructing four types of node systems and five edge types, the multimodal dialogue semantics are made explicit as a graph structure. Through nodes carrying multimodal features and edges encoding semantic associations, structured modeling of dialogue interaction relationships is achieved, providing a visual semantic network for cross-modal reasoning of emotional causes and significantly improving the interpretability of complex dialogue semantics. When constructing the heterogeneous graph, the complexity is precisely controlled through temporal unidirectionality, contextual windows, association-specific constraints, and global collaborative constraints. Temporal constraints ensure the temporal consistency of dialogue logic, preventing future discourses from influencing the past; window constraints focus on nearest-neighbor contextual associations; association constraints strengthen the specific semantic boundaries of emotions or causes, avoiding cross-discourse emotion or cause confusion; and global constraints ensure dialogue-level semantic propagation, allowing local discourse understanding to benefit from the global context. By constructing the heterogeneous graph through constraints, the accuracy and efficiency of emotion-cause association mining are significantly improved while reducing computational costs.

[0061] 4. In this invention, by introducing meta-paths and attention mechanisms, the semantics of multimodal heterogeneous graphs are decomposed into global, sentiment, and causal dimensions. Semantic association rules are defined through meta-paths, and graph attention dynamically learns node importance, achieving hierarchical extraction and fusion of multimodal semantics. This provides feature support for the generation of sentiment context embedding and causal context embedding, solving the problems of semantic ambiguity and weak cross-modal collaboration in traditional methods.

[0062] 5. This invention addresses the challenge of cross-discourse matching of emotion-cause pairs through semantic feature input, positional encoding, and MLP prediction. Combining multimodal semantic embedding and relative positional encoding significantly improves the accuracy of emotion-cause pairing and avoids misjudgments of distant, irrelevant discourse. This invention employs a Gaussian kernel function to dynamically encode relative positions and uses kernel-weighted basic positional embedding to enhance the semantic contribution of nearest-neighbor positions, resolving the temporal ambiguity of causal pairs in long dialogues. This makes emotion-cause pairing prediction more aligned with actual dialogue logic, improving the temporal rationality and accuracy of the pairing results. Attached Figure Description

[0063] Figure 1 This is a schematic diagram of the method steps for analyzing the causes of emotions in dialogue in this invention;

[0064] Figure 2 This is a schematic diagram of the workflow of the dialogue emotion cause analysis system in this invention;

[0065] Figure 3 This is an example diagram of a heterogeneous graph for inter-utterance fusion in the embodiment;

[0066] Figure 4 This is an example diagram of heterogeneous graphs for intra-discourse fusion in the embodiment;

[0067] Figure 5 This is a working example diagram of the dialogue emotion cause analysis method in the embodiment;

[0068] Figure 6 This is an example diagram of dialogue-based multimodal heterogeneous graph construction in the embodiment. Detailed Implementation

[0069] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0070] Compared to traditional text-based emotion causation localization tasks, multimodal dialogue emotion causation localization tasks not only need to identify emotions and causes in the text, but also need to integrate speech and visual information to provide a more comprehensive emotion understanding. For example, in customer service systems, user emotions are not only expressed through language, but emotional changes in speech and visual feedback also affect the system's perception of user emotions and their underlying causes. By introducing multimodal information, emotion causation localization models can more accurately extract emotion and cause pairs in a conversation, thereby improving the response quality of the dialogue agent.

[0071] Furthermore, emotions and causes in dialogue often manifest as multi-scale semantic cues, spanning different modalities and even exhibiting discontinuous or non-grammatical forms. Single phrases, non-lexical statements, and even silent visual signals can convey significant emotional information. Simultaneously, emotion and cause pairs in conversations are often geographically dispersed, spanning multiple clauses or different modalities, increasing the complexity of emotion transmission. Multimodal emotion cause localization not only needs to handle these multi-scale semantic cues but also must address synchronous or asynchronous emotional expressions across different modalities. Therefore, multimodal emotion cause localization has become a forward-looking and highly challenging research topic in the field of affective computing.

[0072] Example 1

[0073] In this embodiment, a dialogue emotion cause analysis method for the field of multimodal emotion computing is adopted, and the method steps are as follows: Figure 1 As shown, it specifically includes:

[0074] S1. Extract modal features and local context features from multimodal data containing text, audio, and video modalities, and output multimodal local context dialogue features;

[0075] S2. Map the multimodal local context dialogue features to nodes and edges in a heterogeneous graph to construct a multimodal heterogeneous graph;

[0076] S3. Based on multimodal heterogeneous graphs and attention mechanisms, multi-scale semantic fusion is performed to output discourse vectors containing emotional context embeddings and causal context embeddings.

[0077] S4. Match different discourse vectors to generate triplet analysis results containing emotion category, emotion discourse index, and cause discourse index.

[0078] The local contextual features of the text modality are obtained through masked multi-head self-attention encoding, and the specific process is as follows:

[0079] H t =MHSA(E t M),

[0080] Among them, H t The text modality represents the local contextual features after masked multi-head self-attention encoding. MHSA is a masked multi-head self-attention mechanism. t M is the sequence representation of each discourse text modality, and M is the mask.

[0081] The local context features of the audio and video modalities are obtained based on the implicit feature layer alignment method of the recurrent encoder. The specific process is as follows:

[0082] E ′m =LN(E m +GRU(E m )),

[0083] H m =LN(E m +E ′m +FFN(E′ m )),

[0084] Among them, E′ m For intermediate features of a single modality after GRU encoding and LN, global information and temporal dependencies are fused, e m This is a sequence representation of a single mode. For a single modality, the local context features are obtained after residual fusion and LN, where m∈{a,v} is the modality identifier, LN represents layer normalization, and FFN represents the feedforward neural network.

[0085] The specific process of constructing a multimodal heterogeneous graph in S2 includes:

[0086] S21. Construct nodes for a multimodal heterogeneous graph. The nodes include four types of nodes: discourse supernodes, dialogue supernodes, emotion context nodes, and cause context nodes. Each discourse supernode contains multimodal local context dialogue features, and the dialogue supernode is initialized with the average pooling result of the multimodal local context dialogue features.

[0087] S22. Determine the edge types and the relationships between the nodes corresponding to each edge. Edge types include edges with the same speaker, edges with different speakers, global connection edges, emotional context connection edges, and causal context connection edges.

[0088] The relationships between the corresponding nodes include:

[0089] Same speaker edge: A one-way connection between two speech supernodes of the same speaker;

[0090] Different speaker edge: A one-way connection between two speech supernodes of different speakers;

[0091] Global connection edges: bidirectional connections between discourse supernodes and dialogue supernodes;

[0092] Emotional context connection edge: a one-way connection between a discourse supernode and its corresponding emotional context node;

[0093] Cause-Context Connection Edge: A one-way connection between a discourse supernode and its corresponding cause-context node;

[0094] S23. Establish constraints and construct a multimodal heterogeneous graph based on nodes, edges, and relationships between nodes.

[0095] The constraints established in S23 include temporal unidirectional constraints, context window constraints, association constraints, and global constraints, specifically:

[0096] Temporal unidirectional constraint: Only past time-time discourse supernodes are allowed to establish pointing connections to future time-time discourse supernodes; future time-time discourse supernodes are not allowed to construct pointing connections to past time-time discourse supernodes in reverse.

[0097] Context window constraint: Introducing a direct context window parameter K for a discourse. For any discourse, its direct context is the set of the first K discourses that have the same speaker attribute as that discourse. Only when other discourses belong to the direct context of that discourse can an edge connection relationship with that discourse be established with the same speaker edge or a different speaker edge. Among them, the same speaker edge is used to connect discourse supernodes with the same speaker attribute, and the different speaker edge is used to connect discourse supernodes with different speaker attributes, thereby limiting the redundant construction of invalid edges.

[0098] Association constraint: Each utterance is only connected to its corresponding emotion node and cause node by edge type of emotion context edge connection and cause context connection, and has no edge connection with the emotion nodes and cause nodes of other utterances;

[0099] Global constraint: All speech nodes are connected to dialogue nodes by bidirectional edges with global connections.

[0100] In S3, when performing multi-scale semantic fusion based on multimodal heterogeneous graphs and attention mechanisms, meta-path-level multimodal fusion is used. The specific steps include:

[0101] S31. Constructing meta-paths based on multimodal heterogeneous graphs;

[0102] S32. Select the target node and use the graph attention mechanism to propagate features between nodes in the metapath to obtain the attention weights of neighboring nodes to the target node.

[0103] S33. Based on attention weights, aggregate the features of the target node based on meta-paths to obtain the emotion context embedding and the cause context embedding.

[0104] S34. Repeat S31-S33 until all nodes have been selected once, and output the discourse vector containing the emotion context embedding and the cause context embedding.

[0105] The meta-path constructed in S31 is specifically represented as follows:

[0106]

[0107] Where φ is the set of metapaths. For different node types v i and v j The corresponding edge type, Φ(v i v j ) represents v i Connected to v via edge type r′ j All paths. By using different meta-paths Φ, the semantic information of different modalities and scales in a multimodal heterogeneous graph can be revealed.

[0108] The formulas for calculating emotion context embedding and causal context embedding are as follows:

[0109]

[0110] in, For the neighbor v j For target node v i Attention weights are assigned, with larger values ​​indicating stronger semantic associations. σ represents the activation function. Represents node v i (Including itself) neighbors based on meta-path Φ, a φ It is the node-level attention vector of the meta-path Φ. For node v i Based on the characteristics of the metapath Φ; since Φ∈{Φ um Φ cg Φ fg The calculation results include and Embed the emotional context for the i-th utterance. Embedding the causal context for the i-th utterance, Φ um Φ cg and Φ fg These are the global meta-path, the emotion meta-path, and the causal meta-path, respectively.

[0111] The specific steps for matching different utterance vectors in S4 include:

[0112] S41. Input the emotion context embedding and the cause context embedding into the multilayer perceptron to obtain the emotion category prediction result and the cause clause prediction result;

[0113] S42. Select the emotional context embedding of the i-th utterance and the causal context embedding of the j-th utterance, and calculate the relative position encoding of the two utterances ji.

[0114] S43. Input the emotional context embedding of the i-th utterance, the causal context embedding of the j-th utterance, and the relative position encoding of the two utterances ji into the multilayer perceptron, and output the emotion-cause pair prediction result of the emotion of the i-th utterance and the cause of the j-th utterance. If the result is greater than the preset value, then output i as the emotional utterance index, j as the causal utterance index, and the emotion category prediction result of the i-th utterance as the analysis result.

[0115] Relative position encoding is based on a Gaussian kernel function, and its specific formula is as follows:

[0116]

[0117] Where, r′ p For the dynamic embedding of the current relative position p; r j Based on position embedding, K p (j) is the Gaussian kernel function, describing the influence strength of relative position p on relative position j, where j∈{-P, ..., P} is one of all possible relative positions traversed, p is the currently focused relative position, and σ k These are the shape control parameters for the kernel function.

[0118] This method fills the gap in current methods for analyzing the causes of emotions in multimodal dialogues within Chinese contexts. It employs a multimodal heterogeneous graph approach to explicitly model the emotional and causal contexts within the dialogue, resulting in more accurate and comprehensive extraction of emotional causes. During multimodal fusion, a heterogeneous graph attention network is used to effectively fuse feature information from different modalities at both inter- and intra-narrative scales, thereby enhancing the model's performance in extracting emotional causes.

[0119] Example 2

[0120] In this embodiment, a dialogue emotion cause analysis system for the field of multimodal emotion computing is adopted. The system includes a multimodal dialogue feature extraction and alignment module, a multi-scale semantic fusion module based on a multimodal heterogeneous graph attention network, and an emotion cause matching module.

[0121] The multimodal dialogue feature extraction and alignment module includes a feature extraction submodule and a feature alignment submodule. The feature extraction submodule is used to extract features of each modality from multimodal data containing text modality, audio modality, and video modality, and to extract local contextual dialogue features of the text modality from the text modality features. The feature alignment submodule is used to align the audio modality features and video modality features, and to extract local contextual dialogue features of the audio modality and video modality from them.

[0122] The multi-scale semantic fusion module includes a multimodal heterogeneous graph construction submodule and a multi-scale semantic fusion submodule. The multimodal heterogeneous graph construction submodule maps multimodal local context dialogue features to nodes and edges in a heterogeneous graph to construct a multimodal heterogeneous graph. The multi-scale semantic fusion submodule performs multi-scale semantic fusion based on the multimodal heterogeneous graph and attention mechanism, and outputs a discourse vector containing emotion context embedding and causal context embedding.

[0123] The emotion cause matching module includes an emotion category prediction submodule and an emotion cause prediction submodule. The emotion category prediction submodule obtains emotion category prediction and cause clause prediction results through the MLP module based on emotion context embedding and cause context embedding. The emotion cause prediction submodule matches different discourse vectors to generate triple analysis results containing emotion category, emotion discourse index and cause discourse index.

[0124] The workflow of the system is as follows Figure 2 As shown, Figure 2 In the diagram, blue circles represent emotion context nodes, green circles represent cause context nodes, yellow circles represent dialogue supernodes, blue lines represent emotion context connections, and green lines represent cause context connections. The system workflow specifically includes:

[0125] (i) The multimodal dialogue feature extraction and alignment module receives multimodal dialogue text, audio, and video input data and extracts and aligns semantic features;

[0126] (ii) The features are mapped to nodes and edges in a multimodal heterogeneous graph through a multi-scale semantic fusion module;

[0127] (III) Based on heterogeneous graph attention network, the propagation and fusion of contextual features within and between discourses are realized to obtain discourse vectors with fused contextual features;

[0128] (iv) Use the emotion cause matching module to match different discourse vectors and generate a triplet output containing emotion category, emotion discourse index and cause discourse index.

[0129] In step (1), the multimodal dialogue feature extraction and alignment module extracts utterance-level features from the text, audio, and video modalities of the input dialogue using Speaker-aware RoBERTa, Wav2Vec2, and VideoMAE, respectively.

[0130] The following describes the construction and working process of the multimodal dialogue feature extraction and alignment module, the multi-scale semantic fusion module based on the multimodal heterogeneous graph attention network, the emotion cause matching module, and the construction process of the multimodal emotion cause analysis dataset.

[0131] (I) Multimodal Dialogue Feature Extraction and Alignment Module

[0132] The goal of this module is to fully extract and align the features of each utterance modality in the dialogue, providing a foundation for subsequent analysis of the causes of emotion. Specifically, this module can be divided into a feature extraction submodule and a feature alignment submodule.

[0133] ①The feature extraction submodule works as follows:

[0134] (1) Obtain the input features of the three modalities of the dialogue;

[0135] For the text modality, all utterances and their corresponding speakers from the dialogue are concatenated, and several special tokens are added to obtain the text modality input sequence: <sep_token> ,in,<cls_token> and<sep_token> S represents the classification labels of a pre-trained language model. i Let i be the text content of the i-th utterance; Let X be the speaker identifier for the i-th utterance. To preserve as much contextual information as possible when inputting dialogues exceeding the maximum input sequence length of the PLM into the PLM, the last marker of the longest utterance in the dialogue is truncated sequentially during preprocessing until the maximum sequence length requirement of the PLM is met. For the audio modality, the audio is sampled to 16kHz. For the video modality, the video is first sampled at equal intervals into a sequence of several frames to obtain the input sequence X for the video modality. v ,in, F is the number of frames after sampling, d f This refers to the dimensions of the image.

[0136] (2) Encode input features using a modality-specific encoder;

[0137] For text, the input sequence X t The input is fed into a pre-trained language model (PLM) to obtain a sequence representation of the entire dialogue:

[0138] I t =PLM(X t ),

[0139] in, Let L be the sequence representation of the entire dialogue, and d be the length of the input sequence. t It is the hidden dimension of PLM.

[0140] To obtain a sequence representation of each utterance, I tThe sequence representation of each utterance is obtained by taking a weighted average of the sequence representations of the tokens for each dialogue. N represents the number of words spoken in the dialogue.

[0141] Audio X a The input is fed into the Audio Feature Extractor (AFE) to obtain a sequence representation of the dialogue audio modality:

[0142] E a =AFE(X) a ),

[0143] in, For the sequence representation of audio modalities, d a This is the hidden layer dimension of the audio feature extraction model. Then, the image sequence X... v The input is fed into the Visual Feature Extractor (VFE) to obtain a sequence representation of the video modalities:

[0144] E v =VFE(X) v ),

[0145] in, d v It is the hidden layer dimension of the video feature extraction model.

[0146] After obtaining the sequence representation of each modality, they are further subjected to single-modal local context encoding. For text, for the sequence representation E... t Multi-Head Self-Attention (MHSA) is used for further encoding. Local contextual information is captured using a mask M and a discourse window w. The mask inside the discourse window is set to 0, and the mask outside the window is set to negative infinity. Then, the mask M is applied to MHSA, and the process can be represented as follows:

[0147] H t =MHSA(E t M),

[0148] Among them, H t M represents the local contextual features of the text after masked multi-head self-attention encoding, where M is the mask.

[0149] ② The feature alignment submodule uses an implicit feature layer alignment method based on a Gated Recurrent Unit (GRU). Its specific working process is as follows:

[0150] A GRU-based coding structure is used for local context information extraction. The specific process can be represented as follows:

[0151] E′ m =LN(E m +GRU(E m )),

[0152] H m =LN(E m +E′ m +FFN(E′ m )),

[0153] Among them, E ′m E represents the intermediate features of a single modality after GRU encoding and LN, which integrates global information and temporal dependencies. m This is a sequence representation of a single mode. The local context features of a single modality are obtained after residual fusion (global features, temporal features, and nonlinear features) and LN, where m∈{a,v} is the modality identifier, LN represents layer normalization, and FFN represents feed-forward network.

[0154] After performing local context encoding on each modality, we obtain the local context sequence representations H for the text, audio, and video modalities. t H a H v Finally, the dimensionality of the sequence representation for each modality is unified through a linear layer, so that d t =d a =d v =d h , where d t d a d v and d h These are the feature dimensions for text, audio, and video, and the unified target dimension, respectively.

[0155] (II) Multi-scale semantic fusion module based on multimodal heterogeneous graph attention network

[0156] The goal of this module is to fully extract contextual information from each utterance in a dialogue by constructing an effective fusion of emotional and causal contexts within and between different modalities. This approach uses multimodal heterogeneous graphs to effectively model utterance interactions in dialogues, achieving a deep understanding of complex dialogue content. Specifically, this module can be subdivided into a multimodal heterogeneous graph construction submodule and a multi-scale semantic fusion submodule.

[0157] ①The working process of the multimodal heterogeneous graph construction submodule is as follows:

[0158] Overall structure It means that among them It is composed of all graph nodes vi The set of nodes ε is composed of all edges (v i ,r,v j The edge set consisting of )∈ε It refers to the relationship between two nodes.

[0159] (1) Select the node type of the multimodal heterogeneous graph;

[0160] There are four types of nodes in total: discourse supernodes HN u Dialogue supernode SN d Emotional context node N e and cause context node N c Each discourse supernode contains features of three modalities {H}. t H a H v The dialogue supernode needs to carry global context information for the entire dialogue; therefore, it is initialized with average pooling of all utterance token features.

[0161] SN d ={AvgPool(H n )},n∈{t,a,v}

[0162] Where AvgPool represents the average pooling operation, N e and N c Initialize as a sequence representation H of the corresponding discourse text modality t .

[0163] (2) Determine the type of edge;

[0164] like Figure 3 Discourse integration of heterogeneous diagrams Figure 4 The diagram illustrates intra-discourse fusion heterogeneity. Blue circles represent emotion context nodes, green circles represent cause context nodes, yellow circles represent dialogue supernodes, blue lines represent emotion connections, and green lines represent cause connections. Five types of edges are defined: same speaker, different speaker, global connection, emotion context connection, and cause context connection. Specifically, for the same speaker and different speakers, each edge can connect nodes of different modalities between different utterances or within the same utterance.

[0165] The relationship R between nodes is determined as follows:

[0166] same speaker: Connects two speech supernodes of the same speaker;

[0167] different speaker: Connects two speech supernodes from different speakers;

[0168] global connection: connects the discourse supernode and the dialogue supernode (bidirectional connection);

[0169] Emotion connection: Connects the discourse supernode with its corresponding emotion context node;

[0170] Cause connection: Connects the discourse supernode to its corresponding cause context node.

[0171] (3) Construct a multimodal heterogeneous graph;

[0172] Construct a multimodal heterogeneous graph using the following constraints:

[0173] (3-1) Past discourse supernodes can point to future discourse supernodes, but future discourse supernodes cannot point to past discourse supernodes;

[0174] (3-2) Set a direct context window K for a discourse U i With the previous K words U i The utterances of the same speaker are called utterances U. i The direct context, only in discourse U i Only utterances within the direct context of a utterance have edge connections with that utterance, either "same speaker" or "different speaker". "Same speaker" and "different speaker" connect to the supernodes of the same speaker and the supernodes of different speakers, respectively.

[0175] (3-3) Each utterance is only connected to its corresponding emotion node and cause node by edge type of emotion connection and cause connection, and there are no edge connections between it and the emotion nodes and cause nodes of other utterances.

[0176] (3-4) All utterance nodes are connected to dialogue nodes by bidirectional edges with global connection. The global connection is used to facilitate the propagation of global context semantics.

[0177] Constraints (3-1) and (3-2) allow for the description of the contextual structure of discourse with relatively low spatiotemporal complexity. Constraint (3-3) effectively establishes independent emotional and causal contexts for each discourse. Constraint (3-4) enables a precise grasp of the overall content of the dialogue. This embodiment, through the above constraints, can... Figure 5 Dialogue building, such as Figure 6 The multimodal heterogeneous graph (K=1).

[0178] ②The multi-scale semantic fusion module is constructed as follows:

[0179] (1) Define metapath;

[0180]

[0181] Where φ is the set of metapaths. For different node types v i and v j The corresponding edge type, Φ(v i v j ) represents v i Connected to v via edge type r′ j All paths. By using different meta-paths Φ, the semantic information of different modalities and scales in a multimodal heterogeneous graph can be revealed.

[0182] (2) Propagation: Feature propagation between nodes is achieved through heterogeneous graph attention;

[0183] Given a pair of nodes (v) connected by a metapath Φ i v j ), Neighbor v j For target node v i The importance can be calculated as:

[0184]

[0185] in, For the neighbor v j For target node v i Attention weights, with larger values ​​indicating stronger semantic associations, H i H j and H k denoted as the node context features of different modalities under the metapath Φ, and σ is the activation function. Represents node v i (Including itself) neighbors based on meta-path Φ, a Φ It is the node-level attention vector of the meta-path Φ, Φ um Φ cg and Φ fg These are the global meta-path, the emotion meta-path, and the causal meta-path, respectively.

[0186] (3) Aggregation;

[0187] Based on attention weights, node v is obtained through aggregation using the following process. i Features based on metapath Φ:

[0188]

[0189] in, For node v i Based on the characteristics of the metapath Φ; since Φ∈{Φ um Φ cg Φ fg The calculation results include and Embed the emotional context for the i-th utterance. Embed the context of the reason for the i-th utterance.

[0190] For a given set of metapaths {Φ um Φ cg Φ fg Finally, we obtain the emotional context embedding and causal context embedding of each discourse by fusing multi-scale and multimodal features.

[0191] (III) Emotional Reason Matching Module

[0192] The goal of this module is to achieve accurate emotion category prediction and emotion cause extraction by pairing the emotion context embeddings and causal context embeddings obtained from a multi-scale semantic fusion module based on a multimodal heterogeneous graph attention network. Specifically, this module can be further divided into an emotion category prediction submodule and an emotion cause prediction submodule.

[0193] The emotion category prediction module works as follows:

[0194] This module obtains the prediction results of sentiment category and cause clause through the MLP module:

[0195]

[0196] in, The predicted sentiment category for the i-th utterance. The prediction result for the cause clause of the i-th utterance. Embed the emotional context for the i-th utterance. Embed the reason context for the i-th utterance.

[0197] The working process of the emotion cause prediction module is as follows:

[0198] (1) Encoding the relative position of discourse;

[0199] The relative positional information between utterances is crucial for indicating sentiment cause pairs. Therefore, relative positional embedding learning is used to embed this information into link prediction. If the relative position of two utterances is too large, the probability of a link between them is small. Therefore, for each possible relative position p∈{-P,…,P}, a kernel function K is used. p (·) is used to describe the impact of discourse on position:

[0200]

[0201] Among them, K p (j) is the Gaussian kernel function, describing the influence strength of relative position p on relative position j, where j∈{-P, ..., P} is one of all possible relative positions traversed, p is the currently focused relative position, and σ k represents the shape control parameter of the kernel function. Position embedding based on the kernel function can be represented as follows:

[0202]

[0203] Where, r′ p For the dynamic embedding of the current relative position p; r j Based on positional embedding. If j is close to m, r j For r′ -p The impact will be greater for more distant relative positions. The relative position encoding is concatenated with the sentiment and cause node embeddings and fed into the MLP for final prediction.

[0204]

[0205] in, Let i be the predicted emotion-cause pair of the emotion of the i-th utterance and the cause of the j-th utterance. Embed the emotional context for the i-th utterance. For the context embedding of the cause of the j-th utterance, r′ j-i Encode the relative positions of the two utterances, ji.

[0206] During training, multi-task joint optimization is performed using a three-part loss function consisting of emotion prediction, cause prediction, and emotion cause link prediction.

[0207] In summary, this solution can effectively and completely extract emotional cause triples from multimodal dialogues, thereby automating a detailed emotional cause analysis of the dialogue. Furthermore, this solution is adaptable to multimodal dialogues in different scenarios and languages, providing valuable information support for sentiment analysis through the extraction of emotional cause triples.

[0208] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in the present invention, and these modifications or substitutions should all be covered within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A method for analyzing the causes of emotions in dialogue based on multimodal emotion computing, characterized in that, The method steps include: S1. Extract modal features and local context features from multimodal data containing text, audio, and video modalities, and output multimodal local context dialogue features; S2. Map the multimodal local context dialogue features to nodes and edges in a heterogeneous graph to construct a multimodal heterogeneous graph; S3. Based on multimodal heterogeneous graphs and attention mechanisms, multi-scale semantic fusion is performed to output discourse vectors containing emotional context embeddings and causal context embeddings. S4. Match different discourse vectors to generate triplet analysis results containing emotion category, emotion discourse index, and cause discourse index.

2. The method for analyzing the causes of emotions in dialogue based on multimodal emotion computing according to claim 1, characterized in that, The local contextual features of the text modality are obtained through masked multi-head self-attention encoding, and the specific process is as follows: H t =MHSA(E t ,M), Among them, H t The text modality represents the local contextual features after masked multi-head self-attention encoding. MHSA is a masked multi-head self-attention mechanism. t M is the sequence representation of each discourse text modality, where M is the mask; The local context features of the audio and video modalities are obtained based on the implicit feature layer alignment method of the recurrent encoder, and the specific process is as follows: AND' m =LN(E m +GRU(E m )), A m =LN(E m +E′ m +FFN(E′ m ))), Among them, E′ m For intermediate features of a single modality after GRU encoding and LN, global information and temporal dependencies are fused, e m This is a sequence representation of a single mode. For a single modality, the local context features are obtained after residual fusion and LN, where m∈{a,v} is the modality identifier, LN represents layer normalization, and FFN represents the feedforward neural network.

3. The method for analyzing the causes of emotions in dialogue based on multimodal emotion computing according to claim 1, characterized in that, The specific process of constructing the multimodal heterogeneous graph in S2 includes: S21. Construct nodes for a multimodal heterogeneous graph. The nodes include four types of nodes: discourse supernodes, dialogue supernodes, emotion context nodes, and cause context nodes. Each discourse supernode contains multimodal local context dialogue features, and the dialogue supernode is initialized with the average pooling result of the multimodal local context dialogue features. S22. Determine the edge type and the relationship between the nodes corresponding to each edge. The edge type includes edge with the same speaker, edge with different speakers, global connection edge, emotional context connection edge, and causal context connection edge. The relationships between the corresponding nodes include: Same speaker edge: A one-way connection between two speech supernodes of the same speaker; Different speaker edge: A one-way connection between two speech supernodes of different speakers; Global connection edges: bidirectional connections between discourse supernodes and dialogue supernodes; Emotional context connection edge: a one-way connection between a discourse supernode and its corresponding emotional context node; Cause-Context Connection Edge: A one-way connection between a discourse supernode and its corresponding cause-context node; S23. Establish constraints and construct a multimodal heterogeneous graph based on nodes, edges, and relationships between nodes.

4. The method for analyzing the causes of emotions in dialogue based on multimodal emotion computing according to claim 3, characterized in that, The constraints established in S23 include temporal unidirectional constraints, context window constraints, association constraints, and global constraints, specifically: Temporal unidirectional constraint: Only past time-time discourse supernodes are allowed to establish pointing connections to future time-time discourse supernodes; future time-time discourse supernodes are not allowed to construct pointing connections to past time-time discourse supernodes in reverse. Context window constraint: Introducing a direct context window parameter K for a discourse. For any discourse, its direct context is the set of the first K discourses that have the same speaker attribute as that discourse. Only when other discourses belong to the direct context of that discourse can an edge connection relationship with that discourse be established with the same speaker edge or a different speaker edge. Among them, the same speaker edge is used to connect discourse supernodes with the same speaker attribute, and the different speaker edge is used to connect discourse supernodes with different speaker attributes, thereby limiting the redundant construction of invalid edges. Association constraint: Each utterance is only connected to its corresponding emotion node and cause node by edge type of emotion context edge connection and cause context connection, and has no edge connection with the emotion nodes and cause nodes of other utterances; Global constraint: All speech nodes are connected to dialogue nodes by bidirectional edges with global connections.

5. The method for analyzing the causes of emotions in dialogue based on multimodal emotion computing according to claim 1, characterized in that, In S3, when performing multi-scale semantic fusion based on multimodal heterogeneous graphs and attention mechanisms, meta-path-level multimodal fusion is used, and its specific steps include: S31. Constructing meta-paths based on multimodal heterogeneous graphs; S32. Select the target node and use the graph attention mechanism to propagate features between nodes in the metapath to obtain the attention weights of neighboring nodes to the target node. S33. Based on attention weights, aggregate the features of the target node based on meta-paths to obtain the emotion context embedding and the cause context embedding. S34. Repeat S31-S33 until all nodes have been selected once, and output the discourse vector containing the emotion context embedding and the cause context embedding.

6. The method for analyzing the causes of emotions in dialogue based on multimodal emotion computing according to claim 5, characterized in that, The meta-path constructed in S31 is specifically represented as follows: φ={…,Φ(v i ,v j ,r′),…}, v i ≠v j , Where φ is the set of metapaths. For different node types v i and v j The corresponding edge type, Φ(v i v j ) represents v i Connected to v via edge type r′ j All paths; through different meta-paths Φ, the semantic information of different modalities and scales of the multimodal heterogeneous graph can be revealed.

7. The method for analyzing the causes of emotions in dialogue based on multimodal emotion computing according to claim 5, characterized in that, The calculation formulas for the emotional context embedding and causal context embedding are as follows: Φ∈{Φ um ,F cg ,F fg }; in, For the neighbor v j For target node v i Attention weights, with larger values ​​indicating stronger semantic associations, H i H j and H k denoted as the node context features of different modalities under the metapath Φ, and σ is the activation function. Represents node v i (Including itself) neighbors based on meta-path Φ, a Φ It is the node-level attention vector of the meta-path Φ. For node v i Based on the characteristics of the metapath Φ; since Φ∈{Φ um Φ cg Φ fg The calculation results include and Embed the emotional context for the i-th utterance. Embedding the causal context for the i-th utterance, Φ um Φ cg and Φ fg These are the global meta-path, the emotion meta-path, and the causal meta-path, respectively.

8. The method for analyzing the causes of emotions in dialogue based on multimodal emotion computing according to claim 1, characterized in that, The specific steps for matching different utterance vectors in S4 include: S41. Input the emotion context embedding and the cause context embedding into the multilayer perceptron to obtain the emotion category prediction result and the cause clause prediction result; S42. Select the emotional context embedding of the i-th utterance and the causal context embedding of the j-th utterance, and calculate the relative position encoding of the two utterances ji. S43. Input the emotional context embedding of the i-th utterance, the causal context embedding of the j-th utterance, and the relative position encoding of the two utterances ji into the multilayer perceptron, and output the emotion-cause pair prediction result of the emotion of the i-th utterance and the cause of the j-th utterance. If the result is greater than the preset value, then output i as the emotional utterance index, j as the causal utterance index, and the emotion category prediction result of the i-th utterance as the analysis result.

9. The method for analyzing the causes of emotions in dialogue based on multimodal emotion computing according to claim 8, characterized in that, The relative position encoding is constructed based on the Gaussian kernel function, and its specific formula is as follows: Where, r′ p For the dynamic embedding of the current relative position p; r j Based on position embedding, K p (j) is the Gaussian kernel function, describing the influence strength of relative position p on relative position j, where j∈{-P, ..., P} is one of all possible relative positions traversed, p is the currently focused relative position, and σ k These are the shape control parameters for the kernel function.

10. A dialogue emotion cause analysis system based on multimodal emotion computing, characterized in that, The system applies a dialogue emotion cause analysis method based on multimodal emotion computing as described in any one of claims 1-9. The system includes a multimodal dialogue feature extraction and alignment module, a multi-scale semantic fusion module, and an emotion cause matching module. The multimodal dialogue feature extraction and alignment module includes a feature extraction submodule and a feature alignment submodule. The feature extraction submodule is used to extract modal features from multimodal data containing text, audio, and video modalities, and to extract local contextual dialogue features of the text modalities from the text modal features. The feature alignment submodule is used to align the audio and video modal features, and to extract local contextual dialogue features of the audio and video modalities. The multi-scale semantic fusion module includes a multimodal heterogeneous graph construction submodule and a multi-scale semantic fusion submodule. The multimodal heterogeneous graph construction submodule maps multimodal local context dialogue features to nodes and edges in a heterogeneous graph to construct a multimodal heterogeneous graph. The multi-scale semantic fusion submodule performs multi-scale semantic fusion based on the multimodal heterogeneous graph and an attention mechanism, and outputs a discourse vector containing emotion context embedding and causal context embedding. The emotion cause matching module includes an emotion category prediction submodule and an emotion cause prediction submodule. The emotion category prediction submodule obtains emotion category prediction and cause clause prediction results through an MLP module based on emotion context embedding and cause context embedding. The emotion cause prediction submodule matches different discourse vectors to generate triple analysis results containing emotion category, emotion discourse index and cause discourse index.

Images