Graph adversarial emotion recognition method and system for conversation analysis in IoT environment

Abstract

The present invention belongs to the field of emotion recognition technology, and discloses a graph adversarial emotion recognition method and system for conversation analysis in an Internet of Things environment, including: constructing a context analysis module and a causal perception module, analyzing the structural dependency in the conversation, and using the structural dependency graph to learn the contextual emotional features between the interlocutors; capturing the causal relationship between the conversations of different speakers at the same time, combining the self-attention mechanism to dynamically adjust the weight of the causal clues in the conversation, obtaining the causal relationship that has the greatest impact on the emotion change, and more accurately understanding the emotions contained in the conversation; finally, introducing a differential regularization term to measure the difference between the outputs of the context analysis module and the causal perception module, and promoting information interaction between the context analysis module and the causal perception module, and obtaining more comprehensive emotional information. The present invention performs better emotion recognition processing on complex conversation scenes in an Internet of Things environment, and improves the recognition accuracy of emotion recognition tasks.

Description

Technical Field

[0001] The present invention belongs to the technical field of emotion recognition, and relates to a graph adversarial emotion recognition method and system for dialogue analysis in an Internet of Things environment. Background Art

[0002] With the continuous development of IoT technology, smart devices have entered people's lives. In this context, a large amount of conversation data is generated and transmitted by smart devices. These data are widely used in various fields such as smart customer service, smart home, medical health, education, etc. However, due to the complexity of conversation data in the IoT environment, more efficient recognition mechanisms are urgently needed to deal with these problems.

[0003] The task of conversation emotion recognition is mainly to infer the emotional state of each speaker in the conversation by analyzing the voice, text or video information. However, the conversation situation in the IoT environment is complex and changeable, especially when multiple speakers interact. The emotion of each speaker depends not only on the content of his or her own speech, but also on the influence of other speakers and the entire conversation context. This multi-party interaction brings highly complex emotional dynamics, which makes it difficult for simple emotion classification models to accurately capture and model. Traditional emotion recognition methods usually use sequence models, such as recursive neural networks and long short-term memory networks to encode text or speech, and perform emotion classification based on the encoding results. However, such methods have difficulty in capturing contextual dependencies in multi-speaker conversations, especially when the conversation is long and the speaker's emotional interaction is complex, and the model tends to ignore the global emotional dynamics. In addition, traditional models have poor recognition effects on low-frequency emotion categories and cannot solve the problem of imbalanced emotion categories. Summary of the invention

[0004] The purpose of the present invention is to solve the problems that traditional models in the prior art are difficult to capture contextual dependencies in multi-speaker conversations, easily ignore global emotional dynamics, have poor recognition effects on low-frequency emotional categories, and cannot solve the problem of imbalanced emotional categories. A graph adversarial emotion recognition method and system for conversation analysis in an Internet of Things environment is provided.

[0005] In order to achieve the above object, the present invention adopts the following technical solutions:

[0006] Graph adversarial emotion recognition method for conversation analysis in IoT environment, including:

[0007] Collecting text data of conversations containing multiple speakers;

[0008] Encode each utterance in the conversation text data based on the pre-trained text model to obtain the text feature representation of the conversation data;

[0009] Build a context analysis module to analyze the context dependency of speakers during conversations based on the discourse dependency graph, and use graph neural networks to aggregate information between text feature representations of conversation data;

[0010] Construct a causal perception module to extract causal relationships in conversations, and use the self-attention mechanism to dynamically adjust the weight of causal clues in conversations to obtain the causal relationships that have the greatest impact on sentiment changes;

[0011] Add small perturbations to the input data to generate adversarial samples, calculate the contrastive learning loss of the original samples and adversarial samples based on contrastive learning loss, and combine them to optimize the sentiment classification performance of the model;

[0012] Obtain the adjacency matrix of the context analysis module and the causal perception module, and measure the difference between the adjacency matrices of the context analysis module and the causal perception module through the differential regularization term, and promote the information interaction between the context analysis module and the causal perception module based on the mutual cross-attention mechanism;

[0013] After completing the information interaction between the context analysis module and the causal perception module, the feature information of the context analysis module and the feature information of the causal perception module are spliced, and the generated feature representation is used as the final input. Sentiment classification is performed through the fully connected layer to obtain the final emotion recognition result.

[0014] A further improvement of the present invention is:

[0015] Furthermore, after including the conversation text data of multiple speakers, the text data is preprocessed by word segmentation and removal of stop words and punctuation marks.

[0016] Furthermore, each utterance in the conversation text data is encoded based on the pre-trained text model to obtain the text feature representation of the conversation data. Specifically, RoBERTa-large is selected as the utterance encoder, the conversation data is preprocessed into a format acceptable to the model, each speech is marked as a separate text unit, and tokenization is performed; given an input sequence ; ; RoBERTa-large uses the Transformer structure to encode it:

[0017] (1);

[0018] in, The feature representation of the conversation data, Represents a text sequence; extracts the feature representation of each speech from the output of the last self-attention layer, and selects The labeled output represents the feature representation of the entire utterance.

[0019] Furthermore, a context analysis module is constructed to analyze the context dependency of the speaker during the conversation based on the discourse dependency graph, and the information between the text feature representations of the conversation data is aggregated using the graph neural network. Specifically:

[0020] Each node in the discourse dependency graph corresponds to a basic discourse unit, and the node set is ; The node set represents all the speeches in the conversation, and each node Corresponding to one of the features of the conversation data ; Edge set Representation Node and The dependency relationship between them, where each edge corresponds to a discourse dependency type;

[0021] For each pair of adjacent nodes and , calculate the attention coefficient between each pair of adjacent nodes , the calculation formula is as follows:

[0022] (2);

[0023] in, Indicates that in the dependency Next, the target discourse To the neighbors The weight of the edge; the dependency r represents the type of edge between two nodes; the edge types include: comment, clarification question, elaboration, confirmation, continuation, explanation, condition, question-answer pair, alternation, question elaboration, result, background, narrative, correction, parallel and contrast; is a learnable parameter weight vector, is the linear weight matrix under the attention mechanism, and The nodes are and nodes The feature representation of represents the vector concatenation operation, and LeakyReLU is the activation function;

[0024] In order to study the relationship between the attention coefficients among all neighboring points, normalization is performed based on the softmax function; the final attention coefficient For adjacent nodes The features of the node are weighted summed to obtain Context-dependent features , specifically:

[0025] (3)

[0026] in, Representation Node Through dependencies A set of connected neighbor nodes.

[0027] Furthermore, we extract the causal relationships in the conversation and use the self-attention mechanism to dynamically adjust the weights of the causal clues in the conversation to obtain the causal relationships that have the greatest impact on sentiment changes. Specifically:

[0028] The causal relationships in the conversation are extracted based on the COMET model. The causal relationships include internal causal relationships and external causal relationships. The internal causal relationships and external causal relationships are input into the COMET network. The last encoding layer of the COMET model fuses the contextual information from the input events and relationship types, extracts the hidden state representation of the encoding layer, and uses it as the causal clue C.

[0029] The causal clues are mapped as edges in the conversation graph, indicating the causal dependency between speeches. The self-attention mechanism is used to calculate the weights of these clues in different speeches. Through the self-attention layer, the model can adjust the attention distribution between different speeches. The larger the weight of the edge, the stronger the impact of the causal relationship on the emotional change, and the more attention is focused on the relevant nodes. The calculation formula is as follows:

[0030] (4);

[0031] (5);

[0032] in, Indicates that Converted into a query vector, Indicates that Converted into a key vector, is a linear transformation; Represents the common sense knowledge graph obtained from ATOMIC and the discourse , relevant internal causal cues; a mask is a mask that ensures that the retrieval process only considers previous utterances by the same speaker; is a mapping function that maps the index of an utterance to the corresponding speaker; Indicates the dimension of the vector, which is used for normalization; in addition to considering the causal clues within the same speaker, it is also necessary to calculate the causal clue association scores between different speakers :

[0033] (6);

[0034] (7);

[0035] in, The cross-speaker causal clue score is calculated, where the causal clues come from the interaction between different speakers; after obtaining the weight scores of internal and external causal clues, they are combined to comprehensively consider all potential causal relationships and finally form a causal-aware context representation:

[0036] (8);

[0037] (9);

[0038] in, and denote the weights of causal cues between internal and external speakers, respectively; and Representation and discourse the collection of utterances internal and external to the speaker; The causal cue characteristics are the comprehensive weights of internal and external causal cues.

[0039] Furthermore, we add small perturbations to the input data to generate adversarial samples. Based on the contrastive learning loss, we calculate the contrastive learning loss of the original sample and the adversarial sample respectively, and combine them to optimize the sentiment classification performance of the model. Specifically,

[0040] Generate adversarial samples by calculating the gradient of the input data:

[0041] (10);

[0042] (11);

[0043] (12);

[0044] Among them, formula (10) is to obtain the input sample The gradient during training , Represents the input sample The gradient calculation of Represents the conditional probability distribution of the model prediction; Formula (11) is based on the gradient information of the input Generate a perturbation vector and normalize it to ensure that the size of the adversarial perturbation is controlled within Within the range; through gradient information Determines how to modify the input sample To increase the loss function, thereby "tricking" the model into making incorrect predictions; Represents the disturbance amplitude, which is used to control the size of the counter-disturbance. Represents the gradient of Norm; Formula (12) is the input sample Add adversarial perturbations to ;

[0045] The contrastive learning loss is introduced to calculate the contrastive learning loss of the original sample and the adversarial sample respectively, and the contrastive loss is calculated by the following formula:

[0046] (13);

[0047] (14);

[0048] Among them, formula (13) is the contrast loss of the original sample; formula (14) is the contrast learning loss of the adversarial sample; Indicates the samples in the current batch The index set of all samples belonging to the same category; Indicates all sample indices in the current sample, except for sample ; Indicates The feature vector of the sample and the feature vector of the nth sample The similarity between is the temperature parameter, used to adjust the distribution of similarity; Indicates The feature vector of the sample and the feature vector of the nth sample The similarity between the two after adding noise adversarial; after calculating the contrastive learning loss of all adversarial samples, it is combined with the loss of the original sample to obtain the final contrastive loss function. The specific formula is as follows:

[0049] (15).

[0050] Furthermore, the adjacency matrix of the context analysis module and the causal perception module is obtained, specifically:

[0051] Causal clue features A collection of causal perception modules to obtain the adjacency matrix ; Based on context-dependent features A collection of to obtain the adjacency matrix of the context analysis module ;

[0052] The difference between the adjacency matrices of the context analysis module and the causal perception module is measured through the differential regularization term, specifically:

[0053] (16);

[0054] in, Represents the Frobenius norm, which is used to measure the difference between the outputs of the context analysis module and the causal perception module.

[0055] Furthermore, the information interaction between the context analysis module and the causal perception module is promoted based on the cross-attention mechanism, specifically:

[0056] The cross-attention mechanism is combined with the information interaction between the context analysis module and the causal perception module to capture the interactive relationship between the context-dependent feature vector and the causal clue feature vector. Specifically:

[0057] (17);

[0058] (18);

[0059] (19);

[0060] (20);

[0061] in, and is the attention matrix, Causal clue features Projection to context-dependent features In the above, it is emphasized that each discourse focuses on the characteristics of different clues. Indicates that context-dependent features Projection onto causal cue features superior; represents the final context feature representation enhanced by causal clue features; represents the final causal cue feature representation enhanced by context-dependent features; and is a learnable parameter matrix; the training goal is to minimize the following overall objective function:

[0062] (twenty one);

[0063] (twenty two);

[0064] in, and is the regularization coefficient and the weight coefficient of contrastive learning; The standard cross entropy loss is used to measure the difference between the model prediction and the true label; B represents the number of dialogues, Indicates The first in the conversation The predicted probability of a statement; is the number of utterances in the Kth dialogue. In different dialogues, the number of utterances Not fixed.

[0065] Furthermore, after completing the information interaction between the context analysis module and the causal perception module, the feature information of the context analysis module and the feature information of the causal perception module are spliced, and the generated feature representation is used as the final input. After the emotion classification is performed through the fully connected layer, the final emotion recognition result is obtained, which is specifically:

[0066] The completing the information interaction between the context analysis module and the causal perception module is specifically: performing L-layer interaction between the context analysis module and the causal perception module information;

[0067] The L-layer interaction includes two modules in each layer: the context-dependent module is used to process the discourse structure information and the causal perception module is used to process the causal information perceived by the speaker; the output of each layer will exchange information between the context analysis module and the causal perception module through the cross-attention mechanism, so that the discourse structure information and the speaker perception information can be transmitted and integrated between different layers to complete the interaction of the current layer; finally, after the L-th layer interaction, the output and are concatenated as the final representation for sentiment prediction;

[0068] The calculation formula of the fully connected layer is as follows:

[0069] (twenty three);

[0070] (twenty four);

[0071] (25);

[0072] Among them, the features of the context analysis module and the causal perception module are combined through formula (23), and the linear transformation and The activation function generates the final feature representation for the next step of calculation; the last layer of the classification task is implemented through formula (24), which converts the model output into a probability estimate for each category; formula (25) is based on the probability distribution predicted by the model , select the category with the highest probability as the final prediction result; , is the weight matrix of the fully connected layer, and is the bias vector of the fully connected layer, is the activation function; is the predicted sentiment label, and the one with the highest probability is taken as the sentiment label; k represents the index of the category, and the value range of k is the set y of categories, that is, , Representation sample The probability of belonging to category k; Indicates The hidden representation of a sample after passing through the fully connected layer.

[0073] Graph adversarial emotion recognition system for conversation analysis in IoT environments, including:

[0074] A collection module, wherein the collection module collects conversation text data containing multiple speakers;

[0075] An encoding module, wherein the encoding module encodes each utterance in the conversation text data based on a pre-trained text model to obtain a text feature representation of the conversation data;

[0076] An analysis module, wherein the analysis module constructs a context analysis module, analyzes the context dependency of the speaker during the conversation based on the discourse dependency graph, and aggregates information between text feature representations of the conversation data using a graph neural network;

[0077] A weight adjustment module, wherein the weight adjustment module constructs a causal perception module, extracts causal relationships in the conversation, and uses a self-attention mechanism to dynamically adjust the weights of causal clues in the conversation to obtain the causal relationships that have the greatest impact on sentiment changes;

[0078] An adversarial learning module, which adds a small perturbation to the input data to generate an adversarial sample, calculates the contrastive learning loss of the original sample and the adversarial sample based on the contrastive learning loss, and combines them to optimize the sentiment classification performance of the model;

[0079] An information interaction module, wherein the information interaction module obtains the adjacency matrix of the context analysis module and the causal perception module, and measures the difference between the adjacency matrix of the context analysis module and the causal perception module through a differential regularization term, and promotes information interaction between the context analysis module and the causal perception module based on a mutual cross-attention mechanism;

[0080] The emotion classification module, after completing the information interaction between the context analysis module and the causal perception module, splices the feature information of the context analysis module and the feature information of the causal perception module, and uses the generated feature representation as the final input, performs emotion classification through the fully connected layer, and obtains the final emotion recognition result.

[0081] Compared with the prior art, the present invention has the following beneficial effects:

[0082] The present invention constructs a context analysis module and a causal perception module respectively, analyzes the structural dependency in the conversation, and uses the structural dependency graph to learn the contextual emotional features between the interlocutors; at the same time, it captures the causal relationship between the conversations of different speakers, dynamically adjusts the weight of the causal clues in the conversation in combination with the self-attention mechanism, obtains the causal relationship that has the greatest impact on emotional changes, and more accurately understands the emotions contained in the conversation; finally, a differential regularization term is introduced to measure the difference between the adjacency matrices of the context analysis module and the causal perception module, and promotes the information interaction between the context analysis module and the causal perception module based on the cross-attention mechanism, so as to obtain more comprehensive emotional information. The present invention performs better emotion recognition processing on complex conversation scenes in the Internet of Things environment, and improves the recognition accuracy of emotion recognition tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0083] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for use in the embodiments are briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present invention and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without creative work.

[0084] Figure 1 It is a schematic diagram of the flow of the graph adversarial emotion recognition method for conversation analysis in an Internet of Things environment of the present invention;

[0085] Figure 2 It is a structural schematic diagram of the graph adversarial emotion recognition system for conversation analysis in an Internet of Things environment of the present invention;

[0086] Figure 3 The embodiment of the present invention analyzes the impact of different adversarial perturbation levels on the overall model performance in IEMOCAP;

[0087] Figure 4 The embodiment of the present invention analyzes the impact of different adversarial perturbation levels on the overall model performance in MELD;

[0088] Figure 5 It is a schematic diagram of analyzing the sensitivity of network depth to model performance under different data set environments in an embodiment of the present invention;

[0089] Figure 6 It is a confusion matrix diagram of an embodiment of the present invention when data distribution is unbalanced in a complex dialog environment;

[0090] Figure 7 It is a confusion matrix diagram of an embodiment of the present invention when data distribution is unbalanced after the anti-attack module is removed. DETAILED DESCRIPTION

[0091] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Generally, the components of the embodiments of the present invention described and shown in the drawings here can be arranged and designed in various different configurations.

[0092] Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the invention claimed for protection, but merely represents selected embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0093] It should be noted that similar reference numerals and letters denote similar items in the following drawings, and therefore, once an item is defined in one drawing, further definition and explanation thereof is not required in subsequent drawings.

[0094] In the description of the embodiments of the present invention, it should be noted that if the terms "upper", "lower", "horizontal", "inner", etc. indicate an orientation or positional relationship based on the orientation or positional relationship shown in the drawings, or the orientation or positional relationship in which the product of the invention is usually placed when in use, it is only for the convenience of describing the present invention and simplifying the description, and does not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation on the present invention. In addition, the terms "first", "second", etc. are only used to distinguish the description, and cannot be understood as indicating or implying relative importance.

[0095] In addition, if the term "horizontal" appears, it does not mean that the component must be absolutely horizontal, but can be slightly tilted. For example, "horizontal" only means that its direction is more horizontal than "vertical", which does not mean that the structure must be completely horizontal, but can be slightly tilted.

[0096] In the description of the embodiments of the present invention, it is also necessary to explain that, unless otherwise clearly specified and limited, the terms "set", "install", "connect", and "connect" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or it can be indirectly connected through an intermediate medium, or it can be the internal connection of two components. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0097] The present invention is further described in detail below in conjunction with the accompanying drawings:

[0098] See also Figure 1 The present invention discloses a graph adversarial emotion recognition method for conversation analysis in an Internet of Things environment, comprising:

[0099] S101, collecting conversation text data containing multiple speakers;

[0100] S102, encoding each utterance in the conversation text data based on the pre-trained text model to obtain a text feature representation of the conversation data;

[0101] After including the conversation text data of multiple speakers, the text data is preprocessed by word segmentation, removal of stop words and punctuation marks.

[0102] Obtain text feature representation of conversation data, specifically: select RoBERTa-large as the discourse encoder, preprocess the conversation data into a format acceptable to the model, mark each speech as a separate text unit, and perform tokenization; given an input sequence ; ; RoBERTa-large uses the Transformer structure to encode it:

[0103] (1);

[0104] in, The feature representation of the conversation data, Represents a text sequence; extracts the feature representation of each speech from the output of the last self-attention layer, and selects The output of the tag represents the feature representation of the entire speech. Each self-attention layer can capture the relationship between different tags in the input sequence, thereby obtaining richer semantic information. After preprocessing the conversation data, a normalized data representation can be obtained. The high-dimensional features extracted in the preprocessing step provide the model with richer semantic information, allowing the emotion recognition model to understand the emotion changes more accurately.

[0105] S103, constructing a context analysis module, analyzing the context dependency of the speaker during the conversation based on the discourse dependency graph, and using a graph neural network to aggregate information between text feature representations of the conversation data;

[0106] In order to fully capture the dynamic changes of emotions in a conversation, it is necessary to analyze how the emotion of each utterance is affected by the previous sentence and the overall context. The context dependency analysis module can not only identify the emotional state of individual utterances, but also reveal subtle emotional changes through a deeper understanding of the context. Therefore, the context dependency analysis module includes a discourse dependency graph to simulate the structural information of conversations in multi-party conversation scenarios in the Internet of Things and smart networks. When the conversation data is preprocessed, these conversations are input into the discourse deep sequence network. These dependencies include common logical and semantic relationships in conversations, such as comments, clarifying questions, elaborations, confirmations, continuations, explanations, conditions, question-answer pairs, alternations, question elaborations, results, backgrounds, narrations, corrections, parallels, and contrasts. The network is trained on a corpus annotated with conversation dependencies to automatically predict and extract different dependencies from conversations. Each node in the discourse dependency graph corresponds to a basic discourse unit, and the node set is ; The node set represents all the speeches in the conversation, and each node Corresponding to one of the features of the conversation data ; Edge set Representation Node and The dependency relationship between them, where each edge corresponds to a discourse dependency type;

[0107] For each pair of adjacent nodes and , calculate the attention coefficient between each pair of adjacent nodes , the calculation formula is as follows:

[0108] (2);

[0109] in, Indicates that in the dependency Next, the target discourse To the neighbors The weight of the edge; the dependency r represents the type of edge between two nodes; the edge types include: comment, clarification question, elaboration, confirmation, continuation, explanation, condition, question-answer pair, alternation, question elaboration, result, background, narrative, correction, parallel and contrast; is a learnable parameter weight vector, is the linear weight matrix under the attention mechanism, and The nodes are and nodes The feature representation of represents the vector concatenation operation, and LeakyReLU is the activation function;

[0110] In order to study the relationship between the attention coefficients among all neighboring points, normalization is performed based on the softmax function; the final attention coefficient For adjacent nodes The features of the node are weighted summed to obtain Context-dependent features , specifically:

[0111] (3);

[0112] in, Representation Node Through dependencies A set of connected neighbor nodes. By weighted summing of the information of adjacent nodes, the feature representation of each node not only contains its own information, but also incorporates the information of other related nodes.

[0113] S104, build a causal perception module to extract the causal relationship in the conversation, and use the self-attention mechanism to dynamically adjust the weight of the causal clues in the conversation to obtain the causal relationship that has the greatest impact on the emotion change;

[0114] Changes in emotional states are usually triggered by interactions between speeches. Capturing the implicit causal relationships between speeches can improve the accuracy of emotion prediction. Based on the COMET model, causal relationships in conversations are extracted, and the generated causal relationships are divided into internal causal relationships and external causal relationships. Internal causal clues are related to the context and emotional changes of the speech itself, and the generated The external causal clues are related to the interaction between other speakers and the overall conversation context, generating The above two types of data are input into the COMET network. The last encoding layer of the COMET model fuses the contextual information from the input events and relationship types, extracts the hidden state representation of the encoding layer, and uses it as the causal clue C. Among them, the causal clues are mapped to the edges in the dialogue graph, indicating the causal dependency between speeches. The self-attention mechanism is used to calculate the weights of these clues in different speeches. Through the self-attention layer, the model can adjust the attention distribution between different speeches. The larger the weight of the edge, the stronger the impact of the causal relationship on the emotional change, and the more focused the attention is on the relevant nodes; the causal clue association score The calculation formula is as follows:

[0115] (4);

[0116] (5);

[0117] in, Indicates that Converted into a query vector, Indicates that Converted into a key vector, is a linear transformation; Represents the common sense knowledge graph obtained from ATOMIC and the discourse , relevant internal causal cues; a mask is a mask that ensures that the retrieval process only considers previous utterances by the same speaker; is a mapping function that maps the index of an utterance to the corresponding speaker; Indicates the dimension of the vector, which is used for normalization; in addition to considering the causal clues within the same speaker, it is also necessary to calculate the causal clue association scores between different speakers :

[0118] (6);

[0119] (7);

[0120] in, The cross-speaker causal clue score is calculated, where the causal clues come from the interaction between different speakers; after obtaining the weight scores of internal and external causal clues, they are combined to comprehensively consider all potential causal relationships and finally form a causal-aware context representation:

[0121] (8);

[0122] (9);

[0123] in, and denote the weights of causal cues between internal and external speakers, respectively; and Representation and discourse the collection of utterances internal and external to the speaker; The causal cue characteristics are the comprehensive weights of internal and external causal cues.

[0124] S105, adding a small perturbation to the input data to generate an adversarial sample, calculating the contrastive learning loss of the original sample and the adversarial sample based on the contrastive learning loss, and combining them to optimize the sentiment classification performance of the model;

[0125] Generate adversarial samples by calculating the gradient of the input data:

[0126] (10);

[0127] (11);

[0128] (12);

[0129] Among them, formula (10) is to obtain the input sample The gradient during training , Represents the input sample The gradient calculation of Represents the conditional probability distribution of the model prediction; Formula (11) is based on the gradient information of the input Generate a perturbation vector and normalize it to ensure that the size of the adversarial perturbation is controlled within Within the range; through gradient information Determines how to modify the input sample To increase the loss function, thereby "tricking" the model into making incorrect predictions; Represents the disturbance amplitude, which is used to control the size of the counter-disturbance. Represents the gradient of Norm; Formula (12) is the input sample Add adversarial perturbations to .

[0130] The contrastive learning loss is introduced to calculate the contrastive learning loss of the original sample and the adversarial sample respectively, and the contrastive loss is calculated by the following formula:

[0131] (13);

[0132] (14);

[0133] Among them, formula (13) is the contrast loss of the original sample; formula (14) is the contrast learning loss of the adversarial sample; Indicates the samples in the current batch The index set of all samples belonging to the same category; Indicates all sample indices in the current sample, except for sample ; Indicates The feature vector of the sample and the feature vector of the nth sample The similarity between is the temperature parameter, used to adjust the distribution of similarity; Indicates The feature vector of the sample and the feature vector of the nth sample The similarity between the two after adding noise adversarial; after calculating the contrastive learning loss of all adversarial samples, it is combined with the loss of the original sample to obtain the final contrastive loss function. The specific formula is as follows:

[0134] (15).

[0135] S106, obtaining the adjacency matrix of the context analysis module and the causal perception module, and measuring the difference between the adjacency matrix of the context analysis module and the causal perception module through a differential regularization term, and promoting information interaction between the context analysis module and the causal perception module based on a mutual cross attention mechanism;

[0136] Get the adjacency matrix of the context analysis module and the causal perception module, specifically:

[0137] Causal clue features A collection of causal perception modules to obtain the adjacency matrix ; Based on context-dependent features A collection of to obtain the adjacency matrix of the context analysis module ;

[0138] The difference between the adjacency matrices of the context analysis module and the causal perception module is measured through the differential regularization term, specifically:

[0139] (16);

[0140] in, Represents the Frobenius norm, which is used to measure the difference between the outputs of the context analysis module and the causal perception module.

[0141] The cross-attention mechanism is combined with the information interaction between the context analysis module and the causal perception module to capture the interactive relationship between the context-dependent feature vector and the causal clue feature vector. Specifically:

[0142] (17);

[0143] (18);

[0144] (19);

[0145] (20);

[0146] in, and is the attention matrix, Causal clue features Projection to context-dependent features In the above, it is emphasized that each discourse focuses on the characteristics of different clues. Indicates that context-dependent features Projection onto causal cue features superior; represents the final context feature representation enhanced by causal clue features; represents the final causal cue feature representation enhanced by context-dependent features; and is a learnable parameter matrix; the training goal is to minimize the following overall objective function:

[0147] (twenty one);

[0148] (twenty two);

[0149] in, and is the regularization coefficient and the weight coefficient of contrastive learning; The standard cross entropy loss is used to measure the difference between the model prediction and the true label; B represents the number of dialogues, Indicates The first in the conversation The predicted probability of a statement; is the number of utterances in the Kth dialogue. In different dialogues, the number of utterances Usually not fixed.

[0150] S107, after completing the information interaction between the context analysis module and the causal perception module, the feature information of the context analysis module and the feature information of the causal perception module are concatenated, and the generated feature representation is used as the final input, and sentiment classification is performed through the fully connected layer to obtain the final emotion recognition result.

[0151] The completing the information interaction between the context analysis module and the causal perception module is specifically: performing L-layer interaction between the context analysis module and the causal perception module information;

[0152] The L-layer interaction includes two modules in each layer: the context-dependent module is used to process the discourse structure information and the causal perception module is used to process the causal information perceived by the speaker; the output of each layer will exchange information between the context analysis module and the causal perception module through the cross-attention mechanism, so that the discourse structure information and the speaker perception information can be transmitted and integrated between different layers to complete the interaction of the current layer; finally, after the L-th layer interaction, the output and are concatenated as the final representation for sentiment prediction;

[0153] The calculation formula of the fully connected layer is as follows:

[0154] (twenty three);

[0155] (twenty four);

[0156] (25);

[0157] Among them, the features of the context analysis module and the causal perception module are combined through formula (23), and the linear transformation and The activation function generates the final feature representation for the next step of calculation; the last layer of the classification task is implemented through formula (24), which converts the model output into a probability estimate for each category; formula (25) is based on the probability distribution predicted by the model , select the category with the highest probability as the final prediction result; , is the weight matrix of the fully connected layer, and is the bias vector of the fully connected layer, is the activation function; is the predicted sentiment label, and the one with the highest probability is taken as the sentiment label; k represents the index of the category, and the value range of k is the set y of categories, that is, , Representation sample The probability of belonging to category k; Indicates The hidden representation of a sample after passing through the fully connected layer.

[0158] See also Figure 2 The present invention discloses a graph adversarial emotion recognition system for conversation analysis in an Internet of Things environment, comprising:

[0159] A collection module, wherein the collection module collects conversation text data containing multiple speakers;

[0160] An encoding module, wherein the encoding module encodes each utterance in the conversation text data based on a pre-trained text model to obtain a text feature representation of the conversation data;

[0161] An analysis module, wherein the analysis module constructs a context analysis module, analyzes the context dependency of the speaker during the conversation based on the discourse dependency graph, and aggregates information between text feature representations of the conversation data using a graph neural network;

[0162] A weight adjustment module, wherein the weight adjustment module constructs a causal perception module, extracts causal relationships in the conversation, and uses a self-attention mechanism to dynamically adjust the weights of causal clues in the conversation to obtain the causal relationships that have the greatest impact on sentiment changes;

[0163] An adversarial learning module, which adds a small perturbation to the input data to generate an adversarial sample, calculates the contrastive learning loss of the original sample and the adversarial sample based on the contrastive learning loss, and combines them to optimize the sentiment classification performance of the model;

[0164] An information interaction module, wherein the information interaction module obtains the adjacency matrix of the context analysis module and the causal perception module, and measures the difference between the adjacency matrix of the context analysis module and the causal perception module through a differential regularization term, and promotes information interaction between the context analysis module and the causal perception module based on a mutual cross-attention mechanism;

[0165] The emotion classification module, after completing the information interaction between the context analysis module and the causal perception module, splices the feature information of the context analysis module and the feature information of the causal perception module, and uses the generated feature representation as the final input, performs emotion classification through the fully connected layer, and obtains the final emotion recognition result.

[0166] Experimental results analysis

[0167] The proposed method is compared with several baseline models on four datasets, mainly analyzing the performance of emotion recognition in conversation scenarios and the robustness of the model in the face of imbalanced emotion labels. The specific results are shown in Table 1. The proposed method is compared with the currently popular baseline models, using Weighted-F1 as the evaluation index on the MELD, EmoryNLP and IEMOCAP datasets, and micro-F1 as the evaluation index on the DailyDialog dataset.

[0168] Experimental results show that the proposed method outperforms most of the baselines, showing significant performance improvements on all four datasets. These improvements highlight the effectiveness of the proposed method in capturing emotional nuances and improving the accuracy of emotion recognition in different conversational contexts. In the IEMOCAP dataset, the proposed method achieves an F1 score of 68.34%, slightly lower than the 69.22% of the SACL-LSTM model, but exceeds most other baseline models. The IEMOCAP dataset is characterized by long conversations, with an average of more than 50 utterances, which increases the challenge of capturing complex emotional cues and may lead to the gradient vanishing problem. The superiority of SACL-LSTM on this dataset is due to its combination of contrastive learning and hierarchical emotion modeling, which enables it to effectively manage emotional changes in extended conversations. On the EmoryNLP and MELD datasets, the proposed method achieves F1 scores of 39.86% and 66.99%, respectively, which are the best performers in both cases. Despite the short length of conversations in these datasets and the presence of multiple speakers, the proposed method uses context and causal dependency mechanisms to effectively integrate conversation structure and capture potential causal relationships. This integration enhances the model's ability to recognize complex emotional changes. For the DailyDialog dataset, where the sentiment labels are highly imbalanced - most utterances are labeled "neutral", our method achieves an F1 score of 59.71%, significantly outperforming the second best model, COSMIC, which scores 58.48%. This improvement may be due to the addition of adversarial samples and contrastive learning loss in our method, which enhances its generalization ability on imbalanced datasets, allowing it to more accurately identify non-neutral sentiment labels.

[0169] Table 1: Comparison of the proposed method with popular baseline models:

[0170]

[0171] In order to better understand the importance of each module in enhancing the model to capture emotional nuances and improve recognition accuracy on different datasets. An ablation study was conducted by gradually removing three key modules: context analysis module, causal perception module, and adversarial attack module to evaluate the contribution of each component to the overall performance of the model. The specific experimental results of this ablation study are shown in Table 2;

[0172] Table 2:

[0173]

[0174] When the context analysis module is removed, the performance of the proposed method decreases on all datasets, with a decrease of 3.27% on the MELD dataset. The MELD dataset includes multiple conversation participants and shorter conversation lengths. The context analysis module builds a commonsense knowledge graph, which plays a vital role in accurately capturing contextual structural dependencies in conversations and enhancing the understanding of the relationships between conversation participants. This result highlights the key role of the context analysis module in capturing and utilizing situational emotional information. Removing the causal awareness module also leads to a decrease in performance, especially a 1.80% decrease on the EmoryNLP dataset. The causal awareness module improves the model's reasoning ability by capturing implicit causal relationships in conversations. Without this module, the model cannot fully utilize internal and external causal clues between utterances, resulting in a decrease in emotion recognition performance. This emphasizes the importance of extracting and utilizing causal clues in conversations to enhance conversational emotion recognition, and demonstrates the effective complementarity between the context analysis module and the causal awareness module.

[0175] To evaluate the contribution of the adversarial attack module, the adversarial training component was removed from the model and experimental analysis was performed. The adversarial training module significantly improves the robustness and generalization ability of the model by generating adversarial samples. This is particularly evident on the DailyDialog dataset, where the performance of the model drops by 2.77% after removing the adversarial training module. Considering the label imbalance in the DailyDialog dataset, adversarial training effectively mitigates the negative impact of this imbalance on the model performance, further verifying the importance of the module.

[0176] Figure 3 and Figure 4 The effects of different adversarial perturbation magnitudes on overall model performance are shown. In both the IEMOCAP and DailyDialog datasets, moderate adversarial perturbations (0.1 for IEMOCAP and 2 for DailyDialog) significantly improve the F1 score of the model. This suggests that moderate adversarial perturbations can enhance the robustness and generalization of the model, enabling it to better capture emotional features. However, when the perturbation strength is too large, the F1 score of the model drops significantly. This is because excessive perturbations make the model's input too far from the true data distribution, resulting in performance degradation. Adversarial training consistently improves the performance of models on a variety of datasets. For the IEMOCAP dataset, even small perturbations are sufficient to interfere with contextual and emotional clues in long conversations, helping the model learn more powerful features. In contrast, the DailyDialog dataset consists of shorter conversations and exhibits a significant imbalance in emotional labels, requiring larger adversarial perturbations to effectively destroy the sparse information in rare emotional categories, thereby enhancing the robustness and generalization of the model.

[0177] Figure 5 The sensitivity of different datasets to network depth is analyzed. Generally speaking, moderately increasing the number of network layers can improve the performance of the model, enabling it to better capture emotional information in conversations. However, excessive network depth increases the complexity of the model, which may introduce unnecessary noise and impair the model's ability to effectively capture complex emotional dynamics in shorter conversations, leading to an overfitting tendency.

[0178] Figure 6 and Figure 7 In order to further analyze the robustness of the present invention in the face of uneven data distribution in a complex dialogue environment, the present invention is compared with the one without removing the adversarial module to verify its ability to adapt to uneven data distribution environments. Figure 7 It shows a higher error rate when classifying sadness and disgust categories, and has a clear tendency to misclassify these emotions as neutral emotions. The invention introduces an adversarial attack module into the model to enhance the robustness of the model in distinguishing difficult emotion categories and reduce the possibility of misclassifying sadness and disgust as neutral emotions.

[0179] The above are only preferred embodiments of the present invention and are not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A graph adversarial emotion recognition method for conversation analysis in IoT environments, characterized by: include: Collecting text data of conversations containing multiple speakers; Encode each utterance in the conversation text data based on the pre-trained text model to obtain the text feature representation of the conversation data; Build a context analysis module to analyze the context dependency of speakers during conversations based on the discourse dependency graph, and use graph neural networks to aggregate information between text feature representations of conversation data; Construct a causal perception module to extract causal relationships in conversations, and use the self-attention mechanism to dynamically adjust the weight of causal clues in conversations to obtain the causal relationships that have the greatest impact on sentiment changes; Add small perturbations to the input data to generate adversarial samples, calculate the contrastive learning loss of the original samples and adversarial samples based on contrastive learning loss, and combine them to optimize the sentiment classification performance of the model; Obtain the adjacency matrix of the context analysis module and the causal perception module, and measure the difference between the adjacency matrices of the context analysis module and the causal perception module through the differential regularization term, and promote the information interaction between the context analysis module and the causal perception module based on the mutual cross-attention mechanism; After completing the information interaction between the context analysis module and the causal perception module, the feature information of the context analysis module and the feature information of the causal perception module are spliced, and the generated feature representation is used as the final input. Sentiment classification is performed through the fully connected layer to obtain the final emotion recognition result.

2. The method for graph adversarial emotion recognition for conversation analysis in an Internet of Things environment according to claim 1 is characterized in that: After including the conversation text data of multiple speakers, the text data is preprocessed by word segmentation, removal of stop words and punctuation marks.

3. The method for graph adversarial emotion recognition for conversation analysis in an Internet of Things environment according to claim 2 is characterized in that: The pre-trained text model is used to encode each utterance in the conversation text data to obtain the text feature representation of the conversation data. Specifically, RoBERTa-large is selected as the utterance encoder, the conversation data is preprocessed into a format acceptable to the model, each speech is marked as a separate text unit, and tokenization is performed; given an input sequence ; ; RoBERTa-large uses the Transformer structure to encode it: （1）； in, The feature representation of the conversation data, Represents a text sequence; extracts the feature representation of each speech from the output of the last self-attention layer, and selects The labeled output represents the feature representation of the entire utterance.

4. The method for graph adversarial emotion recognition for conversation analysis in an Internet of Things environment according to claim 3 is characterized in that: The context analysis module is constructed to analyze the context dependency of the speaker during the conversation based on the discourse dependency graph, and to aggregate the information between the text feature representations of the conversation data using the graph neural network, specifically: Each node in the discourse dependency graph corresponds to a basic discourse unit, and the node set is ; The node set represents all the speeches in the conversation, and each node Corresponding to one of the features of the conversation data ; Edge set Representation Node and The dependency relationship between them, where each edge corresponds to a discourse dependency type; For each pair of adjacent nodes and , calculate the attention coefficient between each pair of adjacent nodes , the calculation formula is as follows: （2）； in, Indicates that in the dependency Next, the target discourse To the neighbors The weight of the edge; the dependency r represents the type of edge between two nodes; the edge types include: comment, clarification question, elaboration, confirmation, continuation, explanation, condition, question-answer pair, alternation, question elaboration, result, background, narrative, correction, parallel and contrast; is a learnable parameter weight vector, is the linear weight matrix under the attention mechanism, and The nodes are and nodes The feature representation of represents the vector concatenation operation, and LeakyReLU is the activation function; In order to study the relationship between the attention coefficients among all neighboring points, normalization is performed based on the softmax function; the final attention coefficient For adjacent nodes The features of the node are weighted summed to obtain Context-dependent features , specifically: （3）； in, Representation Node Through dependencies A set of connected neighbor nodes.

5. The method for graph adversarial emotion recognition for conversation analysis in an Internet of Things environment according to claim 4 is characterized in that: The causal relationship in the conversation is extracted, and the weight of the causal clues in the conversation is dynamically adjusted using the self-attention mechanism to obtain the causal relationship that has the greatest impact on the change in emotion. Specifically: The causal relationships in the conversation are extracted based on the COMET model. The causal relationships include internal causal relationships and external causal relationships. The internal causal relationships and external causal relationships are input into the COMET network. The last encoding layer of the COMET model fuses the contextual information from the input events and relationship types, extracts the hidden state representation of the encoding layer, and uses it as the causal clue C. The causal clues are mapped as edges in the conversation graph, indicating the causal dependency between speeches. The self-attention mechanism is used to calculate the weights of these clues in different speeches. Through the self-attention layer, the model can adjust the attention distribution between different speeches. The larger the weight of the edge, the stronger the impact of the causal relationship on the emotional change, and the more attention is focused on the relevant nodes. The calculation formula is as follows: （4）； （5）； in, Indicates that Converted into a query vector, Indicates that Converted into a key vector, is a linear transformation; Represents the common sense knowledge graph obtained from ATOMIC and the discourse , relevant internal causal cues; a mask is a mask that ensures that the retrieval process only considers previous utterances by the same speaker; is a mapping function that maps the index of an utterance to the corresponding speaker; Indicates the dimension of the vector, which is used for normalization; in addition to considering the causal clues within the same speaker, it is also necessary to calculate the causal clue association scores between different speakers : （6）； （7）； in, The cross-speaker causal clue score is calculated, where the causal clues come from the interaction between different speakers; after obtaining the weight scores of internal and external causal clues, they are combined to comprehensively consider all potential causal relationships and finally form a causal-aware context representation: （8）； （9）； in, and denote the weights of causal cues between internal and external speakers, respectively; and Representation and discourse the collection of utterances internal and external to the speaker; The causal cue characteristics are the comprehensive weights of internal and external causal cues.

6. The method for graph adversarial emotion recognition for conversation analysis in an Internet of Things environment according to claim 5 is characterized in that: The method adds a small perturbation to the input data to generate an adversarial sample, calculates the contrastive learning loss of the original sample and the adversarial sample based on the contrastive learning loss, and combines them to optimize the sentiment classification performance of the model. Specifically, the adversarial sample is generated by calculating the gradient of the input data: （10）； (11)； (12)； Among them, formula (10) is to obtain the input sample The gradient during training , Represents the input sample The gradient calculation of Represents the conditional probability distribution of the model prediction; Formula (11) is based on the gradient information of the input Generate a perturbation vector and normalize it to ensure that the size of the adversarial perturbation is controlled within Within the range; through gradient information Determines how to modify the input sample To increase the loss function, thereby "tricking" the model into making wrong predictions; Represents the disturbance amplitude, which is used to control the size of the counter-disturbance. Represents the gradient of Norm; Formula (12) is the input sample Add adversarial perturbations to ; The contrastive learning loss is introduced to calculate the contrastive learning loss of the original sample and the adversarial sample respectively, and the contrastive loss is calculated by the following formula: （13）； (14)； Among them, formula (13) is the contrast loss of the original sample; formula (14) is the contrast learning loss of the adversarial sample; Indicates the samples in the current batch The index set of all samples belonging to the same category; Indicates all sample indices in the current sample, except for sample ; Indicates The feature vector of the sample With n The feature vector of the sample The similarity between is the temperature parameter, used to adjust the distribution of similarity; Indicates The feature vector of the sample and the feature vector of the nth sample The similarity between the two after adding noise adversarial; after calculating the contrastive learning loss of all adversarial samples, it is combined with the loss of the original sample to obtain the final contrastive loss function. The specific formula is as follows: (15)。 7. The method for graph adversarial emotion recognition for conversation analysis in an Internet of Things environment according to claim 6 is characterized in that: The adjacency matrix of the context analysis module and the causal perception module is obtained as follows: Causal clue features A collection of causal perception modules to obtain the adjacency matrix ; Based on context-dependent features A collection of to obtain the adjacency matrix of the context analysis module ; The difference between the adjacency matrices of the context analysis module and the causal perception module is measured through the differential regularization term, specifically: (16)； in, Represents the Frobenius norm, which is used to measure the difference between the outputs of the context analysis module and the causal perception module.

8. The graph adversarial emotion recognition method for conversation analysis in an Internet of Things environment according to claim 7 is characterized in that: The cross-attention mechanism is used to promote information interaction between the context analysis module and the causal perception module, specifically: The cross-attention mechanism is combined with the information interaction between the context analysis module and the causal perception module to capture the interactive relationship between the context-dependent feature vector and the causal clue feature vector. Specifically: （17）； （18）； （19）； （20）； in, and is the attention matrix; Causal clue features Projection to context-dependent features In the above, it is emphasized that each discourse focuses on the characteristics of different clues. Indicates that context-dependent features Projection onto causal cue features superior; represents the final context feature representation enhanced by causal clue features; represents the final causal cue feature representation enhanced by context-dependent features; and is a learnable parameter matrix; the training goal is to minimize the following overall objective function: (21)； (22)； in, and is the regularization coefficient and the weight coefficient of contrastive learning; The standard cross entropy loss is used to measure the difference between the model prediction and the true label; B represents the number of dialogues, Indicates The first in the conversation The predicted probability of a statement; is the number of utterances in the Kth dialogue. In different dialogues, the number of utterances Not fixed.

9. The method for graph adversarial emotion recognition for conversation analysis in an Internet of Things environment according to claim 8, characterized in that: After completing the information interaction between the context analysis module and the causal perception module, the feature information of the context analysis module and the feature information of the causal perception module are spliced, and the generated feature representation is used as the final input, and the emotion classification is performed through the fully connected layer to obtain the final emotion recognition result, which is specifically: The completing the information interaction between the context analysis module and the causal perception module is specifically: performing L-layer interaction between the context analysis module and the causal perception module information; The L-layer interaction includes two modules in each layer: the context-dependent module is used to process the discourse structure information and the causal perception module is used to process the causal information perceived by the speaker; the output of each layer will exchange information between the context analysis module and the causal perception module through the cross-attention mechanism, so that the discourse structure information and the speaker perception information can be transmitted and integrated between different layers to complete the interaction of the current layer; finally, after the L-th layer interaction, the output and are concatenated as the final representation for sentiment prediction; The calculation formula of the fully connected layer is as follows: （23）； (24)； (25)； Among them, the features of the context analysis module and the causal perception module are combined through formula (23), and the linear transformation and The activation function generates the final feature representation for the next step of calculation; the last layer of the classification task is implemented through formula (24), which converts the model output into a probability estimate for each category; formula (25) is based on the probability distribution predicted by the model , select the category with the highest probability as the final prediction result; , is the weight matrix of the fully connected layer, and is the bias vector of the fully connected layer, is the activation function; is the predicted sentiment label, and the one with the highest probability is taken as the sentiment label; k represents the index of the category, and the value range of k is the set y of categories, that is, , Representation sample The probability of belonging to category k; Indicates The hidden representation of a sample after passing through the fully connected layer.

10. A graph adversarial emotion recognition system for conversation analysis in IoT environments, characterized by: include: A collection module, wherein the collection module collects conversation text data containing multiple speakers; An encoding module, wherein the encoding module encodes each utterance in the conversation text data based on a pre-trained text model to obtain a text feature representation of the conversation data; An analysis module, wherein the analysis module constructs a context analysis module, analyzes the context dependency of the speaker during the conversation based on the discourse dependency graph, and aggregates information between text feature representations of the conversation data using a graph neural network; A weight adjustment module, wherein the weight adjustment module constructs a causal perception module, extracts causal relationships in the conversation, and uses a self-attention mechanism to dynamically adjust the weights of causal clues in the conversation to obtain the causal relationships that have the greatest impact on sentiment changes; An adversarial learning module, which adds a small perturbation to the input data to generate an adversarial sample, calculates the contrastive learning loss of the original sample and the adversarial sample based on the contrastive learning loss, and combines them to optimize the sentiment classification performance of the model; An information interaction module, wherein the information interaction module obtains the adjacency matrix of the context analysis module and the causal perception module, and measures the difference between the adjacency matrix of the context analysis module and the causal perception module through a differential regularization term, and promotes information interaction between the context analysis module and the causal perception module based on a mutual cross-attention mechanism; The emotion classification module, after completing the information interaction between the context analysis module and the causal perception module, splices the feature information of the context analysis module and the feature information of the causal perception module, and uses the generated feature representation as the final input, performs emotion classification through the fully connected layer, and obtains the final emotion recognition result.

Images