Conversation emotion recognition method and system based on dynamic complementary graph convolutional network

Abstract

This disclosure relates to the technical field of conversation emotion recognition. It proposes a conversation emotion recognition method and system based on a dynamic complementary graph convolutional network. This method uses external knowledge to supplement the contextual information of the utterance itself. To prevent redundancy and loss of utterance information during contextual dependency modeling, the system creatively proposes the concept of utterance density. This concept dynamically determines the dependency structure within an utterance, fully integrating contextual information into the representation of each sentence. In this process, graph convolution is used to update the utterance representation, ultimately enabling each sentence to incorporate its own unique knowledge and contextual background, thereby achieving more accurate conversation emotion recognition.

Description

Technical Field

[0001] The present disclosure relates to the technical field related to conversation emotion recognition, and more specifically, to a conversation emotion recognition method and system based on a dynamic complementary graph convolutional network. Background Art

[0002] The statements in this section merely provide background information related to the present disclosure and do not necessarily constitute prior art.

[0003] In recent years, with the rapid development of computer technology and social networks, intelligent devices such as conversational robots, with their powerful human-computer interaction capabilities, have gradually replaced traditional human workers in many application areas, such as smart healthcare, early childhood education, and various service platforms. To ensure the logical communication of conversational robots, it is necessary not only to evaluate the dialogue settings but also the emotions expressed by humans during the conversation. Therefore, research on emotion detection in conversations is becoming increasingly important. Emotion is an important biological attribute of humans, and conversation is a common medium for expressing this attribute. Emotion recognition in conversations aims to identify the personal emotions hidden behind every sentence in the conversation. With the high demand for human-computer interaction technology in intelligent devices, such as robot conversation technology, accurate emotion recognition can enhance the user experience of human-computer interaction.

[0004] The key to emotion recognition in conversations is to utilize limited utterance information to create a richer representation that accurately reflects the speaker's real-time emotions. However, emotion recognition in speech differs from conventional text-based emotion recognition. Conversations inherently lack information, making it difficult to accurately discern emotion. Therefore, many approaches incorporate external knowledge and utilize various methods to obtain contextual information about the conversation in order to obtain more comprehensive emotional information.

[0005] During their research, the inventors discovered that different conversations require different contexts. Recurrent neural network-based approaches only focus on information close to the utterance and lack the ability to integrate more distant context. Graph-based modeling approaches, some with fixed window sizes and others with external structures, prevent the edges in the utterance graph from dynamically adjusting to the utterance, limiting the acquisition of contextual information. Summary of the Invention

[0006] In order to solve the above problems, the present disclosure proposes a conversation emotion recognition method and system based on a dynamic complementary graph convolutional network, which can fully capture contextual information and external knowledge, and dynamically determine the dependency structure within the discourse through a dynamically adjusted undirected graph, thereby achieving more accurate conversation emotion recognition.

[0007] In order to achieve the above objectives, the present disclosure adopts the following technical solutions:

[0008] One or more embodiments provide a method for conversation emotion recognition based on a dynamic complementary graph convolutional network, comprising the following steps:

[0009] Obtain the conversation text to be recognized, encode the conversation text and context-aware speech separately, extract the conversation text context information, and obtain the speech representation of the conversation;

[0010] Abstractly represent the words in the text to be recognized based on the relevant relationships in the external knowledge graph, and then encode them to obtain the knowledge representation of the conversation;

[0011] Construct a discourse density graph based on the speaker's discourse density, as well as a discourse knowledge graph related to knowledge representation. Dynamically determine the dependency structure within the discourse, fully integrate contextual information and external knowledge into the discourse representation of each sentence, and update the discourse representation based on graph convolution.

[0012] Classification is performed based on the updated discourse representation to obtain the conversation emotion category.

[0013] One or more embodiments provide a conversation emotion recognition system based on a dynamic complementary graph convolutional network, including:

[0014] The speech encoder is configured to obtain the conversation text to be recognized, encode the conversation text and the context-aware speech separately, extract the context information of the conversation text, and obtain the speech representation of the conversation;

[0015] Knowledge encoder: This encoder is configured to abstractly represent the words in the text to be recognized based on the relevant relationships in the external knowledge graph, and then encode them to obtain the knowledge representation of the conversation;

[0016] Discourse Representation Update Module: This module constructs a discourse density graph based on the speaker's discourse density, as well as a discourse knowledge graph related to knowledge representation. It dynamically determines the dependency structure within the discourse, fully integrates contextual information and external knowledge into the discourse representation of each sentence, and updates the discourse representation based on graph convolution.

[0017] Classification module: Classification is performed based on the updated discourse representation to obtain the dialogue sentiment category.

[0018] An electronic device includes a memory and a processor, and computer instructions stored in the memory and executed on the processor. When the computer instructions are executed by the processor, the steps of the above method are completed.

[0019] A computer-readable storage medium is used to store computer instructions, and when the computer instructions are executed by a processor, the steps of the above method are completed.

[0020] Compared with the prior art, the present invention has the following beneficial effects:

[0021] This paper innovatively uses the concept of discourse density to construct a graph structure between conversations. Through a dynamically adjusted undirected graph, it dynamically determines the dependency structure within the discourse, fully integrates contextual information and external knowledge into the representation of each sentence, and uses graph convolution to update the representation of the discourse in this process. Ultimately, each sentence can contain the knowledge unique to the discourse itself, the contextual background, and external knowledge, thereby achieving more accurate conversation emotion recognition.

[0022] The advantages of the present disclosure and additional advantages will be described in detail in the following specific embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] The accompanying drawings, which constitute a part of the present disclosure, are used to provide a further understanding of the present disclosure. The exemplary embodiments of the present disclosure and their descriptions are used to explain the present disclosure but do not constitute a limitation of the present disclosure.

[0024] Figure 1 This is an information transmission diagram between multiple people in a conversation when the fixed window is 3 in the existing conversation emotion recognition in Example 1 of the present disclosure;

[0025] Figure 2 1 is a flow chart of a method for identifying emotions in conversations according to Embodiment 1 of the present disclosure;

[0026] Figure 3 This is a simulation example of embodiment 1 of the present disclosure, in the process of speech density mapping, using different reference densities to obtain results;

[0027] Figure 4 This is a comparison chart of the F1 scores of graph convolution on the DKG branch on different data sets using the method of this embodiment in a simulation example of Example 1 of the present disclosure;

[0028] Figure 5 This is a comparison chart of the F1 scores of graph convolution on the DDG branch on different data sets using the method of this embodiment in a simulation example of embodiment 1 of the present disclosure. DETAILED DESCRIPTION

[0029] The present disclosure will be further described below with reference to the accompanying drawings and embodiments.

[0030] It should be noted that the following detailed descriptions are exemplary and intended to provide further explanation of the present disclosure. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the art to which the present disclosure belongs.

[0031] It should be noted that the terms used herein are only for the purpose of describing specific embodiments and are not intended to limit the exemplary embodiments according to the present disclosure. As used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. In addition, it should be understood that when the terms "comprising" and / or "including" are used in this specification, they indicate the presence of features, steps, operations, devices, components and / or combinations thereof. It should be noted that, in the absence of conflict, the various embodiments in the present disclosure and the features in the embodiments can be combined with each other. The embodiments will be described in detail below with reference to the accompanying drawings.

[0032] This paper proposes a novel conversation emotion recognition method based on dynamic complementary graph convolutional networks. It can extract as much background information as possible from limited conversation information and accurately identify the correct emotion of each sentence.

[0033] The background information in a conversation mainly exists in two forms: one is the external knowledge such as the speaker's cognition and intention hidden behind the words themselves, and the other is the contextual information in the conversation context. Existing methods often cause information redundancy when integrating context using a fixed window, and it is difficult to achieve deep integration of different knowledge behind the words themselves. Figure 1 As shown in the figure, a graph structure with a fixed window size of 3 is used to model the context. In this conversation, this leads to redundancy when the utterance receives contextual information. Using the same fixed window to receive information for different conversations will cause similar problems and fail to effectively transmit contextual information.

[0034] Therefore, in order to solve this problem, the present disclosure proposes a dynamic complementary graph convolutional network (DCGCN) for dialogue emotion recognition. In this method, in order to fully enrich the information of the dialogue, external knowledge is used to supplement the background information of the discourse itself. At the same time, in order to prevent the redundancy and loss of discourse information in the dependency modeling of contextual information, the concept of discourse density is creatively proposed. The discourse density is used to dynamically determine the dependency structure within the discourse, and the contextual information is fully integrated into the representation of each sentence. In this process, graph convolution is used to update the representation of the discourse, so that each sentence can eventually contain the knowledge and contextual background unique to the discourse itself, thereby achieving more accurate dialogue emotion recognition. The following is an explanation with specific embodiments.

[0035] Example 1

[0036] In the technical solutions disclosed in one or more embodiments, Figure 2 As shown in FIG, a conversation emotion recognition method based on a dynamic complementary graph convolutional network includes the following steps:

[0037] Step 1: Obtain the dialogue text to be recognized, encode the dialogue text and context-aware utterances separately, extract the dialogue text context information, and obtain the utterance representation of the dialogue;

[0038] Step 2: Abstractly represent the words in the text to be recognized based on the relevant relationship r in the external knowledge graph, and encode them to obtain the knowledge representation of the conversation;

[0039] Step 3: Construct a discourse density graph based on the speaker's discourse density, as well as a discourse knowledge graph related to knowledge representation. Dynamically determine the dependency structure within the discourse, fully integrate contextual information and external knowledge into the discourse representation of each sentence, and update the discourse representation based on graph convolution.

[0040] Step 4: Classify based on the updated discourse representation to obtain the dialogue emotion category.

[0041] In this embodiment, the concept of discourse density is creatively used to construct a graph structure between conversations. The dependency structure within the discourse is dynamically determined through a dynamically adjusted undirected graph, and contextual information and external knowledge are fully integrated into the representation of each sentence. In this process, graph convolution is used to update the representation of the discourse, so that each sentence can ultimately contain the knowledge unique to the discourse itself, the contextual background, and external knowledge, thereby achieving more accurate conversation emotion recognition.

[0042] Problem Definition:

[0043] The data for this task is a transcript of a conversation and the speaker information of each utterance. The goal is to identify the emotion of each utterance from a set of predefined emotions. In conversation emotion recognition, the data consists of multiple conversations {c1, c2, ..., c n}, each dialogue consists of several utterances c i =[u1,u2,…,u m ] and emotional labels where S represents the category of emotion. For a discourse, it consists of some words u i ={w1,w2,…,w L}Composition. Session c i Each utterance in is uttered by a certain speaker and can be expressed as p(c i )=[p(u1),p(u2),…,p(u m )],p(u i )∈P, i=1.2.3...m, where P represents the category or name of the speaker. Therefore, the whole problem can be expressed as obtaining the sentiment label of each utterance based on the context and speaker information in a conversation:

[0044] Optionally, emotion labels may include: happy, frustrated, surprised, neutral, fearful, disgusted, angry, etc.

[0045] In step 1, utterance-level features are extracted from the acquired conversation to be recognized. The utterance editor used can be the pre-trained language model RoBERTa. RoBERTa Large follows the original BERT Large architecture, with 24 layers, 16 self-attention heads in each block, a hidden dimension of 1024, and a total of 355M parameters.

[0046] Before use, fine-tune the pre-trained language model RoBERTa. For each utterance u i ={w1,w2,…,w L}, a special token [CLS] is attached to the beginning of the utterance, resulting in the input sequence {[CLS],w1,w2,…,w L}; Input sequence {[CLS],w1,w2,…,w L Fine-tune the pre-trained RoBERTa model on the utterance-level sentiment classification task and input the [CLS] token from the last hidden layer into a pooling layer to obtain the sentiment classification result;

[0047] After the fine-tuning process, in order to obtain each utterance-level feature vector u′ represented by the [CLS] token i , with {[CLS],w1,w2,···,w L Enter each utterance in the same format:

[0048] u′ i =RoBERTa([CLS],w1,w2,…,w L ) (1)

[0049] in, and d m is the dimension of the hidden state in RoBERTa, and the [CLS] tokens from the last four layers are averaged to obtain the utterance-level feature vector for each utterance. Each utterance vector u′ is then i Transformed to d by linear projection h The dimension of the hidden state changes after the dimension is changed by d m Transformed to d h . Define each session c i The vectorized representation of is {s1,s2,…,s m}.

[0050] Furthermore, it is necessary to extract the order information in the utterances, which is important for predicting the sentiment label of each utterance. This is to extract the contextual information of the conversation text in step 1, including the following:

[0051] Step 11: In order to obtain the order information in the utterance, a separate encoder is used to encode the features of the conversation text context-aware utterance;

[0052] Step 12: Use a bidirectional long short-term memory (Bi-LSTM) network to encode the sequential contextual information of the conversation text. The encoded utterance of each conversation is represented as:

[0053]

[0054] In step 2, external knowledge is obtained and encoded. The external knowledge can be obtained using the ATOMIC knowledge graph, which contains 870,000 pieces of reasoning common sense. The architecture of the ATOMIC common sense graph consists of three types of relations, nine relations, and 877K (event, relation, event) triples.

[0055] In this embodiment, the pre-trained COMET model can be used on an ATOMIC knowledge graph, and the phrase generation decoder module can be discarded. i As the subject, connect it with the relation phrase r in the ATOMIC knowledge graph. Passed to the COMET encoder and the activations are extracted from the last time step.

[0056] Specifically, in this embodiment, four relationships of the ATOMIC spectrum are used, as follows:

[0057] X reaction: X's feelings after this event;

[0058] X want: what X wants to do after this event;

[0059] X intent: why X wants to initiate this event;

[0060] Other reaction: How other people feel after this event;

[0061] Among them, X is the speaker and Other is the listener.

[0062] Optionally, use the relationship of the ATOMIC knowledge graph to extract features from the recognized conversation text and the word u in the recognized text i Specifically, in this embodiment, performing this feature extraction operation will generate four different vectors for each utterance in the conversation (corresponding to four different relationships). These vectors are 768-dimensional. The abstract representation is as follows:

[0063]

[0064] In step 3, contextual information and external knowledge are fully integrated into the discourse representation of each sentence, and the discourse representation is updated based on graph convolution, which includes the following steps:

[0065] Step 31: Construct a discourse density graph based on the speaker's discourse density and discourse representation, dynamically determine the dependency structure within the discourse, incorporate contextual information into the discourse representation of each sentence, and update the discourse representation based on graph convolution to obtain discourse features with contextual information;

[0066] Step 32: Construct a discourse knowledge graph based on the speaker's discourse density, discourse representation, and knowledge representation. Dynamically determine the dependency structure within the discourse, incorporate contextual information and external knowledge into the discourse representation of each sentence, and update the discourse representation based on graph convolution to obtain discourse features with external knowledge.

[0067] The following is a detailed description of step 3.

[0068] To effectively capture information from the conversational context and simulate the mutual influence in real conversations, a discourse density graph (DDG) was designed to facilitate information propagation between discourses. DDG = (V, E, W).

[0069] The nodes in DDG are the representations of the discourse in the conversation, i.e. V = {h1,h2,...,h m};edge(i,j,A i,j )∈E represents the i The information transmitted to h j , where A i,j ∈W is the edge weight information; the edge weight size W∈(0,1),

[0070] Discourse density maps can include:

[0071] Node: Each node in the discourse density graph is represented by the original discourse feature h i Represented and initialized to A primitive utterance is represented by h i is a node. Given a conversation containing N utterances, a graph with N nodes can be constructed.

[0072] Edges: represent the connection between each utterance and other utterances in the same conversation.

[0073] Furthermore, to represent the connections between nodes in the discourse density graph, the concept of discourse density is constructed to determine whether an discourse propagates information to another discourse.

[0074] Specifically, the concept of speech density is: the proportion of the speech of a certain speaker in a conversation to the total number of speeches in the entire conversation is the speech density of the speaker.

[0075] The speech density D = [d1, d2, ..., d a ], a is the number of speakers appearing in this conversation, and is the speech density of the speaker in this conversation.

[0076] A further technical solution to determine the connection relationship between discourses in DDG includes the following steps:

[0077] (1) According to the location of the target utterance, the entire conversation is divided into forward conversation and backward conversation.

[0078] (2) Calculate the speech density of the corresponding speaker for the two segments of the forward dialogue and the backward dialogue respectively, and obtain the minimum speech density in the two segments of the dialogue;

[0079] The minimum density of speaker utterances in the forward dialogue is:

[0080] The minimum density of speakers’ speech in the backward dialogue is:

[0081] (3) Perform forward and backward attention on the target utterance, search forward and backward from the target utterance according to the dialogue time until the speaker utterance density corresponding to the searched utterance is not less than the minimum value in the corresponding direction, and establish connections with the utterance nodes in the search results that are less than the minimum value of the utterance density in the corresponding direction.

[0082] When the target utterance is directed towards other utterances, it is divided into the forward attention process and the backward attention process. Taking the forward attention process as an example, specifically:

[0083] First, an empty list L is created. The target speech is noticed from the self position to the earlier speech in turn, and the speech density of the speaker in the list L is calculated. If the speech density of the speaker in L is When , the attention words are added to the list L and the search continues. When a certain word is added, there are Then attention will no longer continue to move forward. At this time, the discourse in L is the content that needs to be integrated with the target discourse.

[0084] Optionally, the discourse nodes are connected, that is, each node is connected through an edge, wherein the edge weight is calculated according to the similarity between the two nodes, and the higher the similarity, the greater the edge weight.

[0085] Edge weight: If the similarity between two nodes is higher, the information interaction between them is more important, and the edge weight between them should be higher. In order to capture the similarity between node representations, angular similarity can be used to represent the edge weight between two nodes.

[0086] The edge weight is calculated as:

[0087]

[0088] Among them, n i and n j Represents the feature representation of the i-th and j-th nodes in the graph.

[0089] It can be seen that the node connection relationship in this embodiment is determined based on the speaker's speech density. Corresponding to different conversations, the speech density graph is different, which realizes the dynamic determination of the dependency structure within the speech, fully integrates the context information into the representation of each sentence, and uses graph convolution to update the representation of the speech in this process, so that each sentence can finally contain the knowledge and contextual background unique to the speech itself, thereby achieving more accurate conversation emotion recognition.

[0090] Specifically, such as Figure 2 As shown in the figure, l layer is set in the graph convolution.

[0091] In this embodiment, the speech density map constructed by speech density no longer simply sets the size of a fixed window. Instead, each utterance can dynamically adjust the conversation information that needs to be paid attention to based on the different interlocutors and the location of the utterance, thereby achieving effective transmission of contextual information.

[0092] In step 3, in order to incorporate sufficient background knowledge into the dialogue representation and to leverage the contextual information of the discourse to influence knowledge fusion, a discourse knowledge graph (DKG) is constructed to carry out this process, where DKG = (V, E, W).

[0093] The nodes in DKG are the discourse and knowledge representations in the conversation, that is, V = [H, K], where H = {h1,h2,...,h m}, where the discourse representation is the dialogue text feature, Represents the knowledge nodes, that is, the knowledge representation of the dialogue. i,j )∈E represents the information transmission between two nodes, where e i,j ∈W is the weight information of the edge, and the weight size of the edge is W∈(0,1).

[0094] The process of constructing a discourse knowledge graph is as follows:

[0095] Node: Each discourse node in the discourse knowledge graph includes discourse representation and knowledge representation corresponding nodes; that is, it includes discourse representation nodes, which are represented by circles in the figure, and knowledge representation nodes. Figure 2 Represented by boxes.

[0096] Each discourse node in the discourse knowledge graph is represented by the original discourse feature h i Represented and initialized to The knowledge nodes are represented by k i,r Represented and initialized to Therefore, given a conversation with N utterances, we construct a graph with 5N nodes.

[0097] Edges: The discourse representation node of each utterance is connected to the corresponding knowledge representation node, and edges are created between the discourse representation nodes based on the speaker's discourse density.

[0098] Specifically, the discourse knowledge graph contains two types of nodes. To enable interaction between knowledge and discourse in context, different connection strategies are used for these two types of nodes. Edges are created between discourse nodes in the same manner as in the discourse density graph. Each utterance is connected to its corresponding knowledge, facilitating the transfer of knowledge information to discourse.

[0099] Edge weight: In this section, you can define the similarity between nodes in the same way as the weight setting in the discourse density graph. That is, the higher the node similarity, the greater the edge weight.

[0100] There are two types of edges in the discourse knowledge graph: 1) edges connecting discourse representation nodes, and 2) edges connecting discourse representation nodes with discourse knowledge nodes. Different edge weight strategies can be used. For the first type of edge, the edge weight is calculated as formula (4), and for the second type of edge, the edge weight is calculated as:

[0101]

[0102] Among them, n i and n j represents the feature representation of the i-th and j-th nodes in the graph, and γ is a hyperparameter.

[0103] The discourse representation is updated through the discourse knowledge graph, and graph convolution is used to update the discourse representation in this process, so that each sentence can ultimately contain the knowledge unique to the discourse itself, contextual background and external knowledge, thereby achieving more accurate conversation emotion recognition.

[0104] Specifically, such as Figure 2 As shown in the figure, t layer is set in the graph convolution.

[0105] In this embodiment, the constructed discourse density graph and discourse knowledge graph are undirected graphs, respectively, and the relationship between discourse representations and the relationship between discourse representations and external knowledge are obtained.

[0106] The discourse representation is updated based on graph convolution. Specifically, a deep graph convolutional network is constructed to further encode context dependencies and extract deeper features.

[0107] Specifically, given an undirected graph G = (V, E, W), let is the renormalized graph Laplacian matrix of G:

[0108]

[0109] in represents the adjacency matrix, represents the diagonal matrix of graph G, represents the identity matrix. The DDG iterations from different layers can be expressed as:

[0110]

[0111] Among them: α and β l are two hyperparameters, σ represents the activation function, W l is a learnable weight matrix. To ensure that the decay of the weight matrix increases adaptively when more layers are stacked, set Where η is also a hyperparameter. To the first layer A residual connection is added to represent In the same way, the identity mapping is added to the weight matrix With such a residual connection, the speech graph of this embodiment can obtain deeper information to further improve performance.

[0112] In this embodiment, the relationship between discourse representations is determined by discourse density graph and the discourse representation is updated by graph convolution to obtain discourse features with contextual information. The relationship between discourse representations and the dependency between discourse representations and knowledge representations are determined through the knowledge density graph. The discourse representation is updated through graph convolution to obtain a discourse representation that includes external knowledge.

[0113] Before classification, concatenate discourse features with contextual information and discourse representations that include external knowledge To generate the final feature representation of each utterance:

[0114]

[0115] Then ei The input is fed into a multi-layer perceptron (MLP) with fully connected layers to predict the sentiment label y of the utterance i :

[0116] l i =RELU(W l e i +b l ) (9)

[0117] P i =Softmax(W smax l i +b smax ) (10)

[0118]

[0119] The above process can be realized by Figure 2 The network implementation shown is a dynamic complementary graph convolutional neural network (DCGCN), which includes a discourse encoder and a knowledge encoder, a convolutional update of the discourse density graph, a convolutional update of the discourse knowledge graph, and a sentiment prediction module.

[0120] Speech encoder: configured to obtain the dialogue text to be recognized, encode the dialogue text and the context-aware speech respectively, extract the context information of the dialogue text, and obtain the speech representation of the dialogue;

[0121] Knowledge encoder: This is configured to abstractly represent the words in the text to be recognized based on the relevant relations r in the external knowledge graph, and encode them to obtain the knowledge representation of the conversation;

[0122] The convolutional update module of the utterance density graph is configured to construct an utterance density graph based on the number of speakers' utterances and utterance representations, dynamically determine the dependency structure within the utterance, incorporate contextual information into the utterance representation of each sentence, and update the utterance representation based on graph convolution to obtain utterance features with contextual information;

[0123] The convolutional update module of the discourse knowledge graph is configured to construct a discourse knowledge graph based on the number of speakers' utterances, utterance representations, and knowledge representations, dynamically determine the dependency structure within the utterance, incorporate contextual information and external knowledge into the utterance representation of each sentence, and update the utterance representation based on graph convolution to obtain utterance features with external knowledge;

[0124] Emotion prediction module: configured to perform classification based on the updated speech representation to obtain the conversation emotion category

[0125] During training, categorical cross entropy and L2 regularization can be used as loss functions during training:

[0126]

[0127] Where N is the number of dialogues, c(i) is the number of utterances in dialogue i, and P i,j is the probability distribution of the predicted sentiment label of utterance j in conversation i, y i,j is the expected class label of utterance j in conversation i, λ is the L2 regularization weight, and θ is the set of all trainable parameters. The network is trained using stochastic gradient descent with the Adam optimizer, and grid search is used to optimize the hyperparameters.

[0128] To demonstrate the effectiveness of the method in this embodiment, we conducted experiments evaluating the proposed model on four commonly used public benchmark datasets for conversational emotion recognition. The results demonstrate the effectiveness of the Dynamic Complementary Graph Convolutional Network (DCGCN), significantly outperforming other state-of-the-art methods on popular metrics. The proposed DCGCN achieves performance that surpasses baselines. Furthermore, through comprehensive evaluation and ablation studies, the advantages of the DCGCN proposed in this embodiment and the impact of its modules improve the performance of speech emotion recognition.

[0129] Results comparing the method in this example with the most advanced baselines. The test score is calculated based on the best score averaged over five rounds of the test set. As shown in Table 1, DCGCN achieves new state-of-the-art results across all datasets.

[0130] Table 1

[0131]

[0132] In Table 1, the horizontal axis is the data set:

[0133] IEMOCAP dataset: The full name is the Interactive Emotional Dyadic Motion Capture, which is a multimodal emotion recognition dataset;

[0134] The MELD dataset is primarily used for emotion recognition in conversations. Some papers also use it for emotional conversation generation or multimodal conversation generation. It contains 1,433 conversations and 13,708 sentences, providing three categories of coarse-grained emotion labels and seven types of fine-grained emotion labels. The videos are segmented into short videos divided by sentence, and the corresponding video can be retrieved by using the sentence index.

[0135] DailyDialog dataset: The DailyDialog dataset consists of 13,118 dialogues and 102,000 sentences, annotated with seven emotions, four types of dialogue actions (DAs), and 10 dialogue topics. It can be used for dialogue emotion recognition, dialogue action recognition, and emotional dialogue generation.

[0136] EmoryNLP dataset: From the American TV series "Friends", the emotion labels contained in this dataset are happy, neutral, strong, crazy, sad, fearful and calm.

[0137] DialogueRNN (2019): This is a recursion-based approach that models the conversational context using several recurrent neural networks.

[0138] KET (2019): This is a hierarchical transformer that uses their proposed graph attention model to extract information from the knowledge base for conversational modeling.

[0139] RGAT-POS (2020): is a relation-aware graph attention network that leverages the proposed relation position encoding to incorporate position information into conversation graphs.

[0140] DialogXL (2021): is a conversational relation model that helps with conversational sentiment recognition tasks by extracting relations in the conversation.

[0141] DAG-ERC (2021): This model establishes a graph structure based on different interlocutors and uses the idea of recurrent neural networks in the graph fusion process to achieve excellent results.

[0142] SKAIG (2021): This model utilizes psychological concepts in the process of using external common sense knowledge, realizes the integration of knowledge and discourse, and increases the interpretability of the model.

[0143] COSMIC (2020): This model also utilizes external knowledge, but it uses a bidirectional gated recurrent neural network to model the speaker state in a more detailed manner.

[0144] CoG-BART (2022): This model uses a pre-trained model to extract the content of the discourse in detail, and introduces contrastive learning to distinguish similar emotions in the conversation.

[0145] TODKAT (2021): This model identifies the topics and common sense knowledge behind the conversation and emphasizes conversation modeling based on these overall properties of the conversation (i.e., conversation topic and tone).

[0146] The model constructed in this embodiment performs well on four data sets, including the complete model and the model after removing certain common sense knowledge. The scores in Table 2 are F1 scores. The score comparison shows that the introduction of external common sense knowledge can greatly improve the recognition accuracy.

[0147] Table 2

[0148]

[0149] Figure 3 :In the process of constructing the discourse density map, different benchmark densities are used to obtain the results. The five points on the X-axis are the five benchmark points set according to the discourse density in the experiment, taking the minimum and maximum discourse density in the conversation, as well as their upper quartile, lower quartile and average. The Y-axis represents the F1 score of a certain dataset. In the discourse density map construction method, each discourse not only obtains the discourse of each individual appearing in the conversation, but also prevents redundancy caused by excessive individual discourse. Figure 3 It can be shown that the minimum discourse density graph proposed in this embodiment is an effective method to construct a discourse context graph.

[0150] Figure 4 and Figure 5 : Use different convolutional layers to model the performance of four datasets under different graph convolution branches. The X-axis represents the number of convolution layers; the Y-axis represents the F1 score. The broken lines of different shapes and lines represent the F1 scores of different datasets on the graph convolution branch. The corresponding shaded areas represent the differences from the complete model (the number of convolution layers of the two graphs in the complementary operation uses the same value). Figure 4 and Figure 5 It can be shown that any single discourse graph branch in the model proposed in this embodiment cannot fully capture sufficient context and external knowledge information. Specifically, during the fusion process of the discourse knowledge graph, the inclusion of external knowledge and the heavy weight assigned to it affect the representation of contextual information in the discourse knowledge graph, resulting in a loss of contextual background within the discourse knowledge graph. Furthermore, the lack of commonsense knowledge information in the discourse density graph branch leads to insufficient model performance. Therefore, the complementary operations provided are necessary.

[0151] Example 2

[0152] Based on Example 1, this embodiment provides a conversation emotion recognition method based on a dynamic complementary graph convolutional network, including:

[0153] The speech encoder is configured to obtain the conversation text to be recognized, encode the conversation text and the context-aware speech separately, extract the context information of the conversation text, and obtain the speech representation of the conversation;

[0154] Knowledge encoder: This encoder is configured to abstractly represent the words in the text to be recognized based on the relevant relationships in the external knowledge graph, and then encode them to obtain the knowledge representation of the conversation;

[0155] Discourse Representation Update Module: This module constructs a discourse density graph based on the speaker's discourse density, as well as a discourse knowledge graph related to knowledge representation. It dynamically determines the dependency structure within the discourse, fully integrates contextual information and external knowledge into the discourse representation of each sentence, and updates the discourse representation based on graph convolution.

[0156] Classification module: Classification is performed based on the updated discourse representation to obtain the dialogue sentiment category.

[0157] It should be noted here that the various modules in this embodiment correspond one-to-one to the various steps in Example 1, and the specific implementation processes are the same, which will not be repeated here.

[0158] Example 3

[0159] This embodiment provides an electronic device, including a memory and a processor, and computer instructions stored in the memory and running on the processor. When the computer instructions are run by the processor, the steps described in the method of embodiment 1 are completed.

[0160] Example 4

[0161] This embodiment provides a computer-readable storage medium for storing computer instructions. When the computer instructions are executed by a processor, the steps described in the method of embodiment 1 are completed.

[0162] The foregoing description is merely a preferred embodiment of the present disclosure and is not intended to limit the present disclosure. Those skilled in the art will readily appreciate that various modifications and variations are possible. Any modifications, equivalent substitutions, or improvements made within the spirit and principles of the present disclosure shall be included within the scope of protection of the present disclosure.

[0163] Although the above describes the specific implementation methods of the present disclosure in conjunction with the accompanying drawings, it is not intended to limit the scope of protection of the present disclosure. Those skilled in the art should understand that on the basis of the technical solution of the present disclosure, various modifications or variations that can be made by those skilled in the art without creative work are still within the scope of protection of the present disclosure.

Claims

1. A conversation emotion recognition method based on dynamic complementary graph convolutional network, characterized by: The steps include: Obtain the conversation text to be recognized, encode the conversation text and context-aware speech separately, extract the conversation text context information, and obtain the speech representation of the conversation; Abstractly represent the words in the text to be recognized based on the relevant relationships in the external knowledge graph, and then encode them to obtain the knowledge representation of the conversation; Construct a discourse density graph based on the speaker's discourse density, as well as a discourse knowledge graph related to knowledge representation. Dynamically determine the dependency structure within the discourse, fully integrate contextual information and external knowledge into the discourse representation of each sentence, and update the discourse representation based on graph convolution. Classify the conversation based on the updated speech representation to obtain the conversation emotion category; Discourse density is the ratio of the words spoken by a speaker in a conversation to the total number of words spoken in the entire conversation; The constructed discourse density graph includes nodes and edges connecting nodes; Node of the utterance density graph: a raw utterance representation For a node; Edges in the utterance density graph: represent the connection between each utterance and other utterances in the same conversation; The method for determining the connection relationship of discourses in the discourse density graph includes the following steps: According to the location of the target utterance, the entire conversation is divided into forward conversation and backward conversation; Calculate the speech density of the corresponding speaker for the two segments of the forward dialogue and the backward dialogue respectively, and obtain the minimum speech density in the two segments of the dialogue; Perform forward and backward attention on the target utterance, and search forward and backward from the target utterance according to the dialogue time until the speaker utterance density corresponding to the searched utterance is not less than the minimum value in the corresponding direction. Then establish connections with the utterance nodes in the search results that are less than the minimum value of the utterance density in the corresponding direction, and determine the edge weights based on the similarity of the nodes.

2. The method for conversation emotion recognition based on dynamic complementary graph convolutional network according to claim 1, characterized in that: Extract the contextual information of the conversation text, including the following: A separate encoder is used to encode features of the conversational text context-aware utterances; A bidirectional long short-term memory network is used to encode the sequential contextual information of the conversation text and obtain the encoded discourse representation of each sentence.

3. The method for conversation emotion recognition based on a dynamic complementary graph convolutional network according to claim 1, characterized in that: The process of fully integrating contextual information and external knowledge into the discourse representation of each sentence and updating the discourse representation based on graph convolution includes the following steps: Based on the speaker's speech density and speech representation, a speech density graph is constructed to dynamically determine the dependency structure within the speech. Contextual information is incorporated into the speech representation of each sentence, and the speech representation is updated based on graph convolution to obtain speech features with contextual information. A discourse knowledge graph is constructed based on the speaker's discourse density, discourse representation and knowledge representation. The dependency structure within the discourse is dynamically determined, and contextual information and external knowledge are integrated into the discourse representation of each sentence. The discourse representation is updated based on graph convolution to obtain discourse features with external knowledge.

4. The method for conversation emotion recognition based on a dynamic complementary graph convolutional network according to claim 1, characterized in that: The discourse knowledge graph includes nodes and edges connecting nodes. Each discourse node in the discourse knowledge graph includes discourse representation and nodes corresponding to knowledge representation. The discourse representation node of each discourse is connected to the corresponding knowledge representation node, and edges are created between discourse representation nodes based on the speaker's discourse density.

5. The method for conversation emotion recognition based on dynamic complementary graph convolutional network according to claim 1, characterized in that: For the discourse density graph and the updated discourse knowledge graph, a deep graph convolutional network is used to further encode contextual dependencies and extract deeper features.

6. The method for conversation emotion recognition based on dynamic complementary graph convolutional network according to claim 1, characterized in that: The updated connections of the graph convolution are passed through a multi-layer perceptron to determine the sentiment category.

7. A conversation emotion recognition system based on dynamic complementary graph convolutional network, characterized by: include: The speech encoder is configured to obtain the conversation text to be recognized, encode the conversation text and the context-aware speech separately, extract the context information of the conversation text, and obtain the speech representation of the conversation; Knowledge encoder: This encoder is configured to abstractly represent the words in the text to be recognized based on the relevant relationships in the external knowledge graph, and then encode them to obtain the knowledge representation of the conversation; Discourse Representation Update Module: This module constructs a discourse density graph based on the speaker's discourse density, as well as a discourse knowledge graph related to knowledge representation. It dynamically determines the dependency structure within the discourse, fully integrates contextual information and external knowledge into the discourse representation of each sentence, and updates the discourse representation based on graph convolution. Classification module: Classify the conversation based on the updated speech representation to obtain the conversation emotion category; Discourse density is the ratio of the words spoken by a speaker in a conversation to the total number of words spoken in the entire conversation; The constructed discourse density graph includes nodes and edges connecting nodes; Node of the utterance density graph: a raw utterance representation For a node; Edges in the utterance density graph: represent the connection between each utterance and other utterances in the same conversation; The method for determining the connection relationship of discourses in the discourse density graph includes the following steps: According to the location of the target utterance, the entire conversation is divided into forward conversation and backward conversation; Calculate the speech density of the corresponding speaker for the two segments of the forward dialogue and the backward dialogue respectively, and obtain the minimum speech density in the two segments of the dialogue; Perform forward and backward attention on the target utterance, and search forward and backward from the target utterance according to the dialogue time until the speaker utterance density corresponding to the searched utterance is not less than the minimum value in the corresponding direction. Then establish connections with the utterance nodes in the search results that are less than the minimum value of the utterance density in the corresponding direction, and determine the edge weights based on the similarity of the nodes.

8. An electronic device, characterized in that: The method comprises a memory and a processor, and computer instructions stored in the memory and executed on the processor, wherein when the computer instructions are executed by the processor, the steps of any one of the methods of claims 1 to 6 are completed.

9. A computer-readable storage medium, characterized in that Used to store computer instructions, which, when executed by a processor, complete the steps of any one of the methods of claims 1 to 6.

Images