Emotion classification method based on adaptive interaction graph network

Abstract

The application discloses an emotion classification method based on an adaptive interaction graph network, and comprises the following steps: constructing an adaptive interaction graph network, the structure of which comprises a feature extractor, a Gumbel-Softmax discrete sampler, an Utterance-level Encoder, a core component and an emotion classifier; calling a dialogue emotion dataset in a dialogue emotion database, inputting the multi-modal features into the model to start training; obtaining node modal features in the Utterance-level Encoder, then performing graph convolution operation through original GIN and GIN with NESP, and obtaining a final GIN prediction result by using the integration of GSP; after the node modal features and the final GIN prediction result are spliced, the spliced result is input into the emotion classifier, and after the holding loss and the minimum descent loss of the calculation model are calculated, the training is completed. The application can adaptively select specific single-modal and multi-modal relationships, and the accuracy and running speed of emotion classification are improved.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of human-computer signal recognition, and particularly relates to an emotion classification method based on an adaptive interactive graph network. BACKGROUND

[0002] Contextual information in dialogue provides important clues for sentiment analysis, which is confirmed in domain research. Unlike ordinary sentence-level sentiment analysis, the main feature of ERC is the complexity of interactive links in dialogue. According to the emotional dynamics in the dialogue, the ERC model needs to model the context dependency at the speaker level, including cycle-based networks, transformation-based networks and graph-based networks. However, it is challenging to capture the contextual interaction between modalities, which hinders the effective use of multi-modalities. Recent research work explores the modeling of single-modal and cross-modal interactions within the graph framework, taking advantage of the complementarity of different modalities in sentiment tracking. However, the graph-based fusion method tends to accumulate redundant information by aggregating context information in a fixed semantic space in each layer. This ultimately limits the ability to fully understand the context between modalities. Although the multi-modal dynamic fusion network (MMDFN) is a graph-based architecture that considers the interaction between different modalities and the interaction of context information within the same modality. Although progress has been made in modeling context in MERC, these methods have not addressed the need to balance intra-speaker and inter-speaker contextual dependencies (i.e., balance empathy and sentiment inertia) in single-modal interaction relationships.

[0003] Multi-modal fusion aims to combine information from different modalities, mainly including feature-level fusion, decision-level fusion and model-level fusion. Feature-level fusion involves concatenating multi-modal features into a joint feature vector at the input layer, but this method may be affected by data sparsity due to high-dimensional feature sets. Therefore, decision-level fusion combines single-modal decision values through voting, averaging or weighted summation, but ignores the correlation between modalities. Model-level fusion is a compromise between the two, which fuses intermediate representations of different modalities. Some advanced methods have been recently proposed, such as mapping multi-modal features to the same feature space, gradient mixing, and combining fusion targets with single-modal distillation, as described in. Generally, inspired by the idea of machine translation, Yang et al. translate another modality feature into the text modality, enhancing the common features. However, due to the instability of inter-modal interactions in MERC, it is challenging to model multi-modal interactions. Some researchers have begun to explore graph-based fusion modules to capture intra- and inter-modal interaction information. However, these graph structures do not take into account the practical feasibility of predicting the current utterance based on future utterances. In addition, they have limitations in capturing intra- and inter-modal interaction information involving conflicting content expressing different emotions. To solve this problem, the present embodiment proposes a directed AdaGIN to adaptively select certain intra- and inter-modal interaction relationships. SUMMARY

[0004] The main purpose of the present application is to overcome the shortcomings and deficiencies of the prior art, and to provide an emotion classification method based on an adaptive interaction graph network, which adaptively selects specific single-modal and multi-modal relationships by introducing Gumbel-Softmax sampling of discrete random variables, and designs a task-specific loss function to prioritize the text modality and the internal context of the speaker to meet the requirements of the MERC task.

[0005] In order to achieve the above purpose, the technical scheme adopted by the present application is as follows:

[0006] In a first aspect, the present application provides an emotion classification method based on an adaptive interaction graph network, comprising the following steps:

[0007] An adaptive interaction graph network is constructed, which includes a feature extractor, a Gumbel-Softmax discrete sampler, an Utterance-level Encoder, a core component and an emotion classifier. The Gumbel-Softmax discrete sampler includes an NESP component and a GSP component, and the Utterance-level Encoder includes an Utterance Encoder and a Speaker Encoder.

[0008] The training process of the adaptive interactive graph network is as follows:

[0009] The dialogue sentiment dataset is retrieved from the dialogue sentiment database, and then the audiovisual features in the dialogue sentiment dataset, including original text features, original visual features and original speech features, are extracted using a feature extractor.

[0010] Audiovisual features are input into the Utterance-level Encoder, and after processing, node modal features are obtained. These node modal features are then input into the core component for graph convolution operations. Specifically, the node modal features are input into the original GIN and a GIN with NESP. When the graph convolution reaches a set number of layers, the GIN prediction result and the GIN prediction result with NESP are output. The GIN prediction result and the GIN prediction result with NESP are concatenated and then input into the final GIN formed by the GSP component and the GIN. After prediction, the final GIN prediction result is obtained. The GIN with NESP is formed by combining another original GIN with pseudo-labels and the NESP component.

[0011] The node modal features are concatenated with the final GIN prediction results, then input into the sentiment classifier for classification. The model loss is then iteratively calculated, including the preservation loss and the minimum descent loss. Training is complete when the parameters converge.

[0012] As a preferred technical solution, the Utterance Encoder processes audiovisual features as follows:

[0013] Use a bidirectional GRU to extract the feature sequence of the original text. Mapping to a fixed-size representation Simultaneously use a fully connected layer Feature sequences of original visual features and original speech features Perform the same processing as follows:

[0014]

[0015]

[0016] in, The text feature sequence representing the i-th utterance. This represents the hidden state of the text modality of the i-th utterance. Represents a fully connected layer Given separate parameters for the feature sequences of the original visual features and the original speech features, Text modality representing feature sequences, Let ξ represent the visual and speech feature sequences output by the fully connected layer, where ξ∈{a,v} represents the visual and speech modalities of the feature sequences.

[0017] As a preferred technical solution, the processing process of the Speaker Encoder on the audio-visual feature is:

[0018] by GPU P The context prompt of the corresponding speaker captures the features of each utterance-specific speaker information As follows:

[0019]

[0020]

[0021] Among them, The hidden state of the audio-visual feature of the kth utterance, the kth utterance and the ith utterance come from the same speaker, λ η Indicates a manually set hyperparameter, indicating the weight of the speaker feature, For integrating speaker information.

[0022] As a preferred technical solution, the original GIN performs graph convolution operation on node modal features, specifically:

[0023] Connect the features from the same node modal in the dialogue, and connect the three nodes from different modalities in the utterance;

[0024] The edge weight W between two nodes i, j ij According to the formula, as follows:

[0025]

[0026] Among them, Indicates the mapping value of the feature sequence of node i, Indicates the mapping value of the feature sequence of node j, η∈{a,v,t} indicates the text, visual and speech modalities of the feature sequence;

[0027] The graph convolution operation mode is as follows:

[0028]

[0029]

[0030]

[0031] Among them, Is the graph convolution matrix with regularization, Is the degree matrix of d q Indicates the maximum value of the sequence length of the data, Is the adjacency matrix of the corresponding graph, is an identity matrix, representing the connection relationship between nodes, initialized as are trainable parameters, is the output layer of the lth and is the weight matrix of the (l-1)th layer, and α and β are two hyperparameters, and σ(.) represents ReLU activation.

[0032] As a preferred technical solution, the NESP component includes an ESP and an NSP, which are used to learn binary random variables for each node and edge in the dialogue wherein represents the selected state of the node v and the edge δ of the ζth utterance; the ESP is used for node selection strategy, thereby adaptively selecting modal representation, and the NSP is used to classify the current utterance context into self and inter-speaker categories, allowing the edge selection strategy to select the context category for adaptive utterance representation.

[0033] As a preferred technical solution, the node selection strategy specifically includes:

[0034] Setting attribute values for dialogue edges based on Gumbel-Softmax distribution Then, samples in the approximate discrete classification distribution are sampled, as follows:

[0035]

[0036] wherein, is an independent and identically distributed edge sample extracted from a Unif(0,1) distribution, indicates the probability that the node v of the ζth utterance is selected, and τ represents a temperature parameter, which is greater than 0, and when it approaches 0, the Gumbel-softmax sampling result approximates to discrete sampling;

[0037] The calculated value is [0,1] or [1,0], wherein the 0 category refers to selecting the inter-speaker category, and the 1 category refers to selecting the intra-speaker category.

[0038] The edge selection strategy samples samples in the approximate discrete classification distribution in the same way as the node selection strategy.

[0039] As a preferred technical solution, the forming process of the GIN with NESP includes the following steps:

[0040] S1, generating pseudo labels;

[0041] (1) Train a new GIN using two modal information training sets, then compare the emotion recognition results of the new GIN with the recognition results of the GIN trained using a training set including all modal information, label the sentences with the same recognition results, and obtain MASK η ; the modal information η includes text modal t, visual modal v and speech modal a;

[0042] (2) Remove the nodes from the same modal connected in the dialogue, continue to train the new GIN in step (1), and obtain Indicate that the corresponding edge does not need to be trained, and then set the value to [1, 1], indicating that in the graph convolution, the node will introduce the context information between itself and the speaker at the same time;

[0043] S2, obtain the GIN with NESP by combining the pseudo-label and NESP components.

[0044] As a preferred technical solution, the prediction process of the GIN with NESP is a selection mechanism, that is:

[0045]

[0046]

[0047] wherein Gumbel-Softmax() represents the Gumbel-softmax sampling process, represents the attribute value of the visual modal, represents the attribute value of the dialogue edge, represents the selected state of the node v of the ζ utterance, represents the selected state of the edge δ of the ζ utterance.

[0048] At the same time, the node and edge weights of the GIN with NESP are adjusted:

[0049]

[0050]

[0051] wherein ο represents an element-wise multiplication operator, p indicates the node corresponding to the η modal of the i utterance, i represents the speaker of the utterance i.

[0052] As a preferred technical solution, the classification process of the emotion classifier, that is, the linear unit is used to predict the emotion distribution, includes the following steps:

[0053]

[0054]

[0055] wherein, denotes a series operation, and b c ∈R do are trainable parameters, where d o is a category of emotion, represents the modal representation corresponding to the i-th utterance after l-layer convolution, is a set of predicted emotional labels of utterances in the data set.

[0056] As a preferred technical solution, the model loss L is specifically:

[0057] L=L ce +γL m +ωL k +φL d +μL n

[0058]

[0059] wherein, L ce denotes a classification loss, Θ denotes a parameter set, denotes a coefficient of L2 regularization, γ, φ, ω, μ all denote the contribution degrees of the corresponding components of the loss function, denotes a predicted emotion distribution and a cross-entropy loss of the set of predicted emotional labels e of utterances in the data set;

[0060] The maintenance loss includes L m and L k , and is specifically:

[0061]

[0062]

[0063] wherein, n denotes the ordinal number of the dialogue, N denotes the dialogue round, ξ∈{a,v} denotes the visual modality v and the speech modality a of the feature sequence, t denotes the text modality, ρ g [] denotes GSP, which is selected for the complete graph and the graph with NESP, similar to NESP, which is also a quadratic random variable, indicating the selection probability, denotes the NSP of the text modality, i.e., the probability that the t modality node of the n-th utterance is selected, denotes the NSP of the speech and visual modalities, i.e., the probability that the a, v modality node of the n-th utterance is selected, denotes the ESP within the speaker, i.e., the edge that selects to retain the node consistent with the speaker of the current utterance, ESP represents the ESP between speakers, that is, the edge of the node selected to reserve the inconsistency with the current speech speaker;

[0064] The minimum drop loss includes L d And L n Specifically,

[0065]

[0066]

[0067] Compared with the prior art, the present application has the following advantages and beneficial effects:

[0068] (1) The present application introduces Gumbel-Softmax skill sampling of discrete random variables, thereby adaptively selecting specific single mode and multi-mode relationship, and further solving the monotony of current emotion classification work, and improving the ability of cross-modal interaction of conflict content expressing different emotions.

[0069] (2) The present application designs a task-specific loss function to preferentially select text modalities and internal speaker context to meet the requirements of MERC task, and improves the robustness of emotion classification model.

[0070] (3) The present application uses predefined pseudo-labels to determine to ignore unnecessary speeches, thereby reducing the computational complexity of NESP and improving the training speed of the model. BRIEF DESCRIPTION OF DRAWINGS

[0071] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following will briefly introduce the drawings needed in the embodiment description. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can also be obtained by those skilled in the art without creative labor.

[0072] Figure 1 The flow chart of the emotion classification method of the embodiment of the present application based on adaptive interaction graph network;

[0073] Figure 2 The schematic diagram of the adaptive interaction graph network of the embodiment of the present application;

[0074] Figure 3 The schematic diagram of the classifier training of the adaptive interaction graph network of the embodiment of the present application. DETAILED DESCRIPTION

[0075] In the interest of clarity, understanding, and ease of explanation, the present application will be described in connection with the embodiments disclosed herein. It will be appreciated that the description set forth herein is intended as a description of various embodiments of the application and is not intended to limit the scope of the application to these described embodiments. The description set forth herein is not intended to be exhaustive or to be assumed to be complete. It is intended, however, to be as complete and thorough as is reasonably possible and to cover all modifications, equivalents, and alternatives that are within the scope of the application.

[0076] Reference to "an embodiment" or "one embodiment" in this application means that a particular feature, structure, or characteristic described in connection with the embodiment is included in at least one embodiment of the application. The appearances of the phrase "in one embodiment" or "in an embodiment" in various places in the specification are not necessarily all referring to the same embodiment, nor are they necessarily all directed to the same embodiment, or to a single alternative embodiment. It is expressly understood that any of the

[0077] Referring now to the drawings, in which like reference numerals refer to similar elements throughout the several views, and initially to FIG. 1, an example system 100 is shown. The system 100 includes a computing device 102, which can be a desktop computer, a laptop computer, a tablet computer, a smartphone, a server, or any other computing device. The computing device 102 includes a processor 104, a memory 106, and a communication interface 108. The processor 104 can be any suitable processor, such as a central processing unit (CPU), a graphics processing unit (GPU), or a digital signal processor (DSP). The memory 106 can be any suitable memory, such as a random access memory (RAM), a read-only memory (ROM), or a combination of both. The communication interface 108 can be any suitable communication interface, such as a wired or wireless network interface, a Bluetooth interface, or a USB interface. Figure 1 The method provided by the embodiment includes the following steps:

[0078] S1, an adaptive interaction graph network is constructed, and the structure includes a feature extractor, a Gumbel-Softmax discrete sampler, an utterance-level encoder, a core component, and an emotion classifier; the Gumbel-Softmax discrete sampler includes an NESP component and a GSP component, and the utterance-level encoder includes an utterance encoder and a speaker encoder;

[0079] In this embodiment, the adaptive graph interactive network (AdaGIN) technique is mainly used to adaptively select specific single-modal and multi-modal relationships, where the dialogue sentiment features involve multiple modalities, and the modal information η includes the text modality t, the visual modality v, and the speech modality a. More specifically, this embodiment first constructs a GIN to model these relationships, and then introduces node- and edge-level selection policies (NESP) to guide the selection of nodes and edges. This embodiment also provides a graph-level selection policy (GSP) to integrate the original GIN and the GIN with NESP. In addition, this embodiment designs a task-specific loss function to preferentially select the text modality and the inter-speaker context to meet the requirements of the MERC task. Finally, this embodiment uses predefined pseudo-labels to determine which utterances do not need to be selected, thereby reducing the computational complexity of NESP.

[0080] Before data processing, an adaptive graph interactive network needs to be established, and then the network is trained by a training set formed by the dialogue emotion samples after feature processing. The feature extractor actually needs multiple feature extraction models to extract features of different dimensions of the dialogue emotion samples. Specifically, this embodiment uses three feature extraction models to extract text features, visual features, and speech features of the samples. For further implementation, see step S2.

[0081] For the Gumbel-Softmax discrete sampler, it is actually a Gumbel-Softmax distribution, which is a method for sampling from a discrete probability distribution. It achieves this purpose by introducing a Gumbel distribution. Specifically, it adds a noise term sampled from the Gumbel distribution to the log probability, and then transforms the result into a probability distribution through the Softmax operation, thereby achieving sampling of discrete random variables. This method is commonly used in deep learning for modeling discrete variables, especially in reinforcement learning and generative models. It has differentiability, which allows it to be trained in the backpropagation algorithm, thereby obtaining effective gradient information in the model optimization process. In the following text, the original GIN will be mentioned, which requires the use of this Gumbel-Softmax discrete sampler for its creation and operation, which will not be described here.

[0082] Furthermore, the Gumbel-Softmax discrete sampler includes the NESP component and the GSP component. The NESP component comprises NSP and ESP, designed to balance intra-speaker and inter-speaker contexts respectively, and to resolve conflicting content across different modalities. The goal of this component is to learn binary random variables for each node and edge in the dialogue. in This indicates whether node v and edge δ of the ζ-th utterance are selected. For NSP, the node selection strategy is completed through NSP to adaptively select modal representations. For ESP, it can be used to classify the current utterance context into categories of itself and between speakers, allowing the edge selection strategy to select the context category, that is, to select the category of the edge, for adaptive utterance representation.

[0083] Based on the fact that the Gumbel-Softmax distribution is a continuous distribution that can be used to approximate samples in discrete classification distributions, an attribute value was designed for the dialogue node. It is a set of binary random variables ρ, where This indicates the probability that node v of the ζ-th utterance is selected. During the training phase, Gumbel-Softmax is used to generate the following formula: Value:

[0084]

[0085] in, These are independent and identically distributed edge samples extracted from the Unif(0,1) distribution. The probability of node v being selected for the ζ-th utterance is indicated by τ, which represents the temperature parameter. When τ approaches 0, the Gumbel-softmax sampling result approximates discrete sampling.

[0086] During the prediction phase, classes 0 and 1 are mutually exclusive, therefore The value will only be [0,1] or [1,0]. The same method produces... Its category 0 refers to selecting the inter-speaker category, and category 1 refers to selecting the intra-speaker category.

[0087] It's worth explaining that neither the node selection strategy nor the edge selection strategy needs manual adjustment; instead, they are trained using the reverse backpropagation (BP) algorithm, just like the network weights. Furthermore, since binary random variables are discrete and cannot be directly trained using the reverse BP algorithm, the Gumbel-Softmax technique is introduced to address this issue.

[0088] In this embodiment, the Utterance-level Encoder includes an Utterance Encoder and a Speaker Encoder, please refer to Figure 2 The left side is shown. First, the Utterance Encoder is to capture contextual information and handle inconsistent dimensions in multi-modal data. Specifically, his processing process for audio-visual features is as follows:

[0089] The feature sequence of the original text feature is mapped to a fixed-size representation by a bidirectional GRU At the same time, the feature sequence of the original visual feature and the original speech feature is processed by a fully connected layer As follows:

[0090]

[0091]

[0092] Wherein, represents the i-th layer of the predicted text feature sequence, represents the hidden state of the text modality, represents the fully connected layer The individual parameters of the feature sequence of the original visual feature and the original speech feature, represents the text modality of the feature sequence, represents the visual and speech feature sequence output by the fully connected layer, and ξ∈{a,v} represents the visual modality and speech modality of the feature sequence.

[0093] Second, considering the individual differences in emotional expression, this embodiment adds a Speaker Encoder component in the Utterance-level Encoder to model the speaker's state information, which can capture the speaker-specific features of each utterance through the GPU P Contextual cues for the corresponding speaker Specifically as follows:

[0094]

[0095]

[0096] Wherein, represents the hidden state of the audio-visual feature of the k-th utterance, represents the feature sequence of the i-th dialogue, λ η represents a manually set hyperparameter indicating the weight of the speaker feature, Used to integrate speaker information. ​​

[0097] Finally, the obtained multi-modal feature sequences are summed to form node modal features.

[0098] In addition, the bidirectional GRU and GPU P used in the Utterance-level Encoder belong to BiGRU. BiGRU is the abbreviation of Bidirectional Gated Recurrent Unit, which is a neural network structure used in natural language processing. BiGRU combines the capabilities of recurrent neural networks (RNN) and bidirectional processing, which can capture the context information in the text and improve the performance of text processing. In natural language processing tasks such as text classification, sequence labeling, etc., BiGRU is widely used.

[0099] In the adaptive interactive graph network, the core is to have a core component with adaptive graph interaction capabilities, such as Figure 3 shown in the intermediate Adaptive Graph Interactive Network. This core component can dynamically change the layout, size, color, and other properties of the graph, as well as the way of interaction and response speed, by analyzing and understanding the user's behavior and needs, to improve user experience and efficiency. Adaptive graph interaction can be applied to various scenarios such as data visualization, games, virtual reality, etc., so that users can more intuitively and conveniently interact with the graphical interface. Specifically, the working mode of the core component is as follows: first, build an original GIN to model the intra-modal and inter-modal relationships, then introduce NESP to guide the selection of nodes and edges to build another graph, then perform graph convolution operations on the two graphs, and finally integrate the original GIN and the GIN with NESP by GSP to form the final GIN.

[0100] GIN is a graph structure, and each modal feature of each utterance in the dialogue after encoding by the Utterance-level Encoder is taken as a node of the graph. The number of nodes for a dialogue is 3*N, where N is the number of dialogue rounds, i.e. the number of utterances. The interaction between speakers and the self-speaker is modeled by edge connection. The edge connection rules are as follows:

[0101] First, nodes from the same modality are connected in the dialogue;

[0102] Second, three nodes from different modalities are connected in the utterance.

[0103] The edge weight W ij between two nodes i, j is calculated according to the formula as follows:

[0104]

[0105] wherein, represents the mapping value of the feature sequence of node i, represents the mapping value of the feature sequence of node j, η∈{a,v,t} represents the text, visual and voice modalities of the feature sequence;

[0106] The graph convolution operation mode is as follows:

[0107]

[0108]

[0109]

[0110] wherein, is a graph convolution matrix with regularization, is a degree matrix of , d q represents the maximum value of the sequence length of the data, is an adjacency matrix of the corresponding graph, is a unit matrix, representing the connection relationship between nodes, is initialized as is a trainable parameter, is the output layer of the lth, and is the weight matrix of the (l-1)th layer, and α and β are two hyperparameters, and σ(.) represents ReLU activation.

[0111] In this embodiment, the GIN with NESP is obtained by combining pseudo labels and NESP, specifically including the following steps:

[0112] I. Generating pseudo labels

[0113] (1) A new GIN is trained using the training set of two modalities of information, and then the emotion recognition result of the new GIN is compared with the recognition result of the GIN trained using the training set including all modalities of information, sentences with the same recognition result are labeled, and MASK is obtained η ;

[0114] (2) Remove the connection of nodes from the same modality in the dialogue, continue to train the new GIN in step (1), and obtain indicates that the corresponding edge does not need to be trained, and then the value is set to [1, 1], indicating that in the graph convolution, the node will introduce the context information between itself and the speaker at the same time;

[0115] II. Obtain the GIN with NESP by combining the pseudo label and the NESP component.

[0116] It is necessary to explain that for the setting of the "original" prefix, in fact, in order to distinguish the two GIN states of unprocessed and processed, through the function of the core component, the task of the original GIN is to model the intra-modal relationship and the inter-modal relationship, and the GIN with NESP is to introduce NESP to guide the selection of nodes and edges. When they complete the graph convolution, they can capture the information on the text, and then improve the robustness of the model.

[0117] It should be noted that, due to the large number of nodes and edges generated by the data set dialogue sentence, training each strategy alone will result in high computational complexity, therefore, the pseudo label mechanism is introduced to identify the node and edge strategy that does not need to be trained, see step I. In addition, it has been shown in the literature in MERC that the text modality has the best performance, and the result of the new GIN trained by removing the text has high contingency, and cannot provide high reliability for judging whether the text node needs to be trained. Based on this, the training data of the embodiment only contains two modal information such as voice and text. And the emotion recognition result of the GIN is compared with the recognition result of the original GIN trained using three modal information. The sentences with the same recognition result will be marked as MSAK v , indicating that the influence of the visual modality feature of the sentence is optional. The same method is used to obtain MASK a .

[0118] Further, as Figure 3 shown on the left, the final selection mechanism of the GIN with NESP is represented as follows

[0119]

[0120]

[0121] wherein Gumbel-Softmax() represents the Gumbel-softmax sampling process, represents the attribute value of the visual modality, represents the attribute value of the dialogue edge, represents the selection state of the node v of the ζ dialogue, represents the selection state of the edge δ of the ζ dialogue;

[0122] At the same time, the node and edge weights of the GIN with NESP are adjusted:

[0123]

[0124]

[0125] wherein o represents the element-wise multiplication operator, p indicates the node corresponding to the η modality of the i dialogue,i a speaker representing the utterance i.

[0126] The so-called emotion classifier Emotion Classifier can linear unit to predict the emotion distribution, see step S2.

[0127] S2, the training process of the adaptive interaction graph network is as follows:

[0128] S21, the dialogue emotion dataset is called in the dialogue emotion database, and then the audio-visual features in the dialogue emotion dataset are extracted by using the feature extractor, including the original text features, the original visual features and the original speech features;

[0129] For the text aspect, the last four hidden layers of the Roberta Large model are adopted to extract the text features of the sample by using layer normalization and average pooling operations; for the visual aspect, specifically the color, line, saturation, gray scale, brightness and other dimensions that can be distinguished by the naked eye of the picture or video, the speech level features can be extracted from the acoustic and visual patterns by using the OpenSmile model; and for the speech aspect, the feature extraction is realized by using the audio feature extraction toolkit and the pre-trained DenseNet model.

[0130] S22, input the audio-visual features into the Utterance-level Encoder, obtain the node modal features after processing; input the node modal features into the core component for graph convolution operation, that is, input the node modal features into the original GIN and the GIN with NESP, when the graph convolution reaches the set number of layers, output the GIN prediction result and the GIN prediction result with NESP; splice the GIN prediction result and the GIN prediction result with NESP, and then input into the final GIN formed by the GSP component and the GIN, obtain the final GIN prediction result after prediction; the GIN with NESP is another original GIN combined with pseudo label and NESP component;

[0131] S23, splice the node modal features and the final GIN prediction result, then input into the emotion classifier for classification, and then iteratively calculate the model loss, including the keep loss and the minimum descent loss, and the training is completed when the parameters converge, as shown in the right side. Figure 2

[0132] In this embodiment, the emotion classifier uses linear unit to predict the emotion distribution, including the following steps:

[0133]

[0134] ​

[0135] wherein, denotes a series operation, and are trainable parameters, wherein d o is a category of emotion, represents the modal representation of the ith utterance after l-layer convolution, is a set of predicted emotion labels of utterances in the dataset.

[0136] As shown in Figure 3 , the model loss L is specifically:

[0137] L=L ce +γL m +ωL k +φL d +μL n

[0138]

[0139] wherein, L ce denotes a classification loss, Θ denotes a parameter set, denotes a coefficient of L2 regularization, γ, φ, ω, μ all denote the contribution degrees of the corresponding components of the loss function, denotes a predicted emotion distribution and a cross-entropy loss of the predicted emotion label set e of utterances in the dataset;

[0140] The preservation loss includes L m and L k , and is specifically:

[0141]

[0142]

[0143] wherein, n denotes the ordinal number of the dialogue, N denotes the dialogue round, ξ∈{a,v} denotes the visual modality v and the speech modality a of the feature sequence, t denotes the text modality, ρ g [] denotes a GSP, which is selected for a complete graph and a graph with NESP, similar to the NESP, which is also a quadratic random variable, indicating the selection probability, denotes a NSP of the text modality, i.e., the probability of selecting the t modality node of the nth utterance, denotes a NSP of the speech and visual modalities, i.e., the probability of selecting the a, v modality node of the nth utterance, denotes an ESP within a speaker, i.e., selecting an edge that keeps the node consistent with the speaker of the current utterance, denotes an ESP between speakers, i.e., selecting an edge that keeps the node inconsistent with the speaker of the current utterance;

[0144] The minimum drop loss includes L d and L n , specifically:

[0145]

[0146]

[0147] The four loss terms are mainly designed based on the keep-or-drop strategy, which is used to constrain the selection strategy learning. The learning process is shown in the figure. The loss component L m and L k are the components of the keep loss. Minimizing L k encourages the selection of the text modality and the intra-speaker contextual edge for each modality. Minimizing L m encourages the selection of information from other modalities, inter-speaker contextual edges, and graphs. By minimizing the drop loss L d and L n , the effect is opposite to the keep loss L k and L m . Overall, in order to meet the MERC task, the embodiment is more inclined to the text modality information and the intra-speaker contextual information.

[0148] After the training is completed, the model parameters are as follows:

[0149] Table 1

[0150]

[0151] The embodiment does not use curriculum learning to train the selection strategy, but instead directly trains the strategy for the entire dialogue. In the test phase, the discrete decisions of nodes, edges and graphs are sampled from the trained selection strategy distribution. And the validation set is used to perform hyperparameter search on AdaGIN on each dataset. The experimental results of the comparative method are shown in the following table:

[0152] Table 2

[0153]

[0154] As can be seen, in the same two modes, although there are some flaws in the predicted results of some emotion classifications, it does not hinder the outstanding performance of most emotion classifications. From “happy” to “disappointed”, in the IEMOCAP mode, except for the case that the scores of “angry” and “excited” classifications are slightly low, the scores of other emotion classifications and the total score all perform very well, showing high-level emotion classification performance. Similarly, similar results can be obtained in the MELD mode.

[0155] It should be noted that, for the foregoing method embodiments, the purposes of brief description, they are all expressed as a series of action combinations, but those skilled in the art should know that the present application is not limited to the order of the actions described, because according to the present application, certain steps can be performed in other orders or simultaneously. The technical features of the above embodiments can be combined in any way. In order to make the description simple, not all possible combinations of the technical features in the above embodiments are described, but as long as the combination of the technical features does not exist contradictory, it should be considered as the scope of the present application.

[0156] The above embodiments are the preferred embodiments of the present application, but the embodiments of the present application are not limited to the above embodiments, and any changes, modifications, substitutions, combinations, simplifications made without departing from the spirit and principles of the present application are equivalent replacement methods and are included in the protection scope of the present application.

Claims

1. An emotion classification method based on an adaptive interaction graph network, characterized in that, The method comprises the following steps: An adaptive interaction graph network is constructed, which comprises a feature extractor, a Gumbel-Softmax discrete sampler, an utterance-level encoder, a core component and an emotion classifier; the Gumbel-Softmax discrete sampler comprises an NESP component and a GSP component, and the utterance-level encoder comprises an utterance encoder and a speaker encoder; The training process of the adaptive interaction graph network is as follows: The dialogue emotion dataset is called in the dialogue emotion database, and then the audio-visual features in the dialogue emotion dataset are extracted by using the feature extractor, including original text features, original visual features and original speech features; The audio-visual features are input into the utterance-level encoder, and the node modal features are obtained after processing; the node modal features are input into the core component for graph convolution operation, that is, the node modal features are input into the original GIN and the GIN with the NESP, and when the graph convolution is performed to a set number of layers, the GIN prediction result and the GIN prediction result with the NESP are output; the GIN prediction result and the GIN prediction result with the NESP are spliced and then input into the final GIN formed by the GSP component and the GIN, and the final GIN prediction result is obtained after prediction; the GIN with the NESP is another original GIN combined with the pseudo label and the NESP component; The node modal features and the final GIN prediction result are spliced and then input into the emotion classifier for classification, and then the model loss is iteratively calculated, the loss comprises a keep loss and a minimum descent loss, and the training is completed when the parameters converge; The NESP component includes an ESP and an NSP for learning binary random variables for each node and edge in a dialogue wherein denote the selected state of node v and edge δ of the ζth utterance; the ESP is for a node selection policy to adaptively select a modality representation, and the NSP is for classifying the current utterance context into self and inter-speaker categories to allow an edge selection policy to select a context category for an adaptive utterance representation; The node selection strategy is specifically: Setting attribute values for dialogue edges based on gumbel-softmax distributions Samples from the approximated discrete categorical distribution are then sampled as follows: wherein, υ (ζ) ∈R 2 are independent and identically distributed side samples drawn from a Unif(0, 1) distribution, denotes the probability that node v indicating the z-th utterance is selected, τ denotes a temperature parameter, which is greater than 0, and when it tends to 0, the Gumbel-softmax sampling result approximates to discrete sampling; The calculated value is [0, 1] or [1, 0], wherein 0 represents the selection of the speaker inter-class, and 1 represents the selection of the speaker intra-class; The edge selection strategy adopts the same method as the node selection strategy to sample the samples in the approximate discrete classification distribution. 2.The emotion classification method based on adaptive interaction graph network according to claim 1, characterized in that, The processing process of the utterance encoder on the audio-visual features is as follows: Use a bidirectional GRU to extract the feature sequence of the original text. Mapping to a fixed-size representation Simultaneously use a fully connected layer Feature sequences of original visual features and original speech features Perform the same processing as follows: wherein, represents a text feature sequence of the i-th utterance, represents a hidden state of a text modality of the i-th utterance, represents a fully connected layer individual parameters for the feature sequence of the original visual feature and the original speech feature, represents a text modality of the feature sequence, represents a visual and speech feature sequence of the fully connected layer output, and represents a visual modality and a speech modality of the feature sequence, a represents the speech modality, v represents the visual modality, and t represents the text modality. 3.The emotion classification method based on adaptive interaction graph network according to claim 2, characterized in that, The processing process of the speaker encoder on the audio-visual features is as follows: By GPU P Contextual cues for respective speakers capture features specific to the speaker of each utterance The following equation: wherein, denotes the hidden state of the audio-visual features of the kth utterance, the kth utterance and the ith utterance being from the same speaker, λ η denotes a manually set hyper-parameter, indicating the weight of the speaker feature, for integrating the speaker information. 4.The emotion classification method based on adaptive interaction graph network according to claim 1, characterized in that, The graph convolution operation of the original GIN on the node modal features is specifically: The nodes from the same node modal features are connected in the dialogue, and the three nodes from different modalities are connected in the utterance; The edge weight W between two nodes i, j ij The calculation is made according to the formula as follows: wherein, represents a mapping value of the feature sequence of the node i, represents a mapping value of the feature sequence of the node j, η∈{a,v,t} represents the text, visual and speech modalities of the feature sequence; The graph convolution operation mode is as follows: wherein, is a graph convolution matrix with regularization, is is a degree matrix of d q denotes the maximum value of the sequence length of data, is an adjacency matrix of the corresponding graph, is an identity matrix, representing the connection relationship between nodes, is initialized as is a trainable parameter, is the output layer of the lth and is the weight matrix of the (l-1)th layer, and a and b are two hyperparameters, and s(.) represents the ReLU activation.

5. The emotion classification method based on adaptive interaction graph network according to claim 1, characterized in that, The forming process of the GIN with the NESP comprises the following steps: S1, generating a pseudo label; (1) training a new GIN using the training set of two modal information, and then comparing the emotion recognition result of the new GIN with the recognition result of the GIN trained using the training set including all modal information, labeling the sentences with the same recognition result to obtain the MASK η ; the modal information η includes a text modal t, a visual modal v and a speech modal a; (2) Remove the connections of nodes from the same modality in the dialogue, continue to train the new GIN in step (1), and obtain indicates that the node will introduce its own and the speaker's context information in the graph convolution; S2, obtaining the GIN with the NESP by combining the pseudo label and the NESP component.

6. The emotion classification method based on adaptive interaction graph network according to claim 5, characterized in that, The prediction process of the GIN with the NESP is a selection mechanism, that is: where Gumbel-Softmax() represents a Gumbel-softmax sampling process, represents an attribute value of a visual modality, represents an attribute value of a dialogue edge, represents a node v of the z-th utterance is selected state, represents an edge d of the z-th utterance is selected state; Meanwhile, the node and edge weights of the GIN with the NESP are adjusted: where denotes the element-wise multiplication operator, denotes the node corresponding to the ithutterance's η modality, i denotes the speaker of the ithutterance.

7. The emotion classification method based on adaptive interaction graph network according to claim 1, characterized in that, The classification process of the emotion classifier, that is, the emotion distribution is predicted by using a linear unit, comprises the following steps: wherein, denotes a concatenation operation, and are trainable parameters, wherein d o is a class of emotion, represents the modal representation of the ith utterance after l layers of convolution, is a set of predicted sentiment labels of utterances in the dataset. 8.The emotion classification method based on adaptive interaction graph network according to claim 1, characterized in that, The model loss L is specifically: L = L ce + γL m + ωL k + φL d + μL n wherein L ce denotes the classification loss, Θ denotes the set of parameters, denotes the coefficient of L2 regularization, γ, φ, ω, μ all denote the contribution degree of the corresponding components of the loss function, denotes the predicted emotion distribution and the cross-entropy loss between the predicted emotion label set e of the utterance in the data set; The holding loss includes L m and L k , in particular: where n denotes the ordinal of the dialogue, N denotes the dialogue turn, ξ ∈ {a, v} denotes the visual modality v and the speech modality a of the feature sequence, t denotes the text modality, ρ g [] denotes the GSP, selected for both the complete graph and the graph with NESP, similar to NESP, also a quadratic random variable, indicating the selection probability, denotes the NSP for the text modality, i.e., the probability that the t modality node of the nthutterance is selected, denotes the NSP for the speech and visual modalities, i.e., the probability that the a, v modality node of the nthutterance is selected, denotes the intra-speaker ESP, i.e., the selection of edges that preserve the consistency with the speaker of the current utterance, denotes the inter-speaker ESP, i.e., the selection of edges that preserve the inconsistency with the speaker of the current utterance; The minimum descent loss includes L d and L n , specifically:

Images