A multimodal emotion recognition method based on consistency constrained graph convolutional network

Abstract

The present invention relates to a multimodal emotion recognition method based on a consistency-constrained graph convolutional network, and belongs to the field of emotion recognition technology. Currently, most multimodal studies focus on simply modeling multimodal information through feature splicing, or using graph convolutional networks to learn different discourses (samples) and classification features with emotion labels; the present invention uses discourse as a node, and constructs a graph convolutional network according to the different modal relationships between two nodes. At the same time, the consistency loss of the same speaker in the conversation and the consistency loss of different modal features in the same discourse are added as regular terms to the model loss. By doing so, the corresponding model of the present invention adjusts the discourse features to better predict the emotion category. The method was evaluated on two public benchmark datasets, IEMOCAP and MELD, and the results showed that the method is superior to other methods in a multimodal conversation environment.

Description

Technical Field

[0001] The invention relates to a multimodal emotion recognition method based on a consistency-constrained graph convolutional network, and belongs to the technical field of emotion recognition. Background Art

[0002] With the rapid development of social media, multimodal emotion recognition (MERC) in conversation has attracted more and more attention from the research community. Due to the introduction of multimodal information, emotion recognition (ERC) in conversation can analyze emotional expressions from multiple perspectives. In terms of the utilization of multimodal information, researchers pay much attention to the extraction and fusion strategies of different modal features. For the ERC task, existing works mainly consider the relationship between different discourses and modal representations in the conversation. Unlike traditional monologues and other emotion recognition, the discourse relationship and context of individuals in the conversation are very helpful for ERC. Li et al. proposed to extract personalized information from the speaker's acoustic information in APersona-Based NeuralConversation Model. MMGCN proposed a speaker encoder to distinguish different speakers. However, these models have high processing costs in context information extraction or have limitations in speaker feature learning. Summary of the invention

[0003] The present invention provides a multimodal emotion recognition method based on a consistency constrained graph convolutional network. The present invention takes discourse as a node and constructs a graph convolutional network according to different modal relationships between two nodes. At the same time, the consistency loss of the same speaker in a conversation and the consistency loss of different modal features in the same discourse are added to the model loss as rule items to adjust discourse features to better predict emotions.

[0004] The technical solution of the present invention is: a multimodal emotion recognition method based on a consistency-constrained graph convolutional network, the method comprising the following steps:

[0005] Step 1: Preprocess the data set;

[0006] Step 2, encode different modal information;

[0007] Step 3: Build the basic architecture of the graph convolutional network, including the nodes and edges of the adjacency matrix;

[0008] Step 4: Learn the features of the graph;

[0009] Step 5: Extract speaker-level consistency constraints;

[0010] Step 6: Extract consistency constraints between different modes;

[0011] Step 7: Send the features passed through the graph convolutional network into the emotion classifier to obtain the emotion classification of the node.

[0012] As a further solution of the present invention, the Step 3 includes constructing graph nodes of the graph convolutional network adjacency matrix and constructing edges of the graph convolutional network adjacency matrix; the nodes are represented at the sentence level, and in the conversation data set, each utterance in each conversation is taken as each node. Under the multimodal representation, each sentence has three modal representations respectively, so when there are N utterances, there are 3N nodes.

[0013] As a further solution of the present invention, the nodes and edges of the adjacency matrix constructed in Step 3 have different construction methods for the edges between nodes for the same mode and different modes, which are divided into two cases;

[0014] In the first case, if two nodes come from different conversations, their different modal features will not be transferred; in this case, the edge weights of the two nodes are set to 0; angular similarity is used to measure the weight of the edge between any two nodes;

[0015]

[0016] in, Respectively represent the initial features of the same mode of the i-th and j-th utterances, a, t, v represent speech modality, text modality, and image modality, and sim is the cosine similarity function;

[0017] In the second case, if the two nodes come from the same conversation, the different modal features between them are also related, and feature transfer is required. The weight of the edge between the two nodes is also measured by the angular similarity:

[0018]

[0019] in, Respectively represent the initial features of the two different modes of the i-th and j-th utterances,

[0020] As a further solution of the present invention, in Step 4, after constructing the adjacency matrix according to the weight calculation method of the edges between nodes, for a certain modal feature of the node, three adjacency matrices are constructed to perform feature transfer and learning; for a certain modal feature of the node, the relationship between this certain modal feature and its own modality and the other two modalities is considered respectively to construct, wherein the modality of the node includes speech modality, text modality, and image modality.

[0021] As a further solution of the present invention, in Step 5, regarding the consistency constraint at the speaker level, after being encoded by the l-layer deepGCN model, each node integrates multiple modal information of other nodes; assuming that there are M speakers in the data set, the m-th speaker has K utterances in the same dialogue, and the same speaker will not appear in two dialogues, that is, there are K graph convolution feature vectors, and the graph convolution feature vector of the k-th utterance of the m-th speaker is set to be expressed as Concatenate these K node feature vectors to get the speaker's feature matrix Then perform meanpooling to get the average feature vector of the speaker.

[0022]

[0023] where · represents the dot product operation, ||·|| represents the norm of the l2 vector, and d l is a limited dimension, then the total speaker consistency loss L of the dataset is calculated spk :

[0024]

[0025] As a further solution of the present invention, in Step 6, regarding the consistency constraint between modalities, the text modality feature is used as a reference to make the speech modality feature and the image modality feature close to the text modality feature; specifically, u is calculated. i The cosine similarity between speech features, image features, and text features is used as a consistency loss:

[0026]

[0027] Then, the consistency losses of the utterances in the dataset are summed up to obtain the total modality consistency loss:

[0028]

[0029] is the feature representation of the language modality of the l+1 layer, is the feature representation of the text modality of the l+1 layer, is the feature representation of the image modality of layer l+1, u i The i-th utterance.

[0030] As a further solution of the present invention, in Step 7, the convolution feature of the node It is sent to an MLP with two fully connected layers and a Softmax function to obtain the probability that the node belongs to different emotion categories:

[0031]

[0032] Select the emotion category corresponding to the maximum probability value as the i-th utterance u i Emotional tags And use cross entropy as the category loss function:

[0033]

[0034] where p i [j] is utterance u of type j i The model predicts the probability, y i [j] is utterance u of type j i emotional labels;

[0035] In model training, the model combines classification loss, speaker consistency loss, modality consistency loss and 12 regularization. The specific formula is:

[0036] L=L cls +λ1L spk +λ2L mod +λ3||θ||2

[0037] Among them, L spk is the total speaker consistency loss of the dataset, L mod is the total modality consistency loss, λ1, λ1, λ1 are the weights of the three regularization terms, and θ represents the set of trainable parameters.

[0038] The beneficial effects of the present invention are as follows: First, the present invention constructs a graph convolutional network to transmit information between multimodal nodes and learns the unimodal features of each sample; then, the average features of the same speaker in the conversation are calculated, and each sample feature of the speaker is made close to the average feature through the speaker consistency constraint; similarly, through the inter-modal relationship consistency constraint, the speech features and image features are made close to the text features of each sample; finally, the features of the three modalities are connected to obtain the final representation of each sample and predict the emotion category; the present method is evaluated on two public benchmark datasets IEMOCAP and MELD. The results show that in a multimodal conversation environment, the present method is superior to other methods, and the effectiveness of each component of the model is verified by ablation experiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1 is a flow chart of the present invention;

[0040] Figure 2 The present invention constructs a graph convolutional network graph based on speech modal features. DETAILED DESCRIPTION

[0041] Example 1: Figure 1-Figure 2 As shown, a multimodal emotion recognition method based on a consistency-constrained graph convolutional network comprises the following steps:

[0042] Step 1: Preprocess the dataset. Experiments were conducted on two multimodal conversation datasets, IEMOCAP and MELD. Both datasets contain three modalities: text, image, and audio. The datasets were divided into training set, validation set, and test set in a ratio of 8:1:1.

[0043] Table 1 Distribution of IEMOCAP and MELD datasets

[0044]

[0045] IEMOCAP: This dataset contains 12 hours of binary dialogue videos, each video contains a single binary dialogue, with a total of 7433 utterances and 151 dialogues. Each utterance in the dialogue is labeled with six categories of emotions, including happy, sad, neutral, angry, excited, and frustrated.

[0046] MELD: This dataset is a multimodal dialogue dataset like IEMOCAP, but MELD is a multivariate dialogue dataset. MELD contains text, speech, and image information of more than 1,400 dialogues and 13,000 utterances from the TV series "Friends". Each utterance in each dialogue is annotated into one of seven emotion categories: anger, disgust, sadness, happiness, surprise, fear, or neutral.

[0047] Step 2, encode different modal information; use the RoBERTa model to extract features from the text modality in the multimodal dataset. First, fine-tune the RoBERTa Large model. The RoBERTa Large model follows the original BERTLarge model architecture, with a total of 24 layers, 16 self-attention heads in each block, and a hidden dimension of 1024, resulting in a total of 355M parameters. The cls labels of the last four layers are averaged, and then BiLSTM is applied to encode the contextual information in the text, and finally a 200-dimensional feature vector is obtained; for the auditory and visual modalities, a fully connected network is used for encoding, which are unified into 200-dimensional feature vectors. All three modalities contain sentence-level features of contextual information;

[0048] Step 3. Build the basic architecture of the graph convolutional network. First, build the graph nodes of the graph convolutional network adjacency matrix, and then build the edges of the graph convolutional network adjacency matrix. The nodes are represented at the sentence level. In the conversation dataset, each utterance in each conversation is taken as a node. Under the multimodal representation, each sentence has three modal representations, so when there are N utterances, there are 3N nodes.

[0049] Constructing the edges of the graph convolutional network adjacency matrix: For the edges between nodes, there are different construction methods for the same modality and different modalities, which are divided into two cases;

[0050] In the first case, if two nodes come from different conversations, their different modal features will not be transferred; in this case, the edge weights of the two nodes are set to 0; angular similarity is used to measure the weight of the edge between any two nodes;

[0051]

[0052] in, Respectively represent the initial features of the same mode of the i-th and j-th utterances, a, t, v represent speech modality, text modality, and image modality, and sim is the cosine similarity function;

[0053] In the second case, if the two nodes come from the same conversation, the different modal features between them are also related, and feature transfer is required. The weight of the edge between the two nodes is also measured by the angular similarity:

[0054]

[0055] in, Respectively represent the initial features of the two different modes of the i-th and j-th utterances,

[0056] Step 4, learn the features of the graph; construct an adjacency matrix based on the weight calculation method of the edges between nodes. For a certain modal feature of the node, construct three adjacency matrices for feature transfer and learning; for a certain modal feature of the node, consider the relationship between this modal feature and its own modality and the other two modalities to construct, where the modalities of the node include speech modality, text modality, and image modality. Take the feature learning of the speech modality a of the node as an example, Figure 2 As shown, considering the relationship between speech modality a and its own speech modality a, text modality t and image modality v, three graph adjacency matrices A can be constructed: aa , A at , A av , for the initial feature matrix Xa(0) to update.

[0057] Step 5, extract the consistency constraint at the speaker level: Everyone has his or her own independent way of thinking. In a conversation, we know that each speaker will have certain emotional expressions for a certain event or topic, which represents the expression of personal opinions. The speaker's emotional expression and emotional performance are consistent in the same conversation. That is, in the same conversation, the same speaker will tend to maintain a relatively stable emotional state under the influence of emotional inertia. After being encoded by the l-layer deepGCN model, each node integrates multiple modal information of other nodes, such as Figure 1 As shown in the model diagram; assuming that there are M speakers in the data set, the m-th speaker has K utterances in the same dialogue (the same speaker will not appear in two dialogues), the same speaker will not appear in two dialogues, that is, there are K graph convolution feature vectors, and the graph convolution feature vector of the k-th utterance of the m-th speaker is set to be expressed as Concatenate these K node feature vectors to get the speaker's feature matrix Then perform meanpooling to get the average feature vector of the speaker.

[0058]

[0059] where · represents the dot product operation, ||·|| represents the norm of the l2 vector, and d l We take 100 dimensions as a limited dimension, and then calculate the total speaker consistency loss L of the dataset spk :

[0060]

[0061] Step 6. Extract consistency constraints between different modalities: There is consistency and complementarity between the representations of different modalities. Consistency means that different information sources represent the same pattern or result; complementarity means that each modality has independent or mutually exclusive information. If there is no complementarity between modalities, that is, the information of any modality is sufficient to represent the pattern or result, then the multimodal representation has no meaning. This method aims to make the different modal information under the same sample as similar as possible, that is, to make the absolute difference in features between each sample as small as possible. The approach is similar to the speaker information constraint. The text modal features are used as a benchmark to make the speech modal features and image modal features close to the text modal features. Specifically, u is calculated. i The cosine similarity between speech features, image features, and text features is used as a consistency loss:

[0062]

[0063] Then, the consistency losses of the utterances in the dataset are summed up to obtain the total modality consistency loss:

[0064]

[0065] is the feature representation of the language modality of the l+1 layer, is the feature representation of the text modality of the l+1 layer, is the feature representation of the image modality of layer l+1, u i The i-th utterance.

[0066] Step 7: Send the features of the graph convolutional network to the emotion classifier to obtain the emotion classification of the node. It is sent to an MLP with two fully connected layers and a Softmax function to obtain the probability that the node belongs to different emotion categories:

[0067]

[0068] Select the emotion category corresponding to the maximum probability value as the i-th utterance u i Emotional tags And use cross entropy as the category loss function:

[0069]

[0070] where p i [j] is utterance u of type j i The model predicts the probability, y i [j] is utterance u of type j i emotional labels;

[0071] In model training, the model combines classification loss, speaker consistency loss, modality consistency loss and 12 regularization. The specific formula is:

[0072] L=L cls +λ1L spk +λ2L mod +λ3||θ||2

[0073] Among them, L spk is the total speaker consistency loss of the dataset, L mod is the total modality consistency loss, λ1, λ1, λ1 are the weights of the three regularization terms, and θ represents the set of trainable parameters.

[0074] Table 2 shows the comparison test between the model corresponding to the method of the present invention (CCMGCN) and other base models under the same multimodal dataset. Table 2 shows the comparison of MicroF1 values ​​obtained by the model corresponding to the method of the present invention and other models under the same multimodal dataset. CCMGCN improves by 1.14% on the IEMOCAP dataset and 0.99% on the MELD dataset.

[0075] Table 2 MicroF1 values ​​of different models on ERC (%)

[0076]

[0077] Experiments To study the contribution of the two consistency constraints proposed in CCMGCN, ablation studies are performed on the two datasets. Consider the following two settings:

[0078] w / o L spk &L mod The speaker consistency constraint loss and modality consistency constraint loss are eliminated from CCMGCN.

[0079] w / o L mod Remove the modality consistency constraint loss from CCMGCN and only use the speaker consistency constraint loss.

[0080] w / o L spk Remove the speaker consistency constraint loss from CCMGCN and only use the modality consistency constraint loss.

[0081] Table 3 Study on the ablation of these two consistency constraints

[0082]

[0083] To further test the influence of the central modality, the experiment uses image features or speech features instead of text features as the central modality features. The comparison results are shown in Table 4. The conclusion shows that CCMGCN has the best effect. This shows that for dialogue data such as IEMOCAP and MELD, the text modality is more prominent than the speech and image modalities in expressing the speaker's emotions.

[0084] Table 4 Core modality ablation experiment (%)

[0085]

[0086] The specific implementation modes of the present invention are described in detail above in conjunction with the accompanying drawings, but the present invention is not limited to the above implementation modes, and various changes can be made within the knowledge scope of ordinary technicians in this field without departing from the purpose of the present invention.

Claims

1. A multimodal emotion recognition method based on consistency-constrained graph convolutional network, characterized by: The method comprises the following steps: Step 1: Preprocess the data set; Step 2: Encode different modal information; Step 3: Build the basic architecture of the graph convolutional network, including the nodes and edges of the adjacency matrix; Step 4: Learn the features of the graph; Step 5: Extract speaker-level consistency constraints; Step 6: Extract consistency constraints between different modes; Step 7: Send the features of the graph convolutional network to the emotion classifier to obtain the emotion classification of the node; Step 3 includes constructing graph nodes of the graph convolutional network adjacency matrix and constructing edges of the graph convolutional network adjacency matrix; the nodes are represented at the sentence level. In the conversation data set, each utterance in each conversation is taken as each node. Under the multimodal representation, each sentence has three modal representations, so when there are N utterances, there are 3N nodes; In the Step 4, after constructing the adjacency matrix according to the weight calculation method of the edges between nodes, for a certain modal feature of the node, three adjacency matrices are constructed to perform feature transfer and learning; for a certain modal feature of the node, the relationship between the certain modal feature and its own modality and the other two modalities is considered to construct, wherein the modality of the node includes speech modality, text modality, and image modality; In Step 5, regarding the consistency constraint at the speaker level, after being encoded by the l-layer deepGCN model, each node integrates the multi-modal information of other nodes; assuming that there are M speakers in the dataset, the m-th speaker has K utterances in the same dialogue, and the same speaker will not appear in two dialogues, that is, there are K graph convolution feature vectors, and the graph convolution feature vector of the k-th utterance of the m-th speaker is set to be expressed as Concatenate these K node feature vectors to get the speaker's feature matrix Then perform meanpooling to get the average feature vector of the speaker. where · represents the dot product operation, ||·|| represents the norm of the l2 vector, and d l is a limited dimension, then the total speaker consistency loss L of the dataset is calculated spk : In Step 6, regarding the consistency constraint between modalities, the text modality feature is used as a benchmark to make the speech modality feature and the image modality feature close to the text modality feature; specifically, u is calculated. i The cosine similarity between speech features, image features, and text features is used as a consistency loss: Then, the consistency losses of the utterances in the dataset are summed up to obtain the total modality consistency loss: is the feature representation of the language modality of the l+1 layer, is the feature representation of the text modality of the l+1 layer, is the feature representation of the image modality of layer l+1, u i The i-th utterance.

2. The multimodal emotion recognition method based on consistency-constrained graph convolutional network according to claim 1, characterized in that: The nodes and edges of the adjacency matrix constructed in Step 3 have different construction methods for the edges between nodes for the same mode and different modes, which are divided into two cases; In the first case, if two nodes come from different conversations, their different modal features will not be transferred; in this case, the edge weights of the two nodes are set to 0; angular similarity is used to measure the weight of the edge between any two nodes; in, Respectively represent the initial features of the same mode of the i-th and j-th utterances, a, t, v represent speech modality, text modality, and image modality, and sim is the cosine similarity function; In the second case, if the two nodes come from the same conversation, the different modal features between them are also related, and feature transfer is required. The weight of the edge between the two nodes is also measured by the angular similarity: in, Respectively represent the initial features of the two different modes of the i-th and j-th utterances, 3. The multimodal emotion recognition method based on consistency-constrained graph convolutional network according to claim 1, characterized in that: In Step 7, the convolution feature of the node It is sent to an MLP with two fully connected layers and a Softmax function to obtain the probability that the node belongs to different emotion categories: Select the emotion category corresponding to the maximum probability value as the i-th utterance u i Emotional tags And use cross entropy as the category loss function: where p i [j] is utterance u of type j i The model predicts the probability, y i [j] is utterance u of type j i emotional labels; In model training, the model combines classification loss, speaker consistency loss, modality consistency loss and l2 regularization. The specific formula is: L=L cls +λ1L spk +λ2L mod +λ3||θ||2 Among them, L spk is the total speaker consistency loss of the dataset, L mod is the total modality consistency loss, λ1, λ1, λ1 are the weights of the three regularization terms, and θ represents the set of trainable parameters.

Images