A multi-modal sentiment recognition method based on a graph convolution network

Abstract

The application discloses a kind of multi-modal sentiment recognition methods based on graph convolution network, including steps S1, video, audio and text original feature are preprocessed, obtain text, video and audio features;S2, construct the discourse time sequence relationship between text, video and audio features, embed speaker information, adopt graph convolution network GCN to fuse any two modalities in text modal, video modal and audio modal, obtain three kinds of modal high-dimensional fusion features;S3, further construct the time sequence relationship of three kinds of high-dimensional fusion features, graph convolution network GCN is further fused to three kinds of high-dimensional fusion features, and the precision of classification is promoted, and prediction result is obtained by full connection layer FC.The application is fused and trained by two stages of graph convolution network GCN to audio, video, text three kinds of modal characteristics, the potential connection between multi-modal sentiment modal features is deeply fused and trained, and the classification precision of multi-modal sentiment recognition is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of multi-modal emotion recognition, and particularly relates to a multi-modal emotion recognition method based on a graph convolution network. BACKGROUND

[0002] As a popular field of human-computer interaction, multi-modal emotion recognition has been applied in the fields of medicine, education, safe driving and e-commerce. Multi-modal emotion can be expressed and classified using audio, video and text. However, the single emotion expression mode and the specific expressed emotion are not a simple linear relationship. Multi-modal emotion is a comprehensive embodiment of a series of human behaviors and environmental factors. The recognition of multi-modal emotion should be studied from multiple dimensions such as audio, video and text.

[0003] The existing method based on local features mainly uses audio, video and text to obtain the time sequence relationship through a recurrent neural network (RNN), then uses a graph structure such as a graph convolution network (GCN) or a graph neural network (GNN) to fuse different modalities, and finally uses a fully connected layer for classification. The existing method usually focuses on how to better use the graph structure to collect information between speeches, and does not pay attention to mining the potential relationship between intra-modal or inter-modal features. The accuracy of multi-modal emotion recognition is low.

[0004] In reality, in order to improve the accuracy of multi-modal emotion recognition, attention needs to be paid to the fusion of intra-modal or inter-modal features, and further fusion and training are needed to improve the accuracy of multi-modal emotion recognition classification.

[0005] The patent document with the document number CN112559835B discloses a multi-modal emotion recognition method, which comprises extracting frame-level audio features, frame-level video features and word-level text features respectively; the extracted features are input into a feature encoder for modeling to obtain encoded audio encoding, video encoding and text encoding features; the encoded features are first input into respective self-attention modules to model the interaction relationship within the modal, and are then input into a cross-modal attention module to model the interaction relationship between two modes; the outputs of the self-attention modules and the cross-modal attention module are subjected to time sequence pooling to obtain global interaction features within each mode and global interaction features between two modes; the global interaction features within the mode and between the modes are weighted and fused by using an attention mechanism to obtain feature representations within the mode and between the modes of the entire sample to be tested, and the two are spliced and input into a fully connected network to obtain a final emotion classification result. However, the method focuses on collecting information between utterances by using a graph structure, and the features within the multi-modal or between the multi-modal are not deeply fused, which is not conducive to improving the accuracy of multi-modal emotion recognition. SUMMARY

[0006] The present application aims to provide a multi-modal emotion recognition method based on a graph convolution network, which solves the problem of not deepening the fusion training of the potential relationship between the features within the multi-modal emotion modal or between the modal, and the low accuracy of multi-modal emotion recognition.

[0007] The object of the present application can be achieved by the following technical solution: a multi-modal emotion recognition method based on a graph convolution network, comprising the following steps:

[0008] S1, preprocessing video, audio and text original features to obtain text, video and audio features;

[0009] S2, constructing a speech time sequence relationship between the text, video and audio features, embedding speaker information, using a graph convolution network GCN to fuse any two modalities in the text modal, video modal and audio modal, obtaining three kinds of high-dimensional fusion features, and training the high-dimensional fusion features;

[0010] S3, further constructing a time sequence relationship of the three kinds of high-dimensional fusion features, performing graph convolution network GCN re-fusion on the three kinds of high-dimensional fusion features, improving the accuracy of multi-modal emotion recognition classification, obtaining a prediction result through a fully connected layer FC, and then improving the accuracy of multi-modal emotion recognition classification through training.

[0011] As a further scheme of the present application: the S1 performs preprocessing on the video, audio and text original features, including using a TextCNN algorithm to extract and process the text original features, using an OpenSmile toolkit with an IS10 configuration to extract and process the audio original features, and using a DenseNet algorithm to pre-train and extract the visual facial expression video original features on a facial expression recognition Plus (FER+) corpus.

[0012] As a further scheme of the present application: the S2 includes the following steps for obtaining the three high-dimensional fusion features:

[0013] S21, obtaining the discourse time sequence relationship between the video and audio features by using a bidirectional long short-term memory network (LSTM);

[0014] S22, embedding the discourse time sequence relationship between the video and audio into speaker information and constructing a full-connected graph;

[0015] S23, establishing an edge connection between the video modality and the audio modality corresponding to the same discourse node, using a graph convolution network (GCN) to fuse the video and audio modalities, and obtaining video and audio high-dimensional fusion features;

[0016] S24, training the video and audio high-dimensional fusion features, obtaining prior information through the training, and obtaining better video and audio high-dimensional fusion features.

[0017] S25, repeating the methods of S21 to S24 to obtain better video and text high-dimensional fusion features and text and audio high-dimensional fusion features after training;

[0018] As a further scheme of the present application: the S3 includes the following steps:

[0019] S31, constructing a time sequence relationship of the three high-dimensional fusion features by using a bidirectional GRU;

[0020] S32, fusing the three high-dimensional fusion features by using a graph convolution network (GCN) to obtain high-dimensional high fusion features;

[0021] S33, extracting discourse sentiment tendencies by using a multi-head attention mechanism on the high-dimensional high fusion features after retraining, and mining potential relationships among the video, text and audio modalities;

[0022] S34, obtaining a prediction result by using a fully connected layer (FC);

[0023] S35, improving the accuracy of multi-modal sentiment recognition classification through training.

[0024] As a further scheme of the present application: the training uses classification cross-entropy and L2 regularization as the loss function of training:

[0025]

[0026] Wherein N represents the number of dialogues, c(i) represents the number of utterances in dialogue i, P i,j is the probability distribution of the predicted emotional label of utterance j in dialogue i, y i,j is the real emotional label of utterance j in dialogue i, and lambda is the weight of L2 regularization, and theta is the trainable parameter.

[0027] As a further scheme of the present application: the multi-head attention in S33 has two attention heads.

[0028] 1. The present application further fuses the potential relationship between multi-modal emotional modalities by two-stage graph convolution network (GCN) fusion and training of audio, video and text modalities, and improves the multi-modal emotion recognition classification precision.

[0029] 2. The speech timing relationship of audio, video and text is obtained by bidirectional LSTM, the speaker information is embedded, the full-connected graph is constructed, two modalities among audio, video and text are fused by using graph convolution network (GCN), the high-dimensional fusion features of video and audio are obtained, and the high-dimensional fusion features among audio, video and text are improved.

[0030] 3. The timing relationship of the three high-dimensional fusion features is further constructed by bidirectional GRU, the high-dimensional high-fusion features are obtained by further graph convolution network (GCN) fusion of the three high-dimensional fusion features, the potential relationship between multi-modal emotional modalities is improved, and the multi-modal emotion recognition classification precision is improved. BRIEF DESCRIPTION OF DRAWINGS

[0031] Figure 1 It is a first-stage video and audio fusion training schematic diagram of the present application.

[0032] Figure 2 It is a twice fusion training schematic diagram of audio, video and text modalities of the present application. DETAILED DESCRIPTION

[0033] The embodiments of the present application are described in detail below, and examples of the embodiments are shown in the drawings, wherein the same or similar symbols represent the same or similar elements or elements with the same or similar functions throughout. The embodiments described below by referring to the drawings are exemplary and are only used to explain the present application, and cannot be understood as a limitation of the present application.

[0034] As Figures 1-2As shown, the application discloses a multi-modal emotion recognition method based on a graph convolution network, comprising the following steps: S1, preprocessing video, audio and text original features to obtain text, video and audio features; S2, constructing a speech time sequence relationship between the text, video and audio features, embedding speaker information, using a graph convolution network (GCN) to fuse any two modalities in the text mode, video mode and audio mode, and obtaining three kinds of high-dimensional fusion features; S3, further constructing a time sequence relationship of the three kinds of high-dimensional fusion features, performing graph convolution network (GCN) re-fusion on the three kinds of high-dimensional fusion features to improve the classification accuracy, and obtaining a prediction result through a fully connected layer (FC).

[0035] S2 is the first stage of fusion training of the text, video and audio three modal features, and S3 is the second stage of deepening fusion training of the text, video and audio three modalities based on the first stage.

[0036] Through two-stage graph convolution network (GCN) fusion and training of the audio, video and text three modal features, the potential relationship between the multi-modal emotion modalities is further fused, and the multi-modal emotion recognition classification accuracy is improved.

[0037] Embodiment: first, the text original feature is extracted by using TextCNN, the textCNN is a convolutional neural network (CNN) used for text classification, the textCNN is a simple structure model and can be used as a benchmark model for text classification, and the training speed and classification performance are excellent, the audio original feature is extracted by using the OpenSmile tool package with IS10 configuration, the OpenSmile tool package is a modular and flexible audio feature extractor, and the IS10 configuration makes the extracted audio features rich, and the visual facial expression video feature is extracted by using DenseNet pre-trained on the facial expression recognition Plus (FER+) corpus.

[0038] Then three kinds of features are obtained through preprocessing The three kinds of features are audio, video and text features, two of the three kinds of features are taken as examples, and the audio and video features are taken as examples, as shown in Figure 1 The audio and video features are input into a long short-term memory (LSTM) network to obtain audio and video time sequence results, which are represented by .

[0039] The speaker information is also important for multi-modal emotion recognition, in order to encode the identity information of the speaker, the speaker embedding information is added to the feature, and then the graph is constructed and the speaker information is embedded into the feature, the original speaker information is represented by a vector s i , and the speaker embedding S i is calculated in the following manner:

[0040]

[0041] in For trainable bias, where W s These are trainable weights.

[0042] Then, a Graph Convolutional Network (GCN) is used to fuse the features embedded with speaker information. Each utterance consists of three nodes. Indicates the use of Initialize the node. Features derived from a Long Short-Term Memory (LSTM) network and embedded with speaker information, where

[0043] In the first stage, we constructed a graph with 2N nodes, where N represents the total number of utterances, i.e., inputting features from two modalities, namely video and audio features. After constructing a deep graph neural network using an undirected graph, to better encode the context, for a given G = (V, ε), the graph is constructed as follows:

[0044]

[0045] Where A represents the adjacency matrix, D represents the diagonal matrix of graph G, and I represents the identity matrix. The iterative formulas for the Graph Convolutional Network (GCN) from different layers are calculated as follows:

[0046]

[0047] Where α and β (l) These are hyperparameters, where α represents the activation function and W... (l) H represents the learnable weight matrix. (0) Added as a residual join

[0048] The fused results are used for training, and then better results are obtained. The features of this result are extracted, and then... Figure 1 The examples are replaced with video and text, and text and audio features. The same method is used to obtain the features extracted from video and text, and text and audio. This stage obtains high-dimensional fusion features of video and audio, high-dimensional fusion features of video and text, and high-dimensional fusion features of text and audio, and incorporates prior knowledge through training.

[0049] During training, classification cross-entropy and L2 regularization are used as the loss functions:

[0050]

[0051] where N represents the number of dialogues, c(i) represents the number of utterances in dialogue i, P i,j is the probability distribution of the predicted sentiment label of utterance j in dialogue i, y i,j is the true sentiment label of utterance j in dialogue i, λ is the weight of L2 regularization, and θ is a trainable parameter.

[0052] On the basis of the first stage, a second stage of multi-modal feature fusion and training is further performed. The three high-dimensional fusion features obtained in the first stage are constructed to have a time sequence relationship through a gated recurrent unit (GRU), and then a graph with 3N nodes is constructed. The same graph convolution network (GCN) fusion iteration method as in the first stage is used to further fuse the three modalities to obtain high-dimensional high-fusion features E i The fused results are subjected to multi-head attention with 2 attention heads, and for a given sequence E i , the output C i is obtained.

[0053]

[0054] where V and K are obtained by calculation from E i , d k = d a / h, h represents the number of attention heads, and the multi-head attention mechanism with 2 attention heads can achieve good output results. For convenience, the formula can be abbreviated as:

[0055] C i = ATT(E i )

[0056] Finally, a fully connected layer (FC) is used for sentiment classification, and the final classification result is obtained through training.

[0057] During training, the same classification cross-entropy and L2 regularization as in the first stage are used as the loss function for training.

[0058] The present application is trained through staged fusion and staged training. After the multi-modal features are trained by the fusion network in the first stage, three high-dimensional fusion features containing prior knowledge generated by training are obtained. The high-dimensional fusion features are used as the input of the fusion network in the second stage. The training of the second stage network is completely independent of the training of the first stage network, so that the potential connection between intra-modal and inter-modal features is deeply mined, and the classification accuracy of multi-modal sentiment recognition is improved.

[0059] The above description is only a preferred specific embodiment of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art can make equivalent substitutions or changes to the technical solutions and inventive concepts of the present application within the technical scope disclosed by the present application, which should be covered within the protection scope of the present application.

[0060] Furthermore, the terms "first", "second", "third", "fourth", "fifth" and the like in the description and in the claims, are used for distinguishing between similar elements and not necessarily presented chronologically. It is to be understood that the terms so used are interchangeable under appropriate circumstances and embodiments. By definition, the term "a" or "an" means "one or more". It is to be understood that the terms so used are interchangeable under appropriate circumstances and embodiments. The term "plurality" encompasses both the meaning of two or more and the meaning of three or more unless otherwise indicated by the specific context.

Claims

1. A multi-modal sentiment recognition method based on a graph convolution network, characterized in that, The method comprises the following steps: S1, preprocessing video, audio and text original features to obtain text, video and audio features; S2, constructing a discourse time sequence relationship between the text, video and audio features, embedding speaker information, using a graph convolution network (GCN) to fuse any two of the text, video and audio modalities to obtain high-dimensional fusion features of the three modalities, and training the high-dimensional fusion features; S3, further constructing a time sequence relationship of the three high-dimensional fusion features, performing GCN re-fusion on the three high-dimensional fusion features to improve the accuracy of multi-modal sentiment recognition classification, obtaining a prediction result through a fully connected layer (FC), and then training to improve the accuracy of multi-modal sentiment recognition classification; The step of obtaining the three high-dimensional fusion features in S2 comprises the following steps: S21, obtaining a discourse time sequence relationship between video and audio features through a bidirectional long short-term memory network (LSTM); S22, embedding speaker information in the discourse time sequence relationship between the video and audio, and constructing a fully connected graph; S23, establishing an edge connection between video modalities and audio modalities corresponding to the same discourse node, using a graph convolution network (GCN) to fuse the video and audio modalities to obtain high-dimensional fusion features of the video and audio; S24, training the high-dimensional fusion features of the video and audio, and obtaining prior information through the training to obtain better high-dimensional fusion features of the video and audio; S25, repeating the method of S21 to S24 to obtain better high-dimensional fusion features of the video and text, and high-dimensional fusion features of the text and audio after training.

2. The method of claim 1, wherein the method is a graph convolution network-based multi-modal sentiment recognition method. The preprocessing of the video, audio and text original features in S1 comprises using a TextCNN algorithm to extract and process the text original features, and using an OpenSmile toolkit with an IS10 configuration to extract and process the audio original features; The visual facial expression video features are extracted through pre-training on a facial expression recognition Plus (FER+) corpus using a DenseNet algorithm.

3. The method of claim 1, wherein the method is based on a graph convolutional network. The step S3 comprises: S31, constructing a time sequence relationship of the three high-dimensional fusion features through a bidirectional GRU; S32, fusing the three high-dimensional fusion features through a graph convolution network (GCN) to obtain high-dimensional high fusion features; S33, extracting discourse sentiment tendencies through multi-head attention from the high-dimensional high fusion features after re-training, and mining potential relationships among the video, text and audio modalities; S34, obtaining a prediction result through a fully connected layer (FC); S35, training to improve the accuracy of multi-modal sentiment recognition classification.

4. The method of claim 1 or 3, wherein the method is based on a graph convolutional network. The training uses classification cross-entropy and L2 regularization as a loss function for training: where N represents the number of conversations, c(i) represents the number of utterances in conversation i, P i,j is the predicted probability distribution of the sentiment label of utterance j in conversation i, y i,j is the true sentiment label of utterance j in conversation i, λ is the weight of L2 regularization, and θ is the training parameter.

5. The method of claim 3, wherein the method is based on a graph convolutional network. The multi-head attention in S33 has two attention heads.

Images