Multi-modal emotion recognition method for feature extraction and modal fusion based on voice text image

Abstract

The invention provides a multi-modal emotion recognition method for performing feature extraction and modal fusion based on a voice text image. A three-modal feature extractor is designed to accurately capture voice, text and image features contained in an original video, and a cross-modal attention mechanism is introduced to fuse feature sequences extracted from different modals. In order to solve the problem of lack of control during fusion of different modal feature sequences, a modal complementary control gate structure is introduced into the model. Meanwhile, in a cross-modal attention mechanism, a convolutional layer guide model is introduced to pay attention to local features of different modals. Experiments on a Chinese multi-mode emotion recognition data set CH-SIMSv2.0 prove the effectiveness of the network provided by the invention.

Description

Technical Field

[0001] The present invention is a multimodal emotion recognition method based on speech, text and image feature extraction and modality fusion. Background Art

[0002] In recent years, with the rapid development of human-computer interaction, sentiment analysis, as an important part of the dialogue between intelligent entity dialogue agents and users, has attracted the attention of more and more researchers. In the early studies of unimodal sentiment analysis, most studies used text data for sentiment recognition, because with the rise of the Internet and social media, people can easily express and share their emotional states through text information. Therefore, text has become one of the most common data sources for sentiment recognition. However, traditional unimodal sentiment analysis will inevitably face challenges in more complex situations such as text ambiguity and irony, which will lead to errors in sentiment analysis results. In order to cope with this situation, in recent years, researchers have introduced multimodal fusion methods into emotion recognition tasks. Multimodal fusion is a very critical research direction in the fields of artificial intelligence, multimodal machine learning, etc. Its main goal is to build a model for processing and understanding multimodal information, combining data from different modalities to achieve more robust and accurate predictions of the same event.

[0003] Multimodal sentiment analysis introduces information from multiple modalities such as text, video, and audio, and analyzes sentiment by extracting information expressed in different modal data. Compared with early research on sentiment analysis of single-modal data, this method can not only overcome the problems of low accuracy and poor robustness of single-modal sentiment analysis, but also use the complementarity between different modalities to improve the representation ability of sentiment features, thereby improving the accuracy and reliability of sentiment analysis.

[0004] Most of the past work in the field of multimodal emotion recognition has focused on exploring more effective feature fusion methods. Zadeh et al. proposed a tensor fusion network that fuses features extracted from different modalities by performing Cartesian product on different modalities, and uses a low-rank weight fusion method to reduce the computational time and space complexity; Wei et al. proposed a "multimodal information maximum" framework to improve the performance of multimodal emotion recognition tasks by maximizing the mutual information between the multimodal fusion features and the single modality before fusion. As the attention mechanism has shown great potential in the field of deep learning in recent years, a large number of multimodal emotion recognition methods based on the attention mechanism have been proposed. Han et al. proposed a method for dynamically extracting inter-modal information based on a gating mechanism, which effectively solves the problem of unbalanced information extraction between modalities; Tsai et al. proposed an end-to-end multimodal emotion recognition network based on a cross-modal attention mechanism, which avoids explicit alignment of data while potentially capturing common information between different modalities.

[0005] Although the above-mentioned attention-based methods have achieved remarkable success in the field of multimodal emotion recognition, there are still some defects in practicality. For example, most research works are based on the feature vectors extracted in advance from the existing multimodal emotion recognition dataset, and do not consider the step of extracting features from the original video in practical applications; secondly, the existing Chinese multimodal emotion dataset contains a small number of labeled samples, which is prone to overfitting in model training; finally, most of the existing multimodal emotion recognition methods require a long time for data processing and cannot meet the real-time requirements in practical applications. In view of the above defects, the present invention proposes an end-to-end multimodal fusion network based on a cross-modal attention mechanism, which realizes an end-to-end multimodal emotion recognition method from original video to emotion recognition results through a feature extraction module. Summary of the invention

[0006] The present invention proposes an end-to-end multimodal fusion network based on a cross-modal attention mechanism, which realizes an end-to-end multimodal emotion recognition method from raw video to emotion recognition results through a feature extraction module. Figure 1 As shown, the technical solution for implementing the present invention includes two processes: network training and reasoning.

[0007] The network training process includes the following steps:

[0008] Step 1: Obtain audio video clips and corresponding labels, and divide them into training sets and test sets;

[0009] Step 2: Use two-dimensional convolutional layers and residual blocks to build a facial feature extraction network, and train the network with the training set to ensure that the network parameters have good effects on the test set;

[0010] Step 3: Extract frames from the image part of the audio video clip according to the time dimension, and input the extracted image frames into the pre-trained facial feature extraction network to obtain the image feature vector;

[0011] Step 4: Use one-dimensional convolutional layers and residual blocks to build an audio feature extraction network, and train the network with the training set to ensure that the network parameters have good effects on the test set;

[0012] Step 5: Serialize the audio part of the audio video clip according to the time dimension, and input the serialized audio into the pre-trained audio feature extraction network to obtain an audio feature vector;

[0013] Step 6: Use the two-dimensional convolutional layer and the Self-attention Transformer block to build a text feature extraction network, and train the network with the training set to ensure that the network parameters have a good effect on the test set;

[0014] Step 7: Use 2D convolution and Cross-attention Transformer block to build image-text modality fusion layer BBFNC visual With audio-text modality fusion layer BBFNC audio ;

[0015] Step 8: Use the multimodal emotion recognition dataset to train the modal fusion layer and feature extraction layer, and use the AdamW optimizer to update the network weight gradient;

[0016] Step 9: The features obtained in the above step 8, which are finally obtained through the feature extraction layer and the modal fusion layer, are passed through a residual connection feedforward neural network to obtain the final predicted emotion recognition result.

[0017] In step 3, for each video clip, the visual feature extractor first detects whether each frame contains a face, then extracts 16 frames at the same interval step size from the video frames containing faces, and cuts out images based on the original video frames according to the face position of the participant as the facial feature extraction frames of the video. Each facial extraction frame will be sent to the facial feature extraction layer to extract facial features and image features, and then the facial features and image features will be fused into a 16-dimensional 2D tensor {v1,v2,…,v 16} as the extracted facial feature sequence.

[0018] The step 5 uses a pre-trained one-dimensional residual convolutional neural network to encode each audio that does not exceed 5.12s into a 512-dimensional 2D tensor as the audio feature {a1, a2, ..., a 512}.

[0019] The specific steps for building the feature fusion layer used in step 8 are as follows:

[0020] Step 8.1: Feed the extracted 512-dimensional text features into the self-attn transformer block;

[0021] Step 8.2: Add position encoding to the embedding vector. The encoding method adopts absolute position encoding. The encoded embedding vector is also input into the self-attn transformer block.

[0022] Step 8.3: Normalize the embedding dimension of the embedding vector containing position information through the normalization layer, and then obtain the Q, K, and V matrices through three fully connected layers to calculate the attention matrix Where d is the dimension of the embedding vector;

[0023] Step 8.4: Normalize the weighted information block again and input it into the fully connected layer to obtain the output of Transformerblock;

[0024] Step 8.5: The extracted text features are normalized by the normalization layer and then fed into the cross-attention layer as the Q matrix of the cross-attn transformer block;

[0025] Step 8.6: The extracted 512-dimensional text features are used as the K matrix and V matrix of the cross-attention layer and sent to the cross-attn transformer block;

[0026] Step 8.7: Calculate the attention matrix in the criss-cross attention layer After that, the result is sent to the conformer-block to learn local features and obtain the output of the i-th modality fusion layer

[0027] Step 8.8: Based on the idea of ​​the gating mechanism, the i-th modal fusion layer collects the output of the previous modal fusion layer After that, use the bidirectional GRU to Perform serial encoding, then use average pooling to obtain sequence-level hidden features and use them as control parameters of the modal complementary control gate in represents the learnable parameters of the i-th layer bidirectional GRU;

[0028] Step 8.9: Use the modal complementary control gate parameters of different modalities to obtain the control mode fusion layer output sequence containing new sequence information With old sequence information The proportionality factor in represents the scale factor of the old sequence, represents the proportional coefficient of the new sequence, || represents the concatenation of the two;

[0029] Step 8.10: Based on the above two proportional coefficients, the output of the i-th modal fusion layer can be obtained: Where LN represents layer normalization, and FFN represents feed-forward neural network;

[0030] Repeat steps 8.1 to 8.10 above to obtain all modal fusion layers of the network model.

[0031] The network reasoning process includes the following steps: first, decomposing the video into text, speech and image, respectively inputting them into the trained text, speech and image feature extraction modules to obtain the feature vectors corresponding to different modalities; second, using a multi-layer modal fusion layer to learn the information in different modal features and the correlation between different modalities; then, using a gating mechanism to fuse the information extracted between different modal fusion layers, so that each modal fusion layer pays attention to different information; finally, the output of the modal fusion layer is calculated through a fully connected layer to obtain the final emotion prediction result.

[0032] The beneficial effect of this method is that in order to deal with the problem of lack of control when fusing feature sequences of different modalities, a modality complementary control gate structure is introduced into the model. At the same time, in the cross-modal attention mechanism, the convolutional layer conformerblock is introduced to guide the model to focus on the local features of different modalities. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Attached Figure 1 This is a flow chart of a multimodal emotion recognition method based on speech, text and image feature extraction and modality fusion described in the present invention.

[0034] Attached Figure 2 It is the specific structure of any multimodal fusion layer in the multimodal fusion module. DETAILED DESCRIPTION

[0035] A typical embodiment of a multimodal emotion recognition method for feature extraction and modal fusion based on speech, text and image of the present invention is described in detail below, and the method is further described in detail. It is necessary to point out here that the following embodiments are only used to further illustrate the method and cannot be understood as limiting the scope of protection of the method. Personnel skilled in the art in this field may make some non-essential improvements and adjustments to the method based on the above content of the method, which still fall within the scope of protection of the present invention.

[0036] For step 1, we directly use the dataset CH-SIMSv2.0, which includes 4402 supervised instances and 10161 unsupervised instances. The average video length of each instance is 3.6 seconds. It contains rich acoustic and visual emotion-bearing backgrounds, highlighting the role of non-language modalities as emotion carriers. Each supervised instance in the dataset contains a video, the multimodal emotion label of the video, and the unimodal emotion labels corresponding to the visual, text, and audio modalities. Each emotion label is a real number distributed in [-1.0, +1.0], and this interval is divided into five equal parts, representing Negative, Weakly_Negative, Neutral, Weakly_Positive, and Positive from small to large.

[0037] For step 2, a facial feature extraction network needs to be constructed. The network consists of a residual network consisting of 1 layer of 3×3 convolution kernels, 64 channels, stride 1, and padding 1, 3 residual blocks, and 1 layer of 3×3 convolution kernels, 1 channel, stride 1, and padding 1, and a multiplier connection between the output of the last convolution layer and the facial features. The residual block consists of two layers of 3×3 convolution kernels, 64 channels, stride 1, and padding 1, and a residual connection between the input of the residual block and the output of the second convolution layer. Unless otherwise specified, no activation function is added after the second convolution layer of the residual block, and GeLU activation function is added after all other convolution layers, which will not be repeated later.

[0038] For step 3, the input image is extracted from a given video clip. For a video clip with a time length of t seconds, according to the idea of ​​averaging, every For a given input frame rate image I, its dimension is (B, 3, 256, 256), where B represents the batch size used for training, 3 represents the number of channels of the image, and the last two dimensions represent the length and width of the image respectively. The feature of the image obtained by the first layer of convolution is fea = conv1 (I), then the feature of the image is I fea =conv2(Resnet(fea)), where Resnet(.) represents a residual network consisting of 3 residual blocks.

[0039] For step 4, it is necessary to build an audio feature extraction network, which consists of four 1D convolutions, where the convolution kernel size is 2, the padding is 1, and the step sizes are 1, 2, 3, and 4. The final output dimension of the feature extraction network is (B, 1, 64, 256), where B represents the batch size used during training.

[0040] For step 5, the original input audio is extracted through the audio feature extraction network built in step 4. Specifically, the input audio is first truncated to an integer multiple of 640 dimensions, and then sequentially passes through the built one-dimensional convolutional layers of different step sizes to extract audio features of different time granularities. The final audio feature dimension is (B, 1, 64, 256), which will be sent to the subsequent multimodal fusion layer as the audio modality.

[0041] For step 7, it is necessary to construct a modal complement control gate in the modal fusion layer. The modal complement control gate controls the ratio of each modal fusion layer to the previous layer. Specifically, the i-th modal fusion layer collects the output of the previous modal fusion layer. After that, use the bidirectional GRU to Perform serial encoding, then use average pooling to obtain sequence-level hidden features, and use them as control parameters of the modal complementary control gate: in Represents the learnable parameters of the bidirectional GRU in the i-th layer. After obtaining the control parameters of the modality complementary control gate, the modality fusion layer uses the cross-modality attention mechanism to fuse the sequence information of two different modalities and obtain new sequence information Conf means using convolutional layers to extract local detail information, and MHA means cross-modal attention mechanism.

[0042] For step 8, the modal complementary control gate parameters of different modalities are used to obtain the control mode fusion layer output sequence containing new sequence information With old sequence information The proportionality factor in represents the scale factor of the old sequence, represents the scale factor of the new sequence, and || represents the concatenation of the two. Next, based on the above two scale factors, the output of the i-th modal fusion layer can be obtained: Finally, the modal fusion layer output of each layer is Where LN stands for layer normalization and FFN stands for feed-forward neural network.

[0043] For step 9, after obtaining BBFNC visual Modules and BBFNC audio The last multimodal fusion layer output of the module and back, and They will be concatenated and sent to the residual connection feedforward neural network to obtain the final predicted emotion recognition result. That is, the final output of the above steps is Final prediction results

[0044] To ensure that each modal fusion layer of the multimodal fusion module can maintain the independence of its own modal fusion sequence before and after modal fusion, for the modality m,n∈{t,a,v}, we transform the prediction result p obtained by the modal complementary control gate in the i-th modal fusion layer into i and the corresponding pseudo-labels y automatically generated during training i The binary cross entropy between

[0045]

[0046] Where t represents the total number of modality fusion layer stacks, and B is the batch size during training.

[0047] At the same time, in order to ensure that the pre-trained model for extracting visual features pays enough attention to the facial expressions of the characters during feature extraction, we use the mean square error loss to constrain the feature sequence after visual feature extraction based on the visual unimodal labels of the multimodal Chinese dataset CHSIMS. Specifically, for the extracted visual feature sequence X v , there is the following visual feature loss L v :

[0048]

[0049]

[0050] FFN1 and FFN2 are feedforward neural networks independent of the feature fusion module, which are used to extract the visual feature sequence X. v Predicting visual emotion.

[0051] For the final predicted multimodal emotion recognition results We use mean square error loss as the loss function L m :

[0052]

[0053] According to the above losses, the final loss function of the present invention is:

[0054] Loss = L m +λ1L v +λ2L sep

[0055] Where λ1 and λ2 are both hyperparameters.

[0056] In the example of the present invention, the data set used for network training is the multimodal Chinese emotion recognition data set CH-SIMSv2.0, which is carried out in the Pytorch1.10.0 environment, VsCode software, Ubuntu20.04 LTS operating system, 128GB memory, Intel(R)Xeon(R)Bronze 3204CPU@1.90GHz CPU, NVIDIA RTX 3090GPU platform. During the model training process, the optimizer uses the Adam gradient optimizer, and the gradient learning rate is set to 5×10-4; the coefficient λ1 of the loss function is set to 0.15, and λ2 is set to 0.1; the batch size during network training is 4, and the entire training process is iterated 24 times. The training takes about 3 days, and the model after each iteration is verified on the test set using evaluation indicators. The number of iterations for the best model effect is about 12 times.

[0057] Table 1 Three models in CH-SIMS vComparison of evaluation results on the 2.0 dataset

[0058]

[0059] It can be seen from the evaluation results in Table 1 that the various indicators of the multimodal emotion recognition method based on feature extraction and modality fusion of speech, text and image proposed in the present invention are better than the model effects proposed recently.

[0060] For the multimodal emotion recognition task, the present invention proposes a multimodal emotion recognition method based on speech, text and image for feature extraction and modal fusion. The invention designs a trimodal feature extractor to accurately capture the speech, text and image features contained in the original video, and introduces a cross-modal attention mechanism to fuse the feature sequences extracted from different modalities. In order to address the problem of lack of control when fusing feature sequences of different modalities, we introduced a modal complementary control gate structure into the model. At the same time, in the cross-modal attention mechanism, we introduce a convolutional layer to guide the model to focus on the local features of different modalities. On the Chinese multimodal emotion recognition dataset CH-SIMS v Experiments on 2.0 demonstrate the effectiveness of the network proposed in this invention.

Claims

1. A multimodal emotion recognition method based on feature extraction and modality fusion of speech, text and image, characterized in that: It includes two processes: network training and reasoning: The network training process is specifically described as follows: Step 1: Obtain audio video clips and corresponding labels, and divide them into training sets and test sets; Step 2: Use two-dimensional convolutional layers and residual blocks to build a facial feature extraction network, and train the network with the training set to ensure that the network parameters have good effects on the test set; Step 3: Extract the image part of the audio video clip according to the time dimension, and input the extracted image frame into the pre-trained facial feature extraction network to obtain the image feature vector X visual ; Step 4: Use multiple layers of one-dimensional convolutional layers and residual blocks to build an audio feature extraction network, and train the network with the training set to ensure that the network parameters have a good effect on the test set; Step 5: Serialize the audio part of the audio video clip according to the time dimension, and input the serialized audio into the pre-trained audio feature extraction network to obtain the audio feature vector X audio ; Step 6: Use the two-dimensional convolutional layer and the Self-attention Transformer block to build a text feature extraction network, and input the text information in the audio video clip into the text feature extraction network to obtain the text feature vector X text ; Step 7: Use two-dimensional convolution and Cross-attention Transformer block to build the image-text modality fusion layer BBFNC visual With audio-text modality fusion layer BBFNC audio ; Step 8: Substitute the different modal feature vectors X obtained in steps 3, 5, and 6 above into i ,i∈{visual,audio,tex} inputs the image-text modality fusion layer and audio-text modality fusion layer built in step 7, and uses the AdamW optimizer to update the network weight gradient. The specific process is described as follows: a, extract the 512-dimensional text feature X text Send it to the self-attn transformer block; b. Add position encoding to the embedding vector. The encoding method adopts absolute position encoding. The encoded embedding vector is also input into the self-attn transformer block. c. The embedding vector containing position information is normalized through the normalization layer to normalize the embedding dimension, and then the Q, K, and V matrices are obtained through three fully connected layers to calculate the attention matrix Where d is the dimension of the embedding vector; d. Normalize the weighted information block again and input it into the fully connected layer to obtain the output of the Transformer block; e. The extracted text features are normalized through the normalization layer and then sent to the cross-attention layer as the Q matrix of the cross-attntransformerblock; f, calculate the attention matrix in the cross-attention layer After that, the result is sent to the conformer-block to learn local features and obtain the output of the i-th modality fusion layer g, based on the idea of ​​gating mechanism, the i-th modal fusion layer collects the output of the previous modal fusion layer After that, use the bidirectional GRU to Perform serial encoding, then use average pooling to obtain sequence-level hidden features and use them as control parameters of the modal complementary control gate in represents the learnable parameters of the i-th layer bidirectional GRU; h, using the modal complementary control gate parameters of different modalities to obtain the control The output sequence of the modal fusion layer contains new sequence information With old sequence information The proportionality factor in represents the scale factor of the old sequence, represents the proportional coefficient of the new sequence, || represents the concatenation of the two; i, based on the above two proportional coefficients, the output of the i-th modal fusion layer can be obtained: Where LN represents layer normalization, and FFN represents feed-forward neural network; Step nine, the features finally obtained in the above step eight through the feature extraction layer and the modality fusion layer are used to obtain the final predicted emotion recognition result through a residual connection feedforward neural network; The network reasoning process is specifically described as follows: Step 1: Decompose the video into text, speech and image, and input them into the trained text, speech and image feature extraction modules respectively to obtain feature vectors corresponding to different modalities; Step 2: Use multi-layer modal fusion layers to learn the information within different modal features and the correlation between different modalities; Step 3: Use the gating mechanism to fuse the information extracted between different modal fusion layers, so that each modal fusion layer pays attention to different information; Step 4: The output of the modal fusion layer is calculated through the fully connected layer to obtain the final emotion prediction result.

2. The multimodal emotion recognition method based on feature extraction and modality fusion of speech, text and image according to claim 1 is characterized in that: In the network training process, the modal fusion layer in step 7 uses modal complementary control to control the ratio of each modal fusion layer to the previous layer: Specifically, the i-th modal fusion layer collects the output of the previous modal fusion layer After that, use the bidirectional GRU to Perform serial encoding, then use average pooling to obtain sequence-level hidden features, and use them as control parameters of the modal complementary control gate: in represents the learnable parameters of the i-th layer bidirectional GRU.

Images