A continuous emotion recognition method based on multimodal fusion and optimized TCN

Abstract

The present invention belongs to the field of machine vision technology, and discloses a continuous emotion recognition method based on optimized TCN and multimodal fusion, including: establishing discrete emotion and continuous emotion conditional probability models, generating pseudo-label completion data sets based on the models, obtaining complete label data, extracting audio data and image data from video data, performing preliminary feature extraction on the audio data and image data, using optimized TCN to perform temporal feature extraction on the features, obtaining temporal features of audio features and video features, inputting the initial temporal features into the attention model for cross-modal feature fusion, and then combining global maximum pooling to extract the features after cross-modal feature fusion, and after vector splicing the initial image temporal features and the global cross-modal features, obtaining the final result of continuous emotion recognition. The present invention simultaneously takes into account the audio information, image information and label information of the video, and can effectively improve the accuracy of continuous emotion recognition.

Description

Technical Field

[0001] The present invention belongs to the field of machine vision technology, and specifically relates to a continuous emotion recognition method based on multimodal fusion and optimized TCN. Background Art

[0002] Video emotion recognition is becoming a crucial component of many human-computer interaction systems. Existing emotion recognition models are primarily categorized as discrete emotion models and continuous emotion models. Discrete emotion models have garnered widespread attention in the field of emotion recognition due to their simplicity and intuitiveness. However, because they consistently use specific adjectives to represent emotions, they suffer from the following drawbacks: 1) a limited range of emotions they can represent; 2) an inability to describe the emotional development process; and 3) high correlations between different emotion categories, making this correlation difficult to measure and process using discrete models. To overcome these drawbacks of discrete models, researchers have developed a dimensional emotion model, or continuous emotion model. This model has two dimensions: valence and arousal. Valence measures a person's level of pleasure, while arousal measures a person's psychological activity and level of alertness. Emotions are continuously distributed along this two-dimensional coordinate system.

[0003] Since emotion recognition networks based on continuous emotion models can better describe the development of emotions, in-depth research on continuous emotion recognition is essential. However, existing continuous emotion recognition networks often suffer from inaccurate recognition due to missing labels in the dataset itself. To address this issue, existing methods generally increase the data volume by merging other datasets, but this leads to problems such as excessive data volume and high network training costs. Furthermore, existing methods often increase the number of network layers to improve recognition rates, which in turn causes the network to lose shallow information, resulting in difficulties in network training and gradient vanishing. Summary of the Invention

[0004] In order to overcome the shortcomings of the existing technology, the present invention proposes a continuous emotion recognition method based on multimodal fusion and optimized TCN. This continuous emotion recognition method can increase data by generating pseudo labels without using additional data sets; at the same time, an optimized TCN network is provided. Compared with the traditional TCN network, the present invention adds a feature extraction submodule to the TCN basic module to increase the learning ability of the network; at the same time, in order to solve the problems of gradient disappearance and difficulty in training large models, the present invention will also add a residual module while adding the feature extraction submodule to retain shallow features.

[0005] In order to achieve the above object, the present invention is achieved through the following technical solutions:

[0006] The present invention is a continuous emotion recognition method based on multimodal fusion and optimized TCN, which includes the following steps:

[0007] Step 1: Construct data samples, obtain discrete emotion data labels and continuous emotion data set labels, and establish discrete emotion and continuous emotion conditional probability models.

[0008] Step 2: Generate pseudo labels using the discrete emotion and continuous emotion conditional probability models established in step 1 to fill in the missing pseudo label datasets;

[0009] Step 3: Obtain the video data to be analyzed and convert it into audio data and image data. Extract the audio data and image data from the video data. Then use the Vggish network and MFCC network to extract preliminary features of the audio data. Then use the ResNet50 network to extract preliminary features of the image data.

[0010] Step 4: Use the optimized TCN network to extract the time series features of the preliminary features of the audio data and image data to obtain the audio time series output features, audio time series output features and image time series output features.

[0011] Step 5: Position-encode the audio temporal output features, audio temporal output features, and image temporal output features obtained in step 4, multiply them by the weight matrix respectively, and then perform cross-modal feature fusion to obtain cross-modal features;

[0012] Step 6: Input the cross-modal features obtained in step 5 into the multi-head attention model to fuse the self-attention mechanism of the cross-modal features, and extract the cross-modal self-attention features by combining the global maximum pooling;

[0013] Step 7: Based on the image temporal output features obtained in step 4 and the cross-modal self-attention features extracted in step 6, vector splicing is performed to obtain the global cross-modal features, which are input into the fully connected layer to obtain the final result of continuous emotion recognition.

[0014] The further improvement of the present invention is that: Step 1 specifically comprises: constructing a data sample, statistically analyzing the distribution of continuous emotion data labels in the discrete emotion data labels in the data sample, and obtaining a discrete emotion and continuous emotion conditional probability model:

[0015]

[0016] Among them, v and a represent the valence dimension and arousal dimension of continuous emotion respectively, i represents seven discrete emotions plus one neutral emotion classification, E is a set of eight integers from 0 to 7, and n i (v,a) represents the number of valence and arousal dimension labels in discrete emotion classification of class i.

[0017] A further improvement of the present invention is that the step 2 specifically includes the following steps:

[0018] Step 2-1: When the continuous emotion label value of a video frame is determined by the discrete emotion and continuous emotion conditional probability model, the discrete emotion label value with the largest probability is selected as a pseudo label to fill the label value of the video frame that has a continuous emotion label but lacks a discrete emotion label;

[0019] Step 2-2: Randomly extract a pair of (v, a) values from the continuous emotion distribution coordinates corresponding to the discrete emotion as the pseudo-label of the frame. During network training, randomly extract a set of (v, a) labels based on the labeled values of the discrete emotion. After each training session, verify the network performance on the validation set, and output and save the set of pseudo-labels corresponding to the network with the best performance on the validation set.

[0020] A further improvement of the present invention is that step 3 specifically includes the following steps:

[0021] Step 3-1, first convert the video file to a mono audio file;

[0022] Step 3-2: Extract the time code files of the video and audio respectively, compare the time codes of the video and audio, and interpolate black frames in the video for the time code difference;

[0023] Step 3-3: Extract the face image from the video file to obtain the image data. At the same time, convert the audio into a Mel-spectrogram. The spectrogram is then cut into smaller sub-spectrograms. The center time of the sub-spectrogram is aligned with the current frame to obtain the audio data.

[0024] Step 3-4: Input the audio data into the Vggish network and the MFCC network respectively, extract two audio features as two branches of audio features, and then use the ResNet50 network to extract features of the image data as a branch of image features.

[0025] A further improvement of the present invention is that the optimized TCN network includes 4 TCN basic modules, each TCN basic module includes three feature extraction submodules, each feature extraction submodule includes an expanded convolution layer, a normalization layer, an activation layer and a dropout layer, and a residual connection is added between each two adjacent feature extraction submodules to retain shallow features. At the same time, a 1×1 convolution module is added to the first residual connection to preliminarily extract shallow features.

[0026] A further improvement of the present invention is that: Step 4 is: the preliminary audio features X of the audio data extracted by the Vggish network and the MFCC network are V , preliminary audio features XM And the preliminary image features X extracted by the ResNet50 network for image data RN Input the optimized TCN network, use the optimized TCN network to establish a temporal connection between audio features and image features, and obtain the audio temporal output feature X Vt , audio timing output characteristics X Mt And image temporal output feature X RNt , specifically including the following steps:

[0027] Step 4-1: The preliminary image feature X RN Divide the image sequence input feature X into frames RNT , the preliminary audio feature X V and preliminary audio features X M Extract audio features of length 1 to t from the starting frame time of the corresponding video as the audio sequence input feature X VT and audio timing input characteristics X MT ,set up are the input feature vector and output feature vector with time step T and length F0, where the time step T is in the interval 1≤T≤t;

[0028] Step 4-2: Input the audio sequence into feature X VT , audio timing input characteristics X MT And image sequence input features X RNT Input into 4 TCN basic modules, each of which contains L layers of dilated convolution sequence, represents the activation function of the l-th layer dilated convolution sequence of the j-th optimized TCN module, each S (j，1) (X t ) are all inputs of the previous module S (j，l )(X t ), the dilated convolution of each layer should only use the features of two time steps, that is, the features of the current time t and the previous time ts, and the filter F equipped by the feature extraction module w Parameterized as W = (W (1) , W (2)}, Let the bias vector be Assume that the LeakyReLU activation function is f(·), and bring in the above parameters to get S (j，l) (X t )About S (j，l-1) (X t ) and S (j，l-2) (X t ) equation:

[0029]

[0030]

[0031] Let X t ={X VT , X MT , X RNT}, V={V (1) , V (2)}, are the weight matrix and bias vector of the residual connection respectively;

[0032] Step 4-3: Establish the audio feature X V , X M and image features X RN The timing connection between them is used to obtain the audio timing output feature X Vt , audio timing output characteristics X Mt And image temporal output feature X RNt

[0033] The beneficial effects of the present invention are:

[0034] The continuous emotion recognition method based on multimodal fusion and optimized TCN proposed in this paper uses a conditional probability model between discrete emotions and continuous emotions to generate pseudo labels to supplement the missing labels of video frames, thereby improving the utilization rate of video data.

[0035] The present invention uses an optimized TCN network to extract the temporal features of audio features and video features, and reasonably adds a feature extraction module and a residual module to enhance the network learning ability while making the network easier to train.

[0036] The present invention adopts a multi-head attention mechanism to fuse audio features and video features, obtains the splicing features of cross-modal features and video features to perform continuous emotion recognition tasks, and improves the network's noise resistance. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1 is a flow chart of the method of the present invention;

[0038] Figure 2 It is a model structure diagram of the present invention;

[0039] Figure 3 Schematic diagram of the optimized TCN network feature extraction submodule structure in an embodiment of the present invention. DETAILED DESCRIPTION

[0040] The following diagrams illustrate embodiments of the present invention. For clarity, many practical details are included in the following description. However, it should be understood that these practical details are not intended to limit the present invention. In other words, in some embodiments of the present invention, these practical details are not essential.

[0041] like Figure 1 As shown, the present invention is an aspect-level sentiment analysis method based on graph convolutional networks and selective attention, which specifically includes the following steps:

[0042] Step 1: Obtain the Aff-wild2 dataset publicly available online to construct data samples, obtain discrete emotion data labels and continuous emotion dataset labels, establish discrete emotion and continuous emotion conditional probability models, and use the discrete emotion and continuous emotion conditional probability models to generate pseudo labels to fill in the datasets with missing labels. The specific steps are as follows:

[0043] Step 1.1: Count the distribution of continuous emotion data labels in the discrete emotion data labels in the data sample. Obtain the discrete emotion and continuous emotion conditional probability model:

[0044]

[0045] Where v and a represent the valence dimension and arousal dimension of continuous emotion respectively, i represents seven discrete emotions plus one neutral emotion classification, E is a set of eight integers from 0 to 7, and n i (v,a) represents the number of valence and arousal dimension labels in discrete emotion classification of class i.

[0046] Step 1.2: Through the discrete emotion and continuous emotion conditional probability model, when the continuous emotion label value of the video frame is determined, the discrete emotion label value with the largest probability is selected as the pseudo label to fill the label value of the video frame that has a continuous emotion label but lacks a discrete emotion label.

[0047] Step 1.3: When the discrete emotion labels are completed using the aforementioned method, the present invention randomly extracts a pair of (v, a) values from the continuous emotion distribution coordinates corresponding to the discrete emotion as a pseudo-label for the frame. In subsequent network training, a set of (v, a) labels is randomly extracted based on the discrete emotion label values. After each training session, the network performance is verified on the validation set, and the set of pseudo-labels corresponding to the network that performs best on the validation set is output and saved.

[0048] Step 2: Obtain the video data to be analyzed and convert it into audio data and image data. The specific steps are as follows:

[0049] Step 2.1: Use the MKVtoolnix tool to convert the video file to MKV format, and then use the mkvmerge command to extract the 16K sampling rate WAV format mono audio file from the MKV file.

[0050] Step 2.2: Extract the time code files of the video and audio respectively, compare the time codes of the video and audio, and interpolate black frames in the video for the time code difference.

[0051] Step 2.3: Use the Openface tool to extract a 48×48×3 face image from the video file to obtain the image data. Simultaneously, use the TorchAudio package in PyTorch to convert the audio into a mel-spectrogram. The spectrogram is then split into smaller sub-spectrograms, with the center time t of each sub-spectrogram aligned with the current frame to obtain the audio data.

[0052] Step 3: Input the completed data samples, aligned audio samples and image samples described in steps 1 and 2 into the network, as shown in Figure 2 As shown in the figure, the pre-trained Vggish network, MFCC network and ResNet50 network are used to perform preliminary feature extraction on the audio data and image data respectively to obtain the preliminary audio feature X V , X M and preliminary image features X RN .

[0053] Step 4: Convert the preliminary audio features X v , X M and preliminary image features X RN Input optimized TCN network, the optimized TCN network includes 4 TCN basic modules, each TCN basic module includes three feature extraction submodules, each feature extraction submodule includes an expansion convolution layer, a normalization layer, an activation layer and a dropout layer, and a residual connection is added between each two adjacent feature extraction submodules to retain shallow features. At the same time, a 1×1 convolution module is added to the first residual connection to preliminarily extract shallow features, such as Figure 3 As shown. Using the optimized TCN network, a temporal connection is established between the audio features and the image features to obtain the audio temporal output feature X Vt , X Mt And image temporal output feature X RNt , the specific steps are as follows:

[0054] Step 4.1: The preliminary image feature X RN Divide the image sequence feature X into frames RNT ; The preliminary audio feature X V and X MExtract audio features of length 1 to t from the start frame time of the corresponding video as the audio sequence feature X VT and XMT. Let XRNT, XVT, is the input feature vector and output feature vector with time step T and length F0, where the time step T is in the interval of , where 1≤T≤t.

[0055] Step 4.2: Input the audio feature sequence and image feature sequence into 4 TCN basic modules, each module contains L layers of dilated convolution sequence, represents the activation function of the l-th layer dilated convolution sequence of the j-th optimized TCN module. Each S (j，1) (X t ) are all inputs of the previous module S (j，l) (X t ) output. The dilated convolution of each layer should only use the features of two time steps, namely the features of the current time t and the previous time ts. The filter F equipped with the feature extraction module w Parameterized as W = {W (1) , W (2)}, Let the bias vector be Assume that the LeakyReLU activation function is f(•), and bring in the above parameters to get S (j，l) (X t )About S (j，l-1) (X t ) and S (j，l-2) (X t ) equation:

[0056]

[0057]

[0058]

[0059] Let X t ={X VT , X MT , X RNT}, V={V (1) , V (2)}, are the weight matrix and bias vector of the residual connection respectively.

[0060] Step 4.3: Establish the audio feature X V , X M and image features X RN The timing connection between them is used to obtain the audio timing output feature X Vt, audio timing output characteristics X Mt And image temporal output feature X RNt .

[0061] Step 5: Output the audio time series as feature X Vt , audio timing output characteristics X Mt And image temporal output feature X RNt Multiply them by the weight matrix respectively, and then perform cross-modal feature fusion to obtain cross-modal features. The specific steps are as follows:

[0062] Step 5.1: Output the audio time series as feature X Vt , audio timing output characteristics X Mt And image temporal output feature X RNt Perform position encoding.

[0063] Step 5.2: Multiply the time series features after position encoding by the three coefficient matrices Get a query vector Q respectively i , a key vector K i and a value vector V i , where i represents 3 modes.

[0064] Step 5.3: Transform the query vector Q i , key vector K i Sum value vector V i Recombining the connections, forming a cross-modal feature vector Q = [Q1, Q2, Q3]. Then using the multi-head attention calculation formula, and then applying the maximum pooling process to it to obtain the attention feature vector:

[0065]

[0066] where d k =32 is the dimension of the key vector. Set the attention feature vector as the cross-modal attention feature vector X Att .

[0067] (4) The cross-modal feature attention vector X Att Combined with the temporal feature vector of the image data to obtain the global cross-modal feature vector X Att+RN :

[0068] X Att+RN = X att + X RNt (6)

[0069] Step 5: Input the obtained cross-modal features into the multi-head attention model to fuse the self-attention mechanism of cross-modal features, and extract cross-modal self-attention features by combining global maximum pooling;

[0070] Step 6: Based on the obtained image temporal output features and the extracted cross-modal self-attention features, vector splicing is performed to obtain the global cross-modal features, which are input into the fully connected layer to obtain the final result of continuous emotion recognition.

[0071] The continuous emotion recognition method of the present invention greatly improves the utilization rate of video frames, increases the learning ability of the network, better retains the shallow features of the network, makes the network easier to train, and can effectively predict and recognize continuous emotions.

Claims

1. A continuous emotion recognition method based on multimodal fusion and optimized TCN, characterized by: The continuous emotion recognition method comprises the following steps: Step 1: Construct data samples, obtain discrete emotion data labels and continuous emotion data set labels, and establish discrete emotion and continuous emotion conditional probability models; Step 2: Generate pseudo labels using the discrete emotion and continuous emotion conditional probability models established in step 1 to fill in the missing pseudo label datasets. The specific steps include the following: Step 2-1: When the continuous emotion label value of a video frame is determined by the discrete emotion and continuous emotion conditional probability model, the discrete emotion label value with the largest probability is selected as a pseudo label to fill the label value of the video frame that has a continuous emotion label but lacks a discrete emotion label; Step 2-2: Randomly extract a pair of (v, a) values from the continuous emotion distribution coordinates corresponding to the discrete emotion as the pseudo-label for the frame. During network training, randomly extract a set of (v, a) labels based on the discrete emotion label values. After each training session, verify the network performance on the validation set. The set of pseudo-labels corresponding to the network with the best performance on the validation set is output and saved. v and a represent the valence dimension labeling and arousal dimension labeling of the continuous emotion, respectively. Step 3: Obtain the video data to be analyzed and convert it into audio data and image data. Extract the audio data and image data from the video data. Then use the Vggish network and MFCC network to extract preliminary features of the audio data. Then use the ResNet50 network to extract preliminary features of the image data. Step 4: Use the optimized TCN network to extract temporal features from the preliminary features of the audio data and image data to obtain two audio temporal output features and image temporal output features. The optimized TCN network includes four TCN basic modules, each of which includes three feature extraction submodules. Each feature extraction submodule includes an expanded convolution layer, a normalization layer, an activation layer, and a dropout layer. A residual connection is added between each two adjacent feature extraction submodules to retain shallow features. At the same time, a 1×1 convolution module is added to the first residual connection to preliminarily extract shallow features. Step 5: Position-encode the audio temporal output features and the image temporal output features obtained in step 4, multiply them by the weight matrix respectively, and then perform cross-modal feature fusion to obtain cross-modal features; Step 6: Input the cross-modal features obtained in step 5 into the multi-head attention model to fuse the self-attention mechanism of the cross-modal features, and extract the cross-modal self-attention features by combining the global maximum pooling; Step 7: Based on the image temporal output features obtained in step 4 and the cross-modal self-attention features extracted in step 6, vector splicing is performed to obtain the global cross-modal features, which are input into the fully connected layer to obtain the final result of continuous emotion recognition.

2. The continuous emotion recognition method based on multimodal fusion and optimized TCN according to claim 1 is characterized in that: The step 1 specifically includes: constructing a data sample, counting the distribution of continuous emotion data labels in the discrete emotion data labels in the data sample, and obtaining a discrete emotion and continuous emotion conditional probability model: Among them, v and a represent the valence dimension and arousal dimension of continuous emotion respectively, i represents seven discrete emotions plus a neutral emotion classification, E is a set of eight integers from 0 to 7, and n i (v,a) represents the number of valence and arousal dimension labels in discrete emotion classification of class i.

3. The continuous emotion recognition method based on multimodal fusion and optimized TCN according to claim 1 is characterized in that: The step 3 specifically includes the following steps: Step 3-1, first convert the video file to a mono audio file; Step 3-2: Extract the time code files of the video and audio respectively, compare the time codes of the video and audio, and interpolate black frames in the video for the time code difference; Step 3-3: Extract the face image from the video file to obtain the image data. At the same time, convert the audio into a Mel-spectrogram. The spectrogram is then cut into smaller sub-spectrograms. The center time of the sub-spectrogram is aligned with the current frame to obtain the audio data. Step 3-4: Input the audio data into the Vggish network and the MFCC network respectively, extract two audio features as two branches of audio features, and then use the ResNet50 network to extract features of the image data as a branch of image features.

4. The continuous emotion recognition method based on multimodal fusion and optimized TCN according to claim 1 is characterized in that: The step 4 is: the preliminary audio features X of the audio data extracted by the Vggish network and the MFCC network v , preliminary audio features X M And the preliminary image features X extracted by the ResNet50 network for image data RN Input the optimized TCN network, use the optimized TCN network to establish a temporal connection between audio features and image features, and obtain the audio temporal output feature X Vt , audio timing output characteristics X Mt And image temporal output feature X RNt , specifically including the following steps: Step 4-1: The preliminary image feature X RN Divide the image sequence input feature X into frames RNT , the preliminary audio feature X V and preliminary audio features X M Extract audio features of length 1 to t from the starting frame time of the corresponding video as the audio sequence input feature X VT and audio timing input characteristics X MT ,set up are the input feature vector and output feature vector with time step T and length F0, where the time step T is in the interval 1≤T≤t; Step 4-2: Input the audio sequence into feature X VT , audio timing input characteristics X MT And image sequence input features X RNT Input into 4 TCN basic modules, each of which contains L layers of dilated convolution sequence, represents the activation function of the l-th layer dilated convolution sequence of the j-th optimized TCN module, each S (j,1) (X t ) are all inputs of the previous module S (j,l) (X t ), the dilated convolution of each layer should only use the features of two time steps, that is, the features of the current time t and the previous time ts, and the filter F equipped by the feature extraction module w Parameterized as Let the bias vector be Assume that the LeakyReLU activation function is f(·), and bring in the above parameters to get S (j,l) (X t )About S (j,l-1) (X t ) and S (j,l-2) (X t ) equation: set up are the weight matrix and bias vector of the residual connection respectively; Step 4-3: Establish the audio feature X V , X M and image features X RN The timing connection between them is used to obtain the audio timing output feature X Vt , audio timing output characteristics X Mt And image temporal output feature X RNt .

Images