Multimodal emotion recognition method based on spatiotemporal feature fusion

Abstract

The present invention proposes a multimodal emotion recognition method based on spatiotemporal feature fusion, comprising the following steps: establishing a multimodal emotion database including speech, expression and posture; constructing a combined network based on an asymmetric non-local neural network and a space-time LSTM; inputting expression images, spectrograms and posture images into a ResNet101 model to obtain a deep feature map; constructing a deep feature sequence from the deep feature map as the input of a spatial LSTM to learn the spatial structural correlation of the feature sequence; using the output of the spatial LSTM as the input of a temporal LSTM to learn the temporal correlation on each frame of the image; fusing the expression, speech and posture features, and inputting the fused features into a DBN network for further fusion training, and finally inputting a softmax layer to obtain a multi-classification result. The present invention extracts speech, expression and posture emotion features based on spatiotemporal LSTM and an asymmetric non-local neural network and performs multimodal feature fusion classification.

Description

Technical Field

[0001] The present invention relates to an emotion recognition method, in particular to a multimodal emotion recognition method of speech, expression and gesture based on spatiotemporal feature fusion, belonging to the technical field of emotion computing and pattern recognition. Background Art

[0002] With the rapid development of computer technology and artificial intelligence technology, the birth of various intelligent machines in life has provided more convenient services for people's lives. However, current intelligent machines can only perform simple tasks and cannot perfectly interact with humans. In this context, emotion recognition technology plays a very important role in intelligent perception and recognition between humans and computers. In the past, the use of single modal information for emotion recognition could not capture enough information for accurate emotion recognition. In recent years, multimodal fusion can provide more diverse and relevant emotional information for emotion recognition, thereby improving the emotion recognition rate and increasing reliability. However, many current studies are emotion recognition on a single modality of speech or expression, or emotion recognition on dual modalities such as speech-text, expression-gesture, and speech-expression. Few people have studied multimodal emotion recognition, such as multimodal emotion recognition with expression, speech, and gesture trimodality or with the addition of other physiological signals such as EEG. The fusion of multiple modalities not only retains the independence of each modal feature, but also contains the correlation of multiple modalities, providing sufficient and relevant emotional information for more accurate emotion recognition.

[0003] After searching, it was found that the Chinese patent with publication number CN109614895A provides a method for multimodal emotion recognition based on attention feature fusion, which mainly uses data from three modalities, text, voice and video, for the final emotion recognition. First, feature extraction is performed on the data of the three modalities respectively. For text, bidirectional LSTM is used to perform text feature extraction.

[0004] The speech modality uses a convolutional neural network for feature extraction, and the video modality uses a 3D convolutional neural network model to extract video features. Then, the features of the three modalities are fused based on the attention feature layer fusion method, which makes full use of the complementary information between different modalities and assigns certain weights to the features of different modalities.

[0005] In recent years, with the rise of deep learning technology, emotion recognition based on traditional methods has gradually been replaced. Most of the previous emotion recognition research was based on traditional methods. Compared with deep neural networks, traditional methods extract fewer and less rich parameters, and have a heavy artificial design component, which will affect subsequent emotion recognition. The deep neural network that replaced it can use different convolution kernels to extract a large amount of useful deep feature information, thereby obtaining better emotion recognition effects and higher emotion recognition rates. Summary of the invention

[0006] The technical problem to be solved by the present invention is to overcome the shortcomings of the prior art and provide a multimodal emotion recognition method of speech, expression and posture based on the fusion of spatiotemporal features, partially transform and combine the existing deep neural network to meet the needs of computer emotion recognition effect. Compared with traditional methods, the effect is more ideal, and multimodal information is fully utilized to improve the accuracy of computer recognition of human emotions.

[0007] The present invention provides a multimodal emotion recognition method based on spatiotemporal feature fusion, comprising the following steps:

[0008] S1. Establish a multimodal emotion database including speech, expression and gesture, and keep the number of valid samples of each modality consistent;

[0009] S2. Construct a combined network based on an asymmetric non-local neural network and a space-time LSTM. The combined network includes an asymmetric non-local module, an effective channel attention module, a spatial LSTM module, and a temporal LSTM module. The asymmetric non-local module and the efficient channel attention module are connected through a ResNet101 network.

[0010] S3, input the expression image, spectrogram and posture image processed in step S1 into the ResNet101 model combined with the network in step S2 to obtain a deep feature map; construct a deep feature sequence from the deep feature map as the input of the spatial LSTM, so as to learn the spatial structure correlation of the feature sequence; use the output of the spatial LSTM as the input of the temporal LSTM to learn the temporal correlation on each frame of the image, and obtain the expression features, speech features and posture features;

[0011] S4. The facial expression features, speech features and posture features are fused through the Asymmetric Non-Local module, and the fused features are input into the DBN network for further fusion training, and finally input into the softmax layer to obtain the classification results.

[0012] Among them, LSTM (Long Short-Term Memory), long short-term memory network. Deep Belief Network (DBN network for short). DBN is composed of a number of stacked restricted Boltzmann Machine (RBM) structural units. The model performs bottom-up autonomous learning through greedy layer-by-layer unsupervised pre-training, fine-tunes the model parameters through back propagation, and finally automatically extracts deep features. The softmax layer is mainly used for normalization. If there are one hundred types of image classification, the output of the softmax layer is a one-hundred-dimensional vector, and the value in the vector is the recognition rate.

[0013] The present invention uses pre-trained ResNet101 and LSTM network as the basic network, and proposes an asymmetric non-local neural network structure and a space-time LSTM network structure, including the following steps: 1) First, the network is constructed by adding an Asymmetric Non-Local module to the pre-trained ResNet101 network to connect high- and low-order feature relationships, and adding an Efficient Channel Attention module to capture the dependency between adjacent channels, thereby obtaining an improved ResNet101 network. Secondly, the processed expression image, spectrogram, and posture image are respectively input into the improved ResNet101 model to obtain a deep feature map; then, a deep feature sequence is constructed for the deep feature map as the input of the spatial LSTM, thereby learning the spatial structural correlation of the feature sequence; finally, the output of the spatial LSTM is used as the input of the temporal LSTM to learn the temporal correlation on each frame of the image.

[0014] 2) The final extracted deep speech, expression and posture emotion features are fused through the Asymmetric Non-Local module.

[0015] 3) The fused features are input into the DBN network for further fusion training, and finally input into the softmax layer to obtain the results.

[0016] The technical solution further optimized by the present invention is as follows:

[0017] In the step S1, the key parts of the video in the database (face, action) are cut out to the same length and divided into frames, that is, the video is converted into a sequence of pictures to obtain expression images and posture images, and each frame of the picture is used as a structural input; Fourier transform is used for the audio part in the database to extract a two-dimensional spectrogram as a structural input.

[0018] In step S2, the ResNet101 network that bridges the Asymmetric Non-Local module and the Efficient ChannelAttention module includes 5 convolution modules, that is, ResNet101 is composed of 5 groups of convolution layers, each of which includes several convolution layers, pooling layers, BN layers, and ReLU layers; the 5 convolution modules are represented by CONV1, CONV2_x, CONV3_x, CONV4_x, and CONV5_x, where the parameters of CONV1 are as follows:

[0019] conv1:f=7×7,c=64,s=2,p=3

[0020] maxpool:f=3×3,s=2

[0021] The output size calculation formula of its convolutional layer is as follows:

[0022]

[0023] In the formula, output_size represents the output size of the convolution layer, input_size represents the input size of the convolution layer, f represents the size of the convolution kernel or pooling, c represents the number of input channels, s represents the step size of the convolution kernel or pooling, and p represents the padding value;

[0024] The parameters of the CONV2_x convolution module are as follows:

[0025]

[0026] The parameters of the CONV3_x convolution modules are as follows:

[0027]

[0028] An Efficient Channel Attention module is connected across the structure of the convolution modules CONV3_x and CONV4_x to achieve local cross-channel interaction without dimensionality reduction, thereby improving performance.

[0029] The convolution layer parameters in CONV4_x and CONV5_x convolution modules are as follows:

[0030]

[0031]

[0032] An Asymmetric Non-Local module is connected across the convolution modules of CONV4_x and CONV5_x to capture long-range dependencies and fuse high- and low-order features.

[0033] In this step, an improved ResNet101 network is obtained by adding an Asymmetric Non-Local module to the pre-trained ResNet101 network to connect high- and low-order feature relationships, and adding an Efficient Channel Attention module to capture the dependencies between adjacent channels.

[0034] In step S2, the Efficient Channel Attention module mainly includes three parts. The first part is to generate a feature map of size 1*1*C by global average pooling of the output of CONV3_x, that is, to ignore its spatial structure for compression and focus on its channel structure; the second part is to calculate the adaptive convolution kernel size; the third part is to obtain the dependency between adjacent channels through one-dimensional convolution 1D-Conv, obtain the weight of each channel, and weight it to the output of CONV4_x to complete local cross-channel interaction;

[0035] The calculation formula for the one-dimensional convolution kernel size is:

[0036]

[0037] In the formula, |Δ| represents the odd number closest to Δ, γ=2, b=1 is set to consider the size of the convolution kernel and the number of parameters, and C represents the number of channels;

[0038] The Asymmetric Non-Local module mainly includes two parts, namely Asymmetric Fusion Non-local Block (AFNB module) and Asymmetric Pyramid Non-local Block (APNB module). The outputs of CONV4_x and CONV5_x are fused by the AFNB module, and the fused features enter the APNB module for further fusion.

[0039] In the step S3, the features obtained by CONV4_x are called low-level features, and the features obtained by CONV5_x are called high-level features. These two features are input into the AFNB module for fusion; the AFNB module uses the high-level features obtained by CONV5_x as Query, and the low-level features obtained by CONV4_x as Key and Value. First, the Query, Key and Value are respectively reduced in dimension using a 1×1 convolution kernel, and then the Key and Value are further extracted by performing a PyramidPooling operation through a spatial pyramid pooling module to obtain a smaller feature representation; then the correlation matrix of the Query and Key is calculated, and finally the result is transposed and multiplied with the Value feature to obtain a high-level and low-level fused feature, and finally it is spliced ​​with the high-level features output by the original CONV5_x to obtain the updated feature representation Y of the original image. F .

[0040] In step S3, the updated feature representation Y F Entering the APNB module, the 1×1 convolution operation is also used to obtain the Query, Key and Value in the Non-Local module. In order to reduce the amount of calculation, only one pyramid pooling operation is performed on the Key and Value through the spatial pyramid aggregation module to obtain a new feature representation. The other processes are the same as the AFNB process. The correlation matrix is ​​calculated based on the Query and Key, and then the correlation matrix is ​​multiplied by the transpose of the Value feature to obtain the final feature, and then this feature is combined with the updated feature representation Y. F Splicing to get the final image feature representation Y P , and finally the bilinear interpolation method is used to restore the feature resolution.

[0041] In step S3, the spatial pyramid aggregation module, i.e., the sampling module, ensures that the output size remains unchanged by sampling S representative points instead of feeding all spatial points; the process is to perform multiple pooling operations of different scales on the input feature map, and obtain 1*1, 3*3, 6*6 and 8*8 matrices respectively, keep the 1*1 and 3*3 matrices unchanged, and add the obtained 6*6 and 8*8 matrices to the Spatial Attention module respectively to capture the spatial dependency of the two matrices, and then perform the Flatten Concat operation on the four matrices to turn them into a vector, and finally obtain the 110 sampling points, and then use a Spatial Attention module to capture their spatial dependency again to obtain 110 representative points.

[0042] In step S3, the Spatial Attention module further captures its spatial dependency. The main process is: given a feature map f of size H×W×C, sum along the channel axis to generate a two-dimensional matrix A of size H×W:

[0043]

[0044] The softmax function is then applied to the flattened matrix A so that each spatial position is assigned a value p(i,j) indicating how important the position is; for the corresponding spatial position, the resulting value is multiplied by all activations along the f channel axis; therefore, the output F of the spatial attention layer can be written as:

[0045] F k (i,j)=f k (i,j)p(i,j),

[0046] In the formula, f k (i,j) represents the input feature map, e A (i,j) indicates the importance of a spatial position.

[0047] The CONV5_X module outputs a 28×28×2048 feature map as the input of the spatial LSTM module. The spatial LSTM can effectively learn the spatial structural correlation between different regions of the expression image, spectrogram or posture image. After learning through the spatial LSTM, each original expression image, spectrogram or posture image can obtain a deep feature after integrating the regional correlation, and then use this deep feature as the input of the next step of the temporal LSTM.

[0048] The temporal LSTM module further extracts the temporal correlation of deep features, and finally obtains an enhanced emotional feature of an expression, spectrogram or gesture image.

[0049] In step S4, the features after trimodal fusion are input into the DBN network for further feature fusion. DBN uses a bottom-up autonomous learning process, and RBM networks at different levels are reconstructed and reproduced alternately, and finally the nonlinear fusion features of the input are automatically extracted. Finally, the nonlinear fusion features are passed through the softmax layer to obtain the multi-classification results.

[0050] Compared with the prior art, the present invention adopts the above technical solution and has the following technical effects:

[0051] 1) The present invention extracts speech, expression and posture emotion features based on spatiotemporal LSTM and asymmetric non-local neural network and performs multimodal feature fusion classification.

[0052] 2) Use the Asymmetric Non-Local module to obtain high- and low-order feature dependency information and enhance the global display.

[0053] 3) Use the Efficient Channel Attention module to achieve local cross-channel interaction without dimensionality reduction, thereby improving performance.

[0054] 4) Use LSTM to obtain the spatial and temporal correlation of image features, extract features containing key information of speech, expression and posture for multimodal fusion classification, and obtain better emotion recognition effect than the method based on traditional feature learning. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] Figure 1 This is a structural framework diagram of the present invention based on spatiotemporal LSTM and asymmetric non-local neural network.

[0056] Figure 2 This is the structural block diagram of the Asymmetric Non-Local module.

[0057] Figure 3 This is a structural diagram of the fusion classification of facial expressions, speech and gestures. DETAILED DESCRIPTION

[0058] The technical solution of the present invention is further described in detail below in conjunction with the accompanying drawings: This embodiment is implemented on the premise of the technical solution of the present invention, and a detailed implementation method and specific operation process are given, but the protection authority of the present invention is not limited to the following embodiments.

[0059] Example 1

[0060] This embodiment proposes a multimodal emotion recognition method based on spatiotemporal feature fusion, which includes the following steps:

[0061] S1. Preprocessing. A multimodal emotion database including speech, expression and gesture is established. The number of valid samples of each modality in the database is consistent. The key parts of the video in the database are cut to the same length and framed to obtain expression images and gesture images. Each frame of the picture is used as a structural input; Fourier transform is used for the audio part in the database to extract the spectrogram of the two-dimensional structure as a structural input. The specific operation is as follows: Experiments are conducted using the Multimodal database. The database includes seven types of emotions: anger, disgust, fear, happiness, calmness, sadness and surprise, and is composed of emotional data of 8 men and 8 women. Each person's data is recorded separately, and all emotions are expressed 5 times, so there are 80 sets of video samples for each emotion, and a total of 560 video samples. The database is preprocessed. First, the face part of the video in the database is cut out to obtain a video containing only the face, and then 8 frames are selected at equal intervals to obtain an expression picture sequence. 8 frames are selected at equal intervals for the audio part of the database, that is, the audio is converted into an audio sequence, and then Fourier transform is used to obtain a spectrogram of the two-dimensional structure to obtain a speech picture sequence. The action part of the database video is cut out to obtain a video containing only the action, and then 8 frames are selected at equal intervals to obtain a posture image sequence. Through the above steps, a three-channel RBG image with a length and width of 224 is obtained.

[0062] S2. Construct a combined network based on asymmetric non-local neural network and space-time LSTM. The combined network includes an Asymmetric Non-Local module, an Efficient Channel Attention module, a spatial LSTM module and a temporal LSTM module. The Asymmetric Non-Local module and the Efficient Channel Attention module are connected through a ResNet101 network. Figure 1 As shown, the functions of each part are as follows:

[0063] S201, 5 convolution modules of ResNet101 that bridge the Asymmetric Non-Local module and the Efficient Channel Attention module. ResNet101 consists of 5 groups of convolution layers, each of which includes several convolution layers, pooling layers, BN layers, and ReLU layers. Use five convolution modules CONV1, CONV2_x, CONV3_x, CONV4_x, and CONV5_x to simply represent the ResNet101 network. The deep network ResNet101 is mainly built with a Bottleneck structure.

[0064] Among them, the parameters of CONV1 are as follows:

[0065] conv1:f=7×7,c=64,s=2,p=3

[0066] maxpool:f=3×3,s=2

[0067] The output size calculation formula of its convolutional layer is as follows:

[0068]

[0069] Among them, output_size represents the output size of the convolution layer, input_size represents the input size of the convolution layer, f represents the size of the convolution kernel or pooling, c represents the number of input channels, s represents the step size of the convolution kernel or pooling, and p represents the padding value.

[0070] In the CONV2_x convolution module, there is no downsampling process, and its parameters are as follows:

[0071]

[0072] In the CONV3_x convolution module, conv1 reduces the number of channels, and conv2 is responsible for reducing the size; due to the change in size, the input is downsampled. The parameters of the CONV3_x convolution module are as follows:

[0073]

[0074] An Efficient Channel Attention module is connected across the convolution module structure of CONV3_x and CONV4_x to achieve local cross-channel interaction without dimensionality reduction, thereby achieving performance improvement. This module mainly consists of three parts. The first part: the output of CONV3_x is subjected to global average pooling to generate a 1*1*512 feature map, that is, its spatial structure is ignored for compression, and its channel structure is focused on. The second part: the adaptive convolution kernel size is calculated. The third part: the dependency between adjacent channels is obtained through one-dimensional convolution 1D-Conv, the weight of each channel is obtained, and the weight is added to the output of CONV4_x to complete local cross-channel interaction.

[0075] The kernel size of the one-dimensional convolution is adapted through a function, so that the layer with a larger number of channels can better interact across channels. Since the coverage of cross-channel information interaction (i.e., the kernel size k of the one-dimensional convolution) is proportional to the channel dimension C, an exponential function with a base of 2 is used here to represent the nonlinear mapping relationship. Therefore, given the channel dimension C, the calculation formula for the convolution kernel size is:

[0076]

[0077] Where |Δ| represents the odd number closest to Δ. Considering the size of the convolution kernel and the number of parameters, we set γ=2, b=1, and C is the number of channels, which is 512.

[0078] The convolution module structure of CONV4_x and CONV5_x is the same as that of CONV3_x, except that the number of channels is increased and the output size is reduced. An Asymmetric Non-Local module is connected across the convolution modules of CONV4_x and CONV5_x to capture long-range dependencies and fuse high- and low-order features. Because the input and output of the connected modules need to keep the same size, the downsampling layer in the convolution module structure of CONV4_x and CONV5_x is deleted, and dilated convolution is used to keep the feature resolution obtained in the last two stages at the original image. The convolutional layer parameters in the last two convolutional modules are as follows:

[0079]

[0080]

[0081] The structure of the Asymmetric Non-Local module is as follows Figure 2 This module mainly consists of two parts, namely Asymmetric Fusion Non-local Block (AFNB) and Asymmetric Pyramid Non-local Block (APNB).

[0082] The outputs of CONV4_x and CONV5_x are fused using the AFNB module, where the features obtained by CONV4_x are called low-level features, and the features obtained by CONV5_x are called high-level features. These two features are input into the AFNB module for fusion. The AFNB module uses the high-level features obtained by CONV5_x as the query, and the low-level features obtained by CONV4_x as the key and value. First, the Query, Key, and Value are reduced in dimension using a 1×1 convolution kernel, and then the Key and Value are further extracted through the Pyramid Pooling operation using the spatial pyramid pooling module to obtain a smaller feature representation. The correlation matrix of the query and key is then calculated, and finally this result is transposed and multiplied with the Value feature to obtain the high- and low-level fused features, which are finally spliced ​​with the high-level features output by the original CONV5_x to obtain the updated feature representation Y of the original image. F .

[0083] The updated feature representation Y FEnter the APNB module and use the same 1×1 convolution operation to obtain the Query, Key, and Value in the Non-Local module. In order to reduce the amount of calculation, only one pyramid pooling operation is performed on the Key and Value through the spatial pyramid pooling module to obtain a new feature representation. The other processes are the same as the AFNB process. The correlation matrix is ​​calculated based on the Query and Key, and then the correlation matrix is ​​multiplied by the transpose of the Value feature to obtain the final feature, and then this feature is combined with the updated feature representation Y F Splicing to get the final image feature representation Y P , and finally the bilinear interpolation method is used to restore the feature resolution.

[0084] Spatial pyramid pooling module, also known as sampling module. This module ensures that the output size remains unchanged by sampling S representative points instead of feeding all spatial points. The process is to perform multiple pooling operations of different scales on the input feature map to obtain 1*1, 3*3, 6*6 and 8*8 matrices respectively, keep the 1*1 and 3*3 matrices unchanged, and add the obtained 6*6 and 8*8 matrices to the Spatial Attention module to capture the spatial dependency of the two matrices, and then perform Flatten Concat operation on the four matrices to turn them into a vector. Finally, after obtaining the 110 sampling points, a Spatial Attention module is used to capture their spatial dependency again to obtain 110 representative points.

[0085] The Spatial Attention module further captures its spatial dependency. Its main process is: given a feature map f of size H×W×C, summing along the channel axis produces a two-dimensional matrix A of size H×W:

[0086]

[0087] The softmax function is then applied to the flattened matrix A so that each spatial position is assigned a value p(i,j) indicating how important that position is. For the corresponding spatial position, the resulting value is multiplied by all activations along the f channel axis. Therefore, the output F of the spatial attention layer can be written as:

[0088] F k (i,j)=f k (i,j)p(i,j),

[0089] In the formula, f k (i,j) represents the input feature map, e A(i,j) indicates the importance of a spatial position.

[0090] S202, spatial LSTM module. Through the above-mentioned spatiotemporal LSTM and asymmetric non-local neural network, we can obtain deep texture features related to expression, speech or posture, but we cannot learn the spatial structural correlation between the various regions of the expression image, spectrogram or posture image, and this correlation can also provide important emotional information. Therefore, we add a spatial LSTM module after the B1 module. The output of the CONV5 module is not pooled, and its convolutional layer parameters are as follows:

[0091]

[0092] A feature map of 28×28×2048 is obtained, and the output feature map size is 28×28. There are 2048 channels, that is, there are 2048 feature maps. In the present invention, 784 feature vectors are formed along the channel direction at 784 positions of 28×28, each vector is 2048-dimensional, and these 784 feature vectors correspond to a small area of ​​an input frame image. Each feature vector is used as the input of LSTM, so that the spatial structural correlation between different areas of the expression image, spectrogram or posture image can be effectively learned through spatial LSTM. After learning through spatial LSTM, each original expression image, spectrogram or posture image can obtain a deep feature after fusion of regional correlation, and then use this deep feature as the input of the next step of time LSTM.

[0093] S203, Temporal LSTM module: A set of feature vectors containing information related to different regions of an image output by part B2 is used as input to LSTM to further extract temporal correlation.

[0094] S201-S203 is the processing process of a frame of image. After a video is divided into frames, all images are processed by the above network. The image of the next frame integrates the temporal LSTM features of the current frame and the previous frame at B3, and finally obtains an enhanced emotional feature of an expression, spectrogram or posture image.

[0095] S3, on the Multimodal database, after preprocessing, images with the same number of expression, voice and posture frames are obtained. A person-independent strategy is adopted, with a total of 16 folds of independent experiments, each fold has 35 samples for verification and 525 samples for training. These samples are input into the network constructed by B1-B3 for training, and the network parameters are updated. After the training, the network model with the best verification effect is saved. The three-modal features of expression, voice and posture extracted using the network model with the best verification effect are fused using the Asymmetric Non-Local module. When in use, the test video is preprocessed and then input into the network. After being processed by S1 and S2, a set of features are obtained respectively. Finally, the obtained expression features, voice features and posture features are fused through the Asymmetric Non-Local module. The specific method is as follows: Figure 3 The features after trimodal fusion are input into DBN network for further feature fusion. DBN uses a bottom-up autonomous learning process. RBM networks at different levels are reconstructed and reproduced alternately, and finally the nonlinear fusion features of the input are automatically extracted. Finally, the nonlinear fusion features are passed through the softmax layer to obtain the multi-classification results.

[0096] The above description is only a specific implementation of the present invention, but the protection scope of the present invention is not limited thereto. Any person familiar with the technology can understand and think of any changes or substitutions within the technical scope disclosed by the present invention, which should be included in the scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.

Claims

1. A multimodal emotion recognition method based on spatiotemporal feature fusion, characterized in that: The following steps are involved: S1. Establish a multimodal emotion database including speech, expression and gesture, and keep the number of valid samples of each modality consistent; S2. Construct a combined network based on an asymmetric non-local neural network and a space-time LSTM. The combined network includes an asymmetric non-local module, an effective channel attention module, a spatial LSTM module, and a temporal LSTM module. The asymmetric non-local module and the effective channel attention module are connected through a ResNet101 network. The ResNet101 network that bridges the asymmetric non-local module and the effective channel attention module includes 5 convolutional modules, each of which includes several convolutional layers, pooling layers, BN layers, and ReLU layers. The 5 convolutional modules are represented by CONV1, CONV2_x, CONV3_x, CONV4_x, and CONV5_x, where the parameters of CONV1 are as follows: conv1:f=7×7,c=64,s=2,p=3 max pool:f=3×3,s=2 The output size calculation formula of its convolutional layer is as follows: In the formula, output_size represents the output size of the convolution layer, input_size represents the input size of the convolution layer, f represents the size of the convolution kernel or pooling, c represents the number of input channels, s represents the step size of the convolution kernel or pooling, and p represents the padding value; The parameters of the CONV2_x convolution module are as follows: The parameters of the CONV3_x convolution modules are as follows: A valid channel attention module is connected across the structure of the convolution modules CONV3_x and CONV4_x; The convolution layer parameters in CONV4_x and CONV5_x convolution modules are as follows: An asymmetric non-local module is connected across the convolution modules of CONV4_x and CONV5_x to capture long-range dependencies and fuse high- and low-order features; S3, input the expression image, spectrogram and posture image processed in step S1 into the ResNet101 model combined with the network in step S2 to obtain a deep feature map; construct a deep feature sequence from the deep feature map as the input of the spatial LSTM, so as to learn the spatial structure correlation of the feature sequence; use the output of the spatial LSTM as the input of the temporal LSTM to learn the temporal correlation on each frame of the image, and obtain the expression features, speech features and posture features; S4. The facial expression features, speech features and posture features are fused through the asymmetric non-local module, and the fused features are input into the DBN network for further fusion training, and finally input into the softmax layer to obtain multi-classification results.

2. The multimodal emotion recognition method based on spatiotemporal feature fusion according to claim 1 is characterized in that: In the step S1, the key parts of the video in the database are cut to the same length and divided into frames to obtain expression images and posture images, and each frame of the image is used as a structural input; Fourier transform is used for the audio part in the database to extract a two-dimensional spectrogram as a structural input.

3. The multimodal emotion recognition method based on spatiotemporal feature fusion according to claim 2 is characterized in that: In step S2, the effective channel attention module mainly includes three parts. The first part is to generate a feature map of size 1*1*C by global average pooling of the output of CONV3_x; the second part is to calculate the adaptive convolution kernel size; the third part is to obtain the dependency between adjacent channels through one-dimensional convolution 1D-Conv, obtain the weight of each channel, and weight it to the output of CONV4_x to complete local cross-channel interaction; The calculation formula for the one-dimensional convolution kernel size is: In the formula, |D| represents the odd number closest to Δ, g=2, b=1 is set to consider the size of the convolution kernel and the number of parameters, and C represents the number of channels; The asymmetric non-local module mainly includes an AFNB module and an APNB module. The outputs of CONV4_x and CONV5_x are fused by the AFNB module, and the fused features enter the APNB module for further fusion.

4. The multimodal emotion recognition method based on spatiotemporal feature fusion according to claim 3 is characterized in that: In the step S3, the features obtained by CONV4_x are called low-level features, and the features obtained by CONV5_x are called high-level features. These two features are input into the AFNB module for fusion; the AFNB module uses the high-level features obtained by CONV5_x as Query, and the low-level features obtained by CONV4_x as Key and Value. First, the Query, Key and Value are respectively reduced in dimension using a 1×1 convolution kernel, and then the Key and Value are further extracted by a pyramid pooling operation through a spatial pyramid pooling module to obtain a smaller feature representation; then the correlation matrix of the Query and Key is calculated, and finally the result is transposed and multiplied with the Value feature to obtain a high-level and low-level fused feature, and finally it is spliced ​​with the high-level features output by the original CONV5_x to obtain the updated feature representation Y of the original image. F .

5. The multimodal emotion recognition method based on spatiotemporal feature fusion according to claim 4 is characterized in that: In step S3, the updated feature representation Y F Entering the APNB module, the 1×1 convolution operation is also used to obtain the Query, Key, and Value in the Non-Local module. In order to reduce the amount of calculation, only one pyramid pooling operation is performed on the Key and Value through the spatial pyramid pooling module to obtain a new feature representation. The correlation matrix is ​​calculated based on the Query and Key, and then the correlation matrix is ​​multiplied by the transpose of the Value feature to obtain the final feature, and then this feature is combined with the updated feature representation Y. F Splicing to get the final image feature representation Y P , and finally the bilinear interpolation method is used to restore the feature resolution.

6. The multimodal emotion recognition method based on spatiotemporal feature fusion according to claim 5 is characterized in that: In step S3, the spatial pyramid pooling module samples S representative points to ensure that the output size remains unchanged. The process is to perform multiple pooling operations of different scales on the input feature map to obtain matrices of 1*1, 3*3, 6*6 and 8*8 respectively, keep the 1*1 and 3*3 matrices unchanged, add the obtained 6*6 and 8*8 matrices to the spatial attention module respectively to capture the spatial dependency of the two matrices, and then perform Flatten Concat operation on the four matrices to convert them into a vector. Finally, after obtaining the 110 sampling points, a Spatial Attention module is used to capture the spatial dependency again to obtain 110 representative points.

7. The multimodal emotion recognition method based on spatiotemporal feature fusion according to claim 6 is characterized in that: In step S3, the spatial attention module further captures its spatial dependency. The main process is: given a feature map f of size H×W×C, summing along the channel axis generates a two-dimensional matrix A of size H×W: The softmax function is then applied to the flattened matrix A so that each spatial position is assigned a value p(i,j) indicating how important the position is; for the corresponding spatial position, the resulting value is multiplied by all activations along the f channel axis; therefore, the output F of the spatial attention layer can be written as: F k (i,j)=f k (i,j)p(i,j), In the formula, f k (i,j) represents the input feature map, e A (i,j) indicates the importance of a spatial position.

8. The multimodal emotion recognition method based on spatiotemporal feature fusion according to claim 7 is characterized in that: The CONV5_X module outputs a 28×28×2048 feature map as the input of the spatial LSTM module. The spatial LSTM can effectively learn the spatial structural correlation between different regions of the expression image, spectrogram or posture image. After learning through the spatial LSTM, each original expression image, spectrogram or posture image can obtain a deep feature after integrating the regional correlation, and then use this deep feature as the input of the next step of the temporal LSTM. The temporal LSTM module further extracts the temporal correlation of deep features, and finally obtains an enhanced emotional feature of an expression, spectrogram or gesture image.

9. The multimodal emotion recognition method based on spatiotemporal feature fusion according to claim 8, characterized in that: In step S4, the features after trimodal fusion are input into the DBN network for further feature fusion. DBN uses a bottom-up autonomous learning process, and RBM networks at different levels are reconstructed and reproduced alternately, and finally the nonlinear fusion features of the input are automatically extracted. Finally, the nonlinear fusion features are passed through the softmax layer to obtain the multi-classification results.

Images