Multimodal emotion recognition method based on spatio-temporal bidirectional expansion causal convolution and transfomer

Abstract

This invention provides a multimodal emotion recognition method based on spatiotemporally bidirectional dilated causal convolution and Transformer, comprising: processing videos of facial expressions and speech modalities to obtain facial expression sequences and spectrogram sequences, and extracting pose features to obtain pose feature sequences; processing facial expressions and speech modalities using a ResNet18 network to obtain feature maps, and rearranging the feature maps to form corresponding spatial feature sequences; capturing spatial dependencies in the obtained spatial feature sequences using a spatial attention bidirectional dilated causal convolutional network; then feeding the facial expression and speech feature sequences and pose feature sequences into a temporal attention bidirectional dilated causal convolutional network to capture temporal dependencies; stacking the facial expression, speech, and pose features to obtain a feature matrix, and achieving feature fusion using Transformer. This invention captures the spatiotemporal dependencies within a modality through a spatiotemporally bidirectional dilated causal convolutional network, obtaining high-level modal features that aggregate spatiotemporal information, and uses Transformer to capture feature interactions between modalities, thus achieving multimodal feature fusion.

Description

Technical Field

[0001] This invention relates to a multimodal emotion recognition method based on spatiotemporal bidirectional dilated causal convolution and Transformer, belonging to the field of emotion computing technology. Background Technology

[0002] In recent years, artificial intelligence (AI) technology has developed rapidly, attracting widespread attention and recognition from researchers and the general public. Affective computing, as a branch of AI, has benefited from the booming development of natural language processing, machine learning, deep learning, and image recognition technologies, resulting in advancements in its algorithms and recognition performance. Affective computing uses algorithms to enable computers to learn human emotions from modal features such as facial expressions, speech, posture, and physiological states, and to classify these emotions. The goal is to enable computers to perceive, understand, and express emotions like humans, thereby achieving human-computer interaction. Currently, affective computing can be applied in areas such as intelligent customer service, market research, and mental health monitoring.

[0003] Multimodal emotion recognition is a method in emotion computing that combines and analyzes various forms of human emotion expression (such as facial expressions, speech, posture, text, and physiological states) to achieve the goal of recognizing and classifying human emotions. Compared to unimodal emotion recognition, multimodal emotion recognition can obtain more comprehensive and richer emotional information and learn from it to achieve more accurate recognition results. Multimodal emotion recognition mainly includes important steps such as preprocessing, feature extraction, and feature fusion. Preprocessing refers to processing raw data such as audio, video, and physiological signals to obtain data suitable for artificial intelligence algorithms. Feature extraction involves extracting features from a single modality using relevant algorithms to obtain features that better reflect human emotions. These features can reflect the emotional tendencies of different modalities such as text, speech, and images. Feature fusion involves fusing the features obtained from multiple modalities after feature extraction for downstream classification.

[0004] In the field of artificial intelligence, CNNs (Convolutional Neural Networks) and RNNs (Recurrent Neural Networks) are widely used in algorithm research. Traditionally, it is believed that RNNs and Long Short-Term Memory (LSTM) networks excel at handling sequential data due to their ability to capture temporal dependencies. However, in recent years, CNNs have also achieved significant breakthroughs in sequence modeling tasks, particularly for certain types of sequential data, such as text, audio, and time series data. To better adapt to sequential data, several variants of CNN models have been proposed, such as dilated causal convolution. Dilated causal convolution combines causal convolution with dilated convolution, effectively handling time series problems and capturing long-term temporal dependencies. For a specific moment, dilated causal convolution ensures that information from the future is not leaked to that moment; that is, this specific moment is only related to information from previous moments. Simultaneously, dilated convolution guarantees a larger receptive field, enabling it to capture long-term temporal dependencies. This invention proposes to introduce spatially bidirectional dilated causal convolution, which can take into account information from the past and future and model long-term dependencies. It constructs a spatiotemporal bidirectional dilated causal convolution and considers that information from both directions may have different effects on the recognition effect. It adopts attention-based dynamic fusion and finally fuses the obtained high-level features of facial expression, speech and posture through Transformer.

[0005] In view of this, it is indeed necessary to provide a multimodal emotion recognition method based on spatiotemporal bidirectional dilated causal convolution and Transformer to solve the above problems. Summary of the Invention

[0006] The purpose of this invention is to construct a spatiotemporal bidirectional extended causal convolutional network and use Transformer to achieve feature fusion between modalities, providing a new method to improve the accuracy of computer recognition of human emotions.

[0007] To achieve the above objectives, this invention provides a multimodal emotion recognition method based on spatiotemporal bidirectional dilated causal convolution and Transformer, comprising the following steps:

[0008] Step 1: Process the facial expression modality using the OpenCV library to obtain a video sequence. Use the Dlib library to obtain the faces in the video sequence to obtain facial expression images. Extract 8 frames of facial expression images at equal intervals from the facial expression images, and the 8 frames of facial expression images constitute a facial expression image sequence. Use the Spicy library to convert the speech modality from the video sequence into an audio sequence. Use the pydub library to segment the audio sequence to obtain 5 audio segments and a complete audio segment. Extract Log-Mel spectrograms, first-order spectrograms, and second-order spectrograms from the 5 audio segments and the complete audio segment to obtain a spectrogram sequence of length 6. Extract 10 frames of pose features at equal intervals from the video sequence to obtain a pose feature sequence.

[0009] Step 2: Process the facial expression modality and the speech modality of each frame using a ResNet18 network to obtain facial expression modality feature maps and speech modality feature maps. Rearrange the facial expression modality feature maps and the speech modality feature maps to form the facial expression space feature sequence and speech space feature sequence corresponding to each frame of the facial expression and speech modality sequence.

[0010] Step 3: Capture the spatial dependencies of the facial expression space feature sequence and the speech space feature sequence respectively using a spatial attention bidirectional extended causal convolutional network;

[0011] Step 4: The facial expression space feature sequence, the speech space feature sequence, and the pose feature sequence obtained after processing by the spatial bidirectional dilated causal convolutional network are fed into the temporal attention bidirectional dilated causal convolutional network to capture temporal dependencies;

[0012] Step 5: Stack the facial expression features, speech features, and posture features obtained in Step 4 to obtain a feature matrix, and then perform feature fusion using Transformer.

[0013] As a further improvement of the present invention, the parameters of the first four modules of the ResNet18 network used in step two are as follows:

[0014] conv1:k=7,s=2,p=3,d=64,

[0015] Maxpooling:k=3,s=2,p=1,

[0016] Where k represents the size of the convolution kernel or pooling, s represents the stride of the convolution kernel or pooling, p represents the padding size, and d represents the number of output channels;

[0017]

[0018] The size of the convolutional kernel is 3×3, the size of the padding and the stride is 1, and the number of output channels is 64, resulting in a 56×56×64 feature map;

[0019]

[0020] After the third convolutional layer, the size of the feature map is 28×28, and there are 128 feature maps.

[0021]

[0022] After the fourth convolutional layer, the feature map size is 14×14, and there are 256 feature maps. After the first four convolutional layers of the ResNet18 network, a CNN layer is used to further reduce the size of the feature maps. The parameters of the CNN are as follows:

[0023] conv:k=1,d=32,s=1,p=1,

[0024] Maxpooling:k=2,s=1,p=1,

[0025] After passing through the above convolutional network, the 224×224×3 image is processed to obtain a 7×7×32 feature map, and the 7×7×32 feature map is rearranged.

[0026] As a further improvement of the present invention, the rearrangement method is as follows: the 7×7×32 feature map is arranged into 49 feature vectors along the channel direction, and the dimension of each feature vector is 32.

[0027] As a further improvement of the present invention, in step three, the spatial attention bidirectional dilated causal convolutional network processes facial expression space features and speech space features, captures the spatial information of the facial expression space features and the speech space features, and obtains corresponding feature vectors that capture the spatial information by passing the facial expression space feature sequence and the speech space feature sequence obtained in step two through attention bidirectional dilated causal convolution. The feature vectors obtained by the facial expression space features and the speech space features corresponding to different frame images constitute a temporal feature sequence.

[0028] As a further improvement of the present invention, in step four, the temporal attention bidirectional dilated causal convolutional network further captures the temporal dependence of the temporal feature sequence through the temporal attention bidirectional dilated causal convolutional network.

[0029] As a further improvement of the present invention, in step four, the pose feature sequence is directly fed into the temporal attention bidirectional extended causal convolutional network for processing.

[0030] As a further improvement of the present invention, in step five, the Transformer network fuses the feature vectors of facial expressions, speech and posture modalities that capture spatial and temporal information obtained in step four into a joint vector, and feeds the joint vector into the softmax layer to obtain the multi-class classification result.

[0031] The beneficial effects of this invention are as follows:

[0032] Compared with existing technologies, the significant advantages of this invention are: fully utilizing the advantages of dilated causal convolution for sequence modeling problems, constructing a spatiotemporal bidirectional dilated causal convolutional network, and dynamically fusing features from both directions to capture spatial and temporal dependencies; fusing extracted facial expression, speech, and posture features based on Transformer, which can better capture information interaction between modalities, and using the fused features for classifier classification, achieving better emotion recognition results than other traditional methods. Attached Figure Description

[0033] Figure 1 This is a flowchart of the method of the present invention;

[0034] Figure 2 This is a schematic diagram of the spatiotemporally dilated causal convolutional network of the present invention;

[0035] Figure 3 This is a block diagram of the attention-based bidirectional dilatational causal convolution structure;

[0036] Figure 4 This is a block diagram of the Transformer Encoder structure. Detailed Implementation

[0037] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be described in detail below with reference to the accompanying drawings and specific embodiments.

[0038] It should be noted that, in order to avoid obscuring the present invention with unnecessary details, only the structures and / or processing steps closely related to the present invention are shown in the accompanying drawings, while other details that are not closely related to the present invention are omitted.

[0039] Additionally, it should be noted that the terms “comprising,” “including,” or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus.

[0040] like Figure 1As shown, this invention presents a multimodal emotion recognition method based on spatiotemporal bidirectional dilated causal convolution and Transformer. Building upon existing research, this invention proposes an attention-based spatiotemporal bidirectional dilated causal convolutional network, fully leveraging the advantages of dilated causal convolution in processing sequential data to capture spatial and temporal dependencies within modalities, such as... Figure 2 As shown, speech and facial expression images are first processed by a CNN to obtain feature maps. These feature maps are then rearranged, with information from the same position in different feature maps forming a vector, and all vectors from different positions forming a spatial feature sequence. This sequence is then processed by an attention-based bidirectional dilated causal convolution to obtain spatially dependent feature vectors. Images at different time steps are processed in the same way to obtain temporal feature sequences. These temporal sequences are then fed into the attention-based bidirectional dilated causal convolution again to obtain feature vectors capturing both spatial and temporal dependencies, which are used for feature fusion. The structure of the attention-based bidirectional dilated causal convolution network is as follows: Figure 3 As shown, each residual module includes two layers of dilated causal convolutions, weight normalization, a ReLU activation function, and a two-dimensional Dropout layer for regularization after each dilated causal convolution. The feature fusion part adopts a Transformer-based fusion approach, using the Transformer Encoder module as shown... Figure 4 As shown, it mainly includes four important steps: position encoding, multi-head attention mechanism, feedforward neural network, and residual connection and normalization.

[0041] This invention discloses a multimodal emotion recognition method based on spatiotemporal bidirectional dilated causal convolution and Transformer, comprising the following steps:

[0042] Step 1: After processing the video of facial expression modality, obtain 8 frames of facial expression images at equal intervals to form an facial expression sequence. Segment the audio to obtain 5 audio segments. Extract Log-Mel spectrograms, first-order spectrograms, and second-order spectrograms from these 5 audio segments and the complete audio to obtain a speech sequence of length 6. Extract 10 frames of pose features at equal intervals, with each frame being a 150-dimensional vector to form a pose feature sequence.

[0043] Step 2: For facial expressions and speech modalities, each frame is processed through a ResNet18 network to obtain feature maps. Then, the feature maps are rearranged to form the spatial feature sequence corresponding to each frame of the facial expression and speech modal sequence.

[0044] Step 3: Capture the spatial dependencies of the spatial feature sequences corresponding to each frame of the facial expression and speech modal obtained in the previous step using a spatial attention bidirectional dilated causal convolutional network.

[0045] Step 4: Feed the facial expression and speech feature sequences and pose feature sequences obtained after processing by the spatially bidirectional dilated causal convolutional network into the temporally attention-based bidirectional dilated causal convolutional network to capture temporal dependencies;

[0046] Step 5: Stack the facial expression, speech, and posture features obtained through the above steps to obtain a feature matrix, and then use Transformer to achieve feature fusion, resulting in better emotion recognition performance than other traditional methods.

[0047] The following will provide a detailed explanation of steps one through five.

[0048] Step one mainly involves preprocessing the database. For the facial expression modality, the video is processed using the OpenCV library to obtain a video sequence. Then, the faces in the video are obtained using the Dlib library to obtain facial expression images. Eight frames of facial expression images are then obtained at equal intervals to form an image sequence. For the speech modality, the video is converted into audio using the Spicy library. The audio is then segmented using the pydub library to obtain five audio segments and a complete audio segment. Log-Mel spectrograms, first-order spectrograms, and second-order spectrograms are extracted from these six audio segments to form a spectrogram sequence.

[0049] The ResNet18 network used in step two includes its first four modules, with parameters as shown below:

[0050] conv1:k=7,s=2,p=3,d=64

[0051] Maxpooling: k = 3, s = 2, p = 1

[0052] Where k represents the size of the convolution kernel or pooling, s represents the stride of the convolution kernel or pooling, p represents the padding size, and d represents the number of output channels.

[0053]

[0054] The convolution kernel size is 3×3, the padding and stride size is 1, and the output channels are 64. This layer does not change the data size or the number of channels, resulting in a 56×56×64 feature map.

[0055]

[0056] After the third convolutional layer, the output feature map size is 28×28, with a total of 128 feature maps.

[0057]

[0058] After the fourth convolutional layer, the output feature map size is 14×14, with a total of 256 feature maps.

[0059] After the first four convolutional layers of the ResNet18 network, a CNN layer is used to further reduce the feature map size, with the following parameters:

[0060] conv:k=1,d=32,s=1,p=1

[0061] Maxpooling: k = 2, s = 1, p = 1

[0062] After passing through the above convolutional network, the 224×224×3 image is processed to obtain a 7×7×32 feature map.

[0063] To obtain the spatial feature sequence, the obtained feature map is rearranged as follows: the 7×7×32 feature map is used to form 49 feature vectors along the channel direction, with each feature vector having a dimension of 32. This method constructs the spatial feature sequence for each frame of the speech and facial expression modal, where each feature vector contains information from the same spatial location in different channels.

[0064] The implementation of the attention-based bidirectional dilated causal convolutional network used in steps three and four to capture spatial and temporal dependencies involves: reversing the sequence, passing the forward and reverse sequences through several dilated causal convolutional modules respectively, and then dynamically fusing the two directions. Each dilated convolutional module mainly includes two dilated causal convolutions and residual connections. The dilated causal convolutions have two characteristics: 1. The convolutions in the architecture are causal, meaning there is no information "leakage" between the future and the past; 2. A larger receptive field is obtained by selecting the filter size k and the dilated convolution stride d. For a one-dimensional sequence input x∈R... n And the filter f: {0,...,k-1}→R, the unwinding convolution operation F on sequence elements s is defined as:

[0065]

[0066] As the network depth increases, a larger receptive field is obtained by exponentially increasing the expansion step size d. In the i-th layer of the network, d = 2. i-1 The results from both directions are dynamically fused, implemented as follows: g out =w1·g+w2g inverse , g is the sequence after passing through an extended causal convolutional network, g inverse The sequence is the result of the inverse input sequence passing through an extended causal network, and w1 and w2 are trainable parameters.

[0067] In step five, to achieve better cross-modal feature interaction, a Transformer-based feature fusion approach is adopted. The Transformer's Encoder mainly includes four important steps: positional encoding, multi-head attention mechanism, feedforward neural network, and residual connection and normalization. Transformer-based feature fusion primarily utilizes the multi-head attention mechanism, where the Transformer employs an attention module defined as a scaling dot product function.

[0068]

[0069] Q, K, and V are the query matrix, key matrix, and value matrix, respectively. The Transformer computes multiple such parallel attentions, where the output of each attention is called a head. The formula for calculating the i-th head is:

[0070] head i =Attention(QW i q ,KW i k VW i v )

[0071] These are header-specific parameters used to linearly project a matrix into a local space.

[0072] In our fusion process, we first obtain the feature matrix. h v h a h g These are facial expression, speech, and pose features, respectively. We then utilize a multi-head self-attention mechanism based on the Transformer Encoder to achieve intermodal interaction. We set:

[0073]

[0074] Multihead(M)=Concat(head1,head2,...,head n W o

[0075] W o Is and Using a similar weight matrix, we finally construct a joint vector from the Transformer's output. Used for downstream classification, the fused vector is fed into the softmax layer to obtain the multi-class classification result.

[0076] In summary, this invention proposes a spatiotemporal bidirectional extended causal convolutional network to capture spatial and temporal dependencies within a modality, thereby obtaining high-level modal features that aggregate spatiotemporal information. It also employs a Transformer to capture feature interactions between modalities, thus achieving multimodal feature fusion.

[0077] The above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention.

Claims

1. A multimodal emotion recognition method based on spatiotemporal bidirectional dilated causal convolution and Transformer, characterized in that, Includes the following steps: Step 1: Process the facial expression modality using the OpenCV library to obtain a video sequence. Use the Dlib library to obtain the faces in the video sequence to obtain facial expression images. Extract 8 frames of facial expression images at equal intervals from the facial expression images, and the 8 frames of facial expression images constitute a facial expression image sequence. Use the Spicy library to convert the speech modality from the video sequence into an audio sequence. Use the pydub library to segment the audio sequence to obtain 5 audio segments and a complete audio segment. Extract Log-Mel spectrograms, first-order spectrograms, and second-order spectrograms from the 5 audio segments and the complete audio segment to obtain a spectrogram sequence of length 6. Extract 10 frames of pose features at equal intervals from the video sequence to obtain a pose feature sequence. Step 2: Process the facial expression modality and the speech modality of each frame using a ResNet18 network to obtain facial expression modality feature maps and speech modality feature maps. Rearrange the facial expression modality feature maps and the speech modality feature maps to form the facial expression space feature sequence and speech space feature sequence corresponding to each frame of the facial expression and speech modality sequence. Step 3: Capture the spatial dependencies of the facial expression space feature sequence and the speech space feature sequence respectively using a spatial attention bidirectional extended causal convolutional network; Step 4: The facial expression space feature sequence, the speech space feature sequence, and the pose feature sequence obtained after processing by the spatial bidirectional dilated causal convolutional network are fed into the temporal attention bidirectional dilated causal convolutional network to capture temporal dependencies; Step 5: Stack the facial expression features, speech features, and posture features obtained in Step 4 to obtain a feature matrix, and then perform feature fusion through Transformer; The Transformer network fuses the feature vectors of facial expression, speech, and posture modalities that capture spatial and temporal information obtained in Step 4 into a joint vector, and feeds the joint vector into a softmax layer to obtain the multi-class classification result; The implementation of the attention-based bidirectional dilated causal convolutional network used in steps three and four to capture spatial and temporal dependencies includes: reversing the sequence, passing the forward and reverse sequences through several dilated causal convolutional modules respectively, and then dynamically fusing the two directions. Each dilated convolutional module includes two dilated causal convolutions and residual connections.

2. The multimodal emotion recognition method based on spatiotemporal bidirectional dilated causal convolution and Transformer as described in claim 1, characterized in that, The parameters of the first four modules of the ResNet18 network used in step two are as follows: ， ， in, Indicates the size of the convolution kernel or pooling layer. Indicates the stride of the convolution kernel or pooling. Indicates the fill size. Indicates the number of output channels; ， The size of the convolution kernel is 3.

3. The padding and step size are both 1, and the number of output channels is 64, resulting in 56. 56 64 feature maps; ， After the third convolutional layer, the size of the feature map is 28. 28, There are 128 feature maps; ， After the fourth convolutional layer, the feature map has a size of 14.

14. The feature map consists of 256 images; after the first four convolutional layers of the ResNet18 network, a CNN layer is used to further reduce the size of the feature map. The parameters of the CNN are as follows: ， ， 224 224 Image processing of 3 yields 7 7 32 feature maps, for the 7 7 The 32 feature maps are rearranged.

3. The multimodal emotion recognition method based on spatiotemporal bidirectional dilated causal convolution and Transformer as described in claim 2, characterized in that, The rearrangement method is as follows: The 7 7 The 32 feature maps form 49 feature vectors along the channel direction, and each feature vector has a dimension of 32.

4. The multimodal emotion recognition method based on spatiotemporal bidirectional dilated causal convolution and Transformer as described in claim 1, characterized in that, In step three, the spatial attention bidirectional dilated causal convolutional network processes facial expression space features and speech space features, captures the spatial information of the facial expression space features and the speech space features, and uses the facial expression space feature sequence and the speech space feature sequence obtained in step two to obtain corresponding feature vectors that capture the spatial information through attention bidirectional dilated causal convolution. The feature vectors obtained from the facial expression space features and the speech space features corresponding to different frame images constitute a temporal feature sequence.

5. The multimodal emotion recognition method based on spatiotemporal bidirectional dilated causal convolution and Transformer as described in claim 4, characterized in that, In step four, the temporal attention bidirectional dilated causal convolutional network further captures the temporal dependence of the temporal feature sequence.

6. The multimodal emotion recognition method based on spatiotemporal bidirectional dilated causal convolution and Transformer as described in claim 1, characterized in that, In step four, the pose feature sequence is directly fed into the temporal attention bidirectional dilated causal convolutional network for processing.

Images