Transform-based cross-modal fusion multi-modal emotion recognition method

Abstract

The invention discloses a Transform-based cross-modal fusion multi-modal emotion recognition method and device, which are used for solving the problems of modal isomerism, difficulty in time alignment and insufficient dynamic emotion modeling in a multi-modal emotion recognition task, and the method takes the accuracy and robustness of emotion recognition as performance evaluation indexes. Firstly, feature information of three modes of vision, voice and text is obtained, feature extraction is performed on each mode through a deep learning model, then features of different modes are fused by using a cross-mode Transform module, and a complex dependency relationship between the modes is dynamically modeled through a multi-head self-attention mechanism, so that more accurate emotion recognition is realized, and the emotion recognition efficiency is improved. And finally, performing emotion prediction on the fused features based on time sequence modeling and an emotion classification module. According to the method, the problems of modal isomerism, difficulty in time alignment and insufficient dynamic emotion modeling in multi-modal emotion recognition can be effectively solved.

Description

Technical Field

[0001] The present invention relates to a multimodal emotion recognition method based on cross-modal Transformer fusion, belonging to the technical fields of artificial intelligence, emotion recognition and human-computer interaction. Background Art

[0002] With the rapid development of artificial intelligence and deep learning technologies, multimodal emotion recognition (MER) has become an important research direction in the fields of human-computer interaction and affective computing. Emotion recognition aims to identify a user's emotional state by analyzing input data from multiple modalities, such as facial expressions, speech, and text. However, existing emotion recognition methods primarily focus on single-modality processing, relying solely on a single information source such as vision, speech, or text. This approach often fails to fully exploit the complementary nature of multimodal data, resulting in limitations in the accuracy and robustness of emotion recognition.

[0003] With the widespread application of multimodal data and the advancement of deep learning technology, researchers have gradually begun to adopt multimodal emotion recognition methods to improve the performance of emotion recognition by fusing information from multiple modalities. However, this process faces a series of challenges. First, the heterogeneity between modalities (such as differences in the dimensions and temporal characteristics of visual, speech, and text features) may lead to information loss or inconsistency when directly fused. Secondly, how to effectively align the temporal information of different modalities to ensure the synchronization of emotional information at different time steps is an urgent problem to be solved. In addition, the emotional information of different modalities changes dynamically in time. How to model these dynamic changes and adaptively adjust the contribution of each modality remains a difficulty in the field of emotion recognition.

[0004] To address these issues, researchers have proposed a variety of cross-modal fusion strategies. Among them, the Transformer-based cross-modal fusion method has shown great potential in capturing complex relationships and dynamic temporal information between modalities due to its powerful self-attention mechanism. The Transformer's self-attention mechanism can dynamically adjust the information weights between multiple modalities, thereby avoiding information loss while maintaining efficient feature extraction. Despite this, existing multimodal emotion recognition methods still face many challenges, such as how to more effectively fuse information from different modalities, how to adaptively model complex dependencies between modalities, and how to reduce computational complexity in high-dimensional feature spaces.

[0005] Therefore, it is particularly important to propose a multimodal emotion recognition method based on cross-modal Transformer fusion. This paper aims to comprehensively solve the problems of modal heterogeneity, time alignment difficulties, and insufficient dynamic emotion modeling in multimodal emotion recognition by innovatively designing a cross-modal Transformer fusion module and combining multimodal feature extraction and time series modeling methods, thereby further improving the accuracy and robustness of emotion recognition. Summary of the Invention

[0006] Purpose of the Invention: To address the existing challenges of multimodal emotion recognition, such as modal heterogeneity, difficulty in temporal alignment, and insufficient dynamic emotion modeling, this paper proposes a multimodal emotion recognition method based on cross-modal Transformer fusion. This method effectively integrates multiple modal information, including visual, speech, and text, to improve the accuracy and robustness of emotion recognition, particularly for addressing emotional expressions in dynamic and complex environments.

[0007] Technical solution: The multimodal emotion recognition method based on cross-modal Transformer fusion described in the present invention includes the following steps:

[0008] Step 1: Standardize the multimodal data of vision, speech, and text to eliminate heterogeneity between modalities and ensure effective feature fusion. In the visual modality, facial regions are detected using MTCNN combined with OpenCV, and then cropped to a uniform size, normalized, and grayscaled. In the speech modality, high-frequency components are enhanced through pre-emphasis to extract fine-grained speech features. In the text modality, the WordPiece tokenizer is used for subword segmentation. The pre-trained BERT model is used to embed text, and the Transformer encoder is used to model text dependencies and extract deep semantic features.

[0009] Step 2: Use deep learning models to extract emotional features from each modality. The visual modality uses a combination of 3D-CNN and ConvLSTM to extract the spatiotemporal features of facial expressions. The speech modality uses a CNN combined with a BiLSTM network to extract the time-frequency features and temporal dependencies of speech. The text modality uses a combination of BERT and Transformer to extract deep semantic features of the text and capture contextual information.

[0010] Step 3: A cross-modal Transformer module is used to fuse modal features. The sentiment features extracted from visual, speech, and text modalities are projected onto multiple subspaces and the associations between the modalities are computed in parallel using a multi-head self-attention mechanism. This process linearly projects the features of each modality onto the same dimension and fuses the features of each modality to generate a comprehensive feature representation, thereby enhancing the ability to understand and capture sentiment features.

[0011] Step 4: The sentiment classification module uses the fused features to identify emotion. The fused features output by the cross-modal Transformer module are fed into the BiLSTM-Attention network for time series modeling to capture the dynamic evolution of emotion. The attention mechanism performs weighted fusion of the time series features to obtain the final sentiment feature representation. Finally, a fully connected layer and a softmax function are used to predict the sentiment category and output the probability of the sentiment category.

[0012] An electronic device comprises a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the method for cross-modal fusion multimodal emotion recognition based on Transformer is implemented.

[0013] A computer-readable storage medium having computer instructions stored thereon, characterized in that when the computer instructions are executed by a processor, the Transformer-based cross-modal fusion multimodal emotion recognition method is implemented.

[0014] Compared with the prior art, the present invention has the following advantages:

[0015] (1) Cross-modal fusion enhances emotion recognition capabilities: This paper effectively addresses the problem of modal heterogeneity by introducing a Transformer-based cross-modal fusion module. Through the multi-head self-attention mechanism, it can dynamically model the complex dependencies between modalities, enhance the complementarity of information from different modalities, and thus significantly improve the accuracy and robustness of emotion recognition.

[0016] (2) Temporal dynamic modeling improves emotion recognition: This paper uses a BiLSTM-Attention network to perform temporal modeling of multimodal features, fully capturing the dynamic evolution of emotions and resolving the problem that traditional methods fail to fully model the temporal dependence of emotions. This temporal modeling capability effectively improves the ability to recognize emotion changes over long periods of time, further enhancing the real-time and adaptability of the system.

[0017] (3) Flexibility and efficiency of multimodal information fusion: Compared with traditional single-modality methods, the present invention integrates information from visual, speech, and text modalities simultaneously, and comprehensively utilizes multiple emotional features such as facial expressions, voice pitch, speaking speed, and text semantics, thereby greatly improving the comprehensiveness and accuracy of emotion recognition, especially in complex and changing environments. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 A framework diagram for implementing a multimodal emotion recognition method based on cross-modal Transformer fusion.

[0019] Figure 2 This is the network structure diagram of the visual modality feature extraction of the present invention.

[0020] Figure 3 This is a network structure diagram for speech modal feature extraction in the present invention.

[0021] Figure 4 This is the network structure diagram of the text modality feature extraction in the present invention.

[0022] Figure 5 This is a diagram of the multimodal feature fusion network structure of the present invention. DETAILED DESCRIPTION

[0023] In order to deepen the knowledge and understanding of the present invention, the present invention is further illustrated below with reference to specific embodiments.

[0024] Figure 1 The overall implementation framework of a multimodal emotion recognition method based on cross-modal Transformer fusion proposed in the present invention is demonstrated. The framework diagram clearly presents the main process of the method, including four core steps: data preprocessing, feature extraction, cross-modal fusion and emotion classification. First, the three modal data of vision, speech and text are preprocessed respectively, and the data format is standardized through tools such as MTCNN, COVAREP and BERT. Subsequently, each modality extracts emotional features through an independent deep learning model (step 2), such as 3D-CNN and ConvLSTM for visual modality, CNN and BiLSTM for speech modality, and BERT and Transformer encoder for text modality. Then, the three modal features are fused using a cross-modal Transformer module, and the complex dependencies between modalities are modeled through a multi-head self-attention mechanism to generate a comprehensive feature representation. Finally, the fused features are subjected to temporal modeling and emotion classification through the BiLSTM-Attention network and the fully connected layer, and the emotion category probability is output. Figure 1 It intuitively demonstrates how the present invention improves the accuracy and robustness of multimodal emotion recognition through cross-modal fusion.

[0025] Figure 2The network structure of the present invention for visual modality feature extraction is shown. The figure describes in detail the process of visual modality extraction of emotional features from video data. The network structure is mainly composed of 3D convolutional neural network (3D-CNN) and convolutional long short-term memory network (ConvLSTM). First, the video clip is preprocessed, the face area is detected by MTCNN and OpenCV, and it is cropped and normalized to a uniform size. Then, the 3D-CNN module extracts spatiotemporal features from the video clip, captures the joint features of space and time through three-dimensional convolution kernels, and reduces the dimension using maximum pooling operation. Subsequently, the ConvLSTM module further models the serialized features and captures the temporal dependencies between frames through a convolution-based gating mechanism (input gate, forget gate, output gate). Finally, the feature sequence generates a feature representation of the visual modality through global average pooling. Figure 2 It clearly demonstrates how to effectively extract the spatiotemporal variation features of facial expressions and provide high-quality visual input for cross-modal fusion.

[0026] Figure 3 The network structure of the present invention for speech modality feature extraction is shown. The figure details the process of extracting emotional features from speech signals. The network structure mainly includes two modules: a CNN and a bidirectional long short-term memory network (BiLSTM). First, the speech signal is preprocessed, and low-level features are extracted through pre-emphasis and COVAREP. Phoneme-level alignment is performed using P2FA to generate a Mel spectrum feature map. Subsequently, the CNN module uses multiple two-dimensional convolutional layers (Conv2D) to extract local time-frequency features from the Mel spectrum, using 3×3 convolution kernels to capture patterns on the time and frequency axes. Batch Normalization, ReLU activation, maximum pooling, and Dropout operations are combined to stabilize training and prevent overfitting. Next, the feature sequence output by the CNN is compressed in dimension through global average pooling and input into the BiLSTM module for time series modeling to capture the temporal dependencies of the speech signal. Finally, the temporal features output by the BiLSTM are weighted and fused through the attention mechanism to generate the final feature representation of the speech modality. Figure 3 It intuitively demonstrates how to extract high-quality emotional features from speech signals to provide support for subsequent multimodal fusion.

[0027] Figure 4The network structure of the present invention for text modal feature extraction is shown. The figure describes in detail the process of extracting sentiment features from text data. The network structure mainly consists of a BERT pre-trained model and a Transformer encoder. First, the text data is pre-processed, and sub-word segmentation is performed by the WordPiece word segmenter to remove meaningless words, and the text is converted into an embedding vector of fixed dimension using the BERT pre-trained model. Next, BERT learns the contextual information of the word through a bidirectional Transformer structure, generates a hidden layer feature vector, and extracts a global feature representation of the classification tag position. Subsequently, the Transformer encoder further processes the features output by BERT, captures the semantic dependencies between words through a multi-head self-attention mechanism, and enhances the globality of the text representation. Finally, the features output by the Transformer encoder are subjected to global average pooling and dimensionality reduction to generate the final feature representation of the text modality. Figure 4 It clearly demonstrates how to effectively extract deep semantic information from text and provide reliable text feature input for cross-modal fusion.

[0028] Figure 5 The network structure of the present invention for multimodal feature fusion is shown. The figure describes in detail the process of fusing visual, speech and text modal features based on the cross-modal Transformer module. The network structure is centered on the cross-modal Transformer. First, the features of the three modalities are unified in time scale through feature alignment operations (such as interpolation completion and zero padding) to generate an aligned feature sequence. Subsequently, the features of each modality are mapped to the same dimension through linear projection and spliced into a unified input feature. The cross-modal Transformer module calculates the correlation between modalities in parallel through a multi-head self-attention mechanism (Multi-Head Self-Attention, MHA), and dynamically models the complex dependencies between different modal features. During the fusion process, the outputs of the attention heads are spliced and mapped to the final dimension, and a comprehensive feature representation is generated by combining residual connections and layer normalization. Figure 5 It intuitively demonstrates how the cross-modal Transformer enhances the ability to understand and capture emotional features through deep fusion, providing high-quality input for subsequent emotion classification.

[0029] Example 1: A multimodal emotion recognition method based on cross-modal Transformer fusion is proposed. The method first collects and preprocesses visual, speech and text multimodal data; then, a deep learning model is used to extract emotion features for each modality to provide effective input features for cross-modal fusion. Then, the cross-modal Transformer module is used to fuse the features between modalities, and the complex dependencies between different modalities are dynamically modeled through the multi-head self-attention mechanism to improve the accuracy and robustness of emotion recognition. Finally, in order to enhance the temporal modeling capability of the emotion recognition model, the BiLSTM-Attention network is used to perform time series modeling on the fused multimodal features, further improving the dynamic evolution feature recognition of emotions, and outputting the final emotion prediction results through the emotion classification module. For specific models, see Figures 1 to 5 The specific implementation steps are as follows:

[0030] Step 1: Preprocess the visual, speech, and text multimodal data.

[0031] Sub-step 1-1: Visual modality standardization: parse the video clips frame by frame, use the MTCNN model combined with the OpenCV cascade classifier to detect the face area, and crop the detected face area to a uniform size, normalize it, and grayscale it.

[0032] Sub-steps 1-2 normalize the speech modality, pre-emphasize the speech signal, extract low-level speech features using the COVAREP tool library, use P2FA for phoneme-level alignment, and further extract speech features using Mel-frequency cepstral coefficients (MFCC).

[0033] In sub-steps 1-3, the text modality is normalized. The WordPiece word segmenter is used to segment the text data at the subword level, meaningless words with low semantic contribution are removed. The BERT pre-trained model is used to embed the text data, and the text context dependency is modeled through the Transformer encoder.

[0034] Step 2: Extract sentiment features from each modality through a deep learning model. Use independent deep learning models to extract sentiment features from the visual, speech, and text modalities respectively.

[0035] Sub-step 2-1 Visual modality feature extraction uses a combination of 3D convolutional neural network (3D-CNN) and convolutional long short-term memory network (ConvLSTM) to extract and model the spatiotemporal variation characteristics of facial expressions. The video data segment input to the 3D-CNN module can be represented as a four-dimensional tensor X∈R C×T×H×W, C is the number of channels, T is the number of frames, i.e. the time dimension, H and W are the height and width of the frame. The joint features of space and time are captured by 3D convolution operation, and the feature extraction process can be expressed as:

[0036]

[0037] is the output value of the kth convolution kernel at time t, height h, and width w, ReLU is the activation function, is the weight parameter of the kth convolution kernel on the cth input channel, with a size of D×H×W (3D kernel), X c,t+t′,h+h′,w+w′ is the value of the cth input channel of the original input feature map at the offset position (t+t',h+h',w+w'), the bth (k) is the bias term. To capture local features in the spatiotemporal dimension, the input sequence is continuously cropped from a sequence of consecutive video frames. Each 3D convolution kernel slides simultaneously in both spatial and temporal dimensions, enhancing the spatiotemporal correlation between feature maps. To reduce the spatial dimensionality of the feature maps and alleviate the computational complexity of the subsequent ConvLSTM module, a max pooling operation is also performed on the feature maps, thereby improving model efficiency.

[0038]

[0039] is the output value of the kth convolution kernel at time t, height h, and width w, ReLU is the activation function, is the weight parameter of the kth convolution kernel on the cth input channel, with a size of D×H×W (3D kernel), X c,t+t′,h+h′,w+w′ is the value of the cth input channel of the original input feature map at the offset position (t+t',h+h',w+w'), the bth (k) is the bias term. The ConvLSTM module further models the serialized input features to establish the dynamic change pattern between video time series features. ConvLSTM uses a gating mechanism based on convolution operations, which enables it to not only accurately grasp the spatial relationship within the same frame, but also effectively capture the temporal dependencies between different frames. The specific calculation formula of the ConvLSTM unit is as follows:

[0040]

[0041] The input gate, forget gate, and output gate are respectively t 、f t 、o t Indicates, σ is the Sigmoid function, tanh is the hyperbolic tangent function, * indicates the convolution operation, represents element-wise multiplication, X t is the current input feature map, Ht-1 、C t-1 is the hidden state and memory unit of the previous moment, H t 、C t is the hidden state and memory unit at the current moment, W xi 、W xo 、W xc 、W hc is the convolution weight, W hi 、W hf 、W hc 、W ho is the weight matrix W from the hidden state to the four gates ci 、W cf 、W co is the weight of cell state (memory) to gate, b i 、b f 、b c 、b o is the bias vector of each gate. The ConvLSTM output feature sequence is subjected to global average pooling to obtain the final visual modality feature representation:

[0042]

[0043] To indicate that the dimension of the output V is a T×d v The matrix, through the above visual modality feature extraction, effectively captures the spatial and temporal variation characteristics of facial expressions, and provides reliable visual feature input for subsequent cross-modal fusion.

[0044] Sub-step 2-2 Speech modal feature extraction, combining CNN to extract local time-frequency features, and using bidirectional long short-term memory network (BiLSTM) to capture the temporal dependency of speech signals, thereby generating high-quality speech emotion features. The CNN module is used to extract local time-frequency features from the Mel spectrum. This module contains multiple two-dimensional convolutional layers (Conv2D), which use 3×3 convolution kernels to slide on the time and frequency axes to learn the local patterns of speech signals. Each convolution layer is followed by Batch Normalization for normalization to accelerate convergence and stabilize training. At the same time, nonlinear characteristics are introduced through the ReLU activation function to enhance feature expression capabilities. In addition, in order to reduce computational complexity and avoid overfitting, MaxPooling and Dropout layers are added after each convolution layer to compress the feature map and randomly discard some neurons to improve the generalization ability of the model. The Mel spectrum feature map obtained after preprocessing is X∈R T×F, where T represents the uniform time step length and F represents the number of Mel frequency bands. The feature map X is used as the input of the CNN module. The CNN model is used to extract local features from the Mel frequency spectrum:

[0045]

[0046] in, is the value at position (i, j) of the feature map after the l-th layer convolution, is the weight in the convolution kernel of the lth layer, b (l) is the bias term, and M×N is the convolution kernel size. To further reduce the feature dimension, reduce the complexity of model training, and prevent overfitting problems, a maximum pooling layer is used after each convolution layer to perform feature compression:

[0047]

[0048] Ω i,j is the (i,j)th pooling area, P i,j is the value of the output feature map at position (i, j) after pooling. The feature vector output by CNN is recorded as H t,f,c , whose dimensions are (T′, F′, C′), where T′, F′, and C′ are the processed time series length, spectral dimension, and number of feature channels, respectively. The feature sequence output by CNN is represented more compactly through global average pooling:

[0049]

[0050] Among them, H t,f,c is the value of the tth time sequence, the fth frequency band, and the cth channel in the CNN feature map, H t is the feature vector after global average pooling of the t-th time series. The processed CNN output sequence features {H1,H2,…,H T′ As the input of the subsequent BiLSTM module, the time series modeling of the speech signal is further performed to capture the temporal dependencies of the speech signal:

[0051]

[0052] is the global pooling feature extracted by CNN at the t-th time step, is the hidden state of the BiLSTM output at the tth time step, representing the attention representation after time series modeling. The final feature representation of the speech modality is obtained by weighted fusion of the time series features output by BiLSTM through the attention mechanism:

[0053]

[0054] A is the final feature, α tis the attention weight of the t-th time step, usually calculated by the Softmax layer, Dimensions T×d a A two-dimensional tensor of , T is the uniform time step length, d a is the dimension of the feature at each time step. Through the above speech modality feature extraction process, the local time-frequency features and global temporal dependencies in the speech signal are effectively captured, providing high-quality speech feature input for multimodal emotion recognition.

[0055] Sub-step 2-3 text modality feature extraction adopts a combined structure based on the BERT pre-trained model and the Transformer encoder to effectively capture the deep features and contextual dependencies of the text semantic information. Transformer calculates the dependencies between different words through the self-attention mechanism (Self-Attention), thereby enhancing the globality of text representation. The text data is preprocessed, including tokenization, embedding, and sequence padding. This paper uses WordPiece Tokenizer for subword level segmentation to improve the model's generalization ability for unregistered words (Out-of-Vocabulary, OOV), and uses the BERT pre-trained model to convert the text into a fixed-dimensional embedding vector. Assuming the length of the input text sequence is N, the input sequence can be expressed as:

[0056]

[0057] The word embedding dimension is d embed In the feature extraction stage, this paper uses the BERT pre-trained model as a text encoder to obtain deep semantic features. BERT learns the context information of words through a bidirectional Transformer structure and generates hidden layer feature vectors.

[0058]

[0059] On this basis, the global feature representation h of the classification mark position is extracted cls , as the overall semantic information of the sentence, for further processing by the subsequent Transformer encoder. In order to enhance the temporal modeling capability of sentiment features, this paper uses the Transformer encoder to further learn the contextual dependencies of the text based on the BERT semantic features.

[0060]

[0061] The Transformer encoder effectively captures the semantic dependencies between words in the text through multiple parallel attention heads. The feature H output by the Transformer encoder trans Then, the global average pooling is used to reduce the dimension to the final text modality feature representation:

[0062]

[0063] The above text modal feature extraction effectively captures the key semantic information in the text sentiment recognition task, providing reliable text feature input for cross-modal fusion.

[0064] Step 3: Based on the Transformer-based cross-modal fusion framework, deep fusion is achieved through the multi-head self-attention mechanism (MHA) and feedforward neural network. The modal features are projected into multiple subspaces, and the correlation between modalities is calculated in parallel through multi-head attention. The output of the attention heads is fused to generate a comprehensive feature representation to improve the ability to understand and capture emotional features. The visual modal features are denoted as Speech modal features are recorded as Text modality features are denoted as Where T represents the uniform time step length, d v d a d t are the dimensions of visual, speech, and text features, respectively. Since the three modalities have different time scales, a feature alignment method is used to ensure temporal synchronization during cross-modal fusion:

[0065] X′ m =Align(X m ),X m ∈V,A,T x (14)

[0066] Among them, Align(·) is the alignment operation, which specifically includes interpolation and zero padding to make the data of different modes consistent in the time dimension, forming the aligned feature sequences V′, A′, T x ′∈R T′×d To achieve unified input for the cross-modal Transformer, linear projection and concatenation of modal features are required. First, the aligned modal features must be mapped to the same dimension d:

[0067]

[0068] W v 、W a 、W t is the linear projection matrix, b v 、ba 、b t is the corresponding bias term. Then the mapped features are concatenated:

[0069]

[0070] The cross-modal Transformer fusion module is introduced to realize the information interaction between different modal features through the multi-head self-attention mechanism. The multi-head self-attention mechanism calculates the definition query (Query, Q = X fusion W Q ), key (Key, K = X fusion W K ) and value (Value, V = X fusion W V ) matrix, the attention of each head is calculated as head i =Attention(QWi i Q ,KWi i K ,VW i V ). The attention mechanism is defined as:

[0071]

[0072] In the formula, d k is the dimension of the key matrix. Concatenate multiple head outputs and map them to the final dimension MultiHead(Q,K,V)=Concat(head1,…,head h )W O ,Among them, Wi i Q 、W i K 、W i V 、W O are all trainable weight parameter matrices, and h is the number of attention heads. To further stabilize the training process, the Transformer fusion module also includes residual connections and layer normalization to obtain comprehensive features:

[0073] X trans =LayerNorm(X fusion +MultiHead(Q,K,V)) (18)

[0074] Step 4: Use the emotion classification module to perform emotion recognition on the fused features. Based on the multimodal features obtained in step 3, the output feature X transBring in the BiLSTM-Attention network for time series modeling.

[0075]

[0076] And use the Attention mechanism to perform weighted fusion of temporal features.

[0077]

[0078] Finally, the emotion category probability prediction is achieved through the full connection layer and Softmax.

[0079] Y=Softmax(W fc H+b fc ) (twenty one)

[0080] Y is the final sentiment prediction result, W fc 、b fc are the learnable classification layer parameters, representing the fully connected layer weight matrix and the fully connected layer bias term.

[0081] Based on the same inventive concept, the present invention describes a method for multimodal emotion recognition based on Transformer cross-modal fusion, which includes a memory, a processor, and a computer program stored in the memory and runnable on the processor. When the computer program is loaded into the processor, it implements the above-mentioned cross-modal fusion module, time series modeling module, and emotion classification module for multimodal emotion recognition.

[0082] Those skilled in the art will appreciate that the embodiments described herein are intended to help readers understand the principles of the present invention. It should be understood that the embodiments are only used to illustrate the present invention and are not intended to limit the scope of the present invention. After reading the present invention, modifications to various equivalent forms of the present invention made by those skilled in the art fall within the scope defined by the claims appended to this application.

Claims

1. A Transformer-based cross-modal fusion multimodal emotion recognition method, characterized by: The steps include: Step 1: Preprocess visual, speech, and text multimodal data; Step 2: Extract emotional features from each modality through a deep learning model; Step 3: Use the cross-modal Transformer module to fuse modal features; Step 4: Use the sentiment classification module to perform sentiment recognition on the fused features.

2. The method for cross-modal fusion multimodal emotion recognition based on Transformer according to claim 1 is characterized in that: Step 1: Preprocess the visual, speech, and text multimodal data as follows: Sub-step 1-1: Visual modality standardization: parse the video clips frame by frame, use the MTCNN model combined with the OpenCV cascade classifier to detect the face area, and crop the detected face area to a uniform size, normalize and grayscale it. Sub-steps 1-2 normalize the speech modality, pre-emphasize the speech signal, extract low-level speech features using the COVAREP tool library, use P2FA for phoneme-level alignment, and further extract speech features using Mel-Frequency Cepstral Coefficients (MFCC). In sub-steps 1-3, the text modality is normalized. The WordPiece word segmenter is used to segment the text data at the subword level, meaningless words with low semantic contribution are removed. The BERT pre-trained model is used to embed the text data, and the text context dependency is modeled through the Transformer encoder.

3. The method for cross-modal fusion multimodal emotion recognition based on Transformer according to claim 1, characterized in that: Step 2: Extract emotional features from each modality through a deep learning model. Extract emotional features from vision, speech, and text modalities through independent deep learning models. Specifically, it includes the following sub-steps: Sub-step 2-1: Visual modality feature extraction. Use a combination of 3D convolutional neural network (3D-CNN) and convolutional long short-term memory network (ConvLSTM) to extract and model the spatiotemporal variation characteristics of facial expressions. The video data segment input to the 3D-CNN module can be represented as a four-dimensional tensor X∈R C×T×H×W , C is the number of channels, T is the number of frames, that is, the time dimension, H and W are the height and width of the frame, and the joint features of space and time are captured through 3D convolution operation. The feature extraction process is expressed as: is the output value of the kth convolution kernel at time t, height h, and width w, ReLU is the activation function, is the weight parameter of the kth convolution kernel on the cth input channel, with a size of D×H×W (3D kernel), X c,t+t′,h+h′,w+w′ is the value of the cth input channel of the original input feature map at the offset position (t+t',h+h',w+w'), the bth (k) The bias term is used. In order to capture local features in the spatiotemporal dimension, the input sequence is continuously cropped from the continuous frame sequence of the video. Each 3D convolution kernel can slide in both the spatial and temporal dimensions to improve the spatiotemporal correlation between feature maps. In order to reduce the spatial dimension of the feature map and reduce the computational complexity of the subsequent ConvLSTM module, the feature map is subjected to the maximum pooling operation, thereby improving the efficiency of the model. is the value of the feature map at position (i, j) after the k-th layer pooling, Represents the local area (pooling window) Ω i,j The value of all positions (m,n) in takes the maximum value, is the value of the input feature map at position (m,n) in the kth convolution channel, Ω i,j is the set of positions covered by the i,jth pooling region (i.e., sliding window). The ConvLSTM module further models the serialized input features to establish the dynamic change pattern between the video time series features. The ConvLSTM uses a gating mechanism based on convolution operations, which enables it to not only accurately grasp the spatial relationship within the same frame, but also effectively capture the temporal dependency between different frames. The specific calculation formula of the ConvLSTM unit is as follows: The input gate, forget gate, and output gate are respectively t 、f t 、o t Indicates, σ is the Sigmoid function, tanh is the hyperbolic tangent function, * indicates the convolution operation, ° indicates element-by-element multiplication, X t is the current input feature map, H t-1 、C t-1 is the hidden state and memory unit of the previous moment, H t 、C t is the hidden state and memory unit at the current moment, W xi 、W xo 、W xc 、W hc is the convolution weight, W hi 、W hf 、W hc 、W ho is the weight matrix W from the hidden state to the four gates ci 、W cf 、W co is the weight of cell state (memory) to gate, b i 、b f 、b c 、b o is the bias vector of each gate, and the ConvLSTM output feature sequence is subjected to global average pooling (GAP) to obtain the final visual modality feature representation: To indicate that the dimension of the output V is a T×d v The matrix of , through the above visual modality feature extraction, effectively captures the spatial and temporal variation characteristics of facial expressions, and provides reliable visual feature input for subsequent cross-modal fusion. Sub-step 2-2 speech modal feature extraction, combining CNN to extract local time-frequency features, and using bidirectional long short-term memory network (BiLSTM) to capture the time dependency of speech signals, thereby generating high-quality speech emotion features. The CNN module is used to extract local time-frequency features of the Mel spectrum. The module contains multiple two-dimensional convolutional layers (Conv2D), using 3×3 convolution kernels to slide on the time and frequency axes to learn the local pattern of the speech signal. Each convolution layer is followed by BatchNormalization for normalization to accelerate convergence and stabilize training. At the same time, nonlinear characteristics are introduced through the ReLU activation function to improve feature expression capabilities. In addition, in order to reduce computational complexity and avoid overfitting, the maximum pooling (MaxPooling) and Dropout layers are added after each convolution layer to compress the feature map and randomly discard some neurons to improve the generalization ability of the model. The Mel spectrum feature map obtained after preprocessing is X∈R T×F , where T represents the uniform time step length, F represents the number of Mel frequency bands, and the feature map X is used as the input of the CNN module. The CNN model is used to extract local features from the Mel spectrum: in, is the value at position (i, j) of the feature map after the l-th layer convolution, is the weight in the convolution kernel of the lth layer, b (l) is the bias term, M×N is the convolution kernel size, which further reduces the feature dimension, reduces the complexity of model training and prevents overfitting problems. Each convolution layer is followed by a maximum pooling layer for feature compression: Ω i,j is the (i,j)th pooling area, P i,j is the value of the output feature map at position (i, j) after pooling, and the feature vector output by CNN is recorded as H t,f,c , whose dimensions are (T′, F′, C′), where T′, F′, and C′ are the processed time series length, spectrum dimension, and number of feature channels, respectively. The feature sequence output by CNN is represented more compactly by global average pooling: Among them, H t,f,c is the value of the tth time sequence, the fth frequency band, and the cth channel in the CNN feature map, H t is the feature vector after global average pooling of the t-th time series, and the processed CNN output sequence features {H1,H2,…,H T′ As the input of the subsequent BiLSTM module, the time series modeling of the speech signal is further performed to capture the temporal dependencies of the speech signal: is the global pooling feature extracted by CNN at the t-th time step, is the hidden state of the BiLSTM output at the tth time step, representing the attention representation after time series modeling. The attention mechanism is used to weight the time series features output by the BiLSTM to obtain the final feature representation of the speech modality: A is the final feature, α t is the attention weight of the t-th time step, usually calculated by the Softmax layer, Dimensions T×d a A two-dimensional tensor of , T is the uniform time step length, d a It is the dimension of each time step feature. Through the above speech modal feature extraction process, the local time-frequency features and global temporal dependencies in the speech signal are effectively captured, providing high-quality speech feature input for multimodal emotion recognition. Sub-step 2-3: Text modality feature extraction. A combination of the BERT pre-trained model and the Transformer encoder is used to effectively capture the deep features and contextual dependencies of text semantic information. The Transformer calculates the dependencies between different words through the self-attention mechanism, thereby enhancing the globality of text representation. The text data is pre-processed, including tokenization, embedding, and sequence padding. The WordPiece Tokenizer is used for sub-word segmentation to improve the model's generalization ability for out-of-vocabulary (OOV). The BERT pre-trained model is used to convert the text into a fixed-dimensional embedding vector. Assuming the input text sequence length is N, the input sequence can be expressed as: The word embedding dimension is d embed In the feature extraction stage, the BERT pre-trained model is used as the text encoder to obtain deep semantic features. BERT learns the context information of words through the bidirectional Transformer structure and generates hidden layer feature vectors. On this basis, the global feature representation h of the classification mark position is extracted cls , as the overall semantic information of the sentence, for further processing by the subsequent Transformer encoder. In order to enhance the temporal modeling capability of sentiment features, this paper uses the Transformer encoder to further learn the contextual dependencies of the text based on the BERT semantic features. The Transformer encoder is applied to effectively capture the semantic dependencies between words in the text through multiple parallel attention heads. The feature H output by the Transformer encoder is trans Then, the global average pooling is used to reduce the dimension to the final text modality feature representation: The above text modal feature extraction effectively captures the key semantic information in the text sentiment recognition task, providing reliable text feature input for cross-modal fusion.

4. The method for cross-modal fusion multimodal emotion recognition based on Transformer according to claim 3 is characterized in that: Step 3 is as follows: establish a cross-modal fusion framework based on Transformer, achieve deep fusion through multi-head self-attention mechanism (MHA) and feedforward neural network, project modal features into multiple subspaces, calculate the correlation between modalities in parallel through multi-head attention, fuse the output of attention heads, generate comprehensive feature representation, improve the ability to understand and capture emotional features, and record visual modal features as Speech modal features are recorded as Text modality features are denoted as Where T represents the uniform time step length, d v d a d t are the dimensions of visual, speech, and text features, respectively. Since the three modalities have different time scales, a feature alignment method is used to ensure temporal synchronization during cross-modal fusion: X′ m =Align(X m ),X m ∈V,A,T x (14) Among them, Align(·) is the alignment operation, which specifically includes interpolation and zero padding to make the data of different modes consistent in the time dimension, forming the aligned feature sequences V′, A′, T x ′∈R T′×d To achieve unified input of the cross-modal Transformer, linear projection and concatenation of modal features are required. The aligned modal features must first be mapped to the same dimension d: W v 、W a 、W t is the linear projection matrix, b v 、b a 、b t is the corresponding bias term, and then the mapped features are spliced: The cross-modal Transformer fusion module is introduced to realize the information interaction between different modal features through the multi-head self-attention mechanism. The multi-head self-attention mechanism calculation first defines the query (Query, Q = X fusion W Q ), key (Key, K = X fusion W K ) and value (Value, V = X fusion W V ) matrix, the attention of each head is calculated as The attention mechanism is defined as: In the formula, d k The dimension of the key matrix is used to concatenate multiple head outputs and map them to the final dimension MultiHead(Q,K,V)=Concat(head1,…,head h )W O , where W i Q 、 W i V 、W O are all trainable weight parameter matrices, h is the number of attention heads, and to further stabilize the training process, the Transformer fusion module also includes residual connections and layer normalization to obtain comprehensive features: X trans =LayerNorm(X fusion +MultiHead(Q,K,V)) (18)。 5. The method for cross-modal fusion multimodal emotion recognition based on Transformer according to claim 1, characterized in that: Step 4 uses the sentiment classification module to perform sentiment recognition on the fused features. Based on the multimodal features obtained in step 3, the output feature X trans Bring in the BiLSTM-Attention network for time series modeling, And use the Attention mechanism to perform weighted fusion of temporal features. Finally, the emotion category probability prediction is achieved through the full connection layer and Softmax. Y=Softmax(W fc H+b fc ) (21)Y is the final emotion prediction result, W fc 、b fc are the learnable classification layer parameters, representing the fully connected layer weight matrix and the fully connected layer bias term.

6. The multimodal emotion recognition device according to the multimodal emotion recognition method according to claims 1-5, characterized in that: The device includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the computer program is loaded into the processor, the computer program implements the cross-modal fusion module, the temporal modeling module, and the emotion classification module for multimodal emotion recognition.

7. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, it implements the Transformer-based cross-modal fusion multimodal emotion recognition method as described in any one of claims 1 to 5 above.

8. A computer-readable storage medium having computer instructions stored thereon, characterized in that: When the computer instruction is executed by the processor, the Transformer-based cross-modal fusion multimodal emotion recognition method as described in any one of claims 1 to 5 is implemented.

Images