An emotion recognition method and device

Abstract

This application proposes an emotion recognition method and device. First, time-aligned multimodal raw data is obtained. Then, a trained emotion recognition model is used to process the multimodal raw data to obtain the emotion recognition result. The emotion recognition model includes a multimodal feature extraction module, a multi-layer attention fusion module, and a classification recognition module, which respectively perform modal feature extraction, attention mechanism fusion, and classification recognition. In this way, multimodal features can be fully expressed and fused during the emotion recognition process, thereby achieving high-accuracy emotion recognition.

Description

Technical Field

[0001] This application relates to the field of data processing technology, and in particular to an emotion recognition method and device. Background Technology

[0002] Emotion recognition is a key technology in human-computer interaction, intelligent monitoring, and public opinion analysis. Traditional emotion recognition technologies mainly rely on single-modal data, such as sentiment analysis based solely on text, speech, or video images. With the development of social media and smart terminals, multimodal emotion recognition (MER) has gradually become a research hotspot. Existing multimodal emotion recognition technologies typically collect users' text, audio, and video data, extract features from each modality using deep learning networks, then concatenate or perform simple weighted fusion of these features, and finally input them into a classifier to obtain the emotional state, such as positive, negative, or neutral.

[0003] The existing technology has at least the following problems: (1) Insufficient expression of single-modal features: Existing text feature extraction mostly uses static word vectors, such as GloVe and Word2Vec, which cannot capture contextual semantic information; existing video modal feature extraction is mostly limited to facial expressions or general image features, ignoring auxiliary features such as body movements and speaker attributes, such as gender and age, which have an important impact on emotional expression, resulting in insufficient visual representation ability.

[0004] (2) Simple intermodal fusion mechanism: Existing technologies mostly adopt simple feature splicing or early fusion strategies, without fully considering the heterogeneity and complementarity between different modal data. Simple splicing cannot capture the deep interaction relationship between modalities, making it difficult for the model to distinguish which modal information plays a dominant role in the current sentiment judgment.

[0005] (3) Lack of fine-grained attention allocation: In the process of fusion, existing methods often assign the same weight to all modalities or all time steps, lacking an attention mechanism for key emotional information, which leads to noise data interfering with the emotional classification results and reducing the accuracy and robustness of recognition. Summary of the Invention

[0006] This application proposes an emotion recognition method and device that can solve one of the problems existing in the background art.

[0007] To achieve the above objectives, this application adopts the following technical solution: Firstly, an emotion recognition method is provided, including: Obtain time-aligned raw multimodal data to be processed, which includes: raw text data, raw audio data, and raw video data; The trained emotion recognition model is used to process the original multimodal data to obtain the emotion recognition results. The emotion recognition model includes: The multimodal feature extraction module is used to extract features from the original data of each modality to obtain the extracted features for each modality. The multi-layer attention fusion module is used to achieve deep fusion of features extracted from various modalities through an attention mechanism to obtain a fused feature vector; In addition, there is a classification and recognition module, which is used to classify emotions based on fused feature vectors and output the emotion recognition results.

[0008] In one possible design of the first aspect, the multimodal feature extraction module includes: a text feature extraction module, an audio feature extraction module, and a video feature extraction module. The video feature extraction module includes: a first convolutional neural network for extracting a basic visual feature matrix from the original video data; a multimedia processing pipeline for calling a pose estimation subunit and a face feature analysis subunit to extract pose features and face features from the original video data, respectively; a linear projection layer for dimensionally aligning the pose features and face features; a fusion layer for fusing the dimensionally aligned pose features and face features to obtain an auxiliary attribute feature matrix; a feature concatenation unit for concatenating the basic visual feature matrix and the auxiliary attribute feature matrix to obtain an enhanced multi-source feature sequence; a first bidirectional long short-term memory network for capturing a bidirectional contextual hidden state sequence from the enhanced multi-source feature sequence; a self-attention mechanism unit for weighted aggregation of the bidirectional contextual hidden state sequence using a self-attention mechanism to obtain a weighted aggregated sequence; and a video feature output unit for obtaining a video modal feature vector from the weighted aggregation sequence.

[0009] In one possible design of the first aspect, the text feature extraction module includes a cascaded language model and a second bidirectional long short-term memory network, and the audio feature extraction module includes a cascaded second convolutional neural network and a third bidirectional long short-term memory network.

[0010] In one possible design of the first aspect, the multi-layer attention fusion module includes: an intra-modal self-attention layer for processing features extracted from each modality through a self-attention mechanism to obtain updated modal features; a cross-modal attention interaction layer for capturing inter-modal correlations of the updated modal features through an attention matrix to obtain cross-modal fusion features; and a soft attention weighted fusion layer for weighted fusion of the cross-modal fusion features based on importance scores of the cross-modal fusion features to obtain the fusion feature vector.

[0011] In one possible design of the first aspect, the intramodal self-attention layer is specifically used to: map the extracted features of each modality to a query matrix, a key matrix, and a value matrix; obtain an attention score matrix from the query matrix and the key matrix; normalize the attention score matrix to obtain an attention weight matrix; and multiply the attention weight matrix with the value matrix to obtain the updated modal features after self-attention weighting.

[0012] In one possible design approach of the first aspect, the cross-modal attention interaction layer is specifically used for: constructing query-key / value cross-modal attention pairs based on updated modal features of different modalities; for each cross-modal attention pair, projecting the updated modal features onto a unified attention space, obtaining cross-modal attention scores from the query modality and key modality, normalizing the cross-modal attention scores to obtain cross-modal attention weights, multiplying the cross-modal attention weights by the value modality to obtain cross-modal context features after cross-modal attention weighting; fusing the cross-modal context features with the updated modal features to obtain intermediate features; and performing a nonlinear transformation on the intermediate features to obtain the cross-modal fused features.

[0013] In one possible design approach of the first aspect, the soft attention-weighted fusion layer is specifically used for: scoring the importance of the cross-modal fusion features to obtain the original importance scores of each modality; normalizing the original importance scores of each modality to obtain the attention weight coefficients of each modality; fusing the cross-modal fusion features using the attention weight coefficients based on a gating mechanism to obtain global fusion features; and performing a nonlinear transformation on the global fusion features to obtain the fusion feature vector.

[0014] In one possible design of the first aspect, the classification and recognition module includes a cascaded fully connected layer and a Softmax classifier.

[0015] In one possible design of the first aspect, the emotion recognition method further includes: acquiring the raw multimodal data to be processed from the same video file.

[0016] In a second aspect, an electronic device is provided, comprising: a processor, and a memory coupled to the processor, the memory for storing a computer program; the processor for executing the computer program stored in the memory to cause the electronic device to perform the emotion recognition method as described in any possible implementation of the first aspect.

[0017] Beneficial effects: Based on the above technical solution, firstly, time-aligned multimodal raw data is obtained. Then, a trained emotion recognition model is used to process the multimodal raw data to obtain the emotion recognition result. The emotion recognition model includes a multimodal feature extraction module, a multi-layer attention fusion module, and a classification recognition module, which respectively perform modal feature extraction, attention mechanism fusion, and classification recognition. In this way, multimodal features can be fully expressed and fused during the emotion recognition process, thereby achieving high-accuracy emotion recognition. Attached Figure Description

[0018] To more clearly illustrate the technical solutions in the embodiments of this application, the drawings used in the description of the embodiments or related technologies will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0019] Figure 1 This is a flowchart of the emotion recognition method provided in Embodiment 1 of this application; Figure 2 This is a structural diagram of the emotion recognition model provided in Embodiment 1 of this application; Figure 3 This is an overall flowchart of the multimodal emotion recognition method provided in Embodiment 2 of this application; Figure 4 This is a structural diagram of the audio feature extraction module provided in Embodiment 2 of this application; Figure 5 This is a structural diagram of the video feature extraction module provided in Embodiment 2 of this application; Figure 6 This is a structural diagram of the attention fusion module provided in Embodiment 2 of this application. Detailed Implementation

[0020] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0021] It should be noted that although functional modules are divided in the device schematic diagram and the logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the device or the order in the flowchart. The terms "first," "second," etc., in the specification and the above-mentioned figures are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence.

[0022] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.

[0023] Example 1 like Figure 1 As shown, this embodiment provides an emotion recognition method, including: S101, Obtain time-aligned multimodal raw data to be processed, the multimodal raw data to be processed includes: text raw data, audio raw data and video raw data; S102, using the trained emotion recognition model, the original multimodal data to be processed is processed to obtain the emotion recognition result.

[0024] like Figure 2 As shown, the emotion recognition model includes: The multimodal feature extraction module 201 is used to extract features from the original data of each modality to obtain the extracted features of each modality; The multi-layer attention fusion module 202 is used to achieve deep fusion of features extracted from various modalities through an attention mechanism to obtain a fused feature vector; And, a classification and recognition module 203, used to perform emotion classification based on fused feature vectors and output emotion recognition results.

[0025] Based on the above technical solution, firstly, time-aligned multimodal raw data is obtained. Then, a trained emotion recognition model is used to process the multimodal raw data to obtain the emotion recognition result. The emotion recognition model includes a multimodal feature extraction module, a multi-layer attention fusion module, and a classification recognition module, which respectively perform modal feature extraction, attention mechanism fusion, and classification recognition. In this way, multimodal features can be fully expressed and fused during the emotion recognition process, thereby achieving high-accuracy emotion recognition.

[0026] In one possible implementation, the multimodal feature extraction module includes: a text feature extraction module, an audio feature extraction module, and a video feature extraction module. The video feature extraction module includes: a first convolutional neural network (CNN) for extracting a basic visual feature matrix from the original video data; a multimedia processing pipeline (MediaPipe) for calling a pose estimation subunit and a face feature analysis subunit to extract pose features and face features from the original video data, respectively; a linear projection layer for dimensionally aligning the pose features and face features; a fusion layer for fusing the dimensionally aligned pose features and face features to obtain an auxiliary attribute feature matrix; a feature concatenation unit for concatenating the basic visual feature matrix and the auxiliary attribute feature matrix to obtain an enhanced multi-source feature sequence; a first bidirectional long short-term memory network (Bi-LSTM) for capturing a bidirectional contextual hidden state sequence from the enhanced multi-source feature sequence; a self-attention mechanism unit for weighted aggregation of the bidirectional contextual hidden state sequence using a self-attention mechanism to obtain a weighted aggregated sequence; and a video feature output unit for obtaining a video modal feature vector from the weighted aggregation sequence.

[0027] In one possible design of the first aspect, the text feature extraction module includes a cascaded language model (BERT) and a second bidirectional long short-term memory network, and the audio feature extraction module includes a cascaded second convolutional neural network and a third bidirectional long short-term memory network.

[0028] In one possible implementation, the multi-layer attention fusion module includes: an intra-modal self-attention layer for processing features extracted from each modality through a self-attention mechanism to obtain updated modal features; a cross-modal attention interaction layer for capturing inter-modal correlations of the updated modal features through an attention matrix to obtain cross-modal fusion features; and a soft attention weighted fusion layer for weighted fusion of the cross-modal fusion features based on importance scores of the cross-modal fusion features to obtain the fusion feature vector.

[0029] In one possible implementation, the intramodal self-attention layer is specifically used to: map the extracted features of each modality to a query matrix, a key matrix, and a value matrix; obtain an attention score matrix from the query matrix and the key matrix; normalize the attention score matrix to obtain an attention weight matrix; and multiply the attention weight matrix with the value matrix to obtain the updated modal features after self-attention weighting.

[0030] In one possible implementation, the cross-modal attention interaction layer is specifically used for: constructing query-key / value cross-modal attention pairs based on updated modal features of different modalities; for each cross-modal attention pair, projecting the updated modal features onto a unified attention space, obtaining cross-modal attention scores from the query modality and key modality, normalizing the cross-modal attention scores to obtain cross-modal attention weights, multiplying the cross-modal attention weights by the value modality to obtain cross-modal context features after cross-modal attention weighting; fusing the cross-modal context features with the updated modal features to obtain intermediate features; and performing a nonlinear transformation on the intermediate features to obtain the cross-modal fused features.

[0031] In one possible implementation, the soft attention-weighted fusion layer is specifically used to: score the importance of the cross-modal fusion features to obtain the original importance scores of each modality; normalize the original importance scores of each modality to obtain the attention weight coefficients of each modality; fuse the cross-modal fusion features using the attention weight coefficients based on a gating mechanism to obtain global fusion features; and perform a nonlinear transformation on the global fusion features to obtain the fusion feature vector.

[0032] In one possible implementation, the classification and recognition module includes a cascaded fully connected layer and a Softmax classifier.

[0033] In one possible implementation, the emotion recognition method further includes: obtaining the raw multimodal data to be processed from the same video file.

[0034] Example 2 like Figure 3 As shown, the multimodal emotion recognition technology provided in this embodiment includes: a data acquisition and preprocessing module for acquiring raw data in three streams: text, audio, and video; a multimodal feature extraction module for independently encoding the three streams of data and outputting text feature vector T, audio feature vector A, and video feature vector V; a multi-layer attention fusion module for achieving deep feature fusion through self-attention, cross-modal attention, and soft attention mechanisms; and a classification output module for performing emotion classification based on the fused features and outputting the recognition results.

[0035] The multimodal emotion recognition method provided in this embodiment has the following specific steps: Step S1: Multimodal data acquisition and preprocessing The multimodal data comes from publicly available multimodal emotion recognition benchmark datasets (MOSI and CH-SIMS). These datasets contain time-aligned text, audio, and video data, and each sample is labeled with an emotion polarity label or emotion intensity score.

[0036] S1.1 Acquisition of Text Modal Data (1) The text transcription content obtained from the audio track of the video file through automatic speech recognition technology, and then manually proofread to form a text sequence with word-level or character-level timestamp alignment; (2) The text sequence is encoded using a pre-trained language model word segmenter to generate a word index sequence; (3) The word index sequence is truncated or padded to a preset maximum sequence length L. text The resulting shape is L text ×D embed The text feature input tensor, where D embed For word embedding dimension.

[0037] S1.2 Acquisition of Audio Modal Data (1) Extract the original pulse code modulation audio signal from the audio track of the video file and resample it to 16kHz. (2) The original audio signal is divided into frames with a frame length of 25 milliseconds (ms) and a frame shift of 10 milliseconds (ms), and a Hamming window is applied to reduce spectral leakage; (3) Use the Mel spectrum or Mel frequency cepstral coefficient (MFCC) method to convert the time-domain audio signal into a time-frequency domain feature matrix; (4) The time-frequency domain feature matrix is ​​deep encoded using a pre-trained audio encoder, and the output shape is T. audio ×D audio The audio feature sequence, where T audio D is the audio time step. audio For audio feature dimensions.

[0038] S1.3 Acquisition of Video Modal Data (1) Sample video frame sequences from video files at a fixed frame rate, and perform face region detection and cropping on each frame, unifying the resolution to H×W pixels; perform normalization processing on the cropped face image sequence, including scaling the pixel values ​​to the [0,1] range and adjusting the channel order to RGB format; (2) Spatial feature encoding of the video frame sequence is performed using a pre-trained convolutional neural network, and the output shape is T. video ×D video Let T be a visual feature sequence. video D is the video time step. video This represents the video feature dimension.

[0039] Step S2: Multimodal Independent Feature Extraction Feature encoding is performed on the three preprocessed modal data respectively to generate corresponding modal feature vectors.

[0040] S2.1 Text Feature Extraction: A pre-trained language model (BERT) is used to encode the text modal data. The text is input into the pre-trained model to obtain word vector representations containing contextual semantic information, and text sequence features are extracted through a bidirectional long short-term memory network (Bi-LSTM) to generate text feature vectors.

[0041] S2.2 Audio Feature Extraction: An audio feature extraction network is constructed, which includes a convolutional neural network (CNN) layer and a Bi-LSTM layer. First, the CNN is used to extract the local acoustic features of the audio, and then the Bi-LSTM is used to extract the time-series dependent features of the audio, generating an audio feature vector.

[0042] like Figure 4 As shown, the audio feature extraction module starts from the original audio signal and goes through the following processing stages in sequence: Preprocessing stage: The raw audio signal is converted into a time-frequency feature matrix through framing (25ms window, 10ms frame shift), windowing (Hamming window) and feature extraction (MFCC spectrum); CNN Feature Extraction Stage: The time-frequency feature matrix is ​​input into the 1D / 2D CNN module, and local acoustic features are extracted through multi-layer convolution, batch normalization and ReLU activation operations, and the output is a local acoustic feature sequence of shape T×D_cnn; Bi-LSTM temporal modeling stage: Local acoustic feature sequences are input into the Bi-LSTM module, and forward and backward temporal dependencies are captured by two LSTM layers, respectively, and the output is a temporal dependency feature sequence of shape T×D_lstm; Global pooling stage: The time-dependent feature sequence is aggregated in the time dimension T by global average pooling operation to generate an audio feature vector of shape 1×D_audio, which serves as the final representation of the audio modality.

[0043] Note: T is the time step, D_cnn is the dimension of the local acoustic features extracted by CNN, D_lstm is the dimension of the temporal dependent features extracted by Bi-LSTM, and D_audio is the dimension of the final audio feature vector.

[0044] S2.3 Video Feature Extraction (Key Improvement): Construct an enhanced video feature extraction network, such as... Figure 5 As shown.

[0045] like Figure 5As shown, this embodiment constructs an enhanced video feature extraction network. This network employs a dual-stream parallel architecture to extract basic visual semantic features and multi-dimensional human attribute features separately. Through feature concatenation, bidirectional temporal modeling, and a self-attention weighting mechanism, it deeply integrates spatial, attribute, and dynamic temporal information, ultimately outputting a highly discriminative video modality feature vector V_out. The specific implementation steps are as follows: Step S2.3.1: Basic visual feature extraction, specifically implemented through the following sub-steps: (1) The video input module receives the preprocessed video frame sequence, performs time-dimension segmentation and frame alignment, and generates a frame-by-frame image data stream; (2) The visual CNN feature extraction unit performs deep spatial feature encoding on the frame-by-frame image data stream. The visual CNN feature extraction unit has a built-in pre-trained ResNet backbone model, which extracts high-level visual semantics through multi-layer convolution and residual connections; then, the visual CNN feature extraction unit calls the global average pooling layer to compress the spatial dimension of the feature map, eliminate positional redundancy and retain channel global statistical information, and output the basic visual feature matrix F_vis.

[0046] Step S2.3.2: Auxiliary attribute features, specifically implemented by the following sub-steps: (1) The frame-by-frame image data stream is synchronously input to the MediaPipe processing unit for parallel feature decoupling and inference; (2) The MediaPipe processing unit calls the pose estimation subunit, the face analysis subunit (gender) and the face analysis subunit (age) respectively; wherein, the pose estimation subunit extracts the coordinate sequence of key points of human skeleton and encodes it as pose features; the face analysis subunit and the face analysis subunit perform attribute classification reasoning on the cropped face region respectively, and output gender feature vector and age feature vector, which are collectively referred to as face features; the pose estimation subunit adopts a three-dimensional human pose estimation model (BlazePose) based on a lightweight convolutional neural network, and the face analysis subunit adopts a fair face attribute recognition model (FairFace) based on a residual network. Each sub-model runs in reasoning mode, the weight parameters are frozen, and the output features are fused with the baseline visual features after encoding.

[0047] (3) The attribute feature fusion unit receives the pose features and face features, first maps the heterogeneous attributes to a unified latent space and performs dimensional alignment through a linear projection layer, and then performs splicing and fusion in the channel dimension to output the auxiliary attribute feature matrix F_attr; Step S2.3.3: Multi-source feature stitching The feature splicing unit receives the basic visual feature matrix F_vis output from step S2.3.1 and the auxiliary attribute feature matrix F_attr output from step S2.3.2, and performs tensor splicing operation in the feature channel dimension to construct an enhanced multi-source feature sequence.

[0048] Step S2.3.4: Bidirectional Temporal Dependency Modeling The enhanced multi-source feature sequence is received by an LSTM network unit, and long-range dynamic dependency modeling is performed using a bidirectional long short-term memory network architecture. The bidirectional long short-term memory network unit includes a forward propagation layer and a backward propagation layer, which respectively capture the emotional evolution context of the video sequence from the past to the future and from the future to the past, and output a bidirectional contextual hidden state sequence.

[0049] Step S2.3.5: Self-attention weighting and feature output (1) The self-attention mechanism unit receives the bidirectional context hidden state sequence and calculates the scaling dot product correlation weights between each time step within the sequence. The self-attention mechanism unit dynamically highlights key frames carrying high emotional discrimination through a multi-head attention mechanism, adaptively suppresses background redundancy or occluded noise frames, and realizes the weighted aggregation of global context information; (2) The video feature output module performs global pooling and dimension normalization on the weighted aggregated sequence and outputs a fixed-dimensional final video modal feature vector Vout as the input of the subsequent multimodal fusion module.

[0050] like Figure 6 As shown, this module contains three levels: The first level is intramodal self-attention: self-attention processing is performed on the features of the three modalities of text, audio and video respectively. Through Q / K / V matrix calculation and layer normalization, each modality can pay attention to the key information in its own sequence and output the updated modal features T', A' and V'.

[0051] The second layer of cross-modal attention interaction: Six sets of bidirectional cross-modal attention interaction pairs are constructed (text↔audio, text↔video, audio↔video). The correlation between modalities is captured by calculating the attention matrix, and then the cross-modal fusion feature T' is obtained through feature concatenation and self-attention processing. cross A' cross V' cross .

[0052] The third layer is soft attention weighted fusion: the importance of cross-modal features of the three modalities is scored, and the weight coefficients α_T, α_A, and α_V of each modality are obtained by Softmax normalization. The weighted fusion is then performed to obtain the global feature F. Finally, a nonlinear transformation is performed through a multilayer perceptron (MLP) to output the final fused feature vector.

[0053] Constructing a multi-layer attention-interaction deep fusion network, such as Figure 6 As shown, the text feature vector, audio feature vector and video feature vector generated in step S2 are fused.

[0054] S3.1 Intramodal Self-Attention Processing: A self-attention mechanism is applied to the text feature vector, audio feature vector, and video feature vector generated in step S2, respectively. This aims to capture long-distance dependencies between different time steps within the sequence of each modality, enabling the model to focus on key segments in its own sequence that contribute significantly to sentiment. The specific processing is parallel and independent for the three modalities. Taking the text modality as an example, the specific sub-steps are as follows: S3.1.1 Linear Mapping Projection: Input text feature vector Each matrix is ​​mapped to a query matrix using three different learnable linear transformation matrices. ), key matrix ) and value matrix ).

[0055] The calculation formula logic is as follows: , , in, , , This is the learnable parameter matrix during model training. (Audio modality) and video modality Using the same processing method, generate the corresponding... , , and , , .

[0056] S3.1.2 Calculate the attention score matrix: Calculate the query matrix AND key matrix The dot product between the transposes is used to measure the correlation between different positions in the sequence. To prevent the dot product from becoming too large and causing the softmax gradient to vanish, the dot product result is divided by a scaling factor. (in (where is the dimension of the key vector).

[0057] The calculation formula logic is as follows: S3.1.3 Normalization yields the attention weights: Attention rating matrix Each row is normalized using the Softmax function to obtain the attention weight matrix. Each value in this weight matrix represents the degree of attention a feature at the current time step pays to features at other time steps; a larger weight indicates a higher relevance. The calculation formula is as follows: S3.1.4 Weighted summation yields contextual features: attention weight matrix AND-value matrix Multiplying them together yields the text context feature representation after self-attention weighting. This step aggregates important information from the sequence into the current representation. The calculation formula logic is as follows: For audio and video modalities: Repeat steps S3.1.1 to S3.1.5 above, respectively, to input... and Updated to and Among them, the audio modality and the video modality have independent learnable parameter matrices (i.e., , , (Not shared across modalities) to adapt to the data distribution characteristics of different modalities.

[0058] Output: After this step, the updated multimodal feature vector is output. , and This serves as the input for the cross-modal attention interaction in step S3.2.

[0059] S3.2 Cross-modal attention interaction: An improved cross-modal attention mechanism is employed. One modality (text) is used as the query, and the other modalities (video, audio) are used as keys and values. Relevance weights between modalities are calculated. Through multiple interactions, such as text focusing on video and video focusing on audio, complementary information between modalities is captured, generating cross-modal fusion features.

[0060] Step S3.2: Cross-modal attention interaction An improved cross-modal attention mechanism is adopted, which realizes information interaction and complementarity between different modalities by constructing multiple cross-modal attention sub-modules. The core idea of ​​this mechanism is to use the features of one modality as the query and the features of other modalities as the key and value, and calculate the relevance weights between the modalities, so that the model can "focus" on important information in another modality from the perspective of one modality.

[0061] S3.2.1 Constructing cross-modal attention pairs Based on the updated features T', A', and V' output in step S3.1, six sets of bidirectional cross-modal attention interaction pairs are constructed: (1) Text → Audio Interaction: Text features are used as queries, and audio features are used as keys / values; (2) Audio → Text Interaction: Audio features are used as queries, and text features are used as keys / values; (3) Text → Video Interaction: Using text features as queries and video features as keys / values; (4) Video → Text Interaction: Use video features as queries and text features as keys / values; (5) Audio → Video Interaction: Use audio features as queries and video features as keys / values; (6) Video → Audio Interaction: Use video features as queries and audio features as keys / values.

[0062] S3.2.2 Cross-modal attention calculation (taking text→audio interaction as an example) For each cross-modal attention pair, perform the following computation steps: (1) Linear mapping projection: Query modality (text) ) and key-value modal (audio) Each is projected onto a unified attention space: , , in, , , This is a learnable parameter matrix for text-to-audio interaction.

[0063] (2) Calculate cross-modal attention scores: Calculate the query matrix AND key matrix The dot product of the transposes measures the correlation between each time step in the text sequence and each time step in the audio sequence: in, Let be the dimension of the key vector. This is the scaling factor.

[0064] (3) Normalization yields the cross-modal attention weights: Attention rating matrix Normalization is performed using the Softmax function: The dimension of the weight matrix is ,in The length of the text sequence. The length of the audio sequence. Elements in the weight matrix. Indicates the text number The time step for the audio The level of attention given to each time step.

[0065] (4) Weighted summation yields cross-modal context features: Multiplying the attention weight matrix by the value matrix yields the cross-modal contextual features of the text extracted from the audio: S3.2.3 Calculation of other cross-modal interaction pairs: Repeat step S3.2.2 to calculate the remaining five cross-modal attention interaction pairs, and obtain: Text-to-video interaction features: Audio-to-text interaction features: Audio → Video Interaction Features: ; Video → Text Interaction Features: Video → Audio Interaction Features: S3.2.4 Cross-modal feature fusion: The original modal features are fused with their corresponding cross-modal context features, taking text modality as an example: in, This indicates a feature concatenation operation (usually performed along the feature dimension).

[0066] Similarly, the cross-modal fusion features of audio and video are obtained: S3.2.5 Nonlinear transformation of fused features: To enhance the model's expressive power, the concatenated cross-modal features undergo a nonlinear transformation through a fully connected layer: in, For learnable weight matrix, For bias terms, This is the activation function (such as GELU or ReLU). The same logic applies to audio and video modalities. and .

[0067] Output: After this step, the output is the cross-modal interactive fusion feature vector. , and This serves as the input for the soft attention weighted fusion step S3.3.

[0068] S3.3 Soft Attention Weighted Fusion: A soft attention mechanism is introduced to dynamically learn the weight coefficients of the three modalities based on the characteristics of the current input data. The features after cross-modal fusion are weighted and summed to obtain the final global fused feature vector.

[0069] A soft attention mechanism is introduced to dynamically learn the weight coefficients of text, audio, and video modalities based on the characteristics of the current input data. This mechanism can adaptively adjust the contribution of different modalities to the final emotion recognition result, and perform weighted summation of the cross-modal fused features to obtain the final global fused feature vector.

[0070] S3.3.1 Modal Importance Score Calculation: For the cross-modal interaction fusion features output in step S3.2 , and Calculate the importance score for each modality separately.

[0071] (1) Text modality importance score: First, text features Mapped to the scalar scoring space via a fully connected layer: in, For learnable weight vectors, For bias terms, The original importance score for the text modality.

[0072] (2) Audio modality importance score: in, For learnable weight vectors, For bias terms, The original importance score for the audio modality.

[0073] (3) Video modality importance score: in, For learnable weight vectors, For bias terms, The original importance score for the video modality.

[0074] S3.3.2 Attention weight normalization: To obtain the normalized weight coefficients for the three modalities (i.e., the sum of the weights is 1), the original importance scores are... , , Normalization is performed using the Softmax function: , in, , , Let be the attention weight coefficients for text, audio, and video modalities, respectively, satisfying: and .

[0075] S3.3.3 Modal feature weighting: Based on the calculated attention weight coefficients, the features of the three modalities are weighted: , , in, , , These are the weighted text, audio, and video modal features (scalar and vector / matrix broadcast multiplication).

[0076] S3.3.4 Global Feature Fusion: The weighted features of the three modalities are fused to obtain a global fused feature vector. This embodiment provides two fusion strategies: Strategy 1: Weighted Summation and Fusion This strategy is simple and efficient, and is suitable for scenarios where there is strong complementarity between modalities.

[0077] Strategy 2: Weighted splicing and fusion (preferred) in, This indicates a feature concatenation operation (performed along the feature dimension). This strategy preserves the independent information of each modality and is suitable for scenarios with significant differences between modalities.

[0078] S3.3.5 Nonlinear transformation of fused features: To further enhance the expressive power of the fused features, a nonlinear transformation is performed on the global fused features using a multilayer perceptron (MLP): in, For learnable weight matrix, For bias terms, For activation functions (such as GELU or ReLU). The fusion features obtained in step S3.3.4 ( or ), This is the global eigenvector after nonlinear transformation.

[0079] S3.3.6 Gating Fusion Mechanism To further enhance the fusion effect, a gating mechanism can be introduced to dynamically control the information flow: (1) Calculate the gating vector: in, It is the Sigmoid activation function. For learnable weight matrix, For bias terms, Let be a gated vector, and its range is... .

[0080] (2) Gated feature transformation: in, This represents element-wise multiplication. This is the final global fusion feature vector after gating control.

[0081] Output: After this step, the final global fused feature vector is output. (or (), which serves as the input for sentiment classification in step S4.

[0082] Step S4: Sentiment Classification and Output The globally fused feature vector is input into a fully connected layer, and a Dropout layer is used to prevent overfitting. Finally, a Softmax classifier is used to calculate the probability of belonging to each sentiment category. The output sentiment recognition result includes sentiment category labels (e.g., positive, neutral, negative) and the corresponding confidence scores.

[0083] Step S5: Detailed Implementation and Effect Verification S5.1 Experimental Environment Configuration Operating System: Linux / Ubuntu 16.04 or equivalent Processor: Intel Xeon CPU E5-2686 v4 or equivalent CPU Graphics Processor: NVIDIA GTX 3090 (24GB VRAM) Deep learning frameworks: TensorFlow 2.13 / PyTorch 1.10+ Programming language: Python 3.10 S5.2 Dataset Application Example The method in this embodiment is validated on two publicly available multimodal sentiment datasets: The MOSI dataset is a product review video dataset collected from the YouTube platform. It contains 2199 short video clips ranging from 3 to 15 seconds in length, in English. Its modal composition includes: manually proofread and word-timestamp aligned subtitles, original audio with a sampling rate of 16kHz, and a sequence of face video frames captured at a sampling rate of 5 frames / second and cropped to a resolution of 112×112 pixels. Each video clip is labeled with an emotion intensity score, ranging from -3 to +3, where -3 represents strong negative emotion, 0 represents neutral emotion, and +3 represents strong positive emotion. According to the standard division method of this dataset, all samples are divided into a training set of 1284, a validation set of 229, and a test set of 686. In terms of task definition, following the common practice in the field of multimodal emotion recognition, continuous emotion intensity scores are mapped to a three-class classification task, specifically: the score interval [-3, -1] is mapped to negative emotion, the score interval [0] is mapped to neutral emotion, and the score interval [+1, +3] is mapped to positive emotion.

[0084] The CH-SIMS dataset is a multimodal sentiment analysis video clip dataset collected from Chinese movies and TV series, containing 2281 video samples in Chinese. Its modal components include: segmented Chinese dialogue text sequences, Chinese audio with a sampling rate of 16kHz, and face video frame sequences acquired at a sampling rate of 5 frames / second and cropped to a resolution of 112×112 pixels. Each video clip is labeled with sentiment polarity (positive / neutral / negative) and fine-grained modality-specific annotations. Following the standard partitioning method for this dataset, all samples are divided into a training set of 1368 samples, a validation set of 456 samples, and a test set of 457 samples, maintaining a balanced distribution across categories. In terms of task definition, it supports multi-granularity sentiment recognition tasks, including: a three-class classification task (positive / neutral / negative), a five-class classification task (strong negative / weak negative / neutral / weak positive / strong positive), and a two-class classification task (negative category includes weak negative and strong negative, non-negative category includes neutral and positive), to comprehensively evaluate the model's discriminative ability at different sentiment granularities.

[0085] S5.3 Effect Verification Example In the MOSI dataset task, the method of this embodiment was implemented using the above-mentioned optimized parameters, and the comparison results with the baseline model are as follows: Performance validation results on the MOSI dataset demonstrate that the proposed method achieves superior technical performance. Specifically, in the five-class emotion recognition task (Acc-5), the proposed method achieves an accuracy of 55.29%, an absolute improvement of 1.82 percentage points compared to the current state-of-the-art baseline model, Self-MM (53.47%). In the more challenging fine-grained seven-class emotion recognition task (Acc-7), the proposed method achieves an accuracy of 46.22%, an absolute improvement of 3.92 percentage points compared to the previously reported best baseline model, MISA (42.3%). These experimental results confirm that the proposed method can effectively capture and distinguish subtle changes in emotion intensity across the entire emotion spectrum, demonstrating significant technical advantages in fine-grained emotion discrimination tasks.

[0086] In the CH-SIMS dataset task, the method of this embodiment was implemented using the above-mentioned optimized parameters, and the comparison results with the baseline model are as follows: The performance validation results on the Chinese CH-SIMS dataset are shown in Table 3. Experimental results show that the method in this embodiment consistently outperforms the baseline models on the CH-SIMS dataset. Specifically, compared with the current strongest baseline model Self-MM (Acc-5: 41.53%, Acc-2: 80.04%), the method in this embodiment achieves a performance of 43.11% in the five-class classification task (Acc-5), an absolute improvement of 1.58 percentage points; and an performance of 81.05% in the two-class classification task (Acc-2), an absolute improvement of 1.01 percentage points; at the same time, it also achieves a performance of 80.64% in the weighted F1 score, which is better than the baseline model's 80.44%. The above experimental results in cross-linguistic scenarios confirm that the method in this embodiment can effectively adapt to the linguistic characteristics and cultural background differences of Chinese emotional expression, and has excellent technical robustness and generalization ability in different language and cultural contexts.

[0087] Ablation experiments validate the contributions of each module (using the CH-SIMS dataset as an example): To verify the technical contribution of auxiliary attributes in video feature extraction, this embodiment conducts ablation experiments under a fixed hierarchical attention fusion mechanism. As shown in the table, when all auxiliary attributes (including human posture, gender, and age) are removed from the complete model, the model's weighted F1 score and five-class classification accuracy (Acc-5) both decrease by 2.80 percentage points. Specifically, the F1 score decreases from 80.64% to 77.84%, and the Acc-5 score decreases from 43.11% to 40.31%. This experimental data confirms that introducing demographic features and behavioral cues can provide effective intramodal complementary information for traditional facial expression analysis, thereby significantly improving the accuracy of emotion recognition. Furthermore, introducing human posture features alone can improve the F1 score by 1.19 percentage points, and adding gender and age attributes on top of this can provide an additional performance gain of 0.82 percentage points. The above results fully verify the significant technical effect of the multi-attribute enhanced visual feature extraction strategy described in this embodiment in improving the discriminative power and anti-interference ability of video modality representation.

[0088] This application also provides an electronic device, including: a processor, and a memory coupled to the processor, the memory being used to store a computer program; the processor being used to execute the computer program stored in the memory, so that the electronic device performs the method as described in any of the above embodiments.

[0089] Electronic devices can be computing devices such as desktop computers, laptops, handheld computers, and cloud servers. These electronic devices may include, but are not limited to, processors and memory.

[0090] The processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor can be a microprocessor or any conventional processor. The processor is the control center of the electronic device, connecting various parts of the device via various interfaces and lines.

[0091] The memory can be used to store the computer program, and the processor implements various functions of the electronic device by running or executing the computer program stored in the memory and calling the data stored in the memory.

[0092] The memory may primarily include a program storage area and a data storage area. The program storage area may store the operating system, applications required for at least one function, etc.; the data storage area may store data created based on the use of the mobile phone, etc. In addition, the memory may include high-speed random access memory, and may also include non-volatile memory, such as hard disk, memory, plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, at least one disk storage device, flash memory device, or other volatile solid-state storage device.

[0093] This application also provides a storage medium, which is a computer-readable storage medium. The computer program is stored in the computer-readable storage medium, and when executed by a processor, the computer program can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable file, or some intermediate form. The computer-readable medium can include: any entity or device capable of carrying the computer program code, a recording medium, a USB flash drive, a portable hard drive, a magnetic disk, an optical disk, a computer memory, a read-only memory (ROM), a random access memory (RAM), an electrical carrier signal, a telecommunication signal, and a software distribution medium, etc.

[0094] This application also provides a computer program product, including: a computer program or instructions that, when the computer program or instructions are run on a computer, cause the computer to perform any of the above possible implementation methods.

[0095] The above description is the preferred embodiment of this application. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of this application, and these improvements and modifications are also considered to be within the scope of protection of this application.

Claims

1. An emotion recognition method, characterized in that, include: Obtain time-aligned raw multimodal data to be processed, which includes: raw text data, raw audio data, and raw video data; The trained emotion recognition model is used to process the original multimodal data to obtain the emotion recognition results. The emotion recognition model includes: The multimodal feature extraction module is used to extract features from the original data of each modality to obtain the extracted features for each modality. The multi-layer attention fusion module is used to achieve deep fusion of features extracted from various modalities through an attention mechanism to obtain a fused feature vector; In addition, there is a classification and recognition module, which is used to classify emotions based on fused feature vectors and output the emotion recognition results.

2. The emotion recognition method as described in claim 1, characterized in that, The multimodal feature extraction module includes: a text feature extraction module, an audio feature extraction module, and a video feature extraction module. The video feature extraction module includes: a first convolutional neural network for extracting a basic visual feature matrix from the original video data; a multimedia processing pipeline for calling a pose estimation subunit and a face feature analysis subunit to extract pose features and face features from the original video data, respectively; a linear projection layer for dimensionally aligning the pose features and face features; a fusion layer for fusing the dimensionally aligned pose features and face features to obtain an auxiliary attribute feature matrix; a feature concatenation unit for concatenating the basic visual feature matrix and the auxiliary attribute feature matrix to obtain an enhanced multi-source feature sequence; a first bidirectional long short-term memory network for capturing a bidirectional contextual hidden state sequence from the enhanced multi-source feature sequence; a self-attention mechanism unit for weighted aggregation of the bidirectional contextual hidden state sequence using a self-attention mechanism to obtain a weighted aggregated sequence; and a video feature output unit for obtaining a video modal feature vector from the weighted aggregation sequence.

3. The emotion recognition method as described in claim 2, characterized in that, The text feature extraction module includes a cascaded language model and a second bidirectional long short-term memory network, and the audio feature extraction module includes a cascaded second convolutional neural network and a third bidirectional long short-term memory network.

4. The emotion recognition method as described in claim 1, characterized in that, The multi-layer attention fusion module includes: an intra-modal self-attention layer for processing features extracted from each modality through a self-attention mechanism to obtain updated modal features; a cross-modal attention interaction layer for capturing inter-modal correlations of the updated modal features through an attention matrix to obtain cross-modal fusion features; and a soft attention weighted fusion layer for weighted fusion of the cross-modal fusion features based on importance scores of the cross-modal fusion features to obtain the fusion feature vector.

5. The emotion recognition method as described in claim 4, characterized in that, The intramodal self-attention layer is specifically used for: mapping the extracted features of each modality into a query matrix, a key matrix, and a value matrix; obtaining an attention score matrix from the query matrix and the key matrix; and normalizing the attention score matrix to obtain an attention weight matrix. Furthermore, the attention weight matrix is ​​multiplied by the value matrix to obtain the updated modal features after self-attention weighting.

6. The emotion recognition method as described in claim 4, characterized in that, The cross-modal attention interaction layer is specifically used for: constructing query-key / value cross-modal attention pairs based on the updated modal features of different modalities; for each cross-modal attention pair, projecting the updated modal features onto a unified attention space, obtaining cross-modal attention scores from the query modality and key modality, normalizing the cross-modal attention scores to obtain cross-modal attention weights, and multiplying the cross-modal attention weights with the value modality to obtain cross-modal context features after cross-modal attention weighting; The cross-modal context features are fused with the updated modal features to obtain intermediate features; and the intermediate features are subjected to a nonlinear transformation to obtain the cross-modal fused features.

7. The emotion recognition method as described in claim 4, characterized in that, The soft attention weighted fusion layer is specifically used to: score the importance of the cross-modal fusion features to obtain the original importance scores of each modality; Normalize the original importance scores of each modality to obtain the attention weight coefficients of each modality; Based on the gating mechanism, the attention weight coefficients are used to fuse the cross-modal fusion features to obtain global fusion features; and the global fusion features are subjected to a nonlinear transformation to obtain the fusion feature vector.

8. The emotion recognition method as described in claim 1, characterized in that, The classification and recognition module includes a cascaded fully connected layer and a Softmax classifier.

9. The emotion recognition method as described in claim 1, characterized in that, The emotion recognition method further includes: obtaining the original multimodal data to be processed from the same video file.

10. An electronic device, characterized in that, The electronic device includes: a processor, and a memory coupled to the processor. The memory is used to store computer programs; The processor is configured to execute the computer program stored in the memory, so that the electronic device performs the emotion recognition method as described in any one of claims 1-9.

Images