Audio-visual multi-modal emotion recognition method and system based on bidirectional cross attention

Abstract

The application discloses a method and system for audio-visual multi-modal emotion recognition based on bidirectional cross attention, relates to the technical field of emotion recognition, and through integration of facial action unit and acoustic feature analysis, utilizes a bidirectional cross attention network to relieve modality heterogeneity interference and enhance cross-modality interaction modeling, dynamically captures key interaction information of audio-visual modalities through the bidirectional cross attention, realizes effective relief of modality heterogeneity and deep mining of cross-modality complementary information. The application firstly extracts visual facial action unit features and audio acoustic features, maps to a unified dimension for alignment, then utilizes a bidirectional cross attention layer for global cross-modality interaction and feature enhancement, and finally inputs a classifier to output an emotion prediction result. The method significantly improves the emotion recognition accuracy, has good stability and generalization ability.

Description

Technical Field

[0001] This invention relates to the field of emotion recognition technology, specifically to a method and system for audiovisual multimodal emotion recognition based on bidirectional cross-attention. Background Technology

[0002] Traditional emotion recognition methods, while achieving some success by relying on single-modal information, often suffer from insufficient robustness in complex real-world environments. Facial expression recognition's visual signals are easily affected by factors such as lighting conditions, head posture, and partial occlusion; speech emotion recognition is susceptible to environmental noise, individual acoustic characteristics, and differences in expression habits. This inherent incompleteness and instability of single-modal information limits the model's generalization ability. Existing multimodal emotion recognition methods, although fusing features through feature concatenation or simple attention mechanisms, face challenges such as effectively mitigating heterogeneous interference between audio and visual modalities, insufficient cross-modal interaction modeling, and difficulty in dynamically capturing the crucial audiovisual dependencies for emotion discrimination. Furthermore, they lack the expertise to fully model bidirectional cross-attention to capture cross-modal emotional interaction dependencies, resulting in lower accuracy in multimodal emotion recognition. Summary of the Invention

[0003] To address the shortcomings mentioned in the background art, the present invention aims to provide a method and system for audiovisual multimodal emotion recognition based on bidirectional cross-attention, which solves the problems of modal heterogeneity interference and insufficient cross-modal interaction modeling in the prior art, and achieves emotion recognition with higher accuracy and better generalization ability.

[0004] Firstly, the objective of this invention can be achieved through the following technical solution: a multimodal audiovisual emotion recognition method based on bidirectional cross-attention, the method comprising the following steps: Obtain the sentiment features of the first modality and the sentiment features of the second modality; The emotional features of the first modality and the emotional features of the second modality are subjected to bidirectional cross-attention processing to obtain the first modality enhanced features and the second modality enhanced features; Wherein, the first modality enhancement feature is generated based on the first attention weight from the first modality to the second modality; the second modality enhancement feature is generated based on the second attention weight from the second modality to the first modality; The first modality enhancement features and the second modality enhancement features are fused to obtain multimodal global fusion features; the multimodal global fusion features are then used for sentiment classification to obtain the sentiment recognition result.

[0005] In conjunction with the first aspect, in some implementations of the first aspect, the method further includes: the process of performing bidirectional cross-attention fusion processing on the emotional features of the first modality and the emotional features of the second modality, comprising the following steps: Linear projections are performed on the sentiment features of the first modality and the sentiment features of the second modality to obtain the query vector, key vector and value vector corresponding to the first modality and the second modality respectively; The first attention weight is calculated using the query vector of the first modality and the key vector of the second modality. The value vector of the second modality is then weighted based on the first attention weight to obtain the enhanced features of the first modality. The second attention weight is calculated by using the query vector of the second modality and the key vector of the first modality. The value vector of the first modality is then weighted based on the second attention weight to obtain the enhanced features of the second modality.

[0006] In conjunction with the first aspect, in some implementations of the first aspect, the method further includes: the first modality being an audio modality and the second modality being a visual modality.

[0007] In conjunction with the first aspect, in some implementations of the first aspect, the method further includes: the first attention weight is an audio-to-visual attention weight, and the second attention weight is a visual-to-audio attention weight; The first and second attention weights are calculated using a scaled dot product, and adaptive information selection is achieved through a Sigmoid gate. The calculation formula is as follows: in It is the attention weight from audio to visual. It is the attention weight from visual to audio. The query, key, and value vector is obtained by linear projection of the audio modality. This is the vector corresponding to the visual modality. For the projection dimension.

[0008] In conjunction with the first aspect, in some implementations of the first aspect, the method further includes: fusing the first modality enhancement features and the second modality enhancement features to obtain multimodal global fusion features, as shown below: in, For multimodal global fusion features, Indicates cascading splicing. It is the value vector of the visual modality. It is the value vector of the audio modality. and These are the parameters of the bidirectional cross-attention fusion module.

[0009] In conjunction with the first aspect, in some implementations of the first aspect, the method further includes: the extraction process of the emotional features of the audio modality and the emotional features of the visual modality, as follows: Acquire and preprocess voice and facial video data; We applied manual feature extraction methods to extract features from the acoustic information of audio modalities to obtain the emotional features of the audio modalities. The facial action unit extraction method is used to extract features from the visual modality to obtain the emotional features of the visual modality.

[0010] In conjunction with the first aspect, in some implementations of the first aspect, the method further includes: the process of extracting features from the audio modal acoustic information using a manual feature extraction method, comprising: Using the open-source speech feature extraction toolkit openSMILE and the standard configuration of the INTERSPEECH 2010 Paraling Challenge IS10_paraling, the original audio segments in the speech data at the preset sampling rate are processed into frames, and the various acoustic parameters of each short frame are statistically aggregated to obtain a static representation. The design of an audio encoder based on a multilayer perceptron further extracts abstract emotional features from static representations through layer-by-layer nonlinear transformation and reduces the dimensionality to obtain discourse-level audio features, which serve as the emotional features of the audio modality.

[0011] In conjunction with the first aspect, in some implementations of the first aspect, the method further includes: the process of extracting features from the visual modality using the facial action unit extraction method, comprising: Face detection and alignment were performed using the open-source face behavior analysis toolkit OpenFace. A preset number of motion unit intensity values ​​and corresponding binary indicators were extracted. The multidimensional facial motion unit vectors of each frame of the video were averaged in the time dimension to obtain fixed-dimensional discourse-level visual features, which served as the emotional features of the visual modality. Among them, the binary indicator value was 1 when it was presented and 0 when it was not presented.

[0012] Secondly, in order to achieve the above objectives, this invention discloses an audiovisual multimodal emotion recognition system based on bidirectional cross-attention, comprising: The feature acquisition module is used to acquire the sentiment features of the first modality and the sentiment features of the second modality. A bidirectional cross-attention processing module is used to perform bidirectional cross-attention processing on the emotional features of the first modality and the emotional features of the second modality to obtain the first modality enhanced features and the second modality enhanced features; Wherein, the first modality enhancement feature is generated based on the first attention weight from the first modality to the second modality; the second modality enhancement feature is generated based on the second attention weight from the second modality to the first modality; The emotion recognition module is used to fuse the first modality enhancement features and the second modality enhancement features to obtain multimodal global fusion features; and to classify the multimodal global fusion features for emotion to obtain the emotion recognition result.

[0013] The beneficial effects of this invention are: This invention uses hand-defined features to represent audio acoustic information and facial expression action units that are closely related to emotions. It combines a bidirectional cross-attention mechanism to fully and effectively integrate audio modality and visual modality features, effectively alleviating the problems of modal heterogeneity interference and insufficient cross-modal interaction modeling. It achieves adaptive capture of emotional dependencies between audiovisual modalities and improves the accuracy of emotion recognition. Attached Figure Description

[0014] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort. Figure 1 This is a schematic diagram of the method flow of the present invention; Figure 2 This is a schematic diagram of the emotion recognition framework of the present invention; Figure 3 This is a schematic diagram of the overall emotion recognition process of the present invention; Figure 4 This is a schematic diagram of the system structure of the present invention. Detailed Implementation

[0015] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0016] Example 1: like Figure 1 As shown, the audiovisual multimodal emotion recognition method based on bidirectional cross-attention includes the following steps: S101: Obtain the sentiment features of the first modality and the sentiment features of the second modality; The first modality is the audio modality, and the second modality is the visual modality.

[0017] The extraction process of emotional features from the audio modality and the visual modality is as follows: Acquire and preprocess voice and facial video data; We applied manual feature extraction methods to extract features from the acoustic information of audio modalities to obtain the emotional features of the audio modalities. The facial action unit extraction method is used to extract features from the visual modality to obtain the emotional features of the visual modality.

[0018] The process of acquiring and preprocessing voice and facial video data is as follows: Facial data is captured using a high-resolution camera or video frames, along with corresponding audio signals. The audio signals undergo digital signal processing preprocessing, including denoising and normalization. The facial images are then denoised and brightness-equalized to ensure consistent data quality.

[0019] The process of extracting features from audio modal acoustic information using manual feature extraction methods includes: Using the IS10 paraling standard configuration of the openSMILE toolkit, a raw audio segment with a sampling rate of 16kHz was processed into frames (25ms frame length, 10ms frame shift). Static characterization was obtained by statistically aggregating multiple acoustic parameters for each short frame. The preset sampling rate can be selected between 8kHz and 48kHz according to the actual application scenario, and is preferably 16kHz in this embodiment; the preset frame length can be selected between 20ms and 30ms, and the preset frame shift can be selected between 10ms and 15ms, and is preferably 25ms and 10ms in this embodiment, respectively. The preliminary audio features contain key acoustic information in the time and frequency domains of the speech signal. The energy and zero-crossing rate characterize the rhythm and intensity changes of the speech, and the Mel-frequency cepstral coefficients (MFCC) encode the vocal tract resonance characteristics. These complementary information together constitute a highly discriminative acoustic representation, providing a reliable emotional cue basis for subsequent multimodal emotion analysis. To reduce high-dimensional acoustic features To address redundancy, an audio encoder based on a multilayer perceptron (MLP) is designed. This encoder further extracts abstract emotional features and reduces dimensionality to obtain speech-level audio features through layer-by-layer nonlinear transformation. The GELU activation function is introduced into the hidden layer of the MLP, and a dropout mechanism with a preset probability is used to suppress overfitting. The specific value of the preset probability can be adaptively adjusted according to the actual network depth and data size, and is usually set in the range of 0.1 to 0.5. In this embodiment, in order to best balance feature representation and overfitting risk, a value of 0.3 is preferred, and its forward propagation is defined as follows: in and These are parameters of the MLP audio encoder. and It is the output of the hidden layer. It is a speech-level audio emotional feature.

[0020] The process of extracting features from visual modalities using facial action unit extraction methods includes: Face detection and alignment were performed using the open-source face behavior analysis toolkit (OpenFace). Intensity values ​​(0 to 5) of 17 action units and 18 corresponding binary indicators (1 indicating presentation, 0 indicating absence) were extracted. The 35-dimensional action unit vectors of each frame of the video were then averaged and pooled over time to obtain speech-level visual features. In other embodiments, other facial feature extraction tools or newer versions may be used to extract action unit features of different dimensions; no specific limitations are imposed here.

[0021] This method, based on action unit-based feature extraction, effectively avoids the problems of high dimensionality and redundancy in the original image, while retaining key facial semantic information highly relevant to emotion judgment. Its forward propagation is defined as: in and These are the parameters of the MLP visual encoder. It is the output of the hidden layer. It is a discourse-level visual emotional feature.

[0022] Multilayer perceptron networks are applied to further extract emotional features from the handcrafted features of the two modalities and achieve modal dimension alignment. In this embodiment, independent MLP encoders perform multilayer linear projection and nonlinear activation transformation on audio features and visual features respectively, projecting them uniformly into a 128-dimensional space. The GELU activation function and Dropout mechanism are introduced to enhance feature representation ability and generalization performance.

[0023] S102: Perform bidirectional cross-attention processing on the emotional features of the first modality and the emotional features of the second modality to obtain the first modality enhanced features and the second modality enhanced features; Wherein, the first modality enhancement feature is generated based on the first attention weight from the first modality to the second modality; the second modality enhancement feature is generated based on the second attention weight from the second modality to the first modality; The process of performing bidirectional cross-attention fusion processing on the emotional features of the first modality and the emotional features of the second modality includes the following steps: Linear projections are performed on the sentiment features of the first modality and the sentiment features of the second modality to obtain the query vector, key vector and value vector corresponding to the first modality and the second modality respectively; The first attention weight is calculated using the query vector of the first modality and the key vector of the second modality. The value vector of the second modality is then weighted based on the first attention weight to obtain the enhanced features of the first modality. The second attention weight is calculated by using the query vector of the second modality and the key vector of the first modality. The value vector of the first modality is then weighted based on the second attention weight to obtain the enhanced features of the second modality.

[0024] In this embodiment, the audio encoding features and visual encoding features are first linearly projected to obtain their respective query vectors, key vectors, and value vectors. Then, the audio-to-visual attention weight is calculated using the audio query vector and the visual key vector. Next, the visual-to-audio attention weight is calculated using the visual query vector and the audio key vector. Finally, the bidirectional attention outputs are weighted and fused to obtain a global fusion feature containing modal complementarity information. The attention calculation uses a scaled dot product and an adaptive information selection is achieved through a Sigmoid activation function. The calculation formula is as follows: in These are learnable parameters. The query, key, and value vector is obtained by linear projection of the audio modality. This is the vector corresponding to the visual modality. It is the attention weight from audio to visual. It is the attention weight from visual to audio. The projection dimension is used for scaling to avoid numerical instability in attention scores during dot product calculations.

[0025] The construction process of bidirectional cross-attention fusion is as follows: The construction and training processes of bidirectional cross-attention fusion are detailed below: (1) Construction of bidirectional cross-attention network and selection of basic network: When constructing a bidirectional cross-attention network, the choice of the base encoder network and the attention mechanism directly affects the model's feature representation ability and computational efficiency.

[0026] For the audio encoder, the optional base network includes one-dimensional convolutional neural networks, recurrent neural networks, and multilayer perceptrons. Considering that the preliminary acoustic features extracted by this invention are speech-level statistical features of a preset dimension and do not have temporal structure, this invention preferably uses a multilayer perceptron (MLP) as the audio encoder. In this embodiment, the specific construction process is as follows: through a linear layer containing three learnable weight matrices, the preset high-dimensional audio features of 1582 dimensions are nonlinearly reduced layer by layer (e.g., successively reduced to 512 dimensions, 256 dimensions), and finally mapped to a unified 128-dimensional space; during this process, batch normalization, GELU activation function, and a random dropout mechanism with a preset probability (e.g., 0.3) are combined as shown in the attached figure.

[0027] For the visual encoder, the optional base network includes convolutional neural networks, 3D convolutional neural networks, and multilayer perceptrons. Given that the facial action unit features extracted in this invention are semantic-level handcrafted features of a preset dimension (in this embodiment, 35-dimensional features are composed of 17 action unit intensity values ​​and 18 corresponding binary indicators), lacking spatial topological structure, this invention preferably uses a multilayer perceptron (MLP) as the visual encoder. In this embodiment, the specific construction process is as follows: The aforementioned visual features of the preset dimension (e.g., 35-dimensional) are mapped to a preset hidden layer dimension (e.g., 64-dimensional) through a linear layer containing two learnable weight matrices, and then uniformly projected to a 128-dimensional space; similarly, batch normalization, the GELU activation function, and a preset probability random deactivation mechanism are used, as shown in the accompanying drawings.

[0028] For the cross-attention mechanism, options include traditional multi-head attention and single-head attention mechanisms. Since the audio and video features in this invention are all utterance-level global vectors with a sequence length of 1, on the one hand, multi-head mechanisms introduce redundant parameters; on the other hand, if the standard Softmax operation in traditional mechanisms is used, the attention weights will degenerate to a constant 1, causing dynamic weighting to fail. Therefore, this invention ultimately chooses a single-head bidirectional cross-attention mechanism based on Sigmoid gating, as a global cross-modal correlation gating mechanism to dynamically scale the feature representation of the current modality.

[0029] (2) The specific training process of the bidirectional cross-attention network: In the specific training process of the model, the input data are preprocessed (preferably Z-score normalized) audio modal acoustic features and visual modal facial action unit features, as well as the corresponding real emotion category labels; the output data are the posterior probability distributions of six basic emotions, namely anger, disgust, fear, happiness, neutrality and sadness, output by the classifier module through the Softmax activation function.

[0030] The end-to-end optimization training of the model uses the cross-entropy loss function to effectively measure the difference between the predicted probability distribution and the true label.

[0031] In terms of parameter optimization and hyperparameter settings, the Adam optimization algorithm is used to update network weights; the initial learning rate is set to a preset value (e.g., 0.0001); to suppress the risk of overfitting, not only is a random deactivation mechanism with a preset probability used in the hidden layers of the network, but the weight decay coefficient of the optimizer is also set to a preset value (e.g., 0.004); the training batch size is set to a preset size (e.g., 64).

[0032] S103: The first modality enhancement features and the second modality enhancement features are fused to obtain multimodal global fusion features; the multimodal global fusion features are used for emotion classification to obtain the emotion recognition result.

[0033] The multimodal global fusion feature is represented as: in For global features after full cross-modal fusion, Indicates cascading splicing. It is the value vector of the visual modality. It is the value vector of the audio modality. and These are the parameters of the bidirectional cross-attention fusion module.

[0034] Sentiment classification using multimodal global fusion features employs a multilayer perceptron structure, with the fused global features as input. The algorithm outputs posterior probability distributions of six basic emotions—anger, disgust, fear, happiness, neutrality, and sadness—after a fully connected layer and a Softmax activation function. End-to-end optimization training uses the cross-entropy loss function, whose forward propagation definition is: ReLU stands for Rectified Linear Unit, which is the activation function used in the hidden layer. and These are the parameters of the classifier module. It is the output of the hidden layer. This is the result of the sentiment category prediction.

[0035] Specifically, the present invention will be further illustrated below through embodiments: like Figure 2As shown, the emotion recognition framework mainly consists of an audio / video encoder, a bidirectional cross-attention fusion network, and a classifier. Facial actions and acoustic features are dimensionality-reduced by independent encoders and mapped to the same dimension for alignment. Subsequently, the fusion network performs interactive modeling of the two modal features, extracting key cross-modal information. Finally, the classifier processes the global fused features and outputs the emotion prediction result.

[0036] like Figure 3 As shown, the emotion recognition process is divided into three core stages: the modality encoding stage, where audiovisual features are processed by independent networks (including linear layers, batch normalization, etc.) and mapped to the same-dimensional space for alignment; the cross-modal interaction fusion stage, where bidirectional attention weights are generated using a bidirectional cross-attention module and a Sigmoid gating to dynamically strengthen and splice multimodal features; and the emotion discrimination stage, where the classifier outputs the final emotion recognition result through linear mapping and a Softmax activation function.

[0037] The effectiveness of this method is further illustrated through comparative experiments: Dataset Description: This experiment was conducted extensively on the CREMA-D and RAVDESS datasets to test the adaptability of the proposed method under different data distribution conditions. The CREMA-D dataset contains 7442 audio and video samples recorded by 91 actors, covering six basic emotions: anger, disgust, fear, happiness, neutrality, and sadness. This dataset exhibits significant intra-class variability due to the wide range of actors and diverse demographic attributes, placing high demands on the model's generalization ability. The RAVDESS dataset consists of recordings by 24 professional actors, with a subset of 1440 audio samples covering eight basic emotions: anger, disgust, fear, happiness, neutrality, sadness, calmness, and surprise. To ensure consistency across datasets, this paper aligned the emotion labeling system, removing the "calmness" and "surprise" categories from the RAVDESS dataset, retaining only the six common emotions.

[0038] Evaluation metrics description: The experiment uses the commonly used evaluation metrics for classification tasks, accuracy and unweighted average recall (UAR), to characterize the performance of the present invention from the aspects of overall recognition accuracy, class balance and comprehensive discrimination ability.

[0039] Experimental Results: Under the same experimental settings, this invention was compared on the CREMA-D and RAVDESS datasets.

[0040] Table 1: Experimental results of different models on the CREMA-D dataset Table 2: Experimental results of different models on the RAVDESS dataset Table 3: Ablation Experiment Results Based on the above experimental comparisons, it can be found that our proposed method outperforms some existing state-of-the-art methods on two benchmark datasets, demonstrating good cross-dataset stability and good generalization ability under different emotional expressions. On the basis of unimodal emotion recognition, the introduction of multimodal fusion can significantly improve the overall performance. Simple feature concatenation with multimodal features has already brought significant performance gains on both datasets, and our proposed method further achieves stable improvement by introducing a bidirectional cross-attention fusion module to fully interact audio and visual features. In addition to average performance, our proposed method shows smaller performance fluctuations on CREMA-D, and its UAR standard deviation is significantly lower than that of the comparative methods, indicating that it has better stability and robustness under different random initialization conditions.

[0041] Example 2: To achieve the above objective, such as Figure 4 As shown, based on Embodiment 1, this invention discloses an audiovisual multimodal emotion recognition system based on bidirectional cross-attention, comprising: The feature acquisition module 11 is used to acquire the sentiment features of the first modality and the sentiment features of the second modality; The bidirectional cross-attention processing module 12 is used to perform bidirectional cross-attention processing on the emotional features of the first modality and the emotional features of the second modality to obtain the enhanced features of the first modality and the enhanced features of the second modality. Wherein, the first modality enhancement feature is generated based on the first attention weight from the first modality to the second modality; the second modality enhancement feature is generated based on the second attention weight from the second modality to the first modality; The emotion recognition module 13 is used to fuse the first modality enhancement features and the second modality enhancement features to obtain multimodal global fusion features; and to perform emotion classification on the multimodal global fusion features to obtain the emotion recognition result.

[0042] Based on the same inventive concept, this invention also provides a computer device, comprising: one or more processors, and a memory for storing one or more computer programs; the programs include program instructions, and the processor executes the program instructions stored in the memory. The processor may be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. It is the computing and control core of the terminal, used to implement one or more instructions, specifically for loading and executing one or more instructions stored in a computer storage medium to implement the above-described method.

[0043] It should be further explained that, based on the same inventive concept, the present invention also provides a computer storage medium storing a computer program, which, when executed by a processor, performs the above-described method. This storage medium can be any combination of one or more computer-readable media. The computer-readable medium can be a computer-readable signal medium or a computer-readable storage medium. The computer-readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of computer-readable storage media (a non-exhaustive list) include: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In the present invention, the computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0044] In the description of this specification, references to terms such as "an embodiment," "example," "specific example," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of this disclosure. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0045] The foregoing has shown and described the basic principles, main features, and advantages of this disclosure. Those skilled in the art should understand that this disclosure is not limited to the above embodiments. The embodiments and descriptions in the specification are merely illustrative of the principles of this disclosure. Various changes and modifications can be made to this disclosure without departing from its spirit and scope, and all such changes and modifications fall within the scope of this disclosure as claimed.

Claims

1. A multimodal audiovisual emotion recognition method based on bidirectional cross-attention, characterized in that, The method includes the following steps: Obtain the sentiment features of the first modality and the sentiment features of the second modality; The emotional features of the first modality and the emotional features of the second modality are subjected to bidirectional cross-attention processing to obtain the first modality enhanced features and the second modality enhanced features; Wherein, the first modality enhancement feature is generated based on the first attention weight from the first modality to the second modality; the second modality enhancement feature is generated based on the second attention weight from the second modality to the first modality; The first modality enhancement features and the second modality enhancement features are fused to obtain multimodal global fusion features; the multimodal global fusion features are then used for sentiment classification to obtain the sentiment recognition result.

2. The audiovisual multimodal emotion recognition method based on bidirectional cross-attention according to claim 1, characterized in that, The process of performing bidirectional cross-attention fusion processing on the emotional features of the first modality and the emotional features of the second modality includes the following steps: Linear projections are performed on the sentiment features of the first modality and the sentiment features of the second modality to obtain the query vector, key vector and value vector corresponding to the first modality and the second modality respectively; The first attention weight is calculated using the query vector of the first modality and the key vector of the second modality. The value vector of the second modality is then weighted based on the first attention weight to obtain the enhanced features of the first modality. The second attention weight is calculated by using the query vector of the second modality and the key vector of the first modality. The value vector of the first modality is then weighted based on the second attention weight to obtain the enhanced features of the second modality.

3. The audiovisual multimodal emotion recognition method based on bidirectional cross-attention according to claim 2, characterized in that, The first modality is an audio modality, and the second modality is a visual modality.

4. The audiovisual multimodal emotion recognition method based on bidirectional cross-attention according to claim 3, characterized in that, The first attention weight is the attention weight from audio to vision, and the second attention weight is the attention weight from vision to audio; The first and second attention weights are calculated using a scaled dot product, and adaptive information selection is achieved through a Sigmoid gate. The calculation formula is as follows: in It is the attention weight from audio to visual. It is the attention weight from visual to audio. The query, key, and value vector is obtained by linear projection of the audio modality. This is the vector corresponding to the visual modality. For the projection dimension.

5. The audiovisual multimodal emotion recognition method based on bidirectional cross-attention according to claim 1, characterized in that, The first modality enhancement feature and the second modality enhancement feature are fused to obtain the multimodal global fusion feature, as shown below: in, For multimodal global fusion features, Indicates cascading splicing. It is the value vector of the visual modality. It is the value vector of the audio modality. and These are the parameters of the bidirectional cross-attention fusion module.

6. The audiovisual multimodal emotion recognition method based on bidirectional cross-attention according to claim 5, characterized in that, The extraction process of the emotional features of the audio modality and the emotional features of the visual modality is as follows: Acquire and preprocess voice and facial video data; We applied manual feature extraction methods to extract features from the acoustic information of audio modalities to obtain the emotional features of the audio modalities. The facial action unit extraction method is used to extract features from the visual modality to obtain the emotional features of the visual modality.

7. The audiovisual multimodal emotion recognition method based on bidirectional cross-attention according to claim 6, characterized in that, The process of extracting features from audio modal acoustic information using the manual feature extraction method includes: Using the open-source speech feature extraction toolkit openSMILE and the standard configuration of the INTERSPEECH 2010 Paraling Challenge IS10_paraling, the original audio segments in the speech data at the preset sampling rate are processed into frames, and the various acoustic parameters of each short frame are statistically aggregated to obtain a static representation. The design of an audio encoder based on a multilayer perceptron further extracts abstract emotional features from static representations through layer-by-layer nonlinear transformation and reduces the dimensionality to obtain discourse-level audio features, which serve as the emotional features of the audio modality.

8. The audiovisual multimodal emotion recognition method based on bidirectional cross-attention according to claim 7, characterized in that, The process of extracting features from visual modalities using the facial action unit extraction method includes: Face detection and alignment were performed using the open-source face behavior analysis toolkit OpenFace. A preset number of motion unit intensity values ​​and a preset number of corresponding binary indicators were extracted. The multidimensional facial motion unit vectors of each frame of the video were averaged in the time dimension to obtain fixed-dimensional discourse-level visual features, which served as the emotional features of the visual modality. Among them, the corresponding value of 1 in the binary indicator indicates that the action unit was presented, and the corresponding value of 0 indicates that the action unit was not presented.

9. An audiovisual multimodal emotion recognition system based on bidirectional cross-attention, employing the audiovisual multimodal emotion recognition method based on bidirectional cross-attention as described in any one of claims 1 to 8, characterized in that, include: The feature acquisition module is used to acquire the sentiment features of the first modality and the sentiment features of the second modality. A bidirectional cross-attention processing module is used to perform bidirectional cross-attention processing on the emotional features of the first modality and the emotional features of the second modality to obtain the first modality enhanced features and the second modality enhanced features; Wherein, the first modality enhancement feature is generated based on the first attention weight from the first modality to the second modality; the second modality enhancement feature is generated based on the second attention weight from the second modality to the first modality; The emotion recognition module is used to fuse the first modality enhancement features and the second modality enhancement features to obtain multimodal global fusion features; and to classify the multimodal global fusion features for emotion to obtain the emotion recognition result.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is loaded and executed by the processor, it employs the audiovisual multimodal emotion recognition method based on bidirectional cross-attention as described in any one of claims 1 to 8.

Images