Audio and video dual-mode emotion recognition method and system based on adapter fusion

Abstract

The invention relates to the technical field of artificial intelligence and emotion calculation, in particular to an audio and video dual-mode emotion recognition method and system based on adapter fusion. The method comprises the following steps: acquiring a video frame sequence and an audio signal, and preprocessing the video frame sequence and the audio signal; constructing an emotion recognition model; based on a bimodal feature extraction module, a space adapter and a global adapter are embedded in sequence, and corresponding modal enhanced space features and global features are obtained in sequence; generating intermediate representations of the corresponding modes based on the global features, and performing feature fusion according to the intermediate representations to obtain fusion features of the corresponding modes; the fusion features are spliced, time sequence features are extracted, and final features are obtained; inputting the final features into a classifier to obtain a predicted emotion category, training an emotion recognition model by adopting a loss function, and determining an optimal emotion recognition model; and inputting a to-be-recognized video frame sequence and an audio signal into the emotion recognition model, and outputting a recognition result.

Description

Technical Field

[0001] The present invention relates to the technical fields of artificial intelligence and affective computing, and particularly relates to an audio-visual bimodal emotion recognition method and system based on adapter fusion. Background Art

[0002] As an important technology for understanding and responding to human emotional states, emotion recognition is playing an increasingly crucial role in many practical application scenarios such as intelligent healthcare, online education, intelligent customer service, in-vehicle systems, and human-computer interaction; compared with static images or unimodal signals, audio-visual bimodal emotion recognition can capture the multi-dimensional signal characteristics of humans when expressing emotions more comprehensively, thereby significantly improving the accuracy and robustness of emotion recognition.

[0003] Traditional emotion recognition methods mostly rely on unimodal inputs, such as images, speech, or text, and have limitations in dealing with complex dynamic facial expressions; in practical applications, emotions are usually a comprehensive reflection of multiple dimensions and channels, so emotion recognition is usually based on multimodal signals, especially audio-visual data, to comprehensively reflect emotions.

[0004] Audio-visual bimodal fusion has information complementarity, can simultaneously capture the visual changes of facial expressions and the acoustic characteristics of speech, and is closer to the natural emotion expression method of humans; however, there are still certain limitations in the existing emotion recognition for audio-visual bimodality: firstly, the multimodal fusion strategy is not flexible, lacking fine-grained feature alignment and interaction modeling between modalities, and easily ignoring the fine-grained dependency relationships between different modalities; secondly, the number of model parameters is large and the training cost is high. Existing methods usually require joint training of the entire network and are difficult to deploy on devices with limited computing power; thirdly, the cross-modal temporal modeling ability is weak. Generally, the dynamic characteristics of different modalities evolving over time are not fully exploited, resulting in insufficient understanding ability for complex emotional states; fourthly, the transferability and generalization ability are limited. When facing the data distribution changes in the real environment, the model performance is likely to decline significantly. Summary of the Invention

[0005] In order to solve the technical problems of the existing modeling limitations in emotion recognition for audio-visual bimodality, which cannot be flexibly fused, have insufficient understanding ability for complex emotions, have a large amount of calculation, high cost, poor transferability and generalization, and affect the emotion recognition effect, the purpose of the present invention is to provide an audio-visual bimodal emotion recognition method based on adapter fusion, and the specific technical solutions adopted are as follows:

[0006] Obtain a video frame sequence and an audio signal, and perform preprocessing;

[0007] Construct an emotion recognition model, where the emotion recognition model includes a bimodal feature extraction module, a bimodal fusion adapter structure, a cross-modal temporal modeling module, and a classifier;

[0008] The dual-modal feature extraction module is sequentially embedded with a spatial adapter and a global adapter, and the preprocessed video frame sequence and audio signal are processed in sequence, and the spatial features and global features enhanced for the corresponding modality are obtained in sequence;

[0009] Based on the global features, intermediate representations for the corresponding modality are generated through a dual-modal fusion adapter structure, and feature fusion is performed according to the intermediate representations to obtain the fusion features for the corresponding modality respectively;

[0010] The fusion features are concatenated and input into a cross-modal temporal modeling module for temporal feature extraction to obtain the final features;

[0011] The final features are input into a classifier to obtain the predicted emotion categories, and a loss function is used to train the emotion recognition model, adjust the parameters of the emotion recognition model, and determine the optimal emotion recognition model;

[0012] The video frame sequence and audio signal to be recognized are obtained and input into the optimal emotion recognition model to output the recognition result.

[0013] Preferably, the video frame sequence and audio signal are obtained and preprocessed, including:

[0014] The video frame sequence is decoded frame by frame, the size of each video frame is unified, and normalization processing is performed; the audio track is extracted from the normalized video frame sequence, and log-Mel spectrogram calculation is performed to obtain the corresponding audio signal;

[0015] The input lengths of the preprocessed video frame sequence and audio signal are unified.

[0016] Preferably, the dual-modal feature extraction module is sequentially embedded with a spatial adapter and a global adapter, and the preprocessed video frame sequence and audio signal are processed in sequence, and the spatial features and global features enhanced for the corresponding modality are obtained in sequence, including:

[0017] The dual-modal feature extraction module uses a Transformer encoder as the feature extractor, and a spatial adapter and a global adapter are sequentially embedded after the Transformer encoder;

[0018] The preprocessed video frame sequence and audio signal are input into the Transformer encoder to obtain the output features for the corresponding modality respectively;

[0019] After normalization processing based on the output features, dimensionality reduction, non-linear activation, and dimensionality increase processing are sequentially performed using the spatial adapter to obtain the spatial features enhanced for the corresponding modality respectively;

[0020] After normalization based on spatial features, a global adapter is used for processing to obtain the enhanced global features corresponding to each modality respectively.

[0021] Preferably, the preprocessed video frame sequence and audio signal are input into a Transformer encoder to obtain the output features corresponding to each modality respectively, including:

[0022] Determine the corresponding input forms based on the preprocessed video frame sequence and audio signal, and the input forms are patch tokens and classification tokens; align each modality structure and input it into the Transformer encoder to obtain the output features corresponding to each modality, denoted as and , where represents the output feature corresponding to the video frame sequence; represents the output feature corresponding to the audio signal; represents the number of frames of the video frame sequence; 、 both represent the number of patches; represents the dimension of the output feature.

[0023] Preferably, after normalization based on the output features, a spatial adapter is used to perform dimensionality reduction, non-linear activation, and dimensionality increase processing in sequence to obtain the enhanced spatial features corresponding to each modality respectively, including:

[0024] Based on the normalized output features, use a spatial adapter to obtain the initial spatial features corresponding to each modality, and the corresponding calculation formula is:

[0025]

[0026] where represents the initial spatial feature corresponding to the video frame sequence; represents the initial spatial feature corresponding to the audio signal; represents the current Transformer encoder; represents the adjacent upper block of the Transformer encoder; represents the spatial adapter; represents the normalized output feature corresponding to the video frame sequence; represents the normalized output feature corresponding to the audio signal;

[0027] Through the spatial adapter, perform dimensionality reduction, non-linear activation, and dimensionality increase processing on the normalized output features in sequence, and combine with the output features to obtain the enhanced spatial features corresponding to each modality respectively, and the corresponding calculation formula is:

[0028]

[0029]

[0030] Among them, represents the enhanced spatial features corresponding to the video frame sequence; represents the enhanced spatial features corresponding to the audio signal; and respectively represent the output features corresponding to the video frame sequence and the audio signal; represents the MHSA operation in the bimodal feature extraction module.

[0031] Preferably, after performing normalization processing on the spatial features, a global adapter is used for processing to obtain the enhanced global features corresponding to each modality, including:

[0032] The initial global features corresponding to each modality are obtained by using the global adapter based on the normalized spatial features, and the corresponding calculation formula is:

[0033]

[0034] Among them, represents the initial global feature corresponding to the video frame sequence; represents the initial global feature corresponding to the audio signal; represents the global adapter; represents the normalized spatial feature corresponding to the video frame sequence; represents the normalized spatial feature corresponding to the audio signal;

[0035] The initial global features are enhanced to obtain the enhanced global features corresponding to each modality, and the corresponding calculation formula is:

[0036]

[0037]

[0038] Among them, represents the enhanced global feature corresponding to the video frame sequence; represents the enhanced global feature corresponding to the audio signal; represents the MLP operation in the bimodal feature extraction module.

[0039] Preferably, based on the global features, intermediate representations corresponding to each modality are generated through a bimodal fusion adapter structure, and feature fusion is performed according to the intermediate representations to obtain the fusion features corresponding to each modality, including:

[0040] The bimodal fusion adapter structure includes a two-branch adapter in the first stage and a two-branch adapter in the second stage;

[0041] Input the global features into the dual-branch adapter in the first stage to generate the intermediate representations corresponding to each modality. The corresponding calculation formula is:

[0042]

[0043]

[0044] where, and represent the intermediate representations corresponding to the video frame sequence and the audio signal respectively; and represent the enhanced global features corresponding to the video frame sequence and the audio signal respectively; and both represent the operations of the dual-branch adapter in the first stage; and both represent the learnable parameters of the gating mechanism;

[0045] Input the intermediate representations into the dual-branch adapter in the second stage for feature fusion, and obtain the fused features corresponding to each modality respectively. The corresponding calculation formula is:

[0046]

[0047]

[0048] where, and represent the fused features corresponding to the video frame sequence and the audio signal respectively; and both represent the operations of the dual-branch adapter in the second stage; and both represent the learnable parameters of the gating mechanism; represents replicating along the time axis for times, represents the number of frames of the video frame sequence, represents the dimension of the intermediate representation, represents the number of feature channels of the intermediate representation, which is used to match the size of the fused feature ; represents compressing into through average pooling operation.

[0049] Preferably, concatenate the fused features and input them into the cross-modal temporal modeling module to extract temporal features, and obtain the final features, including:

[0050] Collect the fused features after passing through L pairs of dual-modal fusion adapter structures, and denote them respectively as and For the spliced and fused features, the corresponding calculation formula is: where represents the splicing result; represents the splicing operation; 、 respectively represent the fused features of the audio signal and the video frame sequence corresponding to the L pairs of bimodal fusion adapter structures;

[0051] Input the splicing result into the transformer-encoder in the cross-modal temporal modeling module to extract temporal features, and the corresponding calculation formula is:

[0052]

[0053] where represents the final feature; [[ID=2**]]represents the operation of extracting temporal features; represents the positional encoding; represents the class encoding.

[0054] Preferably, input the final feature into the classifier to obtain the predicted emotion category, and use the loss function to train the emotion recognition model and adjust the parameters of the emotion recognition model to determine the optimal emotion recognition model, including:

[0055] Input the final feature into the classifier to obtain the predicted emotion category, and the corresponding calculation formula is:

[0056]

[0057] where represents the predicted emotion category; represents the classifier operation; represents the linear operation; represents the final feature; [[ID=**]]

[0058] Use the loss function to train the emotion recognition model, and the corresponding calculation formula is:

[0059]

[0060] where represents the loss function; represents the total number of emotion categories; represents the one-hot encoding that truly belongs to the th emotion category, ; represents the probability that the emotion recognition model predicts as the th emotion category, . It should be noted that there is a small error in the original text where "2**" is used in line 26 which should be corrected. Also, there is a "**" in line 52 which seems to be an error. The above translation is based on the corrected understanding of the text.

[0061] To solve the above technical problems, the present application further provides: an audio-visual bimodal emotion recognition system based on adapter fusion. The system stores program data, and when the program data is executed, it realizes the steps of an audio-visual bimodal emotion recognition method as described in any one of the foregoing.

[0062] The present invention has the following beneficial effects:

[0063] 1. Based on the collaborative work of each module in the emotion recognition model, the overall performance and practicality of the emotion recognition model are improved; a lightweight adapter module is adopted. On the basis of freezing the backbone network of the emotion recognition model, the parameters of the pre-trained backbone network are fixed so that they do not participate in backpropagation and update, reducing the training parameters; respectively realizing in-modal feature enhancement, inter-modal interaction fusion and cross-modal temporal modeling. That is, compared with the traditional splicing and average fusion strategies, based on the bimodal feature extraction module, a spatial adapter and a global adapter are embedded, the spatial features and global features are determined in turn, and then feature fusion is performed through the bimodal fusion adapter structure and then spliced. It can effectively retain the semantic complementary information between modalities, avoid redundant propagation, make the obtained fusion features more distinguishable, and the training parameter amount can be reduced by the adapter, making the emotion recognition model highly practical and energy-efficient in mobile devices and edge deployment; the introduced cross-modal temporal modeling module obtains the final features, making it show stronger emotion tracking ability during the emotion transition process to solve the defect that single-modal data ignores the time dimension; the overall trained emotion recognition model not only significantly reduces the training parameter amount and resource consumption of the model, but also improves the model's ability to recognize complex emotions in the natural environment, especially more discriminative for fine-grained categories.

[0064] 2. An audio-visual bimodal emotion recognition system based on adapter fusion provided by the present invention has the same beneficial effects as an audio-visual bimodal emotion recognition method provided by the present invention, which will not be elaborated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0065] In order to more clearly illustrate the technical solutions and advantages in the embodiments of the present invention or the prior art, the following will briefly introduce the drawings required to be used in the description of the embodiments or the prior art. Obviously, the following drawings are only some embodiments of the present invention. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0066] Figure 1 It is a schematic diagram of the overall framework of an emotion recognition model of an audio-visual bimodal emotion recognition method provided by an embodiment of the present invention;

[0067] Figure 2 A comparison schematic diagram of the performance of an audio-visual bimodal emotion recognition method based on adapter fusion provided by an embodiment of the present invention in different modalities;

[0068] Figure 3 A comparison of parameters and performance of an audio-visual bimodal emotion recognition method based on adapter fusion provided by an embodiment of the present invention on the DFEW dataset, where the bubble size corresponds to the model size;

[0069] Figure 4 A visualization diagram of t-SNE high-dimensional embedding features in the last fold (fd5) of the emotion recognition model of an audio-visual bimodal emotion recognition method based on adapter fusion provided by an embodiment of the present invention on the DFEW and MAFW datasets;

[0070] Figure 5 A visualization diagram of t-SNE high-dimensional embedding features in the last fold (fd5) of the emotion recognition model of an audio-visual bimodal emotion recognition method based on adapter fusion provided by an embodiment of the present invention on the MAFW dataset; Figure 6 A heat map of attention visualization of the emotion recognition model of an audio-visual bimodal emotion recognition method based on adapter fusion provided by an embodiment of the present invention on the DFEW dataset; Figure 7 A heat map of attention visualization of the emotion recognition model of an audio-visual bimodal emotion recognition method based on adapter fusion provided by an embodiment of the present invention on the MAFW dataset. Detailed implementation manners

[0071] In order to further elaborate on the technical means and effects adopted by the present invention to achieve the predetermined invention purpose, the following describes in detail the specific implementation manners, structures, features and effects of an audio-visual bimodal emotion recognition method and system based on adapter fusion proposed according to the present invention in conjunction with the accompanying drawings and preferred embodiments. In the following description, different "one embodiment" or "another embodiment" do not necessarily refer to the same embodiment. In addition, the specific features, structures or characteristics in one or more embodiments can be combined in any suitable form.

[0072] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the technical field to which the present invention belongs.

[0073] The following specifically describes the specific solutions of an audio-visual bimodal emotion recognition method and system based on adapter fusion provided by the present invention in conjunction with the accompanying drawings.

[0074] Please refer toFigure 1 , a method for audio - visual bimodal emotion recognition based on adapter fusion provided by the first embodiment of the present invention, the method comprising:

[0075] Step S1: Obtain a video frame sequence and an audio signal, and perform pre - processing;

[0076] Step S2: Construct an emotion recognition model, the emotion recognition model including a bimodal feature extraction module, a bimodal fusion adapter structure, a cross - modal temporal modeling module, and a classifier;

[0077] Step S3: Based on the bimodal feature extraction module, embed a spatial adapter and a global adapter in sequence, input the pre - processed video frame sequence and audio signal for processing in sequence, and obtain the spatially enhanced features and global features corresponding to the modalities after enhancement in sequence;

[0078] Step S4: Generate intermediate representations corresponding to the modalities through the bimodal fusion adapter structure based on the global features, perform feature fusion according to the intermediate representations, and obtain the fusion features corresponding to the modalities respectively;

[0079] Step S5: Concatenate the fusion features and input them into the cross - modal temporal modeling module to extract temporal features, obtaining the final features;

[0080] Step S6: Input the final features into the classifier to obtain the predicted emotion categories, and use a loss function to train the emotion recognition model, adjust the parameters of the emotion recognition model, and determine the optimal emotion recognition model;

[0081] Step S7: Obtain the video frame sequence and audio signal to be recognized, and input them into the optimal emotion recognition model to output the recognition result.

[0082] For better illustration, due to differences in multiple aspects such as data structure, temporal density, and semantic expression between audio and video, there is a natural heterogeneity between the audio and video modalities, making it difficult for simple concatenation or unified processing to capture the fine and dynamic emotion - related features between modalities; in addition, most existing emotion recognition methods rely on joint optimization of the model. Although the performance is acceptable, the training resource consumption is large, lacking flexibility and scalability, especially difficult to deploy in devices with limited computing power or scenarios with frequent task changes, that is, there are significant deficiencies in multi - modal feature fusion, cross - modal interaction modeling, and temporal information extraction.

[0083] Emotional states themselves are highly time-sequential and subjective, often showing subtle and dynamic change trends in natural interactions. Traditional recognition methods mostly limit their modeling in the time dimension to single-modal or flat fusion, ignoring the co-variation between audio and video modalities in time evolution. The way of making fusion among multi-modal data is not flexible enough, making it difficult to fully exploit the fine-grained semantic connections between audio and video modalities, resulting in low accuracy in recognizing complex emotional states. Especially in the aspect of differentiating fine-grained emotion categories, for example, in the case of easily confused categories such as anger, the existing recognition methods have limited discriminative ability and weak cross-modal time modeling ability, and cannot accurately capture the dynamic evolution of emotions during natural expression; moreover, due to data noise, background changes and the subjectivity of emotion labels in real environments, the model structure is huge and the parameters are redundant, further exacerbating the training difficulty and generalization risk of the model, resulting in high training costs, inconvenient deployment, unstable model performance, and being difficult to meet the engineering requirements of lightweight and efficient migration. In response to this, this application proposes an audio-visual bimodal model architecture, namely an emotion recognition model, which is flexible in structure, efficient in training, precise in fusion and applicable to dynamic emotion recognition scenarios, so as to improve the recognition accuracy, enhance the fusion efficiency and strengthen the robustness and transferability of the emotion recognition model.

[0084] As an alternative implementation, the emotion recognition model provided in this application can be integrated and applied with existing product forms such as intelligent terminals, wearable devices, and online platforms, providing effective emotion perception capabilities and decision-making support for multiple fields such as human-computer interaction, mental health, education analysis, recommendation systems, and public safety, and having good prospects for practical transformation and market application value; for example, deploying the emotion recognition model in an intelligent terminal or a medical assistance system to daily monitor the emotional state of users, especially applicable to emotion recognition and early warning of patients with mental illnesses, high-risk groups of depression, etc., and assisting doctors in mental health assessment and personalized intervention.

[0085] Further, in step S1, it includes:

[0086] Step S11: Decode the video frame sequence frame by frame, unify the size of each video frame, and perform normalization processing; extract the audio track from the normalized video frame sequence, and perform log-Mel spectrogram calculation to obtain the corresponding audio signal;

[0087] Step S12: Unify the input lengths of the preprocessed video frame sequence and the audio signal.

[0088] Optionally, the video frame sequence is the DFEW (Dynamic Facial Expression in the Wild) dataset. The data collection scenarios are from real scenarios such as movies, TV dramas, interviews, and online videos, with obvious facial expression actions, changes in language intonation, and background noise disturbances. It is an important benchmark for evaluating multi-modal emotion recognition algorithms in real-world situations. It contains thousands of video materials with clear emotion labels. Each video contains facial expressions and synchronized speech, and the labels are based on Ekman's six basic emotions and their extended definitions, covering multiple categories such as "happy", "sad", "angry", "surprised", "disgusted", "fearful", and "neutral".

[0089] The audio signal is the MAFW (Multi-Action Facial Expression in the Wild) dataset, which focuses on the fine-grained emotion expression that fuses expressions and speech, contains a higher proportion of micro-expressions and emotion transformation processes, and provides clear audio-visual time alignment annotations, facilitating the subsequent model to synchronously extract and analyze the temporal information between modalities, and helping to evaluate the model's perception ability of dynamic emotion transformation processes.

[0090] Specifically, for the DFEW dataset, the OpenCV (Open Source Computer Vision Library) and FFmpeg (Flexible Framework for Multimedia) toolchain are used for frame-by-frame decoding. In this embodiment, 16 key frames are uniformly intercepted from each video material as visual inputs, and the image corresponding to each key frame is adjusted to a unified size of 224×224 and normalized to the [0,1] interval. Then, the FFmpeg toolchain is used to extract the audio track from the video frame sequence constructed by the key frames, the sampling rate is set to 16 kHz, and the librosa library is used for log-Mel (Logarithmic Mel-Spectrogram) spectrogram calculation to determine the audio signal for subsequent analysis. Among them, the log-Mel spectrogram is a method of audio feature representation that captures the spectral characteristics of the audio signal by mapping the spectral energy to the Mel frequency scale.

[0091] And the determined video frame sequence and audio signal for analysis are divided into a training set, a validation set, and a test set according to a certain proportion, and five-fold cross-validation is used for evaluation to ensure the robustness and universality of the results. It should be noted that to avoid data leakage, the division of the training set, the validation set, and the test set strictly ensures that the same video segment does not appear in the training set and the test set at the same time.

[0092] It is explained that in step S2, an emotion recognition model is constructed. The emotion recognition model includes a bimodal feature extraction module, a bimodal fusion adapter structure, a cross-modal temporal modeling module, and a classifier. That is, this emotion recognition model fully combines the strong expression ability and the parameter efficiency of the adapter mechanism. By constructing corresponding modules, it respectively strengthens the unimodal representation, improves the cross-modal interaction modeling ability, and depicts the evolution path of emotions over time. Furthermore, a bimodal emotion recognition framework with high parameter efficiency, fine fusion, and flexible structure is constructed. By only fine-tuning a small number of inserted modules, the training resource requirements are significantly reduced, and the flexibility of model migration is improved.

[0093] Furthermore, in step S3, it includes:

[0094] Step S31: The bimodal feature extraction module uses a Transformer encoder as the feature extractor, and a spatial adapter and a global adapter are sequentially embedded after the Transformer encoder.

[0095] Optionally, in this embodiment, the bimodal feature extraction module (Modality-Specific Adapter, MSA) uses a two-stream pre-trained Vision Transformer (ViT) as the backbone network for audio and video bimodal processing. That is, a Transformer encoder is used as the feature extractor to process visual and auditory information respectively, and it can efficiently capture complex features in multi-modal data.

[0096] It can be understood that since there are significant heterogeneities in the structure, semantic level, and temporal density between the audio and video modalities, such as different speech rates and frame rates, and non-linear relationships between speech spectra and facial movements, etc., traditional emotion recognition models are difficult to adapt to the data characteristics of both modalities at the same time, resulting in unbalanced feature expression capabilities. Therefore, based on the idea of parameter-efficient fine-tuning in the field of NLP (Natural Language Processing), customized adapter structures, namely spatial adapters (S-Adapter) and global adapters (G-Adapter), are proposed to be inserted for audio and video respectively in each layer of the Transformer. By using lightweight modules to strengthen the spatial and global feature modeling capabilities within each modality, the intra-modal modeling ability and the inter-modal interaction expression ability are enhanced, thereby improving the representation robustness of the model on different modalities, improving the recognition accuracy of dynamic complex emotional states, and effectively improving the integrity of intra-modal feature expression.

[0097] It is explained that the spatial adapter S-Adapter is embedded in the multi-head self-attention (MHSA) stage to enhance local spatial features; the global adapter G-Adapter is embedded in the multi-layer perceptron (MLP) stage to capture global context information; that is, the local structure modeling ability is improved through the spatial adapter, and the cross-region semantic expression ability is enhanced by the global adapter. Combining the overall dual-modal feature extraction module MSA realizes the reinforcement of the corresponding modal features, enhances the consistency and integrity of the internal representation of the modality, so that both the audio and video dual-modalities can obtain richer and more stable feature representations corresponding to each other.

[0098] Step S32: Input the preprocessed video frame sequence and audio signal into the Transformer encoder to obtain the output features of the corresponding modality respectively.

[0099] Furthermore, in step S32, it includes:

[0100] Determine the corresponding input forms based on the preprocessed video frame sequence and audio signal. The input forms are patch tokens and classification tokens; align each modal structure and input it into the Transformer encoder to obtain the output features of the corresponding modality, denoted as and , where, represents the output features corresponding to the video frame sequence; represents the output features corresponding to the audio signal; represents the number of frames of the video frame sequence; , both represent the number of patches; represents the dimension of the output features.

[0101] It can be explained that in the foregoing processing steps, the video frame sequence and audio signal have been processed into fixed-length input segments. Then, the processing of the video frame sequence is converted into a spatial feature representation through frame sampling and Patch Embedding; the audio signal generates time-frequency features through the Mel spectrogram and is converted into Patch Tokens, that is, the feature representations of the two different modalities are defined as output features. Preferably, in this embodiment, the input forms are scattered patch tokens and classification tokens respectively. Based on the video frame sequence, each frame of image is divided into 16×16 patches, and the embedding dimension is 768; based on the audio signal, each log-Mel spectrogram is cut into 2D patches, and an embedding method equivalent to the visual path is adopted to keep the structure aligned.

[0102] Step S33: After performing normalization processing based on the output features, use a spatial adapter to perform dimensionality reduction, non-linear activation, and dimensionality increase processing in sequence, and respectively obtain the spatially enhanced features corresponding to each modality.

[0103] For better illustration, in this embodiment, take the Transformer block as an example for illustration, that is, the Transformer blocks in the video and audio bimodal branches form pairs; then determine the input in the pair of Transformer blocks is the output of the adjacent previous pair of Transformer blocks. Through step S32, the video frame sequence and the audio signal are respectively determined as and , where represents the number of frames in the video frame sequence; represents the dimension of the output features; represents the number of feature channels, which reflects the richness of feature information on a specific dimension and frame.

[0104] Further, in step S33, it includes:

[0105] Step S331: Use a spatial adapter based on the normalized output features to obtain the initial spatial features corresponding to each modality. The corresponding calculation formula is:

[0106]

[0107] where represents the initial spatial features corresponding to the video frame sequence; represents the initial spatial features corresponding to the audio signal; represents the current Transformer encoder; represents the previous adjacent Transformer encoder; represents the spatial adapter; represents the normalized output features corresponding to the video frame sequence; represents the normalized output features corresponding to the audio signal.

[0108] Step S332: Use the spatial adapter to perform dimensionality reduction, non-linear activation, and dimensionality increase processing on the normalized output features in sequence, and combine with the output features to respectively obtain the spatially enhanced features corresponding to each modality. The corresponding calculation formula is:

[0109]

[0110]

[0111] where Represents the enhanced spatial features corresponding to the video frame sequence; Represents the enhanced spatial features corresponding to the audio signal; 、 Respectively represent the output features corresponding to the video frame sequence and the audio signal; Represents the MHSA operation in the bimodal feature extraction module.

[0112] Specifically, in the attention (Multi-Head Self Attention, MHSA) stage, the normalized output features are applied to the spatial adapter S-Adapter in the MSA to obtain the initial spatial features of the corresponding modalities, that is, and ; Then, the spatial adapter S-Adapter performs dimensionality reduction - non-linear activation - dimensionality increase operations on the normalized features, that is, compresses the high-dimensional output features to a lower dimension, introduces non-linear characteristics through the non-linear activation function, and finally expands back to the original dimension, not only retaining the key information but also enhancing the expression ability of the output features; Then, it is connected with the output residual in the MHSA stage, making the information flow smoother, avoiding information loss, and further obtaining the enhanced spatial features of the corresponding modalities respectively.

[0113] Step S34: After performing normalization processing based on the spatial features, use the global adapter for processing to obtain the enhanced global features of the corresponding modalities respectively.

[0114] Furthermore, in step S34, it includes:

[0115] Step S341: Use the global adapter based on the normalized spatial features to obtain the initial global features of the corresponding modalities, and the corresponding calculation formula is:

[0116]

[0117] Among them, Represents the initial global feature corresponding to the video frame sequence; Represents the initial global feature corresponding to the audio signal; Represents the global adapter; Represents the normalized spatial feature corresponding to the video frame sequence; Represents the normalized spatial feature corresponding to the audio signal;

[0118] Step S342: Perform enhancement processing on the initial global features to obtain the enhanced global features of the corresponding modalities respectively, and the corresponding calculation formula is:

[0119]

[0120]

[0121] Among them, represents the enhanced global feature corresponding to the video frame sequence; represents the enhanced global feature corresponding to the audio signal; represents the MLP operation in the bimodal feature extraction module.

[0122] Specifically, in the MLP stage, the global adapter G-Adapter similarly processes the normalized spatial features and enhances them respectively to obtain the global features enhanced for the corresponding modalities.

[0123] Furthermore, in step S4, it includes:

[0124] Step S41: The bimodal fusion adapter structure includes a two-branch adapter in the first stage and a two-branch adapter in the second stage.

[0125] It can be understood that there are problems of semantic complementarity but large expression differences between audio and video. Simple splicing or shared attention mechanisms are prone to redundant cross-interference, so problems such as modal conflict, redundant propagation, and semantic shift are usually faced. Based on the concept of "first modal enhancement and then interactive fusion", a bimodal fusion adapter structure (Bimodal Fusion Adapter, BFA) is constructed, and a gating mechanism is introduced to dynamically adjust the information flow intensity between modalities. At the same time, token expansion and compression operations are used to achieve cross-modal feature alignment, allowing the emotion recognition model to automatically adjust the information flow intensity between modalities during training, enhancing semantic complementarity while retaining the modal ontology information, more effectively capturing the fine-grained emotion interaction features between speech and vision, and improving the cross-modal adaptation ability.

[0126] Explanation is made that based on the backbone network determined in the aforementioned step S31, a symmetric bimodal fusion adapter structure is designed in the middle layer of the Transformer to independently enhance audio and video respectively, and the gating mechanism is used to control the direction and intensity of cross-modal information flow, enabling close association between modalities at the feature level and effectively improving the accuracy of fusion expression; specifically, the bimodal fusion adapter structure BFA is used to enhance the intra-modal feature representation and achieve cross-modal information interaction. It is inserted after each pair of Transformer blocks and includes two stages of two-branch adapters. That is, in this embodiment, after the pair of Transformer blocks, a two-branch adapter in the first stage and a two-branch adapter in the second stage are inserted respectively. By dynamically controlling the opening degree of the feature channels between modalities and combining copy-aggregation operations, information coupling and decoupling between audio and video are achieved, significantly enhancing the cross-modal expression ability.

[0127] Step S42: Input the global features into the dual-branch adapter in the first stage to generate intermediate representations for corresponding modalities. The corresponding calculation formula is:

[0128]

[0129]

[0130] where, and represent the intermediate representations corresponding to the video frame sequence and the audio signal respectively; and represent the enhanced global features corresponding to the video frame sequence and the audio signal respectively; and both represent the operations of the dual-branch adapter in the first stage; and both represent the learnable parameters of the gating mechanism;

[0131] Step S43: Input the intermediate representations into the dual-branch adapter in the second stage for feature fusion to obtain the fusion features for corresponding modalities respectively. The corresponding calculation formula is:

[0132]

[0133]

[0134] where, and represent the fusion features corresponding to the video frame sequence and the audio signal respectively; and both represent the operations of the dual-branch adapter in the second stage; and both represent the learnable parameters of the gating mechanism; means replicating along the time axis times, represents the number of frames of the video frame sequence, represents the dimension of the intermediate representation, represents the number of feature channels of the intermediate representation, which is used to match the size of the fusion feature ; means compressing into through average pooling operation.

[0135] Specifically, the global features after the MLP stage are processed by the dual-branch adapter and the gating mechanism in the first stage to generate the intermediate representations for corresponding modalities, which are and ; Then, the intermediate representation is input into the two-branch adapter in the second stage for feature fusion. The gating mechanism is also introduced to obtain the fusion features corresponding to each modality, denoted as and .

[0136] It can be understood that emotions have the characteristic of dynamic evolution. Existing emotion recognition methods usually process the time dimension of audio or video separately, which easily ignores the coupled changes in time series between the two modalities and lacks effective time series processing capabilities. Therefore, a cross-modal time series modeling module is constructed. The fusion features corresponding to the modalities obtained above are fed into the Transformer time series encoder, and time series and modality information are injected through modality type embedding and time position embedding to achieve collaborative modeling. Then, the overall emotion evolution path is modeled by inputting into the Transformer structure, enabling the emotion recognition model to accurately track the emotion change trajectory and enhancing the model's perception ability of complex dynamic emotion change processes.

[0137] It should be noted that after the dual-modal fusion adapter structure, a cross-modal temporal modeling module (Cross-modal Temporal Modeling, CTM) is introduced to capture the temporal collaborative relationship between audio and video and enhance the modeling ability of the emotion recognition model for the real emotion evolution process. Among them, the cross-modal temporal modeling module CTM includes a transformer-encoder, which consists of multiple encoding layers. Each layer contains a self-attention sub-layer and a feed-forward neural network sub-layer, and the stability and expression ability of the model are improved through residual connection and layer normalization techniques.

[0138] Furthermore, in step S5, it includes:

[0139] Step S51: Collect the fusion features after passing through L pairs of dual-modal fusion adapter structures, which are respectively denoted as and . The fused features are concatenated, and the corresponding calculation formula is: , where represents the concatenation result; represents the concatenation operation; , respectively represent the fusion features corresponding to the audio signal and the video frame sequence after passing through L pairs of dual-modal fusion adapter structures;

[0140] Step S52: Input the concatenation result into the transformer-encoder in the cross-modal temporal modeling module to extract temporal features, and the corresponding calculation formula is:

[0141]

[0142] where Represents the final feature; Represents the operation of extracting temporal features; Represents positional encoding; Represents class encoding.

[0143] Specifically, the fused features after passing through L pairs of bimodal fusion adapter structures are collected. That is, after passing through L pairs of Transformer blocks equipped with bimodal feature extraction modules MSA and bimodal fusion adapter structures BFA, they are input into the cross-modal temporal modeling module to obtain fused features that capture the temporal dependencies and interactions between audio and video, and after splicing, they are input into the transformer-encoder for the extraction of temporal features.

[0144] Preferably, in this embodiment, a linear classifier is used for training, that is, the final feature is mapped onto a straight line to achieve the discrimination of different emotion categories.

[0145] Furthermore, in step S6, it includes:

[0146] Step S61: Input the final feature into the classifier to obtain the predicted emotion category, and the corresponding calculation formula is:

[0147]

[0148] where represents the predicted emotion category; represents the classifier operation; represents a linear operation; represents the final feature;

[0149] Step S62: Use the loss function to train the emotion recognition model, and the corresponding calculation formula is:

[0150]

[0151] where represents the loss function; represents the total number of emotion categories; represents being truly in the th class of emotion category's one-hot encoding, ; represents the probability that the emotion recognition model predicts as the th class of emotion category, .

[0152] Make an explanation, represents a linear operation, which represents a fully connected layer; [[ID=6

[0153] Specifically, during the training process of the emotion recognition model, a unified training process and standardized parameter settings are adopted to ensure the comparability of the performance of the emotion recognition model and the reproducibility of the experiment; in this embodiment, a single sampling method is adopted for each video frame sequence, and the same number of key frame images are extracted from each video frame sequence, that is , each key frame is adjusted to a unified size, that is , to adapt to the input requirements of the Transformer (ViT) module in the backbone network of the emotion recognition model. The optimizer uses AdamW (Adam Weight Decay Regularization, that is, Adaptive Moment Estimation Weight Decay Regularization), which has good generalization ability and regularization control effect. Based on the video frame sequence and audio signal, corresponding initial learning rates are set, that is, set to on the DFEW dataset, and set to on the MAFW dataset, and the cosine annealing strategy is used to gradually decay to 0 within 25 training epochs, that is, the learning rate is adjusted through a smooth cosine curve to improve the convergence stability. It can not only avoid the training stagnation caused by the premature decrease of the learning rate, but also prevent the slow training speed caused by too low learning rate, ensuring that the emotion recognition model maintains the best learning efficiency and stability during the training process; and the training batch size is uniformly set to 8 to ensure that the amount of data processed in each iteration is appropriate, which helps to balance the training speed and performance; the weight decay coefficient is set to to prevent overfitting; the remaining optimizer hyperparameters, that is, learning rate, momentum, etc. adopt the default configuration of PyTorch to simplify the parameter tuning process.

[0154] It should be noted that during the training stage of the emotion recognition model, only the parameters of the specific modality adaptation, bimodal fusion adapter structure, cross-modal temporal modeling module, and classifier are updated and adjusted, that is, the model parameters of the backbone network in the bimodal feature extraction module are kept frozen, greatly reducing the parameter overhead to reduce the training cost and avoid catastrophic forgetting, and improving the application feasibility and transfer generalization ability of the emotion recognition model in resource-constrained devices.

[0155] As an alternative implementation, each module for constructing the emotion recognition model can be optimized or replaced according to the actual situation. For example, the adapter can be set to explore task-adaptive structure search or to be a conditionally activated dynamic adapter to achieve stronger generalization and flexibility; in the dual-modal fusion adapter structure, in addition to introducing the gating mechanism, multi-scale attention fusion, multi-path interaction attention mechanism, integrated fusion strategy, etc. can also be introduced to enhance the modeling ability of the semantic boundary and coupling strength between modalities; currently, the Transformer structure is used to model the unified time series, and in the future, it can be replaced with models such as causal convolution, structured RNN (Recurrent Neural Networks), or graph time series network to improve the balance of short-term and / or long-term dynamic modeling; in real scenarios, there may be modality missing or asynchronous phenomena, and by introducing modality compensation or modality adversarial discrimination mechanisms, the overall model robustness can be improved.

[0156] To make an explanation, to interpret the attention distribution within the adapter and the cross-modal information flow path, Grad-CAM (Gradient-weighted Class Activation Mapping) and multi-modal visualization analysis methods can be combined, which is helpful for task tuning and verification by domain experts; in addition, to address the instability caused by few samples or label subjectivity, emotion feature-preserving data augmentation methods can be introduced, such as feature perturbation, cross-modal generation, etc., and even combined with adversarial learning for style transfer-based sample expansion to enhance the generalization ability of the model.

[0157] Please refer to Figure 2 , which makes an explanation. It conducts a performance comparison of the emotion recognition model proposed in this application on the DFEW and MAFW datasets for the dual-modal of only video, only audio, and combined audio and video respectively to verify the recognition rate for emotion categories. Among them, the performance metrics are the unweighted average recall (UAR, i.e., Unweighted Average Recall) and the weighted average recall (WAR, i.e., Weighted Average Recall). UAR represents the average value of the recall rates for all emotion categories, without considering the differences in the number of samples in each emotion category; WAR is weighted and averaged according to the number of samples in each emotion category, making the emotion categories with more samples have a greater impact on the final result; it can be shown that the recognition rates based on the dual-modal for emotion recognition are all relatively high, and the model performance is good.

[0158] Please refer to Figure 3, for better illustration, in this embodiment, the dynamic emotion recognition datasets DFEW and MAFW in two real scenarios are verified. The experiment adopts the standard five-fold cross-validation scheme. The emotion recognition model constructed in this application, denoted as ABVnet, is compared with existing mainstream methods to evaluate the performance of the emotion recognition model constructed in this application in the multi-class emotion recognition task. Among them, the performance metrics are the average unweighted recall rate UAR and the weighted recall rate WAR; and then Table 1 is obtained, which is the performance comparison between the emotion recognition method proposed in this application and the mainstream methods on the DFEW and MAFW datasets.

[0159] Table 1 Performance Comparison Table of Emotion Recognition Methods and Mainstream Methods on DFEW and MAFW Datasets

[0160] It can be shown that on the DFEW dataset, the average unweighted recall rate UAR of the emotion recognition model ABVnet in this application reaches 67.52%, and the weighted recall rate WAR reaches 78.28%, ranking first among all comparison methods; at the same time, the UAR of the lightweight emotion recognition model ABVnet-S with fewer parameters reaches 67.23% and the WAR reaches 78.13%, showing good accuracy and model efficiency; on the MAFW dataset, the UAR of the emotion recognition model ABVnet reaches 44.46% and the WAR reaches 58.41%, ranking second, further verifying the stability and robustness of the emotion recognition model under different data conditions of audio and video.

[0161] Please refer to Figures 4 - 7 , in which, the t-SNE (t-distributed Stochastic Neighbor Embedding) high-dimensional embedding feature visualization diagrams of some existing emotion recognition methods and the emotion recognition model provided in this application on the DFEW and MAFW datasets are provided; it can be shown that the emotion recognition model can effectively and clearly divide the emotion categories, especially the boundaries between similar emotions such as "anger" and "sadness", "neutral" and "boring" are more distinct, indicating that the emotion recognition model is effective in extracting discriminative features; and through the attention heat map display, it shows that the emotion recognition model dynamically focuses on the corners of the mouth, the eye region and the speech emotion intensity paragraphs in the multi-frame sequence, showing that the emotion recognition model has good perception ability for the key regions and changing rhythms of emotions.

[0162] In addition, in the specific emotion categories based on the DFEW dataset, the emotion recognition model provided in this application is compared with existing methods for emotion category recognition, and then Table 2 is obtained. It is a performance comparison table of the emotion recognition method proposed in this application and mainstream methods for emotion category recognition based on the DFEW dataset. Among them, Ha (Happiness) represents happiness; Fe (Fear) represents fear; Sa (Sadness) represents sadness; Ne (Neutral) represents neutral; An (Anger) represents anger; Su (Surprise) represents surprise; Di (Disgust) represents disgust. It can be stated that in Figure 6 are the visualization heatmaps of "fear" and "surprise" respectively; in Figure 7 are the visualization heatmaps of "fear" and "anxiety" respectively.

[0163] Table 2 Performance comparison table of emotion recognition methods and mainstream methods for emotion category recognition based on the DFEW dataset

[0164] It can be stated that the classification accuracies in emotion categories such as "happiness" and "fear" reach 96.73% and 55.89% respectively, both higher than existing methods, indicating that the emotion recognition model provided in this application also has advantages in recognizing marginal emotions.

[0165] It can be understood that based on the collaborative work of each module in the emotion recognition model, the overall performance and practicality of the emotion recognition model are improved; a lightweight adapter module is adopted. On the basis of freezing the backbone network of the emotion recognition model, the parameters of the pre-trained backbone network are fixed so that they do not participate in backpropagation and update, reducing the training parameters; modal-internal feature enhancement, inter-modal interaction fusion, and cross-modal temporal modeling are respectively implemented. That is, compared with traditional splicing and average fusion strategies, based on the dual-modal feature extraction module, a spatial adapter and a global adapter are embedded. The spatial features and global features are determined in sequence, and then after feature fusion through the dual-modal fusion adapter structure and splicing, it can effectively retain the semantic complementary information between modalities, avoid redundant propagation, making the obtained fusion features more distinguishable, and the training parameter amount can be reduced through the adapter, making the emotion recognition model highly practical and energy-efficient in mobile device and edge deployment; the introduced cross-modal temporal modeling module obtains the final features, making it show stronger emotion tracking ability during the emotion transition process to solve the defect that single-modal data ignores the time dimension; the overall trained emotion recognition model not only significantly reduces the training parameter amount and resource consumption of the model, but also improves the model's ability to recognize complex emotions in the natural environment, especially more discriminative in fine-grained categories.

[0166] The second embodiment of the present invention proposes an audio-visual bimodal emotion recognition system based on adapter fusion. The system stores program data, and when the program data is executed, it implements the steps of an audio-visual bimodal emotion recognition method based on adapter fusion as described in any of the foregoing embodiments; this system has the same beneficial effects as the foregoing provided audio-visual bimodal emotion recognition method based on adapter fusion, and will not be elaborated here.

[0167] It can be understood that when the modules of an audio-visual bimodal emotion recognition system based on adapter fusion are operating, they need to utilize the audio-visual bimodal emotion recognition method based on adapter fusion provided in the foregoing embodiments. Therefore, whether the method and the program data are integrated or different hardware is configured to produce functions similar to the effects achieved by the present invention, they all fall within the protection scope of the present invention.

[0168] It should be noted that the above sequence of embodiments of the present invention is only for description and does not represent the superiority or inferiority of the embodiments. The processes depicted in the drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0169] Each embodiment in this specification is described in a progressive manner. The same or similar parts among the embodiments can be referred to each other, and the key points of each embodiment are the differences from other embodiments.

Claims

1. An audio-visual bimodal emotion recognition method based on adapter fusion, characterized in that The method includes: Obtaining a video frame sequence and an audio signal, and performing preprocessing; Constructing an emotion recognition model, where the emotion recognition model includes a bimodal feature extraction module, a bimodal fusion adapter structure, a cross-modal temporal modeling module, and a classifier; Based on the bimodal feature extraction module, embedding a spatial adapter and a global adapter in sequence, inputting the preprocessed video frame sequence and audio signal for processing in sequence, and obtaining the spatially enhanced features and globally enhanced features corresponding to each modality in sequence; Generating intermediate representations corresponding to each modality based on the global features through the bimodal fusion adapter structure, performing feature fusion according to the intermediate representations, and respectively obtaining the fusion features corresponding to each modality; Concatenating the fusion features and inputting them into the cross-modal temporal modeling module to extract temporal features, obtaining the final features; Inputting the final features into the classifier to obtain the predicted emotion category, and training the emotion recognition model using a loss function, adjusting the parameters of the emotion recognition model, and determining the optimal emotion recognition model; Obtaining the video frame sequence and audio signal to be recognized, and inputting them into the optimal emotion recognition model to output the recognition result.

2. The audio - video bimodal emotion recognition method based on adapter fusion according to claim 1, characterized in that, Obtaining a video frame sequence and an audio signal, and performing preprocessing, including: Performing frame-by-frame decoding on the video frame sequence, unifying the size of each video frame, and performing normalization processing; extracting the audio track from the normalized video frame sequence, performing log-Mel spectrogram calculation, and obtaining the corresponding audio signal; Unifying the input lengths of the preprocessed video frame sequence and audio signal.

3. A method for audio-visual bimodal emotion recognition based on adapter fusion according to claim 1, characterized in that Based on the bimodal feature extraction module, embedding a spatial adapter and a global adapter in sequence, inputting the preprocessed video frame sequence and audio signal for processing in sequence, and obtaining the spatially enhanced features and globally enhanced features corresponding to each modality in sequence, including: The bimodal feature extraction module uses a Transformer encoder as the feature extractor, and embeds a spatial adapter and a global adapter in sequence after the Transformer encoder; Inputting the preprocessed video frame sequence and audio signal into the Transformer encoder to respectively obtain the output features corresponding to each modality; After performing normalization processing on the output features, using the spatial adapter to perform dimensionality reduction, non-linear activation, and dimensionality increase processing in sequence to respectively obtain the spatially enhanced features corresponding to each modality; After performing normalization processing on the spatial features, using the global adapter for processing to respectively obtain the globally enhanced features corresponding to each modality.

4. The method for audio-visual bimodal emotion recognition based on adapter fusion according to claim 3, wherein Inputting the preprocessed video frame sequence and audio signal into the Transformer encoder to respectively obtain the output features corresponding to each modality, including: Determine the corresponding input form based on the preprocessed video frame sequence and audio signal, where the input form is patch tokens and classification tokens; align each modal structure and input it into the Transformer encoder to obtain the output features of the corresponding modality, denoted as and , where represents the output features corresponding to the video frame sequence; represents the output features corresponding to the audio signal; represents the number of frames of the video frame sequence; and both represent the number of patches; represents the dimension of the output features.

5. A method for audio-visual bimodal emotion recognition based on adapter fusion according to claim 4, characterized in that After performing normalization processing on the output features, using the spatial adapter to perform dimensionality reduction, non-linear activation, and dimensionality increase processing in sequence to respectively obtain the spatially enhanced features corresponding to each modality, including: Obtaining the initial spatial features corresponding to each modality using the spatial adapter based on the normalized output features, and the corresponding calculation formula is: ； Among them, represents the initial spatial features corresponding to the video frame sequence; represents the initial spatial features corresponding to the audio signal; represents the current Transformer encoder; represents the adjacent previous block of the Transformer encoder; represents the spatial adapter; represents the output features after normalization corresponding to the video frame sequence; represents the output features after normalization corresponding to the audio signal; Performing dimensionality reduction, non-linear activation, and dimensionality increase processing on the normalized output features through the spatial adapter in sequence, and combining with the output features to respectively obtain the spatially enhanced features corresponding to each modality, and the corresponding calculation formula is: ； ； Among them, represents the enhanced spatial features corresponding to the video frame sequence; represents the enhanced spatial features corresponding to the audio signal; , respectively represent the output features corresponding to the video frame sequence and the audio signal; represents the MHSA operation in the bimodal feature extraction module.

6. The method for audio-visual bimodal emotion recognition based on adapter fusion according to claim 5, characterized in that After performing normalization processing based on spatial features, a global adapter is used for processing to obtain the enhanced global features corresponding to each modality, including: Based on the normalized spatial features, the initial global features corresponding to each modality are obtained using a global adapter, and the corresponding calculation formula is: ； Among them, represents the initial global feature corresponding to the video frame sequence; represents the initial global feature corresponding to the audio signal; represents the global adapter; represents the normalized spatial feature corresponding to the video frame sequence; represents the normalized spatial feature corresponding to the audio signal; The initial global features are enhanced to obtain the enhanced global features corresponding to each modality, and the corresponding calculation formula is: ； ； Among them, represents the enhanced global feature corresponding to the video frame sequence; represents the enhanced global feature corresponding to the audio signal; represents the MLP operation in the bimodal feature extraction module.

7. A method for audio-visual bimodal emotion recognition based on adapter fusion according to claim 1, characterized in that, Based on the global features, the intermediate representations corresponding to each modality are generated through a bimodal fusion adapter structure, and feature fusion is performed according to the intermediate representations to obtain the fusion features corresponding to each modality, including: The bimodal fusion adapter structure includes a two-branch adapter in the first stage and a two-branch adapter in the second stage; The global features are input into the two-branch adapter in the first stage to generate the intermediate representations corresponding to each modality, and the corresponding calculation formula is: ； ； Among them, and respectively represent the intermediate representations corresponding to the video frame sequence and the audio signal; and respectively represent the enhanced global features corresponding to the video frame sequence and the audio signal; and both represent the dual-branch adapter operations in the first stage; and both represent the learnable parameters of the gating mechanism; The intermediate representations are input into the two-branch adapter in the second stage for feature fusion to obtain the fusion features corresponding to each modality, and the corresponding calculation formula is: ； ； Among them, , respectively represent the fusion features corresponding to the video frame sequence and the audio signal; , both represent the double-branch adapter operations in the second stage; , both represent the learnable parameters of the gating mechanism; represents replicating along the time axis for times, represents the number of frames of the video frame sequence, represents the dimension of the intermediate representation, represents the number of feature channels of the intermediate representation, used to match the size of the fusion feature ; represents compressing into through an average pooling operation.

8. A method for audio-visual bimodal emotion recognition based on adapter fusion according to claim 1, characterized in that, The fused features are concatenated and input into a cross-modal temporal modeling module for extracting temporal features to obtain the final features, including: Collect the fusion features after passing through L pairs of bimodal fusion adapter structures, and respectively denote them as and . Concatenate the fusion features, and the corresponding calculation formula is: where represents the concatenation result; represents the concatenation operation; and respectively represent the fusion features corresponding to the audio signal and the video frame sequence after passing through L pairs of bimodal fusion adapter structures; The concatenated result is input into the transformer-encoder in the cross-modal temporal modeling module for extracting temporal features to obtain the final features, and the corresponding calculation formula is: ； Among them, represents the final feature; represents the operation of extracting temporal features; represents the positional encoding; represents the class encoding.

9. A method for audio-visual bimodal emotion recognition based on adapter fusion according to claim 1, characterized in that The final features are input into a classifier to obtain the predicted emotion categories, and a loss function is used to train the emotion recognition model and adjust the parameters of the emotion recognition model to determine the optimal emotion recognition model, including: The final features are input into a classifier to obtain the predicted emotion categories, and the corresponding calculation formula is: ； Among them, represents the predicted emotion category; represents the classifier operation; represents the linear operation; represents the final feature; A loss function is used to train the emotion recognition model, and the corresponding calculation formula is: ； Among them, represents the loss function; represents the total number of emotion categories; represents the one - hot encoding that truly belongs to the th emotion category, ; represents the probability that the emotion recognition model predicts as the th emotion category, .

10. An audio-visual bimodal emotion recognition system based on adapter fusion, characterized in that, The system stores program data, and when the program data is executed, the steps of a method for audio-visual bimodal emotion recognition based on adapter fusion as described in any one of claims 1-9 are implemented.

Images