An audio and video dual-modal emotion recognition method and system based on adapter fusion

Abstract

The present invention relates to the field of artificial intelligence and emotional computing technology, and in particular to an audio and video dual-modal emotion recognition method and system based on adapter fusion; the method comprises: obtaining a video frame sequence and an audio signal, and performing preprocessing; constructing an emotion recognition model; embedding a spatial adapter and a global adapter in sequence based on a dual-modal feature extraction module, and obtaining spatial features and global features after enhancement of the corresponding modalities in sequence; generating an intermediate representation of the corresponding modality based on the global features, performing feature fusion according to the intermediate representation, and obtaining fusion features of the corresponding modalities respectively; splicing the fused features, extracting temporal features, and obtaining final features; inputting the final features into a classifier to obtain a predicted emotion category, training the emotion recognition model using a loss function, and determining the optimal emotion recognition model; inputting the video frame sequence and audio signal to be recognized into the emotion recognition model, and outputting a recognition result.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence and emotion computing technology, and in particular to an audio and video dual-modal emotion recognition method and system based on adapter fusion. Background Art

[0002] Emotion recognition, as an important technology for understanding and responding to human emotional states, is playing an increasingly critical role in many practical application scenarios such as smart healthcare, online education, intelligent customer service, in-vehicle systems, and human-computer interaction. Compared with static images or single-modal signals, audio and video dual-modal emotion recognition can more comprehensively capture the multi-dimensional signal characteristics of humans when expressing emotions, thereby significantly improving the accuracy and robustness of emotion recognition.

[0003] Traditional emotion recognition methods mostly rely on single-modal inputs, such as images, voice or text, and have limitations when processing complex dynamic facial expressions. In practical applications, emotions are usually a comprehensive reflection of multiple dimensions and channels. Therefore, in order to fully reflect emotions, emotion recognition is usually based on multimodal signals, especially audio and video data.

[0004] The fusion of audio and video dual modalities has information complementarity and can simultaneously capture the visual changes in facial expressions and the acoustic characteristics of speech, which is closer to the natural way humans express emotions. However, the existing emotion recognition for audio and video dual modalities still has certain limitations: First, the multimodal fusion strategy is inflexible and lacks fine-grained inter-modal feature alignment and interaction modeling, which easily ignores the fine-grained dependencies between different modalities. Second, the model parameters are large and the training cost is high. Existing methods usually require the joint training of the entire network, which is difficult to deploy on devices with limited computing power. Third, the cross-modal time series modeling capability is weak. In general, it fails to fully explore the dynamic characteristics of different modalities evolving over time, resulting in insufficient understanding of complex emotional states. Fourth, the transferability and generalization capabilities are limited. When facing changes in data distribution in real environments, the model performance is prone to significant decline. Summary of the Invention

[0005] In order to solve the technical problems of the existing modeling limitations of emotion recognition for audio and video dual modalities, such as the inability to flexibly integrate, insufficient ability to understand complex emotions, large computational complexity, high cost, poor transferability and generalization, which affect the effect of emotion recognition, the purpose of the present invention is to provide an audio and video dual modal emotion recognition method based on adapter fusion. The technical solutions adopted are as follows:

[0006] Obtain video frame sequences and audio signals and perform preprocessing;

[0007] Constructing an emotion recognition model, wherein the emotion recognition model includes a bimodal feature extraction module, a bimodal fusion adapter structure, a cross-modal temporal modeling module, and a classifier;

[0008] Based on the dual-modal feature extraction module, the spatial adapter and the global adapter are embedded in sequence, and the pre-processed video frame sequence and audio signal are input and processed in sequence, and the spatial features and global features after corresponding modality enhancement are obtained in sequence;

[0009] Based on the global features, the intermediate representation of the corresponding modality is generated through the dual-modal fusion adapter structure, and feature fusion is performed according to the intermediate representation to obtain the fusion features of the corresponding modality respectively;

[0010] The fusion features are spliced ​​and input into the cross-modal time series modeling module to extract time series features and obtain the final features;

[0011] The final features are input into the classifier to obtain the predicted emotion category, and the emotion recognition model is trained using the loss function to adjust the emotion recognition model parameters and determine the optimal emotion recognition model;

[0012] Obtain the video frame sequence and audio signal to be recognized, input them into the optimal emotion recognition model, and output the recognition results.

[0013] Preferably, obtaining a video frame sequence and an audio signal and performing preprocessing include:

[0014] Decode the video frame sequence frame by frame, unify the size of each frame, and normalize it; extract the audio track from the normalized video frame sequence, calculate the log-Mel spectrogram, and obtain the corresponding audio signal;

[0015] Unify the input length of preprocessed video frame sequences and audio signals.

[0016] Preferably, based on the dual-modal feature extraction module, the spatial adapter and the global adapter are sequentially embedded, the pre-processed video frame sequence and the audio signal are input and processed in sequence, and the spatial features and global features after the corresponding modality enhancement are obtained in sequence, including:

[0017] The bimodal feature extraction module adopts the Transformer encoder as the feature extractor, and embeds the spatial adapter and the global adapter in sequence after the Transformer encoder;

[0018] The preprocessed video frame sequence and audio signal are input into the Transformer encoder to obtain the output features of the corresponding modalities respectively;

[0019] After normalization based on the output features, the spatial adapter is used to perform dimensionality reduction, nonlinear activation and dimensionality increase in sequence to obtain the spatial features after corresponding modality enhancement.

[0020] After normalization based on spatial features, global adapter is used for processing to obtain the global features after corresponding modality enhancement.

[0021] Preferably, the preprocessed video frame sequence and audio signal are input into the Transformer encoder to obtain the output features of the corresponding modalities, including:

[0022] Based on the pre-processed video frame sequence and audio signal, the corresponding input form is determined, which is the patch token and classification token; each modal structure is aligned and input into the Transformer encoder to obtain the output features of the corresponding modality, which are recorded as and ,in, Represents the output features corresponding to the video frame sequence; Represents the output features corresponding to the audio signal; Indicates the frame number of the video frame sequence; 、 Both represent the number of patches; Indicates the dimension of the output features.

[0023] Preferably, after normalization processing is performed based on the output features, a spatial adapter is used to sequentially perform dimensionality reduction, nonlinear activation, and dimensionality increase processing to obtain spatial features after corresponding modality enhancement, including:

[0024] Based on the normalized output features, the spatial adapter is used to obtain the initial spatial features of the corresponding modality. The corresponding calculation formula is:

[0025]

[0026] in, Represents the initial spatial features corresponding to the video frame sequence; Represents the initial spatial features corresponding to the audio signal; Represents the current Transformer encoder; Represents the adjacent previous Transformer encoder; Represents a spatial adapter; Represents the normalized output features corresponding to the video frame sequence; Represents the normalized output features corresponding to the audio signal;

[0027] The spatial adapter is used to perform dimensionality reduction, nonlinear activation, and dimensionality increase on the normalized output features in sequence, and the output features are combined to obtain the spatial features after corresponding modal enhancement. The corresponding calculation formula is:

[0028]

[0029]

[0030] in, Represents the enhanced spatial features corresponding to the video frame sequence; Represents the enhanced spatial features corresponding to the audio signal; 、 Represent the output features corresponding to the video frame sequence and audio signal respectively; Represents the MHSA operation in the bimodal feature extraction module.

[0031] Preferably, after normalization processing is performed based on spatial features, a global adapter is used for processing to obtain global features after corresponding modality enhancement, including:

[0032] Based on the normalized spatial features, the global adapter is used to obtain the initial global features of the corresponding modality. The corresponding calculation formula is:

[0033]

[0034] in, Represents the initial global features corresponding to the video frame sequence; Represents the initial global features corresponding to the audio signal; Represents a global adapter; Represents the normalized spatial features corresponding to the video frame sequence; Represents the normalized spatial features corresponding to the audio signal;

[0035] The initial global features are enhanced to obtain the enhanced global features of the corresponding modes. The corresponding calculation formula is:

[0036]

[0037]

[0038] in, Represents the enhanced global features corresponding to the video frame sequence; Represents the enhanced global features corresponding to the audio signal; Represents the MLP operation in the bimodal feature extraction module.

[0039] Preferably, an intermediate representation of the corresponding modality is generated through a dual-modal fusion adapter structure based on the global features, and feature fusion is performed according to the intermediate representation to obtain fusion features of the corresponding modality, including:

[0040] The dual-modality fusion adapter structure includes a first-stage dual-branch adapter and a second-stage dual-branch adapter;

[0041] The global features are input into the first-stage dual-branch adapter to generate the intermediate representation of the corresponding modality. The corresponding calculation formula is:

[0042]

[0043]

[0044] in, 、 Represent the intermediate representations corresponding to the video frame sequence and audio signal respectively; 、 Represent the enhanced global features corresponding to the video frame sequence and audio signal respectively; 、 Both represent the first stage of dual-branch adapter operation; 、 Both represent the learnable parameters of the gating mechanism;

[0045] The intermediate representation is input into the second-stage dual-branch adapter for feature fusion to obtain the fusion features of the corresponding modalities. The corresponding calculation formula is:

[0046]

[0047]

[0048] in, 、 Represent the fusion features corresponding to the video frame sequence and audio signal respectively; 、 Both represent the second-stage dual-branch adapter operation; 、 Both represent the learnable parameters of the gating mechanism; Indicates that along the time axis copy Second-rate, Indicates the number of frames in the video frame sequence, represents the dimension of the intermediate representation, Indicates the number of feature channels in the intermediate representation, used to match the fusion features size; Indicates that the average pooling operation Compressed into .

[0049] Preferably, the fusion features are spliced ​​and input into the cross-modal time series modeling module to extract time series features to obtain the final features, including:

[0050] The fusion features after collecting L pairs of dual-modal fusion adapter structures are recorded as and , splicing fusion features, the corresponding calculation formula is: ,in, Indicates the splicing result; Represents a splicing operation; 、 Respectively represent the fusion features of the audio signal and video frame sequence after L pairs of dual-modal fusion adapter structures;

[0051] The concatenated results are input into the transformer-encoder in the cross-modal time series modeling module to extract time series features and obtain the final features. The corresponding calculation formula is:

[0052]

[0053] in, Indicates the final feature; Represents the operation of extracting time series features; Indicates positional encoding; Indicates category code.

[0054] Preferably, the final features are input into a classifier to obtain a predicted emotion category, and a loss function is used to train the emotion recognition model, adjust the emotion recognition model parameters, and determine the optimal emotion recognition model, including:

[0055] The final features are input into the classifier to obtain the predicted emotion category. The corresponding calculation formula is:

[0056]

[0057] in, represents the predicted emotion category; Represents a classifier operation; Represents linear operations; Indicates the final feature;

[0058] The loss function is used to train the emotion recognition model. The corresponding calculation formula is:

[0059]

[0060] in, represents the loss function; Indicates the total number of emotion categories; Indicates that the real One-hot encoding of emotion-like categories, ; Indicates that the emotion recognition model predicts The probability of the emotion class, .

[0061] In order to solve the above technical problems, the present application also provides: an audio and video dual-modal emotion recognition system based on adapter fusion, the system stores program data, and when the program data is executed, it implements the steps of an audio and video dual-modal emotion recognition method based on adapter fusion as described in any of the above items.

[0062] The present invention has the following beneficial effects:

[0063] 1. Based on the collaborative work of various modules in the emotion recognition model, the overall performance and practicality of the emotion recognition model are improved; a lightweight adapter module is used to freeze the backbone network of the emotion recognition model and fix the parameters of the pre-trained backbone network so that it does not participate in back propagation and update, thereby reducing training parameters; intra-modal feature enhancement, inter-modal interaction fusion and cross-modal time series modeling are realized respectively, that is, compared with the traditional splicing and average fusion strategies, the spatial adapter and the global adapter are embedded based on the dual-modal feature extraction module, and the spatial features and the global features are determined in turn. Then, the features are fused and spliced ​​through the dual-modal fusion adapter structure, which can effectively The semantic complementary information between modalities is retained to avoid redundant propagation, making the obtained fusion features more recognizable, and the adapter can reduce the amount of training parameters, making the emotion recognition model highly practical and energy-efficient in mobile devices and edge deployments; the introduced cross-modal time series modeling module obtains the final features, enabling it to exhibit stronger emotion tracking capabilities during the emotion transition process, in order to solve the defect of single-modal and other data ignoring the time dimension; the overall trained emotion recognition model not only significantly reduces the model's training parameter volume and resource consumption, but also improves the model's ability to recognize complex emotions in natural environments, especially for fine-grained categories.

[0064] 2. The audio and video dual-modal emotion recognition system based on adapter fusion provided by the present invention has the same beneficial effects as the audio and video dual-modal emotion recognition method based on adapter fusion provided by the present invention, which will not be described in detail here. BRIEF DESCRIPTION OF THE DRAWINGS

[0065] In order to more clearly illustrate the technical solutions and advantages of the embodiments of the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the prior art descriptions. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0066] Figure 1 A schematic diagram of the overall framework of an emotion recognition model of an audio and video dual-modal emotion recognition method based on adapter fusion provided by one embodiment of the present invention;

[0067] Figure 2 A schematic diagram comparing the performance of different modalities of an audio and video dual-modal emotion recognition method based on adapter fusion provided by one embodiment of the present invention;

[0068] Figure 3 Comparison of parameters and performance of an audio and video dual-modal emotion recognition method based on adapter fusion on the DFEW dataset provided by one embodiment of the present invention, where the bubble size corresponds to the model size;

[0069] Figure 4 A visualization of the t-SNE high-dimensional embedding features of the emotion recognition model in the last fold (fd5) of the DFEW and MAFW datasets of an audio-video dual-modal emotion recognition method based on adapter fusion provided by one embodiment of the present invention;

[0070] Figure 5 A visualization of the t-SNE high-dimensional embedding features of an emotion recognition model in the last fold (fd5) of the MAFW dataset in an audio-video dual-modal emotion recognition method based on adapter fusion provided by one embodiment of the present invention;

[0071] Figure 6 An attention visualization heat map of the emotion recognition model of an audio and video dual-modal emotion recognition method based on adapter fusion provided by one embodiment of the present invention on the DFEW dataset;

[0072] Figure 7 An embodiment of the present invention provides an audio and video dual-modal emotion recognition method based on adapter fusion, and an attention visualization heat map of the emotion recognition model on the MAFW dataset. DETAILED DESCRIPTION

[0073] In order to further illustrate the technical means and effects adopted by the present invention to achieve the predetermined purpose of the invention, the following, in conjunction with the accompanying drawings and preferred embodiments, describes in detail the specific implementation, structure, features and effects of the audio and video dual-modal emotion recognition method and system based on adapter fusion proposed by the present invention. In the following description, different "one embodiment" or "another embodiment" does not necessarily refer to the same embodiment. In addition, specific features, structures or characteristics of one or more embodiments may be combined in any suitable form.

[0074] Unless defined otherwise, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs.

[0075] The following describes in detail a specific solution of an audio and video dual-modal emotion recognition method and system based on adapter fusion provided by the present invention in conjunction with the accompanying drawings.

[0076] See also Figure 1 The first embodiment of the present invention provides an audio and video dual-modal emotion recognition method based on adapter fusion, the method comprising:

[0077] Step S1: Obtain video frame sequence and audio signal and perform preprocessing;

[0078] Step S2: constructing an emotion recognition model, which includes a bimodal feature extraction module, a bimodal fusion adapter structure, a cross-modal temporal modeling module, and a classifier;

[0079] Step S3: Based on the dual-modal feature extraction module, the spatial adapter and the global adapter are sequentially embedded, the pre-processed video frame sequence and audio signal are input and processed in sequence, and the spatial features and global features after corresponding modality enhancement are obtained in sequence;

[0080] Step S4: Based on the global features, an intermediate representation of the corresponding modality is generated through a dual-modal fusion adapter structure, and feature fusion is performed according to the intermediate representation to obtain fused features of the corresponding modality;

[0081] Step S5: Splice and fuse the features and input them into the cross-modal time series modeling module to extract time series features to obtain the final features;

[0082] Step S6: Input the final features into the classifier to obtain the predicted emotion category, and use the loss function to train the emotion recognition model, adjust the emotion recognition model parameters, and determine the optimal emotion recognition model;

[0083] Step S7: Obtain the video frame sequence and audio signal to be recognized, input them into the optimal emotion recognition model, and output the recognition result.

[0084] To better illustrate, audio and video have natural heterogeneity due to differences in data structure, temporal density, semantic expression and other aspects, making it difficult for simple splicing or unified processing to capture the subtle and dynamic emotion-related features between modalities; in addition, most existing emotion recognition methods rely on the joint optimization of models. Although the performance is acceptable, they consume a lot of training resources and lack flexibility and scalability. They are especially difficult to deploy on devices with limited computing power or in scenarios where tasks change frequently. In other words, they have significant deficiencies in multimodal feature fusion, cross-modal interaction modeling and temporal information extraction.

[0085] Emotional states themselves are highly temporal and subjective, and often show subtle and dynamic trends of change in natural interactions. Traditional recognition methods are mostly limited to single-modal or flat fusion in modeling in the time dimension, ignoring the coordinated changes between audio and video modalities in time evolution. The fusion method for multimodal data is not flexible enough, and it is difficult to fully explore the fine-grained semantic connections between audio and video modalities, resulting in low accuracy in recognizing complex emotional states, especially in distinguishing fine-grained emotion categories. For example, for easily confused categories such as anger, existing recognition methods have limited discrimination capabilities and weak cross-modal time modeling capabilities, and cannot Accurately capture the dynamic evolution of emotions in the process of natural expression; and based on the data noise, background changes and subjectivity of emotion labels generated in the real environment, the model structure is huge and the parameters are redundant, which further aggravates the difficulty of training the model and the risk of generalization, resulting in high training costs, inconvenient deployment, unstable model performance, and difficulty in meeting the engineering requirements of lightweight and efficient migration; in this regard, the present application proposes an audio and video dual-modal model architecture with flexible structure, efficient training, accurate fusion and suitable for dynamic emotion recognition scenarios, namely, an emotion recognition model, to improve recognition accuracy, improve fusion efficiency and enhance the robustness and transferability of the emotion recognition model.

[0086] As an optional implementation method, the emotion recognition model provided in this application can be integrated and applied with existing smart terminals, wearable devices, online platforms and other product forms to provide effective emotion perception capabilities and decision-making support for multiple fields such as human-computer interaction, mental health, educational analysis, recommendation systems, public safety, etc., and has good implementation and transformation prospects and market application value; for example, the emotion recognition model is deployed in a smart terminal or medical assistance system to conduct daily monitoring of the user's emotional state, which is particularly suitable for emotion recognition and early warning of patients with mental illness, people at high risk of depression, etc., and assists doctors in mental health assessment and personalized intervention.

[0087] Furthermore, step S1 includes:

[0088] Step S11: Decode the video frame sequence frame by frame, unify the size of each frame, and normalize it; extract the audio track from the normalized video frame sequence, calculate the log-Mel spectrogram, and obtain the corresponding audio signal;

[0089] Step S12: unifying the input lengths of the pre-processed video frame sequence and audio signal.

[0090] Optionally, the video frame sequence is the DFEW (Dynamic Facial Expression in the Wild) dataset. The data collection scenes are from real scenes such as movies, TV dramas, interviews, and online videos. They have obvious facial expressions, language intonation changes, and background noise disturbances. It is an important benchmark for evaluating multimodal emotion recognition algorithms in real-world situations. It contains thousands of video materials with clear emotion labels. Each video contains facial expressions and synchronized speech, and the labels are based on Ekman's six basic emotions and their extended definitions, covering multiple categories such as "happy", "sad", "angry", "surprise", "disgust", "fear" and "neutral".

[0091] The audio signal is from the MAFW (Multi-Action Facial Expression in the Wild) dataset, which focuses on fine-grained emotional expression that integrates facial expressions and speech, including a higher proportion of micro-expressions and emotional transitions. It provides clear audio and video time alignment annotations, facilitating the subsequent model's simultaneous extraction and analysis of inter-modal timing information, and helps evaluate the model's ability to perceive dynamic emotional transitions.

[0092] Specifically, the DFEW dataset is decoded frame by frame using the OpenCV (Open Source Computer Vision Library) and FFmpeg (Flexible Framework for Multimedia) toolchain. In this embodiment, 16 key frames are uniformly captured from each video clip as visual input, and the image corresponding to each key frame is resized to a uniform size of 224×224 and normalized to the interval [0,1]. The FFmpeg toolchain is then used to extract the audio track from the video frame sequence constructed from the key frames. The sampling rate is set to 16kHz, and the librosa library is used to calculate the log-Mel (Logarithmic Mel-Spectrogram) spectrogram to determine it as the audio signal for subsequent analysis. The log-Mel spectrogram is an audio feature representation method that captures the spectral characteristics of the audio signal by mapping the spectral energy to the Mel frequency scale.

[0093] The video frame sequences and audio signals determined for analysis are divided into training set, validation set and test set in proportion, and evaluated using five-fold cross-validation to ensure the robustness and universality of the results. It can be explained that in order to avoid data leakage, the division of training set, validation set and test set strictly ensures that the same video clip does not appear in the training set and test set at the same time.

[0094] It is explained that in step S2, an emotion recognition model is constructed, and the emotion recognition model includes a bimodal feature extraction module, a bimodal fusion adapter structure, a cross-modal time series modeling module and a classifier; that is, the emotion recognition model fully combines the strong expression ability with the parameter efficiency of the adapter mechanism, and by constructing corresponding modules, it strengthens the single modal representation, improves the cross-modal interaction modeling ability and depicts the evolution path of emotions over time, thereby constructing a bimodal emotion recognition framework with efficient parameters, fine fusion and flexible structure, and only fine-tunes a small number of inserted modules, which greatly reduces the demand for training resources and improves the flexibility of model migration.

[0095] Furthermore, step S3 includes:

[0096] Step S31: The bimodal feature extraction module uses the Transformer encoder as the feature extractor, and embeds the spatial adapter and the global adapter in sequence after the Transformer encoder.

[0097] Optionally, in this embodiment, the dual-modal feature extraction module (Modality-Specific Adapter, MSA) uses a dual-stream pre-trained Vision Transformer (ViT) as the backbone network for audio and video dual-modal processing, that is, it uses a Transformer encoder as a feature extractor to process visual and auditory information respectively, which can efficiently capture complex features in multimodal data.

[0098] Understandably, due to the significant heterogeneity of audio and video modalities in structure, semantic level and temporal density, such as different speaking speeds and frame rates, and the nonlinear relationship between speech spectrum and facial movements, traditional emotion recognition models find it difficult to adapt to the data features of the two modalities at the same time, resulting in unbalanced feature expression capabilities. Therefore, based on the idea of ​​efficient parameter fine-tuning in the field of NLP (Natural Language Processing), it is proposed to insert customized adapter structures for audio and video in each layer of Transformer, namely spatial adapter (S-Adapter) and global adapter (G-Adapter), respectively. Through lightweight modules, the spatial and global feature modeling capabilities within each modality are strengthened to enhance the intra-modal modeling capabilities and the inter-modal interactive expression capabilities, thereby improving the representation robustness of the model in different modalities, improving the recognition accuracy of dynamic and complex emotional states, and effectively improving the integrity of intra-modal feature expression.

[0099] It is explained that the spatial adapter S-Adapter is embedded in the Multi-Head Self-Attention (MHSA) stage to enhance local spatial features; the global adapter G-Adapter is embedded in the Multilayer Perceptron (MLP) stage to capture global contextual information; that is, the local structure modeling capability is improved through the spatial adapter, and the cross-regional semantic expression capability is enhanced by the global adapter. Combined with the overall bimodal feature extraction module MSA, the corresponding modal features are reinforced, and the consistency and completeness of the internal representation of the modality are enhanced, so that both audio and video modalities can obtain corresponding richer and more stable feature representations.

[0100] Step S32: Input the preprocessed video frame sequence and audio signal into the Transformer encoder to obtain the output features of the corresponding modalities.

[0101] Furthermore, step S32 includes:

[0102] Based on the preprocessed video frame sequence and audio signal, the corresponding input form is determined, which is the patch token and classification token; each modal structure is aligned and input into the Transformer encoder to obtain the output features of the corresponding modality, which are recorded as and ,in, Represents the output features corresponding to the video frame sequence; Represents the output features corresponding to the audio signal; Indicates the frame number of the video frame sequence; 、 Both represent the number of patches; Indicates the dimension of the output features.

[0103] It can be explained that in the aforementioned processing steps, the video frame sequence and audio signal have been processed into fixed-length input segments, and then the video frame sequence is converted into spatial feature representation through frame sampling and Patch Embedding; the audio signal generates time-frequency features through the Mel-spectrogram and is converted into Patch Tokens, that is, the feature representations of two different modalities are defined as output features; preferably, in this embodiment, the input forms are respectively scattered patch tokens and classification tokens, based on the video frame sequence, each frame image is divided into 16×16 patches, and the embedding dimension is 768; based on the audio signal, each log-Mel spectrogram is cut into 2D patches, and an embedding method equivalent to the visual path is used to maintain structural alignment.

[0104] Step S33: After normalization processing is performed based on the output features, a spatial adapter is used to perform dimensionality reduction, nonlinear activation and dimensionality increase in sequence to obtain the spatial features after corresponding modality enhancement.

[0105] For better explanation, in this embodiment, Take the Transformer block as an example, that is, the Transformer blocks in the video and audio bimodal branches form a pair; then determine the The input to the Transformer block is the output of the adjacent pair of Transformer blocks. The video frame sequence and the audio signal are determined by step S32 to be and ,in, Indicates the frame number of the video frame sequence; Indicates the dimension of the output feature; Indicates the number of feature channels to reflect the richness of feature information in a specific dimension and frame.

[0106] Furthermore, step S33 includes:

[0107] Step S331: Based on the normalized output features, the spatial adapter is used to obtain the initial spatial features of the corresponding modality. The corresponding calculation formula is:

[0108]

[0109] in, Represents the initial spatial features corresponding to the video frame sequence; Represents the initial spatial features corresponding to the audio signal; Represents the current Transformer encoder; Represents the adjacent previous Transformer encoder; Represents a spatial adapter; Represents the normalized output features corresponding to the video frame sequence; Represents the normalized output features corresponding to the audio signal;

[0110] Step S332: The normalized output features are sequentially subjected to dimensionality reduction, nonlinear activation, and dimensionality increase by the spatial adapter, and the output features are combined to obtain the spatial features after corresponding modality enhancement. The corresponding calculation formula is:

[0111]

[0112]

[0113] in, Represents the enhanced spatial features corresponding to the video frame sequence; Represents the enhanced spatial features corresponding to the audio signal; 、 Represent the output features corresponding to the video frame sequence and audio signal respectively; Represents the MHSA operation in the bimodal feature extraction module.

[0114] Specifically, in the Multi-Head Self Attention (MHSA) stage, the normalized output features are applied to the spatial adapter S-Adapter in MSA to obtain the initial spatial features of the corresponding modality, namely and ; Then, the normalized features are subjected to dimensionality reduction-nonlinear activation-dimensionality increase operations through the spatial adapter S-Adapter, that is, the high-dimensional output features are compressed to a lower dimension, nonlinear characteristics are introduced through the nonlinear activation function, and finally expanded back to the original dimension, which not only retains the key information but also enhances the expressive power of the output features; and then connected with the output residual of the MHSA stage to make the information flow smoother and avoid information loss, thereby obtaining the spatial features after corresponding modal enhancement.

[0115] Step S34: After normalization based on the spatial features, a global adapter is used for processing to obtain global features after corresponding modality enhancement.

[0116] Furthermore, step S34 includes:

[0117] Step S341: Based on the normalized spatial features, a global adapter is used to obtain the initial global features of the corresponding modality. The corresponding calculation formula is:

[0118]

[0119] in, Represents the initial global features corresponding to the video frame sequence; Represents the initial global features corresponding to the audio signal; Represents a global adapter; Represents the normalized spatial features corresponding to the video frame sequence; Represents the normalized spatial features corresponding to the audio signal;

[0120] Step S342: Enhance the initial global features to obtain the enhanced global features of the corresponding modalities. The corresponding calculation formula is:

[0121]

[0122]

[0123] in, Represents the enhanced global features corresponding to the video frame sequence; Represents the enhanced global features corresponding to the audio signal; Represents the MLP operation in the bimodal feature extraction module.

[0124] Specifically, in the MLP stage, the global adapter G-Adapter processes the normalized spatial features in the same way and enhances them to obtain the enhanced global features of the corresponding modalities.

[0125] Furthermore, step S4 includes:

[0126] Step S41: The dual-modality fusion adapter structure includes a first-stage dual-branch adapter and a second-stage dual-branch adapter.

[0127] Understandably, there is a problem between audio and video, which have semantic complementarity but large differences in expression. Simple splicing or shared attention mechanisms can easily lead to redundant cross-interference, and therefore usually face problems such as modal conflict, redundant propagation, and semantic shift. Based on the concept of "modal enhancement first, then interactive fusion", a bimodal fusion adapter (BFA) structure is constructed, and a gating mechanism is introduced to dynamically adjust the intensity of information flow between modalities. At the same time, token expansion and compression operations are used to achieve cross-modal feature alignment, allowing the emotion recognition model to automatically adjust the information flow intensity between modalities during training, thereby enhancing semantic complementarity while retaining the modal ontology information, more effectively capturing the fine-grained emotional interaction features between speech and vision, and improving cross-modal adaptation capabilities.

[0128] To illustrate, based on the backbone network determined in the aforementioned step S31, a symmetrical dual-modal fusion adapter structure is designed in the middle layer of the Transformer to independently enhance the audio and video respectively, and control the direction and intensity of the cross-modal information flow through a gating mechanism, so that a close association is established between the modalities at the feature level, effectively improving the accuracy of the fusion expression; specifically, the dual-modal fusion adapter structure BFA is used to enhance the intra-modal feature representation and realize cross-modal information interaction. It is inserted after each pair of Transformer blocks and includes a two-stage dual-branch adapter, that is, in this embodiment, in the first The first-stage dual-branch adapter and the second-stage dual-branch adapter are inserted after the Transformer block respectively. By dynamically controlling the degree of opening of feature channels between modalities and combining copy-aggregation operations, information coupling and decoupling between audio and video are achieved, significantly enhancing cross-modal expression capabilities.

[0129] Step S42: Input the global features into the dual-branch adapter of the first stage to generate the intermediate representation of the corresponding modality. The corresponding calculation formula is:

[0130]

[0131]

[0132] in, 、 Represent the intermediate representations corresponding to the video frame sequence and audio signal respectively; 、 Represent the enhanced global features corresponding to the video frame sequence and audio signal respectively; 、 Both represent the first stage of dual-branch adapter operation; 、 Both represent the learnable parameters of the gating mechanism;

[0133] Step S43: Input the intermediate representation into the second-stage dual-branch adapter for feature fusion to obtain the fusion features of the corresponding modalities. The corresponding calculation formula is:

[0134]

[0135]

[0136] in, 、 Represent the fusion features corresponding to the video frame sequence and audio signal respectively; 、 Both represent the second-stage dual-branch adapter operation; 、 Both represent the learnable parameters of the gating mechanism; Indicates that along the time axis copy Second-rate, Indicates the number of frames in the video frame sequence, represents the dimension of the intermediate representation, Indicates the number of feature channels in the intermediate representation, used to match the fusion features size; Indicates that the average pooling operation Compressed into .

[0137] Specifically, the global features after the MLP stage are processed by the dual-branch adapter and gating mechanism in the first stage to generate the intermediate representation of the corresponding modality, which are and ; Then the intermediate representation is input to the second stage dual-branch adapter for feature fusion, and the gating mechanism is also introduced to obtain the fusion features of the corresponding modalities, which are recorded as and .

[0138] Understandably, emotions have dynamic evolution characteristics. Existing emotion recognition methods usually process the time dimension of audio or video separately, easily ignoring the coupling changes between the two modalities in time sequence, and lack effective time sequence processing capabilities. Therefore, a cross-modal time sequence modeling module is constructed, and the corresponding modal fusion features obtained above are sent to the Transformer time sequence encoder, and the time sequence and modal information are injected through modal type embedding and time position embedding to achieve collaborative modeling, and then the overall emotion evolution path is input into the Transformer structure modeling, so that the emotion recognition model can accurately track the emotion change trajectory and improve the model's perception of complex dynamic emotion change processes.

[0139] It is explained that after the bimodal fusion adapter structure, the cross-modal temporal modeling module (CTM) is introduced to capture the temporal synergy between audio and video and improve the emotion recognition model's ability to model the real emotion evolution process. Among them, the cross-modal temporal modeling module CTM includes a transformer-encoder, which is composed of multiple encoding layers. Each layer contains a self-attention sublayer and a feedforward neural network sublayer. The residual connection and layer normalization technology are used to improve the stability and expression ability of the model.

[0140] Furthermore, step S5 includes:

[0141] Step S51: Collect the fusion features after L pairs of dual-modal fusion adapter structures, which are respectively recorded as and , splicing fusion features, the corresponding calculation formula is: ,in, Indicates the splicing result; Represents a splicing operation; 、 Respectively represent the fusion features of the audio signal and video frame sequence after L pairs of dual-modal fusion adapter structures;

[0142] Step S52: Input the splicing result into the transformer-encoder in the cross-modal temporal modeling module to extract temporal features and obtain the final features. The corresponding calculation formula is:

[0143]

[0144] in, Indicates the final feature; Indicates the operation of extracting time series features; Indicates positional encoding; Indicates category code.

[0145] Specifically, the fusion features are collected after passing through L pairs of bimodal fusion adapter structures, that is, after passing through L pairs of Transformer blocks equipped with bimodal feature extraction modules MSA and bimodal fusion adapter structures BFA, they are input into the cross-modal temporal modeling module to obtain fusion features that capture the temporal dependency and interaction between audio and video, and after splicing, they are input into the transformer-encoder for temporal feature extraction.

[0146] Preferably, in this embodiment, a linear classifier is used for training, that is, the final features are mapped onto a straight line, thereby achieving differentiation of different emotion categories.

[0147] Furthermore, step S6 includes:

[0148] Step S61: Input the final features into the classifier to obtain the predicted emotion category. The corresponding calculation formula is:

[0149]

[0150] in, represents the predicted emotion category; Represents a classifier operation; Represents linear operations; Indicates the final feature;

[0151] Step S62: Use the loss function to train the emotion recognition model. The corresponding calculation formula is:

[0152]

[0153] in, represents the loss function; Indicates the total number of emotion categories; Indicates that the real One-hot encoding of emotion-like categories, ; Indicates that the emotion recognition model predicts The probability of the emotion class, .

[0154] Make an explanation, Represents a linear operation, which represents a fully connected layer; Represents the classifier operation, which is used to map the output of the fully connected layer into a probability distribution, and then obtain the predicted emotion category of the final output.

[0155] Specifically, during the training of the emotion recognition model, a unified training process and standardized parameter settings are adopted to ensure the comparability of the performance of the emotion recognition model and the reproducibility of the experiment. In this embodiment, a single sampling method is adopted for each video frame sequence, and the same number of key frame images are extracted from each video frame sequence, that is, , each key frame is adjusted to a uniform size, i.e. To adapt to the input of the backbone network Transformer (ViT) module in the emotion recognition model, the optimizer is required to use AdamW (Adam Weight DecayRegularization, i.e., adaptive moment estimation weight decay regularization), which has good generalization ability and regularization control effect. The corresponding initial learning rate is set based on the video frame sequence and audio signal, that is, it is set to on the DFEW dataset. , set on the MAFW dataset as , and uses the cosine annealing strategy to gradually decay to 0 within 25 training cycles (epochs). That is, the learning rate is adjusted by a smooth cosine curve to improve convergence stability. This can not only avoid training stagnation caused by premature decline in the learning rate, but also prevent slow training speed caused by too low a learning rate, ensuring that the emotion recognition model maintains optimal learning efficiency and stability during training. The training batch size is uniformly set to 8 to ensure that the amount of data processed in each iteration is moderate, which helps to balance training speed and performance. The weight decay coefficient is set to , to prevent overfitting; the remaining optimizer hyperparameters, such as learning rate and momentum, use the PyTorch default configuration to simplify the parameter adjustment process.

[0156] It can be explained that during the emotion recognition model training stage, only the parameters of the specific modality adaptation, bimodal fusion adapter structure, cross-modal time series modeling module and classifier are updated and adjusted, that is, the model parameters of the backbone network in the bimodal feature extraction module remain frozen, which greatly reduces the parameter overhead, thereby reducing the training cost and avoiding catastrophic forgetting, and improving the application feasibility and migration generalization ability of the emotion recognition model in resource-constrained devices.

[0157] As an optional implementation method, the various modules of the emotion recognition model can be optimized or replaced according to actual conditions. For example, the adapter can be set to explore task-adaptive structural search, or set to a dynamic adapter that can be conditionally activated to achieve stronger generalization and flexibility; in the dual-modal fusion adapter structure, in addition to referencing the gating mechanism, multi-scale attention fusion, multi-pathway interactive attention mechanism, integrated fusion strategy, etc. can also be introduced to enhance the modeling ability of semantic boundaries and coupling strength between modalities; the Transformer structure is currently used to model the unified time series, which can be replaced by causal convolution, structured RNN (Recurrent Neural Networks) or graph temporal network models in the future to improve the balance of short-term and / or long-term dynamic modeling; in real scenarios, there may be modal missing or asynchronous phenomena, and the overall model robustness can be improved by introducing modal compensation or modal adversarial discrimination mechanism.

[0158] It is explained that in order to explain the attention distribution and cross-modal information flow path within the adapter, Grad-CAM (Gradient-weighted Class Activation Mapping) and multimodal visualization analysis methods can be combined to facilitate task tuning and domain expert verification; in addition, to address the instability caused by few samples or label subjectivity, emotion feature-preserving data enhancement methods can be introduced, such as feature perturbation, cross-modal generation, etc., and even combined with adversarial learning for style transfer sample expansion to enhance the generalization ability of the model.

[0159] See also Figure 2 , it is explained that the performance of the emotion recognition model proposed in this application is compared on the DFEW and MAFW datasets for video only, audio only, and dual modality of audio and video, respectively, to verify the recognition rate of emotion categories. Among them, the performance indicators are the average unweighted recall rate (UAR) and the weighted recall rate (WAR). UAR represents the average recall rate of all emotion categories, without considering the difference in the number of samples in each emotion category; WAR performs a weighted average according to the number of samples in each emotion category, so that the emotion category with a larger number of samples has a greater impact on the final result. It can be explained that the recognition rate of emotion recognition based on dual modality is high and the model performance is good.

[0160] See also Figure 3For better explanation, in this embodiment, verification is carried out on two dynamic emotion recognition datasets DFEW and MAFW in real scenarios. The experiment adopts a standard five-fold cross-validation scheme, and the emotion recognition model constructed in this application, denoted as ABVnet, is compared with the existing mainstream methods to evaluate the performance of the emotion recognition model constructed in this application in multi-category emotion recognition tasks, where the performance indicators are the average unweighted recall rate UAR and the weighted recall rate WAR; Table 1 is then obtained, which shows the performance comparison of the emotion recognition method proposed in this application and the mainstream methods on the DFEW and MAFW datasets.

[0161] Table 1 Performance comparison of emotion recognition methods and mainstream methods on DFEW and MAFW datasets

[0162]

[0163] It can be shown that on the DFEW dataset, the average unweighted recall rate UAR of the emotion recognition model ABVnet of this application reached 67.52%, and the weighted recall rate WAR reached 78.28%, ranking first among all compared methods; at the same time, the lightweight version of the emotion recognition model ABVnet-S with fewer parameters achieved a UAR of 67.23% and a WAR of 78.13%, showing good accuracy and model efficiency; on the MAFW dataset, the emotion recognition model ABVnet achieved a UAR of 44.46% and a WAR of 58.41%, ranking second, further verifying the stability and robustness of the emotion recognition model under different audio and video data conditions.

[0164] Please combine Figure 4-Figure 7 , among which, t-SNE (t-distributed Stochastic Neighbor Embedding) high-dimensional embedding feature visualization diagrams of some existing emotion recognition methods on the DFEW and MAFW datasets and the emotion recognition model provided by this application are provided; it can be shown that the emotion recognition model can effectively divide the emotion categories clearly, especially the boundaries between similar emotions such as "anger" and "sadness", "neutral" and "boredom" are clearer, indicating that the emotion recognition model is effective in extracting discriminative features; and through the attention heat map display, it is shown that the emotion recognition model dynamically pays attention to the corners of the mouth, eye areas and speech emotion intensity segments in multi-frame sequences, showing that the emotion recognition model has good perception of key emotional areas and changing rhythms.

[0165] In addition, based on the specific emotion categories of the DFEW dataset, the emotion recognition model provided by this application is compared with the existing methods for emotion category recognition, and then Table 2 is obtained, which is a performance comparison table of the emotion recognition method proposed in this application and the mainstream method in emotion category recognition based on the DFEW dataset, among which Ha (Happiness) represents happiness; Fe (Fear) represents fear, Sa (Sadness) represents sadness; Ne (Neutral) represents neutral; An (Anger) represents anger; Su (Surprise) represents surprise; Di (Disgust) represents disgust; It can be seen that in Figure 6 In the figure, there are heat maps of “fear” and “surprise”; in Figure 7 In the figure, the heatmaps for “fear” and “anxiety” are visualized respectively.

[0166] Table 2 Performance comparison of emotion recognition methods and mainstream methods in emotion recognition based on the DFEW dataset

[0167]

[0168] It can be shown that the classification accuracy rates in emotion categories such as "happiness" and "fear" reached 96.73% and 55.89% respectively, both higher than the existing methods, indicating that the emotion recognition model provided by this application also has advantages in identifying marginal emotions.

[0169] It can be understood that the various modules in the emotion recognition model work together to improve the overall performance and practicality of the emotion recognition model; a lightweight adapter module is used to freeze the backbone network of the emotion recognition model and fix the parameters of the pre-trained backbone network so that it does not participate in back propagation and update, thereby reducing training parameters; intra-modal feature enhancement, inter-modal interaction fusion and cross-modal time series modeling are realized respectively, that is, compared with the traditional splicing and average fusion strategies, the spatial adapter and the global adapter are embedded based on the dual-modal feature extraction module, and the spatial features and the global features are determined in turn, and then the features are fused and spliced ​​through the dual-modal fusion adapter structure, which can It effectively retains the semantic complementary information between modalities and avoids redundant propagation, making the obtained fusion features more recognizable, and the adapter can reduce the amount of training parameters, making the emotion recognition model highly practical and energy-efficient in mobile devices and edge deployments; the introduced cross-modal time series modeling module obtains the final features, enabling it to show stronger emotion tracking capabilities during the emotion transition process, in order to solve the defect of single-modal and other data ignoring the time dimension; the overall trained emotion recognition model not only significantly reduces the model's training parameter volume and resource consumption, but also improves the model's ability to recognize complex emotions in natural environments, especially for fine-grained categories.

[0170] The second embodiment of the present invention proposes an audio and video dual-modal emotion recognition system based on adapter fusion. The system stores program data, and when the program data is executed, it implements the steps of an audio and video dual-modal emotion recognition method based on adapter fusion as described in any of the aforementioned embodiments; this system has the same beneficial effects as the aforementioned audio and video dual-modal emotion recognition method based on adapter fusion, and will not be repeated here.

[0171] It can be understood that when a module of an audio and video dual-modal emotion recognition system based on adapter fusion is in operation, it is necessary to utilize an audio and video dual-modal emotion recognition method based on adapter fusion provided by the aforementioned embodiment. Therefore, whether the method is integrated with program data or different hardware is configured to produce functions similar to the effects achieved by the present invention, it falls within the scope of protection of the present invention.

[0172] It should be noted that the order in which the embodiments of the present invention are described above is for illustrative purposes only and does not necessarily represent the superiority or inferiority of the embodiments. The processes depicted in the accompanying drawings do not necessarily require the specific order or sequential order shown to achieve the desired results. In certain embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0173] The various embodiments in this specification are described in a progressive manner, and the same or similar parts between the various embodiments can be referred to each other. Each embodiment focuses on the differences from other embodiments.

Claims

1. An audio and video dual-modal emotion recognition method based on adapter fusion, characterized in that: The method comprises: Obtain video frame sequences and audio signals and perform preprocessing; Constructing an emotion recognition model, wherein the emotion recognition model includes a bimodal feature extraction module, a bimodal fusion adapter structure, a cross-modal temporal modeling module, and a classifier; Based on the dual-modal feature extraction module, the spatial adapter and the global adapter are embedded in sequence. The pre-processed video frame sequence and audio signal are input and processed in sequence, and the spatial features and global features after corresponding modality enhancement are obtained in sequence, including: The bimodal feature extraction module adopts the Transformer encoder as the feature extractor, and embeds the spatial adapter and the global adapter in sequence after the Transformer encoder; The preprocessed video frame sequence and audio signal are input into the Transformer encoder to obtain the output features of the corresponding modalities respectively; After normalization based on the output features, the spatial adapter is used to perform dimensionality reduction, nonlinear activation, and dimensionality increase in sequence to obtain the spatial features after corresponding modality enhancement, including: Based on the normalized output features, the spatial adapter is used to obtain the initial spatial features of the corresponding modality. The corresponding calculation formula is: ； in, Represents the initial spatial features corresponding to the video frame sequence; Represents the initial spatial features corresponding to the audio signal; Represents the current Transformer encoder; Represents the adjacent previous Transformer encoder; Represents a spatial adapter; Represents the normalized output features corresponding to the video frame sequence; Represents the normalized output features corresponding to the audio signal; The spatial adapter is used to perform dimensionality reduction, nonlinear activation, and dimensionality increase on the normalized output features in sequence, and the output features are combined to obtain the spatial features after corresponding modal enhancement. The corresponding calculation formula is: ； ； in, Represents the enhanced spatial features corresponding to the video frame sequence; Represents the enhanced spatial features corresponding to the audio signal; 、 Represent the output features corresponding to the video frame sequence and audio signal respectively; Represents the MHSA operation in the bimodal feature extraction module; After normalization based on spatial features, global adapter is used for processing to obtain the enhanced global features of the corresponding modalities; Based on the global features, the intermediate representation of the corresponding modality is generated through the dual-modal fusion adapter structure, and feature fusion is performed according to the intermediate representation to obtain the fusion features of the corresponding modality respectively; The fusion features are spliced ​​and input into the cross-modal time series modeling module to extract time series features and obtain the final features; The final features are input into the classifier to obtain the predicted emotion category, and the emotion recognition model is trained using the loss function to adjust the emotion recognition model parameters and determine the optimal emotion recognition model; Obtain the video frame sequence and audio signal to be recognized, input them into the optimal emotion recognition model, and output the recognition results.

2. The audio and video dual-modal emotion recognition method based on adapter fusion according to claim 1 is characterized in that: Obtain video frame sequences and audio signals and perform preprocessing, including: Decode the video frame sequence frame by frame, unify the size of each frame, and normalize it; extract the audio track from the normalized video frame sequence, calculate the log-Mel spectrogram, and obtain the corresponding audio signal; Unify the input length of preprocessed video frame sequences and audio signals.

3. The audio and video dual-modal emotion recognition method based on adapter fusion according to claim 1 is characterized in that: The preprocessed video frame sequence and audio signal are input into the Transformer encoder to obtain the output features of the corresponding modalities, including: Based on the pre-processed video frame sequence and audio signal, the corresponding input form is determined, which is the patch token and classification token; each modal structure is aligned and input into the Transformer encoder to obtain the output features of the corresponding modality, which are recorded as and ,in, Represents the output features corresponding to the video frame sequence; Represents the output features corresponding to the audio signal; Indicates the frame number of the video frame sequence; 、 Both represent the number of patches; Indicates the dimension of the output features.

4. The audio and video dual-modal emotion recognition method based on adapter fusion according to claim 3 is characterized in that: After normalization based on spatial features, global adapters are used for processing to obtain the global features after corresponding modality enhancement, including: Based on the normalized spatial features, the global adapter is used to obtain the initial global features of the corresponding modality. The corresponding calculation formula is: ； in, Represents the initial global features corresponding to the video frame sequence; Represents the initial global features corresponding to the audio signal; Represents a global adapter; Represents the normalized spatial features corresponding to the video frame sequence; Represents the normalized spatial features corresponding to the audio signal; The initial global features are enhanced to obtain the enhanced global features of the corresponding modes. The corresponding calculation formula is: ； ； in, Represents the enhanced global features corresponding to the video frame sequence; Represents the enhanced global features corresponding to the audio signal; Represents the MLP operation in the bimodal feature extraction module.

5. The audio and video dual-modal emotion recognition method based on adapter fusion according to claim 1 is characterized in that: Based on the global features, the intermediate representation of the corresponding modality is generated through the dual-modal fusion adapter structure. Feature fusion is performed based on the intermediate representation to obtain the fusion features of the corresponding modality, including: The dual-modality fusion adapter structure includes a first-stage dual-branch adapter and a second-stage dual-branch adapter; The global features are input into the first-stage dual-branch adapter to generate the intermediate representation of the corresponding modality. The corresponding calculation formula is: ； ； in, 、 Represent the intermediate representations corresponding to the video frame sequence and audio signal respectively; 、 Represent the enhanced global features corresponding to the video frame sequence and audio signal respectively; 、 Both represent the first stage of dual-branch adapter operation; 、 Both represent the learnable parameters of the gating mechanism; The intermediate representation is input into the second-stage dual-branch adapter for feature fusion to obtain the fusion features of the corresponding modalities. The corresponding calculation formula is: ； ； in, 、 Represent the fusion features corresponding to the video frame sequence and audio signal respectively; 、 Both represent the second-stage dual-branch adapter operation; 、 Both represent the learnable parameters of the gating mechanism; Indicates that along the time axis copy Second-rate, Indicates the number of frames in the video frame sequence, represents the dimension of the intermediate representation, Indicates the number of feature channels in the intermediate representation, used to match the fusion features size; Indicates that the average pooling operation Compressed into .

6. The audio and video dual-modal emotion recognition method based on adapter fusion according to claim 1 is characterized in that: The fusion features are spliced ​​and input into the cross-modal time series modeling module for time series feature extraction to obtain the final features, including: The fusion features after collecting L pairs of dual-modal fusion adapter structures are recorded as and , splicing fusion features, the corresponding calculation formula is: ,in, Indicates the splicing result; Represents a splicing operation; 、 Respectively represent the fusion features of the audio signal and video frame sequence after L pairs of dual-modal fusion adapter structures; The concatenated results are input into the transformer-encoder in the cross-modal time series modeling module to extract time series features and obtain the final features. The corresponding calculation formula is: ； in, Indicates the final feature; Represents the operation of extracting time series features; Indicates positional encoding; Indicates category code.

7. The audio and video dual-modal emotion recognition method based on adapter fusion according to claim 1 is characterized in that: The final features are input into the classifier to obtain the predicted emotion category, and the emotion recognition model is trained using the loss function. The emotion recognition model parameters are adjusted to determine the optimal emotion recognition model, including: The final features are input into the classifier to obtain the predicted emotion category. The corresponding calculation formula is: ； in, represents the predicted emotion category; Represents a classifier operation; Represents linear operations; Indicates the final feature; The loss function is used to train the emotion recognition model. The corresponding calculation formula is: ； in, represents the loss function; Indicates the total number of emotion categories; Indicates that the real One-hot encoding of emotion-like categories, ; Indicates that the emotion recognition model predicts The probability of the emotion class, .

8. An audio and video dual-modal emotion recognition system based on adapter fusion, characterized in that: The system stores program data, and when the program data is executed, the steps of the audio and video dual-modal emotion recognition method based on adapter fusion as described in any one of claims 1 to 7 are implemented.

Images