Dynamic facial expression recognition method and device, medium and equipment

Abstract

The invention discloses a dynamic facial expression recognition method and device, a medium and equipment, and relates to the technical field of artificial intelligence and computer vision. In the multi-layer video feature and audio feature extraction process, feature fusion and recombination based on frame alignment are introduced, and video and audio cross-modal time consistency is realized, so that finer-grained dynamic facial expression changes are captured in the feature extraction stage. In a multi-level feature extraction fusion stage, a lightweight Mama module is utilized, fusion interaction of a current level is performed after feature extraction codes of each level are aligned, so that cross-modal interaction can be realized in different levels in the overall multi-level feature extraction stage, the discriminability and robustness of features are enhanced, and the accuracy of feature extraction is improved. When the dynamic facial expression recognition is carried out based on the audio features and the video features of the last level, the accuracy and robustness of the dynamic facial expression recognition are improved.

Description

Technical Field

[0001] This invention relates to the fields of artificial intelligence and computer vision technology, and in particular to a method, apparatus, medium and device for dynamic facial expression recognition. Background Technology

[0002] Currently, facial expression recognition, as one of the core tasks in affective computing and human-computer interaction, has wide-ranging applications in psychological research, intelligent monitoring, education, medical assistance, and virtual reality. In recent years, with the development of deep learning, static image expression recognition methods have made significant progress, but dynamic facial expression recognition in complex environments still faces considerable challenges.

[0003] In existing technologies, dynamic facial expression recognition methods mainly focus on single-modal feature modeling, such as video-based expression recognition or audio-based emotion recognition. Single-modal methods are prone to performance degradation and lack robustness when affected by factors such as noise, lighting changes, and pose shifts. To address this, researchers have proposed a multimodal fusion method that combines visual and audio information to improve the accuracy and stability of recognition.

[0004] However, existing multimodal dynamic facial expression recognition methods only align features from different modalities at a global level, making it difficult to capture fine-grained dynamic information about facial expressions changing over time. Furthermore, feature fusion between different modalities mostly remains at a shallow level, lacking deep and dynamic cross-modal interaction mechanisms, resulting in insufficient utilization of multimodal information. Therefore, existing methods still have shortcomings in cross-modal feature alignment and interaction modeling, leading to low accuracy and robustness in dynamic facial expression recognition. Summary of the Invention

[0005] Therefore, it is necessary to provide a dynamic facial expression recognition method, device, medium, and equipment to address the aforementioned technical problems.

[0006] The present invention adopts the following technical solution: This invention provides a dynamic facial expression recognition method, comprising: Acquire video and audio data captured from the target user's face, and perform multi-level feature fusion on the video and audio data; In each level of feature fusion, the initial video features and initial audio features including global and local features are extracted based on the input features. In the first level of feature fusion, the input features are video data and audio data, and in subsequent levels of feature fusion, the input features are the output features of the previous level. The local features in the initial audio features of the current level are divided into multiple parts according to the number of frames of the video features of the current level, and global features are added to each part to obtain multi-frame speech features. The multi-frame speech features and the initial video features are concatenated in the token dimension to obtain multimodal fusion features. The Mamba model is used to fuse and encode the multimodal fusion features and then reverse the feature dimension to obtain the audio and video features of the current level after cross-modal interactive fusion and update, thus completing the feature fusion of the current level. The evolution of dynamic facial expressions over time is modeled based on the global features in the audio and video features at the last level, resulting in a temporal feature vector. Emotion classification is then performed based on the temporal feature vector to obtain the emotional category of the target user's facial expressions.

[0007] Optionally, the step of dividing the local features in the initial audio features of the current level into multiple parts according to the number of frames of the video features of the current level, and adding global features to each part to obtain multi-frame speech features, specifically includes: The local features in the initial audio features of the current level are divided into multiple parts according to the number of frames of the video features of the current level. Global features are added to each part using the following formula to obtain multi-frame speech features: ; in, For the first Frame speech features After evenly dividing the local features in the initial audio features of the current level, the th Patch token, For the class token in the audio features, Indicates splicing.

[0008] Optionally, the step of concatenating multi-frame speech features and initial video features along the token dimension to obtain multimodal fusion features specifically includes: The following formula unifies the dimensionality of multi-frame speech features and initial video features through linear mapping: , ; The multimodal fusion feature is obtained by concatenating the unified multi-frame speech features and the initial video features along the token dimension using the following formula: ; in, For the first Frame speech features For multi-frame speech features after unifying dimensions, This indicates a dimensionality reduction operation. This indicates normalization processing. For video features, The initial video features after unifying dimensions, Indicates splicing, This is a multimodal fusion feature.

[0009] Optionally, the step of modeling the evolution of dynamic facial expressions over time based on global features in the final level of audio and video features to obtain a temporal feature vector specifically includes: The frame-level representation of the global features of the final-level video features is determined. The global features of the final-level audio features are copied according to the number of frames in the frame-level representation. The global features of the final-level audio features are then fused with the frame-level representation of the global features of the final-level video features and the embedding of each frame position to obtain the input sequence features. The Transformer Encoder models the evolution of dynamic facial expressions over time based on the features of the input sequence, resulting in a temporal feature vector.

[0010] Optionally, multi-level feature fusion is performed on video and audio data using a pre-trained improved mask autoencoder as a visual feature extractor and a pre-trained improved audio mask autoencoder as an audio spectral feature extractor. Both the mask autoencoder and the audio mask autoencoder include stacked Transformer blocks of the same layer. Each layer of Transformer blocks in the mask autoencoder and the audio mask autoencoder are parallel and sequentially connected to the feature recombination module and the Mamba model to form the improved mask autoencoder and the improved audio mask autoencoder. In the feature fusion at each level, the Transformer block in the improved mask autoencoder is used to extract the initial video features of the current level; the Transformer block in the improved audio mask autoencoder is used to extract the initial audio features of the current level; the feature reorganization module is used to divide the local features in the initial audio features of the current level into multiple parts according to the number of frames of the video features of the current level, and add global features to each part to obtain multi-frame speech features. The multi-frame speech features and the initial video features are concatenated in the token dimension to obtain multimodal fusion features. The evolution of dynamic facial expressions over time is modeled using a multimodal temporal module based on global features in the final layer of audio and video features, resulting in a temporal feature vector. The facial expression emotion category of the target user is obtained by classifying emotions based on temporal feature vectors using a classifier.

[0011] Optionally, the Mamba model, multimodal temporal module, and classifier are trained while the visual feature extractor and audio spectral feature extractor are frozen.

[0012] This invention provides a dynamic facial expression recognition device, comprising: The acquisition module is used to acquire video and audio data captured from the target user's face, and to perform multi-level feature fusion on the video and audio data respectively. The extraction module is used to extract the initial video features and initial audio features including global and local features of the current level based on the input features in each level of feature fusion. The input features in the first level of feature fusion are video data and audio data, and the input features in the subsequent levels of feature fusion are the output features of the previous level. The alignment module is used to divide the local features in the initial audio features of the current level into multiple parts according to the number of frames of the video features of the current level, and add global features to each part to obtain multi-frame speech features. The multi-frame speech features and the initial video features are concatenated in the token dimension to obtain multimodal fusion features. The fusion module is used to fuse and encode multimodal fusion features using the Mamba model and reverse-restore them along the feature dimension to obtain the audio and video features of the current level after cross-modal interactive fusion and update, thus completing the feature fusion of the current level. The recognition module is used to model the evolution of dynamic facial expressions over time based on global features in the audio and video features of the last layer, to obtain a temporal feature vector, and to classify emotions based on the temporal feature vector to obtain the emotional category of the target user's facial expression.

[0013] The present invention provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described dynamic facial expression recognition method.

[0014] The present invention provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the above-described dynamic facial expression recognition method.

[0015] The above-mentioned at least one technical solution adopted in this invention can achieve the following beneficial effects: This invention introduces frame-aligned feature fusion and recombination during multi-layered video and audio feature extraction, achieving fine-grained temporal consistency across video and audio modalities. This allows for the capture of more granular dynamic facial expression changes during the feature extraction stage. In the multi-level feature extraction and fusion stage, a lightweight Mamba module is used to perform fusion interaction at each level after feature extraction encoding alignment. This enables cross-modal interaction at different levels in the overall multi-level feature extraction stage, improving the utilization rate of cross-modal information and enhancing the discriminativeness and robustness of the updated audio and video features after cross-modal interaction fusion. When performing dynamic facial expression recognition based on the final level's audio and video features, the accuracy and robustness of dynamic facial expression recognition are improved. Attached Figure Description

[0016] The accompanying drawings, which are included to provide a further understanding of the invention and form part of this invention, illustrate exemplary embodiments of the invention and are used to explain the invention, but do not constitute an undue limitation of the invention. In the drawings:

[0017] Figure 1 This is a schematic diagram of a dynamic facial expression recognition method provided by the present invention; Figure 2 A flowchart illustrating a dynamic facial expression recognition method based on a self-supervised lightweight model provided by the present invention; Figure 3 This is a schematic diagram of a dynamic facial expression recognition device provided by the present invention; Figure 4 This is a schematic diagram of a computer device for implementing a dynamic facial expression recognition method according to the present invention. Detailed Implementation

[0018] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below in conjunction with specific embodiments and corresponding drawings. Obviously, the described embodiments are only a part of the embodiments of this invention, and not all of them. Based on the embodiments of this invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this invention.

[0019] Currently, most existing multimodal dynamic facial expression recognition methods rely on full-parameter fine-tuning, which still has shortcomings in cross-modal feature alignment and interaction modeling. Specifically, full-parameter fine-tuning requires updating all model parameters, which can easily lead to the forgetting of pre-trained knowledge, thus affecting generalization ability. At the same time, existing methods mostly stop at shallow concatenation of features between different modalities, lacking deep and dynamic cross-modal interaction mechanisms. In addition, some methods only align features of different modalities at the global level, making it difficult to capture fine-grained dynamic information of facial expressions changing over time. Furthermore, full-parameter training not only brings large computational and storage overhead, but also limits the deployment and expansion of the methods in practical applications.

[0020] Therefore, how to effectively model the deep interactions between audio and video modalities while ensuring parameter efficiency, and improve the accuracy and robustness of dynamic facial expression recognition, has become a pressing technical problem that needs to be solved.

[0021] The technical solutions provided by the various embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

[0022] Figure 1 This is a schematic diagram of a dynamic facial expression recognition method according to the present invention, which specifically includes the following steps: S101: Acquire video and audio data captured from the target user's face, and perform multi-level feature fusion on the video and audio data.

[0023] S102: In the feature fusion of each level, the initial video features of the current level and the initial audio features including global and local features are extracted based on the input features; the input features in the feature fusion of the first level are video data and audio data, and the input features in the feature fusion of subsequent levels are the output features of the previous level.

[0024] S103: Divide the local features in the initial audio features of the current level into multiple parts according to the number of frames of the video features of the current level, and add global features to each part to obtain multi-frame speech features. Concatenate the multi-frame speech features and the initial video features in the token dimension to obtain multimodal fusion features.

[0025] S104: The multimodal fusion features are fused and encoded using the Mamba model and then reversed in the feature dimension to obtain the audio and video features of the current level after cross-modal interactive fusion and update, thus completing the feature fusion of the current level.

[0026] S105: Model the evolution of dynamic facial expressions in the time dimension based on the global features in the audio and video features of the last level, obtain the temporal feature vector, and classify emotions based on the temporal feature vector to obtain the facial expression emotion category of the target user.

[0027] For ease of explanation, the following description focuses solely on the server as the executing entity. The server mentioned in this invention can be a server set up on a business platform, or a device such as a desktop computer or laptop computer capable of executing the solution of this invention.

[0028] To ensure efficient parameter tuning while effectively modeling deep interactions between audio and video modalities and improving the accuracy and robustness of dynamic facial expression recognition, this invention proposes a multimodal dynamic facial expression recognition method based on efficient parameter fine-tuning. This method comprises four stages: feature extraction, feature alignment, modality fusion, and temporal modeling. In the feature extraction stage, pre-trained visual feature extractors and audio spectral feature extractors are used to extract multimodal features, and encoder parameters are frozen to fully retain pre-trained knowledge and avoid catastrophic forgetting. In the feature alignment stage, the extracted audio and video features are reconstructed using a frame-aligned feature reassembly method to achieve temporal consistency alignment between audio and video modalities, ensuring fine-grained correspondences between different modalities. In the modality fusion stage, the aligned features are input to a multimodal fusion adapter placed after each modal encoder layer. This adapter employs a lightweight Mamba-based structure, enabling efficient cross-modal interaction and fusion at different levels, thereby capturing more discriminative emotion-related features. Finally, in the temporal modeling stage, the fused features are input into the cross-modal temporal modeling module to capture the temporal evolution of dynamic facial expressions, and the final emotion category prediction result is output through the classifier.

[0029] This invention features a simple structure and enables cross-modal fusion with only minor adjustments to a small number of adapter parameters, effectively avoiding the catastrophic forgetting problem caused by full parameter fine-tuning. Through frame-level alignment strategies and hierarchical fusion mechanisms, this invention can better capture the fine-grained dynamic features of facial expressions changing over time, significantly improving the accuracy and robustness of dynamic facial emotion recognition while ensuring parameter efficiency.

[0030] Specifically, in one or more embodiments of the present invention, the server may first acquire video and audio data captured from the target user's face, and then perform multi-level feature fusion on the video and audio data. Each level of feature fusion includes:

[0031] Step 1: For the feature extraction stage, acquire the input audio data. With video data The inputs are respectively fed into a pre-trained audio spectral feature extractor. and visual feature extractor Feature extraction is performed.

[0032] (1) in, , .in, For frame number, For the number of patches, The number of features is denoted as . The encoder parameters are frozen during training to ensure that the knowledge from the pre-training phase is fully preserved and to avoid catastrophic forgetting, thereby obtaining initial high-level feature representations for both modalities.

[0033] Step Two: In the feature alignment stage, frame-based alignment is performed on the extracted audio and video features. A frame-based feature recombination method is used to ensure consistency across different modalities over time. The result of frame-level audio feature alignment can be represented as:

[0034] (2) in, For frame alignment-based segmentation operations, This step removes the initial audio features of the global feature (class token). It enables fine-grained correspondence of cross-modal features, providing a reliable foundation for subsequent fusion learning.

[0035] Step 3: Input the aligned multimodal features into the fusion adapter module located after each layer of the encoder. Normalization and dimensionality reduction operations are required first.

[0036] (3) This module, based on a lightweight Mamba architecture, enables the interaction and fusion of audio and video features at different levels, thereby enhancing the ability to discriminate emotion-related features and effectively suppressing redundant and noisy information. Its formula is as follows:

[0037] (4) (5) (6) in, and pass The decomposition yields US, a dimensionality-up operation used to map features from the fusion space back to the representation space of their respective modalities. Then, through average pooling of audio class tokens and recombination of patch tokens, the feature processing for the current level is completed.

[0038] After completing multi-level feature fusion, the evolution of dynamic facial expressions over time can be modeled based on the class tokens in the final level of audio and video features, resulting in a temporal feature vector. This temporal feature vector is then input into a classifier to output the final emotion category prediction.

[0039] Figure 2 This is a flowchart illustrating a dynamic facial expression recognition method based on a self-supervised lightweight model according to the present invention. (Refer to...) Figure 2 This paper describes the visual feature extractor, audio spectrum feature extractor, feature recombination module, feature fusion encoder, multimodal temporal module, and training loss function proposed in this invention.

[0040] (1) Image Encoder visual feature extractor: In self-supervised learning-based facial expression analysis, in one or more embodiments of this invention, a pre-trained improved masked autoencoder (MAE) can be used as a visual feature extractor. This pre-trained model, through self-supervised reconstruction learning on a large-scale facial image dataset of approximately 2.17 million images, has been able to capture facial representations with strong generalization capabilities. Its core mechanism involves applying a high proportion of random masks to the input images during training. The encoder only processes unmasked image patches and extracts high-level semantic features, while the decoder reconstructs the masked regions based on the latent representations output by the encoder. In practical applications, this pre-trained model can be directly loaded as the encoder baseline model without requiring retraining. This encoder possesses robust reasoning capabilities for local occlusion (such as obscuring facial features) and can generate high-quality facial representations by combining global contextual information. These representations contain rich semantic information, including various features such as identity, expression, and pose, and can serve as an effective visual front-end for downstream facial expression classification tasks.

[0041] (2) Audio Encoder audio spectrum feature extractor: In audio feature extraction based on self-supervised learning, in one or more embodiments of this invention, a pre-trained improved audio mask autoencoder can be used as an audio spectral feature extractor. This model has been pre-trained on a large-scale audio dataset through a self-supervised reconstruction task, enabling it to learn audio representations with strong generalization capabilities. Its core pre-training strategy is to apply a high-proportion random mask to the input Mel spectrogram; the encoder only processes the unmasked visible blocks and extracts high-level semantic features, while the decoder reconstructs the masked regions based on the encoder output. This invention can directly load the pre-trained audio mask autoencoder as an audio baseline model without retraining the mask reconstruction. This encoder possesses the ability to model local correlations in the time-frequency domain, effectively capturing structured information in the spectrum thanks to the local window attention mechanism introduced during pre-training. The extracted audio representation contains rich semantic features and can serve as a general audio front-end representation, combined with images and videos, for direct use in downstream face emotion classification tasks.

[0042] Both the mask autoencoder and the audio mask autoencoder consist of stacked Transformer blocks of the same layer. Each layer of Transformer blocks in the mask autoencoder and the audio mask autoencoder are parallelized and sequentially connected to the feature recombination module and the Mamba model to form the improved mask autoencoder and the improved audio mask autoencoder.

[0043] In practical applications, Image Encoder and Audio Encoder can be feature extractors based on Vision Transformer. In the improved mask autoencoder, the Transformer block is used to extract the initial video features of the current level; in the improved audio mask autoencoder, the Transformer block is used to extract the initial audio features of the current level; the feature reorganization module is used to divide the local features in the initial audio features of the current level into multiple parts according to the number of frames of the video features of the current level, and add global features to each part to obtain multi-frame speech features. The multi-frame speech features and the initial video features are concatenated in the token dimension to obtain multimodal fusion features.

[0044] (3) Frame-Aligned Feature Reorganization Module: First, we need to obtain the initial audio feature matrix for the current level. and the initial video feature matrix Separate the global features (class token) and local features (patch token) in the initial audio features, denoted as:

[0045] (7) The patch token in the initial audio features is divided into M equal parts according to the number of frames of the current level video features. Then, the same class token is added to each patch token to form multi-frame speech features: (8) Multi-frame speech features and initial video features are linearly mapped to a unified dimension d to obtain: (9) Here, DS is a dimensionality reduction operation. Subsequently, the aligned speech features and the initial video features are concatenated along the token dimension to obtain the multimodal fusion features: (10) In the formula, For the first Frame speech features The first of the audio features Patch token, For the class token in the audio features, For multi-frame speech features after unifying dimensions, This indicates a dimensionality reduction operation. This indicates normalization processing. For video features, The initial video features after unifying dimensions, Indicates splicing, This is a multimodal fusion feature.

[0046] (4) Mamba-Fusion Adapter—a feature fusion module based on Mamba: The Mamba-Fusion Adapter utilizes State Space Models, a pure SSM approach that demonstrates great potential in modeling long sequences, handling them better than models such as Transformer.

[0047] The multimodal fusion features are fused and encoded using the Mamba module to obtain the output representation: (11) Furthermore, the output can be separated according to modality, with the first part being speech features. The latter part consists of video features. Each feature is then restored to its original dimension using a reverse mapping layer.

[0048] (12) Here, US is a dimension-up operation used to map the features of the fusion space back to the representation space of their respective modalities. The updated speech features are obtained through cross-modal interaction fusion. These are the video features updated through cross-modal interactive fusion.

[0049] For speech modalities, an average pooling operation is also performed on all class tokens of the speech features to obtain a global semantic representation. Subsequently, patches were applied to all speech features. The concatenation operation is performed and combined with the pooled class token to obtain the final audio features of the current level after cross-modal interaction fusion and update.

[0050] (13) (14) In the formula, The current level of speech features is updated through cross-modal interaction fusion. This refers to the current level video features updated through cross-modal interactive fusion.

[0051] This adapter, through modal mapping, fusion, and recombination, can achieve efficient interaction of cross-modal information while ensuring the independence of each modal feature, thereby improving the modeling capability and multimodal fusion effect of long sequence tasks.

[0052] (5) Multimodal Temporal Module (MTM): The MTM module extracts class labels from the final layer of video and audio features, fuses them with location information, and feeds them into the Transformer Encoder for feature learning. MTM aims to capture dynamic dependencies across time dimensions and learn powerful facial expression representations for real-world scenes.

[0053] Specifically, in one or more embodiments of the present invention, the server may first determine the frame-level representation of the class token of the final-level video features, copy the class token of the final-level audio features according to the number of frames in the frame-level representation, and fuse it with the frame-level representation of the class token of the final-level video features and the embedding of each frame position to obtain the input sequence features; then, the Transformer Encoder models the evolution of dynamic facial expressions in the time dimension based on the input sequence features to obtain the temporal feature vector.

[0054] For example, after feature extraction and feature alignment fusion at the final level, the final representation with both video and audio features can be obtained. and First, from Extract video class token ,from Extract audio class token .Will Proposed as a frame-level representation:

[0055] (15) Then, and Added together, because There is only one, so a copy operation is needed using repeat and Frame count Corresponding to, and with, a newly generated learnable class tag. Combined, they form a bimodal time fusion characteristic. : (16) This fusion feature Further through location embedding To enrich and form features of the input sequence : (17) in, This is used to encode temporal features by fusing the relative position information of frames in the sequence, enabling the model to capture inter-frame order and temporal dependencies. Features The input is fed into the Transformer Encoder for temporal modeling and cross-modal interaction. During the multi-head self-attention stage, the input is... Apply layer normalization to obtain normalized features. Then, the output of the multi-head self-attention head is connected to the residual connection... Combining to produce intermediate features :

[0056] (18) In the MLP stage, Layer normalization is performed to obtain The output of the MLP is connected to another residual connection. Combining, to produce the final feature :

[0057] (19) After using the MTM module, Final classification is then performed. Sentiment category is predicted using a linear layer. :

[0058] (20) (6) Cross-EntropyLoss function: Dynamic facial expression recognition is essentially a multi-class classification problem that determines the facial expression state in a video. Therefore, after extracting video frame features using VisionMamba and inputting them into the temporal Mamba module, a multi-class fully connected layer is introduced. The Cross-Entropy Loss function is used to measure the classification error generated during training. The function is shown in the following equation:

[0059] (twenty one) in, It is the number of emotion categories. For the true labels of the samples, It is the first The probability results of each category prediction.

[0060] based on Figure 1 The dynamic facial expression recognition method described herein introduces frame-aligned feature fusion and recombination during the multi-layered video and audio feature extraction process. This achieves cross-modal temporal consistency between video and audio, thereby capturing more fine-grained dynamic facial expression changes during the feature extraction stage. In the multi-level feature extraction and fusion stage, a lightweight Mamba module is used to perform fusion interaction at each level after feature extraction encoding alignment. This enables cross-modal interaction at different levels in the overall multi-level feature extraction stage, enhancing the discriminative power and robustness of the features. When performing dynamic facial expression recognition based on the final level's audio and video features, the accuracy and robustness of dynamic facial expression recognition are improved.

[0061] The method of this invention has a clear flow, a reasonable structural design, low parameter count, and is easy to implement, significantly reducing training overhead and dependence on large-scale labeled data. By freezing the audio spectral feature extractor and visual feature extractor during the pre-training stage, this invention effectively preserves existing knowledge and avoids catastrophic forgetting caused by full-parameter fine-tuning. The introduction of frame-aligned feature reorganization in the feature alignment stage achieves cross-modal temporal consistency, thereby capturing more fine-grained dynamic facial expression changes. In the fusion stage, a lightweight Mamba fusion adapter is embedded after each encoder layer, enabling the model to achieve cross-modal interaction at different levels, enhancing the discriminativeness and robustness of features.

[0062] On the DFEW dataset, this invention achieves significantly better results than the baseline and full-parameter fine-tuning schemes with only an increase of approximately 3.3M trainable parameters. Both unweighted average recall (UAR) and weighted average recall (WAR) are substantially improved, by 0.57 and 1.23 respectively compared to traditional full-parameter fine-tuning methods. The number of trainable parameters is reduced by 6 to 30 times compared to currently known methods, demonstrating the dual advantages of this invention in parameter efficiency and performance. Compared to traditional full-parameter training methods, this invention not only has lower computational and storage costs but also maintains stable recognition performance even when facing class imbalance and noise interference, making it suitable for widespread application in real-world scenarios.

[0063] When applying the dynamic facial expression recognition method provided by this invention, it is not necessary to rely on... Figure 1 The steps shown are executed in sequence. The specific execution order of each step can be determined as needed, and this invention does not impose any restrictions on it.

[0064] The above describes a dynamic facial expression recognition method provided by one or more embodiments of the present invention. Based on the same idea, the present invention also provides a corresponding dynamic facial expression recognition device, such as... Figure 3 As shown.

[0065] Figure 3 A schematic diagram of a dynamic facial expression recognition device provided by the present invention includes: The acquisition module 201 is used to acquire video and audio data obtained from the facial capture of the target user, and to perform multi-level feature fusion on the video and audio data. The extraction module 202 is used to extract the initial video features and initial audio features including global and local features of the current level based on the input features in each level of feature fusion; the input features in the first level of feature fusion are video data and audio data, and the input features in the subsequent levels of feature fusion are the output features of the previous level; Alignment module 203 is used to divide the local features in the initial audio features of the current level into multiple parts according to the number of frames of the video features of the current level, and add global features to each part to obtain multi-frame speech features. The multi-frame speech features and the initial video features are concatenated in the token dimension to obtain multimodal fusion features. The fusion module 204 is used to fuse and encode the multimodal fusion features through the Mamba model and reverse the feature dimension to obtain the audio and video features of the current level after cross-modal interactive fusion and update, thus completing the feature fusion of the current level. The recognition module 205 is used to model the evolution of dynamic facial expressions in the time dimension based on the global features in the audio and video features of the last level, to obtain a temporal feature vector, and to classify emotions based on the temporal feature vector to obtain the facial expression emotion category of the target user.

[0066] Specific limitations regarding the dynamic facial expression recognition device can be found in the limitations of the dynamic facial expression recognition method described above, and will not be repeated here. Each module in the aforementioned dynamic facial expression recognition device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device in hardware form, or stored in the memory of a computer device in software form, so that the processor can call and execute the corresponding operations of each module.

[0067] The present invention also provides a computer-readable storage medium storing a computer program that can be used to execute the above-described... Figure 1 The provided method for dynamic facial expression recognition.

[0068] The present invention also provides Figure 4 The schematic diagram of the computer device shown is as follows: Figure 4As shown, at the hardware level, this computer device includes a processor, internal bus, network interface, memory, and non-volatile memory, and may also include other hardware required for business operations. The processor reads the corresponding computer program from the non-volatile memory into memory and then executes it to achieve the above. Figure 1 The provided method for dynamic facial expression recognition.

[0069] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the methods described above. Any references to memory, storage, databases, or other media used in the embodiments provided by this invention can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, or optical storage, etc. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM), etc.

[0070] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this invention.

Claims

1. A dynamic facial expression recognition method, characterized in that, The method comprises the following steps: obtaining video data and audio data collected from a target user's face, and performing multi-level feature fusion on the video data and the audio data; in each level of feature fusion, initial video features of the current level and initial audio features including global features and local features are extracted from input features; in the first level of feature fusion, the input features are the video data and the audio data, and in subsequent levels of feature fusion, the input features are the output features of the previous level; the local features in the initial audio features of the current level are evenly divided into multiple parts according to the frame number of the video features of the current level, and each part is added with global features to obtain multiple frames of speech features, and the multiple frames of speech features and the initial video features are spliced in the token dimension to obtain multi-modal fusion features; the multi-modal fusion features are fused and encoded by a Mamba model and are inversely restored in the feature dimension to obtain audio features and video features of the current level which are updated through cross-modal interaction fusion, and the feature fusion of the current level is completed; the evolution process of the dynamic facial expression in the time dimension is modeled according to the global features in the audio features and the video features of the last level to obtain a time sequence feature vector, and emotion classification is performed according to the time sequence feature vector to obtain a facial expression emotion category of the target user.

2. The dynamic facial expression recognition method of claim 1, wherein, The local features in the initial audio features of the current level are evenly divided into multiple parts according to the frame number of the video features of the current level, and each part is added with global features to obtain multiple frames of speech features, and the multiple frames of speech features and the initial video features are spliced in the token dimension to obtain multi-modal fusion features, which specifically comprises: The local features in the initial audio features of the current level are evenly divided into multiple parts according to the frame number of the video features of the current level, and each part is added with global features to obtain multiple frames of speech features, and the multiple frames of speech features and the initial video features are spliced in the token dimension to obtain multi-modal fusion features, which specifically comprises: ； wherein, is the first frame speech feature, is the local feature split of the initial audio feature at the current level, is the patch token, is the class token in the audio feature, denotes concatenation.

3. The dynamic facial expression recognition method of claim 1, wherein, The multiple frames of speech features and the initial video features are unified in dimension by linear mapping respectively by the following formula: The multiple frames of speech features and the initial video features are spliced in the token dimension to obtain multi-modal fusion features by the following formula: ， ； The evolution process of the dynamic facial expression in the time dimension is modeled according to the global features in the audio features and the video features of the last level to obtain a time sequence feature vector, which specifically comprises: ； wherein, is the first frame speech feature, is the multi-frame speech feature after uniform dimension, represents a dimension reduction operation, represents a normalization processing, is the video feature, is the initial video feature after uniform dimension, represents splicing, is the multi-modal fusion feature.

4. The dynamic facial expression recognition method of claim 1, wherein, The frame-level representation of the global features of the video features of the last level is determined, the global features of the audio features of the last level are copied according to the frame number of the frame-level representation, and are fused with the frame-level representation and the frame position embedding of the global features of the video features of the last level to obtain input sequence features; The evolution process of the dynamic facial expression in the time dimension is modeled according to the input sequence features by a Transformer Encoder to obtain a time sequence feature vector. The video data and the audio data are fused by a pre-trained improved mask autoencoder as a visual feature extractor and a pre-trained improved audio mask autoencoder as an audio spectrum feature extractor.

5. The dynamic facial expression recognition method of claim 1, wherein, ​ The mask autoencoder and the audio mask autoencoder each include stacked same layers of Transformer blocks, and the mask autoencoder and the audio mask autoencoder are each parallel and sequentially connected with a feature reorganization module and a Mamba model to form an improved mask autoencoder and an improved audio mask autoencoder; In the feature fusion of each level, the Transformer block in the improved mask autoencoder is used to extract initial video features of the current level; The Transformer block in the improved audio mask autoencoder is used to extract initial audio features of the current level; The feature reorganization module is used to divide local features in the initial audio features of the current level into multiple parts according to the number of frames of the video features of the current level, and each part is added with global features to obtain multiple frames of speech features, and the multiple frames of speech features and the initial video features are spliced in the token dimension to obtain multi-modal fusion features; The evolution process of the dynamic facial expression in the time dimension is modeled according to the global features in the audio features and the video features of the last level through the multi-modal temporal module to obtain a temporal feature vector; The emotion classification of the target user is obtained through the classifier according to the temporal feature vector.

6. The dynamic facial expression recognition method of claim 5, wherein, The Mamba model, the multi-modal temporal module and the classifier are trained under the premise of freezing the visual feature extractor and the audio spectrum feature extractor.

7. A dynamic facial expression recognition apparatus, characterized by, Comprise: An acquisition module is configured to acquire video data and audio data collected from a target user's face, and perform multi-level feature fusion on the video data and the audio data, respectively; An extraction module is configured to extract initial video features and initial audio features including global features and local features of a current level according to input features in the feature fusion of each level; the input features in the feature fusion of the first level are the video data and the audio data, and the input features in the feature fusion of subsequent levels are output features of a previous level; An alignment module is configured to divide local features in the initial audio features of the current level into multiple parts according to the number of frames of the video features of the current level, and add global features to each part to obtain multiple frames of speech features, and splice the multiple frames of speech features and the initial video features in the token dimension to obtain multi-modal fusion features; A fusion module is configured to fuse and encode the multi-modal fusion features through the Mamba model and reversely restore the features in the feature dimension to obtain audio features and video features of the current level updated through cross-modal interaction fusion, and complete the feature fusion of the current level; An identification module is configured to model the evolution process of a dynamic facial expression in the time dimension according to global features in audio features and video features of the last level to obtain a temporal feature vector, and perform emotion classification according to the temporal feature vector to obtain a facial expression emotion category of a target user.

8. A computer-readable storage medium, characterized in that, The storage medium stores a computer program, and the computer program is executed by the processor to implement the method of any one of claims 1-6. The storage medium stores a computer program, and the computer program is executed by the processor to implement the method of any one of claims 1-6.

9. A computer device, comprising: A computer program product comprising a memory, a processor and a computer program stored on the memory and loadable on the processor, the processor implementing the method according to any one of claims 1 to 6 when executing the computer program.

Images