Bimodal emotion recognition method based on adaptive feature fusion
By employing an adaptive feature fusion-based bimodal emotion recognition method that combines facial and gait features, the shortcomings of single-modal methods in environmental sensitivity and complex scenarios are addressed, achieving higher accuracy and robustness.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-12-17
- Publication Date
- 2026-04-17
AI Technical Summary
Existing single-modal emotion recognition methods are sensitive to the environment in facial expression recognition, have poor accuracy in gait emotion recognition in complex scenes, and have unstable modality fusion, making it difficult to perform end-to-end training on multi-source datasets.
A dual-modal emotion recognition method based on adaptive feature fusion is adopted. Features are extracted by facial feature extraction module and gait feature extraction module respectively, and dynamic weighted fusion is performed by adaptive feature fusion module, combined with emotion classification module for emotion recognition.
It improves the accuracy and stability of emotion recognition, enhances the coordination between features and the generalization performance of the model, and maintains the robustness of recognition performance in different environments.
Smart Images
Figure CN121884407A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of facial expression recognition technology, and in particular to a bimodal emotion recognition method based on adaptive feature fusion. Background Technology
[0002] Existing research on emotion recognition largely focuses on the visual feature analysis of facial images. Typical methods are based on convolutional neural networks (CNNs) or visual Transformer models, which classify and recognize emotions such as anger, happiness, surprise, and sadness by modeling local features and global semantics of facial images. In recent years, models such as Poster V2, ResNet, and VGG-Face have achieved high accuracy on public datasets (such as RAF-DB and FER2013). However, these methods mainly rely on facial expression information and are quite sensitive to external factors such as lighting, occlusion, and pose changes; when the subject's expression is not obvious or there is occlusion, the model performance will significantly decrease.
[0003] Gait, as a non-contact biometric feature, can provide clues to emotional states in situations where facial expressions are absent or at a distance. Existing research often employs methods such as Graph Convolutional Networks (GCNs) and Temporal Convolutional Networks (TCNs) to extract gait features from the dynamic changes of skeleton point sequences or pose key points. Typical methods, such as InfoGCN, capture the spatiotemporal correlation features of gait through information propagation mechanisms, thereby identifying an individual's emotion category. However, the expressive power of gait features is limited by motion capture accuracy and individual differences, is sensitive to the different physical characteristics of different subjects, and lacks the ability to characterize fine-grained emotional signals such as facial expressions. Models relying solely on gait data still have insufficient emotion recognition capabilities in complex scenarios.
[0004] To overcome the limitations of unimodal methods, recent research has begun to explore multimodal emotion recognition, which integrates multi-source data such as facial images, speech, posture, and gait to comprehensively analyze emotional features. Mainstream methods often employ feature concatenation or attention-weighted fusion, but due to the significant differences in feature dimensions across modalities and their complex correlations, simple fusion often leads to feature redundancy or information loss. Furthermore, existing multimodal datasets are mainly concentrated in "face + speech" scenarios, making it difficult to support end-to-end model training. Summary of the Invention
[0005] To overcome the shortcomings of existing technologies, such as facial expression recognition being overly sensitive to the environment, poor accuracy of gait emotion recognition in complex scenarios, and unstable modality fusion, this invention proposes a dual-modal emotion recognition method based on adaptive feature fusion. This method integrates facial features and gait features for emotion recognition, thereby improving the accuracy of expression recognition.
[0006] This invention proposes a bimodal emotion recognition method based on adaptive feature fusion. First, a bimodal emotion model is trained on a dataset {facial image data, gait sequence data; emotion label}. The bimodal emotion recognition model extracts facial features and gait features based on facial expression images and gait sequence data, respectively. Then, the two are fused to obtain fused features, and the emotion category is identified based on the fused features. During emotion recognition, gait sequence data and facial expression images are extracted from the data of the person being analyzed and input into a bimodal emotion model to identify the emotion category.
[0007] Preferred bimodal emotion models include: The facial feature extraction module is used to process facial image data and extract facial features; The gait feature extraction module is used to process gait sequence data and extract gait features; The adaptive feature fusion module is used to dynamically weight and fuse facial features and gait features to obtain fused features; The emotion classification module classifies emotions based on fused features.
[0008] Preferably, the adaptive feature fusion module includes a sequentially connected dimension alignment unit, a weight generation unit, and a weighted fusion unit; The dimension alignment unit performs dimension alignment on the facial features and gait features to obtain aligned facial vectors and gait vectors; The weight generation unit first uses a multilayer perceptron to process the concatenated vector of the face vector and gait vector to generate weight scores for gait features and face features; then the two weight scores are normalized using the Softmax function to obtain gait weights and face weights. The weighted fusion unit uses gait weights and facial weights to sum the gait features and facial features to generate fused features.
[0009] Preferably, the training method for the bimodal emotion model includes the following steps: S1. Construct a bimodal dataset {facial image data, gait sequence data; emotion labels}; S2. Instruct the bimodal emotion model to learn from the bimodal dataset to train the adaptive feature fusion module and the emotion classification module until the model converges.
[0010] Preferably, the method for constructing the bimodal dataset in step S1 is as follows: first, obtain a facial image dataset {facial image data, emotion label} and a gait sequence dataset {gait sequence data; emotion label}; then, select the corresponding emotion labels from the facial image dataset and the gait sequence dataset as the emotion labels of the bimodal dataset; based on the emotion labels of the bimodal dataset, select the corresponding facial image data and gait sequence data from the facial image dataset and the gait sequence dataset to construct the bimodal dataset.
[0011] Preferably, the facial image dataset uses the AffectNet dataset, and the gait sequence dataset uses the EWalk dataset.
[0012] Preferably, the facial feature extraction module and the gait feature extraction module adopt the pre-trained Poster V2 model and the Info GCN model, respectively.
[0013] Preferably, the emotion classification module can specifically use Transformer.
[0014] The present invention proposes a bimodal emotion recognition system based on adaptive feature fusion, comprising a memory and a processor. The memory stores a computer program, and the processor is connected to the memory. The processor is used to execute the computer program to realize the bimodal emotion recognition method based on adaptive feature fusion.
[0015] The present invention proposes a storage medium storing a computer program, which, when executed, is used to implement the aforementioned bimodal emotion recognition method based on adaptive feature fusion.
[0016] The advantages of this invention are: (1) The present invention proposes a dual-modal emotion recognition method based on adaptive feature fusion. First, a dual-channel feature extraction architecture is built by parallel facial feature extraction module and gait feature extraction module to realize the joint extraction of emotion features of different modalities. It can comprehensively utilize the information of dynamic behavior features and static facial expression features to improve the accuracy and stability of emotion recognition and overcome the problem of insufficient single-modal feature information.
[0017] (2) This invention designs an adaptive feature fusion module that aligns gait features and facial features in the same dimensional space and achieves adaptive feature fusion, thereby improving the coordination between features, feature selection ability, and generalization performance of the model. This invention retrains the model on a self-constructed bimodal dataset, enabling the feature extraction network and fusion module to be optimized collaboratively, thereby improving the overall accuracy and robustness of emotion recognition.
[0018] (3) In this invention, the adaptive feature fusion module utilizes the multilayer perceptron (MLP) and Softmax mechanism to automatically adjust the fusion weights of gait and facial features according to the modal feature quality of the input sample, thereby maintaining high recognition performance under different environmental conditions.
[0019] (4) In this invention, the effectiveness of fusion calculation is ensured by aligning the dimensions of facial features and gait features, and the semantic consistency of feature expression is improved.
[0020] (5) The present invention adopts a Transformer-based emotion classification module, models the global dependency relationship between features through a multi-head self-attention mechanism, fully explores the contextual information of multimodal fusion features, and further improves the generalization ability and robustness of emotion recognition. Attached Figure Description
[0021] Figure 1 This is a structural diagram of a bimodal emotion model proposed in this invention; Figure 2 for Figure 1 Simplified structural diagram of the adaptive feature fusion module; Figure 3 This is a flowchart of a dual-modal emotion recognition method based on adaptive feature fusion proposed in this invention. Detailed Implementation
[0022] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present invention.
[0023] This embodiment proposes a bimodal emotion recognition model based on adaptive feature fusion, referred to as the bimodal emotion model, which identifies emotions based on facial image data and gait sequence data. Reference Figure 1 This bimodal emotion model includes: The facial feature extraction module is used to process facial image data and extract facial features; specifically, the Poster V2 model can be used. The gait feature extraction module is used to process gait sequence data (i.e., gait skeleton point sequence) and extract gait features; specifically, the Info GCN model can be used. The adaptive feature fusion module is used to dynamically weight and fuse facial features and gait features to obtain fused features; The emotion classification module classifies emotions based on fused features.
[0024] In this embodiment, the gait feature extraction module generates a depth feature vector representing gait features based on the gait skeleton sequence (i.e., gait data), and the facial feature extraction module generates a depth feature vector representing facial features based on the facial image data corresponding to the gait data.
[0025] Specifically, the Poster V2 model, as a facial feature extraction module, extracts local texture features and global semantic information of the face region through multi-layer convolution and attention mechanisms, and can effectively capture features of emotion-related regions such as expression changes, eye and mouth corner details.
[0026] The Info GCN model, as a gait feature extraction module, extracts the spatiotemporal features of gait sequences. The Info GCN model models the gait keypoint sequence as temporal graph structure nodes, capturing the dynamic relationships and posture change patterns between joints through an information propagation mechanism, thereby obtaining dynamic features reflecting emotional states.
[0027] The Poster V2 model and the Info GCN model are loaded with publicly available pre-trained weights, which are not retrained in this invention but used only as fixed feature extractors to ensure the stability and universality of the extracted features. In this way, the model can utilize the representational capabilities of existing mature networks, avoid redundant training, and improve overall training efficiency and stability. The model training phase mainly focuses on parameter optimization for the subsequent adaptive feature fusion module, achieving adaptive alignment and fusion of cross-modal features through learning the fusion weights.
[0028] Facial features have a higher dimensionality than gait features, providing a multi-layered source of feature information for the subsequent adaptive feature fusion module.
[0029] Reference Figure 2 The adaptive feature fusion module includes sequentially connected dimension alignment units, weight generation units, and weighted fusion units.
[0030] Dimensional alignment units are used to align high-dimensional facial features. Dimensionality reduction and gait features Compared with the dimensionality reduction facial features Activation is performed to ensure the consistency of facial features and gait features in numerical distribution and nonlinear feature space.
[0031] Facial features after dimensionality reduction Gait characteristics The dimensions are the same to ensure consistency of feature dimensions during fusion.
[0032] Assuming, , The formula for processing dimension alignment units is expressed as follows: ; in, The parameter matrix is a learnable matrix that is automatically updated via backpropagation during model training to adaptively extract facial features. The most discriminative semantic information in the middle. It is a learnable bias vector used to translate the transformation results, enabling the model to fit the data distribution more flexibly.
[0033] The activation operation of dimension-aligned units can use the ReLU activation function, and the operation process is as follows: ; in, Gait features The gait vector is obtained after activation processing. Facial features The facial vector obtained after activation processing; .
[0034] The ReLU activation function can effectively suppress abnormal negative values, increase the nonlinear expressive power of the model, and ensure that the two features are in the same positive domain space before fusion, thereby improving the numerical stability and feature comparability during fusion.
[0035] The weight generation unit includes a multilayer perceptron (MLP) and a softmax activation function.
[0036] The weight generation unit first generates the gait vector and facial vectors Align and concatenate to obtain the concatenated vector z. Then, the concatenated vector z is processed by a multilayer perceptron to generate a score array s, which includes the weighted scores of the gait features. Weighted score of facial features The formula is expressed as: ; in, It is a non-linear activation function (such as ReLU). , For the intermediate hidden layer dimension; and As weight, and This is a bias term.
[0037] The Softmax function normalizes the score array s to obtain the final fusion weights. , For gait weights, Facial weighting; ; ; Where e is the natural base.
[0038] Softmax normalization ensures that the weights are between 0 and 1, and that the sum is always 1, making the weights interpretable and balanced. When the facial features of the input sample are affected by factors such as lighting, occlusion, or blurring, leading to a decrease in feature quality, the distribution of the high-dimensional features extracted by Poster V2 in the feature space will deviate from the high-confidence region of the training samples, thus affecting the weight score of the facial modality. The relative reduction is achieved through backpropagation. During training, the model automatically learns the correspondence between this distribution change and the classification results. When facial modalities fail to provide effective discriminative information, the gait weight scores generated by the multilayer perceptron (MLP) are used. The gait weights will increase relatively, and after Softmax normalization, higher gait weights are obtained. This allows gait features to account for a larger proportion of the fusion result, enabling adaptive adjustment of modal reliability.
[0039] Weighted fusion units are used to process gait features and facial features Perform a weighted summation to generate a unified fusion feature. The formula is expressed as follows: ; Fusion features By integrating dynamic structural information from gait with static semantic information from the face, this method possesses stronger emotional expression and discriminative capabilities. This fused feature serves as input to the subsequent emotion classification module for performing emotion recognition tasks.
[0040] The emotion classification module can specifically adopt the Transformer classification module to capture the global dependencies and semantic interactions between different feature dimensions at a higher level.
[0041] The Transformer classification module utilizes a multi-head self-attention mechanism to fuse features. Global information modeling is performed, and the interaction relationships between features are learned in parallel from multiple attention subspaces, enabling the model to more comprehensively understand the potential emotional information in the fused features. Compared with traditional convolutional or recurrent neural networks, the Transformer structure can simultaneously consider long-range dependencies between features and multimodal semantic alignment, thereby enhancing the discriminativeness and stability of feature representation.
[0042] In practical implementation, fusion features After passing through the Transformer encoding layer, the extracted global contextual semantic representation is input into the linear mapping layer and the Softmax classification layer to output the final emotion label (such as "angry", "happy", "neutral", "sad"). The Transformer module does not require additional modality alignment operations during this process, allowing for direct deep semantic modeling of the fused features, thereby further improving the model's accuracy and generalization ability in multimodal emotion recognition tasks. In this way, the system can better cope with situations where single-modal information is missing or feature noise is high, maintaining the robustness and consistency of the recognition results.
[0043] In the bimodal emotion model, the facial feature extraction module and the gait feature extraction module use the pre-trained Poster V2 model and the Info GCN model, respectively; only the adaptive feature fusion module and the emotion classification module are updated during the bimodal emotion model process.
[0044] Specifically, training a bimodal emotion model includes the following steps: S1. Construct a bimodal dataset {facial image data, gait sequence data; emotion labels}.
[0045] Specifically, in this step, we can first obtain a facial image dataset {facial image data, emotion labels} and a gait sequence dataset {gait sequence data; emotion labels}, and then fuse the facial image dataset and the gait sequence dataset based on the emotion labels to obtain a bimodal dataset.
[0046] For example, in subsequent embodiments, the facial image dataset uses the AffectNet dataset, which contains eight discrete emotion categories: neutral, as well as anger, disgust, fear, happiness, sadness, surprise, and contempt. The gait sequence dataset uses the EWalk dataset, which mainly covers four emotion categories: happy, angry, sad, and neutral.
[0047] Thus, in this embodiment, four types of samples—happy, angry, sad, and neutral—are selected from the AffectNet dataset, consistent with the gait emotion label, and then aligned with the EWalk label of the gait sequence dataset to obtain a bimodal dataset.
[0048] In this embodiment, the sample data of each type selected from the AffectNet dataset are balanced to ensure the consistency and comparability of multimodal data in terms of emotion categories.
[0049] S2. Instruct the bimodal emotion model to learn from the bimodal dataset to train the adaptive feature fusion module and the emotion classification module until the model converges.
[0050] The specific convergence condition for the model can be set as follows: the number of model updates reaches a set value, or the model's loss function converges.
[0051] The loss function used in the model training process can be various classification losses such as mean squared error loss or cross-entropy loss.
[0052] Reference Figure 3 This embodiment proposes a bimodal emotion recognition method based on adaptive feature fusion. First, data of the person to be identified is acquired, and facial image data and gait sequence data are extracted respectively. The extracted facial image data and gait sequence data are processed by a bimodal emotion model, and the emotion category is output.
[0053] The bimodal emotion model described above will be validated in the following specific embodiments.
[0054] In this embodiment, four comparison algorithms are used to compare and verify the bimodal emotion model proposed in this invention (hereinafter referred to as the method of this invention).
[0055] Compared to Algorithm 1 (Poster V2 model), which identifies emotion categories based on facial images; Compare Algorithm 2 (Info GCN model), which identifies emotion categories based on gait sequence data; Compared to Algorithm 3 (feature splicing and fusion), the Poster V2 model is used to extract facial features from facial images, and the Info GCN model is used to extract gait features from gait sequence data. Facial features are dimensionality reduced to ensure that the dimensions of facial features and gait features are consistent. Then, the facial features and gait features are input into the Transformer classification module, and the Transformer classification module outputs the emotion category. Compared with Algorithm 4 (equal weight fusion), the Poster V2 model is used to extract facial features from facial images, and the InfoGCN model is used to extract gait features from gait sequence data. Facial features are dimensionality reduced to ensure that the dimensions of facial features and gait features are consistent. Then, the facial features and gait features are weighted and fused and input into the Transformer classification module. The Transformer classification module outputs the emotion category. The weights of facial features and gait features are both 0.5.
[0056] In this embodiment, the model of the present invention and each comparison algorithm are trained on the corresponding training dataset, and the accuracy of the algorithms is tested on the test dataset. The results are shown in Table 1 below.
[0057] Table 1: Performance Comparison of Different Models ; The data comparison above shows that the method of the present invention improves the accuracy by nearly 10% compared with the existing commonly used facial image emotion recognition models; and improves the accuracy by 5.3% compared with the fixed-weight bimodal fusion algorithm (equal-weight fusion).
[0058] Of course, those skilled in the art will recognize that the present invention is not limited to the details of the exemplary embodiments described above, but also includes the same or similar structures that can be implemented in other specific forms without departing from the spirit or essential characteristics of the invention. Therefore, the embodiments should be considered illustrative and non-limiting in all respects, and the scope of the invention is defined by the appended claims rather than the foregoing description. Thus, all variations falling within the meaning and scope of equivalents of the claims are intended to be included within the present invention. No reference numerals in the claims should be construed as limiting the scope of the claims.
[0059] Furthermore, it should be understood that although this specification describes embodiments, not every embodiment contains only one independent technical solution. This narrative style is merely for clarity. Those skilled in the art should consider the specification as a whole, and the technical solutions in each embodiment can also be appropriately combined to form other embodiments that can be understood by those skilled in the art.
[0060] The technologies, shapes, and structures not described in detail in this invention are all known technologies.
Claims
1. A dual-modal emotion recognition method based on adaptive feature fusion, characterized in that, First, a bimodal emotion model is trained on the dataset {facial image data, gait sequence data; emotion label}. The bimodal emotion recognition model extracts facial features and gait features based on facial expression images and gait sequence data, respectively. Then, the two are fused to obtain fused features, and the emotion category is identified based on the fused features. During emotion recognition, gait sequence data and facial expression images are extracted from the data of the person being analyzed and input into a bimodal emotion model to identify the emotion category.
2. The dual modal emotion recognition method based on adaptive feature fusion as claimed in claim 1, wherein, Bimodal emotion models include: The facial feature extraction module is used to process facial image data and extract facial features; The gait feature extraction module is used to process gait sequence data and extract gait features; The adaptive feature fusion module is used to dynamically weight and fuse facial features and gait features to obtain fused features; The emotion classification module classifies emotions based on fused features.
3. The dual modal emotion recognition method based on adaptive feature fusion as claimed in claim 2, wherein, The adaptive feature fusion module includes sequentially connected dimension alignment units, weight generation units, and weighted fusion units; The dimension alignment unit performs dimension alignment on facial features and gait features to obtain aligned facial vectors and gait vectors; The weight generation unit first uses a multilayer perceptron to process the concatenated vector of the face vector and gait vector to generate weight scores for gait features and face features; then the two weight scores are normalized using the Softmax function to obtain gait weights and face weights. The weighted fusion unit uses gait weights and facial weights to sum the gait features and facial features to generate fused features.
4. The dual modal emotion recognition method based on adaptive feature fusion as claimed in claim 2, wherein, The training method for a bimodal emotion model includes the following steps: S1. Construct a bimodal dataset {facial image data, gait sequence data; emotion labels}; S2. Instruct the bimodal emotion model to learn from the bimodal dataset to train the adaptive feature fusion module and the emotion classification module until the model converges.
5. The dual modal emotion recognition method based on adaptive feature fusion as claimed in claim 4, wherein, The method for constructing the bimodal dataset in step S1 is as follows: First, obtain the facial image dataset {facial image data, emotion labels} and the gait sequence dataset {gait sequence data; emotion labels}; then, select the corresponding emotion labels from the facial image dataset and the gait sequence dataset as the emotion labels for the bimodal dataset. Based on the emotion labels of the bimodal dataset, corresponding facial image data and gait sequence data are selected from the facial image dataset and gait sequence dataset to construct the bimodal dataset.
6. The dual modal emotion recognition method based on adaptive feature fusion as claimed in claim 5, wherein, The facial image dataset uses the AffectNet dataset, and the gait sequence dataset uses the EWalk dataset.
7. The dual-modal emotion recognition method based on adaptive feature fusion as described in claim 4, characterized in that, The facial feature extraction module and the gait feature extraction module use the pre-trained Poster V2 model and the Info GCN model, respectively.
8. The dual modal emotion recognition method based on adaptive feature fusion as claimed in claim 2, wherein, The emotion classification module can specifically use Transformer.
9. A dual-modal emotion recognition system based on adaptive feature fusion, characterized in that, It includes a memory and a processor, wherein the memory stores a computer program, the processor is connected to the memory, and the processor is used to execute the computer program to implement the bimodal emotion recognition method based on adaptive feature fusion as described in any one of claims 1-8.
10. A storage medium, characterized by A computer program is stored, and when executed, is used to implement the adaptive feature fusion based bimodal emotion recognition method according to any one of claims 1-8.