Facial expression recognition method based on double structured attention

Abstract

The invention discloses a facial expression recognition method based on double structured attention. According to the method, a backbone network is extracted through parallel image and key point features, multi-scale bimodal features are obtained, and the core is a double hierarchical structured attention mechanism. The first adaptive bimodal fusion module performs intelligent weighted selection on bimodal features at each scale by means of a channel-space attention structure, dynamically adjusts the modal contribution degree, and deals with modal information uncertainty caused by low illumination, shielding and the like. And the second cascade cross-scale fusion module globally aggregates fusion features of each scale through a specific to abstract cascade attention structure, introduces strong inductive bias, reduces calculation complexity and guarantees context consistency. The method overcomes the problems of fusion strategy coupling, complex calculation, lack of structured guidance and the like in the prior art, and significantly improves the accuracy and efficiency of facial expression recognition.

Description

Technical Field

[0001] This invention belongs to the field of computer vision and artificial intelligence technology, and specifically relates to a deep learning method for facial expression recognition, especially a multimodal, multi-scale feature fusion method based on dual structured attention. Background Technology

[0002] Facial Expression Recognition (FER) is a key technology for realizing applications such as human-computer interaction and affective computing. Its core challenge lies in how to extract discriminative features from images that are robust to variables irrelevant to individual identity, lighting, and pose, while being highly sensitive to subtle changes in facial expressions.

[0003] To address this challenge, fusing image and facial landmark information has become an effective strategy. However, designing an efficient fusion mechanism remains an open question. Existing methods, such as POSTER V2, while achieving state-of-the-art performance through Transformers, still have limitations in their fusion strategies: 1) Their scale fusion relies on global self-attention after stitching, and the computational complexity is related to the square of the input resolution (…). 1) Poor scalability; 2) Its modal interactions use fixed cross-attention, which cannot dynamically adjust modal contributions based on the quality of the input image. These problems limit further breakthroughs in the performance and efficiency of the FER model. Summary of the Invention

[0004] The purpose of this invention is to provide a novel facial expression recognition method, namely, a dual-structured attention network (DSAN), to solve the problems mentioned in the background art, such as the high computational cost, lack of structured guidance, and lack of adaptability in the fusion strategy of existing FER models.

[0005] To achieve the above objectives, the core technical solution adopted by this invention is:

[0006] A facial expression recognition method based on dual structured attention includes: constructing a hierarchical processing framework containing a first layer of structured attention and a second layer of structured attention to process multimodal and multiscale features extracted from facial images;

[0007] The first structured attention mechanism is implemented by an adaptive dual-modal fusion module, which adaptively fuses the image feature modality and key point feature modality of the face image at each of the multiple feature scales to generate modal fusion features at each scale.

[0008] The second structured attention mechanism is implemented by a cascaded cross-scale fusion module, which aggregates the modal fusion features at multiple scales in a cascaded manner to generate a global feature vector for facial expression classification.

[0009] Furthermore, the specific operating steps are as follows:

[0010] Step S1: Process the input face image Through parallel image backbone network and key backbone network ,extract Image feature set at each scale and key feature set ;

[0011] Step S2: For each scale , feature pair The input is fed into a separate ADMF module, where it is weighted and fused using a channel and spatial attention mechanism to obtain the modal fusion features at this scale. ;

[0012] Step S3: Fuse feature sets of all scales The input is fed into the CCFM module, where it first undergoes feature normalization to obtain a token sequence set with uniform size and dimensions. Then, through cascaded cross-scale encoders and global context encoders, the data is aggregated and refined to finally generate a global feature vector. ;

[0013] Step S4: Transfer the global feature vector The data is input into the classification header, and the final probability distribution of the expression category is calculated. ;

[0014] Step S5: Based on the category probability distribution The loss function is calculated using real facial expression labels, and the network model is trained using the backpropagation algorithm.

[0015] Furthermore, the specific processing procedure of the ADMF module in step 2 is defined by the following formula:

[0016] Preliminary aggregation: ;

[0017] Channel attention gating:

[0018] ,

[0019] ;

[0020] in, This represents a global average pooling operation; Represents a fully connected layer; The one-dimensional vector is reshaped into a two-dimensional matrix; the Softmax function operates along the second dimension (modal dimension) to generate normalized attention weights; The operation ultimately splits it into two weight vectors. ;

[0021] Spatial attention map:

[0022] ,

[0023] ;

[0024] in, It is the Sigmoid activation function; This indicates a splicing operation along the channel dimension; and These represent average pooling and max pooling along the channel axis, respectively; and Represents two independent convolutional layers with non-shared weights, which are used to generate their respective spatial attention maps from the pooling results of image features and keypoint features, respectively.

[0025] Weighted fusion: ,in This represents element-wise multiplication.

[0026] Furthermore, the specific processing procedure of the CCFM module in step 3 is defined by the following formula:

[0027] Feature normalization: ,in, Represents an adaptive average pooling layer; It is Convolutional layer; Represents a reshaping operation;

[0028] Cascaded cross-scale encoders:

[0029] ,

[0030] ,

[0031] ;

[0032] in, LN represents a multi-head cross-attention layer; LN represents layer normalization; FFN represents a feedforward network.

[0033] Global Context Encoder: ,in This represents a bullish self-attention strategy;

[0034] Output layer: ,in This represents the global average pooling operation.

[0035] Furthermore, the loss function in step 5 is the cross-entropy loss function, which is defined as: ,in To predict the category of the model The probability, For category The true label indication value, This represents the total number of categories.

[0036] The beneficial effects of this invention are reflected in the synergistic improvement of performance, efficiency, and robustness. First, through its first layer of structured attention (ADMF module), this invention achieves strong robustness. This module can dynamically adjust the influence of each modal feature based on its discriminative power, effectively addressing the uncertainty of modal information caused by input degradation (such as low illumination or occlusion). Second, its second layer of structured attention (CCFM module) introduces a strong inductive bias through a cascaded information flow from concrete to abstract. This not only ensures the quality of high-level semantic representation construction and contextual consistency but also achieves significant technical advantages. Crucially, this advanced structured design decouples cross-scale fusion computation from input resolution, enabling this high-quality feature aggregation process to be completed while significantly reducing computational complexity and the number of model parameters, greatly improving the model's scalability. Finally, the robustness brought by ADMF and the high-quality, high-efficiency fusion brought by CCFM form a perfect synergy. The entire dual-structured design allows the model to learn more discriminative feature representations with lower computational overhead, achieving superior performance compared to existing methods.

[0037] This invention designs and integrates a dual, hierarchical structured attention mechanism that decouples the complex fusion process of facial expression features into two clearly defined stages: modality fusion and scale aggregation. Specifically, the first layer of the mechanism is implemented by an Adaptive Dual-Modal Fusion Module (ADMF), which uses a channel-space attention structure at each feature scale to intelligently weight and select features from both image and keypoint modalities, thereby dynamically adjusting the contribution of each modality. Subsequently, the second layer of the mechanism is implemented by a Cascaded Cross-Scale Fusion Module (CCFM), which uses a cascaded attention structure from concrete to abstract to efficiently aggregate the fused features at each scale globally. Attached Figure Description

[0038] Figure 1 This is a flowchart illustrating the computational process of the overall DSAN network architecture proposed in this invention, which is a facial expression recognition method based on dual structured attention. Detailed Implementation

[0039] Exemplary embodiments will now be described more fully with reference to the accompanying drawings. However, these exemplary embodiments can be implemented in many forms and should not be construed as limited to the examples set forth herein; rather, they are provided so that the invention will be more comprehensive and complete, and will fully convey the concept of the exemplary embodiments to those skilled in the art. The described features or characteristics may be combined in any suitable manner in one or more embodiments.

[0040] Reference Figure 1 The present invention proposes a facial expression recognition method based on dual structured attention, the specific operation steps of which are as follows:

[0041] Step S1, Multimodal and Multiscale Feature Extraction:

[0042] Input a face image ,in These represent the height and width of the image, respectively. The image backbone network is used in parallel. and key backbone network Extract respectively Features at each scale.

[0043] Among them, the image backbone network It can be any deep neural network capable of extracting multi-scale visual features from an image, such as, but not limited to, convolutional neural networks (CNNs) or visual transformers (ViTs). In a preferred embodiment of the invention, A deep residual network (ResNet) was used.

[0044] Similarly, the key point backbone network This can be any deep learning model capable of extracting structural or positional features related to facial key points. In a preferred embodiment of the invention, to balance efficiency and performance, A lightweight mobile face network (MobileFaceNet) was used.

[0045] The extracted multi-scale feature set is represented by the following formula:

[0046]

[0047]

[0048] in, Representing the first Scale-based image features and keypoint features These represent the height, width, and number of channels of the feature map at this scale.

[0049] Step S2, First layer of structured attention: Adaptive bimodal fusion (ADMF):

[0050] For each scale The fusion is performed using a separate ADMF module, and the calculation process is as follows:

[0051] Step S201, preliminary aggregation:

[0052]

[0053] Step S202, Channel Attention Gating: Calculate the channel weights for modality selection:

[0054]

[0055]

[0056] in, This represents a global average pooling operation; Represents a from Dimension mapping to A fully connected layer of dimensionality; The one-dimensional vector is reshaped into a two-dimensional matrix; the Softmax function operates along the second dimension (modal dimension) to generate normalized attention weights; The operation ultimately splits it into two weight vectors. .

[0057] Step S203, Spatial Attention Map: Calculate the spatial saliency of each modality:

[0058]

[0059]

[0060] in, It is the Sigmoid activation function; This indicates a splicing operation along the channel dimension; and These represent average pooling and max pooling along the channel axis, respectively; and This represents two independent convolutional layers with non-shared weights, which are used to generate their respective spatial attention maps from the pooling results of image features and keypoint features, respectively.

[0061] Step S204, Weighted fusion:

[0062]

[0063] in Represents element-wise multiplication. This is the output of phase two.

[0064] Step S3, Second Layer of Structured Attention: Cascaded Cross-Scale Fusion (CCFM):

[0065] Modality fusion feature sets at various scales are obtained through processing by the ADMF module. Subsequently, this invention further addresses the technical problem of cross-scale feature integration by proposing a cascaded cross-scale fusion module (CCFM). The calculation process of this module is as follows:

[0066] Step S301, Feature Normalization:

[0067]

[0068] in, . Represents an adaptive average pooling layer, reducing dimensionality to a uniform size. ; It is Convolutional layers, projected onto a unified embedding dimension ; This represents a reshaping operation.

[0069] Step S302, cascaded cross-scale encoder:

[0070]

[0071]

[0072]

[0073] in, LN represents a multi-head cross-attention layer; LN represents layer normalization; FFN represents a feedforward network.

[0074] Step S303, Global Context Encoder:

[0075]

[0076] in, This represents a bullish self-attention operation.

[0077] Step S304, Output Layer:

[0078]

[0079] in This represents the global average pooling operation.

[0080] Step S4, Classification and Training:

[0081] eigenvectors The data is fed into the classification head to obtain the probability distribution:

[0082]

[0083] in, For the total number of categories, MLP( ) represents a multilayer perceptron.

[0084] During training, the cross-entropy loss function is used. The model parameters are optimized. For a single sample, the loss function is defined as follows:

[0085]

[0086] in, The model predicts that the sample belongs to a certain category. The probability, i.e., the vector The first in One element; It is an indicator function if the true label of the sample is a category. ,but Otherwise, it is 0.

[0087] The effects of this invention can be further illustrated by the following simulation experiments:

[0088] I. Simulation conditions:

[0089] The simulation experiment in this embodiment was conducted on a high-performance server equipped with four NVIDIA A100 GPUs. In terms of the software environment, the PyTorch deep learning framework based on the Python language was used for model building, training, and testing of the method described in this invention.

[0090] II. Simulation Content:

[0091] The simulation in this embodiment aims to classify facial expressions from real-world facial images. To this end, we use the publicly available RAF-DB dataset for model training and performance evaluation. This dataset contains seven expression categories and follows its official classification. We trained the model using 12,271 images and tested it with 3,068 images.

[0092] In this embodiment, the specific network parameter settings for the constructed DSAN model are as follows: Image backbone An ir-50 network pre-trained on Ms-Celeb-1M was used, and its parameters were fine-tuned during training for this task; key points backbone MobileFaceNet is used, and its network weights are frozen during training to ensure the stability of keypoint extraction. For the first layer of structured attention (ADMF module), its spatial attention map generator... and All use a single output channel of 1. Convolutional layers. For the second layer of structured attention (CCFM module), the unified spatial resolution in its feature normalization step. Set as Unified Embedding Dimension Set to 512. The classification head MLP consists of two linear layers, with GELU used as a non-linear activation function in between.

[0093] The model training process used the following hyperparameter settings: the optimizer was AdamW, with weight decay set to 0.05. The initial learning rate was set to 3.5e-4 and dynamically adjusted using a cosine annealing strategy.

[0094] III. Simulation Effect Analysis:

[0095] To verify the beneficial effects of this invention, we compared the performance of the DSAN model implemented in this embodiment with that of the baseline model POSTER V2, which uses the same backbone network and training settings. The performance index comparison results obtained from the simulation experiment are shown in Table 1.

[0096] Table 1: Performance comparison of the embodiments of the present invention and the baseline model on the RAF-DB test set

[0097] method FLOPs(G) Params(M) Accuracy (%) POSTER V2 (Baseline) 8.4 43.7 92.21 DSAN (An embodiment of the present invention) 8.2 42.6 93.57

[0098] As can be seen from the comparison results in Table 1, the DSAN model implemented in this embodiment of the invention achieves higher classification accuracy with lower computational cost (FLOPs) and fewer parameters (Params). This result fully demonstrates that the dual structured attention mechanism employed in this invention has significant advantages in both performance and efficiency compared to the fusion strategy of the baseline model, thus verifying the inventiveness and beneficial effects of this invention.

[0099] In summary, this invention proposes a novel network architecture called DSAN and its corresponding facial expression recognition method. Its core contribution lies in the pioneering design of a dual, hierarchical structured attention mechanism, decoupling the complex fusion task into two stages: orthogonal modality selection and scale aggregation. This architecture adaptively addresses the uncertainty problem in multimodal information fusion through a first-level attention structure (ADMF), and efficiently achieves cross-scale feature evolution with strong cognitive priors through a second-level attention structure (CCFM). The deep model constructed in this invention can not only be directly applied to facial expression recognition as a high-performance classifier, but more importantly, it provides an advanced and generalizable multidimensional information processing paradigm. The network weights obtained through pre-training on large-scale datasets can serve as valuable parameter initial values ​​or a general feature extractor, which can be transferred to other related downstream facial analysis tasks (such as emotion computing, fatigue detection, etc.), thereby significantly improving the performance and convergence efficiency of downstream models.

[0100] Other embodiments of the invention will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of the invention that follow the general principles of the invention and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of the invention are indicated by the appended claims.

Claims

1. A facial expression recognition method based on dual structured attention, characterized in that, include: A hierarchical processing framework containing first-level and second-level structured attention is constructed to process multimodal and multiscale features extracted from face images. The first structured attention mechanism is implemented by an adaptive dual-modal fusion module, which adaptively fuses the image feature modality and key point feature modality of the face image at each of the multiple feature scales to generate modal fusion features at each scale. The second structured attention mechanism is implemented by a cascaded cross-scale fusion module, which aggregates the modal fusion features at multiple scales in a cascaded manner to generate a global feature vector for facial expression classification.

2. The method according to claim 1, characterized in that, The specific operating steps are as follows: Step S1: Process the input face image Through parallel image backbone network and key backbone network ,extract Image feature set at each scale and key feature set ; Step S2: For each scale , feature pair The input is fed into a separate ADMF module, where it is weighted and fused using a channel and spatial attention mechanism to obtain the modal fusion features at this scale. ; Step S3: Fuse feature sets of all scales The input is fed into the CCFM module, where it first undergoes feature normalization to obtain a token sequence set with uniform size and dimensions. Then, through cascaded cross-scale encoders and global context encoders, the data is aggregated and refined to finally generate a global feature vector. ; Step S4: Transfer the global feature vector The data is input into the classification header, and the final probability distribution of the expression category is calculated. ; Step S5: Based on the category probability distribution The loss function is calculated using real facial expression labels, and the network model is trained using the backpropagation algorithm.

3. The method according to claim 2, characterized in that, The specific processing procedure of the ADMF module in step 2 is defined by the following formula: Preliminary aggregation: ; Channel attention gating: ， ； in, This represents a global average pooling operation; Represents a fully connected layer; The one-dimensional vector is reshaped into a two-dimensional matrix; the Softmax function operates along the second dimension (modal dimension) to generate normalized attention weights; The operation ultimately splits it into two weight vectors. ; Spatial attention map: ， ； in, It is the Sigmoid activation function; This indicates a splicing operation along the channel dimension; and These represent average pooling and max pooling along the channel axis, respectively; and Represents two independent convolutional layers with non-shared weights, which are used to generate their respective spatial attention maps from the pooling results of image features and keypoint features, respectively. Weighted fusion: ,in This represents element-wise multiplication.

4. The method according to claim 2, characterized in that, The specific processing procedure of the CCFM module in step 3 is defined by the following formula: Feature normalization: ,in, Represents an adaptive average pooling layer; It is Convolutional layer; Represents a reshaping operation; Cascaded cross-scale encoders: ， ， ； in, LN represents a multi-head cross-attention layer; LN represents layer normalization; FFN represents a feedforward network. Global Context Encoder: ,in This represents a bullish self-attention strategy; Output layer: ,in This represents the global average pooling operation.

5. The method according to claim 2, characterized in that, The loss function in step 5 is the cross-entropy loss function, which is defined as follows: ,in To predict the category of the model The probability, For category The true label indication value, This represents the total number of categories.

Images