A facial expression recognition method based on a double attention mechanism

Abstract

The application provides a facial expression recognition method based on a double attention mechanism, named Adaptive Spatio-Excitation Transformer (ASET). This method introduces Adaptive Squeeze-and-Excitation (ASE) attention mechanism and spatial attention mechanism, and combines with Transformer network, which significantly improves the accuracy and efficiency of facial expression recognition. The core of the application is to recalibrate the channels of feature maps through ASE mechanism, and focus on the key areas of the face through spatial attention mechanism, so as to more efficiently extract expression features. In addition, the application also introduces visualization technology to highlight the key areas in the face image, helping to understand the decision-making process of the model. Experimental results show that the application has achieved excellent performance on multiple standard datasets such as FER2013, CK+ and JAFFE, and has significantly improved performance compared with existing technology. The application can be widely used in human-computer interaction, sentiment analysis and other fields, providing an efficient and accurate solution for facial expression recognition.

Description

Technical Field

[0001] This invention belongs to the field of computer vision and artificial intelligence technology, specifically relating to a facial expression recognition method based on a dual attention mechanism. Background Technology

[0002] Facial expression recognition is a crucial research area in computer vision and artificial intelligence. Its aim is to accurately identify an individual's emotional state by analyzing facial muscle activity and facial expression features in facial images. Facial expressions are important nonverbal signals of human emotion and can significantly influence interpersonal communication and social interaction. In recent years, with the rapid development of deep learning technology, especially the widespread application of convolutional neural networks (CNNs), facial expression recognition technology has made significant progress. CNNs, with their powerful feature extraction capabilities, can efficiently identify basic expressions such as anger, disgust, fear, happiness, neutrality, sadness, and surprise under controlled laboratory conditions.

[0003] However, existing technologies still have some limitations. First, traditional CNN models struggle to effectively extract key features when dealing with complex backgrounds and varied natural scenes, resulting in performance limitations. Second, while existing attention mechanisms can improve the model's focus on key information, they can only consider unidirectional contextual information, failing to capture bidirectional contextual information, and they also calculate weights for unimportant positions, leading to computational waste. This invention aims to address the performance limitations of existing facial expression recognition technologies in natural scenes by proposing a novel facial expression recognition framework and attention mechanism to improve the accuracy and efficiency of expression recognition.

[0004] To solve the above problems, the existing technologies are as follows:

[0005] Comparison with the patented technology "facial expression recognition based on dual attention mechanism";

[0006] I. Patent CN115661910A employs a traditional single-attention mechanism for facial expression recognition, which can only focus on a single key region in the face image and cannot simultaneously capture important information in both channel and spatial dimensions, resulting in limited recognition performance in complex scenes. This research, however, adopts a facial expression recognition method based on a dual-attention mechanism. It not only introduces the proposed Adaptive Squeeze-and-Excitation (ASE) attention mechanism to recalibrate the channels of the feature map but also combines a spatial attention mechanism to focus on key facial regions. Compared to the single-attention mechanism used in CN115661910A, this invention can simultaneously optimize features from both channel and spatial dimensions, extracting expression features more efficiently and significantly improving the accuracy and efficiency of facial expression recognition, especially demonstrating stronger robustness in complex backgrounds and varied natural scenes. Regarding model training, CN115661910A may use traditional optimization methods and loss functions, lacking optimization for model regularization and generalization capabilities. This invention employs the Adam optimizer from stochastic gradient descent to optimize the loss function during training. Through a carefully designed loss function, including classification loss and regularization terms, and by incorporating dropout technology, it effectively avoids overfitting and enhances the model's generalization ability. This optimization strategy enables the model to be effectively trained from scratch even with small datasets, resulting in superior performance on multiple public datasets and a significant performance improvement compared to CN115661910A.

[0007] Second, while patent CN118823852A uses a traditional Convolutional Neural Network (CNN) architecture for feature extraction, this research not only employs a CNN architecture but also incorporates a Transformer network. The Transformer network can better capture long-range dependencies, further enhancing the model's feature extraction capability and recognition performance. It performs exceptionally well on multiple public datasets, achieving significant performance improvements compared to existing technologies. Specifically, it shows superior performance on FER2013 and CK+ datasets.

[0008] On the JAFFE dataset, it achieved accuracies of 75.14%, 100%, and 95.80%, respectively, which are significantly better than the technical solution adopted in patent CN118823852A. Summary of the Invention

[0009] To address the aforementioned technical problems, this invention proposes a facial expression recognition method based on a dual attention mechanism. In complex backgrounds and varied natural scenes, it learns discriminative features and expression-invariant features using only a small number of labeled face images, aiming to achieve efficient expression recognition in unlabeled natural scene images.

[0010] To achieve the above objectives, the technical solution adopted by the present invention is as follows:

[0011] A facial expression recognition method based on a dual attention mechanism, the specific steps of which are as follows:

[0012] S1: Input the facial image into the ASET architecture;

[0013] S2: In the feature extraction module, features are extracted from the input facial image through convolutional layers, max pooling layers, rectified linear units (ReLU) activation function, and ASE attention mechanism.

[0014] S3: In the attention mechanism module, the feature map output by the feature extraction module is weighted through the spatial attention mechanism to focus on the key regions of the face;

[0015] S4: Input the weighted feature map into the classifier to predict the expression category and realize facial expression recognition.

[0016] As a further improvement of the present invention, the feature extraction module in step S2 includes:

[0017] S21: Seven convolutional layers, each followed by a ReLU activation function and an ASE attention module;

[0018] S22: A 2×2 max pooling layer is set between every two convolutional layers, and a dropout layer with a dropout rate of 35% is connected after the last convolutional layer.

[0019] As a further improvement of the present invention, the ASE attention mechanism in step S2 includes the following steps:

[0020] S23: Perform the Squeeze operation to compress the spatial dimension H×W of the feature map into a global channel descriptor, with the specific formula as follows:

[0021]

[0022] Among them, z c This represents the global descriptor for the c-th channel, u c (i,j) represents the value of the feature map of the c-th channel at position (i,j), where C is the number of channels. This operation aggregates the feature maps of each channel through global average pooling, capturing global information to provide context for subsequent activation operations, helping the network to better understand the relationship between different channels. α represents the learnable scaling parameter.

[0023] S24: Perform the Excitation operation. Based on the global descriptor z generated by the Squeeze operation, dynamically generate weights s for each channel. The specific formula is as follows:

[0024] s=σ(W2δ(W1z+b))

[0025] Where W1∈R C / r×C and W2∈R C×C / r δ is the weight of the two fully connected layers, r is the dimensionality reduction ratio used to reduce the number of parameters; δ is the ReLU activation function used to introduce nonlinearity; σ is the Sigmoid activation function used to normalize the output to the range (0, 1); s is the weight of the generated c-th channel. The Excitation operation learns the nonlinear relationship between channels through a simple self-gating mechanism and generates the weight of each channel. b is the bias term used to adjust the weights.

[0026] S25: Perform feature map recalibration, applying the generated channel weights s to the original feature map U to complete the feature map recalibration. The specific formula is as follows:

[0027]

[0028] in, This is the recalibrated feature map, s c It is the weight of the c-th channel, u c This is the c-th channel of the original feature map. β is a normalization parameter used to adjust the distribution of channel weights.

[0029] As a further improvement of the present invention, the attention mechanism module in step S3 includes:

[0030] S31: Two convolutional layers, each followed by a max pooling layer and a ReLU activation function in sequence;

[0031] S32: Two fully connected layers used to transform parameters through regression analysis, mapping the input data onto the sampling grid T(θ) to achieve spatial weighting of the feature map.

[0032] As a further improvement of the present invention, the spatial attention mechanism in step S3 includes the following steps:

[0033] S33: Spatial Attention Weight Generation: The input feature map is processed by a convolutional kernel to generate a spatial attention weight map M. The specific formula is as follows:

[0034] M=σ(W s δ(V s U))

[0035] Where U is the input feature map; Vs and W s M is the convolutional kernel used to generate spatial attention weights, which is adjusted through learning; M is the spatial attention weight map generated by this process.

[0036] S34: Feature map weighting: The specific formula is as follows:

[0037] U′=M⊙U

[0038] U′ is a weighted feature map; ⊙ is element-wise multiplication, also known as Hadamard multiplication;

[0039] During the training of this model, the Adam optimizer in stochastic gradient descent is used to optimize the loss function. The loss function consists of two parts, and the specific formula is as follows:

[0040]

[0041] S35: Classification loss, using the cross-entropy loss function, is used to detect the difference between the model's predictions and the actual labels, thereby evaluating the model's performance on classification tasks;

[0042] S36: Regularization term, using the L2 norm, normalizes the weights of the last four fully connected layers in the model to enhance the model's generalization ability.

[0043] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0044] By introducing the Adaptive Squeeze-and-Excitation (ASE) attention mechanism and the spatial dual attention mechanism, the model of this invention exhibits stronger robustness in complex scenarios such as image quality degradation, occlusion, and illumination changes. This enables the model to maintain stable recognition performance in diverse real-world application scenarios, significantly improving its applicability in complex environments. The model can efficiently filter key features, reducing the waste of computational resources on unimportant features, thereby significantly improving feature extraction efficiency. This invention performs excellently on multiple public datasets, achieving significant performance improvements compared to existing technologies. Specifically, it achieves accuracies of 75.14%, 100%, and 95.80% on the FER2013, CK+, and JAFFE datasets, respectively. These results demonstrate the significant superiority of this invention in facial expression recognition tasks. In summary, this invention, through its innovative model architecture and optimization strategies, achieves significant performance improvements and broad applicability in the field of facial expression recognition, providing important support for the development of related technologies. Attached Figure Description

[0045] Figure 1This is a flowchart of the facial expression recognition method based on the dual attention mechanism of the present invention;

[0046] Figure 2 This is a structural diagram of the Adaptive Spatio-Excitation Transformer (ASET) architecture in this invention. Detailed Implementation

[0047] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments:

[0048] Example 1:

[0049] The flowchart of the facial expression recognition method is attached. Figure 1 As shown in the diagram, the architecture of the Adaptive Spatio-ExcitationTransformer (ASET) is as follows: Figure 2 As shown, a deep learning-based facial expression recognition method includes the following steps:

[0050] S1: Input the facial image into the Adaptive Spatio-ExcitationTransformer (ASET) architecture;

[0051] S2: In the feature extraction module, features are extracted from the input facial image through convolutional layers, max pooling layers, rectified linear unit (ReLU) activation functions, and the Adaptive Squeeze-and-Excitation (ASE) attention mechanism;

[0052] S3: In the attention mechanism module, the feature map output by the feature extraction module is weighted through the spatial attention mechanism to focus on the key regions of the face;

[0053] S4: Input the weighted feature map into the classifier to predict the expression category and realize facial expression recognition.

[0054] In step S2, the feature extraction module includes the following sub-steps:

[0055] S21: Seven convolutional layers, each followed by a rectified linear unit (ReLU) activation function and an ASE attention module;

[0056] S22: A 2×2 max pooling layer is set between every two convolutional layers, and a dropout layer with a dropout rate of 35% is connected after the last convolutional layer.

[0057] In step S3, the attention mechanism module includes the following sub-steps:

[0058] S31: Two convolutional layers, each followed by a max pooling layer and a rectified linear unit (ReLU) activation function;

[0059] S32: Two fully connected layers used to transform parameters through regression analysis, mapping the input data onto the sampling grid to achieve spatial weighting of the feature map.

[0060] In step S2, the ASE attention mechanism includes the following sub-steps:

[0061] S23: Perform the Squeeze operation to compress the spatial dimension (H×W) of the feature map into a global channel descriptor, using the following formula:

[0062]

[0063] Among them, z c This represents the global descriptor for the c-th channel, u c (i,j) represents the value of the feature map of the c-th channel at position (i,j), where C is the number of channels. This operation aggregates the feature maps of each channel through global average pooling, capturing global information to provide context for subsequent activation operations, helping the network better understand the relationships between different channels. α represents the learnable scaling parameter.

[0064] S24: Perform the Excitation operation. Based on the global descriptor z generated by the Squeeze operation, dynamically generate weights s for each channel. The specific formula is as follows:

[0065] s=σ(W2δ(W1z+b))

[0066] Where W1∈R C / r×C and W2∈R C×C / r Here, r is the weights of two fully connected layers, r is the dimensionality reduction ratio used to reduce the number of parameters; δ is the ReLU activation function used to introduce nonlinearity; σ is the Sigmoid activation function used to normalize the output to the range (0, 1); and s is the weight of the generated c-th channel. The excitation operation learns the nonlinear relationship between channels through a simple self-gating mechanism and generates weights for each channel. These weights can dynamically highlight important feature channels while suppressing unimportant feature channels, thereby enhancing the network's feature representation capability. b is a bias term used to adjust the weights.

[0067] S25: Perform feature map recalibration, applying the generated channel weights s to the original feature map U to complete the feature map recalibration. The specific formula is as follows:

[0068]

[0069] in, This is the recalibrated feature map, s c It is the weight of the c-th channel, u c This is the c-th channel of the original feature map. β is a normalization parameter used to adjust the distribution of channel weights.

[0070] In step S3, the spatial attention mechanism includes the following sub-steps:

[0071] S33: Spatial Attention Weight Generation: The input feature map is processed by a convolutional kernel to generate a spatial attention weight map M. The specific formula is as follows:

[0072] M=σ(W s δ(V s U))

[0073] Where U is the input feature map; V s and W s M is the convolutional kernel used to generate spatial attention weights, which can be adjusted through learning; M is the spatial attention weight map generated by this process.

[0074] S34: Feature map weighting: The specific formula is as follows:

[0075] U′=M⊙U

[0076] U′ is a weighted feature map; ⊙ is element-wise multiplication, also known as Hadamard multiplication.

[0077] The facial expression recognition method employs the Adam optimizer in stochastic gradient descent to optimize the loss function during model training. The loss function includes the following sub-steps:

[0078]

[0079] S35: Classification loss, using the cross-entropy loss function, is used to detect the difference between the model's predictions and the actual labels, thereby evaluating the model's performance on classification tasks.

[0080] S36: Regularization term, employing the L2 norm, normalizes the weights of the last four fully connected layers in the model to enhance its generalization ability. The weights λ of the regularization term are carefully tuned on the validation set for optimal performance. Furthermore, dropout technology is incorporated, combined with the regularization strategy, enabling the model to be effectively trained from scratch even with small datasets. By adding these two parts, a loss function is constructed that considers both model performance and avoids overfitting, effectively guiding the model training process.

[0081] In practice, each model was trained from scratch. Network weights were initialized using a random Gaussian distribution with a mean of zero and a standard deviation of 0.05. A learning rate of 0.0001 was used on the CK+ and FER2013 datasets, while 0.00001 was more suitable for the JAFFE dataset. With a dropout rate of 0.35, the model exhibited good generalization ability on all datasets, especially on the FER2013 dataset, effectively mitigating overfitting. Adjusting the L2 regularization weight (λ), the model showed the most stable performance on the validation set when λ = 0.0001. Furthermore, the Adam optimizer from stochastic gradient descent was used to optimize the loss function, effectively guiding the model training process.

[0082] Experiments were conducted using the FER2013, CK+, and JAFFE datasets. The JAFFE dataset covers seven different facial expressions: happiness, sadness, surprise, anger, disgust, and fear, as well as a neutral expression. The entire dataset consists of 213 grayscale images. The CK+ dataset contains 593 image sequences from 123 participants, recording the transition from neutral expressions to happiness, sadness, surprise, anger, disgust, fear, and contempt. All images in the FER2013 dataset were sourced from the internet, containing 35,887 grayscale images. 28,709 images formed the initial training set, while 3,589 images formed the validation and test sets. The dataset showcases seven different facial expressions: anger (13.8%), disgust (1.5%), fear (14.3%), happiness (25.0%), sadness (16.9%), surprise (11.2%), and neutral (17.3%). Experiments showed that the accuracy reached 75.14%, 100%, and 95.80% on the FER2013, CK+, and JAFFE datasets, respectively.

[0083] Furthermore, the method of this invention is compared with other methods: IACNN, STP, ViTAU, DeepFER, Salient Facial Patch, LBP+SVM, VGG16+LRP, LSTM, DCNN+RLPS, and HCSA. The comparison results are shown in Tables 1, 2, and 3 below.

[0084] Table 1. Comparison of model accuracy on the CK+ dataset

[0085] Model Basic model architecture accuracy IACNN CNN 95.37％ STP SPD+LSTM 96.87％ ViTAU VisionTransformer 99.40％ DeepFER CNN+Attention 99.47％ ASET CNN + Attention + Transformer 100.00％

[0086] Table 2 Comparison of model accuracy on the JAFFE dataset

[0087] Model Basic model architecture accuracy SalientFacialPatch LBP+SVM 91.80％ LBP+SVM LBP+SVM 92.45％ VGG16+LRP VGG16+LRP 96.33％ DeepFER CNN+Attention 93.89％ ASET CNN + Attention + Transformer 95.80％

[0088] Table 3 Comparison of model accuracy on the FER2013 dataset

[0089] Model Basic model architecture accuracy LSTM LSTM 71.00％ DCNN+RLPS CNN+RL 72.35％ HCSA CNN+HFE-Block 73.40％ DeepFER CNN+Attention 74.85％ ASET CNN + Attention + Transformer 75.14％

[0090] As can be seen from Tables 1, 2, and 3, the recognition performance of this invention is superior to other methods, especially on datasets with high-quality annotations. The above experiments verify the effectiveness of this invention.

[0091] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention in any other way. Any modifications or equivalent changes made based on the technical essence of the present invention shall still fall within the scope of protection claimed by the present invention.

Claims

1. A facial expression recognition method based on a dual attention mechanism, comprising the following steps, characterized in that: S1: Input the facial image into the ASET architecture; S2: In the feature extraction module, features are extracted from the input facial image through convolutional layers, max pooling layers, rectified linear units (ReLU) activation function, and ASE attention mechanism. The ASE attention mechanism described in step S2 includes the following steps: S23: Perform the Squeeze operation to compress the spatial dimension H×W of the feature map into a global channel descriptor, with the specific formula as follows: ； in, This represents the global descriptor for the c-th channel. This indicates the feature map of the c-th channel at position. The value of C represents the number of channels. This operation aggregates the feature maps of each channel through global average pooling, capturing global information to provide context for subsequent activation operations, helping the network better understand the relationships between different channels. Indicates the learnable scaling parameter; S24: Perform the Excitation operation. Based on the global descriptor z generated by the Squeeze operation, dynamically generate weights s for each channel. The specific formula is as follows: ； in, and These are the weights of the two fully connected layers, and r is the dimensionality reduction ratio used to reduce the number of parameters; It is the ReLU activation function, used to introduce nonlinearity; is the Sigmoid activation function, used to normalize the output to the range (0, 1); s is the weight of the generated c-th channel. The Excitation operation learns the nonlinear relationship between channels through a simple self-gating mechanism and generates the weight of each channel. b is the bias term used to adjust the weights. S25: Perform feature map recalibration, applying the generated channel weights s to the original feature map U to complete the feature map recalibration. The specific formula is as follows: ； in, This is the recalibrated feature map. It is the weight of the c-th channel. It is the c-th channel of the original feature map. It is a normalization parameter used to adjust the distribution of channel weights; S3: In the attention mechanism module, the feature map output by the feature extraction module is weighted through the spatial attention mechanism to focus on the key regions of the face; S4: Input the weighted feature map into the classifier to predict the expression category and realize facial expression recognition.

2. The facial expression recognition method based on a dual attention mechanism according to claim 1, characterized in that: The feature extraction module in step S2 includes: S21: Seven convolutional layers, each followed by a ReLU activation function and an ASE attention module; S22: A 2×2 max pooling layer is set between every two convolutional layers, and a dropout layer is connected after the last convolutional layer with a dropout rate of 35%.

3. The facial expression recognition method based on a dual attention mechanism according to claim 1, characterized in that: The attention mechanism module mentioned in step S3 includes: S31: Two convolutional layers, each followed by a max pooling layer and a ReLU activation function in sequence; S32: Two fully connected layers used to transform parameters through regression analysis, mapping the input data onto the sampling grid T(θ) to achieve spatial weighting of the feature map.

4. The facial expression recognition method based on a dual attention mechanism according to claim 1, characterized in that: The spatial attention mechanism described in step S3 includes the following steps: S33: Spatial Attention Weight Generation: The input feature map is processed by a convolutional kernel to generate a spatial attention weight map M. The specific formula is as follows: ； in, It is the input feature map; and M is the convolutional kernel used to generate spatial attention weights, which is adjusted through learning; M is the spatial attention weight map generated by this process. S34: Feature map weighting: The specific formula is as follows: ； ⊙ is a weighted feature map; ⊙ is element-wise multiplication, also known as Hadamard multiplication; During the training of this model, the Adam optimizer in stochastic gradient descent is used to optimize the loss function. The loss function consists of two parts, and the specific formula is as follows: ； S35: Classification loss, using the cross-entropy loss function, is used to detect the difference between the model's predictions and the actual labels, thereby evaluating the model's performance on classification tasks; S36: Regularization term, using the L2 norm, normalizes the weights of the last four fully connected layers in the model to enhance the model's generalization ability.

Images