A facial expression recognition method based on attention mechanism and self-distillation
By employing adaptive channel attention and self-distillation techniques, the problem of facial expression recognition models focusing on key regions under field conditions has been solved, achieving efficient recognition on low-end devices and improving the accuracy and efficiency of facial expression recognition.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- HANGZHOU DIANZI UNIV
- Filing Date
- 2023-06-13
- Publication Date
- 2026-05-12
AI Technical Summary
Existing facial expression recognition methods struggle to effectively target key areas in outdoor conditions due to poor lighting, facial expression occlusion, and posture changes. This increases model complexity and hinders deployment on low-end devices.
We employ an attention-based and self-distillation-based facial expression recognition method. During the training phase, we guide the model to focus on important regions through an adaptive channel attention module and a reshaping module. We also utilize a self-distillation network to gradually transfer knowledge from deep networks to shallow networks, thereby reducing the complexity of the inference phase.
Without increasing model complexity, the accuracy and efficiency of facial expression recognition were improved, and the correct classification of seven basic emotions was achieved.
Smart Images
Figure CN116682161B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of facial expression recognition, and more particularly to a facial expression recognition method based on attention mechanisms and self-distillation. Background Technology
[0002] Facial expressions play a crucial role in interpersonal communication, allowing people to clearly perceive the emotions others wish to express through changes in facial expressions. Over the past few decades, the widespread application of facial expression recognition in industries such as digital entertainment, human-computer interaction, driver detection, and healthcare has led to an increasing number of researchers focusing on this field. However, facial expression recognition in outdoor conditions remains challenging due to poor lighting, facial occlusion, and changes in posture.
[0003] Traditional methods typically utilize handcrafted or shallow features for facial expression recognition. These methods take the entire facial expression image as input and determine the category of the facial expression based on the model's final output. However, due to the high similarity between facial expression classes, the model struggles to focus on key regions that are crucial for accurate facial expression classification. Some advanced methods improve the model's ability to grasp important regions by increasing model complexity and attention mechanisms. However, this significantly increases model complexity, severely hindering deployment on low-end devices. Summary of the Invention
[0004] To address the above problems, the purpose of this invention is to provide a facial expression recognition method based on attention mechanisms and self-distillation. This method can adaptively guide the model to focus on certain important areas of facial expressions without increasing model complexity, thereby enabling the model to correctly classify the seven basic emotions of disgust, happiness, anger, fear, surprise, sadness, and neutrality, effectively improving the efficiency and accuracy of facial expression recognition.
[0005] The technical solution adopted by this invention to solve the technical problem is as follows:
[0006] Step 1: Split the three public datasets, Affect-Net, FERPlus, and RAF-DB, into training, testing, and validation sets. Each of the training, testing, and validation sets contains several images of the seven basic facial expressions.
[0007] Step 2: Construct a facial expression recognition model, which includes a feature extraction module, an adaptive channel attention module, a reshaping module, and a self-distillation network.
[0008] Step 3: Align faces in the images input to the training, test, and validation sets, and crop them to a fixed size. Perform image enhancement on the training set input to the model, such as random cropping, horizontal flipping, and random removal, to prevent overfitting.
[0009] Step 4: Input the split training set images into the constructed facial expression recognition model, and train and optimize the learnable parameters in the model until the accuracy of the model no longer improves. The specific training process includes the following sub-steps:
[0010] S41. Input the processed image into the feature extraction module, and extract the output feature map z of each basic block in the feature extraction module. j As features output at each stage of the feature extraction module;
[0011] S42. An adaptive channel attention module is added after the feature map output by each basic block. In this module, max pooling and average pooling are first used to process the feature map z output by each basic block. j The channel information is aggregated to generate two different channel vectors q. max ,q avg Secondly, the adaptive weight module is used for q. max ,q avg Generate the corresponding weight w max ,w avg Then, q max and w max Features after multiplication and q avg and w avg The multiplied features are then added together to obtain the final channel weight feature f. cw Finally, the obtained f cw With input feature map z j Multiplication yields the final output feature map. The calculation formula is as follows:
[0012] q max =MaxPool(z j ),
[0013] q avg =AvgPool(z j ),
[0014] w max =W m1 (σ(W m0 (q max ))),
[0015] w avg =W a1 (σ(W a0 (qavg ))),
[0016] f cw =σ(MLP(q) max )*w max +MLP(q avg )*w avg ),
[0017]
[0018] S43. Utilize the reshaping module to further refine the features output by the shallow basic blocks and finally map the features output by each basic block to the same size dimension to obtain... The calculation formula is as follows:
[0019]
[0020] S44. Calculate each F using a fully connected layer and a softmax function. j The corresponding predicted score f j The calculation formula is as follows:
[0021]
[0022] S45. Knowledge distillation is performed on the model from deep to shallow using a self-distillation network and the corresponding distillation loss, specifically including the following sub-steps:
[0023] S451. Cross-entropy loss is used to measure the difference between the predicted score of the last basic block output and the true label of the training set. The calculation formula is as follows:
[0024] L ce =CrossEntropyLoss(f c ,y).
[0025] S452. The KL divergence loss is used to measure the difference between the prediction score of the current basic block and the prediction score of its next basic block. The calculation formula is as follows:
[0026]
[0027] S453. The L2 loss is used to measure the difference between the features output by the current basic block and the features output by its next basic block. The calculation formula is as follows:
[0028]
[0029] S454. Finally, the total loss consists of the above three types of losses, and its calculation formula is as follows:
[0030] L total =Lce +αL kl +βL2.
[0031] Step 5: Perform inference on the images in the test and validation sets. During the inference phase, the adaptive channel attention module, reshaping module, and self-distillation network are removed. Only the feature extraction module trained during the training phase is used to make the final prediction on the input. The processed training and test set images are input into the model to obtain the corresponding classification results, which are one of the seven basic expressions: disgust, happiness, anger, fear, surprise, sadness, and neutrality.
[0032] This method uses ResNet-50 as the feature extraction module of the model. ResNet-50 consists of 4 basic blocks, each of which contains several convolutional layers. When an image is input into the feature extraction module, the spatial scale of the feature map is halved and the number of channels is doubled after each basic block.
[0033] An adaptive channel attention module is added after the feature map output by each basic block. In this module, max pooling and average pooling are first used to aggregate the channel information of the input feature map to generate two different channel vectors q. max ,q avg Next, these two features are input into the MLP and the corresponding adaptive weight module, which generates the corresponding weights w. max ,w avg Then, q max and w max Features after multiplication and q avg and w avg The multiplied features are then added together to obtain the channel weight feature f. cw Finally, the obtained f cw With input feature map z j Multiplication yields the output feature map. This achieves weighting of the input feature map in the channel dimension.
[0034] Since the semantic information and size of the output features of different basic blocks in ResNet-50 are different, in order to make up for the weak semantic information of shallow basic blocks and to facilitate knowledge distillation later, the reshaping module is used to further refine the features output by shallow basic blocks and map the features output by each basic block to the same size dimension.
[0035] During the training process, this method uses some distillation losses to optimize the model parameters, mainly using cross-entropy loss, KL loss and L2 loss to optimize the model.
[0036] During the inference phase, the model removes the adaptive channel attention module, the reshaping module, and the self-distillation network, retaining only the feature extraction module, and uses the final output of the feature extraction module to predict the input image.
[0037] The essential features of this invention are as follows: By utilizing multi-stage feature extraction in the feature extraction module, the features extracted by the model contain both low-level spatial information from shallow networks and high-level semantic information from deep networks, increasing the robustness of the extracted features; through the adaptive channel attention module, the proportion of max-pooling and average-pooling features is adaptively adjusted, thereby providing more detailed attention to channels and improving the representational power of the output features; by utilizing the reshaping module, the differences between the output features of different stages of the feature extraction module are reduced; by using a novel self-distillation method to guide the shallow network (student network) to continuously learn from the deep network (teacher network), the knowledge of the network is refined and compressed, the network performance is improved, and the network self-learning is realized; by removing the adaptive channel attention module, the reshaping module, and the self-distillation network during the inference stage, and only retaining the feature extraction module trained during the training stage, the complexity of the model during the inference stage is greatly reduced. Attached Figure Description
[0038] Figure 1 This is a flowchart of the facial expression recognition method based on attention mechanism and self-distillation according to the present invention;
[0039] Figure 2 This is a schematic diagram of the overall network architecture of an example of the present invention;
[0040] Figure 3 This is a schematic diagram of the overall network architecture of the adaptive channel attention module in an example of the present invention. Detailed Implementation
[0041] The present invention will now be described in further detail with reference to the accompanying drawings and specific embodiments.
[0042] Research has found that among numerous facial expression recognition methods, convolutional neural networks (CNNs) employing attention mechanisms are a relatively efficient and reliable approach. As a weakly supervised method, using a multi-stage convolutional network as a feature extraction module while utilizing attention mechanisms allows the model to capture more comprehensive information. This is because multi-stage feature extraction enables the model to extract both low-level physical information (such as shape, contour, and edges) and high-level semantic information, thus improving the model's accuracy in recognizing facial expressions. While attention-based facial expression recognition methods have improved accuracy to some extent, they still have limitations. For example, increasing the depth and number of attention modules to enhance their effectiveness and accuracy significantly increases computational cost and hinders deployment on low-end devices. Therefore, an effective model compression method and knowledge transfer system are needed to ensure both accuracy and efficiency without increasing complexity.
[0043] To address the aforementioned issues, this invention proposes a facial expression recognition method based on attention mechanisms and self-distillation. This method effectively improves the accuracy and efficiency of the model by adaptively adjusting the proportions of max pooling and average pooling in channel attention and by using the idea of self-distillation to gradually distill the knowledge extracted from deep networks into shallow networks. Moreover, since the adaptive channel attention module, the reshaping module, and the self-distillation network are only used during the training phase, they do not increase the complexity of the model.
[0044] Specifically, this invention is a facial expression recognition method based on attention mechanisms and self-distillation, such as... Figure 1 As shown, the main steps include the following:
[0045] Step 1: Split the three public datasets, Affect-Net, FERPlus, and RAF-DB, into training, testing, and validation sets. Each set contains several images of the seven basic facial expressions.
[0046] RAF-DB is a real-world dataset containing facial expression images from thousands of people of different races, genders, and ages. It contains a total of 15,339 images (12,271 as the training set and 3,068 as the test set). The FERPlus dataset is an extension of the FER2013 dataset, which is real-world data collected from the Google search engine. In the FER2013 dataset, all images have been cropped to 48x48 after face alignment. However, due to the limitations of FER2013... The 2013 dataset contained many mislabeled images. Therefore, the FERPlus dataset was expanded and relabeled. The FERPlus dataset contains a total of 35,887 images (28,709 images as the training set, 3,589 images as the test set, and 3,589 images as the validation set). The Affect-Net dataset is the largest dataset among all public FER datasets to date, containing a total of 287,401 images (283,901 images as the training set and 3,500 images as the test set).
[0047] Step 2: Construct a facial expression recognition model, such as... Figure 2 As shown, the model includes a feature extraction module, an adaptive channel attention module, a reshaping module, and a self-distillation network.
[0048] In the facial expression recognition model of this invention, the feature extraction module is used to extract features output by the model at different stages. The features extracted by the shallow basic blocks are mainly low-dimensional details such as color, texture, and contour, while the features extracted by the deep basic blocks have higher semantic information and are closer to human understanding of image content. The feature extraction module is mainly composed of ResNet-50, which consists of four basic blocks. Each basic block contains several convolutional layers. When an image is input into the feature extraction module, the spatial scale of the feature map is halved and the number of channels doubles after each basic block. The output feature map z of each basic block in the feature extraction module is... j Features are the outputs of each stage of the feature extraction module.
[0049] In existing technologies, attention modules are widely used in computer vision tasks to guide models to focus on important regions that are beneficial to the model's final prediction. Traditional channel attention modules only consider the direct addition and fusion of average pooling and max pooling features, ignoring the importance of average pooling and max pooling features to the final output features, resulting in a lack of robustness in the final output features.
[0050] To address the above issues, this method proposes an adaptive channel attention module, referencing... Figure 3Specifically, an adaptive channel attention module is added after the feature map output by each basic block. In this module, max pooling and average pooling are first used to aggregate the channel information of the feature map output by each basic block, generating two different channel vectors q. max ,q avg ∈R c×1×1 These represent the max-pooling and average-pooling features of the feature map, respectively. Next, these two features are input into the MLP and its corresponding adaptive weight module. The MLP has only one hidden layer with a size of R. c / r×1×1 , where r represents the reduction ratio. The adaptive weighting module consists of fully connected layers. This module adaptively adjusts the contribution of max pooling and average pooling features to the final output channel weight features, denoted as q. max ,q avg Generate the corresponding weight w max ,w avg Then, q max and w max Features after multiplication and q avg and w avg The multiplied features are then added together to obtain the final channel weight feature f. cw Finally, the obtained f cw With input feature map z j Multiplication yields the final output feature map. This achieves the processing of the input feature map z j Weighting along the channel dimension. This can be specifically formulated as follows:
[0051] q max =MaxPool(z j ),
[0052] q avg =AvgPool(z j ),
[0053] w max =W m1 (σ(W m0 (q max ))),
[0054] w avg =W a1 (σ(W a0 (q avg ))),
[0055] f cw =σ(MLP(q) max )*w max +MLP(q avg )*w avg ),
[0056]
[0057] Where σ represents the ReLU activation function, MaxPool and AvgPool represent the max pooling operation and average pooling operation, respectively, MLP(*) represents the output of the specified vector after entering the MLP, and W(*) represents the parameters of the fully connected layer.
[0058] Since the semantic information and size of the output features of different basic blocks in ResNet-50 vary, a reshaping module is proposed to compensate for the weak semantic information of shallow basic blocks and to facilitate subsequent self-distillation. This module further refines the features output by shallow basic blocks and ultimately maps the features of each basic block output to the same size dimension. Then, fully connected layers and a softmax function are used to compute each... The corresponding predicted score f j The reshaping module mainly consists of several stacked convolutional, normalization, and activation layers. Specifically, it is represented as follows:
[0059]
[0060]
[0061] Where μ(·ε) represents the convolution operation, ε represents the convolution kernel parameter, BN and σ represent the normalization layer and activation layer, respectively, and c represents the number of basic blocks. represent The output after the i-th convolutional layer, normalization layer, and activation layer; f j express The predicted score obtained after passing through the fully connected layer and the Softmax layer, where W represents the parameters of the fully connected layer.
[0062] Step 3: Align faces in the images input to the training, test, and validation sets, and crop them to 224x224. Selective enhancements (such as random cropping, horizontal flipping, and random removal) are applied to the training set images to prevent overfitting.
[0063] Step 4: Input the split training set images into the constructed facial expression recognition model, and train and optimize the learnable parameters in the model until the accuracy of the model no longer improves.
[0064] Throughout the training process, this method utilizes a novel distillation technique to optimize the model. Knowledge distillation is a commonly used model compression method. Traditional knowledge distillation requires significant time to find and train a suitable teacher network, then distills the knowledge from the trained teacher network into the student network, thus achieving model compression. This approach is extremely costly. Self-distillation effectively solves this problem. It eliminates the need to find a teacher network, distilling the knowledge learned by the deep network into the shallow network, thereby enabling the model to learn independently. However, knowledge learning should be diverse and gradual. Traditional self-distillation only considers using the output of the deepest basic block as the output of the teacher network and the output of other basic blocks as the output of the student network. After each training iteration, the student network extracts knowledge from the teacher network, which is not conducive to the student network's learning of diverse knowledge.
[0065] To address the aforementioned issues, this method proposes a novel self-distillation approach. Except for the last basic block, each basic block treats its next basic block as a teacher. During knowledge distillation, distillation proceeds progressively from deep to shallow layers, ensuring that the current basic block learns the knowledge of all subsequent basic blocks. This guarantees that the shallow student network can learn diverse knowledge. This self-distillation approach effectively transfers semantically richer knowledge from deep networks to shallow networks, thereby improving the representational power of shallow network output features and promoting the model's accurate classification of facial expressions. The total self-distillation loss L... total The definition is as follows:
[0066] L total =L ce +αL kl +βL2,
[0067] Where L ce L represents the cross-entropy loss. kl α represents the KL divergence loss, L2 represents the L2 loss, and α and β are hyperparameters used to adjust the contribution of KL loss and L2 loss to the total loss.
[0068] Since the prediction of the last basic block lacks an evaluation metric, this method utilizes cross-entropy loss to calculate the difference between the predicted score of the last basic block after passing through the Softmax layer and the true label on the training set. This provides an evaluation metric for the predicted score of the last basic block, thereby promoting model learning. Specifically, it is expressed as follows:
[0069] L ce =CrossEntropyLoss(f c ,y),
[0070] Where CrossEntropyLoss represents the cross-entropy loss, f c y represents the predicted score of the Cth (deepest) basic block, and y represents the corresponding label.
[0071] KL divergence is primarily used to measure the difference between two probabilities. In this method, KL divergence is used to measure the difference between the predicted score of the current basic block (student) and the predicted score of its next basic block (teacher), thereby guiding the student's prediction of the sample to be as close as possible to the teacher's prediction. Specifically, it is expressed as follows:
[0072]
[0073] Where KL represents the KL divergence loss. This means that gradient updates are canceled during gradient backpropagation.
[0074] L2 loss is used to supervise the features output by the student network. By using L2 loss, the features output by the student network are made as close as possible to the features output by the teacher network, thereby improving the representational power of the student network's output features. Specifically, it is represented as follows:
[0075]
[0076] in This means that gradient updates are canceled during gradient backpropagation.
[0077] Step 5: Perform inference on the images in the test and validation sets. During the inference phase, remove the adaptive channel attention module, the reshaping module, and the self-distillation network. Only use the feature extraction module trained during the training phase to make the final prediction on the input. Input the processed training and test set images into the model to obtain the corresponding classification results, which are one of the seven basic expressions: disgust, happiness, anger, fear, surprise, sadness, and neutrality.
Claims
1. A facial expression recognition method based on attention mechanisms and self-distillation, characterized in that... Includes the following steps: S1. The public dataset is split into a training set, a test set, and a validation set. The training set, the test set, and the validation set each contain several images of the seven basic facial expressions. S2. Construct a facial expression recognition model, which includes a feature extraction module, an adaptive channel attention module, a reshaping module, and a self-distillation network; The self-distillation process is such that, except for the last basic block, each basic block extracts knowledge from its next basic block, and during self-distillation, distillation is carried out step by step from deep to shallow. S3. Perform face alignment on the images in the training, test, and validation sets input to the model, and crop them to a fixed size; perform image augmentation on the training set input to the model to prevent overfitting. S4. Input the split training set images into the constructed facial expression recognition model, and train and optimize the learnable parameters in the model until the accuracy of the model no longer improves. The specific training process includes the following sub-steps: S41. Input the processed image into the feature extraction module, and output the feature map of each basic block in the feature extraction module. As features output at each stage of the feature extraction module; S42. An adaptive channel attention module is added after the feature map output by each basic block. In this module, max pooling and average pooling are first used to process the feature map output by each basic block. The channel information is aggregated to generate two different channel vectors. Secondly, the adaptive weight module is used for... Generate corresponding weights Then, and Features after multiplication and The multiplied features are then added together to obtain the final channel weight features. Finally, the result will be With input feature map Multiplication yields the final output feature map. The calculation formula is as follows: ； ； ； ； ； ； in Represents the ReLU activation function. , These represent max pooling and average pooling operations, respectively. MLP(*) represents the output of the specified vector after it enters the MLP, and W(*) represents the parameters of the fully connected layer. S43. Utilize the reshaping module to further refine the features output by the shallow basic blocks and finally map the features output by each basic block to the same size dimension to obtain... The calculation formula is as follows: ； in This represents the convolution operation. Represents the convolution kernel parameters, BN and These represent the normalization layer and the activation layer, respectively, and c represents the number of basic blocks; S44. Utilize fully connected layers and the softmax function to compute each Corresponding predicted score The calculation formula is as follows: ； W represents the parameters of the fully connected layer; S45. Knowledge distillation is performed on the model from deep to shallow by using a self-distillation network and the corresponding distillation loss. S5. Perform inference on the images in the test and validation sets. During the inference phase, remove the self-distillation and attention modules, and only use the feature extraction module trained during the training phase to make the final prediction on the input. Input the processed training and test set images into the model to obtain the corresponding classification results, which are one of the seven basic expressions: disgust, happiness, anger, fear, surprise, sadness, and neutrality.
2. The facial expression recognition method based on attention mechanism and self-distillation according to claim 1, characterized in that... Step S45 includes the following sub-steps: S451. Cross-entropy loss is used to measure the difference between the predicted score of the last basic block output and the true label of the training set. The calculation formula is as follows: ； in Represents cross-entropy loss, This represents the prediction score obtained by the Cth basic block. Indicates the corresponding label; S452. The KL divergence loss is used to measure the difference between the prediction score of the current basic block and the prediction score of its next basic block. The calculation formula is as follows: ； in Represents the KL divergence loss. This indicates that gradient updates are canceled during gradient backpropagation; S453. The L2 loss is used to measure the difference between the features output by the current basic block and the features output by its next basic block. The calculation formula is as follows: ； Represents L2 loss; S454. Finally, the total self-distillation loss The calculation formula for the loss, which consists of the above three types of losses, is as follows: ； These are hyperparameters used to adjust the contributions of KL loss and L2 loss to the total loss.
3. The facial expression recognition method based on attention mechanism and self-distillation according to claim 1, characterized in that, The adaptive channel attention module consists of a channel attention module and an adaptive weight module; The channel attention module operates on the average pooling features and max pooling features, using an attention mechanism to determine the importance of each channel and weighting each channel according to its importance; the adaptive weight module operates on the average pooling features and max pooling features, using an attention mechanism to determine the importance of the average pooling features and max pooling features and generating corresponding weights.