Expression recognition method for partially occluded faces based on multi-scale attention mechanism

Abstract

This invention relates to a method for facial expression recognition based on a multi-scale attention mechanism for partially occluded faces, belonging to the field of computer vision within computer science. The implementation steps are as follows: First, a training set and a validation set are determined, and the images are preprocessed. Then, an expression recognition model is constructed by adding attention mechanisms after the four residual structures of the ResNet18 network, forming a "multi-scale attention mechanism" to process feature maps of different sizes and extract different feature information. A "partition loss function" is used to guide the four attention mechanisms to focus on different regions, extracting multi-scale feature vectors and inputting them into a Softmax classifier to obtain the facial expression category in the image. Finally, the model is trained using the training set and validated using the validation set. Once the model's accuracy meets the requirements, it is used to recognize the facial expression category of the input image. This method can improve the accuracy of expression recognition for images with partially occluded faces.

Description

Technical Field

[0001] This invention belongs to the field of computer vision under the discipline of computer science, and specifically relates to an expression recognition method for partially occluded faces based on a multi-scale attention mechanism. Background Technology

[0002] Facial expressions are one of the most effective ways for people to directly express their inner emotions, states, and intentions. With the rapid development of deep learning, facial expression recognition has been widely applied in education, affective computing, and human-computer interaction. The image-based facial expression recognition process typically involves three stages: face detection, feature extraction, and expression classification. Among these, the feature extraction capability has a significant impact on the accuracy of facial expression recognition results. However, in real-world scenarios, everyday accessories such as masks, glasses, and hats can obscure facial information. When the face is partially obscured, feature extraction becomes more difficult, affecting the accuracy of facial expression recognition.

[0003] Current research on facial expression recognition typically employs deep neural networks, using convolutional layers to pass features and obtain a final global feature map. A convolutional block attention module (CBAM) then focuses on this global feature map for expression recognition. A CBAM is a module that forces the deep neural network to select meaningful features. In expression recognition models, CBAM helps focus on important facial regions and suppress irrelevant areas and background. While most studies focus on key facial expression regions, extracting expression features solely from the entire image can easily overlook local details and reduce the accuracy of expression recognition.

[0004] In convolutional neural networks (CNNs), shallow feature maps have higher resolution and contain rich texture features from the input image. However, due to fewer convolutional computations, shallow feature maps tend to have less semantic information. Deep feature maps, obtained after more convolutional computations, extract richer semantic features from the input image, but they are also prone to losing detailed features. Combining shallow and deep features allows CNNs to leverage complementary information to improve feature extraction capabilities. This invention extracts and fuses features from the convolutional maps obtained under each residual structure of the ResNet18 network.

[0005] When the face is partially obscured by objects, facial expression features become difficult to extract due to varying degrees of occlusion. To ensure the accuracy of expression recognition, extracting facial expression features from the unoccluded areas becomes crucial. Facial expression features are not limited to a single part of the face but are distributed across different regions. Using only a single convolutional attention module is insufficient to comprehensively extract facial expression features from the unoccluded areas. Therefore, to enhance the network's ability to extract subtle changes in facial expressions from unoccluded areas, this invention incorporates an attention mechanism after each set of residual structures in the ResNet18 network. This mechanism assigns higher weights to unoccluded areas in feature maps of different scales and merges multiple non-overlapping and salient local regions into a single feature map, extracting features at different scales and improving feature accuracy. This results in a facial expression recognition method for partially occluded faces based on a multi-scale attention mechanism. Summary of the Invention

[0006] In view of this, the present invention provides a method for facial expression recognition based on a multi-scale attention mechanism for partially occluded faces, the implementation of which is as follows:

[0007] Step 1: Determine the training set and validation set;

[0008] The training set includes the training sets of the RAF-DB, FER+, and FED-RO public datasets, and the validation set includes the validation sets of the RAF-DB, FER+, and FED-RO public datasets; facial expressions include seven categories: surprise, fear, disgust, happiness, sadness, anger, and neutral.

[0009] Step 2: Preprocess the images in the training and test sets. The specific steps are as follows:

[0010] Step 2.1: Convert the images in the training and test sets to a uniform size;

[0011] Step 2.2: Enhance the training set using the following method:

[0012] Images of uniform size are randomly flipped horizontally, randomly occluded, and randomly cropped to enhance the dataset and prevent overfitting during training.

[0013] Step 2.3: Convert the images in the enhanced training set into tensors;

[0014] Step 3: Construct a "network model for facial expression recognition based on a multi-scale attention mechanism for partially occluded faces," and name this network model Model M. Model M consists of four parts: a ResNet18 network, a multi-scale attention mechanism, a "partition loss function," and multi-scale feature classification. Among them, the ResNet18 network is the base network, the construction method of the multi-scale attention mechanism is as described in Step 4, the setting method of the "partition loss function" is as described in Step 5, and the method of multi-scale feature classification is as described in Step 6.

[0015] Step 4: Add four attention mechanisms to the base network. Each attention mechanism has the same structure. The specific steps are as follows:

[0016] Step 4.1: Use ResNet18 network as the base network, which has four sets of residual structures;

[0017] Step 4.2: Add an attention mechanism after the first set of residual structures in the ResNet18 network and name the attention mechanism attention_1;

[0018] The first set of residual structures of the ResNet18 network outputs a feature map of size 64*56*56, which contains a lot of fine-grained information. After the first set of residual structures, attention_1 is added to process the 64*56*56 feature map into a feature vector of length 289, which can extract local facial expression features with "small expression amplitude" from the fine-grained information.

[0019] Step 4.3: Add an attention mechanism after the second set of residual structures in the ResNet18 network and name the attention mechanism attention_2;

[0020] The second set of residual structures of the ResNet18 network outputs a feature map of size 128*28*28, which has strong geometric features. After the second set of residual structures, attention_2 is added to process the 128*28*28 feature map into a feature vector of length 289, which can extract facial expression features with "smaller expression amplitude" from the geometric features.

[0021] Step 4.4: Add an attention mechanism after the third set of residual structures in the ResNet18 network and name the attention mechanism attention_3;

[0022] The third residual structure of the ResNet18 network outputs a feature map of size 256*14*14, which has a larger receptive field than the feature maps output by the first two residual structures. Attention_3 is added after the third residual structure to process the 256*14*14 feature map into a feature vector of length 289, thereby extracting local facial expression features with "larger expression amplitude".

[0023] Step 4.5: Add an attention mechanism after the fourth set of residual structures in the ResNet18 network and name the attention mechanism attention_4;

[0024] The fourth residual structure of the ResNet18 network outputs a feature map of size 512*7*7, which has strong edge contour information. Adding attention_4 after the fourth residual structure can give relatively high weight to the salient edge contour, and process the 512*7*7 feature map into a feature vector of length 289 to extract high-level semantic features such as facial contour and color.

[0025] Thus, attention_1, attention_2, attention_3, and attention_4, four attention mechanisms, are applied to the four residual structures of the ResNet18 network to process feature maps of different sizes and extract different feature information. Together, these four attention mechanisms form a "multi-scale attention mechanism".

[0026] Step 5: Set the "Partition Loss Function";

[0027] The "partition loss function" can guide the four attention mechanisms to focus on different regions during training, as shown in Equation 1:

[0028]

[0029] in, The mean squared error of the feature vectors generated by the four attention mechanisms on the same dimension, N is the number of feature vectors, C is the dimension of each feature vector, and k is an adjustment factor used to adjust the influence of the inverse mean squared error on the loss function, which is generally set to 10;

[0030] During training, when the four attention mechanisms focus on the same region, the resulting feature similarity is high, and the partitioning loss is greater; when the four attention mechanisms focus on different regions, the resulting feature similarity is low, and the partitioning loss will decrease; thus, the four attention mechanisms focus on different regions respectively.

[0031] Step 6: Multi-scale feature classification;

[0032] The four feature vectors of length 289 extracted through four attention mechanisms are concatenated to obtain a multi-scale feature vector of length 289; the multi-scale feature vector is then input into a Softmax classifier to obtain the facial expression category in the image.

[0033] Thus, the ResNet18 network, the multi-scale attention mechanism described in step four, the "partition loss function" described in step five, and the multi-scale feature classification described in step six together constitute the "expression recognition network model for partially occluded faces based on multi-scale attention mechanism", namely model M.

[0034] Step 7: Train model M using the training set;

[0035] Training is stopped when the accuracy of model M on the training set changes by less than 1% over 10 to 15 consecutive epochs, and the trained model is named model H.

[0036] Step 8: Validate the effectiveness of model H using a validation set. The specific method is as follows:

[0037] The effectiveness of model H is evaluated by its accuracy on the validation set. If the accuracy is higher than 90%, model H is considered effective and step nine can be executed; otherwise, the process exits.

[0038] Step 9: Use model H to perform facial expression recognition;

[0039] The image to be identified is used as the input of model H. After being identified by model H, the corresponding facial expression category of the image is output.

[0040] At this point, the "Expression Recognition Method for Partially Occluded Faces Based on Multi-Scale Attention Mechanism" has been completed.

[0041] Beneficial effects:

[0042] The method proposed in this invention can process feature maps of different sizes and extract different feature information when the face is partially occluded by real objects. It also uses a "partition loss function" to make the multi-scale attention mechanism focus on different regions, thereby improving the accuracy of facial expression category recognition. On the RAF-DB public dataset, the accuracy of model H is 92.03%, which is 3.67%, 3.05% and 2.33% higher than the accuracy of PSR model, DAN model and EfficientFace model, respectively. Attached image description:

[0043] Figure 1 The flowchart of this invention

[0044] Figure 2The structure of model M proposed in this invention

[0045] Figure 3 The structure of the attention mechanism in model M proposed in this invention. Detailed Implementation

[0046] The present invention will now be described in detail with reference to the accompanying drawings and embodiments;

[0047] Step 1: Determine the training set and validation set;

[0048] The training set includes the training sets of the RAF-DB, FER+, and FED-RO public datasets, and the validation set includes the validation sets of the RAF-DB, FER+, and FED-RO public datasets; facial expressions include seven categories: surprise, fear, disgust, happiness, sadness, anger, and neutral.

[0049] In this embodiment, the RAF-DB, FER+, and FED-RO public datasets were downloaded respectively. The download address for RAF-DB is http: / / www.whdeng.cn / raf / model1.html#dataset, the download address for FER+ is https: / / www.kaggle.com / datasets / msambare / fer2013, and the download address for FED-RO is https: / / github.com / mysee1989 / PG-CNN.

[0050] Step 2: Preprocess the images in the training and test sets. The specific steps are as follows:

[0051] Step 2.1: Convert the images in the training and test sets to a uniform size;

[0052] In this embodiment, the images in the training set and the test set are uniformly adjusted to a size of 224*224;

[0053] Step 2.2: Enhance the training set using the following method:

[0054] Images of uniform size are randomly flipped horizontally, randomly occluded, and randomly cropped to enhance the dataset and prevent overfitting during training.

[0055] Step 2.3: Convert the images in the enhanced training set into tensors;

[0056] In this embodiment, the image is converted into a tensor of size 3*224*224;

[0057] Step 3: Construct a "Network Model for Facial Expression Recognition Based on a Multi-Scale Attention Mechanism for Partially Occluded Faces," and name this network model Model M. Model M consists of four parts: a ResNet18 network, a multi-scale attention mechanism, a "partition loss function," and multi-scale feature classification. The ResNet18 network is the base network. The construction method for the multi-scale attention mechanism is as described in Step 4, the setting method for the "partition loss function" is as described in Step 5, and the multi-scale feature classification method is as described in Step 6. The network structure of Model M is shown in the attached figure. Figure 2 As shown;

[0058] Step 4: Add four attention mechanisms to the base network. Each attention mechanism has the same structure. The structure of the attention mechanisms in model M is shown in the attached figure. Figure 3 As shown, the specific steps for adding four attention mechanisms are as follows:

[0059] Step 4.1: Use ResNet18 network as the base network, which has four sets of residual structures;

[0060] Step 4.2: Add an attention mechanism after the first set of residual structures in the ResNet18 network and name the attention mechanism attention_1;

[0061] The first set of residual structures of the ResNet18 network outputs a feature map of size 64*56*56, which contains a lot of fine-grained information. After the first set of residual structures, attention_1 is added to process the 64*56*56 feature map into a feature vector of length 289, which can extract local facial expression features with "small expression amplitude" from the fine-grained information.

[0062] Step 4.3: Add an attention mechanism after the second set of residual structures in the ResNet18 network and name the attention mechanism attention_2;

[0063] The second set of residual structures of the ResNet18 network outputs a feature map of size 128*28*28, which has strong geometric features. After the second set of residual structures, attention_2 is added to process the 128*28*28 feature map into a feature vector of length 289, which can extract facial expression features with "smaller expression amplitude" from the geometric features.

[0064] Step 4.4: Add an attention mechanism after the third set of residual structures in the ResNet18 network and name the attention mechanism attention_3;

[0065] The third residual structure of the ResNet18 network outputs a feature map of size 256*14*14, which has a larger receptive field than the feature maps output by the first two residual structures. Attention_3 is added after the third residual structure to process the 256*14*14 feature map into a feature vector of length 289, thereby extracting local facial expression features with "larger expression amplitude".

[0066] Step 4.5: Add an attention mechanism after the fourth set of residual structures in the ResNet18 network and name the attention mechanism attention_4;

[0067] The fourth residual structure of the ResNet18 network outputs a feature map of size 512*7*7, which has strong edge contour information. Adding attention_4 after the fourth residual structure can give relatively high weight to the salient edge contour, and process the 512*7*7 feature map into a feature vector of length 289 to extract high-level semantic features such as facial contour and color.

[0068] Thus, attention_1, attention_2, attention_3, and attention_4, four attention mechanisms, are applied to the four residual structures of the ResNet18 network to process feature maps of different sizes and extract different feature information. The four attention mechanisms together constitute the "multi-scale attention mechanism".

[0069] Step 5: Set the "Partition Loss Function";

[0070] The "partition loss function" can guide the four attention mechanisms to focus on different regions respectively during training;

[0071] During training, when the four attention mechanisms focus on the same region, the resulting feature similarity is high, and the partitioning loss is greater; when the four attention mechanisms focus on different regions, the resulting feature similarity is low, and the partitioning loss will decrease; thus, the four attention mechanisms focus on different regions respectively.

[0072] In this embodiment, N is 4, C is 289, and k is set to 10;

[0073] When four attention mechanisms focus on the same region, "partition loss" The calculation method for “” is shown in Formula 2:

[0074]

[0075] When four attention mechanisms focus on different regions, "partition loss" The calculation method for “ is shown in Formula 3:

[0076]

[0077] Step 6: Multi-scale feature classification;

[0078] The four feature vectors of length 289 extracted through four attention mechanisms are concatenated to obtain a multi-scale feature vector of length 289; the multi-scale feature vector is then input into a Softmax classifier to obtain the facial expression category in the image.

[0079] In this embodiment, the feature vectors generated by the four attention mechanisms are concatenated in the 0th dimension to obtain a 4*289 dimensional feature vector. Then, the summation is performed in the 0th dimension to obtain a multi-scale feature vector of length 289. The torch.nn.Linear(289, 7) method is used to obtain a 7-dimensional feature vector. The Softmax activation function is used as a facial expression classifier to output a 7-dimensional feature vector with a sum of 1, where each dimension represents the probability of recognizing various facial expressions.

[0080] Thus, the ResNet18 network, the multi-scale attention mechanism described in step four, the "partition loss function" described in step five, and the multi-scale feature classification described in step six together constitute the "expression recognition network model for partially occluded faces based on multi-scale attention mechanism", namely model M.

[0081] Step 7: Train model M using the training set;

[0082] Training is stopped when the accuracy of model M on the training set changes by less than 1% over 10 to 15 consecutive epochs, and the trained model is named model H.

[0083] The specific steps for training model M in this embodiment are as follows:

[0084] (1) Define the hyperparameters of model M: batch_size is 256, epoch is 40, learning rate is 0.01, and learning rate decay is 0.9. The training parameter settings are shown in Table 1.

[0085] Table 1 Training Parameter Settings

[0086] parameter Parameter settings Optimization Algorithm Adam Learning rate 0.01 Learning rate decay 0.9 batch_size 256 epoch 40

[0087] (2) Select different dataset paths, use the datasets.ImageFolder() method in the torchvision library to read the training set of the dataset respectively, use the utils.data.DataLoader() method in the torch library to construct a data loader and split the data into batches, and load the data in batches during training;

[0088] (3) Use model.train() to start training the network;

[0089] (4) When training the data in batches, use optimizer.zero_grad() to clear the gradients, use loss.backward() to backpropagate and calculate the gradients, and use optimizer.step() to update the network parameters;

[0090] (5) Use tqdm.write() to output the average accuracy and average loss of each epoch training set to the console, and use json.dump() to save the average accuracy and average loss of each epoch training set to a json file;

[0091] (6) After model M finishes training, use torch.save() to save the trained model and name it model H;

[0092] Step 8: Validate the effectiveness of model H using a validation set. The specific method is as follows:

[0093] The effectiveness of model H is evaluated by its accuracy on the validation set. If the accuracy is higher than 90%, model H is considered effective and step nine can be executed; otherwise, the process exits.

[0094] In this embodiment, the specific steps for verifying model H are as follows:

[0095] (1) Select different dataset paths, use the datasets.ImageFolder() method in the torchvision library to read the validation set of the dataset respectively, use the utils.data.DataLoader() in the torch library to construct a data loader and split the data into batches, and load the data in batches during validation;

[0096] (2) Load model H using torch.loader();

[0097] (3) Use model.eval() to start the validation of model H;

[0098] (4) Use tqdm.write() to output the average accuracy and average loss on each epoch of the validation set to the console, and use json.dump() to save the average accuracy and average loss on each epoch of the validation set to a json file;

[0099] (5) Model H has an accuracy of 92% on the validation set, which is higher than 90%, so model H is considered usable;

[0100] Step 9: Use model H to perform facial expression recognition;

[0101] The image to be identified is used as the input of model H. After being identified by model H, the corresponding facial expression category of the image is output.

[0102] In this embodiment, after the image is recognized by model H, a 7-dimensional feature vector [0.21, 0.02, 0.02, 0.52, 0.07, 0.10, 0.06] is output, where each dimension represents the probability of recognizing various facial expressions, as shown in Table 2; the facial expression category corresponding to the highest probability is the final expression recognition result, with a maximum probability of 0.52, and the final facial expression recognition category is happiness;

[0103] Table 2. Probabilities corresponding to facial expression categories

[0104] Serial Number probability Facial Expression Categories 1 0.21 surprise 2 0.02 fear 3 0.02 disgust 4 0.52 hapiness 5 0.07 sad 6 0.10 anger 7 0.06 neutral

[0105] At this point, the "Expression Recognition Method for Partially Occluded Faces Based on Multi-Scale Attention Mechanism" has been completed.

[0106] Table 3 shows the comparison of the accuracy of Model H with that of PSR, DAN and EfficientFace facial expression recognition methods proposed in recent years on the RAF-DB public dataset:

[0107] Table 3 Comparison of Model Accuracy

[0108] Model Year the model was proposed Accuracy (%) PSR 2020 88.36 DAN 2021 88.98 EfficientFace 2021 89.70 Model H 2022 92.03

[0109] Model H improved accuracy by 3.67%, 3.05%, and 2.33% compared to the other three models, respectively; and it can accurately identify facial expression categories when the face is partially obscured by objects.

Claims

1. A method for expression recognition of partially occluded faces based on multi-scale attention mechanism, characterized in that, The method comprises the following steps: Step one, determining the training set and the validation set; Wherein the training set comprises the training set of RAF-DB, FER+, and FED-RO public data sets respectively, and the validation set comprises the validation set of RAF-DB, FER+, and FED-RO public data sets; the facial expression comprises seven types of surprise, fear, disgust, happiness, sadness, anger, and neutrality; Step two, preprocessing the images in the training set and the test set, and the specific steps are: Step 2.1, converting the images in the training set and the test set into a unified size; Step 2.2, enhancing the training set, and the specific method is: The unified size image is randomly horizontally flipped, randomly occluded, and randomly cropped to enhance the data set, so as to prevent overfitting in the training process; Step 2.3, converting the enhanced images in the training set into tensors; Step three, constructing an "expression recognition network model for partially occluded faces based on a multi-scale attention mechanism", and naming the network model as model M; the model M is composed of a ResNet18 network, a multi-scale attention mechanism, a "partition loss function", and a multi-scale feature classification; wherein the ResNet18 network is a basic network, the construction method of the multi-scale attention mechanism is as step four, the setting method of the "partition loss function" is as step five, and the method of multi-scale feature classification is as step six; Step four, adding four attention mechanisms to the basic network, and the specific steps are: Step 4.1, using the ResNet18 network as the basic network, and the network has four groups of residual structures; Step 4.2, adding an attention mechanism after the first group of residual structures of the ResNet18 network, and naming the attention mechanism as attention_1; The first group of residual structures of the ResNet18 network outputs a feature map with a size of 64*56*56, and the feature map has more fine-grained information; after the first group of residual structures, the attention_1 processes the feature map with a size of 64*56*56 into a feature vector with a length of 289, which can extract the local facial expression features of "micro-expression amplitude" from the fine-grained information; Step 4.3, adding an attention mechanism after the second group of residual structures of the ResNet18 network, and naming the attention mechanism as attention_2; The second group of residual structures of the ResNet18 network outputs a feature map with a size of 128*28*28, and the feature map has strong geometric features; after the second group of residual structures, the attention_2 processes the feature map with a size of 128*28*28 into a feature vector with a length of 289, which can extract the facial expression features of "smaller expression amplitude" from the geometric features; Step 4.4, adding an attention mechanism after the third group of residual structures of the ResNet18 network, and naming the attention mechanism as attention_3; The third group of residual structures of the ResNet18 network outputs a feature map with a size of 256*14*14, and the receptive field of the feature map is larger than that of the feature maps output by the first two groups of residual structures; an attention_3 is added after the third group of residual structures, which processes the feature map with a size of 256*14*14 into a feature vector with a length of 289, and extracts the local facial expression features with a "relatively large expression amplitude"; Step 4.5, an attention mechanism is added after the fourth group of residual structures of the ResNet18 network, and the attention mechanism is named as attention_4; The fourth group of residual structures of the ResNet18 network outputs a feature map with a size of 512*7*7, and the feature map has strong edge contour information; an attention_4 is added after the fourth group of residual structures, which can give relatively high weights to the significant edge contours, processes the feature map with a size of 512*7*7 into a feature vector with a length of 289, and extracts the contour and color high-level semantic features of the face; At this point, the four attention mechanisms of attention_1, attention_2, attention_3, and attention_4 are added after the four groups of residual structures of the ResNet18 network, respectively, to process feature maps of different sizes and extract different feature information, and the four attention mechanisms together form a "multi-scale attention mechanism"; Step five, set a "partition loss function"; The "partition loss function" can guide the four attention mechanisms to focus on different regions during training, as shown in formula (1): wherein, is the mean square error of the feature vectors produced by the four attention mechanisms in the same dimension, N is the number of feature vectors, C is the dimension of each feature vector, and k is an adjustment factor for adjusting the influence of the inverse of the mean square error on the loss function, which is generally set to 10; During training, when the four attention mechanisms focus on the same region, the similarity of the features generated is high, and the partition loss is large; when the four attention mechanisms focus on different regions, the similarity of the features generated is low, and the partition loss will decrease; so that the four attention mechanisms focus on different regions; Step six, multi-scale feature classification; After the four feature vectors with a length of 289 extracted by the four attention mechanisms are spliced, a multi-scale feature vector with a length of 289 is obtained; the multi-scale feature vector is input into a Softmax classifier to obtain the facial expression category in the image; At this point, the ResNet18 network, the multi-scale attention mechanism described in step four, the "partition loss function" described in step five, and the multi-scale feature classification described in step six together form an "expression recognition network model for partially occluded faces based on a multi-scale attention mechanism", which is model M; Step seven, use the training set to train the model M, and the training parameter settings are shown in Table 1: Table 1. Training parameter settings When the accuracy of the model M on the training set changes within 1% within "10 to 15 consecutive epochs", stop training, and name the model obtained after training as model H; Step eight, use the validation set to verify the effectiveness of the model H, and the specific method is: The accuracy of the model H on the validation set is used to evaluate the effectiveness of the model H, and when the accuracy is higher than 90%, the model H is considered effective, and step nine can be performed, otherwise, exit. Step nine, facial expression recognition is performed using the model H; The image to be recognized is input into the model H, and the corresponding facial expression category of the image is output after recognition by the model H; the "facial expression recognition method for partially occluded faces based on a multi-scale attention mechanism" is completed.

Images