Occluded Facial Expression Recognition Model Based on Facial Hybrid Enhancement Strategy

Abstract

The present invention discloses an occluded face expression recognition model based on a face hybrid enhancement strategy in the field of computer vision technology, including: an occluded face expression recognition module; the occluded face expression recognition module includes a local feature enhancement module that enables the hybrid region to have strong expression-related semantic information, a high-level / intermediate-level learning representation module that learns the non-occluded local features of face features and filters the most discriminative local features, and a prediction module that includes the calculation of a loss function and the output of a prediction result. The face hybrid enhancement strategy FERMix and the learning of intermediate feature representations are used to improve the occlusion robustness of face expression recognition. The main image and the hybrid image are simultaneously input into the network for training to enhance the model's ability to represent global and local information, thereby improving the performance of the model and its robustness to occlusion. The intermediate feature representation layer is used for the first time in the face expression recognition task, and the accuracies on the wild face expression datasets are 89.80%, 89.58%, and 65.54% respectively.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and specifically to an occluded face expression recognition model based on a face hybrid enhancement strategy. Background Art

[0002] Facial expressions are one of the most common and important signals for conveying human emotions and intentions. Therefore, being able to correctly recognize facial expressions is of great significance for understanding user emotions and intentions. As a basic task in the fields of computer vision and emotion analysis, facial expression recognition is widely used in applications such as online education, healthcare, home care, product recommendation, video recommendation, and online monitoring to obtain user preferences and mental states in order to achieve "humanized" human-computer interaction.

[0003] Today, with the popularity of deep learning, the end-to-end facial expression recognition technology based on a large amount of data has become increasingly mature, but there are still many problems that need to be solved and considered. In the real world, the face images obtained by machines are not always complete frontal face images, but often there are facial occlusions. The occluders may be mobile phones, glasses, hands, hair, etc. These situations will cause the loss of key facial information, resulting in a sharp decline in the performance of the expression recognition system. Therefore, many current facial expression recognition studies are dedicated to improving the robustness of the expression recognition system to facial occlusions. There are the following several solutions to this problem.

[0004] 1) Expand the dataset. Add more occluded face images to the dataset. This is the most direct and effective way, but the difficulty lies in the fact that collecting the dataset is time-consuming and laborious, and there is still no directly available large-scale occluded face expression dataset for training.

[0005] 2) Face completion. This approach is to use a generation method to complete the occluded part of the face, and then send the completed face into the expression recognition network for classification. However, such methods also rely on the generation model being able to learn good expression features, and the training difficulty is relatively high. Coupled with the fact that existing research basically uses artificially simulated occluders to replace real occlusion situations, these methods are difficult to generalize to real scenarios.

[0006] 3) Enhance the ability to represent information in local facial regions. Psychological research shows that humans can effectively use local regions and the overall face to perceive the semantics conveyed by incomplete faces. When some parts of the face are occluded, humans can judge the expression based on other non-occluded local regions. Therefore, many methods hope to enhance the ability of local facial regions to represent expression features, so that even when the face is occluded, the expression can be recognized using non-occluded local regions. Currently, this solution has achieved better results.

[0007] Regarding how to enhance the ability to represent local region information, existing research work has proven that providing local face information in the model input can enhance the model's learning of local features, and jointly training global information and local information can effectively improve the model's generalization ability and robustness to occlusion. However, the above work makes the model include a large amount of data input in one training and inference, which is relatively inefficient in the training and inference process. Moreover, the selection of local regions depends on face key point detection. However, in the case of occlusion, face key point detection is not accurate. Therefore, this method may fail in practical applications.

[0008] In order to have better training and inference efficiency while enhancing the model's robustness to occlusion, this application designs a simple and efficient model for occluded face expression recognition, namely FER (Facial Expression Recognition) MixNet, which does not require any additional information.

[0009] Therefore, this application proposes an occluded face expression recognition model based on a face mixing enhancement strategy. Summary of the Invention

[0010] The purpose of the present invention is to provide an occluded face expression recognition model based on a face mixing enhancement strategy to solve the problems raised in the above background technology.

[0011] To achieve the above purpose, the present invention provides the following technical solution: An occluded face expression recognition model based on a face mixing enhancement strategy, including:

[0012] An occluded face expression recognition module;

[0013] The occluded face expression recognition module includes a local feature enhancement module that enables the mixed region to have strong expression-related semantic information, a high-level / intermediate-level learning representation module that learns the non-occluded local features of face features and filters the most discriminative local features, and a prediction module that includes the calculation of the loss function and the output of the prediction result.

[0014] Further, the local feature enhancement module includes a horizontal equal division mixing module for upper and lower face occlusion, a vertical equal division mixing module for left and right face occlusion, and a corner module for single-eye occlusion.

[0015] Further, the high-level / intermediate-level learning representation module includes a deep convolutional network module. A middle-level feature representation layer is added to the deep convolutional network module, and the middle-level feature representation layer includes a × convolutional layer and a max pooling layer.

[0016] Further, the recognition method of the occluded face expression recognition model based on the face mixing enhancement strategy includes the following steps:

[0017] S1: Face mixing enhancement:

[0018] Mix two face images, and try to ensure the integrity of the eye and mouth regions as much as possible during mixing. After face alignment, the face expression dataset has strong particularities. Utilize these special properties to conveniently use the face mixing enhancement strategy (FERMix) without additional information;

[0019] Horizontal equal division mixing: In a face image, a person's two eyes are generally located in the upper half, and the mouth is generally located in the lower half. Therefore, directly cut the image in half horizontally to divide the eye region and the mouth region. This mixing method can make the model pay more attention to the upper half or the lower half region of the face, increasing the model's robustness to upper and lower occlusions;

[0020] Vertical equal division mixing: Directly cut the image in half vertically to divide the left and right face regions. This mixing method can make the model pay more attention to the left half or the right half of the face to increase the model's robustness to left and right occlusions;

[0021] Corner mixing: Corresponds to the left and right eye regions. When calculating the mixed cross-entropy loss, since this region only accounts for 1 / 4 of the original image size, the weight assigned to this region is also small and will not have too much negative impact on the model;

[0022] S2: Randomly shuffle the main image to generate paired images:

[0023] Input two parts including the main image and the FERMix mixed image into the model (FERMixNet). The main image selects a sample from a normal training batch. In a batch of model training, obtain their paired images by randomly shuffling the images in this batch. In each iteration, the model will select a combination of FERMix with a certain probability to generate the mixed image;

[0024] S3: Feature representation:

[0025] The model structure for learning the high-level representation is the same as the standard ResNet18, including conv1, conv2_x, conv3_x, conv4_x, conv5_x, average pooling layer, and fully connected layer, which is mainly used to learn the global features of the input samples. To enhance the model's learning of local non-occluded features, this application adds an intermediate feature representation layer to the standard ResNet18. This application adds an intermediate classification branch after conv4_x, and the branch structure includes a 1×1 convolution, ReLU activation function, and max pooling layer;

[0026] After the original image passes through conv1-conv4_x, a feature map of C×H×W can be obtained. In the spatial dimension of 1×H×W, each point corresponds to the receptive field of the corresponding area in the original image, and this receptive field reflects a small local area of the original image. To more accurately locate the most discriminative local area, the stride of the 1×1 convolution is set to 1. A well-trained 1×1 convolution can have a high response to important local areas. Therefore, the most discriminative local area can be found through global max pooling (GMP). The intermediate classification part can effectively learn the local features with the highest semantic information degree of the input samples. When the face is occluded, the local information in the unoccluded area of the face can, to a great extent, help this application distinguish expressions. Therefore, learning intermediate feature representations can well supplement expression information and enhance the occlusion robustness of the model. The face is divided into multiple regions, and then the attention method is used to locate the unoccluded local areas. The intermediate feature representation layer used in this application is designed to be simpler and more flexible and does not require any additional information;

[0027] S4: Training Loss and Model Decision:

[0028] The standard cross-entropy loss is calculated for the main image and the mixed image in both the advanced classification and the intermediate classification;

[0029] During model training, the advanced representation and intermediate representation parts calculate the total loss through formulas and then update the parameters. During model inference, this application directly makes decision fusion, that is, adds the output units obtained by passing the advanced classification and intermediate classification through the fully connected layer as the final decision structure of the model.

[0030] S5: Experiment:

[0031] A: The RAF-DB (Real-world Affective Faces DataBase) dataset contains 29,672 real-world face images. All images are collected from the Internet, and there are significant differences in the age, gender, race, lighting, skin color, etc. of the objects. For the facial expression recognition task of this patent, 15,339 facial expression images are used, of which 12,271 images are used for training and 3,068 images are used for testing. The expression categories are 7: "Happy", "Surprise", "Sad", "Anger", "Disgust", "Fear", and "Neutral";

[0032] B: The FERPlus dataset is an extension of the standard FER2013 wild face expression dataset. The dataset contains 35,887 grayscale face expression images of size 48×48, including 28,709 in the training set, 3,589 in the validation set, and 3,589 in the test set. There are 8 expression categories, with an additional "Contempt" compared to RAF-DB. To address the problem of poor annotation quality in the FER2013 dataset, FERPlus had 10 annotators label each image, so each image has 10 annotation messages. This application uses the majority voting method to determine the true label of each image;

[0033] C: The AffectNet dataset is the largest wild face expression dataset today, containing 440,000 face images. The dataset contains two benchmarks: AffectNet-7 and AffectNet-8. AffectNet-7 contains 7 expression categories, the same as RAF-DB, with 283,901 in the training set and 3,500 in the test set. AffectNet-8 contains 8 expression categories, with an additional category "Contempt", with 287,651 in the training set and 4,000 in the test set. Since there is a serious problem of sample imbalance in this dataset, this application adopts the method of data downsampling to alleviate this problem;

[0034] D: Occluded face expression dataset. To verify the performance of the face expression recognition model in real-world occlusion situations, FED-RO (Facial Expression Dataset with Real Occlusion) collected and annotated 400 face images with various occlusion situations in real scenarios for model testing. This application jointly trains the model with the training sets of RAF-DB and AffectNet;

[0035] Occlusion-RAF-DB, Occlusion-FERPlus, and Occlusion-AffectNet are sub-datasets for occlusion situations constructed using the RAF-DB, FERPlus, and AffectNet datasets. They contain 735, 605, and 682 test data respectively. These sub-datasets selected occluded face images from the test sets of the original datasets as test data. The occlusion types of the data include top-bottom occlusion, left-right occlusion, glasses / mask occlusion.

[0036] S6: Data comparison:

[0037] a1: Comparison on the occlusion dataset:

[0038] Mainly to improve the robustness of the facial expression recognition model against facial occlusion, the effectiveness of this method is verified on several occlusion datasets, namely FED-RO, Occlusion-RAF-DB, Occlusion-FERPlus, and Occlusion-AffectNet. This application is compared with methods for solving occluded facial expression recognition that have performed well in recent years, including gACNN, RAN, VTFF, EfficientFace, and MAPNet;

[0039] a2: Comparison with SOTA methods:

[0040] To investigate the generality of this method in real-world complex scenarios, this application is compared with some excellent works in current facial expression recognition on three large-scale outdoor facial expression datasets, namely RAF-DB, FERPlus, and AffectNet. SCN, RUL, and DMUE address the problem of noisy labels in facial expression datasets, while DACL and FDRL focus on the problems of high intra-class distance and low inter-class distance of expressions.

[0041] Compared with the prior art, the beneficial effects of the present invention are:

[0042] 1. A model for occluded facial expression recognition, FERMixNet, is proposed. This model adopts the facial mixing enhancement strategy FERMix and learns intermediate feature representations to improve the occlusion robustness of facial expression recognition.

[0043] 2. To strengthen the learning of local facial regions, this application proposes a hybrid enhancement strategy for facial expressions, FERMix. This method simulates real-world occlusion situations by mixing the most semantically informative local regions in pairwise facial images and generates samples containing local facial information, thereby improving the robustness of the model against occlusion. To effectively utilize the local information in the FERMix mixed images, this application inputs the main image and the mixed image into the network for training simultaneously. They promote each other, enhancing the model's ability to represent global and local information, and thus improving the model's performance and robustness against occlusion.

[0044] 3. To learn the feature representations of non-occluded facial regions, this application introduces an intermediate feature representation layer in the backbone model to further improve the model's robustness against facial occlusion. To the best knowledge of this application, this is the first time to use an intermediate feature representation layer in the facial expression recognition task.

[0045] 4. The proposed method has achieved the best results to date on the face occlusion datasets Occlusion-RAF-DB, Occlusion-FERPlus, and FED-RO, with accuracies of 86.67%, 85.95%, and 72.97% respectively. It has also achieved the best results to date on the wild face expression datasets RAF-DB, FERPlus, and AffectNet-7, with accuracies of 89.80%, 89.58%, and 65.54% respectively. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] Figure 1 It is a system block diagram of the occluded face expression recognition module of the present invention;

[0047] Figure 2 It is the occluded face expression recognition model of the present invention;

[0048] Figure 3 It is the face hybrid enhancement strategy FERMix of the present invention;

[0049] Figure 4 It is to generate paired images by randomly shuffling the main image of the present invention.

[0050] In the figure: 1, local feature enhancement module; 2, high-level / intermediate-level learning representation module; 3, prediction module; 4, horizontal equal division mixing module; 5, vertical equal division mixing module; 6, corner module. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0051] Next, the technical solutions in the embodiments of the present invention will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts shall fall within the protection scope of the present invention.

[0052] Please refer to Figures 1-4 , the present invention provides a technical solution: an occluded face expression recognition model based on a face hybrid enhancement strategy, including:

[0053] Occluded face expression recognition module;

[0054] The occluded face expression recognition module includes a local feature enhancement module 1 that enables the mixed area to have strong expression-related semantic information, a high-level / intermediate-level learning representation module 2 that learns the non-occluded local features of the face features and filters the most discriminative local features, and a prediction module 3 that includes the calculation of the loss function and the output of the prediction result;

[0055] The local feature enhancement module 1 includes a horizontal equal division mixing module 4 for upper and lower face occlusion, a vertical equal division mixing module 5 for left and right face occlusion, and a corner module 6 for single-eye occlusion.

[0056] The advanced / intermediate learning representation module 2 includes a deep convolutional network module. A middle-level feature representation layer is added to the deep convolutional network module, and the middle-level feature representation layer contains a 1×1 convolution and a max pooling layer.

[0057] The recognition method of the occlusion face expression recognition model based on the face mixing enhancement strategy includes the following steps:

[0058] S1: Face mixing enhancement:

[0059] Mix two face images. When mixing, try to ensure the integrity of the eye and mouth regions as much as possible. After face alignment, the face expression dataset has strong particularities. Using these special properties, it is convenient to use the face mixing enhancement strategy FERMix without additional information;

[0060] Horizontal equal division mixing: Generally, a person's two eyes in a face image are located in the upper half, and the mouth is generally located in the lower half. Therefore, directly cutting the image in half horizontally can divide the eye region and the mouth region. It is defined by the formula:

[0061]

[0062] Indicates that the mixing area is the upper half of the image, that is, the eye region; Indicates that the mixing area is the lower half of the image, that is, the mouth region. This mixing method is used to simulate the situation of upper and lower face occlusion. Upper and lower face occlusion, such as wearing a mask, will cover the lower half of the face, leaving only the upper half; such as wearing sunglasses, covering the upper half of the face and mainly leaving the lower half. This mixing method can make the model pay more attention to the upper or lower half of the face region, increasing the model's robustness to upper and lower occlusion;

[0063] Vertical equal division mixing: Directly cutting the image in half vertically can divide the left and right face regions. This mixing method can make the model pay more attention to the left or right half of the face to increase the model's robustness to left and right occlusion; It is defined by the formula:

[0064]

[0065] Indicates that the mixing area is the left half of the image, that is, the left face; Indicates that the mixing area is the right half of the image, that is, the right face. This mixing method is for the situation of left and right face occlusion. For example, making a phone call may cause left and right occlusion.

[0066] Left deflection of the face will also cause occlusion of the left face, resulting in a larger visible area on the right half; right deflection will cause occlusion of the right face, resulting in a larger visible area on the left half.

[0067] Corner Mixing: It corresponds to the left and right eye regions. When calculating the mixed cross-entropy loss, since this region only accounts for 1 / 4 of the original image size, the weight assigned to this region is also small and will not have too much negative impact on the model. It is defined by the formula as:

[0068]

[0069] Indicates that the mixed region is the upper left part of the image, that is, the left eye; Indicates that the mixed region is the upper right part of the image, that is, the right eye. This mixing method is for the case where only one eye is occluded, such as wearing a single eye patch, hair occlusion, etc. The mixed region only contains one eye region, and it seems that there is less semantic information. However, this application believes that a single eye region can still be regarded as a complete region, and there is also approximate symmetry between the two eye regions. Therefore, the semantic information amount is sufficient. Moreover, when calculating the mixed cross-entropy loss, since this region only accounts for 1 / 4 of the original image size;

[0070] Let the main image be y A be its label; the paired image is y B be its label. The FERMix mixed image and its label can be expressed as:

[0071]

[0072]

[0073] In formula (4), the mixed region M ∈ {0, 1} W×H , is a binary mask, ⊙ is the dot product operation, the mask of the cropped region of the main image is 0, and the mask of the corresponding mixed region of the paired image is 1, indicating that the cropped part of the main image will be filled with the corresponding part of the paired image. In formula (5), λ ∈ {0.5, 0.75}. When using horizontal equal division mixing (a) and vertical equal division mixing (b), the main image accounts for half of the size of the mixed image, and λ takes the value of 0.5. When using corner mixing (c), the main image accounts for 3 / 4 of the size of the mixed image, and λ takes the value of 0.75;

[0074] Such as Figure 2As shown, the input of FERMixNet consists of two parts: the main image and the FERMix mixed image. The main image is a sample in a normal training batch, and the paired images are obtained in the way given in (9) of this application, that is, in a batch of model training, the paired images are obtained by randomly shuffling the images in this batch. As Figure 3 shown, if a training batch has four samples, the original samples serve as the main images, and the randomly shuffled samples serve as the paired images. In each iteration, the model will select a combination of FERMix with a certain probability to generate the mixed image. This application defines that the probability of using horizontal equal division mixing (a) is 40%, where the mixing region M is and both have a probability of 20%; the probability of using vertical equal division mixing (b) is 40%, where M is and both have a probability of 20%; the probability of using corner mixing (c) is 20%, where M is and both have a probability of 10%. This application believes that for (a) and (b), the mixed images retain sufficient semantic information of the main image and the paired images; while for (c), the semantic information of the paired images in the mixed images is less. Therefore, this application sets a greater probability for (a) and (b);

[0075] It should be noted that the input of the FERMix mixed image only exists in the training stage, and only the main image needs to be input in the inference stage. Moreover, in the training stage, this application also sets a probability p mix for the input of the mixed image. Generate a random value r from a uniform distribution from 0 to 1. If r < p mix , then the model input includes the main image and the mixed image; if r ≥ p mix , then the model input only has the main image and does not include the mixed image.

[0076] S2: Randomly shuffle the main image to generate paired images:

[0077] In the model FERMixNet, the input includes two parts: the main image and the FERMix mixed image. The main image is selected as a sample in a normal training batch. In a batch of model training, the paired images are obtained by randomly shuffling the images in this batch. In each iteration, the model will select a combination of FERMix with a certain probability to generate the mixed image;

[0078] S3: Feature representation:

[0079] The model structure for learning advanced representations is the same as the standard ResNet18, including conv1, conv2_x, conv3_x, conv4_x, conv5_x, average pooling layer, and fully connected layer, which is mainly used to learn the global features of input samples. To enhance the model's learning of local non-occluded features, this application adds an intermediate feature representation layer to the standard ResNet18. This application adds an intermediate classification branch after conv4_x, and the branch structure includes a 1×1 convolution, ReLU activation function, and max pooling layer; Denote the feature map obtained after conv4_x as The expression can be written as:

[0080] F mid = GMP(ReLU(Conv6(F))) #(6)

[0081] represents the obtained intermediate features, Conv6(·) represents the 1×1 convolution operation, and GMP(·) represents the global max pooling operation. The 1×1 convolution can be regarded as a detector for a small area.

[0082] After the original image passes through conv1 - conv4_x, a feature map of C×H×W can be obtained. In the spatial dimension of 1×H×W, each point corresponds to the receptive field of the corresponding area of the original image, and this receptive field reflects a small local area of the original image. To more accurately locate the most discriminative local area, the stride of the 1×1 convolution is set to 1. A well-trained 1×1 convolution can have a high response to important local areas. Therefore, the most discriminative local area can be found through global max pooling (GMP). The intermediate classification part can effectively learn the local features with the highest semantic information degree of the input samples. When the face is occluded, the local information of the non-occluded area of the face can help this application distinguish expressions to a large extent. Therefore, learning intermediate feature representations can well supplement the expression information, enhance the occlusion robustness of the model, divide the face into multiple regions, and then use the attention method to locate the unoccluded local areas. The intermediate feature representation layer used in this application is designed more simply and flexibly and does not require any additional information;

[0083] S4: Training loss and model decision:

[0084] The standard cross-entropy loss is calculated for both the main image and the mixed image in the advanced classification and intermediate classification. The specific expressions are as follows:

[0085]

[0086]

[0087] In the formula, C represents the total number of expression categories, the superscript h represents the high-level classification part, and m represents the middle-level classification part; the subscript main represents the main image classification, and mix represents the mixed image classification. y i represents whether the true label of the main image is i. If it is, take 1; otherwise, take 0; Obtained from formula (5), it represents whether the true label of the mixed image is i. The mixed image includes two parts: the main image and the mixed image. For the main image category, take λ; for the paired image category, take 1 - λ; in other cases, take 0. represents the probability that the model predicts a sample belongs to category i.

[0088] As mentioned before, the input probability of the mixed image is p mix , so the total training loss is:

[0089]

[0090] α ∈ (0, 1) is used to control the loss weight of the mixed image and the main image, and β ∈ (0, 1) is used to control the loss weight of the high-level classification and the middle-level classification. After experiments, the default settings of this application are α = 0.5 and β = 0.5;

[0091] During model training, the high-level representation and the middle-level representation parts calculate the total loss through formula (9), and then update the parameters. During model inference, this application directly performs decision fusion, that is, adding the output units obtained by passing the high-level classification and the middle-level classification through the fully connected layer as the final decision structure of the model.

[0092] S5: Experiment: To verify that the method proposed in this application can indeed improve the robustness of the model to environmental occlusion, this application tests the method on several real-world environmental expression occlusion and pose change datasets, such as Occlusion-RAF-DB, Occlusion-FERPlus, Occlusion-AffectNet, and FED-RO. At the same time, to verify the generality and effectiveness of the method of this application, this application conducts tests on three large-scale wild expression datasets - RAF-DB, FERPlus, and AffectNet.

[0093] A: The RAF-DB (Real-world Affective Faces DataBase) dataset contains 29,672 real-world face images. All images were collected from the Internet, with significant differences in the age, gender, race, lighting, skin color, etc. of the subjects. For the facial expression recognition task of this patent, 15,339 facial expression images were used, of which 12,271 images were used for training and 3,068 images were used for testing. There are 7 expression categories: "Happy", "Surprise", "Sad", "Anger", "Disgust", "Fear", and "Neutral";

[0094] B: The FERPlus dataset is an extension of the standard FER2013 wild face expression dataset. The dataset contains 35,887 grayscale face expression images of size 48×48, including 28,709 in the training set, 3,589 in the validation set, and 3,589 in the test set. There are 8 expression categories, with the addition of "Contempt" compared to RAF-DB. To address the problem of poor annotation quality in the FER2013 dataset, FERPlus had 10 annotators label each picture, so each picture has 10 annotation messages. This application uses the method of majority voting to determine the true label of each picture;

[0095] C: The AffectNet dataset is the largest wild face expression dataset today, containing 440,000 face pictures. This dataset contains two benchmarks: AffectNet-7 and AffectNet-8. AffectNet-7 contains 7 expression categories, the same as RAF-DB, with 283,901 in the training set and 3,500 in the test set. AffectNet-8 contains 8 expression categories, with the addition of the category "Contempt", with 287,651 in the training set and 4,000 in the test set. Due to the serious problem of sample imbalance in this dataset, this application adopts the method of data downsampling to alleviate this problem;

[0096] D: Occluded face expression dataset. To verify the performance of the facial expression recognition model under real-world occlusion conditions, FED-RO (Facial Expression Dataset with Real Occlusion) collected and annotated 400 face pictures of various occlusion situations in real scenarios for model testing. This application jointly trains the model with the training sets of RAF-DB and AffectNet;

[0097] Occlusion-RAF-DB, Occlusion-FERPlus, and Occlusion-AffectNet constructed sub-datasets for occlusion situations using the three datasets RAF-DB, FERPlus, and AffectNet. They contain 735, 605, and 682 test data respectively. These sub-datasets selected face images with occlusion from the test sets of the original datasets as test data. The occlusion types of the data include top-bottom occlusion, left-right occlusion, glasses / mask occlusion.

[0098] Experimental details: All the data in this application directly used the officially face-aligned samples, and their sizes were scaled to 224×224. The backbone model of this application is ResNet18. Like most works, this application pre-trained the backbone model on the MS-Celeb-1M face recognition dataset.

[0099] The RAF-DB and FERPlus datasets were trained for a total of 40 epochs. Data augmentation used random horizontal flipping and random cropping. The Batch_size was set to 32, the initial learning rate was 0.1, and the learning rate was reduced to Adopt momentum stochastic gradient for optimization, with momentum set to 0.9 and weight_decay set to 1e-4.

[0100] The AffectNet dataset was also trained for 40 epochs. Data augmentation used random horizontal flipping and random affine transformation. To address the problem of sample imbalance, this application adopted the method of downsampling, that is, reducing the number of samples of the class with relatively more samples in the training data. The Batch_size was set to 16, the initial learning rate was 0.0001, and the learning rate decayed exponentially every epoch, with the exponent set to 0.8. The Adam optimizer was selected for optimization, and weight_decay was set to 1e-4.

[0101] For some hyperparameters in the method of this application, the input probability P of the mixed image mix was set to 0.8. α in equation (8) was set to 0.5, and β was also 0.5. This application will elaborate in detail in the subsequent ablation experiments.

[0102] The evaluation metric for all experiments was accuracy. In the tables of the experimental results, an upward arrow indicates that the larger the metric, the better, and a downward arrow indicates that the smaller the metric, the better. This application implemented the method of this application using the Pytorch framework, and all experiments were completed with a single NVIDIA GTX 3060 GPU.

[0103] The comparison methods of this application on the occluded facial expression dataset include gACNN, RAN, VTFF, EfficientFace, and MAPNet introduced in the related work section (2.1). On the three wild facial expression datasets of RAF-DB, FERPlus, and AffectNet, in addition to the above methods, this application was also compared with SCN, DACL, RUL, DMUE, and FDRL. SCN is a self-repairing network that can handle noisy labels. This method uses an attention mechanism to assign smaller weights to noisy labels to eliminate the influence of noisy labels. DACL assigns attention weights to each feature dimension of the sample, enabling the model to pay more attention to important feature dimensions, and then uses the Center Loss that can increase the distance between features to increase the class spacing, thereby improving the performance of the facial expression recognition model. RUL performs a Mixup operation on paired image features at the image feature level to learn the uncertainty of different images to determine whether the image label is a noisy label. DMUE takes into account the distribution of sample labels during the training of the model, assists the facial expression recognition task by learning the label distribution, and eliminates the influence of noisy labels on model training. FDRL decomposes and reconstructs facial expression features to find similar and different facial expression information to better obtain fine-grained facial expression features, thereby distinguishing each facial expression.

[0104] S6: Data comparison:

[0105] a1: Comparison on the occluded dataset:

[0106] Mainly to improve the robustness of the facial expression recognition model to facial occlusion, the effectiveness of this method was verified on several occluded datasets, namely FED-RO, Occlusion-RAF-DB, Occlusion-FERPlus, and Occlusion-AffectNet. This application was compared with the excellent methods for solving occluded facial expression recognition in recent years, including gACNN, RAN, VTFF, EfficientFace, and MAPNet;

[0107] a2: Comparison with SOTA methods:

[0108] Table 1 Comparison results on FED-RO

[0109]

[0110] Table 1 shows the comparison results in the FER-RO real-world occluded facial expression dataset. Since VTFF was not tested on this dataset, this application does not compare this method. Baseline is the result obtained by training with the standard ResNet18. The method of this application achieves the best performance on FER-RO. Compared with gACNN and RAN, the method of this application improves the accuracy by 6.47% and 4.99% respectively, improves the model training and inference efficiency, and further enhances the robustness of the model to occlusion without using any additional information. Compared with EfficientFace, the method of this application improves the accuracy by 4.72% and has better performance. Compared with MAPNet, the method of this application improves the accuracy by 1.47%, and this application does not require facial key points as additional information;

[0111] Accuracy comparison of Table 2 on Occlusion-RAF-DB, Occlusion-FERPlus, and Occlusion-AffectNet

[0112]

[0113] Table 2 shows the results on three occlusion datasets, namely Occlusion-RAF-DB, Occlusion-FERPlus, and Occlusion-AffectNet. The baseline is also the result obtained by training with the standard ResNet18. Since there are no experimental results of gACNN and MAPNet on these datasets, these two methods are not compared in this application. Generally speaking, the method of this application has more advantages. Compared with RAN, the method of this application has advantages on the three test sets of Occlusion-RAF-DB, Occlusion-FERPlus, and Occlusion-AffectNet, with the accuracy rates increased by 3.95%, 2.35%, and 4.16% respectively. It can be seen that the method of this application can further improve the performance of the model while reducing the model input and improving the training and inference efficiency. Compared with EfficientFace, the method of this application also has great advantages on Occlusion-RAF-DB and Occlusion-AffectNet (EfficientFace does not provide the results on Occlusion-FERPlus), with the accuracy rates increased by 3.43% and 2.78% respectively. Compared with VTFF, although the accuracy rate of the method of this application is slightly lower by 0.32% on the Occlusion-AffectNet dataset, the method proposed in this patent has great advantages on Occlusion-RAF-DB and Occlusion-FERPlus, with the accuracy rates increased by 2.72% and 1.44% respectively. This application analyzes that AffectNet has a larger data volume compared with the other two datasets, and VTFF uses the Vision Transformer module in the model to obtain the relationship between local regions, so it has more advantages in learning this large dataset. This application only uses ResNet18 as the backbone model, and its ability to learn this large dataset is slightly weaker. However, this makes the number of model parameters of this application much smaller (as shown in Table 4, the number of parameters of VTFF is 51.8M, and the number of parameters of FERMixNet is 11.25M), and the training and inference speeds are also faster. Moreover, as a data augmentation strategy, FERMix is more flexible and can be applied to various large models. When the model trained by this application is used for inference, no additional information is required, while VTFF needs to extract the LBP features of the face additionally, which is also an advantage of the method of this application.

[0114] In summary, the method of this application is slightly inferior to VTFF with a Vision Transformer module on Occlusion-AffectNet. However, the method of this application has higher flexibility, fewer parameters, and does not require any additional information, and also demonstrates the best performance on the Occlusion-RAF-DB, Occlusion-FERPlus, and FEDRO datasets. Therefore, overall, the method of this application has better application prospects;

[0115] Comparison with SOTA methods: To investigate the generality of the method of this application in real complex scenarios, this application compares with some current excellent works in facial expression recognition on three large-scale outdoor facial expression datasets, namely RAF-DB, FERPlus, and AffectNet. SCN, RUL, and DMUE address the problem of noisy labels in facial expression datasets, while DACL and FDRL focus on the problems of high intra-class distance and low inter-class distance of expressions;

[0116] Table 3 Comparison of accuracies on RAF-DB, FERPlus, AffectNet-7, and AffectNet-8

[0117]

[0118]

[0119] Table 4 Comparison of parameter quantities of DMUE, VTFF, and the FERMixNet of this application

[0120]

[0121] Except for AffectNet-8, the method of this application has achieved the best results on three datasets, RAF-DB, FERPlus, and AffectNet-7, with the accuracy rates being 89.80%, 89.58%, and 65.54% respectively. It can be seen that the method of this application also has good generalization on the wild face expression datasets and can effectively handle face expression recognition in real scenarios. Although on AffectNet-8, the accuracy rate of the method of this application is lower than that of DMUE by 0.92%, on the other two datasets, the accuracy rates of the method of this application are both higher than that of DMUE (DMUE does not give the results on AffectNet-7): on RAF-DB, the accuracy rate of the method of this application is 1.04% higher; on FERPlus, the accuracy rate of the method of this application is 0.94% higher. And the model of this application has far fewer parameters than DMUE, as shown in Table 7. Since DMUE provides the source code, the model parameter quantity of DMUE can be calculated to be 78.4M in this application, while the model parameter quantity of this application is 11.25M (for fair comparison, ResNet18 is uniformly set as the backbone in this application). The parameter quantity of DMUE is almost 7 times that of the method of this application. Compared with VTFF, the accuracy rate of the method of this application is slightly lower in the Occlusion-AffectNet test set, but the accuracy rates of the method of this application are higher in these wild datasets, and the method of this application has fewer parameters. Therefore, considering the model size and the evaluation indicators of several datasets comprehensively, the method of this application has achieved better performance.

[0122] To investigate the generality of this method in real complex scenarios, this application makes comparisons with some excellent works in current face expression recognition on three large wild face expression datasets, RAF-DB, FERPlus, and AffectNet. SCN, RUL, and DMUE are aimed at the problem of noisy labels in face expression datasets, while DACL and FDRL focus on the problems of high intra-class distance and low inter-class distance of expressions.

[0123] Embodiment: First, this application simulates the situation of real face occlusion by performing horizontal equal-part mixing, vertical equal-part mixing, and corner mixing on face images while retaining the most semantically rich local regions of the face. In the training of the model, this application adopts the method of jointly training the main image and the mixed image, enabling the model to enhance the attention to local information while focusing on global information. In the design of the model, this application introduces an intermediate feature representation layer to learn the most discriminative non-occluded local features of the face. A large number of experiments prove that the method of this application can improve the robustness of the model to face occlusion, and the method of this application also has the best comprehensive performance on three public wild face expression datasets, RAF-DB, FERPlus, and AffectNet.

[0124] Although embodiments of the present invention have been shown and described, those of ordinary skill in the art will appreciate that various changes, modifications, substitutions and variations can be made to these embodiments without departing from the principles and spirit of the present invention. The scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. An occlusion face expression recognition model based on a face hybrid enhancement strategy, characterized in that, Including: The FERMixNet model, the FERMixNet model includes an occluded face expression recognition module; the occluded face expression recognition module includes a local feature enhancement module, a high-level / intermediate-level learning representation module, and a prediction module; The local feature enhancement module is used to perform hybrid enhancement on the face image to enhance the expression-related semantic information in the hybrid area. The local feature enhancement module includes a horizontal equal division hybrid module, a vertical equal division hybrid module, and a corner module; the horizontal equal division hybrid module is used to perform hybrid for upper and lower face occlusion, cut the face image in half horizontally, divide the two-eye area and the mouth area for hybrid; the vertical equal division hybrid module is used to perform hybrid for left and right face occlusion, cut the face image in half vertically, divide the left and right face areas for hybrid; the corner module is used to perform hybrid for single-eye occlusion, and hybrid the corresponding left-eye and right-eye areas; The high-level / intermediate-level learning representation module is used to learn the non-occluded local features of the face features and filter the most discriminative local features, and the prediction module is used to calculate the loss function and output the prediction result; The high-level / intermediate-level learning representation module includes a deep convolutional network module. The deep convolutional network module is based on the standard ResNet18 architecture and adds an intermediate feature representation layer after conv4_x. The intermediate feature representation layer includes a 1×1 convolution and a max pooling layer for obtaining intermediate features; The recognition method of the FERMixNet model includes the following steps: S1: Input includes two parts, the main image and the FERMix hybrid image. Among them, the main image selects a sample in a normal training batch. In a batch of model training, paired images are obtained by randomly shuffling the images in this batch. The main image is mixed with the paired images, and horizontal equal division hybrid, vertical equal division hybrid, or corner hybrid is randomly selected by probability for each iteration to generate the FERMix hybrid image; S2: Feature representation. The deep convolutional network module extracts the global features of the input samples and learns the local non-occluded features through the intermediate feature representation layer; S3: Training loss and model decision. The standard cross-entropy loss is calculated for both the main image and the hybrid image in high-level classification and intermediate-level classification. When the model is trained, the parameters are updated through the total loss. When inferring, the output units of high-level classification and intermediate-level classification are added as the final decision structure.

Images