A method for facial expression recognition based on prior knowledge

Abstract

The present invention discloses a method for facial expression recognition based on prior knowledge. The specific steps of the method are: (1) preprocessing an image data set; (2) converting the prior distribution of the preprocessed image into the average distribution of the category, and then using the corresponding unique hot label as an index to extract the prior knowledge from it, and then converting the prior knowledge into a posterior probability specific to each data set, and weighted summing the label distribution and the posterior probability to obtain a mixed distribution; (3) inputting the preprocessed facial expression image into a pretrained student model network for feature extraction; (4) outputting the predicted distribution of the extracted features highly correlated with the expression through a fully connected layer, and finally obtaining the classification result through a classifier after smoothing. The present invention utilizes the prior knowledge obtained in advance, and can solve the common expression ambiguity problem in facial expression recognition to a certain extent, and obtain a more realistic prediction.

Description

Technical Field

[0001] The present invention relates to the technical field of facial expression recognition, and in particular to a facial expression recognition method based on prior knowledge. Background Art

[0002] According to surveys, language conveys one-third of the information in human communication, while non-language conveys two-thirds of human communication. Among several non-language components, facial expressions are the most important form of communication. Therefore, in the past few decades, the study of facial expressions has received extensive attention not only in perceptual science and cognitive science, but also in affective computing and computer animation. Therefore, interpreting the visual information contained in the face and identifying facial expressions have become hot research topics. Facial expression recognition is the process of identifying expressions from input images or videos, and the main steps include face detection, feature extraction (extracting global or local features of the face) and expression classification.

[0003] In the late 1960s, Ekman and Friesen compared and analyzed the differences in facial expressions among people and summarized human expressions into the following basic types: sadness, surprise, happiness, disgust, anger, and fear. Later, Matsumoto added contempt to the basic expressions, forming seven basic expressions. Facial expression recognition is a very challenging and innovative research topic. It is formed by combining relevant knowledge in the fields of psychology, computer vision, and deep learning. According to the nature of facial expression recognition, there are currently several important application scenarios: (1) Intelligent robots: giving human-computer interaction capabilities to better perceive people's emotions; (2) Assisted medical care: using expressions to help doctors better perceive the patient's psychological state during the diagnosis process; (3) Classroom teaching: using students' expressions to promptly feedback students' emotions to teachers to improve teaching quality.

[0004] Considering the various unpredictable factors in the facial expression recognition framework and various application scenarios, facial expression recognition still faces the following challenges: (1) Ambiguity of expression: Facial expressions are complex and are often composed of basic expressions of varying degrees. However, the labels of the major public facial expression recognition datasets are often one-hot, and the annotation of expressions is affected by many subjective factors such as the gender and psychology of the annotator. Therefore, the one-hot labels of expressions are ambiguous; (2) Occlusion and posture problems: From the perspective of face detection, expression recognition is easily affected by facial occlusion, posture and lighting in real application scenarios. Accurately detecting and locating the face position from the input image or video and extracting effective and important features is a task that needs to be explored and solved. In summary, facial expression recognition is still an open problem that requires continuous exploration.

[0005] For the problem of facial expression ambiguity, the focus of research is on how to define the degree of ambiguity of samples through weights, and how to define the degree of ambiguity is often complex and multi-stage. Zeng J (Zeng J, Shan S, Chen X. Facial expression recognition with inconsistently annotated datasets [C] / / Proceedings of the European conference on computer vision (ECCV). 2018: 222-237.) solves the problem of facial expression recognition label ambiguity by extracting latent distributions from pseudo labels and input face images. Reference 1 Wang K (Wang K, Peng X, Yang J, et al. Suppressing uncertainties for large-scale facial expression recognition [C] / / Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition. 2020: 6897-6906.) designed a self-healing network in which the expression images of the low-importance expression group divided by the self-attention module will be recalibrated as the class with the maximum prediction probability. Vo TH (VoT H, Lee GS, Yang HJ, et al. Pyramid with super resolution for In-the-Wild facial expression recognition[J]. IEEE Access, 2020, 8:131988-132001.) takes the voting results in the dataset calibration process as prior knowledge and combines it with the label smoothing loss function to obtain a prediction distribution that is more in line with human intuition.On this issue, Shen J (She J, Hu Y, Shi H, et al. Dive into Ambiguity: Latent Distribution Mining and Pairwise Uncertainty Estimation for Facial Expression Recognition [C] / / Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition. 2021: 6248-6257.) has achieved the best performance so far. The proposed method, with the help of the auxiliary multi-branch framework, makes full use of the potential distribution in the label dimension to improve performance, and in the sample dimension, estimates the degree of ambiguity of the sample by learning the relationship between semantic features. At the same time, prior knowledge in expressions, such as action units, can also be used as prior knowledge to improve the performance of expression recognition.

[0006] For occlusion and posture problems, attention modules are often introduced to focus on the unoccluded parts, thereby reducing the impact of occlusion on performance. However, because this type of algorithm is based on cropped areas and cropped blocks, it takes a lot of time to train the model. Li Y (Li Y, Zeng J, Shan S, et al. Occlusion aware facial expression recognition using cnn with attention mechanism [J]. IEEE Transactions on Image Processing, 2018, 28 (5): 2439-2450.) Considering different regions of interest, a block-based attention network and a global and local-based attention network are proposed. The former only focuses on local facial blocks, while the latter fuses local representation and global representation to improve performance. Reference 2 Wang K (Wang K, Peng X, Yang J, et al. Region attention networks for pose and occlusion robust facial expression recognition [J]. IEEE Transactions on Image Processing, 2020, 29: 4057-4069.) proposed a regional self-attention network that can adaptively capture the importance of each region, and defined a regional bias loss function to encourage attention to more important face areas. Summary of the invention

[0007] In view of the problem of expression ambiguity in the field of facial expression recognition, current research has complex and multi-stage defects in how to define the degree of ambiguity. The purpose of this invention is to propose a facial expression recognition method based on prior knowledge, using a simpler and more efficient framework to reduce the impact of the expression ambiguity problem, thereby achieving better classification results.

[0008] To achieve the above object, the technical solution adopted by the present invention is as follows:

[0009] A method for recognizing facial expressions based on prior knowledge, the specific steps comprising:

[0010] Step 1, preprocessing an image data set, wherein the image data set includes corresponding real label data;

[0011] Step 2: Convert the prior distribution of the preprocessed images into the average distribution of the categories, and then use the corresponding one-hot labels as indices to extract prior knowledge from them. Then convert the prior knowledge into posterior probabilities specific to each dataset, and weightedly sum the label distribution and posterior probability to obtain a mixed distribution, which will be used in subsequent steps to guide the student model to better predict results that are more consistent with human intuition.

[0012] Step 3, inputting the preprocessed facial expression image into the pre-trained student model network for feature extraction;

[0013] Step 4: The extracted features that are highly correlated with facial expressions are output through a fully connected layer to predict distribution, and then after smoothing, they are passed through a classifier to finally obtain the facial expression classification result.

[0014] Furthermore, in step 1, the preprocessing is specifically: expanding the image data set by rotation and scaling methods, and normalizing the values.

[0015] Furthermore, in step 2, the method of extracting prior knowledge is:

[0016] Step 21, preprocessed image dataset The prior distribution of each image x in is in Represents v x is a C×1 matrix, C is the number of categories in the image dataset, and ∑v x =1; represents a set of images, Represents the unique hot label corresponding to the image set;

[0017] Step 22, the average distribution of each unique hot label y is:

[0018]

[0019] in, Refers to the image set The subset of samples with one-hot label y in , For subset The number of samples;

[0020] Step 23, using the unique hot label y as an index, convert the average distribution of each unique hot label y into prior knowledge p o (k|x)=d yk , where x represents the image, is the probability of being classified as category k when the unique hot label is y.

[0021] Furthermore, in step 2, the naive Bayes formula is used to convert the prior knowledge into a posterior probability specific to each data set:

[0022]

[0023] The A event represents the image’s unique label y, the B event represents the image being predicted as category k, P(A) is the distribution of the dataset, and P(B k |A) represents the probability that the unique hot label of a known image is y and the image is predicted to be of category k, that is, the prior knowledge p o (k|x), P(A|B k ) is the prediction result of the model during the inference process, that is, the posterior probability p e (k|x).

[0024] Furthermore, in step 2, the calculation method of the mixed distribution is:

[0025] p f (k|x)=(1-α)p e (k|x)+αq(k|x)

[0026] where α controls the weight between distributions, and p e (k|x) is the posterior probability, and q(k|x) is the true label distribution.

[0027] Furthermore, in step 3, the training step of the student model network includes: step 31, inputting the training set into the student model network to obtain the output result of the student model network; step 32, determining the cross entropy loss function based on the output result of the student model network and the real label data, and the loss function includes the mixed distribution obtained in step 2; step 33, using the cross entropy loss function to iteratively train the student model network.

[0028] Furthermore, in step 4, the predicted distribution p(k|x) is smoothed by the temperature coefficient T to obtain

[0029] Furthermore, in step 4, the loss function of the student model network is:

[0030] L = (1-β) L CE +βL f

[0031] Where L CE is the cross entropy function, β controls the weight between loss functions, L f Loss function for the teacher-student model:

[0032]

[0033] Among them, KL is the relative entropy, p f (k|x) is a mixed distribution, is the mixed distribution p f (k|x) is obtained after smoothing with temperature coefficient T, x is the image, Represents a set of images.

[0034] The present invention proposes a method for facial expression recognition based on prior knowledge, which utilizes the prior knowledge obtained in advance and guides the student model with a simpler, more efficient and pre-trained virtual teacher, thereby further improving the performance of facial expression recognition without increasing the model training cost. At the same time, it solves the common expression ambiguity problem in facial expression recognition to a certain extent and obtains a more realistic prediction. The present invention can effectively compress the student model and further improve the performance of the student model in terms of low complexity. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Figure 1 is a flow chart of the method of the present invention;

[0036] Figure 2 Performance comparison between the present invention and existing methods, (a) RAF-DB dataset, (b) FERPlus dataset, (c) AffectNet dataset.

[0037] Figure 3 It is a comparison diagram of the statistical distribution, the control group distribution and the distribution of the present invention.

[0038] Figure 4 2D feature visualization for three facial expression recognition datasets. DETAILED DESCRIPTION

[0039] In order to make the objectives, technical solutions and advantages of the present invention more clear, the embodiments of the present invention will be further described in detail below with reference to the accompanying drawings.

[0040] The ambiguity of facial expressions can be attributed to the strong subjectivity of human annotators and the inherent strong correlation between expressions. For the former, the influence of strong subjectivity can be weakened by replacing the one-hot label with a one-hot label distribution or by assigning specific weights to different samples through learning; for the latter, the correlation and differences between expressions can be explored through prior knowledge such as existing action units, so as to better extract the true emotions behind the expression images.

[0041] This embodiment provides a method for recognizing facial expressions based on prior knowledge, such as Figure 1 As shown, the specific steps are:

[0042] The original facial expression dataset is preprocessed by changing the size, random cropping, random horizontal flipping, random erasing, and numerical normalization, and the input image size is unified to 224×224×3. The preprocessed face image dataset is defined as in represents a set of images, Represents the unique hot label corresponding to the image set, the unique hot label of the input image x The distribution of Where C is the number of categories, which include eight basic expressions: surprise, fear, disgust, happiness, sadness, anger, neutrality and contempt.

[0043] In order to obtain a more realistic distribution, this example uses the FERPlus dataset Prior knowledge of the expression distribution of the face image x is calculated from the voting results of the annotators officially trained on the dataset, where Represents v x is a C×1 matrix, and ∑v x =1.

[0044] The average distribution of each expression category y is:

[0045]

[0046] in, Refers to the image set The subset of samples with the unique hot label y, For subset The number of samples.

[0047] The unique hot label y of the preprocessed facial expression dataset is used as the index to extract the prior knowledge p from the statistical data calculated by the above steps. o (k|x)=d yk , where x represents the image, is the probability of being classified as expression k when the unique hot label is y, as shown in Table 1.

[0048] Table 1 Statistics of the FERPLUS dataset

[0049] surprise Fear disgust Happy sad angry neutral Contempt surprise 0.788 0.089 0.003 0.036 0.010 0.014 0.058 0.002 Fear 0.169 0.659 0.013 0.012 0.073 0.027 0.043 0.004 disgust 0.017 0.008 0.621 0.017 0.089 0.125 0.090 0.033 Happy 0.023 0.002 0.002 0.918 0.005 0.004 0.043 0.003 sad 0.009 0.016 0.011 0.013 0.748 0.023 0.170 0.010 angry 0.060 0.023 0.035 0.025 0.032 0.741 0.075 0.009 neutral 0.017 0.004 0.007 0.033 0.114 0.026 0.778 0.021 Contempt 0.010 0.006 0.060 0.021 0.043 0.034 0.148 0.678

[0050] In order to apply the prior knowledge to different facial expression recognition datasets, the prior knowledge is converted into a posterior probability p specific to each dataset. e (k|x), using the naive Bayes formula:

[0051]

[0052] The A event represents the image’s unique label y, the B event represents the image being predicted as category k, P(A) is the distribution of the dataset, and P(B k |A) represents the probability that the unique hot label of a known image is y and the image is predicted to be of category k, that is, the prior knowledge p o (k|x), P(A|B k ) is the prediction result of the model during the inference process, that is, the posterior probability p e (k|x).

[0053] Taking into account the information in the one-hot label, the one-hot label and the posterior probability are weighted and summed to obtain a mixed distribution:

[0054] p f (k|x)=(1-α)p e (k|x)+αq(k|x)

[0055] where α controls the weight between distributions, and p e (k|x) is the posterior probability, and q(k|x) is the true label distribution.

[0056] The four backbone networks ResNet-18, ResNet50-IBN, ShuffleNetV1 and MobileNetV2 are pre-trained on the large face dataset MS-Celeb-1M, and the guided loss function is the cross entropy loss function;

[0057] The preprocessed facial expression images are input into the pretrained backbone network for feature extraction.

[0058] The extracted features highly correlated with facial expressions are output through the fully connected layer to predict the distribution p(k|x), which is smoothed by the temperature coefficient T to obtain

[0059] Define the teacher-student loss function:

[0060]

[0061] Where KL is the relative entropy, also known as Kullback-Leibler divergence, For p f (k|x) is obtained after smoothing with temperature coefficient T, x is the image, Represents a set of images.

[0062] The total loss function in the student model is:

[0063] L=(1-β)L CE +βL f

[0064] Where L CE is the cross entropy function, and β controls the weights between loss functions.

[0065] Under the guidance of the teacher model, the teacher-student loss function can reduce the overconfident prediction results that are easily caused by the cross entropy function. Therefore, the prediction distribution of the student model is closer to the classification results of the real emotion after the classifier output.

[0066] The beneficial effects of this embodiment can be verified by the following experiments:

[0067] 1. Model Accuracy

[0068] 1. Comparison of different backbone networks

[0069] According to the teacher-student framework proposed in this embodiment, the models trained with four backbone networks, ResNet-18, ResNet-50IBN, ShuffleNetV1 and MobileNetV2, are compared with the baseline model. As shown in Table 2, significant improvements are achieved in the three face image datasets RAF-DB, FERPlus and AffectNet, with an average improvement of 3.10%, 2.76% and 3.62% in accuracy, respectively.

[0070] Table 2 Comparison of the accuracy of different backbone networks

[0071]

[0072] 2. Comparison with existing methods

[0073] According to the teacher-student framework proposed in this embodiment, the trained model obtained impressive results on three facial image datasets, RAF-DB, FERPlus and AffectNet, and was compared with existing facial expression recognition methods.

[0074] Figure 2The performance comparison between the present invention and the existing methods mentioned in the background technology is given. For the fairness of the comparison, except for the method proposed by Vo TH, which uses VGG16 as the backbone network, the backbone networks of the other methods are all ResNet-18. The present invention is pre-trained on the MS-Celeb-1M face recognition dataset, and the best results are achieved under the settings of hyperparameters T = 5, α = 0.50, β = 0.55. Figure 2 As shown in Figure (a), the present invention achieves 89.90% accuracy on the RAF-DB dataset, which is more than 1% higher than that of the previous one; Figure (b) shows that the present invention achieves higher accuracy on the FERPlus dataset with a lighter ResNet-18 than VGG16; and Figure (c) shows that the present invention achieves nearly 2% accuracy improvement on the AffectNet dataset. This shows the superiority of the present invention in model accuracy.

[0075] 2. Visualization Results

[0076] 1. Predicted distribution and correlation

[0077] After training according to the above specific scheme, the obtained model can output a more realistic prediction distribution. The statistical distribution is the voting result of the annotators, and the control group is a baseline model trained only with the cross entropy loss function, using KL divergence as the similarity metric.

[0078] like Figure 3 As shown, by randomly sampling two facial expression images in RAF-DB, its statistical distribution, control group and predicted distribution of the present invention can be obtained. The KL divergence of the calculated statistical distribution and the predicted distribution output by the present invention is significantly lower than that of the statistical distribution and the control group. It can be seen that the present invention can obtain a more realistic predicted distribution and can solve the problem of overconfident prediction of the benchmark model.

[0079] 2. Visualization of 2D features

[0080] By forcing the model output distribution to be consistent with the actual distribution, intra-class differences can be reduced.

[0081] This embodiment uses the t-SNE algorithm to analyze the feature embedding of ResNet-18 trained on three datasets: RAF-DB, FERPlus, and AffectNet.

[0082] like Figure 4 As shown, it can be seen that compared with the baseline model, the present invention significantly reduces the intra-class differences. In addition, under the guidance of the teacher model, the distance between classes is further expanded, and the differences between expressions are better distinguished.

[0083] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for facial expression recognition based on prior knowledge, characterized in that: The steps of the method include: Step 1, preprocessing an image data set, wherein the image data set includes corresponding real label data; Step 2: Convert the prior distribution of the preprocessed image into the average distribution of the category, and then use the corresponding unique hot label as an index to extract the prior knowledge from it. Then convert the prior knowledge into a posterior probability specific to each data set, and weighted sum the label distribution and the posterior probability to obtain a mixed distribution, which is used in subsequent steps to guide the student model to better predict results that are more consistent with human intuition. The method of extracting prior knowledge is: Step 21: Preprocessed image dataset The prior distribution of each image x in is in Represents v x is a C×1 matrix, C is the number of categories in the image dataset, and Σv x =1; represents a set of images, Represents the unique hot label corresponding to the image set; Step 22, the average distribution of each unique hot label y is: in, Refers to the image set The subset of samples with one-hot label y in , For subset The number of samples; Step 23, using the unique hot label y as an index, convert the average distribution of each unique hot label y into prior knowledge p o (k|x)=d yk , where x is the image, is the probability of being classified as category k when the unique hot label is y; Step 3, inputting the preprocessed facial expression image into the pre-trained student model network for feature extraction; Step 4: The extracted features that are highly correlated with facial expressions are output through a fully connected layer to predict distribution, and then after smoothing, they are passed through a classifier to finally obtain the facial expression classification result.

2. A method for facial expression recognition based on prior knowledge according to claim 1, characterized in that: In the step 1, the preprocessing specifically includes: expanding the image data set by means of rotation and scaling, and normalizing the values.

3. The method for facial expression recognition based on prior knowledge according to claim 1, characterized in that: In step 2, the naive Bayes formula is used to convert the prior knowledge into a posterior probability specific to each data set: The A event represents the image’s unique label y, the B event represents the image being predicted as category k, P(A) is the distribution of the dataset, and P(B k |A) represents the probability that the unique hot label of a known image is y and the image is predicted to be of category k, that is, the prior knowledge p o (k|x), P(A|B k ) is the prediction result of the model during the inference process, that is, the posterior probability p e (k|x).

4. The method for facial expression recognition based on prior knowledge according to claim 1, characterized in that: In step 2, the calculation method of the mixed distribution is: p f (k|x)=(1-α)p e (k|x)+αq(k|x) where α controls the weight between distributions, and p e (k|x) is the posterior probability, and q(k|x) is the true label distribution.

5. The method for facial expression recognition based on prior knowledge according to claim 1, characterized in that: In step 3, the training steps of the student model network include: Step 31, input the training set into the student model network to obtain the output result of the student model network; Step 32, determining a cross entropy loss function based on the output result of the student model network and the true label data, wherein the loss function includes the mixed distribution obtained in step 2; Step 33, iteratively training the student model network using the cross entropy loss function.

6. The method for facial expression recognition based on prior knowledge according to claim 1, characterized in that: In step 4, the predicted distribution p(k|x) is smoothed by the temperature coefficient T to obtain 7. The method for facial expression recognition based on prior knowledge according to claim 1, characterized in that: In step 4, the loss function of the student model network is: L=(1-β)L CE +βL f Where L CE is the cross entropy function, β controls the weight between loss functions, L f Loss function for the teacher-student model: Among them, KL is the relative entropy, p f (k|x) is a mixed distribution, is the mixed distribution p f (k|x) is obtained after smoothing with temperature coefficient T, x is the image, Represents a set of images.

Images