Facial expression recognition model training method and facial expression recognition method

Abstract

The invention provides a training method of a facial expression recognition model and a facial expression recognition method, and relates to the technical field of picture recognition, and the training method comprises the steps: carrying out the overturning of an obtained facial image, calculating the consistency between the features of the facial image and the features of an overturned facial image through the natural symmetry features of a human face, obtaining a first loss value, and obtaining a second loss value; the attention of the model is focused on the facial key features; dividing the face image into a clean image subset and a noise image subset, training the clean image by using a supervised learning method to obtain a second loss value, and training the noise image by using a self-supervised learning method to obtain a third loss value; and finally, fusing the three loss values, and adjusting the initial recognition model in combination with the total loss value and the preset loss value, so that the recognition model training efficiency is improved, and the robustness and accuracy of the facial expression recognition model are enhanced.

Description

Technical Field

[0001] The present invention relates to the technical field of picture recognition, and in particular to a training method for a facial expression recognition model and a facial expression recognition method. Background Art

[0002] In today's era of rapid digital development, real-world application scenarios are showing a trend of diversification, and the application areas of facial expression recognition technology are becoming more and more extensive. Whether in the field of emotional computing, by analyzing facial expressions to gain insight into human emotions and achieve more accurate psychological state assessment; or in human-computer interaction, allowing machines to give more intelligent and considerate interactive feedback based on facial expressions, all demonstrate its important value and huge potential. However, this technology faces severe challenges at the data level. Manually labeled data often contains a large number of noise labels, which has become a key factor hindering its further development.

[0003] The erased attention consistency method has brought new opportunities for improving facial expression recognition capabilities. With the help of the attention mechanism, it can focus on the features of key facial areas, thereby optimizing the model's recognition accuracy of expressions.

[0004] However, the erased attention consistency method does not fully consider the negative impact of noisy labels on model performance. When faced with data with noisy labels, it is difficult to distinguish the authenticity of the labels, and it is likely that the wrong labels will be learned as correct information, and the inaccuracy of the labels cannot be effectively handled, which will lead to the model failing to converge to the ideal state during the training process, and the training effect will decline, thus affecting the accuracy and stability of facial expression recognition. Summary of the invention

[0005] The problem to be solved by the present invention is that when the existing erased attention consistency method faces data with noisy labels, it is difficult to distinguish the authenticity of the labels, and it is very likely to learn the wrong labels as correct information, and it is unable to effectively deal with the inaccuracy of the labels.

[0006] To solve the above problems, in a first aspect, the present invention provides a method for training a facial expression recognition model, comprising:

[0007] The acquired facial image is horizontally flipped to obtain a facial flipped image;

[0008] Inputting the facial image and the face flipped image into the initial recognition model respectively, obtaining a first attention map of the facial image, a second attention map of the face flipped image, a first predicted classification probability sequence of the facial image, and a second predicted classification probability sequence of the face flipped image respectively;

[0009] Obtaining a first loss value according to the first attention map and the second attention map;

[0010] Dividing the facial image according to the first predicted classification probability sequence, the second predicted classification probability sequence, and an adaptive threshold corresponding to each expression category to obtain a clean image subset and a noisy image subset;

[0011] Using a supervised learning method to train the clean image subset to obtain a second loss value;

[0012] Using a self-supervised learning method to train the noise image subset to obtain a third loss value;

[0013] The first loss value, the second loss value and the third loss value are used to obtain a total loss value;

[0014] When the fluctuation value of the total loss value is greater than or equal to a preset threshold, adjusting the model parameters of the initial recognition model, and training the adjusted initial recognition model;

[0015] When the fluctuation value of the total loss value is less than a preset threshold, the currently adjusted initial recognition model is used as the facial expression recognition model.

[0016] Optionally, the initial recognition model includes a feature extractor and a classifier, and the feature extractor is connected to the classifier;

[0017] The step of inputting the facial image and the face flipped image into the initial recognition model respectively, and obtaining the first attention map of the facial image, the second attention map of the face flipped image, the first predicted classification probability sequence of the facial image, and the second predicted classification probability sequence of the face flipped image respectively comprises:

[0018] Inputting the facial image and the facial flipped image into the initial recognition model respectively, and outputting the first attention map and the second attention map after passing through the feature extractor;

[0019] The first attention map and the second attention map are respectively input into the classifier to obtain the corresponding first predicted classification probability sequence and the second predicted classification probability sequence, wherein the first predicted classification probability sequence includes the predicted classification probabilities corresponding to the multiple expression categories predicted by the facial image, and the second predicted classification probability sequence includes the predicted classification probabilities corresponding to the multiple expression categories predicted by the facial flipped image.

[0020] Optionally, the first loss value is:

[0021]

[0022] Where N is the number of images, L is the number of expression categories, H is the image height, W is the image width, and A is theij The first attention map representing the j-th expression of the i-th facial image, Fp(A i ' j ) represents the second attention map of the j-th expression in the i-th face flipped image.

[0023] Optionally, dividing the facial image according to the first predicted classification probability sequence, the second predicted classification probability sequence and the adaptive threshold corresponding to each expression category to obtain a clean image subset and a noisy image subset comprises:

[0024] The predicted classification probability corresponding to each expression category in the first predicted classification probability sequence is added to the predicted classification probability corresponding to the same expression category in the second predicted classification probability sequence, and the average value is obtained to obtain the predicted classification probability mean corresponding to each expression category;

[0025] Comparing the predicted classification probability means corresponding to the plurality of expression categories, determining the expression category corresponding to the largest predicted classification probability mean and the largest predicted classification probability mean as the expression category to which the facial image belongs and the predicted classification probability mean corresponding to the expression category;

[0026] According to the maximum predicted classification probability in the first predicted classification probability sequence, classify the facial image into the expression category corresponding to the maximum predicted classification probability, and obtain a predicted sample corresponding to each expression category;

[0027] According to the predicted classification probability of each facial image in the predicted samples of each expression category, a classification probability mean corresponding to each expression category is obtained, and the classification probability mean is used as an adaptive threshold of the expression category;

[0028] When the predicted classification probability mean of the expression category to which the facial image belongs is greater than or equal to the adaptive threshold of the same expression category, the facial image is determined as a clean image to obtain the clean image subset;

[0029] When the predicted classification probability mean of the expression category to which the facial image belongs is less than an adaptive threshold of the same expression category, the facial image is determined to be a noise image to obtain the noise image subset.

[0030] Optionally, the second loss value is:

[0031]

[0032] Where N represents the number of images, Represents the yth fully connected layer of the initial recognition model i The weight of the class label (expression), y irepresents the specified label (specified expression category) of the i-th image, f i represents the i-th feature output by the global average pooling layer of the initial recognition model, L represents the number of expression categories, and W j Represents the weight of the j-th type of expression in the fully connected layer of the initial recognition model.

[0033] Optionally, the third loss value is:

[0034]

[0035] in, n represents the number of face images, represents the probability of image generation after the facial image is weakly enhanced. represents the probability distribution of the image generated after the face flipped image is weakly enhanced, represents the probability of image generation after strong enhancement of facial images, It represents the probability of image generation after strong enhancement of the face flipped image.

[0036] Optionally, the total loss value is:

[0037] L total =L sup +βL feat +γL mse

[0038] Among them, L feat Represents the first loss value, L sup Represents the second loss value, L mse represents the third loss value, β represents the hyperparameter of the first loss value, and γ represents the hyperparameter of the third loss value.

[0039] In a second aspect, the present invention also provides a facial expression recognition method,

[0040] The acquired facial image is input into a facial expression recognition model trained by the training method of the facial expression recognition model as described above, to obtain the expression category of the facial image.

[0041] In a third aspect, the present invention provides an electronic device, including a memory and a processor;

[0042] The memory is used to store computer programs;

[0043] The processor is used to implement the training method of the facial expression recognition model or the facial expression recognition method as described above when executing the computer program.

[0044] In a fourth aspect, the present invention provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the training method for the facial expression recognition model as described above or the facial expression recognition method as described above is implemented.

[0045] The present invention provides a facial expression recognition model training method and a facial expression recognition method. Compared with the prior art, the present invention has the following beneficial effects:

[0046] The acquired facial image is flipped, and the consistency between the features of the facial image and the features of the facial flipped image is calculated by using the features of the natural symmetry of the face to obtain the first loss value, so as to focus the model's attention on the key facial features, thereby reducing the interference of irrelevant features and improving recognition accuracy; in order to reduce the interference of noise labels, the facial images are divided into a clean image subset and a noisy image subset. The clean images are reliable and easy to process. The supervised learning method is used for training to obtain the second loss value, which makes full use of the correct labels and improves the convergence of the model training; the noisy images are trained using a self-supervised learning method to obtain the third loss value. Self-supervised learning does not rely on the given label information, but focuses on the consistency of the facial image itself, reducing the negative interference of the noisy image on the training; finally, the three loss values ​​are fused, combined with the total loss value and the preset loss value, and the initial recognition model is adjusted to improve the efficiency of the recognition model training and enhance the robustness and accuracy of the facial expression recognition model. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0048] Figure 1 A flowchart of a method for training a facial expression recognition model provided by an embodiment of the present invention;

[0049] Figure 2 A schematic diagram of the structure of a facial expression recognition model training system provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0050] In order to make the purpose, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application are clearly and completely described. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0051] The embodiments of the present application provide a training method for a facial expression recognition model and a facial expression recognition method, thereby solving the problem that the erased attention consistency method cannot effectively handle the inaccuracy of labels when facing data with noisy labels, and enables the model to converge to an ideal state during the training process, thereby improving the accuracy and stability of facial expression recognition.

[0052] In order to better understand the above technical solution, the above technical solution will be described in detail below in conjunction with the accompanying drawings and specific implementation methods.

[0053] like Figure 1 As shown, a training method for a facial expression recognition model provided in an embodiment of the present application includes:

[0054] S1: Flip the acquired facial image horizontally to obtain a facial flipped image.

[0055] S2: Input the facial image and the face flipped image into the initial recognition model respectively, and obtain the first attention map of the facial image, the second attention map of the face flipped image, the first predicted classification probability sequence of the facial image, and the second predicted classification probability sequence of the face flipped image respectively.

[0056] S3: Obtain a first loss value according to the first attention map and the second attention map.

[0057] S4: Divide the facial image according to the first predicted classification probability sequence, the second predicted classification probability sequence and the adaptive threshold corresponding to each expression category to obtain a clean image subset and a noisy image subset.

[0058] S5: Using a supervised learning method to train the clean image subset to obtain a second loss value.

[0059] S6: Use a self-supervised learning method to train the noise image subset to obtain a third loss value.

[0060] S7: Add the first loss value, the second loss value and the third loss value to obtain a total loss value.

[0061] S8: When the fluctuation value of the total loss value is greater than or equal to a preset threshold, the model parameters of the initial recognition model are adjusted, and the adjusted initial recognition model is trained.

[0062] S9: When the fluctuation value of the total loss value is less than a preset threshold, the currently adjusted initial recognition model is used as the facial expression recognition model. For example, when the fluctuation value of the total loss value does not exceed 0.1, the training is stopped and the currently adjusted initial recognition model is used as the facial expression recognition model.

[0063] In this embodiment, the acquired facial image is flipped, and the initial facial image and the facial flipped image are respectively input into the initial recognition model to obtain a first attention map and a first predicted classification probability sequence corresponding to the facial image, as well as a second attention map and a second predicted classification probability sequence corresponding to the facial flipped image. The consistency between the features of the facial image and the features of the facial flipped image is calculated by utilizing the features of the natural symmetry of the human face. According to the first attention map and the second attention map, a first loss value is obtained, and the model attention is focused on key facial features, thereby reducing the interference of irrelevant features and improving recognition accuracy. In order to reduce the interference of noise labels, the facial image is first divided into a clean image subset and a noisy image subset according to the first predicted classification probability sequence, the second predicted classification probability sequence and an adaptive threshold, and then respectively adopted Supervised learning methods and self-supervised learning methods are used to learn and train image features to obtain the second loss value and the third loss value. Different learning and training methods are used for clean images and noisy images respectively. Clean images are reliable and easy to process. Supervised learning methods are used for training to obtain the second loss value, which makes full use of correct labels and improves the convergence of model training. Self-supervised learning methods are used for training noisy images to obtain the third loss value. Self-supervised learning does not rely on given label information, but learns by mining the characteristics of the facial image itself, focusing on the consistency of the facial image itself, and reducing the negative interference of the noisy image on training. Finally, the three loss values ​​are fused, combined with the total loss value and the preset loss value, and the initial recognition model is adjusted to improve the efficiency of recognition model training and enhance the robustness and accuracy of the facial expression recognition model.

[0064] Each step is described in detail below.

[0065] S1: Flip the acquired facial image horizontally to obtain a facial flipped image.

[0066] Specifically, we use the large-scale dataset RAF-DB (Real-world Affective Faces Database) collected in real scenes, which contains facial images with a resolution of 100×100 pixels.

[0067] In an optional embodiment of the present application, the initial recognition model includes a feature extractor and a classifier, and the feature extractor is connected to the classifier. The initial recognition model uses ResNet18 as the backbone network, and the size of the input facial image should first be adjusted to 224×224 pixels. In order to improve the generalization ability of the model, data enhancement can also be performed on the input facial image, such as random erasing, random flipping, etc. The initial recognition model uses a ResNet18 network, in which the feature extractor and classifier represent the feature extraction layer and classification layer of the backbone network, respectively.

[0068] S2: Inputting the facial image and the face flipped image into the initial recognition model respectively, obtaining the first attention map of the facial image, the second attention map of the face flipped image, the first predicted classification probability sequence of the facial image, and the second predicted classification probability sequence of the face flipped image respectively, including:

[0069] The facial image and the facial flipped image are respectively input into the initial recognition model, and the first attention map and the second attention map are output after passing through the feature extractor.

[0070] The first attention map and the second attention map are respectively input into the classifier to obtain the corresponding first predicted classification probability sequence and the second predicted classification probability sequence, wherein the first predicted classification probability sequence includes the predicted classification probabilities corresponding to the multiple expression categories predicted by the facial image, and the second predicted classification probability sequence includes the predicted classification probabilities corresponding to the multiple expression categories predicted by the facial flipped image.

[0071] Specifically, the facial image is randomly erased to obtain the resulting image, denoted as I. The image I is horizontally flipped to obtain its flipped image, denoted as I′, and then image I and image I′ are input into the initial recognition model. Next, the class activation map (CAM) is used to generate an attention map from the activation value of the last convolutional layer of the feature extractor to highlight the category-related areas in the image. These attention maps are denoted as F, F′∈N×C×H×W, corresponding to the original image and the flipped image. N, C, H, W represent the number of images, the number of channels, the height, and the width, respectively. We∈L×C represents the weight of the fully connected (FC) layer, and L represents the number of expression categories. A ij Represents the attention map of the jth class of the i-th image. The calculation formula of the attention map is as follows:

[0072]

[0073] Among them, A j(h,w) is the weighted sum of the j-th expression at position (h,w). The attention map of the input image is calculated using the CAM method to visualize the features that the model focuses on when making classification decisions.

[0074] Use consistency loss to minimize the attention map A ij And the attention map Fp(A i ' j ) between the two.

[0075] S3: Obtain a first loss value according to the first attention map and the second attention map. Specifically, the first loss value is:

[0076]

[0077] Where N is the number of images, L is the number of expression categories, H is the image height, W is the image width, and A is the ij The first attention map representing the j-th expression of the i-th facial image, Fp(A′ ij ) represents the second attention map of the j-th expression in the i-th face flipped image.

[0078] Optionally, S4: dividing the facial image according to the first predicted classification probability sequence, the second predicted classification probability sequence and the adaptive threshold corresponding to each expression category to obtain a clean image subset and a noisy image subset includes:

[0079] The predicted classification probability corresponding to each expression category in the first predicted classification probability sequence and the predicted classification probability corresponding to the same expression category in the second predicted classification probability sequence are added together and the average value is obtained to obtain the predicted classification probability mean corresponding to each expression category.

[0080] Compare the predicted classification probability means corresponding to the multiple expression categories, and determine the expression category corresponding to the largest predicted classification probability mean and the largest predicted classification probability mean as the expression category to which the facial image belongs and the predicted classification probability mean corresponding to the expression category.

[0081] Specifically, for example, the facial expression categories include happy, angry and frightened. The first predicted classification probability sequence corresponding to the facial image is {0.8, 0.1, 0.1}, and the second predicted classification probability sequence corresponding to the face flipped image is {0, 0.6, 0.4}. The corresponding addition results in the mean predicted classification probabilities for each facial expression category: 0.4, 0.35 and 0.25. By comparing the sizes of the three, it is determined that the facial image belongs to the happy expression category, and the predicted classification probability mean is 0.4.

[0082] According to the maximum predicted classification probability in the first predicted classification probability sequence, the facial image is classified into the expression category corresponding to the maximum predicted classification probability, and a predicted sample corresponding to each of the expression categories is obtained.

[0083] Specifically, if the first predicted classification probability sequence of the facial image is {0.8, 0.1, 0.1}, the facial image presents a happy expression, and the predicted sample of the happy expression includes the facial expression.

[0084] According to the predicted classification probability of each facial image in the predicted samples of each expression category, a classification probability mean corresponding to each expression category is obtained, and the classification probability mean is used as an adaptive threshold of the expression category.

[0085] Specifically, the adaptive threshold T c for:

[0086]

[0087] in, represents the adaptive threshold corresponding to the k-th expression, N sk represents the number of samples in the predicted sample corresponding to the k-th expression, P i It represents the maximum value of the predicted classification probability corresponding to each image sample in the predicted sample. For different facial image datasets, the adaptive thresholds obtained are different, which are more matched with the corresponding facial image datasets. The image classification is performed based on this threshold, and the classification results are more reasonable.

[0088] When the predicted classification probability mean of the expression category to which the facial image belongs is greater than or equal to the adaptive threshold of the same expression category, the facial image is determined as a clean image to obtain the clean image subset.

[0089] When the predicted classification probability mean of the expression category to which the facial image belongs is less than an adaptive threshold of the same expression category, the facial image is determined to be a noise image to obtain the noise image subset.

[0090] Specifically, the adaptive threshold T c Combined with the predicted classification probability mean p of the facial image, the facial image set is divided into a clean image subset and a noisy image subset. The clean image subset has the highest predicted classification probability mean, and the facial images whose predicted classification probability mean exceeds the threshold are designated as belonging to the clean image subset S. clean .

[0091]

[0092] In contrast, the noisy image subset S noisyMay show feature distributions that overlap with multiple related categories. These facial image samples may contain uncertainties or incorrect annotations, which in turn have a negative impact on the learning process. Therefore, self-supervised learning methods are used to mitigate the negative impact of noisy labels and improve the robustness of the model.

[0093]

[0094] in, Indicates the yth i Adaptive threshold corresponding to the expression class.

[0095] S5: Using a supervised learning method to train the clean image subset to obtain a second loss value.

[0096] Specifically, the clean image subset is characterized by being simple and easy to learn, and is trained using the cross entropy loss. The attention map F is processed through a global average pooling (GAP) layer to obtain a compressed feature representation, denoted as f. This feature is then reshaped into a matrix of size N×C, where N is the batch size and C is the number of categories. Next, the reshaped feature vector is passed through a fully connected (FC) layer to calculate the cross entropy classification loss.

[0097] The second loss value is:

[0098]

[0099] Where N is the number of images, W yi Represents the yth fully connected layer of the initial recognition model i The weight of the class label (expression), y i represents the specified label (specified expression category) of the i-th image, f i represents the i-th feature output by the global average pooling layer of the initial recognition model, L represents the number of expression categories, and W j Represents the weight of the j-th type of expression in the fully connected layer of the initial recognition model.

[0100] S6: Use a self-supervised learning method to train the noise image subset to obtain a third loss value.

[0101] Specifically, for the subset of noisy images, due to the complexity and ambiguity of the image itself, directly using labels may cause significant losses in the learning process. Weak enhancement and strong enhancement are performed on the same image at the same time, where weak enhancement refers to slightly changing the features of the original image. In this application, the image can be scaled; strong enhancement refers to significantly changing the structure or style of the original image. In this application, Gaussian noise can be added to the image, and a self-supervised learning strategy is used to process the noisy image, calculate the consistency of the results, and improve the model effect. The third loss value L mse for:

[0102]

[0103] in, n represents, represents the probability of image generation after the facial image is weakly enhanced. represents the probability distribution of the image generated after the face flipped image is weakly enhanced, represents the probability of image generation after strong enhancement of facial images, It indicates the probability of generating an image after strong enhancement of a face flipped image. Adaptive threshold is used to effectively identify noise samples. Noise samples are trained and learned separately using targeted methods to avoid the impact of mixed training of clean samples and noise samples on the training effect, reduce energy and resource consumption during training, and improve the resource utilization efficiency of the entire model training.

[0104] S7: The first loss value, the second loss value and the third loss value are combined to obtain a total loss value. Specifically, the total loss value is:

[0105] L total =L sup +βL feat +γL mse

[0106] Among them, β represents the hyperparameter of the first loss value, and γ represents the hyperparameter of the third loss value. Since the second loss value is the loss value obtained by training the clean image subset, the training process of the clean image is more accurate, so there is no need to increase the weight before the second loss value, but it is necessary to increase the weight before the first loss value and the third loss value to reduce the impact of noise.

[0107] S8: When the fluctuation value of the total loss value is greater than or equal to a preset threshold, the model parameters of the initial recognition model are adjusted, and the adjusted initial recognition model is cyclically trained so that the model gradually converges to an ideal state.

[0108] S9: When the fluctuation value of the total loss value is less than a preset threshold, the currently adjusted initial recognition model is used as the facial expression recognition model, and the obtained facial expression recognition model has high recognition accuracy and strong stability.

[0109] Specifically, the initial recognition model is continuously trained, and the total loss value gradually decreases until the total loss value finally stabilizes and remains unchanged, for example, the preset threshold is set to 0.1. The above-mentioned fluctuation value refers to the absolute change value of the current total loss value relative to the total loss value obtained last time. The fluctuation value is a positive value, which can be an upward floating value or a downward floating value. When the fluctuation value of the total loss value is greater than or equal to 0.1, it can be regarded as that the fluctuation of the total loss value is still large, and training can continue. When the fluctuation value of the total loss value is less than 0.1, the fluctuation of the total loss value is small at this time, and the total loss value can be regarded as remaining in a basically stable state. At this time, training is stopped, and the currently adjusted initial recognition model is used as the facial expression recognition model.

[0110] In summary, compared with the prior art, the present invention has the following beneficial effects:

[0111] 1. Flip the acquired facial image, use the natural symmetry of the face to calculate the consistency between the features of the facial image and the features of the facial flipped image, and obtain the first loss value, so as to focus the model on the key facial features, thereby reducing the interference of irrelevant features and improving the recognition accuracy; in order to reduce the interference of noise labels, the facial image is divided into a clean image subset and a noisy image subset in advance. The clean image is reliable and easy to process. The supervised learning method is used for training to obtain the second loss value, which makes full use of the correct label and improves the convergence of the model training; the noisy image is trained by the self-supervised learning method to obtain the third loss value. Self-supervised learning does not rely on the given label information, focuses on the consistency of the facial image itself, and reduces the negative interference of the noisy image on the training; finally, the three loss values ​​are fused, combined with the total loss value and the preset loss value, and the initial recognition model is adjusted so that the model gradually converges to the ideal state, improves the efficiency of the recognition model training, and enhances the robustness and accuracy of the facial expression recognition model.

[0112] 2. Adaptive thresholds are used to effectively identify noise samples. Noise samples are trained and learned separately using targeted methods to avoid the impact of mixed training of clean samples and noise samples on the training effect, reduce energy and resource consumption during training, and improve the resource utilization efficiency of the entire model training.

[0113] 3. The three loss values ​​are fused, where the second loss value obtained by training with the clean image subset is not weighted. Based on the second loss value, the corresponding weights are set according to the importance of the first and third loss values, and the total loss value is fused. At this time, the total loss value obtained mainly comes from the second loss value, that is, obtained by training with the clean image subset. At the same time, the consistency loss (that is, the first loss value) calculated during the symmetry enhancement training and the third loss value obtained by training with the noisy image samples are taken into account, so that the total loss takes into account both diversity and accuracy, reducing the negative impact of noise labels.

[0114] like Figure 2 As shown, a facial expression recognition model training system provided in an embodiment of the present application includes:

[0115] The image processing module is used to horizontally flip the acquired facial image to obtain a facial flip image.

[0116] A model prediction module is used to input a facial image and a face flipped image into an initial recognition model respectively, and obtain a first attention map of the facial image, a second attention map of the face flipped image, a first predicted classification probability sequence of the facial image, and a second predicted classification probability sequence of the face flipped image respectively.

[0117] A symmetric enhanced feature decoupling module is used to obtain a first loss value according to the first attention map and the second attention map.

[0118] The image segmentation module is used to segment the facial image according to the first predicted classification probability sequence, the second predicted classification probability sequence and the adaptive threshold corresponding to each expression category to obtain a clean image subset and a noisy image subset.

[0119] The second loss value analysis module is used to train the clean image subset using a supervised learning method to obtain a second loss value.

[0120] The third loss value analysis module is used to train the noise image subset using a self-supervised learning method to obtain a third loss value.

[0121] The total loss value analysis module is used to obtain a total loss value by combining the first loss value, the second loss value and the third loss value. It is also used to adjust the model parameters of the initial recognition model when the fluctuation value of the total loss value is greater than or equal to a preset threshold, and train the adjusted initial recognition model, and when the fluctuation value of the total loss value is less than the preset threshold, use the current adjusted initial recognition model as the facial expression recognition model.

[0122] An adaptive threshold module is used to classify the facial image into the expression category corresponding to the maximum predicted classification probability according to the maximum predicted classification probability in the first predicted classification probability sequence, and obtain the predicted samples corresponding to each of the expression categories; it is also used to obtain the mean classification probability corresponding to each of the expression categories according to the predicted classification probability of each facial image in the predicted samples of each of the expression categories, and use the mean classification probability as the adaptive threshold of the expression category.

[0123] In this embodiment, the beneficial effects of the facial expression recognition model training system are similar to the beneficial effects of the facial expression recognition model training method described above, and will not be repeated here.

[0124] The present application also provides a facial expression recognition method, comprising:

[0125] The acquired facial image is input into a facial expression recognition model trained by the training method of the facial expression recognition model as described above, to obtain the expression category of the facial image.

[0126] The present application also provides a facial expression recognition system, comprising:

[0127] The facial image recognition module is used to input the acquired facial image into the facial expression recognition model trained by the training method of the facial expression recognition model as described above, so as to obtain the expression category of the facial image.

[0128] In this embodiment, the beneficial effects of the facial expression recognition method and the facial expression recognition system are similar to the beneficial effects of the training method of the above-mentioned facial expression recognition model, and will not be repeated here.

[0129] An electronic device provided in an embodiment of the present application includes a memory and a processor;

[0130] The memory is used to store computer programs;

[0131] The processor is used to implement the training method of the facial expression recognition model or the facial expression recognition method as described above when executing the computer program.

[0132] An embodiment of the present application provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the training method of the facial expression recognition model as described above or the facial expression recognition method as described above is implemented.

[0133] In this embodiment, the beneficial effects of the electronic device and the computer-readable storage medium are similar to the beneficial effects of the above-mentioned facial expression recognition model training method, and will not be repeated here.

[0134] It should be noted that the beneficial effects of the electronic device and computer-readable storage medium provided in the present application are similar to the beneficial effects of the training method of the above-mentioned facial expression recognition model, and will not be repeated here.

[0135] Now will describe the electronic device that can be used as the server or client of the present application, it is the example of the hardware device that can be applied to various aspects of the present application.Electronic device is intended to represent the computer device of various forms of digital electronics, such as, laptop computer, desktop computer, workbench, personal digital assistant, server, blade server, mainframe computer and other suitable computer.Electronic device can also represent various forms of mobile device, such as, personal digital processing, cellular phone, smart phone, wearable device and other similar computing device.The parts shown in this article, their connection and relationship, and their functions are only as examples, and are not intended to limit the realization of the present application described and / or required herein.

[0136] The electronic device includes a computing unit, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) or a computer program loaded from a storage unit into a random access memory (RAM). In the RAM, various programs and data required for the operation of the device can also be stored. The computing unit, ROM and RAM are connected to each other through a bus. An input / output (I / O) interface is also connected to the bus.

[0137] A person of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment method can be completed by instructing the relevant hardware through a computer program, and the program can be stored in a computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, the storage medium can be a disk, an optical disk, a read-only memory (ROM) or a random access memory (RAM), etc. In the present application, the unit described as a separate component may or may not be physically separated, and the component displayed as a unit may or may not be a physical unit, that is, it may be located in one place, or it may be distributed on multiple network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the embodiment scheme of the present application. In addition, each functional unit in each embodiment of the present application can be integrated in a processing unit, or each unit can exist physically separately, or two or more units can be integrated in one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units.

[0138] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0139] The above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A training method for a facial expression recognition model, characterized in that: include: The acquired facial image is horizontally flipped to obtain a facial flipped image; Inputting the facial image and the face flipped image into the initial recognition model respectively, obtaining a first attention map of the facial image, a second attention map of the face flipped image, a first predicted classification probability sequence of the facial image, and a second predicted classification probability sequence of the face flipped image respectively; Obtaining a first loss value according to the first attention map and the second attention map; Dividing the facial image according to the first predicted classification probability sequence, the second predicted classification probability sequence, and an adaptive threshold corresponding to each expression category to obtain a clean image subset and a noisy image subset; Using a supervised learning method to train the clean image subset to obtain a second loss value; Using a self-supervised learning method to train the noise image subset to obtain a third loss value; The first loss value, the second loss value and the third loss value are used to obtain a total loss value; When the fluctuation value of the total loss value is greater than or equal to a preset threshold, adjusting the model parameters of the initial recognition model, and training the adjusted initial recognition model; When the fluctuation value of the total loss value is less than a preset threshold, the currently adjusted initial recognition model is used as the facial expression recognition model.

2. The training method of the facial expression recognition model as claimed in claim 1, characterized in that, The initial recognition model includes a feature extractor and a classifier, wherein the feature extractor is connected to the classifier; The step of inputting the facial image and the face flipped image into the initial recognition model respectively, and obtaining the first attention map of the facial image, the second attention map of the face flipped image, the first predicted classification probability sequence of the facial image, and the second predicted classification probability sequence of the face flipped image respectively comprises: Inputting the facial image and the facial flipped image into the initial recognition model respectively, and outputting the first attention map and the second attention map after passing through the feature extractor; The first attention map and the second attention map are respectively input into the classifier to obtain the corresponding first predicted classification probability sequence and the second predicted classification probability sequence, wherein the first predicted classification probability sequence includes the predicted classification probabilities corresponding to the multiple expression categories predicted by the facial image, and the second predicted classification probability sequence includes the predicted classification probabilities corresponding to the multiple expression categories predicted by the facial flipped image.

3. The training method of the facial expression recognition model as claimed in claim 2, characterized in that, The first loss value is: Where N is the number of images, L is the number of expression categories, H is the image height, W is the image width, and A is the ij The first attention map representing the j-th expression of the i-th facial image, Fp(A i ′ j ) represents the second attention map of the j-th expression in the i-th face flipped image.

4. The training method of the facial expression recognition model as claimed in claim 1, wherein: The step of dividing the facial image according to the first predicted classification probability sequence, the second predicted classification probability sequence and the adaptive threshold corresponding to each expression category to obtain a clean image subset and a noisy image subset comprises: The predicted classification probability corresponding to each expression category in the first predicted classification probability sequence is added to the predicted classification probability corresponding to the same expression category in the second predicted classification probability sequence, and the average value is obtained to obtain the predicted classification probability mean corresponding to each expression category; Comparing the predicted classification probability means corresponding to the plurality of expression categories, determining the expression category corresponding to the largest predicted classification probability mean and the largest predicted classification probability mean as the expression category to which the facial image belongs and the predicted classification probability mean corresponding to the expression category; According to the maximum predicted classification probability in the first predicted classification probability sequence, classify the facial image into the expression category corresponding to the maximum predicted classification probability, and obtain a predicted sample corresponding to each expression category; According to the predicted classification probability of each facial image in the predicted samples of each expression category, a classification probability mean corresponding to each expression category is obtained, and the classification probability mean is used as an adaptive threshold of the expression category; When the predicted classification probability mean of the expression category to which the facial image belongs is greater than or equal to the adaptive threshold of the same expression category, the facial image is determined as a clean image to obtain the clean image subset; When the predicted classification probability mean of the expression category to which the facial image belongs is less than an adaptive threshold of the same expression category, the facial image is determined to be a noise image to obtain the noise image subset.

5. The training method of the facial expression recognition model as claimed in claim 1, characterized in that, The second loss value is: Where N represents the number of images, Represents the yth fully connected layer of the initial recognition model i The weight of the class label (expression), y i represents the specified label (specified expression category) of the i-th image, f i represents the i-th feature output by the global average pooling layer of the initial recognition model, L represents the number of expression categories, and W j Represents the weight of the j-th type of expression in the fully connected layer of the initial recognition model.

6. The training method of the facial expression recognition model as claimed in claim 1, characterized in that, The third loss value is: in, n represents the number of face images, represents the probability of image generation after the facial image is weakly enhanced. represents the probability distribution of the image generated after the face flipped image is weakly enhanced, represents the probability of image generation after strong enhancement of facial images, It represents the probability of image generation after strong enhancement of the face flipped image.

7. A training method for a facial expression recognition model as described in any one of claims 1-6, characterized in that: The total loss value is: L total =L sup +βL feat +γL mse Among them, L feat Represents the first loss value, L sup Represents the second loss value, L mse represents the third loss value, β represents the hyperparameter of the first loss value, and γ represents the hyperparameter of the third loss value.

8. A facial expression recognition method, characterized in that: include: The acquired facial image is input into a facial expression recognition model trained by the training method for a facial expression recognition model as described in any one of claims 1 to 7 to obtain the expression category of the facial image.

9. An electronic device, characterized in that: including memory and processor; The memory is used to store computer programs; The processor is used to implement the training method of the facial expression recognition model as described in any one of claims 1 to 7 or the facial expression recognition method as described in claim 8 when executing the computer program.

10. A computer-readable storage medium, characterized in that: The storage medium stores a computer program, and when the computer program is executed by the processor, the training method of the facial expression recognition model as described in any one of claims 1 to 7 or the facial expression recognition method as described in claim 8 is implemented.

Images