Training method of facial expression recognition model and facial expression recognition method

Abstract

The application provides a face expression recognition model training method and a face expression recognition method, relates to the picture recognition technical field, and the training method flips the obtained face image, utilizes the natural symmetry feature of the face, calculates the consistency between the features of the face image and the features of the face flipped image, obtains a first loss value, and concentrates the model attention on the key features of the face; the face image is divided into a clean image subset and a noise image subset, the clean image is trained by using a supervised learning method to obtain a second loss value, and the noise image is trained by using a self-supervised learning method to obtain a third loss value; finally, the three loss values are fused, a total loss value and a preset loss value are combined, and the initial recognition model is adjusted to improve the efficiency of the recognition model training and enhance the robustness and accuracy of the face expression recognition model.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of picture recognition, in particular to a facial expression recognition model training method and a facial expression recognition method. BACKGROUND

[0002] In today's era of rapid digital development, real-world application scenarios are showing a trend of diversification, and the application field of facial expression recognition technology is becoming more and more extensive. Whether it is in the field of affective computing, through analyzing facial expressions to understand human emotions and achieve more accurate psychological state assessment, or in the field of human-computer interaction, allowing machines to give more intelligent and thoughtful interaction feedback based on facial expressions, it highlights its important value and great potential. However, this technology is facing serious challenges at the data level, and artificially annotated data often has a large amount of noisy labels, which is a key factor hindering its further development.

[0003] The erase attention consistency method has brought new opportunities for the improvement of facial expression recognition capability, which can focus on the key area features of the face through the attention mechanism, thereby optimizing the recognition accuracy of the model for expressions.

[0004] However, the erase attention consistency method does not fully consider the negative impact of noisy labels on model performance. When facing data with noisy labels, it is difficult to distinguish the authenticity of the labels, and it is likely to learn from incorrect labels as correct information, and cannot effectively handle the inaccuracy of the labels, thereby causing the model to fail to converge to an ideal state during training, and the training effect is reduced, thereby affecting the accuracy and stability of facial expression recognition. SUMMARY

[0005] The problem to be solved by the present application is that the existing erase attention consistency method, when facing data with noisy labels, is difficult to distinguish the authenticity of the labels, and is likely to learn from incorrect labels as correct information, and cannot effectively handle the inaccuracy of the labels.

[0006] To solve the above problems, in a first aspect, the present application provides a facial expression recognition model training method, comprising:

[0007] horizontally flipping the obtained facial image to obtain a facial flipped image;

[0008] inputting the facial image and the facial flipped image into an initial recognition model respectively to obtain a first attention map of the facial image, a second attention map of the facial flipped image, a first predicted classification probability sequence of the facial image, and a second predicted classification probability sequence of the facial flipped image;

[0009] obtaining a first loss value according to the first attention map and the second attention map;

[0010] According to the first predicted classification probability sequence, the second predicted classification probability sequence, and the adaptive threshold corresponding to each expression category, the face image is divided to obtain a clean image subset and a noise image subset;

[0011] The clean image subset is trained by using a supervised learning method to obtain a second loss value;

[0012] The noise image subset is trained by using a self-supervised learning method to obtain a third loss value;

[0013] The first loss value, the second loss value, and the third loss value are combined to obtain a total loss value;

[0014] When the fluctuation value of the total loss value is greater than or equal to a preset threshold, the model parameters of the initial recognition model are adjusted, and the initial recognition model after adjustment is trained;

[0015] When the fluctuation value of the total loss value is less than the preset threshold, the initial recognition model after the current adjustment is taken as the facial expression recognition model.

[0016] Optionally, the initial recognition model comprises a feature extractor and a classifier, and the feature extractor is connected to the classifier.

[0017] The face image and the face flipped image are respectively input into the initial recognition model to obtain a first attention map of the face image, a second attention map of the face flipped image, a first predicted classification probability sequence of the face image, and a second predicted classification probability sequence of the face flipped image, respectively.

[0018] The face image and the face flipped image are respectively input into the initial recognition model, and the first attention map and the second attention map are output after the feature extractor.

[0019] The first attention map and the second attention map are respectively input into the classifier to obtain the corresponding first predicted classification probability sequence and the second predicted classification probability sequence, wherein the first predicted classification probability sequence comprises predicted classification probabilities corresponding to multiple expression categories obtained by predicting the face image, and the second predicted classification probability sequence comprises predicted classification probabilities corresponding to multiple expression categories obtained by predicting the face flipped image.

[0020] Optionally, the first loss value is:

[0021]

[0022] wherein N represents the number of images, L represents the number of expression categories, H represents the height of the image, W represents the width of the image, and A represents the first attention map.ij a first attention map of a j-th expression of an i-th face image, Fp(A i j a second attention map of a j-th expression of an i-th face image.

[0023] Optionally, the dividing the face images according to the first predicted classification probability sequence, the second predicted classification probability sequence and the adaptive threshold corresponding to each expression category to obtain a clean image subset and a noise image subset comprises:

[0024] averaging the predicted classification probability corresponding to each expression category in the first predicted classification probability sequence and the predicted classification probability corresponding to the same expression category in the second predicted classification probability sequence to obtain a predicted classification probability mean value corresponding to each expression category;

[0025] comparing the predicted classification probability mean values corresponding to multiple expression categories, determining the expression category corresponding to the maximum predicted classification probability mean value and the maximum predicted classification probability mean value as the expression category to which the face image belongs and the predicted classification probability mean value corresponding to the expression category;

[0026] classifying the face image into the expression category corresponding to the maximum predicted classification probability according to the maximum predicted classification probability in the first predicted classification probability sequence to obtain a predicted sample corresponding to each expression category;

[0027] obtaining a classification probability mean value corresponding to each expression category according to the predicted classification probability of each face image in the predicted sample of each expression category, and taking the classification probability mean value as an adaptive threshold of the expression category;

[0028] when the predicted classification probability mean value of the expression category to which the face image belongs is greater than or equal to the adaptive threshold of the same expression category, determining the face image as a clean image to obtain the clean image subset;

[0029] when the predicted classification probability mean value of the expression category to which the face image belongs is less than the adaptive threshold of the same expression category, determining the face image as a noise image to obtain the noise image subset.

[0030] Optionally, the second loss value is:

[0031]

[0032] wherein N represents the number of images, a weight of a y i th class label (expression), y i ​denotes the specified label (the specified expression category) of the ith image, f i denotes the i-th feature of the global average pooling layer output of the initial recognition model, L denotes the number of expression categories, W j denotes the weight of the j-th expression in the fully connected layer of the initial recognition model.

[0033] Optionally, the third loss value is:

[0034]

[0035] wherein, n denotes the number of face images, denotes the probability of generating an image after weak enhancement processing of a face image, denotes the probability distribution of generating an image after weak enhancement processing of a face flip image, denotes the probability of generating an image after strong enhancement processing of a face image, denotes the probability of generating an image after strong enhancement processing of a face flip image.

[0036] Optionally, the total loss value is:

[0037] L total = L sup + βL feat + γL mse

[0038] wherein, L feat denotes the first loss value, L sup denotes the second loss value, L mse denotes the third loss value, β denotes the hyperparameter of the first loss value, and γ denotes the hyperparameter of the third loss value.

[0039] In a second aspect, the present application further provides a facial expression recognition method,

[0040] inputting the obtained face image into a facial expression recognition model trained by the facial expression recognition model training method as described above, to obtain an expression category of the face image.

[0041] In a third aspect, the present application provides an electronic device, comprising a memory and a processor;

[0042] The memory is configured to store a computer program.

[0043] The processor is configured to, when executing the computer program, implement the facial expression recognition model training method as described above or the facial expression recognition method as described above.

[0044] In a fourth aspect, the present application provides a computer readable storage medium, wherein the storage medium stores a computer program, and when the computer program is executed by a processor, the training method of the facial expression recognition model or the facial expression recognition method is realized.

[0045] The present application provides a training method of a facial expression recognition model and a facial expression recognition method.

[0046] The obtained facial image is flipped, the consistency between the features of the facial image and the features of the flipped facial image is calculated by using the natural symmetry feature of the face, a first loss value is obtained, the model attention is focused on the key features of the face, thereby reducing the interference of irrelevant features and improving the recognition accuracy; in order to reduce the interference of noise labels, the facial image is divided into a clean image subset and a noise image subset, the clean image is reliable and easy to process, a supervised learning method is used for training to obtain a second loss value, the correct labels are fully utilized to improve the convergence of the model training; and the noise image is trained by using a self-supervised learning method to obtain a third loss value, the self-supervised learning does not depend on the given label information, and focuses on the consistency of the facial image itself, thereby reducing the negative interference of the noise image on the training; finally, the three loss values are fused, the total loss value and the preset loss value are combined, and the initial recognition model is adjusted to improve the efficiency of the recognition model training and enhance the robustness and accuracy of the facial expression recognition model. BRIEF DESCRIPTION OF DRAWINGS

[0047] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiments or the prior art description. Obviously, the drawings in the following description are only some embodiments of the present application, and for those skilled in the art, other drawings can also be obtained from these drawings without creative labor.

[0048] Figure 1 A flowchart of a training method of a facial expression recognition model provided by the embodiment of the present application;

[0049] Figure 2 A structure diagram of a training system of a facial expression recognition model provided by the embodiment of the present application. DETAILED DESCRIPTION

[0050] In order to make the purposes, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application are described clearly and completely. Obviously, the described embodiments are some of the embodiments of the present application, but not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work fall within the scope of protection of the present application.

[0051] The embodiments of the present application provide a face expression recognition model training method and a face expression recognition method, which solve the problem that the erasing attention consistency method cannot effectively handle the inaccuracy of labels when facing data with noise labels, and realize that the model converges to an ideal state in the training process, thereby improving the accuracy and stability of face expression recognition.

[0052] In order to better understand the above technical solutions, the above technical solutions will be described in detail below in combination with the drawings of the specification and specific embodiments.

[0053] As shown in Figure 1 The face expression recognition model training method provided by the embodiments of the present application comprises:

[0054] S1: horizontally flipping the obtained face image to obtain a face flipped image.

[0055] S2: inputting the face image and the face flipped image into an initial recognition model respectively to obtain a first attention map of the face image, a second attention map of the face flipped image, a first predicted classification probability sequence of the face image and a second predicted classification probability sequence of the face flipped image respectively.

[0056] S3: obtaining a first loss value according to the first attention map and the second attention map.

[0057] S4: dividing the face image according to the first predicted classification probability sequence, the second predicted classification probability sequence and an adaptive threshold corresponding to each expression category to obtain a clean image subset and a noise image subset.

[0058] S5: training the clean image subset by using a supervised learning method to obtain a second loss value.

[0059] S6: training the noise image subset by using a self-supervised learning method to obtain a third loss value.

[0060] S7: obtaining a total loss value by using the first loss value, the second loss value and the third loss value.

[0061] S8: When the fluctuation value of the total loss value is greater than or equal to a preset threshold value, adjusting the model parameters of the initial recognition model, and training the initial recognition model after the adjustment.

[0062] S9: When the fluctuation value of the total loss value is less than the preset threshold value, taking the initial recognition model after the current adjustment as the facial expression recognition model. For example, when the up-and-down fluctuation value of the total loss value does not exceed 0.1, stopping the training, and taking the initial recognition model after the current adjustment as the facial expression recognition model.

[0063] In this embodiment, the obtained facial image is flipped, and the initial facial image and the facial flipped image are input into the initial recognition model respectively to obtain the first attention map and the first predicted classification probability sequence corresponding to the facial image, and the second attention map and the second predicted classification probability sequence corresponding to the facial flipped image. By using the feature of natural symmetry of the face, the consistency between the feature of the facial image and the feature of the facial flipped image is calculated, and the first loss value is obtained according to the first attention map and the second attention map, so that the model attention is focused on the facial key features, thereby reducing the interference of irrelevant features and improving the recognition accuracy. In order to reduce the interference of noise labels, the facial image is first divided into a clean image subset and a noise image subset according to the first predicted classification probability sequence, the second predicted classification probability sequence and the adaptive threshold, and then the image features are learned and trained by using the supervised learning method and the self-supervised learning method respectively to obtain the second loss value and the third loss value. Different learning and training methods are used for the clean image and the noise image respectively. The clean image is reliable and easy to process, and the supervised learning method is used for training to obtain the second loss value, so as to fully utilize the correct label and improve the convergence of the model training. The noise image is trained by using the self-supervised learning method to obtain the third loss value. The self-supervised learning does not depend on the given label information, but learns by mining the features of the facial image itself, focuses on the consistency of the facial image itself, and reduces the negative interference of the noise image on the training. Finally, the three loss values are fused, the total loss value and the preset loss value are combined, the initial recognition model is adjusted, the efficiency of the recognition model training is improved, and the robustness and accuracy of the facial expression recognition model are enhanced.

[0064] The following describes each step in detail.

[0065] S1: Flipping the obtained facial image horizontally to obtain a facial flipped image.

[0066] Specifically, a large-scale dataset RAF-DB (Real-world Affective Faces Database) collected in a real scene is used. The facial images contained in the dataset all have a resolution size of 100x100 pixels.

[0067] In an optional embodiment of the present application, the initial recognition model comprises a feature extractor and a classifier connected to the feature extractor. The initial recognition model uses ResNet18 as the backbone network, and the size of the input face image should be first adjusted to 224x224 pixels. In order to improve the generalization ability of the model, data enhancement such as random erasing and random flipping can also be performed on the input face image. The initial recognition model uses the ResNet18 network, wherein the feature extractor and the classifier represent the feature extraction layer and the classification layer of the backbone network, respectively.

[0068] S2: inputting the face image and the face flipped image into the initial recognition model respectively to obtain a first attention map of the face image, a second attention map of the face flipped image, a first predicted classification probability sequence of the face image, and a second predicted classification probability sequence of the face flipped image, comprising:

[0069] The face image and the face flipped image are input into the initial recognition model, and the first attention map and the second attention map are output after the feature extractor.

[0070] The first attention map and the second attention map are input into the classifier respectively to obtain the corresponding first predicted classification probability sequence and the second predicted classification probability sequence, wherein the first predicted classification probability sequence comprises predicted classification probabilities corresponding to multiple expression categories obtained by predicting the face image, and the second predicted classification probability sequence comprises predicted classification probabilities corresponding to multiple expression categories obtained by predicting the face flipped image.

[0071] Specifically, the face image is randomly erased to obtain a result image, denoted as I. The image I is horizontally flipped to obtain its flipped image, denoted as I', and then the image I and the image I' are input into the initial recognition model. Next, the class activation map (CAM) is used to generate attention maps from the activation values of the last convolution layer of the feature extractor, highlighting the regions in the image related to the category. These attention maps are denoted as F, F'∈NxCxHxW, corresponding to the original image and the flipped image. N, C, H, and W represent the number of images, the number of channels, the height, and the width, respectively. Let We∈LxC represent the weights of the fully connected (FC) layer, and L represent the number of expression categories. ij represent the attention map of the jth category of the ith image. The calculation formula of the attention map is as follows:

[0072]

[0073] wherein A j(h,w) is the weighted sum of the j-th expression at position (h, w). The attention map of the input image is calculated using the CAM method, so that the features that the model focuses on when making a classification decision can be visualized.

[0074] A consistency loss is used to minimize the attention map A ij and the difference between the attention map Fp(A i ′ j ) corresponding to the flipped image of the face image.

[0075] S3: According to the first attention map and the second attention map, a first loss value is obtained. Specifically, the first loss value is:

[0076]

[0077] where N represents the number of images, L represents the number of expression categories, H represents the height of the image, W represents the width of the image, A ij represents the first attention map of the j-th expression of the i-th face image, Fp(A′ ij ) represents the second attention map of the j-th expression of the i-th face flipped image.

[0078] Optionally, S4: According to the first predicted classification probability sequence, the second predicted classification probability sequence, and the adaptive threshold value corresponding to each expression category, the face image is divided to obtain a clean image subset and a noise image subset, including:

[0079] The predicted classification probability corresponding to each expression category in the first predicted classification probability sequence is added to the predicted classification probability corresponding to the same expression category in the second predicted classification probability sequence to obtain the predicted classification probability mean value corresponding to each expression category.

[0080] The predicted classification probability mean values corresponding to a plurality of expression categories are compared, and the expression category corresponding to the maximum predicted classification probability mean value and the maximum predicted classification probability mean value are determined as the expression category to which the face image belongs and the predicted classification probability mean value corresponding to the expression category.

[0081] Specifically, for example, the expression categories are happy, angry and surprised, the first predicted classification probability sequence corresponding to the face image is {0.8, 0.1, 0.1}, and the second predicted classification probability sequence corresponding to the face flipped image is {0, 0.6, 0.4}. After adding, the predicted classification probability mean value corresponding to each expression category is 0.4, 0.35 and 0.25 respectively. By comparing the sizes of the three, it is determined that the expression category to which the face image belongs is happy, and the predicted classification probability mean value is 0.4.

[0082] According to the maximum predicted classification probability in the first predicted classification probability sequence, the face image is classified into the expression category corresponding to the maximum predicted classification probability, to obtain a predicted sample corresponding to each expression category.

[0083] Specifically, if the first predicted classification probability sequence of the face image is {0.8, 0.1, 0.1}, the face image presents a happy expression, and the predicted sample of the happy expression includes the face expression.

[0084] According to the predicted classification probability of each face image in the predicted sample of each expression category, an average classification probability of each expression category is obtained, and the average classification probability is taken as an adaptive threshold of the expression category.

[0085] Specifically, the adaptive threshold T c is:

[0086]

[0087] wherein, T k represents the adaptive threshold corresponding to the kth expression, N sk represents the number of samples in the predicted sample corresponding to the kth expression, P i represents the maximum value of the predicted classification probability corresponding to each picture sample in the predicted sample. For different face image data sets, the obtained adaptive threshold is different, and is more matched with the corresponding face image data set. With the threshold as the standard for image classification, the classification result is more reasonable.

[0088] When the average predicted classification probability of the expression category to which the face image belongs is greater than or equal to the adaptive threshold of the same expression category, the face image is determined as a clean image, and a clean image subset is obtained.

[0089] When the average predicted classification probability of the expression category to which the face image belongs is less than the adaptive threshold of the same expression category, the face image is determined as a noise image, and a noise image subset is obtained.

[0090] Specifically, the adaptive threshold T c In combination with the average predicted classification probability p of the face image, the face image set is divided into a clean image subset and a noise image subset. The clean image subset has the highest average predicted classification probability, and the face image with the predicted classification probability exceeding the threshold is designated as belonging to the clean image subset S clean .

[0091]

[0092] In contrast, the noise image subset S noisyThe face image samples can exhibit feature distribution overlapping with multiple relevant categories. These face image samples can contain uncertain or erroneous labels, which in turn negatively affect the learning process. Therefore, a self-supervised learning method is used to mitigate the negative effects of noisy labels and improve the robustness of the model.

[0093]

[0094] wherein, represents the y i th adaptive threshold value corresponding to the expression category.

[0095] S5: training the clean image subset using a supervised learning method to obtain a second loss value.

[0096] Specifically, the clean image subset is simple and easy to learn, and is trained using a cross-entropy loss. The attention map F is processed through a global average pooling (GAP) layer to obtain a compressed feature representation, denoted as f. This feature is then reshaped into a matrix of size NxC, where N is the batch size and C is the number of categories. Next, the reshaped feature vector is passed through a fully connected (FC) layer to calculate the cross-entropy classification loss.

[0097] The second loss value is:

[0098]

[0099] wherein, N represents the number of images, W yi represents the y i th weight of the global average pooling layer of the initial recognition model, y i represents the specified label (specified expression category) of the i i th feature output by the global average pooling layer of the initial recognition model, L represents the number of expression categories, W j represents the j

[0100] S6: training the noisy image subset using a self-supervised learning method to obtain a third loss value.

[0101] Specifically, for the subset of noisy images, due to the complexity and ambiguity of the images themselves, directly using the labels may cause significant loss in the learning process. Weak enhancement refers to slight changes to the features of the original image, and in this application, the image can be scaled. Strong enhancement refers to a significant change in the structure or style of the original image, and in this application, Gaussian noise can be added to the image. The self-supervised learning strategy is used to process noisy images, calculate the consistency of the results, and improve the model effect. The third loss value L mse is:

[0102]

[0103] wherein, n represents, represents the probability of generating an image after weak enhancement processing of a face image, represents the probability distribution of generating an image after weak enhancement processing of a face flip image, represents the probability of generating an image after strong enhancement processing of a face image, represents the probability of generating an image after strong enhancement processing of a face flip image. An adaptive threshold is used to effectively identify noise samples. Noise samples are individually trained and learned using targeted methods, avoiding the influence of mixed training of clean samples and noise samples on training effect, reducing energy consumption and resource consumption during training, and improving resource use efficiency of the entire model training.

[0104] S7: obtaining a total loss value by combining the first loss value, the second loss value, and the third loss value. Specifically, the total loss value is:

[0105] L total = L sup + βL feat + γL mse

[0106] wherein, β represents a hyperparameter of the first loss value, and γ represents a hyperparameter of the third loss value. Since the second loss value is obtained by training the clean image subset, the training process of the clean image is relatively accurate, so it is not necessary to increase the weight before the second loss value, but it is necessary to increase the weight before the first loss value and the third loss value to reduce the influence of noise.

[0107] S8: when the fluctuation value of the total loss value is greater than or equal to a preset threshold, adjusting the model parameters of the initial recognition model, and performing cyclic training on the adjusted initial recognition model, so that the model gradually converges to an ideal state.

[0108] S9: when the fluctuation value of the total loss value is less than a preset threshold value, the current adjusted initial recognition model is taken as the facial expression recognition model, and the recognition accuracy of the obtained facial expression recognition model is higher and the stability is stronger.

[0109] Specifically, the initial recognition model is continuously trained, and the total loss value gradually decreases until the total loss value is stable, for example, the preset threshold value is set to 0.1. The fluctuation value refers to the absolute change value of the current total loss value relative to the last obtained total loss value. The fluctuation value is a positive value, which can be a floating value upward or a floating value downward. When the fluctuation value of the total loss value is greater than or equal to 0.1, it can be considered that the fluctuation of the total loss value is still large, and the training can be continued. When the fluctuation value of the total loss value is less than 0.1, the fluctuation of the total loss value is small at this time, and it can be considered that the total loss value is kept in a basically stable state, and the training is stopped at this time. The current adjusted initial recognition model is taken as the facial expression recognition model.

[0110] In summary, compared with the prior art, the following beneficial effects are achieved:

[0111] 1. The obtained face image is flipped, the consistency between the features of the face image and the features of the flipped face image is calculated by using the natural symmetry feature of the face, a first loss value is obtained, the model attention is focused on the key features of the face, thereby reducing the interference of irrelevant features and improving the recognition accuracy. In order to reduce the interference of noise labels, the face image is divided into a clean image subset and a noise image subset in advance. The clean image is reliable and easy to process. A supervised learning method is used for training to obtain a second loss value. The correct label is fully utilized to improve the convergence of the model training. The noise image is trained by using a self-supervised learning method to obtain a third loss value. The self-supervised learning does not depend on the given label information, and focuses on the consistency of the face image itself, thereby reducing the negative interference of the noise image on the training. Finally, the three loss values are fused, and the total loss value and the preset loss value are combined to adjust the initial recognition model, so that the model gradually converges to an ideal state, improves the efficiency of the recognition model training, and enhances the robustness and accuracy of the facial expression recognition model.

[0112] 2. An adaptive threshold is used to effectively identify noise samples. Noise samples are trained and learned separately by using a targeted method, so as to avoid the influence of mixed training of clean samples and noise samples on the training effect, reduce the energy consumption and resource consumption in the training process, and improve the resource use efficiency of the entire model training.

[0113] 3. The three loss values are fused, wherein the second loss value obtained by training the clean image subset is not added with a weight, and based on the second loss value, weights corresponding to the importance of the first loss value and the third loss value are set to obtain a total loss value, wherein the total loss value is mainly derived from the second loss value, that is, obtained by training the clean image subset, while taking into account the consistency loss (that is, the first loss value) calculated during the symmetry enhancement training and the third loss value obtained by training the noisy image sample, so that the total loss takes into account diversity and accuracy, and reduces the negative impact of noisy labels.

[0114] As shown in Figure 2 The training system of the facial expression recognition model provided by the embodiments of the present application comprises:

[0115] The picture processing module is configured to horizontally flip the obtained facial image to obtain a flipped facial image.

[0116] The model prediction module is configured to input the facial image and the flipped facial image into the initial recognition model respectively to obtain a first attention map of the facial image, a second attention map of the flipped facial image, a first predicted classification probability sequence of the facial image, and a second predicted classification probability sequence of the flipped facial image.

[0117] The symmetry enhancement feature decoupling module is configured to obtain a first loss value according to the first attention map and the second attention map.

[0118] The picture division module is configured to divide the facial image according to the first predicted classification probability sequence, the second predicted classification probability sequence, and an adaptive threshold corresponding to each expression category to obtain a clean image subset and a noisy image subset.

[0119] The second loss value analysis module is configured to train the clean image subset by using a supervised learning method to obtain a second loss value.

[0120] The third loss value analysis module is configured to train the noisy image subset by using a self-supervised learning method to obtain a third loss value.

[0121] The total loss value analysis module is configured to obtain a total loss value from the first loss value, the second loss value, and the third loss value. The total loss value analysis module is further configured to adjust the model parameters of the initial recognition model when the fluctuation value of the total loss value is greater than or equal to a preset threshold, train the initial recognition model after the adjustment, and use the initial recognition model after the current adjustment as the facial expression recognition model when the fluctuation value of the total loss value is less than the preset threshold.

[0122] An adaptive threshold module is configured to classify the face image into the expression category corresponding to the maximum predicted classification probability according to the maximum predicted classification probability in the first predicted classification probability sequence, to obtain a predicted sample corresponding to each expression category; and configured to obtain a classification probability mean corresponding to each expression category according to the predicted classification probability of each face image in the predicted sample of each expression category, and take the classification probability mean as an adaptive threshold of the expression category.

[0123] In the embodiment, the face expression recognition model training system has similar advantages to the face expression recognition model training method, which will not be repeated here.

[0124] The embodiment of the present application further provides a face expression recognition method, comprising:

[0125] The obtained face image is input into the face expression recognition model trained by the face expression recognition model training method, to obtain an expression category of the face image.

[0126] The embodiment of the present application further provides a face expression recognition system, comprising:

[0127] The face image recognition module is configured to input the obtained face image into the face expression recognition model trained by the face expression recognition model training method, to obtain an expression category of the face image.

[0128] In the embodiment, the face expression recognition method and the face expression recognition system have similar advantages to the face expression recognition model training method, which will not be repeated here.

[0129] The embodiment of the present application provides an electronic device, comprising a memory and a processor;

[0130] The memory is configured to store a computer program;

[0131] The processor is configured to implement the face expression recognition model training method or the face expression recognition method when executing the computer program.

[0132] The embodiment of the present application provides a computer readable storage medium, and the storage medium stores a computer program. When the computer program is executed by a processor, the face expression recognition model training method or the face expression recognition method is implemented.

[0133] In the embodiment, the electronic device and the computer readable storage medium have similar advantages to the face expression recognition model training method, which will not be repeated here.

[0134] It should be noted that the electronic device and the computer readable storage medium provided in the present application have similar beneficial effects to the training method of the facial expression recognition model described above, and will not be described here.

[0135] An electronic device that can be a server or a client of the present application will now be described, which is an example of a hardware device that can be applied to various aspects of the present application. The electronic device is intended to represent a wide variety of digital electronic computing devices, such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other suitable computing devices. The electronic device can also represent a variety of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown in the figures, their connections and relationships, and their functions, are meant to be examples only, and are not intended to limit the implementations of the present application described and / or claimed in this document.

[0136] The electronic device includes a computing unit that can perform various appropriate actions and processes in accordance with a computer program stored in a read-only memory (ROM) or a computer program loaded into a random access memory (RAM) from a storage unit. In the RAM, various programs and data required for device operation can also be stored. The computing unit, the ROM, and the RAM are connected to each other through a bus. An input / output (I / O) interface is also connected to the bus.

[0137] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be completed by instructing the relevant hardware through a computer program, and the program can be stored in a computer readable storage medium. When the program is executed, it can include the processes of the above-mentioned embodiments. The storage medium can be a magnetic disc, an optical disc, a read-only memory (ROM) or a random access memory (RAM), etc. In the present application, the units described as separate components can be or can not be physically separated, and the components shown as units can be or can not be physical units, i.e. they can be located in one place or distributed on multiple network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the embodiments of the present application. In addition, the functional units in each embodiment of the present application can be integrated in one processing unit, or each unit can exist physically, or two or more units can be integrated in one unit. The integrated unit can be realized in the form of hardware or in the form of a software functional unit.

[0138] It is to be noted that, in the present text, relational terms such as first and second and the like can be used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions. Moreover, the terms "comprises", "comprising", or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but can also include other elements not expressly listed or inherent to such process, method, article, or apparatus. An element proceeded by "comprises... a" does not, without more constraints, exclude the existence of additional identical elements in the process, method, article, or apparatus that comprises the element.

[0139] The above examples are merely used to illustrate the technical solutions of the present application, but not to limit it; even though the present application has been described in detail with reference to the foregoing examples, those of ordinary skill in the art should understand that they can still make modifications to the technical solutions recorded in the foregoing examples, or make equivalent replacements to some of the technical features; and these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A training method for a facial expression recognition model, characterized in that: include: Flip the acquired facial image horizontally to obtain a facial flipped image; Inputting the facial image and the face-flipped image into the initial recognition model respectively, obtaining a first attention map of the facial image, a second attention map of the face-flipped image, a first predicted classification probability sequence of the facial image, and a second predicted classification probability sequence of the face-flipped image respectively; Obtaining a first loss value according to the first attention map and the second attention map; Dividing the facial image according to the first predicted classification probability sequence, the second predicted classification probability sequence, and an adaptive threshold corresponding to each expression category to obtain a clean image subset and a noisy image subset; Using a supervised learning method to train the clean image subset to obtain a second loss value; Using a self-supervised learning method to train the noise image subset to obtain a third loss value; The first loss value, the second loss value and the third loss value are used to obtain a total loss value; When the fluctuation value of the total loss value is greater than or equal to a preset threshold, adjusting the model parameters of the initial recognition model, and training the adjusted initial recognition model; When the fluctuation value of the total loss value is less than a preset threshold, the currently adjusted initial recognition model is used as the facial expression recognition model; The step of dividing the facial image according to the first predicted classification probability sequence, the second predicted classification probability sequence, and the adaptive threshold corresponding to each expression category to obtain a clean image subset and a noisy image subset includes: The predicted classification probability corresponding to each expression category in the first predicted classification probability sequence is added to the predicted classification probability corresponding to the same expression category in the second predicted classification probability sequence, and the average value is obtained to obtain the mean predicted classification probability corresponding to each expression category; Comparing the predicted classification probability means corresponding to the plurality of expression categories, and determining the expression category corresponding to the largest predicted classification probability mean and the largest predicted classification probability mean as the expression category to which the facial image belongs and the predicted classification probability mean corresponding to the expression category to which it belongs; Classifying the facial image into the expression category corresponding to the maximum predicted classification probability according to the maximum predicted classification probability in the first predicted classification probability sequence, and obtaining a predicted sample corresponding to each expression category; Obtaining a classification probability mean corresponding to each expression category based on the predicted classification probability of each facial image in the predicted samples of each expression category, and using the classification probability mean as an adaptive threshold for the expression category; When the predicted classification probability mean of the expression category to which the facial image belongs is greater than or equal to the adaptive threshold of the same expression category, the facial image is determined to be a clean image, thereby obtaining the clean image subset; When the predicted classification probability mean of the expression category to which the facial image belongs is less than the adaptive threshold of the same expression category, the facial image is determined to be a noise image, and the noise image subset is obtained; Among them, the adaptive threshold T c for: in, represents the adaptive threshold corresponding to the kth type of expression, N sk represents the number of samples in the predicted sample corresponding to the k-th expression, p i Indicates the maximum value of the predicted classification probability corresponding to each image sample in the predicted sample.

2. The training method of the facial expression recognition model as claimed in claim 1, wherein The initial recognition model includes a feature extractor and a classifier, wherein the feature extractor is connected to the classifier; Inputting the facial image and the face-flipped image into the initial recognition model respectively to obtain a first attention map of the facial image, a second attention map of the face-flipped image, a first predicted classification probability sequence of the facial image, and a second predicted classification probability sequence of the face-flipped image respectively includes: Inputting the facial image and the face flipped image into the initial recognition model respectively, and outputting the first attention map and the second attention map after passing through the feature extractor; The first attention map and the second attention map are respectively input into the classifier to obtain the corresponding first predicted classification probability sequence and the second predicted classification probability sequence, wherein the first predicted classification probability sequence includes the predicted classification probabilities corresponding to the multiple expression categories obtained by predicting the facial image, and the second predicted classification probability sequence includes the predicted classification probabilities corresponding to the multiple expression categories obtained by predicting the facial flipped image.

3. The training method of the facial expression recognition model as claimed in claim 2, wherein The first loss value is: Among them, N represents the number of images, L represents the number of expression categories, H represents the image height, W represents the image width, and A ij The first attention map representing the j-th expression of the i-th facial image, Fp(A′ ij ) represents the second attention map of the j-th expression in the i-th face flipped image.

4. The training method of the facial expression recognition model as claimed in claim 1, wherein The second loss value is: Where N represents the number of images, Represents the yth fully connected layer of the initial recognition model i The weight of the expression-like label, y i Indicates the specified expression category label of the i-th image, f i represents the i-th feature output by the global average pooling layer of the initial recognition model, L represents the number of expression categories, and W j Represents the weight of the j-th type of expression in the fully connected layer of the initial recognition model.

5. The training method of the facial expression recognition model as claimed in claim 1, wherein The third loss value is: in, n represents the number of facial images, represents the probability of image generation after the facial image is weakly enhanced, represents the probability of image generation after the face flipped image is weakly enhanced, represents the probability of image generation after strong enhancement of the facial image, Represents the probability of image generation after strong enhancement of the face flipped image.

6. The training method of the facial expression recognition model according to any one of claims 1 to 5, wherein: The total loss value is: L total =L sup +βL feat +γL mse Among them, L feat Represents the first loss value, L sup Represents the second loss value, L mse Represents the third loss value, β represents the hyperparameter of the first loss value, and γ represents the hyperparameter of the third loss value.

7. A facial expression recognition method, characterized in that: include: The acquired facial image is input into a facial expression recognition model trained by the training method for a facial expression recognition model according to any one of claims 1 to 6 to obtain an expression category of the facial image.

8. An electronic device, characterized in that: including memory and processor; The memory is used to store computer programs; The processor is configured to implement the facial expression recognition model training method according to any one of claims 1 to 6 or the facial expression recognition method according to claim 7 when executing the computer program.

9. A computer-readable storage medium, characterized in that The storage medium stores a computer program, which, when executed by a processor, implements the training method for a facial expression recognition model according to any one of claims 1 to 6 or the facial expression recognition method according to claim 7.

Images