Noise-robust facial expression recognition method and system

Abstract

The present invention provides a noise-robust expression recognition method, system, storage medium, and electronic device, relating to the technical field of expression recognition. In the present invention, an expression recognition model is constructed, comprising a deep neural network, a co-division submodel, and a tri-regularization submodel. The method uses a co-learning strategy for model training, mining clean data, ambiguous data, and noisy label data in a dataset based on the logical relationship between the prediction results of two deep neural networks and the expression labels of samples, thereby improving the model's discriminative ability. Different training strategies are also used to optimize parameters for different data types until the expression recognition model converges, thereby improving the model's noise resistance and the accuracy of expression recognition.

Description

Technical Field

[0001] The present invention relates to the technical field of expression recognition, and in particular to an expression recognition method, system, storage medium and electronic equipment with noise robustness. Background Art

[0002] Facial expression recognition refers to separating specific facial expressions from a given static image or dynamic video sequence, thereby determining the psychological emotions of the identified object.

[0003] At present, some studies have proposed the use of uncertain learning strategies. By constructing an uncertainty estimation module, the uncertainty of each facial expression image is calculated. Samples with large uncertainty tend to be considered as noise samples, and samples with small uncertainty tend to be considered as clean samples, thereby suppressing the impact of uncertain samples on the network.

[0004] However, facial expression noise exists between similar and dissimilar categories, namely ambiguous expressions and mislabeled expressions. Previous methods were unable to distinguish between these two types of noise, causing the model to fit the noise, limiting its generalization ability and recognition accuracy. Summary of the Invention

[0005] (1) Technical problems solved

[0006] In response to the shortcomings of the existing technology, the present invention provides a noise-robust expression recognition method, system, storage medium and electronic device, which solves the technical problem of being unable to distinguish ambiguous expressions and expressions with incorrect labels.

[0007] (2) Technical solution

[0008] To achieve the above objectives, the present invention is implemented through the following technical solutions:

[0009] A noise-robust expression recognition method is provided, which constructs an expression recognition model, wherein the expression recognition model includes a deep neural network, a co-division sub-model, and a tri-regularization sub-model; the method comprises:

[0010] S1. Obtain a training set containing several facial expression images and their original labels;

[0011] S2. Input each facial expression image into the two deep neural networks to obtain the first and second predicted categories;

[0012] S3. Input the original label, the first and second predicted categories into the co-division sub-model at the same time, and divide the training set into three parts: a clean data set, an ambiguous data set, and an incorrectly labeled data set;

[0013] S4, inputting the clean dataset, the ambiguous dataset, and the incorrectly labeled dataset into a tri-regularization sub-model, and optimizing parameters using different training strategies for each of the three datasets until the expression recognition model converges;

[0014] S5. Input the facial expression image to be recognized into the converged model to obtain an expression recognition result that is robust to noise.

[0015] Preferably, the deep neural network adopts the ResNet18 network.

[0016] Preferably, in S3:

[0017] The clean data set is divided according to:

[0018] Pred1=y and Pred2=y

[0019] The basis for dividing the ambiguous dataset is:

[0020] (Pred1=y and Pred2≠y)or (Pred1≠y and Pred2=y)

[0021] The basis for dividing the wrong label data set is:

[0022] Pred1≠y and Pred2≠y

[0023] Among them, Pred1 and Pred2 respectively represent the first and second predicted categories of the same facial expression image, and y represents the original label of the facial expression image.

[0024] Preferably, in S4, a supervised loss function is used to optimize parameters for the clean data set, and the formula is as follows:

[0025] L sup =L CE (Prob1,y)+L CE (Prob2,y)

[0026] Among them, L sup Refers to the supervised loss function; Prob1 and Prob2 represent the category prediction probabilities of the two deep neural networks for each face image; L CE represents the cross entropy loss function.

[0027] Preferably, for the ambiguous data set, a mutuality loss function is used for parameter optimization, and the formula is as follows:

[0028] L mut =(1-λ)*L sup +λ*Lmic

[0029] Among them, L mut is the mutual loss function;

[0030]

[0031] λ is a hyperparameter used to balance the weights of the two loss functions; max Refers to the maximum value of λ, e refers to the current epoch, e r Refers to the epoch when λ reaches its maximum value, and β controls the shape of the function

[0032] L mic =D KL (Prob2||Prob1)+D KL (Prob1||Prob2)

[0033] L mic To mimic the loss function, it consists of the KL divergence between the predicted probabilities of two deep neural networks.

[0034] Preferably, for the mislabeled dataset, an unsupervised consistent loss function is used for parameter optimization, and the formula is as follows:

[0035]

[0036] Among them, L usc is an unsupervised consistent loss function; pre-obtain the enhanced image of each image x in the wrong label dataset Based on two deep neural networks, we can obtain The predicted probability of the category and MSE is the L2 loss function.

[0037] Preferably, the overall loss function of the expression recognition model is as follows:

[0038] L total =L sup +γL mut +ωL usc

[0039] Among them, γ and ω are hyperparameters used to balance the weights of the last two terms.

[0040] A noise-robust expression recognition system is provided, which constructs an expression recognition model comprising a deep neural network, a co-division sub-model, and a tri-regularization sub-model; the system comprises:

[0041] An acquisition module is used to obtain a training set containing several facial expression images and their original labels;

[0042] A prediction module, configured to input each facial expression image into the two deep neural networks to obtain a first and a second predicted category;

[0043] A division module is used to input the original label, the first and second predicted categories into the co-division sub-model at the same time, and divide the training set into three parts: a clean data set, an ambiguous data set, and an incorrectly labeled data set;

[0044] an optimization module, configured to input the clean dataset, the ambiguous dataset, and the mislabeled dataset into a tri-regularization sub-model, and optimize parameters using different training strategies for each of the three datasets until the expression recognition model converges;

[0045] The recognition module is used to input the facial expression image to be recognized into the converged model to obtain the expression recognition result with noise robustness.

[0046] A storage medium stores a computer program for noise-robust expression recognition, wherein the computer program enables a computer to execute the expression recognition method described above.

[0047] An electronic device, comprising:

[0048] one or more processors;

[0049] Memory; and

[0050] One or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, the programs including instructions for executing the expression recognition method as described above.

[0051] (3) Beneficial effects

[0052] The present invention provides a noise-robust expression recognition method, system, storage medium, and electronic device. Compared with the prior art, the present invention has the following advantages:

[0053] In the present invention, an expression recognition model is constructed, which includes a deep neural network, a co-division submodel and a tri-regularization submodel. The method uses a co-learning strategy to train the model, and mines clean data, ambiguous data and noise label data in a data set according to the logical relationship between the prediction results of two deep neural networks and the expression labels of samples, thereby improving the model's discrimination ability. Different training strategies are used to optimize parameters for different data types until the expression recognition model converges, thereby improving the model's noise resistance and the accuracy of expression recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0055] Figure 1 A flowchart of a noise-robust facial expression recognition method provided by an embodiment of the present invention;

[0056] Figure 2 An architectural diagram of an expression recognition model provided by an embodiment of the present invention;

[0057] Figure 3 A RAF-DB test set confusion matrix is ​​provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0058] To make the objectives, technical solutions, and advantages of the embodiments of the present invention more clear, the technical solutions in the embodiments of the present invention are clearly and completely described. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of them. All other embodiments derived by persons of ordinary skill in the art based on the embodiments of the present invention without inventive effort are within the scope of protection of the present invention.

[0059] The embodiments of the present application solve the technical problem of being unable to distinguish between ambiguous expressions and incorrectly labeled expressions by providing a noise-robust expression recognition method, system, storage medium, and electronic device.

[0060] The technical solution in the embodiments of the present application is to solve the above technical problems, and the overall idea is as follows:

[0061] Due to the ambiguity of facial expressions, the subjectivity of annotators, and low-quality facial images, there are a large number of noisy labels in facial expression datasets in real scenarios. Some of this noise occurs in similar categories, that is, ambiguous data; the other part occurs in dissimilar categories, that is, incorrectly labeled data. Previous methods were unable to effectively distinguish between these two types of noise, which restricted the recognition accuracy of the model. The embodiments of the present invention propose a noise-robust expression recognition method and system that can effectively mine three types of samples in the dataset: clean, ambiguous, and incorrectly labeled, thereby better suppressing the model's anti-noise ability and improving the model's generalization and recognition accuracy.

[0062] In order to better understand the above technical solution, the above technical solution will be described in detail below with reference to the accompanying drawings and specific implementation methods.

[0063] Example 1:

[0064] like Figure 1 As shown, the embodiment of the present invention provides a noise-robust expression recognition method, constructing an expression recognition model, such as Figure 2 As shown, the expression recognition model includes a deep neural network, a co-division sub-model and a tri-regularization sub-model; the method includes:

[0065] S1. Obtain a training set containing several facial expression images and their original labels;

[0066] S2. Input each facial expression image into the two deep neural networks to obtain the first and second predicted categories;

[0067] S3. Input the original label, the first and second predicted categories into the co-division sub-model at the same time, and divide the training set into three parts: a clean data set, an ambiguous data set, and an incorrectly labeled data set;

[0068] S4, inputting the clean dataset, the ambiguous dataset, and the incorrectly labeled dataset into a tri-regularization sub-model, and optimizing parameters using different training strategies for each of the three datasets until the expression recognition model converges;

[0069] S5. Input the facial expression image to be recognized into the converged model to obtain an expression recognition result that is robust to noise.

[0070] The embodiments of the present invention propose a noise-robust expression recognition method and system, which can effectively mine three types of samples in the data set: clean, ambiguous and mislabeled samples, thereby better suppressing the model's anti-noise ability and improving the model's generalization and recognition accuracy.

[0071] Next, we will introduce the various steps of the above technical solution in detail based on the specific content:

[0072] In step S1, a training set including several facial expression pictures and their original labels is obtained.

[0073] This embodiment of the present invention uses the real-world facial expression dataset RAF-DB. First, the face images are aligned and cropped using the MTCNN model. Each image is then scaled to a 224x224 resolution and subjected to data augmentation such as random flipping and random erasing.

[0074] The embodiment of the present invention adopts deep learning technology and co-learning (cooperative training strategy), the backbone network selects ResNet18, the two ResNet18 networks are initialized differently, and the model structure is as follows Figure 2 As shown in the figure, the model mainly consists of two modules: co-division and tri-regularization. The training process is as follows:

[0075] In step S2, i.e., model warmup, each facial expression picture is input into the two deep neural networks to obtain the first and second predicted categories.

[0076] Each facial expression image is fed into two ResNet18 networks, which simultaneously perform feature extraction and category prediction, and optimize the model using the cross-entropy loss function. This warm-up process improves the model's discriminative capabilities, enabling the co-division module to perform more accurate data segmentation.

[0077] In step S3, i.e., co-division, the original labels, the first and second predicted categories are simultaneously input into the co-division sub-model to divide the training set into three parts: a clean dataset, an ambiguous dataset, and an incorrectly labeled dataset.

[0078] After the model is warmed up, the two networks' predicted categories Pred1 and Pred2 for each image, along with the image's expression label y, are fed into the co-division module. Based on the logical relationship between the two networks' predictions and the original labels, the co-division module divides the dataset into three parts: clean data, ambiguous data, and noisy label data. The specific divisions are based on the following:

[0079] (1) clean data

[0080] Clean data mainly contains easily recognizable facial expressions. The labels of these expressions match the facial features, and both networks can predict them correctly. Therefore, the data with the same predictions and labels of the two networks are considered clean data. The formula is as follows

[0081] Pred1=y and Pred2=y

[0082] (2)ambiguous data

[0083] Ambiguous data primarily consists of images with difficult-to-recognize facial expressions. The characteristics of these expressions often conflate two or more categories, such as fear being misidentified as surprise. For these expressions, the predictions of the two networks may diverge. Therefore, we consider data with inconsistent predictions between the two networks as ambiguous data, using the following formula:

[0084] (Pred1=y and Pred2≠y)or (Pred1≠y and Pred2=y)

[0085] (3) noisy label data

[0086] For mislabeled data, their features are usually not related to the label. For example, a happy expression is labeled as sad, and the predictions of the two networks are different from the label. Therefore, we regard the data whose predictions of the two networks are inconsistent with the label as mislabeled data. The formula is as follows:

[0087] Pred1≠y and Pred2≠y

[0088] Among them, Pred1 and Pred2 respectively represent the first and second predicted categories of the same facial expression image, and y represents the original label of the facial expression image.

[0089] In step S4, i.e., tri-regularization: the clean dataset, the ambiguous dataset, and the incorrectly labeled dataset are input into the tri-regularization sub-model, and different training strategies are used to optimize parameters for the three datasets until the expression recognition model converges.

[0090] In the previous step, the Co-division module divides the dataset into three subsets: clean data, ambiguous data, and noisy label data. Since these three subsets have obvious differences, three different training strategies are adopted for these three subsets: supervised learning, mutuality learning, and unsupervised learning. Specifically, they include:

[0091] (1)Supervised learning

[0092] Since the features in the clean data are highly correlated with the labels, a supervised loss function is used for parameter optimization for the clean data set. The formula is as follows:

[0093] L sup =L CE (Prob1,y)+L CE (Prob2,y)

[0094] Among them, L sup Refers to the supervised loss function, which consists of two large cross entropy losses of the network and is used to guide the model to learn clean samples; Prob1 and Prob2 represent the category prediction probabilities of the two deep neural networks for each face image; L CE represents the cross entropy loss function.

[0095] (2) Mutuality learning

[0096] Because ambiguous data contains both correctly labeled and incorrectly labeled samples, deep neural networks prioritize learning simple samples in the early stages of training. However, as training progresses, they gradually overfit to noise, resulting in decreased model performance and generalization. To ensure that the two networks in this model can fully learn simple samples early on while minimizing noise interference in the middle and late stages, we use a mutual loss function for parameter optimization for this ambiguous dataset. The formula is as follows:

[0097] L mut =(1-λ)*L sup +λ*L mic

[0098] Among them, L mut is the mutual loss function;

[0099]

[0100] λ is a hyperparameter used to balance the weights of the two loss functions; max Refers to the maximum value of λ, e refers to the current epoch, e r Refers to the epoch when λ reaches its maximum value, and β controls the shape of the function

[0101] L mic =D KL (Prob2||Prob1)+D KL (Prob1||Prob2)

[0102] L mic To mimic the loss function, it consists of the KL divergence between the predicted probabilities of two deep neural networks.

[0103] (3)Unsupervised learning

[0104] Since noisy label data will cause the model to overfit, resulting in a decrease in the recognition accuracy of the model. In order to make full use of this part of the data and reduce the model's fitting of noise, an unsupervised learning strategy is adopted. That is, instead of using these wrong labels, a stronger data augmentation is first performed on each image x in this subset to obtain the enhanced image. Then we get two network pairs respectively. The predicted probability of the category and In order to enable the model to learn more robust features, an unsupervised consistent loss function is used to optimize the parameters for the mislabeled dataset. The formula is as follows:

[0105]

[0106] Among them, L usc is an unsupervised consistent loss function; pre-obtain the enhanced image of each image x in the wrong label dataset Based on two deep neural networks, we can obtain The predicted probability of the category and MSE is the L2 loss function.

[0107] (4)Overall loss function

[0108] That is, the overall loss function of the expression recognition model is as follows

[0109] L total =L sup +ωL mut +ωL usc

[0110] Among them, γ and ω are hyperparameters used to balance the weights of the last two terms.

[0111] In step S5, the facial expression picture to be recognized is input into the converged model to obtain an expression recognition result with noise robustness.

[0112] During the test phase, the co-division and tri-regularization modules are not used. The predicted category of each expression is directly obtained based on the average of the predicted probabilities of the two ResNet18 networks. The recognition accuracy of the embodiment of the present invention on the RAF-DB test set is as high as 90.25%. The confusion matrix is ​​as follows: Figure 3 shown.

[0113] An embodiment of the present invention provides a noise-robust expression recognition system, which constructs an expression recognition model. The expression recognition model includes a deep neural network, a co-division submodel, and a tri-regularization submodel. The system includes:

[0114] An acquisition module is used to obtain a training set containing several facial expression images and their original labels;

[0115] A prediction module, configured to input each facial expression image into the two deep neural networks to obtain a first and a second predicted category;

[0116] A division module is used to input the original label, the first and second predicted categories into the co-division sub-model at the same time, and divide the training set into three parts: a clean data set, an ambiguous data set, and an incorrectly labeled data set;

[0117] an optimization module, configured to input the clean dataset, the ambiguous dataset, and the mislabeled dataset into a tri-regularization sub-model, and optimize parameters using different training strategies for each of the three datasets until the expression recognition model converges;

[0118] The recognition module is used to input the facial expression image to be recognized into the converged model to obtain the expression recognition result with noise robustness.

[0119] An embodiment of the present invention provides a storage medium storing a computer program for noise-robust facial expression recognition, wherein the computer program enables a computer to execute the facial expression recognition method described above.

[0120] An embodiment of the present invention provides an electronic device, including:

[0121] one or more processors;

[0122] Memory; and

[0123] One or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, the programs including instructions for executing the expression recognition method as described above.

[0124] It is understandable that the noise-robust expression recognition system, storage medium and electronic device provided in the embodiments of the present invention correspond to the noise-robust expression recognition method provided in the embodiments of the present invention. The explanations, examples and beneficial effects of the relevant contents can refer to the corresponding parts in the expression recognition method and will not be repeated here.

[0125] In summary, compared with the existing technology, the present invention has the following beneficial effects:

[0126] 1. In the present invention, an expression recognition model is constructed, which includes a deep neural network, a co-division submodel and a tri-regularization submodel; the method uses a co-learning strategy to train the model, and mines clean data, ambiguous data and noise label data in the data set according to the logical relationship between the prediction results of two deep neural networks and the expression labels of the samples to improve the model's discrimination ability; and different training strategies are used to optimize parameters for different data types until the expression recognition model converges, thereby improving the model's noise resistance and the accuracy of expression recognition.

[0127] 2. Using the mutual loss function to constrain ambiguous data can enable the model to fully grasp the clean samples while reducing the fitting of noise.

[0128] 3. Use unsupervised learning strategies for noisy label data to improve the robustness of the model without using incorrect labels.

[0129] It should be noted that, in this document, relational terms such as first and second, etc., are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply the existence of any such actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article, or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or device. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of other identical elements in the process, method, article, or device comprising the element.

[0130] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present invention.

Claims

1. A noise-robust facial expression recognition method, characterized in that: Constructing an expression recognition model, wherein the expression recognition model includes a deep neural network, a co-division sub-model, and a tri-regularization sub-model; the method includes: S1. Obtain a training set containing several facial expression images and their original labels; S2. Input each facial expression image into the two deep neural networks to obtain the first and second predicted categories; S3. Input the original label, the first and second predicted categories into the co-division sub-model at the same time, and divide the training set into three parts: a clean data set, an ambiguous data set, and an incorrectly labeled data set; S4, inputting the clean dataset, the ambiguous dataset, and the incorrectly labeled dataset into a tri-regularization sub-model, and optimizing parameters using different training strategies for each of the three datasets until the expression recognition model converges; S5. Input the facial expression image to be recognized into the converged model to obtain an expression recognition result that is robust to noise.

2. The facial expression recognition method according to claim 1, wherein The deep neural network adopts the ResNet18 network.

3. The facial expression recognition method according to claim 1 or 2, wherein: In the S3: The clean data set is divided according to: Pred1=y and Pred2=y The basis for dividing the ambiguous dataset is: (Pred1=y and Pred2≠y)or (Pred1≠y and Pred2=y) The basis for dividing the wrong label data set is: Pred1≠y and Pred2≠y Among them, Pred1 and Pred2 respectively represent the first and second predicted categories of the same facial expression image, and y represents the original label of the facial expression image.

4. The facial expression recognition method according to claim 3, wherein: In S4, for the clean data set, a supervised loss function is used to optimize the parameters, and the formula is as follows: L sup =L CE (Prob1,y)+L CE (Prob2,y) Among them, L sup Refers to the supervised loss function; Prob1 and Prob2 represent the category prediction probabilities of the two deep neural networks for each face image; L CE represents the cross entropy loss function.

5. The facial expression recognition method according to claim 4, wherein: For the ambiguous dataset, the mutual loss function is used for parameter optimization, and the formula is as follows: L mut =(1-λ)*L sup +λ*L mic Among them, L mut is the mutual loss function; λ is a hyperparameter used to balance the weights of the two loss functions; max Refers to the maximum value of λ, e refers to the current epoch, e r Refers to the epoch when λ reaches its maximum value, and β controls the shape of the function L mic =D KL (Prob2||Prob1)+D KL (Prob1||Prob2) L mic To mimic the loss function, it consists of the KL divergence between the predicted probabilities of two deep neural networks.

6. The facial expression recognition method according to claim 5, wherein: For the incorrectly labeled dataset, an unsupervised consistent loss function is used for parameter optimization, and the formula is as follows: Among them, L usc is an unsupervised consistent loss function; pre-obtain the enhanced image of each image x in the wrong label dataset Based on two deep neural networks, we can obtain The predicted probability of the category and MSE is the L2 loss function.

7. The facial expression recognition method according to claim 6, wherein: The overall loss function of the expression recognition model is as follows: L total =L sup +γL mut +ωL usc Among them, γ and ω are hyperparameters used to balance the weights of the last two terms.

8. A noise-robust facial expression recognition system, characterized in that: Constructing an expression recognition model, the expression recognition model including a deep neural network, a co-division sub-model and a tri-regularization sub-model; the system includes: An acquisition module is used to obtain a training set containing several facial expression images and their original labels; A prediction module, configured to input each facial expression image into the two deep neural networks to obtain a first and a second predicted category; A division module is used to input the original label, the first and second predicted categories into the co-division sub-model at the same time, and divide the training set into three parts: a clean data set, an ambiguous data set, and an incorrectly labeled data set; an optimization module, configured to input the clean dataset, the ambiguous dataset, and the mislabeled dataset into a tri-regularization sub-model, and optimize parameters using different training strategies for each of the three datasets until the expression recognition model converges; The recognition module is used to input the facial expression image to be recognized into the converged model to obtain the expression recognition result with noise robustness.

9. A storage medium, characterized in that: The computer program for noise-robust facial expression recognition is stored therein, wherein the computer program enables a computer to execute the facial expression recognition method according to any one of claims 1 to 7.

10. An electronic device, characterized in that: include: one or more processors; Memory; as well as One or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, and the programs include instructions for executing the expression recognition method according to any one of claims 1 to 7.

Images