Label noisy expression recognition method under double-branch noise extraction suppression

Abstract

The invention discloses a label noisy expression recognition method under double-branch noise extraction suppression. The method comprises the following steps: step 1, constructing a classification recognition model; step 2, sample extraction; adopting a data set containing a noisy sample and a clean real sample as an original data set; inputting the original data set into a classification and recognition model, using a standard cross entropy function as a loss function, and performing classification by using the classification and recognition model to obtain predicted values of various labels corresponding to each sample; constructing a sample extraction module by using an EMA branch calculation model and a CL branch calculation model, and extracting a noise label; 3, manufacturing a pseudo label; 4, retraining to obtain an emotion recognition classification model; step 5, identifying facial expressions; and acquiring a facial image, inputting the facial image as input data into the emotion recognition and classification model, performing facial expression recognition by using the emotion recognition and classification model, and outputting a facial expression recognition result. According to the method, the expression recognition precision and efficiency can be effectively improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of image intelligent recognition, and in particular relates to a label-noised expression recognition method under dual-branch noise extraction suppression. Background Art

[0002] Facial expression is one of the most common and important signals for expressing human emotions and intentions. Expression recognition is the basis of emotion understanding, which makes facial expression recognition a core task in human-computer interaction. In recent years, facial expression recognition (FER) technology has made great progress and has become a key technology in various practical applications, such as medical, surveillance, and virtual reality. However, due to the subjectivity of the annotator, low-quality expression images, and the ambiguity of the expression itself, it is difficult to obtain high-quality and accurate expression labels. Many images are misclassified, and the labels of these images are noisy labels. The FER task with noisy labels is more challenging than the traditional FER task. If there are noisy labels in the dataset, the model will overfit these noisy labels during the training process, which will lead to poor overall performance of the model and thus fail to perform accurate expression recognition. However, in real-world applications, ideal noise-free reference datasets are usually difficult to obtain, and it is not realistic to rely on completely correct labels to identify wrong labels. As a result, it becomes particularly important to effectively deal with the noisy label problem in the absence of completely accurate labels. To this end, it is urgent to provide a new method for label-noisy expression recognition. Summary of the invention

[0003] In view of the problems existing in the above-mentioned prior art, the present invention provides a method for label-noisy expression recognition under dual-branch noise extraction suppression, which can effectively improve the recognition accuracy and efficiency of expressions, has a low degree of dependence on the accurate labels of samples, and can provide reliable technical support for label-noisy recognition. At the same time, it has good robustness and can provide a promising solution for improving the performance of FER models in the presence of noisy labels.

[0004] In order to achieve the above object, the present invention provides a method for label-noised expression recognition under dual-branch noise extraction suppression, comprising the following steps:

[0005] Step 1: Build a classification and recognition model;

[0006] Constructing a classification and recognition model, wherein the classification and recognition model includes a facial key point detection branch, a facial feature detection branch, a Cross-Fusion Transformer Encoder, and a pyramid architecture;

[0007] Step 2: Sample extraction;

[0008] S21: A dataset containing both noisy samples and clean real samples is used as the original dataset;

[0009] S22: Input the original data set into the classification recognition model, use the standard cross entropy function as the loss function, and use the classification recognition model to classify and obtain the model's predicted value and sample label;

[0010] S23: Single-branch noise label judgment is not enough to deal with various noise labels. Therefore, a dual-branch sample extraction module is designed to extract more noise labels. The sample extraction module is constructed using the EMA branch calculation model and the CL branch calculation model, and the obtained predicted labels and the corresponding sample labels in the original data set are input into the sample extraction module, and the noise labels are extracted using the sample extraction module; wherein, the EMA branch calculation model and the CL branch calculation model are used to find samples containing noise labels from two perspectives: the learning of the model on the samples during the training process and the joint distribution of the predicted labels and the sample labels;

[0011] A1: EMA is a statistical method used to calculate the average value of time series data. It is similar to the simple moving average (SMA), but it gives greater weight to the latest observations and is more sensitive to changes in recent data. By calculating the EMA of each sample prediction value, the long-term change trend of the sample prediction label can be more clearly captured, so that a more accurate judgment can be made on whether the sample label is a noise label.

[0012] The EMA branch calculation model is used to calculate the EMA value of each predicted label. For samples with noise labels, the model often predicts large fluctuations during training and often switches between different categories. This fluctuation indicates that the model has uncertainty in classifying the sample, mainly because the model finds that the characteristics of the sample do not match its label. This is used to determine whether the label is a noise label. Therefore, the long-term trend of the EMA value can be used to make a more accurate judgment on whether the sample label is a noise label.

[0013] A11: Obtain the EMA value of each predicted label according to formula (1);

[0014]

[0015] In the formula, represents the EMA value calculated under the kth epoch as the sample prediction value, y represents the label predicted by the model, and y k represents the predicted label of an image under the kth epoch, and α represents the weight parameter of the EMA value calculated under the k-1th epoch;

[0016] A12: The The corresponding sample label Compare them. If they are consistent, the current sample label is determined to be a clean label. If they are inconsistent, the current sample label is determined to be a noise label. At the same time, the sample label corresponding to the noise label is extracted.

[0017] A2: Use the CL branch calculation model to estimate the joint probability distribution of noisy labels and true labels; ConfidentLearning proposes a systematic method to estimate the probability distribution of noisy labels in the data set, which enables the CL branch to quantify the noise level of the data set and continuously update it during the training process, thereby more accurately identifying noisy labels. The CL branch mainly consists of three steps: designing the threshold, calculating the confidence joint count matrix, and estimating the joint probability distribution matrix.

[0018] A21: The average probability of each category label predicted by the model in the current round of training is used as the probability threshold G j , as shown in formula (2); for the predicted probability greater than the probability threshold G j The sample label is judged as a clean true label;

[0019]

[0020] In the formula, represents the sample label, represents the sample label j, x represents the input sample image, θ represents the model parameter, Indicates the predicted label of the sample image x input into the model with parameter θ probability; represents the sample set with sample label j, Indicates the number of samples in the set;

[0021] A22: Calculate the confidence joint count matrix using predicted labels and sample labels As shown in formula (3);

[0022]

[0023] In the formula, y represents the true label, Represents the set of samples with sample label i and true label j;

[0024] A23: Joint count matrix using confidence The value of M[i][j] in The sum of the counts is calculated occupy ratio, and then compare this ratio with Multiply as the calibration formula, as shown in formula (4); use formula (4) to obtain the calibrated confidence joint count matrix

[0025]

[0026] In the formula, m represents the number of categories of samples in the data set;

[0027] A24: Joint count matrix using calibrated confidence Estimate the joint probability distribution matrix of the noise label and the true label As shown in formula (5);

[0028]

[0029] A25: For the confidence joint count matrix after calibration The sample labels in the diagonal units are extracted as the true labels; for the confidence joint count matrix processed after calibration Sample labels in non-diagonal units are distributed according to the joint probability matrix of noise labels and true labels Extract label noisy samples, that is, for each category of sample label i∈1,2,...,m according to probability Sort and extract from low to high samples, these samples are used as label noisy samples, and the remaining clean samples form a clean data set, where n is the number of samples in the data set;

[0030] Step 3: Pseudo label creation;

[0031] After being extracted by the sample extraction module, the original data set is divided into labeled noisy samples and clean data sets. The clean data sets are used to continue training the classification and recognition model to obtain updated network weights. Based on the updated network weights, the labeled noisy samples are tested and the output results are used as pseudo labels corresponding to the noise labels.

[0032] Step 4: Retraining;

[0033] S41: Use pseudo labels to replace corresponding noise labels to form a corrected data set, so as to reduce the proportion of noise samples in the data set;

[0034] S42: Combine the corrected data set and the clean data set into a reconstructed data set, while ensuring the correction of the noise data, maintaining the integrity of the data set as a whole, and retaining the original data distribution characteristics of the data set, so that the model can learn more fully; use the reconstructed data set to train the classification recognition model, and at the same time, use the gradient suppression method and label smoothing technology to design the gradient suppression smoothing function Loss G-sm To enhance the robustness of the classification and recognition model, the sentiment classification and recognition module is obtained after training; among them, the gradient suppression smoothing function LossG-sm The design process is as follows:

[0035] B1: Calculate the gradient modulus g in the loss function GHMC_Loss according to formula (6);

[0036]

[0037] In the formula, is the probability predicted by the model, p * is the sample label;

[0038] B2: Calculate the gradient density GD(g) of the sample using g according to formula (7);

[0039]

[0040] In the formula, δ ε (g k ,g) shows that g is distributed in 1~N samples. The number of samples in the range, l ε (g) Representative The length of the interval;

[0041] B3: Set the weight to the inverse of the sample's gradient density GD(g), and obtain the loss function GHMC_Loss according to formula (8);

[0042]

[0043] In the formula, Loss is the cross entropy loss;

[0044] B4: Introducing label smoothing strategy Loss sm , and according to formula (9), the loss function GHMC_Loss is improved and adjusted to obtain the gradient suppression smoothing function Loss G-sm ;

[0045] Loss G-sm =βLoss GHM-C +γLoss sm (9);

[0046] In the formula, Where N is the number of samples, C is the number of categories, i is the index of the sample, j and k are the index of the category, x is the output of the model, and x i,j represents the output value of the i-th sample in the j-th category, y i represents the sample label value of the i-th sample, τ represents the smoothing factor; β and γ are hyperparameters;

[0047] S43: saving the obtained emotion recognition classification model;

[0048] Step 5: Recognize facial expressions;

[0049] Obtain a facial image and input it as input data into an emotion recognition classification model, use the emotion recognition classification model to recognize facial expressions, and output the facial expression recognition results.

[0050] As a preferred embodiment, in step one, the POSTER model is selected as the classification and recognition model.

[0051] As a preferred embodiment, in step 2 S21, the original data set is a data set modified based on the RAF-DB data set and the FERPlus data set.

[0052] As a preference, in B4 in step three, β=1.

[0053] As a preference, in B4 in step three, γ=2.

[0054] The present invention proposes a new method for solving the problem of facial expression recognition (FER) with noisy labels, which significantly reduces the influence of noisy labels on model prediction from the two perspectives of reducing the noise ratio of the data set and suppressing label noisy samples. The method is mainly divided into three core parts: sample extraction, pseudo label generation and retraining. First, sample extraction is performed by calculating the joint probability distribution matrix of the noisy label and the true label and the sliding exponential average of the model prediction value, so that the remaining samples in the data set are used as a clean data set. Then, reasonable pseudo labels are assigned to the label noisy samples using the training weights of the clean data set, and the noisy labels are replaced with pseudo labels to construct a corrected data set. Then, the corrected data set and the clean data set are composed of a reconstructed data set, thereby effectively reducing the number of noisy labels in the data set. Finally, the model is retrained using the reconstructed data set, which can effectively improve the prediction accuracy of the model. In addition, a gradient suppression smoothing function is designed to suppress the label noisy samples in the data set for the retraining process, which further improves the prediction accuracy of the model, thereby ensuring efficient and accurate recognition of expressions.

[0055] This method combines the design of a robust loss function with the selection of clean samples for training. It does not rely on completely accurate labels, but gradually identifies and corrects noisy labels through the training of the dataset in the model and the predicted labels of the model, thereby avoiding the problem of circular dependency, that is, "recognizing noisy labels may indeed require a reference dataset with correct labels". It has good robustness and can achieve good results even when the dataset contains noisy labels. It is almost unaffected by the proportion of noisy labels in the dataset. Experimental results on two public datasets show that this method is superior to several state-of-the-art solutions for FER tasks with noisy labels. This method can effectively improve the recognition accuracy and efficiency of facial expressions. It has a low degree of dependence on the accurate labels of samples and can provide reliable technical support for label-noisy recognition. At the same time, it can provide a promising solution for improving the performance of FER models in the presence of noisy labels. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] Figure 1 is a flow chart of the present invention;

[0057] Figure 2 is an architecture diagram of the sample extraction module in the present invention;

[0058] Figure 3 It is a flow chart of EMA value calculation in the present invention;

[0059] Figure 4 is a flow chart of the pseudo-label generation process in the present invention;

[0060] Figure 5 is a flow chart of the retraining process in the present invention;

[0061] Figure 6 is an example of the analysis of ambiguous expressions in the present invention;

[0062] Figure 7 is a graph of accuracy at different noise levels in the present invention;

[0063] Figure 8 This is an example diagram of sample extraction in the present invention. DETAILED DESCRIPTION

[0064] The true label refers to the real emotion label of the person in the picture, and the noise label refers to the emotion label that is incorrectly labeled. Noise labels are widely present in facial expression recognition (FER) datasets, and these noise labels seriously damage the performance of FER models. Since noise labels are ubiquitous in datasets, this application aims to enhance the performance of FER models when the dataset contains noise labels, and the experimental dataset used is a dataset containing noise labels. Therefore, the sample labels contain both noise labels and true labels.

[0065] The present invention will be further described below in conjunction with the accompanying drawings. Figure 1 In the figure, black text indicates the true label of the sample, and red text indicates the noise label;

[0066] like Figures 1 to 8 As shown, the present invention provides a method for label-noised expression recognition under dual-branch noise extraction suppression, comprising the following steps:

[0067] Step 1: Build a classification and recognition model;

[0068] Constructing a classification and recognition model, wherein the classification and recognition model includes a facial key point detection branch, a facial feature detection branch, a Cross-Fusion Transformer Encoder, and a pyramid architecture;

[0069] Step 2: Sample extraction;

[0070] S21: A dataset containing both noisy samples and clean real samples is used as the original dataset;

[0071] S22: Input the original data set into the classification recognition model, use the standard cross entropy function as the loss function, and use the classification recognition model to classify and obtain the model's predicted value and sample label;

[0072] S23: Figure 2 As shown, the EMA (Weighted Moving Average) branch calculation model and the CL (Confident Learning) branch calculation model are used to build a sample extraction module, and the obtained prediction labels and the corresponding sample labels in the original data set are input into the sample extraction module, and the noise labels are extracted using the sample extraction module;

[0073] A1: Figure 3 As shown in the figure, the EMA branch calculation model is used to calculate the EMA value of each predicted label, and a more accurate judgment is made on whether the sample label is a noise label based on the long-term change trend of the EMA value;

[0074] A11: Obtain the EMA value of each predicted label according to formula (1);

[0075]

[0076] In the formula, represents the EMA value calculated under the kth epoch as the sample prediction value, y represents the label predicted by the model, and y k represents the predicted label of an image under the kth epoch, and α represents the weight parameter of the EMA value calculated under the k-1th epoch;

[0077] A12: The The corresponding sample label Compare them. If they are consistent, the current sample label is determined to be a clean label. If they are inconsistent, the current sample label is determined to be a noise label. At the same time, the sample label corresponding to the noise label is extracted.

[0078] A2: Use the CL branch calculation model to estimate the joint probability distribution of noise labels and true labels;

[0079] A21: The design of the probability threshold helps to distinguish whether the label of a sample is a noise label; the average probability of each category label predicted by the model in the current round of training is used as the probability threshold G j , as shown in formula (2); for the predicted probability greater than the probability threshold G j The sample label is judged as a clean true label;

[0080]

[0081] In the formula, represents the sample label, represents the sample label j, x represents the input sample image, θ represents the model parameter, Indicates the predicted label of the sample image x input into the model with parameter θ probability; represents the sample set with sample label j, Indicates the number of samples in the set;

[0082] A22: Based on the extracted sample labels and the corresponding predicted labels, the confidence joint count matrix is ​​calculated using the predicted labels and sample labels As shown in formula (3), the rows of the matrix represent the number of predicted label samples, the columns represent the number of true label samples, and each cell M[i][j] represents the number of samples with the true label j and the sample label i.

[0083]

[0084] In the formula, y represents the true label, Represents the set of samples with sample label i and true label j;

[0085] A23: Due to the limitation of probability threshold, the total count in the count matrix will be less than or equal to the number of input sample labels. In order to make the total count consistent with the number of sample labels and to facilitate the calculation of the joint probability distribution of noise labels and true labels, it is necessary to perform weighted operations on the values ​​in the matrix. This step is called calibration of data distribution.

[0086] Specifically, the confidence joint count matrix is ​​used The value of M[i][j] in The sum of the counts is calculated occupy ratio, and then compare this ratio with Multiply as the calibration formula, as shown in formula (4); use formula (4) to obtain the calibrated confidence joint count matrix

[0087]

[0088] In the formula, m represents the number of categories of samples in the data set;

[0089] A24: Joint count matrix using calibrated confidence Estimate the joint probability distribution matrix of the noise label and the true label As shown in formula (5);

[0090]

[0091] A25: The confidence joint count matrix can simulate the distribution of noise labels and true labels. The calculated count matrix can be used to obtain samples whose sample labels are consistent with the true labels. These samples are in the diagonal units of the count matrix, and the non-diagonal units are samples whose sample labels are inconsistent with the true labels. For the confidence joint count matrix after calibration, The sample labels in the diagonal units are extracted as the true labels; for the confidence joint count matrix processed after calibration Sample labels in non-diagonal units are distributed according to the joint probability matrix of noise labels and true labels Extract label noisy samples, that is, for each category of sample label i∈1,2,...,m according to probability Sort and extract from low to high samples, these samples are used as label noisy samples, and the remaining clean samples form a clean data set, where n is the number of samples in the data set;

[0092] Step 3: Pseudo label creation;

[0093] After being extracted by the sample extraction module, the original data set is divided into labeled noisy samples and clean data sets. The clean data sets are used to continue training the classification and recognition model to obtain updated network weights. Based on the updated network weights, the labeled noisy samples are tested and the output results are used as pseudo labels corresponding to the noise labels. Figure 4 As shown, the red text represents the noisy label and the blue text represents the pseudo label.

[0094] Step 4: Retraining;

[0095] S41: Use pseudo labels to replace corresponding noise labels to form a corrected data set, so as to reduce the proportion of noise samples in the data set;

[0096] S42: Figure 5 As shown in the figure, the corrected data set and the clean data set are combined into a reconstructed data set, which ensures that the overall integrity of the data set is maintained while correcting the noise data, and retains the original data distribution characteristics of the data set, so that the model learning is more sufficient; the reconstructed data set is used to train the classification recognition model, and at the same time, the gradient suppression method and label smoothing technology are used to design the gradient suppression smoothing function Loss G-sm To enhance the robustness of the classification and recognition model, the sentiment classification and recognition module is obtained after training; among them, the gradient suppression smoothing function Loss G-sm The design process is as follows:

[0097] B1: Calculate the gradient modulus g in the loss function GHMC_Loss according to formula (6);

[0098]

[0099] In the formula, is the probability predicted by the model, p * is the sample label;

[0100] B2: g is proportional to the difficulty of detection. The larger the g, the more difficult the detection. If the model is forced to pay attention to these samples with large g, it will reduce the accuracy of the model. GHMC_Loss uses gradient density to reduce the weight of samples with large g, while not paying too much attention to easy-to-classify samples. According to formula (7), the gradient density GD(g) of the sample is calculated using g;

[0101]

[0102] In the formula, δ ε (g k ,g) shows that g is distributed in 1~N samples. The number of samples in the range, l ε (g) Representative The length of the interval;

[0103] B3: Set the weight to the inverse of the sample's gradient density GD(g), and obtain the loss function GHMC_Loss according to formula (8);

[0104]

[0105] In the formula, Loss is the cross entropy loss;

[0106] B4: Although label noisy samples and difficult-to-classify samples are essentially different, they are both difficult samples for the model. The idea of ​​suppressing difficult samples can be used to suppress label noisy samples that may exist in the data set. Drawing on the loss function GHMC_Loss, the goal of this loss function is to solve the problem of unbalanced distribution of positive and negative samples in the single-stage target detection process. While reducing the weight of simple samples, it also relatively reduces the weight of difficult samples to ensure that the model's response to these abnormal samples is not too prominent. Since GHMC_Loss mainly focuses on solving the problem of sample imbalance, in order to adapt to the research and make full use of the characteristics of GHMC_Loss, it is necessary to make some improvements and adjustments to the loss function, and then obtain the gradient suppression smoothing function Loss. G-sm Samples with unclear expression information are also a major factor affecting recognition accuracy, such as Figure 6 As shown in the figure, there are a large number of ambiguous expression images in the original dataset. A survey and statistics were conducted on some ambiguous expressions in the original dataset. The statistical results show that people's understanding of these expressions is usually concentrated on two or three labels, indicating that there is uncertainty in the labels of these samples. Ambiguous expressions will cause the model to be uncertain about some samples, which means that the model may not be able to clearly understand the emotions in the image, thus affecting the model's judgment of such images. In order to reduce this impact, the label smoothing strategy Loss is introduced. sm , label smoothing gives these images a certain error tolerance probability, which can prevent the model from over-trusting the sample labels.

[0107] Specifically, the label smoothing strategy Loss is introduced sm , and according to formula (9), the loss function GHMC_Loss is improved and adjusted to obtain the gradient suppression smoothing function Loss G-sm ;

[0108] Loss G-sm =βLoss GHM-C +γLoss sm (9);

[0109] In the formula, Where N is the number of samples, C is the number of categories, i is the index of the sample, j and k are the index of the category, x is the output of the model, and x i,j represents the output value of the i-th sample in the j-th category, y i represents the sample label value of the i-th sample, τ represents the smoothing factor; β and γ are hyperparameters;

[0110] S43: saving the obtained emotion recognition classification model;

[0111] Step 5: Recognize facial expressions;

[0112] Obtain a facial image and input it as input data into an emotion recognition classification model, use the emotion recognition classification model to recognize facial expressions, and output the facial expression recognition results.

[0113] As a preferred method, in step 1, the POSTER model is selected as the classification recognition model. The POSTER model mainly includes four core parts: facial key point detection, facial feature detection, Cross-Fusion Transformer Encoder and pyramid architecture. Given a picture input into POSTER, the picture passes through the facial key point detection branch and the facial image feature detection branch at the same time to obtain the facial key point feature x lm and facial image features x img , and then the facial key point feature x lm and facial image features x img Input into the Cross-Fusion Transformer Encoder, specifically the facial key point features x lm and facial image features x img The self-attention module in the input transformer is mapped into three matrices: query matrix Q, key matrix K and value matrix V. The specific cross-fusion operation is reflected in the interaction of image features and key point features when calculating self-attention, as shown in formula (10);

[0114]

[0115] In the formula, softmax(·) is the activation function of softmax. In order to prevent the gradient from being too small, use is the normalization scale factor.

[0116] In summary, the Cross-Fusion Transformer Encoder can be summarized as shown in formula (11);

[0117]

[0118] Where CFMSA(·) represents the Cross-Fusion MSA block, Norm(·) is the normalization operator, and MLP(·) represents the multi-layer perceptron.

[0119] As a preferred embodiment, in step 2 S21, the original data set is a data set modified based on the RAF-DB data set and the FERPlus data set.

[0120] As a preference, in B4 in step three, β=1, γ=2.

[0121] Example:

[0122] 1. Original dataset;

[0123] RAF-DB (Real-world Affective Faces Database) is the first real-world facial expression dataset containing basic or compound expressions, containing 12,271 training images and 3,068 test images. The images in this dataset have great variability in age, gender, race, head posture, lighting conditions, occlusion (such as glasses, facial hair or self-occlusion), post-processing operations (such as various filters and special effects), etc. Experiments were conducted using images of the six basic expressions (happiness, surprise, sadness, anger, disgust, fear) and neutral expressions in the dataset.

[0124] FERPlus is an extension of the FER2013 dataset for the ICML2013 challenge. FERPlus consists of large-scale expression images collected by Google search engine and new labels provided by Microsoft for FER2013. It contains 28,709 training images, 3,589 validation images, and 3,589 test images with a size of 48×48 pixels. FERPlus includes eight emotional states (six basic emotions, neutral, and contempt).

[0125] The noise dataset used in this application is modified based on RAF-DB and FERPlus. A portion of the training data (10%, 20% and 30%) is randomly selected and randomly assigned to other categories to produce RAF-DB datasets and FERPlus datasets containing 10%, 20% and 30% noise labels.

[0126] 2. Verification process;

[0127] In the experiment, POSTER was used as the classification network, and the mixed data of RAF-DB and FERPlus was used as the original data set. In the process of extracting samples, the loss function used was the standard cross entropy loss. In the process of retraining, the loss function used was Loss G-sm After experiments, the Loss G-sm The parameters are β = 1, γ = 2. The experiments were performed using pytorch on two Nvidia Tesla V100 graphics cards.

[0128] 2.1. DNES (Dual-Branch Noise Extraction and Suppression Method) is compared with other methods on the RAF-DB and FERPlus datasets with 10%, 20% and 30% noise labels. The comparison results are shown in Table 1.

[0129] Table 1: Comparative data table;

[0130]

[0131] Summary on RAF-DB: Table 1 compares DNES with previous methods on the RAF-DB dataset with 10%, 20%, and 30% noise labels in the training set. DNES performs well in accuracy, with average precision of 91.17%, 91.56%, and 91.58% in the last 5 epochs, respectively.

[0132] Summary on FERPlus: Table 1 compares DNES with previous methods on the FERPlus dataset with 10%, 20%, and 30% noise labels in the training set. All images in FERPlus are small-resolution grayscale images, and DNES still achieves excellent results, with average accuracies of 89.91%, 90.17%, and 89.69% in the last 5 epochs, respectively.

[0133] 2.2. Calculate the accuracy difference between various methods on the original data set and the data set with 10%, 20%, and 30% noise label ratio in the training set, represented by s. The specific results are shown in Tables 2 and 3.

[0134] Table 2: Comparative data table of training accuracy decline on the RAF-DB dataset;

[0135]

[0136] Summary on RAF-DB: Compared with the training accuracy on the original RAF-DB dataset, DNES has the least drop in training accuracy under noisy labels, which decreases by 1.01%, 0.62%, and 0.60% respectively.

[0137] Table 3: Comparative data table of training accuracy decline on the FERPlus dataset;

[0138] Method Original FERPlus s(10％) s(20％) s(30％) Baseline 91.62 3.72 4.42 6.02 SCN 88.01 3.73 4.84 5.54 RUL 88.75 1.82 3.7 4.85 PT 86.60 1.56 2.33 2.87 EAC 89.64 2.61 3.57 4.2 ReSup 89.12 1.3 1.77 2.11 Ours 91.72 1.81 1.55 2.03

[0139] Summary on FERPlus: Compared with the training accuracy on the original FERPlus dataset, the training accuracy of DNES under noisy labels decreased by 1.81%, 1.55%, and 2.03%, respectively. This shows that DNES successfully reduces the adverse effects of noisy labels on training accuracy.

[0140] The p-values ​​of various methods on different data sets were calculated to evaluate their statistical significance, as shown in Table 4.

[0141] Table 4: Comparison of p-values ​​on different data sets;

[0142]

[0143]

[0144] Overall, the experimental results on these two datasets demonstrate the effectiveness of DNES in dealing with FER tasks with noisy labels.

[0145] The performance of various models under different noise conditions was studied, and it was found that the accuracy of most models decreased with the increase of noise level, such as Figure 7 As shown in the figure, DNES performs well in this case and still maintains high accuracy in a 30% noise environment. This is because DNES not only uses a sample extraction module with two branches in the sample extraction process to extract more labeled noisy samples, but also creates pseudo labels to maintain the integrity of the dataset. In the retraining process, it uses a gradient suppression smoothing function to suppress the remaining labeled noisy samples in the reconstructed dataset, which is more comprehensive and adequate in dealing with noisy labels than other methods, reflecting the superiority of DNES.

[0146] DNES uses the CL branch and the EMA branch to extract labeled noisy samples from the dataset. It is less sensitive to noise when the noise level is low, and more sensitive when the noise level is moderate, which affects the quality of the reconstructed dataset and leads to an improvement in accuracy when the noise increases from 10% to 30%.

[0147] 3. Ablation experiment;

[0148] The RAF-DB dataset with 20% noisy labels is used as an example for ablation study. First, the effects of the two branches of extracting noisy labels on the FER task with noisy labels are explored, and the results are shown in Table 5.

[0149] Table 5: Comparative data of ablation experiments of CL branch and EMA branch;

[0150]

[0151]

[0152] Based on the experimental results, the two branches of DNES were deeply analyzed. First, it was found that the CL branch found 559 label-noisy samples, and the EMA branch found 429 label-noisy samples. Using these two branches at the same time, more label-noisy samples were found, totaling 669. Figure 8Some noisy label samples were visualized. It was found that the samples found by the CL branch and the EMA branch can complement each other and increase the number of noisy label samples found. According to the visualization results, the CL branch is more likely to find samples that are simply mislabeled, while the EMA branch is more likely to find samples with poor image quality. In terms of accuracy, using both branches at the same time also surpasses the performance of using the CL branch or the EMA branch alone, which confirms the previous speculation.

[0153] In order to ensure that DNES only suppresses noise labels without damaging clean labels, the RAF-DB dataset is taken as an example to explore the changes of clean samples in the original dataset after passing through the Noise Label Extraction Module and Generating Pseudo-labels process of DNES. The specific results are shown in Table 6:

[0154] Table 6: Clean sample difference data table;

[0155] Noise Before After Similarity(%) 10％ 11029 10863 98.49 20％ 9772 9663 98.88 30％ 8548 8477 99.16

[0156] As can be seen from Table 6, DNES has almost no damage to the originally clean samples in the dataset.

[0157] A comparative experiment was designed to explore the effect of Loss G-sm As for the impact of noisy labels on the FER task performance, experiments were conducted on the RAF-DB dataset with 20% noisy labels. The experimental results show that the use of CE_Loss and Loss G-sm The average accuracy on the last five epochs was 91.20% and 91.56% respectively. The improvement effect of GHMC_Loss was explored. The experimental results show that the average accuracy when only GHMC_Loss is used is 91.25%, and when only Loss is used, the average accuracy is 91.56%. sm The average accuracy is 91.12%, which is lower than that of using Loss G-sm This shows that Loss G-sm The effectiveness of is shown in Table 7. At the same time, using Loss on the original RAF-DB dataset G-sm The results show that Loss G-sm It also effectively enhances the performance of the model.

[0158] Table 6: Loss function ablation experiment comparison data table;

[0159]

[0160]

[0161] Experiments on other FER models; In order to prove the universality of the method, experiments were conducted on the equally excellent expression recognition networks ARM and FDRL, and the results are shown in Table 8;

[0162] Table 8: Comparative data table of experimental results of other FER models;

[0163] Noise ARM DNES-ARM 10％ 88.45 89.64 20％ 84.42 90.32 30％ 79.11 89.65

[0164] As shown in Table 8, the accuracy of ARM on the RAF-DB dataset with noise ratios of 10%, 20%, and 30% respectively reaches 88.45%, 84.42%, and 79.11%, and the average accuracy of DNES in the last 5 epochs reaches 89.64%, 90.32%, and 89.65%, respectively.

[0165] Table 9: Comparison data table of experimental results of FDRL;

[0166] Noise FDRL DNES-FDRL 10％ 86.08 88.50 20％ 84.16 88.49 30％ 79.30 87.84

[0167] As shown in Table 9, the accuracy of FDRL on the RAF-DB dataset with noise ratios of 10%, 20%, and 30% respectively reaches 86.08%, 84.16%, and 79.30%, and the average accuracy of DNES in the last 5 epochs reaches 88.50%, 88.49%, and 87.84%, respectively.

[0168] Experiments on other tasks; The algorithm is not only applicable to expression recognition, but also has certain effects on the presence of noisy labels in datasets in other fields. Experiments were conducted on the Category Flower Dataset, and VGG16 was selected as the classification network. Similar to the previous experiments, a portion of the training data (10%, 20%, and 30%) was randomly selected, and noisy labels were generated by assigning them to other random flower categories to destroy their labels. Experiments were conducted on datasets with noise ratios of 10%, 20%, and 30%, respectively, and the accuracy reached 93.13%, 91.17%, and 92.75%, respectively, as shown in Table 10, which proves that the method can be used not only in the field of expression recognition, but also in other fields.

[0169] Table 10: Comparison data table of experimental results in other fields;

[0170] Noise Baseline DNES-VGG16 10％ 89.90 93.13 20％ 85.68 91.17 30％ 79.41 92.75

[0171] The present invention proposes a new method for solving the problem of facial expression recognition (FER) with noisy labels, which significantly reduces the influence of noisy labels on model prediction from the two perspectives of reducing the noise ratio of the data set and suppressing label noisy samples. The method is mainly divided into three core parts: sample extraction, pseudo label generation and retraining. First, sample extraction is performed by calculating the joint probability distribution matrix of the noisy label and the true label and the sliding exponential average of the model prediction value, so that the remaining samples in the data set are used as a clean data set. Then, reasonable pseudo labels are assigned to the label noisy samples using the training weights of the clean data set, and the noisy labels are replaced with pseudo labels to construct a corrected data set. Then, the corrected data set and the clean data set are composed of a reconstructed data set, thereby effectively reducing the number of noisy labels in the data set. Finally, the model is retrained using the reconstructed data set, which can effectively improve the prediction accuracy of the model. In addition, a gradient suppression smoothing function is designed to suppress the label noisy samples in the data set for the retraining process, which further improves the prediction accuracy of the model, thereby ensuring efficient and accurate recognition of expressions.

[0172] This method combines the design of a robust loss function with the selection of clean samples for training. It does not rely on completely accurate labels, but gradually identifies and corrects noisy labels through the training of the dataset in the model and the predicted labels of the model, thereby avoiding the problem of circular dependency, that is, "recognizing noisy labels may indeed require a reference dataset with correct labels". It has good robustness and can achieve good results even when the dataset contains noisy labels. It is almost unaffected by the proportion of noisy labels in the dataset. Experimental results on two public datasets show that this method is superior to several state-of-the-art solutions for FER tasks with noisy labels. This method can effectively improve the recognition accuracy and efficiency of facial expressions. It has a low degree of dependence on the accurate labels of samples and can provide reliable technical support for label-noisy recognition. At the same time, it has good robustness and can provide a promising solution for improving the performance of FER models in the presence of noisy labels.

Claims

1. A method for label-noised expression recognition under dual-branch noise extraction suppression, characterized in that: The steps include: Step 1: Build a classification and recognition model; Constructing a classification and recognition model, wherein the classification and recognition model includes a facial key point detection branch, a facial feature detection branch, a Cross-Fusion Transformer Encoder, and a pyramid architecture; Step 2: Sample extraction; S21: A dataset containing both noisy samples and clean real samples is used as the original dataset; S22: Input the original data set into the classification recognition model, use the standard cross entropy function as the loss function, and use the classification recognition model to classify and obtain the model's predicted value and sample label; S23: construct a sample extraction module using the EMA branch calculation model and the CL branch calculation model, input the obtained prediction labels and the corresponding sample labels in the original data set into the sample extraction module, and use the sample extraction module to extract noise labels; A1: Use the EMA branch calculation model to calculate the EMA value of each predicted label, and make a more accurate judgment on whether the sample label is a noise label based on the long-term change trend of the EMA value; A11: Obtain the EMA value of each predicted label according to formula (1); In the formula, represents the EMA value calculated under the kth epoch as the sample prediction value, y represents the label predicted by the model, and y k represents the predicted label of an image under the kth epoch, and α represents the weight parameter of the EMA value calculated under the k-1th epoch; A12: The The corresponding sample label Compare them. If they are consistent, the current sample label is determined to be a clean label. If they are inconsistent, the current sample label is determined to be a noise label. At the same time, the sample label corresponding to the noise label is extracted. A2: Use the CL branch calculation model to estimate the joint probability distribution of noise labels and true labels; A21: The average probability of each category label predicted by the model in the current round of training is used as the probability threshold G j , as shown in formula (2); for the predicted probability greater than the probability threshold G j The sample label is judged as a clean true label; In the formula, represents the sample label, Indicates that the sample label is j, x represents the input sample image, θ represents the model parameter, Indicates the predicted label of the sample image x input into the model with parameter θ The probability of represents the sample set with sample label j, Indicates the number of samples in the set; A22: Calculate the confidence joint count matrix using predicted labels and sample labels As shown in formula (3); In the formula, y represents the true label, Represents the set of samples with sample label i and true label j; A23: Joint count matrix using confidence The value of M[i][j] in The sum of the counts is calculated occupy ratio, and then compare this ratio with Multiply as the calibration formula, as shown in formula (4); use formula (4) to obtain the calibrated confidence joint count matrix In the formula, m represents the number of categories of samples in the data set; A24: Joint count matrix using calibrated confidence Estimate the joint probability distribution matrix of the noise label and the true label As shown in formula (5); A25: For the confidence joint count matrix after calibration The sample labels in the diagonal units are extracted as the true labels; for the confidence joint count matrix processed after calibration Sample labels in non-diagonal units are distributed according to the joint probability matrix of noise labels and true labels Extract label noisy samples, that is, for each category of sample label i∈1,2,...,m according to probability Sort and extract from low to high samples, these samples are used as label noisy samples, and the remaining clean samples form a clean data set, where n is the number of samples in the data set; Step 3: Pseudo label creation; After being extracted by the sample extraction module, the original data set is divided into labeled noisy samples and clean data sets. The clean data sets are used to continue training the classification and recognition model to obtain updated network weights. Based on the updated network weights, the labeled noisy samples are tested and the output results are used as pseudo labels corresponding to the noise labels. Step 4: Retraining; S41: Use pseudo labels to replace corresponding noise labels to form a corrected data set, so as to reduce the proportion of noise samples in the data set; S42: Combine the corrected data set and the clean data set into a reconstructed data set, while ensuring the correction of the noise data, maintaining the integrity of the data set as a whole, and retaining the original data distribution characteristics of the data set, so that the model can learn more fully; use the reconstructed data set to train the classification recognition model, and at the same time, use the gradient suppression method and label smoothing technology to design the gradient suppression smoothing function Loss G-sm To enhance the robustness of the classification and recognition model, the sentiment classification and recognition module is obtained after training; among them, the gradient suppression smoothing function Loss G-sm The design process is as follows: B1: Calculate the gradient modulus g in the loss function GHMC_Loss according to formula (6); In the formula, is the probability predicted by the model, p * is the sample label; B2: Calculate the gradient density GD(g) of the sample using g according to formula (7); In the formula, δ ε (g k ,g) shows that g is distributed in 1~N samples. The number of samples in the range, l ε (g) Representative The length of the interval; B3: Set the weight to the inverse of the sample's gradient density GD(g), and obtain the loss function GHMC_Loss according to formula (8); In the formula, Loss is the cross entropy loss; B4: Introducing label smoothing strategy Loss sm , and according to formula (9), the loss function GHMC_Loss is improved and adjusted to obtain the gradient suppression smoothing function Loss G-sm ; Loss G-sm =βLoss GHM-C +γLoss sm (9); In the formula, Where N is the number of samples, C is the number of categories, i is the index of the sample, j and k are the index of the category, x is the output of the model, and x i,j represents the output value of the i-th sample in the j-th category, y i represents the sample label value of the i-th sample, τ represents the smoothing factor; β and γ are hyperparameters; S43: saving the obtained emotion recognition classification model; Step 5: Recognize facial expressions; Obtain a facial image and input it as input data into an emotion recognition classification model, use the emotion recognition classification model to recognize facial expressions, and output the facial expression recognition results.

2. According to the method of claim 1, the method is characterized in that: In step one, the POSTER model is selected as the classification and recognition model.

3. According to the method of claim 1, the method is characterized in that: In step 2 S21, the original data set is a data set modified based on the RAF-DB data set and the FERPlus data set.

4. The label-noised expression recognition method under dual-branch noise extraction suppression according to claim 1 or 2, characterized in that: In B4 in step three, β=1.

5. According to the method of claim 4, the method is characterized in that: In B4 in step three, γ=2.

Images