An expression recognition method based on label confidence estimation and multi-network joint training

Abstract

The application provides an expression recognition method based on label confidence estimation and multi-network joint training, which comprises four modules connected in sequence by electrical signals, namely reliable training set initialization, multi-network joint training, label revision and final expression classification network training, wherein: the reliable training set initialization divides the original training set into an absolutely reliable training set and an unreliable training set, the multi-network joint training is used for updating the reliable training set, the label revision modifies the labels of part of relatively reliable samples in the unreliable training set and adds them to the reliable training set, and the final expression classification network training trains the final expression classification network from the reliable training set. The application can accurately eliminate or correct the noise samples, successfully suppresses the uncertainty in expression recognition, solves the problem that the difficult samples and the noise samples are difficult to distinguish, and reduces the influence of overfitting.

Description

Technical Field

[0001] This invention belongs to the field of facial expression recognition technology. More specifically, it relates to a facial expression recognition method based on label confidence estimation and multi-network joint training. Background Technology

[0002] Facial expressions, as one of the most important nonverbal communication tools, can convey one's inner emotions. Psychological research has found that humans are capable of deciphering the meanings of various facial expressions and categorizing them as one of the basic emotions. To enable computers to understand various facial expressions, facial expression recognition has become a research hotspot in the field of computer vision.

[0003] Deep convolutional neural networks (DCNNs) are constructed by mimicking the visual perception mechanisms of biological organisms. They can perform both supervised and unsupervised learning. The shared parameters of the convolutional kernels within their hidden layers and the sparsity of inter-layer connections enable DCNNs to extract gridded features with relatively low computational cost. After achieving great success in computer vision tasks such as face recognition, classic models such as VGG, GoogLeNet, and ResNet have been directly applied to facial expression recognition.

[0004] The performance degradation of models due to the subjectivity of dataset annotators, image quality defects, and the inherent confounding nature of facial expressions is particularly pronounced in large-scale facial expression recognition datasets in natural scenes. This results in high uncertainty in these datasets, hindering further improvements in facial expression recognition methods based on convolutional deep neural networks and supervised learning. Existing expression recognition methods essentially suppress uncertainty by estimating the label confidence of samples in different ways, and then using this label confidence to determine the weight of a sample in loss calculation or whether to retain it in the training set. These methods generally do not specifically address the three completely different causes and influencing processes of label errors, image quality defects, and the inherent confounding nature of facial expressions, thus failing to adequately suppress uncertainty in facial expression recognition in natural scenes. Summary of the Invention

[0005] The technical problem to be solved by this invention is to provide an expression recognition method based on label confidence estimation and multi-network joint training, which addresses the shortcomings or improvement needs of existing technologies. By suppressing the uncertainty in the expression dataset, the method effectively improves the accuracy of expression classification by using multi-network joint training.

[0006] The present invention solves its technical problem by adopting the following technical solution: The facial expression recognition method based on label confidence estimation and multi-network joint training provided by this invention includes four modules connected in sequence by electrical signals: reliable training set initialization, multi-network joint training, label revision, and final facial expression classification network training. Specifically: reliable training set initialization divides the original training set into an absolutely reliable training set and an unreliable training set; multi-network joint training is used to update the reliable training set; label revision selects some relatively reliable samples from the unreliable training set, modifies their labels, and adds them to the reliable training set; and final facial expression classification network training trains the facial expression classification network from the reliable training set.

[0007] Furthermore, the reliable training set initialization employs N-fold cross-validation and a first convolutional neural network; The specific implementation steps of the N-fold cross-validation are as follows: the original training set is evenly divided into N parts, the first part is taken as the validation set, and all other data are used as the training set; the training set is input into the first convolutional neural network for training, and then the validation set is input into the trained first convolutional neural network to obtain the prediction results of the validation set samples; Repeat the above steps N times, where the kth execution takes the kth data as the validation set, so that all samples are used as validation data once, k=2,3,...,N.

[0008] Furthermore, the backbone of the first convolutional neural network is ResNet-18 and pre-trained on the face dataset MS-Celeb-1M.

[0009] Furthermore, the method for dividing the absolutely reliable training set and the unreliable training set is as follows: samples whose prediction results are consistent with the true labels are added to the absolutely reliable training set. Samples whose predicted results do not match the true labels are added to the unreliable training set. .

[0010] Furthermore, the multi-network joint training employs an expression classification network and a confidence estimation network, and includes the following steps: (1) Extraction of sample features and class features: Using a reliable training set A model was trained using an expression classification network to extract sample features and class features from all samples in the dataset; The reliable training set When the information is unknown, use an absolutely reliable training set. initialization; The facial expression classification network employs a second convolutional neural network, which uses a reliable training set. Training is performed; the second convolutional neural network uses the same network structure and pre-training method as the first convolutional neural network; The sample features and class features of the sample are extracted from the second convolutional neural network; (2) Construction of the confidence network training set: Used for absolutely reliable training sets The samples in the dataset are used to generate a positive sample set based on the sample features and class features described in step (1). and negative sample set ; The specific method for generating the positive and negative sample sets is as follows: For For any sample, the sample features obtained in step (1) are concatenated with the class features of the c-th class, where c is the label of the sample. All concatenation results are used as the positive sample set. ;right For any sample, use the sample features obtained in step (1) and the first sample... The class features are concatenated, where For all other classes not equal to c, all concatenation results are treated as the negative sample set. ; (3) Confidence network training: Used to train a confidence network based on the positive and negative sample sets described in step (2), and output an unreliable training set. Confidence scores for all samples in the dataset; The confidence network employs a multilayer perceptron (MLP), consisting of a six-layer fully connected network. The activation function for the first five layers is ReLU, and the activation function for the sixth layer is the sigmoid function. This network, through progressive dimensionality reduction, leverages the powerful nonlinear fitting capability of the fully connected neural network to fully consider the information in the sample features and category features, as well as the correlation between the two, to determine the reasonableness of assigning a label to the sample to that category, thereby estimating the label confidence. The confidence score of the sample is: For any sample in the network, the sample features extracted in step (1) and the c-th type features are concatenated, where c is the label of the sample. The input of the network is the confidence score of the sample. (4) Reliable training set update: Used to update the reliable training set based on the confidence score described in step 3. ; The specific method for updating the reliable training set is as follows: The confidence score of the sample is greater than the threshold Then add it to the training set. Threshold Used to determine whether a sample should be added to the training subset; (5) Iterative training: This is used to repeat steps (1) to (4) to further update the training set. ; The stopping condition for iterative training is reaching the preset number of iterations or ceasing to feed data to the training set. Add more reliable samples.

[0011] Furthermore, the sample feature extraction method is as follows: input any image into the second convolutional neural network, pass through all convolutional layers and the first fully connected layer, and the extracted features are used as sample features; The method for extracting class features of the samples is as follows: extract sample features from all samples belonging to the same category and calculate the average, which is the center of all samples belonging to the same category.

[0012] Furthermore, the label revision is based on the sample's confidence score and a set reasonable threshold. and To change the labels of samples and add them to a reliable training set. middle; The method for revising the labels is as follows: updating the labels after multi-network joint training. The confidence level of the middle sample is less than the threshold And after the update medium confidence level greater than The labels of the samples are modified to the corresponding categories with the highest confidence; the relabeled samples are then added to the training set. .

[0013] Furthermore, the label revision is based on the following two cases: when the confidence score of the instance formed by concatenating the sample features and the corresponding category features is low, i.e., < When the label is modified; when the confidence score of an instance concatenated with sample features and other category features is high, i.e. > Modify its tags in time; among them Set to 0.1, Set as the average confidence level of samples with the same label .

[0014] Furthermore, the threshold setting is used to determine whether to modify the label of a sample.

[0015] Furthermore, the final expression classification network training obtains a reliable training set based on the label revision step. The final facial expression classification network is obtained by training the third convolutional neural network; the third convolutional neural network adopts the same network structure and pre-training as the first convolutional neural network.

[0016] Compared with the prior art, the present invention has the following main advantages: (1) This invention proposes a label confidence estimation method. By utilizing the powerful nonlinear fitting capability of fully connected neural networks, the information in the sample features, category features, and the correlation between the two features is fully extracted to determine the rationality of assigning the category as the label of the sample, so as to estimate the label confidence.

[0017] (2) This invention proposes an uncertainty suppression method that dynamically provides a "clean" training set for the facial expression classification network by correcting erroneous labels.

[0018] (3) The present invention also adopts a multi-network alternating iterative training method. The clean dataset obtained in each round of alternating optimization stage and label correction stage will be perturbed, that is, the samples contained in the dataset will change, and the deep features output by the expression classification network will also be perturbed and fine-tuned, so that the trained network has better robustness.

[0019] (4) This invention utilizes an unsupervised face recognition dataset for pre-training, with the aim of improving the model’s ability to extract low-level texture information in order to enhance its generalization ability. To demonstrate the effectiveness of this method, experiments were conducted using two datasets: RAFDB and FER2013. The RAFDB dataset, released in 2017 by Shan Li et al., contains nearly 30,000 face images (RGB three-channel color images) in natural scenes, labeled in two different ways: the first with 7 basic expression categories, and the second with 11 composite expression categories. We used the subset labeled with the 7 basic expression categories from this dataset. According to the official classification, the training set contains 12,271 samples, and the test set contains 3,068 samples. FER2013, released in 2013 on the Kaggle competition platform, consists of 48*48 pixel grayscale images with manually labeled 7 basic expression categories. The training set contains 28,704 images, and the validation and test sets each contain 3,589 images.

[0020] Furthermore, in the classification experiments, as shown in Table 1, the baseline used ResNet18 pre-trained on the MS-Celeb-1M face recognition dataset; the SCN backbone was also ResNet18. The accuracy of the RAF-DB dataset in the table is the result published by the authors in their paper, while the results for the FER2013 dataset were not published by the authors. We reproduced the results in the table above using the source code published by the authors. The backbone in RLPS is different from the ResNet18 used in this invention. For the FER2013 dataset, a state-of-the-art backbone published on that dataset was used, resulting in better experimental results. However, on the RAF-DB dataset, the authors used a simpler backbone, leading to a lower overall accuracy (the paper only published the accuracy and average accuracy for each class; the overall accuracy of 78.23% was calculated based on the accuracy of each class). Therefore, the experimental results published by this method are less fair compared to other methods. The authors of the CL method did not conduct experiments on expression recognition-related datasets. Since the core of this method is to cleanse samples by predicting probabilities through cross-validation, and it is not coupled with any specific classification model, we reproduced the method using ResNet18 as the classification network and with the experimental settings exactly the same as the baseline. The table above shows our reproduction results. The DMUE backbone is also ResNet18. The test accuracy of the RAF-DB dataset in Table 1 is the result published by the authors in the paper. However, the authors did not publish the results on the FER2013 dataset. Since the authors did not release the source code, we did not reproduce the results of this method on FER2013.

[0021] In summary, under the same datasets, backbone networks, and evaluation metrics, our method achieved high test accuracy on both the RAF-DB and FER2013 datasets. On the RAF-DB dataset, our method significantly outperformed most methods, only 0.56% lower than the published state-of-the-art method DMUW, and 1.47% higher than the baseline, reaching 88.20%. While the experimental results on the FER2013 dataset were slightly lower than RLPS, as mentioned earlier, direct comparison with other methods in the table cannot guarantee fairness. However, under the same backbone network, our method achieved the best result of 71.08% on the FER2013 dataset. Attached Figure Description

[0022] Figure 1 This is a flowchart of the facial expression recognition method based on label confidence estimation and multi-network joint training according to an embodiment of the present invention; Figure 2 This is a diagram of the tag confidence network structure according to an embodiment of the present invention; Figure 3 This is a schematic diagram of the tag confidence estimation process according to an embodiment of the present invention; Figure 4 This is a schematic diagram of the alternating iterative training process of the facial expression classification network and the label confidence network in an embodiment of the present invention; Figure 5 This is a diagram of the facial expression classification network structure according to an embodiment of the present invention; Figure 6 This is the confusion matrix diagram of the present invention on the RAF-DB test set; Figure 7 This is a graph showing the relationship between the number of samples and the test accuracy of this invention. Figure 8 This is a comparison chart showing the performance degradation trend of the method of this invention as noise increases; Figure 9 This is a comparison chart of the performance degradation of the noisy and noiseless algorithms of this invention. Detailed Implementation

[0023] This invention discloses an expression recognition method based on label confidence estimation and multi-network joint training. The method comprises four modules: reliable training set initialization, multi-network joint training, label revision, and final expression classification network training. The reliable training set initialization divides the original training set into an absolutely reliable training set and an unreliable training set. The multi-network joint training updates the reliable training set. The label revision changes the labels of unreliable samples and adds them to the reliable training set. The final expression classification network training obtains the final expression classification network from the reliable training set. This invention cleverly transforms the unsupervised learning problem of label confidence estimation into a supervised learning task through reasonable assumptions, accurately eliminating or correcting labels for noisy samples, successfully suppressing uncertainty in expression recognition, and solving the problem of difficulty in distinguishing between hard and noisy samples, as well as the effects of overfitting.

[0024] The present invention will be further described in detail below with reference to the embodiments and accompanying drawings. It should be understood that the specific embodiments described herein are only for explaining the present invention and are not intended to limit the present invention. Furthermore, the technical features involved in the various embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.

[0025] The facial expression recognition method based on label confidence estimation and multi-network joint training provided by this invention, such as... Figure 1 As shown, the process includes the following steps: S101 Reliable training set initialization, S102 Multi-network joint training, S103 Label revision, and S104 Final expression classification network training.

[0026] S101 is initialized with a reliable training set, and the reliable and unreliable training sets are divided using N-fold cross-validation and the first convolutional neural network 1. The structure of the first convolutional neural network 1 is as follows: Figure 5As shown, its backbone network uses ResNet18 and is pre-trained on the face dataset MS-Celeb-1M.

[0027] The N-fold cross-validation is used to estimate the model's generalization ability, estimate the model's true prediction error, and adjust the model's hyperparameters, while also preventing overfitting. The N-fold cross-validation process involves first randomly dividing the dataset into N equal parts. Then, the model is trained and validated N times, each time using N-1 samples as the training set and the other as the validation set, ensuring that each sample is used as the validation set exactly once. Finally, the average of the N validation accuracies is taken as the final cross-validation result. Generally, samples whose test results in cross-validation match their annotations are considered easy samples, while others are considered hard samples or noisy samples. Therefore, these easy samples constitute an absolutely reliable training set. And used to initialize a reliable training set. Difficult or noisy samples are used to initialize unreliable training sets. :

[0028]

[0029] in Representative sample Predictions in cross-validation tests yes The tag.

[0030] Furthermore, the basic block of the first convolutional neural network 1 consists of four convolutional layers and two skip connections, with ReLU as the activation function. The network input is a 224*224*3 image, which outputs a 7*7*512 feature map after passing through the fourth basic block. Then, after passing through a 7*7 average pooling layer and a flattening operation, it outputs a 512-dimensional feature map. Finally, it passes through two fully connected layers to obtain the output result.

[0031] Furthermore, during the pre-training of the first convolutional neural network 1, after ResNet-18 has been trained on the MS-Celeb-1M dataset, weight parameters are generated. Because the first convolutional neural network 1 uses ResNet-18 as its backbone, the parameters can be directly loaded. Using datasets The pre-trained parameters are obtained after one training cycle. The purpose of pre-training with the MS-Celeb-1M face recognition dataset is to improve the model's ability to extract low-level texture information, thereby enhancing its generalization ability. If using... When initializing the classification network, the last fully connected layer is randomly initialized because the number of categories in the face recognition dataset and the expression recognition dataset is different. and Unlike the last fully connected layer, this random initialization will further lead to different latent semantics for the same dimension of the deep features output by the penultimate fully connected layer between different iterations. Ensuring the semantic similarity of the output deep features plays a crucial role in the convergence of the label confidence network.

[0032] The S102 multi-network joint training employs an expression classification network and a confidence estimation network to update the reliable training set. Specifically, it includes the following steps: sample feature and class feature extraction, confidence network training set construction, confidence network training, reliable training set update, and iterative training.

[0033] (1) Extraction of sample features and class features. (This involves) obtaining... Then, the facial expression classification network is trained on it. Given a facial expression image... Input the facial expression classification network (second convolutional neural network 2) to obtain sample features. :

[0034] in This indicates that the sample has undergone feature extraction operations through all convolutional layers and the first fully connected (FC) layer. This represents the trainable parameters of these layers.

[0035] For a specific emoji category Class features Belonging to The center representation of all samples:

[0036] in Is all tags equal to The sample set.

[0037] (2) Construction of the confidence network training set. An absolutely reliable training set is used. The samples in the dataset are used to generate a positive sample set based on the sample features and class features described in step 1. and negative sample set .

[0038] The generation of the positive and negative sample sets is due to all samples They are all very reliable. It can be seen as The label is then used to construct the sample feature set. Therefore, the sample features and the class features corresponding to the label c of that sample are concatenated to obtain the positive sample set. Sample features and labels are Other class features are concatenated to obtain the negative sample set. , means as follows: , , , in This indicates a feature splicing operation. For sample features and class features The concatenated feature vector, where R is the set of real numbers.

[0039] (3) Confidence network training. The positive sample set obtained in step 2 is used... and negative sample set Used for training the belief network. See flowchart below. Figure 3 The backbone of this network is an MLP (consisting of six fully connected layers). The activation function for the first five layers is ReLU, and the activation function for the sixth layer is the sigmoid function. The structure is as follows: Figure 2 As shown. The input to the confidence network is the deep features of the samples and classes extracted by the facial expression classification network. (256+256 dimensions), the output is the estimated confidence score (1 dimension, ranging from...). (Between). This network, through gradual dimensionality reduction, leverages the powerful nonlinear fitting ability of fully connected neural networks to fully extract information from sample features, category features, and the correlation between the two features, judging the rationality of assigning a label to the class for the sample, and estimating the label confidence. ReLU is used as the activation function for the first five layers, primarily for the following reasons: fully connected neural networks have a large number of parameters and computational cost, and using ReLU can save some computational overhead; it makes the activation values ​​contain more zeros to increase network sparsity and reduce overfitting; and it alleviates gradient vanishing to ensure training efficiency. Experiments show that further increasing the depth or width of the network, or appropriately reducing its size, has very little impact on the results. Therefore, a simple network structure is adopted, with each of the first five layers reduced to half dimensionality and then an output layer added.

[0040] It should be noted that the trainable parameters in an MLP are optimized by minimizing the cross-entropy loss, as shown in the following formula:

[0041] in Output confidence score. These are the trainable parameters of the MLP. It is an instance tag. This is the feature vector obtained by concatenating sample features and class features. When hour, ,when hour, Typically, sample features and class features are close to each other in positive examples and far apart in negative examples. Therefore, MLPs learn a non-linear decision function from these instances, mapping the similarity between samples and classes to confidence scores. When a sample's label is sufficiently reliable, the sample features should be close to the class features of the corresponding label, resulting in a high confidence score for the MLP's output, and vice versa. Therefore, confidence scores can be used to determine whether a sample is reliable and whether it can be added to the training subset.

[0042] (4) Reliable training set update. Based on the confidence score output by the confidence network, the training set is updated from... The training set is then updated by selecting relatively reliable samples. However, after initializing the reliable training set, a large number of reliable samples are still not added to it. Therefore, those samples from... Excluded samples (i.e.) These are input into a confidence network to evaluate their label confidence scores. In an MLP, test instances are represented as... That is, sample features and the corresponding tags Class characteristics The concatenation is performed. If the confidence score is high enough, it indicates that the sample features are close to the class features corresponding to the label, indicating the reliability of the sample label. Therefore, a threshold is set. This determines whether a sample should be added to the training set. Samples with confidence scores greater than a threshold will be added to the training set. The specific update method is as follows:

[0043] in, Output confidence score. Indicates category as The feature vector. "+" indicates a data merging operation.

[0044] (5) Iterative training. This involves repeatedly executing steps 1-4 to further update the training set. After the training set is updated, we can use it to train a new classification network and update the features. Since more reliable samples participate in the model's training, the representational power of the new classification network is improved. Therefore, the newly trained classification network needs to be used to update sample features and class features. Simultaneously, the confidence network can explore more reliable samples and optimize feature representations. In this way, the outputs of the two networks are coupled.

[0045] It should be noted that the iteration terminates when the preset number of iterations is reached or when no more data is added to the training set. Add more reliable samples.

[0046] It should be noted that although new samples were identified as reliable samples and added to the training subset, their actual confidence level was lower than that of the original samples. Therefore, it is unnecessary to use these samples to construct a new set of positive and negative instances, as this may weaken the confidence estimation network's ability to distinguish between reliable and unreliable samples. Therefore, in the alternating optimization, the update... and The instance features in the dataset allow the belief network to adapt to feature changes caused by fine-tuning, but do not add new instances to the positive and negative instance sets.

[0047] Furthermore, the belief network is initialized with random parameters before starting alternating iterative training, and then each iteration "inherits" the parameters from the previous iteration without re-initializing them. Before alternating iterative training, the facial expression classification network is loaded... Then, use a subset of datasets The pre-trained parameters are obtained after one training cycle. In each iteration, the facial expression classification network first loads the pre-trained parameters. Then, the subset of data output by the uncertainty suppression module in that iteration is used for further training. The facial expression classification network is reinitialized in each iteration. This is to avoid severe overfitting in later iterations and to prevent the removal of noisy samples from the current iteration, which may have already been trained in the classification network in previous iterations (the samples removed between iterations have perturbations, especially in the early stages of an iteration, where the perturbations are larger). The confidence network is not reinitialized because the deep features output by the classification network in each iteration are fine-tuned; retaining the previous parameters helps improve the generalization ability of the confidence network.

[0048] S103 Label Revision. This invention proposes a cautious label revision strategy to change the label of a sample and add it to a reliable training set based on the sample's confidence score and a reasonable threshold. See details. Figure 1 .

[0049] In practice, some unreliable data is generated through mislabeling. Using confidence estimation to correct the labels of these samples can further expand the clean training set and improve the performance of representation learning.

[0050] Furthermore, when determining whether a sample has modified its label, two conditions should be considered: (1) instances where the sample features and labeled class features are connected have low confidence scores; (2) instances where the sample features and other class features are connected have high confidence scores. Based on these rules, two thresholds are set. and Select samples whose labels can be modified. When the confidence of instances with label class features is lower than [a certain threshold], [the following applies]. Instances with other class characteristics have a maximum confidence level higher than [a certain level]. At this point, the sample's label will be modified to the class corresponding to the maximum confidence score. The modified sample can then be added to the training set. The updated training set is then fed into the facial expression classification network. The training set update process can be described as follows:

[0051] In the formula: Indicates sample The revised label, , where argmax returns the index of the maximum value. ∩ represents the intersection operation in the set.

[0052] Furthermore, to overcome the class imbalance problem, adaptive thresholds for different classes are set based on the average confidence score of samples with the same label. Instead : .

[0053] Where C is the set of all samples whose labels are equal to c, and |C| represents the total number of samples in set C.

[0054] S104 Final expression classification network training, used to obtain a reliable training set based on the label revision step S103. The final facial expression classification network is obtained by training with the third convolutional neural network 3.

[0055] The third convolutional neural network 3 adopts the same network structure and pre-training as the first convolutional neural network 1.

[0056] In both embodiments, the RAFDB and FER2013 facial expression databases were used. The RAFDB dataset, released in 2017 by Shan Li et al., contains nearly 30,000 images of human faces in natural scenes (RGB three-channel color images), labeled in two different ways: the first with 7 basic facial expression categories, and the second with 11 composite facial expression categories. The images in this dataset were collected from social networking platforms such as Flickr and Facebook and manually labeled. We used a subset of this dataset labeled with the 7 basic facial expression categories. According to the official classification, its training set contains 12,271 samples, and the test set contains 3,068 samples, with a similar proportion of each category as the test set. This dataset is one of the most commonly used datasets for facial expression recognition in natural scenes. FER2013, released in 2013 on the Kaggle competition platform, consists of 48*48 pixel grayscale images with manually labeled 7 basic facial expression categories. This dataset consists of 35,882 face images from natural scenes, with 28,704 images in the training set, and 3,589 images each in the validation and test sets. It is also one of the most commonly used datasets for two-point facial expression recognition in natural scenes.

[0057] Throughout the entire experimental process, a total of 10 alternating iterations of training were conducted. During the first iteration, the uncertainty suppression module was not enabled, meaning that no samples were removed.

[0058] To quantitatively measure and compare the effectiveness of our methods, we use the overall accuracy r, which is most commonly used in facial expression recognition tasks in natural scenes, as the primary evaluation metric. The calculation method is as follows:

[0059] in, The total number of samples in the test set. Predict the correct number of samples for the test set.

[0060] Experiments showed that, under the same datasets, backbone networks, and evaluation metrics, this invention achieved high test accuracy on both the RAF-DB and FER2013 datasets. On the RAF-DB dataset, the method significantly outperformed most methods, only 0.56% lower than the published state-of-the-art (SOTA) method, and 1.47% higher than the baseline, reaching 88.20%. On the FER2013 dataset, under the same backbone network, the method achieved the best result of 71.08%.

[0061] Furthermore, this method also underwent noise-adding experiments on the RAF-DB dataset training set. Specifically, the labels of 10%, 20%, and 30% of the randomly selected samples from the RAF-DB training set were randomly changed to labels other than the original labels. In the 10%, 20%, and 30% noise experiments on the RAF-DB dataset, the performance of this invention significantly outperformed other methods, exceeding the published state-of-the-art (SOTA) results by 4.22%, 5.45%, and 6.31%, respectively, reaching 87.41%, 86.47%, and 85.72%, and also exceeding the baseline by 2.34%, 4.14%, and 7.3%, respectively. Moreover, the advantage of this invention over other methods increased with increasing noise. These results fully demonstrate the superiority of the method of this invention over existing methods in suppressing the uncertainty of facial expression recognition in natural scenes.

[0062] It must be noted that in any of the above embodiments, the methods are not necessarily executed sequentially according to the sequence number. As long as it cannot be inferred from the execution logic that a certain order must be followed, it means that any other possible order can be used. It should be understood that the specific embodiments described herein are only for explaining the present invention. Furthermore, the technical features involved in the various embodiments of the present invention described above can be combined with each other as long as they do not conflict with each other.

[0063] It should be noted that, since this invention only removes noisy samples from the training set and corrects the labels of some samples, it does not modify the network structure of the method like SCN and DMUE, does not increase the model parameters, and does not use a state-of-the-art backbone network for a specific dataset like RLPS, the experimental results are slightly lower than those of other methods.

[0064] Figure 6 The confusion matrix for this invention on the RAF-DB test set shows that Fearful has a relatively high probability of being incorrectly predicted as Surprised and Sad, Disgusted has a relatively high probability of being misclassified as Happy, Sad, and Neutral, and Sad also has a relatively high probability of being predicted as Neutral. The confusion rates from Neutral to Fearful, Sad to Fearful, and Angry to Sad are 0%. Among the prediction accuracy rates for each class, Happy has the highest at 95.53%, while Disgusted and Fearful have lower accuracy rates of 63.12% and 66.22%, respectively. The accuracy rates for the other four expressions are all around 85%. Figure 7The prediction accuracy and sample size for each expression were generally positively correlated. For example, "Happy" had the highest sample size and accuracy, while "Fearful" had both lower sample size and accuracy. Although "Angry" had a low sample size, its accuracy was relatively high, presumably because its features were more pronounced. "Disgusted" had the lowest prediction accuracy, even though its sample size was not the lowest, presumably because it was easily confused with other expressions.

[0065] Furthermore, the noise-adding experiment employed the same processing method as SCN and DMUE, performing noise-adding experiments on the RAF-DB dataset training set. Specifically, the labels of 10%, 20%, and 30% of the randomly selected samples from the RAF-DB dataset training set were randomly changed to labels other than the original labels. After noise addition, the accuracy of each method on the RAF-DB dataset test set is shown in Table 2. The experimental details for the baseline are the same as the previous experiment, and the results for SCN and DMUE in Table 2 are the results published by the method authors.

[0066] In experiments with 10%, 20%, and 30% noise on the RAF_DB dataset, our invention significantly outperforms other methods, achieving performance levels of 87.41%, 86.47%, and 85.72% respectively, exceeding published state-of-the-art (SOTA) results by 4.22%, 5.45%, and 6.31%, respectively, and also surpassing baseline results by 2.34%, 4.14%, and 7.3%. Furthermore, the advantages of our method increase with increasing noise levels compared to other methods.

[0067] It is worth noting that while the performance of other methods degrades rapidly with increasing noise, the degradation trend of our method is very slow. From Figure 8 It can be clearly seen that the decay trends of the Baseline, SCN and DMUE methods are basically the same, while the present invention is significantly better than the former. Figure 9 The results demonstrate the decrease in accuracy of each method under three levels of noise, compared to the absence of noise. At 10% noise, the Baseline, SCN, and DMUE decrease by 1.66%, 4.85%, and 5.57% respectively, while our invention only decreases by 0.79%. At 20% noise, the other three methods decrease by 4.40%, 6.93%, and 7.74% respectively, while our invention only decreases by 1.17%. At 30% noise, the other three methods decrease by 8.31%, 9.57%, and 9.35% respectively, while our invention only decreases by 1.99%. These results fully demonstrate the superiority of our method over existing methods in suppressing uncertainty in facial expression recognition in natural scenes.

[0068] This invention dynamically divides reliable and unreliable training sets through label confidence estimation and multi-network joint training, providing a cleaner training set for the facial expression classification network to enhance its robustness and generalization ability; label revision alleviates the problem of imbalanced class distribution and noise in the dataset, and suppresses the uncertainty problem of facial expression recognition in natural scenes.

[0069] Table 1. Accuracy comparison of the facial expression recognition method of the present invention in natural scenes.

[0070] Table 2. Accuracy comparison of the present invention when adding noise to the RAF-DB dataset.

[0071] Those skilled in the art will readily understand that the above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. An expression recognition method based on label confidence estimation and multi-network joint training, characterized by: It consists of four modules connected in sequence by electrical signals: reliable training set initialization, multi-network joint training, label revision, and final expression classification network training. The reliable training set initialization divides the original training set into an absolutely reliable training set and an unreliable training set. The multi-network joint training is used to update the reliable training set. The label revision selects some relatively reliable samples from the unreliable training set, modifies their labels, and adds them to the reliable training set. Finally, the expression classification network is trained using the reliable training set. Multi-network joint training employs an expression classification network and a confidence network, and includes the following steps: (1) Extraction of sample features and class features: Using a reliable training set The model is trained with an expression classification network, which is used to extract sample features and class features of all samples in the dataset. The sample feature extraction method is as follows: any image is input into the second convolutional neural network, and after passing through all convolutional layers and the first fully connected layer, the extracted features are used as sample features. The class feature extraction method is as follows: the sample features of all samples belonging to the same class are extracted and averaged, that is, the center of all samples belonging to the same class is obtained. The reliable training set When the situation is unknown, use an absolutely reliable training set. initialization; The facial expression classification network employs a second convolutional neural network, which uses a reliable training set. Conduct training; (2) Construction of the confidence network training set: Generate a positive sample set based on the sample features and class features described in step (1). and negative sample set ; The specific method for generating the positive and negative sample sets is as follows: For For any sample, the sample features obtained in step (1) are concatenated with the class features of the c-th class, where c is the label of the sample. All concatenation results are used as the positive sample set. ;right For any sample, use the sample features obtained in step (1) and the first sample... The class features are concatenated, where For all other classes not equal to c, all concatenation results are treated as the negative sample set. ; (3) Confidence network training: Used to train a confidence network based on the positive and negative sample sets described in step (2), and output an unreliable training set. Confidence scores for all samples in the dataset; The confidence network uses a multilayer perceptron, which consists of a six-layer fully connected network. The activation function of the first five layers is ReLU, and the activation function of the sixth layer is the sigmoid function. The confidence score of the sample is: For any sample in the network, extract sample features using the facial expression classification network trained in step (1) and concatenate them with the c-th class features, where c is the label of the sample. Input the confidence network, and the output of its last fully connected layer is the confidence score of the sample. (4) Reliable training set update: Used to update the reliable training set based on the confidence score described in step (3). ; The specific method for updating the reliable training set is as follows: The confidence score of the sample is greater than the threshold Then add it to the training set. Threshold Used to determine whether a sample should be added to the training subset; (5) Iterative training: This is used to repeat steps (1) to (4) to further update the training set. ; The stopping condition for iterative training is reaching the preset number of iterations or ceasing to feed data to the training set. Add more reliable samples.

2. The facial expression recognition method based on label confidence estimation and multi-network joint training as described in claim 1, characterized in that, Reliable training set initialization employs N-fold cross-validation and a first convolutional neural network; The specific implementation steps of the N-fold cross-validation are as follows: the original training set is evenly divided into N parts, the first part is taken as the validation set, and all other data are used as the training set; the training set is input into the first convolutional neural network for training, and then the validation set is input into the trained first convolutional neural network to obtain the prediction results of the validation set samples; Repeat the above steps N times, where the kth execution takes the kth data as the validation set, so that all samples are used as validation data once, k=2,3,...,N.

3. The facial expression recognition method based on label confidence estimation and multi-network joint training as described in claim 2, characterized in that, The backbone of the first convolutional neural network uses ResNet-18 and is pre-trained on the MS-Celeb-1M face dataset.

4. The facial expression recognition method based on label confidence estimation and multi-network joint training as described in claim 1, characterized in that, The method for dividing the absolutely reliable training set and the unreliable training set is as follows: samples whose prediction results are consistent with the true labels are added to the absolutely reliable training set. Samples whose predicted results do not match the true labels are added to the unreliable training set. .

5. The facial expression recognition method based on label confidence estimation and multi-network joint training as described in claim 1, characterized in that, Label revision is based on the sample's confidence score and a reasonable threshold. and To change the labels of samples and add them to a reliable training set. middle; Label revisions are based on the following two scenarios: when the confidence score of the instance formed by concatenating the sample features and the corresponding category features is low (i.e., < 0). Modify its label as needed; When the confidence score of an instance formed by concatenating sample features and other class features is high, that is, > Modify its label as needed; in Set to 0.1, Set as the average confidence level of samples with the same label .

6. The facial expression recognition method based on label confidence estimation and multi-network joint training as described in claim 5, characterized in that, The threshold setting is used to determine whether to modify the label of a sample.

7. The facial expression recognition method based on label confidence estimation and multi-network joint training as described in claim 2, characterized in that, The final facial expression classification network training yields a reliable training set based on the label revision steps. The third convolutional neural network is trained to obtain the final facial expression classification network; the third convolutional neural network adopts the same network structure and pre-training as the first convolutional neural network.

8. The method according to any one of claims 1 to 7, in the application of facial expression recognition.

Images