A two-stage facial expression recognition method based on curriculum learning and label smoothing

Abstract

The invention discloses a two-stage facial expression recognition method based on curriculum learning and label smoothing, comprising the following steps: firstly, processing the labels of the original data set into labels of a secondary structure, that is, each training sample comprises a coarse classification label and a fine classification label; then designing a two-stage hierarchical cascade convolutional neural network comprising a coarse classification network and a fine classification network; then introducing the idea of ​​curriculum learning into facial expression recognition, designing a sample confidence-aware pace learning module and a dynamic label smoothing module, dynamically adjusting the size of the smoothing factor according to the training progress, and gradually introducing training samples from easy to difficult; evaluating the expression recognition effect by using the trained model weight parameters and the constructed recognition network; the invention has good robustness and generalization for facial expressions comprising easily confused labels and noise labels, and both subjective and objective test evaluation results and comparative experiments prove the effectiveness and advancement of the method of the invention.

Description

Technical Field

[0001] The present invention relates to the field of human face facial expression recognition based on deep learning, and in particular to a facial expression recognition method based on curriculum learning and label smoothing. Background Art

[0002] In recent years, with the rapid development of modern science and technology and Internet technology, mankind has entered the information age. The progress of science and technology has also greatly promoted social development and facilitated people's daily lives. Especially in the field of artificial intelligence, many research results mainly in the direction of computer vision have been widely used in people's daily lives. For example, advanced technologies such as face recognition, target detection and automatic driving have built a bridge between the digital world and the real world, providing more information and intelligent services for the real world. Therefore, in the field of artificial intelligence, how to better simulate the human cognitive level, give machines human thinking ability and emotional perception ability, and improve the human-computer interaction experience is a difficult problem that researchers are constantly exploring. Facial expression recognition technology is an important task in the field of computer vision. It aims to accurately identify facial expressions such as happiness, anger, sadness, surprise, etc. from face images or videos. In human-computer interaction, emotion recognition is one of the important research directions. Machines recognize changes in facial expressions, thereby perceiving people's inner thoughts and emotions. By analyzing face images, they can effectively help people better understand human emotions. Facial expression recognition has broad application prospects in the fields of intelligent driving, medical diagnosis and psychological counseling. Therefore, it is of great practical significance to develop an efficient facial expression recognition method so that computers can efficiently and accurately recognize facial expressions in human faces and improve the ability of human-computer interaction.

[0003] In order to solve the problem that facial expression recognition tasks are easily affected by noisy labels and uncertain labels, many methods attempt to mine the potential label distribution of samples to correct the uncertainty in facial expression datasets. Reference 1 (Wang K, Peng X, Yang J, et al. Suppressing uncertainties for large-scale facial expression recognition [C] / / Proceedings of the IEEE / CVF conference on computer vision and pattern recognition. 2020: 6897-6906.) is a method to improve the performance of facial expression recognition from the perspective of suppressing uncertain noisy labels. This method suppresses uncertainty from two aspects. First, the self-attention mechanism on each batch weights each training sample and performs weight level regularization. Then, the sample label of the lowest weight group is corrected through the re-labeling mechanism to better learn facial expression features. This method has strong robustness and generalization performance for facial expression data containing noisy labels. Reference 2 (Jiahui She, Yibo Hu, Hailin Shi, Jun Wang, Qiu Shen, and Tao Mei. Dive into ambiguity: Latent distribution mining and pairwise uncertainty estimation for facial expression recognition. In Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition, pages 6248–6257, 2021.) designed an uncertainty estimation module to dynamically learn from the mined label distribution and the true sample labels. The above methods have achieved good results in uncertain label processing, but these methods have not observed the impact of noisy labels in the presence of an imbalance in the number of samples between classes and an imbalance in the difficulty level within a class.

[0004] Facial expression recognition in natural environments is particularly complex for the following reasons: In natural environments, it is often affected by the subjectivity of the annotator, the ambiguity of different expressions, and the similarity between categories, and contains a large number of samples with uncertain labels. Some expression categories are highly similar in facial muscle changes and are easily confused. In addition, the common expression recognition method based on ordinary convolutional neural networks outputs the probabilities of all categories. When there are too many expression categories and the features of some expression data are similar, some expression categories will be easily confused. Therefore, the stability of the model and the overall recognition performance are greatly affected. Summary of the invention

[0005] In order to overcome the problem of label ambiguity caused by the high similarity of facial changes in some expression categories and the problem of label ambiguity caused by uncertain labels, the present invention proposes a facial expression recognition method based on curriculum learning and label smoothing. The method first constructs a hierarchical cascade convolutional neural network, designs a coarse classification network and a fine classification network respectively, and designs an alternating training method during the training process. Subsequently, the idea of ​​curriculum learning is introduced into facial expression recognition, and a sample confidence-aware pace learning module and a dynamic label smoothing module are designed. The size of the smoothing factor is dynamically adjusted according to the training progress, and training samples from easy to difficult are gradually introduced. The proportion and difficulty of samples with uncertain labels can be dynamically adjusted. Through the above method, the influence of uncertain labels can be effectively reduced and the generalization ability of the model can be improved.

[0006] A facial expression recognition method based on curriculum learning and label smoothing comprises the following steps:

[0007] S1. First, according to the function of the coarse and fine classification network, the labels of the original data set are processed into two-level labels, that is, each sample has a coarse classification label and a fine classification label; the coarse classification label is applied to the coarse classification network, and according to the classification result of the coarse classification network, the fine classification label is applied to the corresponding fine classification network. The specific process of label reprocessing is as follows:

[0008] B1. Reprocessing of labels of coarse classification network expression dataset: The coarse classification network is a two-category network, which modifies the original dataset labels into two categories: easy to confuse and not easy to confuse. That is, the dataset labels need to be processed into a secondary structure, including primary labels and secondary labels. Anger, sadness, disgust, and fear are classified as easily confused expression categories, and happiness, surprise, and neutrality are classified as not easy to confuse expression categories; the dataset labels of the coarse classification network are processed as first-level labels, and all dataset labels are uniformly modified to 0 or 1 at the coarse classification network stage, where 0 represents the label of the not easy to confuse class, and 1 represents the label of the easily confused class;

[0009] B2. Reprocessing of the labels of the fine classification network expression dataset: The fine classification network is divided into two parts, namely, fine classification network A (not easy to confuse) and fine classification network B (easy to confuse). The two networks are responsible for classifying expressions of the not easy to confuse category and the easy to confuse category respectively. Based on the coarse labels, the expression dataset labels are further processed into a two-level structure; the dataset labels of the fine classification network will be processed as the second-level labels, similar to the coarse classification network data labels. The fine classification network data labels use numbers to represent the expression categories, and the fine labels of the four easy to confuse category labels are processed as 0 to 3, and the other three categories are processed as 0 to 2. The expression labels processed by the fine classification network are two-level labels, that is, the structure of coarse labels + fine labels.

[0010] S2, performing face cropping and alignment on the facial expression training set images in step S1, and performing data enhancement and data balancing using an image enhancement algorithm to obtain a preprocessed facial expression image training set;

[0011] S3. Construct a hierarchical cascade convolutional neural network. In the hierarchical cascade convolutional neural network structure design, the coarse classification network and the fine classification network in the network structure will share the first three layers of the convolutional neural network; the total network is divided into three parts, namely the coarse classification network, which is set as a 2-classification network, the fine classification network A (easy to confuse), which is set as a 3-classification network, and the fine classification network B (not easy to confuse), which is set as a 4-classification network. The construction process of the coarse and fine hierarchical cascade convolutional neural network is as follows:

[0012] C1. Design of coarse classification network structure: The coarse classification network is a binary classification network structure. The basic network selects ResNet-18. The network residual block contains 2 convolutional layers with the same number of output channels. Each convolutional layer is followed by a batch normalization layer and a ReLU activation function. The output of ResNet-18 is adjusted to a binary classification output, and the number of input channels is modified to 512, and the output is modified to 64. At the same time, a fully connected layer with 64 channels and 2 outputs is added.

[0013] C2. Design of fine classification network structure: The fine classification network consists of two networks. The two fine classification network structures adopt the idea of ​​transfer learning and share the first three layers of the network with the coarse classification network. The residual blocks of the fine classification networks contain two convolutional layers with the same number of output channels. Each convolutional layer is followed by a batch normalization layer and a ReLU activation function. The entire network has a parallel structure in terms of structure. Fine classification network A will classify expressions that are not easily confused and is designed as a three-classification network. Fine classification network B will classify expressions that are easily confused and is designed as a four-classification network.

[0014] S4. Construct a sample confidence-aware pace learning module, implement confidence-aware pace learning based on a predefined sample library, first use a baseline model trained with cross-entropy loss to calculate the confidence score of each sample and sort them by confidence score; design a confidence-aware pace learning strategy, adjust the hyperparameters initial sample ratio and epoch ratio η, and calculate the pace parameters based on λ and η Where E is the total number of epochs;

[0015] First, we use the self-paced learning idea in course learning. Self-paced learning takes the course design as a regularization term in the objective function. Its formal optimization objective can be expressed as:

[0016]

[0017] Where L(y i ,f(x i , w)) represents the true label y i and predicted label f(x i ,ω), w is the model parameter, υ is the weight vector with length n, and γ is the step parameter; when the step parameter γ is small, only the “easy” samples with smaller loss are introduced for training. As γ increases, the “difficult” samples with larger loss are gradually added to the training.

[0018] S5. Construct a dynamic label smoothing module to dynamically reduce the smoothing factor ∈ in uniform label smoothing (ULS). Specifically, the smoothing factor ∈ is annealed at a decay rate of α in each epoch of training, and the decay process is implemented by exponential functions and linear functions.

[0019] Dynamic label smoothing It can be expressed as:

[0020]

[0021] in, represents the label vector after label smoothing, T one-hot represents the original one-hot encoded label vector, K is the number of label categories, ∈ is the smoothing factor adjusted by the decay rate α, and α<1;

[0022] If P is the predicted probability, then the dynamic label smoothing objective The loss can be expressed as:

[0023]

[0024] S6. Use the facial expression recognition network based on curriculum learning and label smoothing constructed in steps S3, S4 and S5 to train the facial expression image training set preprocessed in step S2; during training, the face image is recognized and pre-positioned through the results output by the classification regression layer, and the face position is marked with a rectangular frame, and different face images are identified to obtain face images with marked frames and expression categories.

[0025] S7. Set the relevant hyperparameters and conventional optimization algorithms required for training the hierarchical cascade convolutional neural network, and save the weight parameter file of the network training when the loss function converges.

[0026] S8. Construct a facial expression recognition network model based on curriculum learning and label smoothing, that is, introduce the weight parameter file obtained in step S7 into the hierarchical cascade network constructed in steps S3, S4 and S5, and use the recognition model to classify the facial image to be recognized.

[0027] Beneficial effects:

[0028] 1. The present invention proposes a facial expression recognition method based on curriculum learning and label smoothing. First, the face detection algorithm is used to crop and align the facial area of ​​the face, and random rotation and random horizontal flipping are used to enhance the data. In addition, data balancing is performed during the data enhancement process to generate more enhanced images for expression categories with less original data, which can effectively reduce the impact of data imbalance on model training.

[0029] 2. The present invention then inputs the facial expression image into the sample confidence perception module, first sorts the training samples from easy to difficult, then introduces the training samples from easy to difficult into the training through the pace learning strategy, extracts facial features of different scales through coarse and fine two-level classification in the hierarchical cascade network, then smoothes the output results through the dynamic label smoothing module, and finally inputs them into the classification regression layer to obtain the prediction results. The structure of the hierarchical cascade network can better characterize the facial expression features, making the model converge better, and using the pace learning strategy and the dynamic label smoothing strategy to guide the model training can effectively reduce the influence of noise labels. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1 The overall structure diagram of the human face facial expression recognition method used in the embodiment of the present invention;

[0031] Figure 2 It is an illustration of the component modules involved in the human face facial expression recognition method used in the embodiment of the present invention; wherein, Figure a1 is a rough classification network structure diagram, Figure a2 is a hierarchical cascade convolutional neural network structure diagram, and Figure a3 is a sample confidence perception pace learning process diagram;

[0032] Figure 3 A sample diagram of a data set used in an embodiment of the present invention;

[0033] Figure 4 It is a general operation flow chart of the method for recognizing facial expressions of human faces according to an embodiment of the present invention;

[0034] Figure 5 This is a partial facial expression recognition effect diagram of an embodiment of the present invention;

[0035] Figure 6 FIG. 4 is a confusion matrix diagram of facial expression recognition results of some faces according to an embodiment of the present invention. DETAILED DESCRIPTION

[0036] In order to further describe the facial expression recognition method based on curriculum learning and label smoothing proposed by the present invention, an embodiment of the facial expression recognition image of the present invention is described in conjunction with the accompanying drawings to enhance the understanding of the purpose, method structure and application effect of the present invention. The embodiment is only a further explanation of the present invention, and researchers in this field can make further non-fundamental technical improvements on the basis of the present invention, which are also within the scope of protection of the present invention.

[0037] A facial expression recognition method based on curriculum learning and label smoothing comprises the following steps:

[0038] S1. First, according to the function of the coarse and fine classification network, the labels of the original data set are processed into two-level labels, that is, each sample has a coarse classification label and a fine classification label; the coarse classification label is applied to the coarse classification network, and according to the classification result of the coarse classification network, the fine classification label is applied to the corresponding fine classification network.

[0039] S2, performing face cropping and alignment on the facial expression training set images in step S1, and performing data enhancement and data balancing using an image enhancement algorithm to obtain a preprocessed facial expression image training set.

[0040] S3. Construct a hierarchical cascade convolutional neural network. In the hierarchical cascade convolutional neural network structure design, the coarse classification network and the fine classification network in the network structure will share the first three layers of the convolutional neural network. The total network is divided into three parts, namely the coarse classification network, the fine classification network A and the fine classification network B. The three network residual blocks all contain two convolutional layers with the same number of output channels, and each convolutional layer is followed by a batch normalization layer and a ReLU activation function; the difference is that the coarse network is set as a 2-classification network, the fine classification network is set as a 3-classification network, and the fine classification network B is a 4-classification network.

[0041] S4. Construct a sample confidence-aware pace learning module, implement confidence-aware pace learning based on a predefined sample library, first use a baseline model trained with cross-entropy loss to calculate the confidence score of each sample and sort them by confidence score; design a confidence-aware pace learning strategy, adjust the hyperparameters initial sample ratio λ and epoch ratio η, and calculate the pace parameters based on λ and η Where E is the total number of epochs.

[0042] S5. Construct a dynamic label smoothing module to dynamically reduce the smoothing factor ∈ in the uniform label smoothing ULS, and anneal the smoothing factor ∈ at a decay rate of α in each epoch of training, where α<1.

[0043] S6. Use the facial expression recognition network based on curriculum learning and label smoothing constructed in steps S3, S4 and S5 to train the training set of facial expression images preprocessed in step S2; during training, the face images are recognized and pre-positioned through the results output by the classification regression layer, and the face positions are marked with rectangular frames, and different face images are identified to obtain face images with marked frames and expression categories.

[0044] S7. Set the relevant hyperparameters and conventional optimization algorithms required for training the hierarchical cascade convolutional neural network, and save the weight parameter file of the network training when the loss function converges.

[0045] S8. Construct a facial expression recognition network model based on curriculum learning and label smoothing, that is, introduce the weight parameter file obtained in step S7 into the hierarchical cascade network constructed in steps S3, S4 and S5, and use the recognition model to classify the facial image to be detected.

[0046] As a specific implementation of the present invention, the step 1 processes the original data set labels into secondary labels, that is, each sample has a coarse classification label and a fine classification label, specifically:

[0047] B1. Reprocessing of labels of coarse classification network expression dataset: The coarse classification network is a two-classification network, which modifies the labels of the original dataset into two categories: easy to confuse and not easy to confuse. Angry, sad, disgusted and afraid are classified as easy to confuse expression categories, and happy, surprised and neutral are classified as not easy to confuse expression categories. The dataset labels of the coarse classification network are processed as first-level labels. In the coarse classification network stage, all dataset labels are uniformly modified to 0 or 1, where 0 represents the label of the not easy to confuse class and 1 represents the label of the easy to confuse class.

[0048] B2. Reprocessing of labels of fine-classification network expression dataset: The fine-classification network is divided into two parts, namely fine-classification network A and fine-classification network B. The two networks are responsible for classifying expressions that are not easily confused and easily confused, respectively. Based on the coarse labels, the expression dataset labels are further processed into a two-level structure; the dataset labels of the fine-classification network will be processed into second-level labels, which are similar to the coarse classification network data labels. The fine-classification network data labels use numbers to represent expression categories. The fine labels of the secondary labels of the four easily confused categories are processed from 0 to 3, and the secondary labels of the other three not easily confused categories are processed from 0 to 2. The expression labels processed by the fine-classification network are two-level labels, that is, the structure of coarse labels + fine labels;

[0049] As a specific implementation of the present invention, the hierarchical cascade convolutional neural network is constructed in step S3, and the specific process is as follows:

[0050] C1. Design of coarse classification network structure: The coarse classification network is a binary classification network structure. The basic network is ResNet-18. The network residual block contains two convolutional layers with the same number of output channels. Each convolutional layer is followed by a batch normalization layer and a ReLU activation function. The output of ResNet-18 is adjusted to a binary classification output, and the number of input channels is changed to 512, the number of output channels is changed to 64, and a fully connected layer with 64 channels and 2 outputs is added at the end.

[0051] C2. Fine classification network structure design: The fine classification network consists of two networks. The two fine classification network structures adopt the idea of ​​transfer learning and share the first few layers of the network with the coarse classification network. The residual blocks of the fine classification network contain two convolutional layers with the same number of output channels. Each convolutional layer is followed by a batch normalization layer and a ReLU activation function. The entire network presents a parallel structure. Fine classification network A will classify expressions that are not easily confused and is designed as a three-classification network. Fine classification network B will classify expressions that are easily confused and is designed as a four-classification network.

[0052] As a specific implementation of the present invention, the sample confidence-aware pace learning module in step S4 has the following construction process and principle:

[0053] First, we use the self-paced learning idea in course learning. Self-paced learning takes the course design as a regularization term in the objective function. Its formal optimization objective can be expressed as:

[0054]

[0055] Where L(y i ,f(x i , w)) represents the true label y i and predicted label f(xi , w), where w is the model parameter, υ is the weight vector of length n, and γ is the step parameter. When the step parameter γ is small, only the "easy" samples with small loss are introduced for training. As γ increases, the "difficult" samples with large loss are gradually added to the training.

[0056] As a specific implementation of the present invention, the construction process and principle of the dynamic label smoothing module described in step S5 are as follows:

[0057] Dynamically reduce the smoothing factor ∈ in uniform label smoothing ULS, anneal the smoothing factor ∈ with a decay rate of α at each epoch of training, where α<1, dynamic label smoothing It can be expressed as:

[0058]

[0059] Among them, ∈ is the smoothing factor adjusted by the smoothing decay rate α.

[0060] If P is the predicted probability, then the dynamic label smoothing objective The loss can be expressed as:

[0061]

[0062] Simulation experiment:

[0063] In order to verify the effectiveness and generalization of the proposed method, experiments and evaluations are conducted on a public natural facial expression dataset. Figure 2 (b1) shows a brief introduction to the two facial expression datasets used in some experiments.

[0064] 1. FER2013: The FER2013 dataset is a facial expression dataset provided by Kaggle. The images are collected from the Google image search engine and contain facial expression images of people of different ages and postures. The dataset consists of 7 basic expressions, including 28,709 images in the training set, 3,589 images in the public test set and 3,589 images in the private test set. In the experiment, the public test set is used as the validation set, and the private test set is used as the test set.

[0065] 2.RAF-DB:RAF-DB dataset is a real-world facial expression dataset collected from the Internet. The dataset contains a single-label subset and a double-label subset. The single-label subset contains 7 basic expressions, and the double-label subset contains 12 compound expressions. The dataset also contains images from different ages, postures, and occlusions. Only 7 categories of basic expressions, a total of 15,339 images, are used as the experimental dataset, including 12,271 training set images and 3,068 test set images.

[0066] 3. In order to study the effectiveness of the proposed method, experimental comparisons were conducted with advanced methods. Figure 5 The effect diagram of facial expression recognition of the method of the present invention is given in FIG. Figure 6 The confusion matrix on the datasets FER2013 and RAF-DB is given. The comparison of the accuracy results of facial expression recognition is shown in Table 1.

[0067] Table 1 Comparison of facial expression recognition accuracy in simulation experiments

[0068]

[0069]

[0070] from Figure 5 It can be seen from Table 1 that the method of the present invention can effectively recognize facial expressions of human faces, and performs well in facial expression recognition with confusing labels and noise labels.

[0071] The above simulation experiment results show that the present invention can efficiently detect the face image in the image and perform facial expression recognition while ensuring the stability of facial expression recognition, which shows the effectiveness of the method of the present invention.

[0072] The above is an explanation of the method proposed in the present invention, and researchers in the field can reproduce the present invention through these explanations. Based on the content of the present invention, other embodiments and results obtained by researchers without creative work should all fall within the scope of protection of the present invention.

Claims

1. A two-stage facial expression recognition method based on curriculum learning and label smoothing, characterized in that: The following steps are involved: S1. First, according to the function of the coarse and fine classification network, the original data set labels are processed into two-level labels, that is, each sample has a coarse classification label and a fine classification label. The coarse classification label is applied to the coarse classification network. According to the classification result of the coarse classification network, the fine classification label is then applied to the corresponding fine classification network. S2, performing face cropping and alignment on the facial expression training set images in step S1, and performing data enhancement and data balancing using an image enhancement algorithm to obtain a preprocessed facial expression image training set; S3. Construct a two-stage hierarchical cascade convolutional neural network. In the hierarchical cascade convolutional neural network structure design, the coarse classification network and the fine classification network in the network structure will share the first three layers of the convolutional neural network; the total network is divided into three parts, namely the coarse classification network, the fine classification network A and the fine classification network B; the three network residual blocks all contain two convolutional layers with the same number of output channels, each convolutional layer is followed by a batch normalization layer and a ReLU activation function. The difference is that the coarse network is set as a 2-classification network, the fine classification network is set as a 3-classification network, and the fine classification network B is a 4-classification network; S4. Construct a sample confidence-aware pace learning module, implement confidence-aware pace learning based on a predefined sample library, first use a baseline model trained with cross-entropy loss to calculate the confidence score of each sample and sort them by confidence score; design a confidence-aware pace learning strategy, adjust the hyperparameters initial sample ratio λ and epoch ratio η, and calculate the pace parameters based on λ and η Where E is the total number of epochs; S5. Construct a dynamic label smoothing module to dynamically reduce the smoothing factor ∈ in the uniform label smoothing ULS, and anneal the smoothing factor ∈ at a decay rate of α in each epoch of training, where α<1; S6, using the two-stage facial expression recognition network based on curriculum learning and label smoothing constructed in steps S3, S4 and S5 to train the facial expression image training set preprocessed in step S2; during training, the face image is recognized and pre-positioned through the results output by the classification regression layer, and the face position is marked with a rectangular frame, and different face images are marked to obtain a face image with a marked frame and an expression category; S7. Set the relevant hyperparameters and conventional optimization algorithms required for training the hierarchical cascade convolutional neural network, and save the weight parameter file of the network training when the loss function converges; S8. Construct a two-stage facial expression recognition network model based on curriculum learning and label smoothing, that is, introduce the weight parameter file obtained in step S7 into the hierarchical cascade network constructed in steps S3, S4 and S5, and use the recognition model to classify the facial image to be detected.

2. A two-stage facial expression recognition method based on curriculum learning and label smoothing according to claim 1, characterized in that: The step S1 processes the original dataset labels into secondary labels, that is, each sample has a coarse classification label and a fine classification label, specifically: B1. Reprocessing of labels of coarse classification network expression dataset: The coarse classification network is a two-classification network. The labels of the original dataset are modified into two categories: easy to confuse and not easy to confuse. Anger, sadness, disgust and fear are classified as easy to confuse expression categories, and happiness, surprise and neutrality are classified as not easy to confuse expression categories. The dataset labels of the coarse classification network are processed as first-level labels. In the coarse classification network stage, all dataset labels are uniformly modified to 0 or 1, where 0 represents the label of the not easy to confuse class and 1 represents the label of the easy to confuse class. B2. Reprocessing of the labels of the fine-classification network expression dataset: Based on the coarse labels, the expression dataset labels are further processed into a two-level structure; the dataset labels of the fine-classification network will be processed as second-level labels, similar to the coarse classification network data labels. The fine-classification network data labels use numbers to represent expression categories, and the fine labels of the four easily confused categories of secondary labels are processed as 0 to 3, and the other three categories of secondary labels are processed as 0 to 2. The expression labels after processing by the fine-classification network are two-level labels, that is, the structure of coarse labels + fine labels.

3. A two-stage facial expression recognition method based on curriculum learning and label smoothing according to claim 1, characterized in that: Step S3 The construction process of the coarse-grained and fine-scale cascaded convolutional neural network is as follows: C1. Design of coarse classification network structure: The coarse classification network is a binary classification network structure. The basic network is ResNet-18. The network residual block contains two convolutional layers with the same number of output channels. Each convolutional layer is followed by a batch normalization layer and a ReLU activation function. The output of ResNet-18 is adjusted to a binary classification output, and the number of input channels is changed to 512, and the number of output channels is changed to 64. At the same time, a fully connected layer with 64 channels and 2 outputs is added at the end. C2. Fine classification network structure design: The fine classification network contains two networks. The two fine classification network structures adopt the idea of ​​transfer learning and share the first three layers of the network with the coarse classification network. The residual blocks of the fine classification networks contain two convolutional layers with the same number of output channels. Each convolutional layer is followed by a batch normalization layer and a ReLU activation function. The entire network has a parallel structure from a structural point of view. Fine classification network A will classify expressions that are not easily confused and is designed as a 3-classification network. Fine classification network B will classify expressions that are easily confused and is designed as a 4-classification network.

4. A two-stage facial expression recognition method based on curriculum learning and label smoothing according to claim 1, characterized in that: The construction process of the sample confidence-aware dynamic label smoothing module in step S4 is as follows: First, we use the self-paced learning idea in course learning. Self-paced learning takes the course design as a regularization term in the objective function. Its formal optimization objective can be expressed as: Where L(y i ,f(x i ,w)) represents the true label y i and predicted label f(x i ,w), where w is the model parameter, v is the weight vector of length n, and γ is the step parameter; when the step parameter γ is small, only "easy" samples with smaller loss are introduced for training. As γ increases, "difficult" samples with larger loss are gradually added to the training.

5. A two-stage facial expression recognition method based on curriculum learning and label smoothing according to claim 1, characterized in that: The construction process of the dynamic label smoothing module DLS in step S5 is as follows: Dynamically reduce the smoothing factor ∈ in uniform label smoothing ULS. Specifically, the smoothing factor ∈ is annealed with a decay rate of α at each epoch of training. Dynamic label smoothing It can be expressed as: Among them, ∈ is the smoothing factor adjusted by the smoothing decay rate α; If P is the predicted probability, then the dynamic label smoothing objective The loss can be expressed as:

Images