A facial expression recognition method integrating uncertainty estimation and active learning

Abstract

The present invention relates to the technical field of deep learning and image processing, and discloses an expression recognition method integrating uncertainty estimation and active learning. The expression recognition method integrating uncertainty estimation and active learning comprises the following steps: 1. comparative self-supervised pre-training; 2. dynamic re-labeling module DRM (Dynamic Re-labeling Module) integrating uncertainty estimation; 3. model fine-tuning based on pre-trained model. The expression recognition method integrating uncertainty estimation and active learning provided by the present invention first utilizes comparative learning based on the underlying feature representation of the overall unlabeled data set, and further enhances the feature extraction capability of the model through an improved uncertainty estimation method; subsequently, active learning is used to fine-tune the model based on the pre-trained model to more efficiently select and label the most informative samples, and the result achieves a significant improvement in recognition accuracy compared to the prior art.

Description

Technical Field

[0001] The present invention relates to the technical field of deep learning and image processing, and in particular to an expression recognition method integrating uncertainty estimation and active learning. Background Art

[0002] It has important applications in the fields of computer vision and human-computer interaction. Traditional expression recognition methods mainly rely on manual feature extraction and machine learning algorithms, which often perform poorly when dealing with large-scale data and complex scenes. With the development of deep learning technology, methods based on deep neural networks have made significant progress in expression recognition tasks. However, the training of deep learning models usually requires a large amount of labeled data, and the data labeling process is time-consuming, labor-intensive and expensive.

[0003] In addition, deep learning models are often highly sensitive to the distribution and quality of training data during training and inference. In particular, in the presence of noisy labels or uneven data distribution, the performance of the model may drop significantly. To address these issues, uncertainty estimation and active learning techniques are introduced. Uncertainty estimation can help identify samples that are difficult to classify or may be misclassified by quantifying the model's confidence in the prediction of each sample. Active learning techniques improve model performance while reducing the cost of labeling by intelligently selecting the most valuable samples for labeling.

[0004] However, there is still little research in the prior art on how to effectively combine these two methods to achieve optimal performance. Therefore, a new expression recognition method is needed that organically combines uncertainty estimation with active learning technology to improve the accuracy and robustness of expression recognition. Summary of the invention

[0005] In order to address the deficiencies of the above-mentioned prior art, the present invention proposes an expression recognition method that integrates uncertainty estimation and active learning to reduce annotation costs, improve model performance and enhance the generalization ability of unseen expressions.

[0006] The expression recognition method integrating uncertainty estimation and active learning provided by the present invention comprises the following steps:

[0007] Step 1: Build the first stage: Compare the pre-training learning framework;

[0008] Step 2: Build SimCLR (Simple Framework for Contrastive Learning of Visual Representations): Contrast the pre-trained expression recognition network, which is composed of a backbone network, a global average pooling layer, a projection head, and a dynamic re-labeling module;

[0009] Step 3: Construct a re-labeling module based on uncertainty estimation, which mainly includes self-attention weight module, logarithmic weighted cross entropy loss, regularized sorting and dynamic threshold re-labeling;

[0010] Step 4: Active learning.

[0011] Preferably, the step 1 comprises the sub-steps of:

[0012] Step 1.1, data sampling: select a data set and obtain a face image data set X = {X1, X2, ···, X k ···X K}, X k represents the kth face image, and saves the label value corresponding to its face separately as y={y1,y2,···,y k ···y K},y k Indicates the label value corresponding to the k-th face image;

[0013] Step 1.2: Resize each face in the dataset X, normalize the pixel values, and then apply random transformations (rotation, cropping, color adjustment, etc.) to obtain the data augmented image X' = {X'1, X'2, ···, X' k ···X' K},X”={X”1,X”2,···,X” k ···X” K}, where X' k and X” k They represent the random data augmented views of the k-th sample respectively, and these two views are regarded as positive sample pairs.

[0014] Preferably, the step 2 comprises the sub-steps of:

[0015] Step 2.1, the backbone network is a ResNet18 network, and includes a first convolution block, a second convolution block, a third convolution block, a fourth convolution block, a fifth convolution block, and a sixth fully connected block in sequence;

[0016] The convolution kernel of the first convolution block is m1×m2, the second convolution block consists of a residual block, the third convolution block consists of a residual block; the fourth convolution block consists of two residual blocks; the fifth convolution block consists of two residual blocks; the sixth fully connected block contains global average pooling and a fully connected layer; each residual block contains two 3×3 convolution kernels;

[0017] Step 2.2: Input X', X" into the backbone network for processing, and after being processed by five convolution blocks in sequence, output the kth enhanced graphic feature map F'. k , F”k , the global average pooling layer compresses the spatial dimension of each feature map to a single value, thereby obtaining two fixed-length feature vectors z i ,z j ;

[0018] Step 2.3: Compare the loss and train the training to calculate the positive sample pair z i ,z j cosine similarity; select negative samples from the current batch; the feature vector of each view is a negative sample of the feature vectors of other views; use formula (1) to maximize the similarity of each positive sample pair and minimize its similarity with all negative sample pairs.

[0019]

[0020] Formula (1) i ,z j where are the embedding representations of sample i and positive sample p, respectively, z a is the embedding representation of negative samples, τ is the temperature parameter, 1 [k≠i] is an indicator function, which is 1 when k≠i and 0 otherwise; P(i)={p∈{1,...,N}|y p =y i ,p≠i} is the positive sample set.

[0021] Preferably, the step 3 comprises the sub-steps of:

[0022] Step 3.1. Calculate the self-attention weight: F” k After input into the Softmax classifier for back propagation, the feature weights of the fully connected FC layer are extracted And perform feature weighting with sample X, and obtain the feature weight corresponding to each image sample through formula (2);

[0023]

[0024] In formula (2), α i is the feature weight of the i-th sample, is the weight coefficient of the fully connected layer, σ is the Sigmod function;

[0025] Step 3.2: Calculate the log-weighted cross entropy loss L using formula (3) WCE

[0026]

[0027] Where W j is the weight of the jth classifier;

[0028] Step 3.3, regularized sorting: First, sort the attention weights in descending order, and then divide them into high standard deviation attention weight groups a s and low standard deviation attention weights group a u two groups; then use rank regularization of formulas (4) and (5) to ensure that the standard deviation attention weight of the high-importance group is slightly higher than the standard deviation attention weight of the low-importance group;

[0029] L RR =max{0,δ1-(α s -α u )}(4)

[0030] In formula (4), δ1 is the marginal value that controls the proportion of the total number of low-importance groups. During training, the comprehensive loss function is defined as:

[0031] L all =γL RR +(1-γ)L WCE (5)

[0032] Formula (5), where γ is the trade-off ratio;

[0033] Step 3.4, Dynamic Threshold Relabeling: Uncertain samples usually show lower importance weights, i.e., a u , the re-labeling module will u The sample data in the Softmax probability method is used to evaluate samples with lower importance weights; the highest predicted probability and the probability assigned to the original label are compared using formulas (6) and (7); when the former exceeds the latter, the sample is reassigned a new pseudo label; the operation of the dynamic threshold relabeling process is as follows:

[0034]

[0035] In formula (6), δ2 is the threshold, P max is the value with the largest predicted probability, P gtInd Represents the probability of a given label value; l max and l org are the index of the original given label and the index of the maximum predicted value, respectively;

[0036] In formula (7), increase and decrease represent the magnitude of each adjustment;

[0037] Step 3.5: Based on the overall loss function L all The contrast supervised pre-trained network is trained. When the number of training iterations reaches the set number or the overall loss function L all When convergence occurs, the training stops, and the optimal facial expression feature extraction network model M is obtained. p, used for subsequent active learning.

[0038] Preferably, step 4 comprises the sub-steps of:

[0039] Step 4.1: Divide the face dataset X into a small-scale labeled dataset

[0040] and a large unlabeled dataset D L The last position initial_points in the data set is the starting point for distinguishing whether there is labeled data or not;

[0041] Step 4.2, minimum confidence method:

[0042] Set the number of learning rounds to 7. In each round of active learning, the model M p From the unlabeled dataset D U Select the batch size of the sample in U Each sample in All feature values ​​are respectively classified by Softmax classifier Determine the expression category and get D U Corresponding expression category prediction probability According to the highest confidence level P max =max(P U ) Sort all unlabeled samples from small to large, and select the sample with the highest confidence and the lowest ranking as the sample selected for active learning, and record it as in represents the selected sample with the smallest k-th highest confidence in the batch; the newly labeled data D new Add to the labeled dataset D L In the unlabeled dataset D U Remove these samples from the

[0043] Step 4.3: Use the updated labeled dataset D L Retrain or fine-tune model M P , generate a new model M P+1 ; Evaluate the performance of the model on the validation set until the budget is reached or 7 epochs are completed.

[0044] Compared with related technologies, the expression recognition method integrating uncertainty estimation and active learning provided by the present invention has the following beneficial effects:

[0045] 1. Improve labeling efficiency: By combining uncertainty estimation with active learning strategies, the most informative samples are dynamically selected for labeling, effectively reducing the amount of labeled data required and reducing the cost and time of data labeling.

[0046] 2. Improve model performance: Use uncertainty estimation methods to identify samples that are difficult for the model to classify or may be misclassified, and use active learning strategies to focus on training these samples, significantly improving the recognition accuracy and robustness of the model.

[0047] 3. Enhance the ability to generalize to unseen expressions: During multiple rounds of cyclic training and model updating, the model continuously adapts to newly added samples, improves the ability to generalize to unseen expressions, and makes it more robust in practical applications.

[0048] 4. Adapt to data imbalance and noisy labels: Even when the data set is imbalanced or contains noisy labels, the method can still effectively improve the model performance through uncertainty estimation and active learning strategies, and reduce the performance degradation caused by data imbalance or noisy labels.

[0049] 5. Reduce the risk of overfitting: Active learning strategies give priority to samples with high uncertainty for labeling and training, avoiding overtraining of known samples, thereby reducing the risk of overfitting and improving the generalization ability of the model. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] Figure 1 A diagram of a comparative learning pre-training framework in the expression recognition method integrating uncertainty estimation and active learning proposed in the present invention;

[0051] Figure 2 The dynamic label re-marking structure diagram of the present invention;

[0052] Figure 3 This is a structural diagram of the active learning part in the method of the present invention. DETAILED DESCRIPTION

[0053] The present invention will be further described below in conjunction with the accompanying drawings and implementation modes.

[0054] The embodiment of the present invention proposes a method for facial expression recognition that integrates uncertainty estimation and active learning, wherein the first stage pre-training part is as follows: Figure 1 As shown, the following steps are followed:

[0055] Step 1: Build the first stage: Compare the pre-training learning framework;

[0056] Step 1.1, data sampling: select a data set and obtain a face image data set X = {X1, X2, ···, X k ···X K}, X k represents the kth face image, and saves the label value corresponding to its face separately as y={y1,y2,···,y k ···y K},y k Represents the label value corresponding to the k-th face image.

[0057] Step 1.2: Resize each face in the dataset X, normalize the pixel values, and then apply random transformations (rotation, cropping, color adjustment, etc.) to obtain the data augmented image X' = {X'1, X'2, ···, X' k ···X' K},X”={X”1,X”2,···,X” k ···X” K}, where X' k and X” k They represent random data augmented views of the kth sample, and these two views are used as positive sample pairs; in this example, the training data is augmented using random cropping and random horizontal flipping.

[0058] Step 2: Build SimCLR (Simple Framework for Contrastive Learning of Visual Representations): The pre-trained expression recognition network is composed of a backbone network, a global average pooling layer, a projection head, and a dynamic re-labeling module:

[0059] Step 2.1, the backbone network is a ResNet18 network, and includes a first convolution block, a second convolution block, a third convolution block, a fourth convolution block, a fifth convolution block, and a sixth fully connected block in sequence;

[0060] The convolution kernel of the first convolution block is m1×m2, the second convolution block consists of a residual block, the third convolution block consists of a residual block; the fourth convolution block consists of two residual blocks; the fifth convolution block consists of two residual blocks; the sixth fully connected block contains global average pooling and a fully connected layer; each residual block contains two 3×3 convolution kernels;

[0061] Step 2.2: Input X', X" into the backbone network for processing, and after being processed by five convolution blocks in sequence, output the kth enhanced graphic feature map F'. k , F” k , the global average pooling layer compresses the spatial dimension of each feature map to a single value, thereby obtaining two fixed-length feature vectors z i ,z j In this example, the enhanced image is input into a 224x224x3 face image, and after passing through a convolutional layer and four residual blocks, a 7x7x512 feature is obtained, which is then downsampled through a global average pooling layer. Finally, the model outputs a feature map of size 1x1x512.

[0062] Step 2.3: Compare the loss and train the training to calculate the positive sample pair z i ,z j The cosine similarity of . Select negative samples from the current batch. The feature vector of each view is a negative sample of the feature vector of other views. Using formula (1), for each positive sample pair, maximize its similarity and minimize its similarity with all negative sample pairs.

[0063]

[0064] In formula (1), z i ,z j are the embedding representations of sample i and positive sample p, respectively, a is the embedding representation of negative samples, τ is the temperature parameter, 1 [k≠i] is an indicator function, which is 1 when k≠i and 0 otherwise. P(i)={p∈{1,...,N}|y p =y i ,p≠i} is the positive sample set.

[0065] Step 3: Construct a re-labeling module based on uncertainty estimation. Its main structure is as follows: Figure 2 As shown in the figure, it mainly includes self-attention weight module, logarithmic weighted cross entropy loss, regularized sorting and dynamic threshold re-labeling.

[0066] Step 3.1. Calculate the self-attention weight: F” k After input into the Softmax classifier for back propagation, the feature weights of the fully connected FC layer are extracted And perform feature weighting with sample X, and obtain the feature weight corresponding to each image sample through formula (2).

[0067]

[0068] In formula (2), α i is the feature weight of the i-th sample, is the weight coefficient of the fully connected layer, and σ is the Sigmod function.

[0069] Step 3.2: Calculate the log-weighted cross entropy loss L using formula (3) WCE

[0070]

[0071] Where W j is the weight of the jth classifier. Establishing the log-weighted cross entropy loss can accurately reflect the process of weighting the loss using the attention weights.

[0072] Step 3.3, regularized sorting: First, sort the attention weights in descending order, and then divide them into high standard deviation attention weight groups a s and low standard deviation attention weights group a u We then use rank regularization in formulas (4) and (5) to ensure that the standard deviation attention weight of the high-importance group is slightly higher than the standard deviation attention weight of the low-importance group; in this example, the high-to-low ratio is 8:2.

[0073] L RR =max{0,δ1-(α s -α u )}(4)

[0074] In formula (4), δ1 is the marginal value that controls the proportion of the total number of low-importance groups. During training, the comprehensive loss function is defined as:

[0075] L all =γL RR +(1-γ)L WCE (5)

[0076] Formula (5), where γ is the trade-off ratio

[0077] In this example, δ1 is set to 0.07.

[0078] Step 3.4, Dynamic Threshold Relabeling: Uncertain samples usually show lower importance weights, i.e., a u , the re-labeling module will u The sample data in is evaluated using the Softmax probability method to evaluate samples with lower importance weights. Comparisons are made between the highest predicted probability and the probability assigned to the original label using formulas (6) and (7). When the former exceeds the latter, the sample is reassigned a new pseudo-label. The operation of the dynamic threshold relabeling process is as follows:

[0079]

[0080] In formula (6), δ2 is the threshold, P max is the value with the largest predicted probability, P gtInd Represents the probability of a given label value. max and l org are the index of the original given label and the index of the maximum predicted value, respectively.

[0081] In formula (7), increase and decrease represent the magnitude of each adjustment.

[0082] In this example, δ2 is set to 0.2, and increase and decrease are set to 0.05.

[0083] Step 3.5: Based on the overall loss function L all The contrast supervised pre-trained network is trained. When the number of training iterations reaches the set number or the overall loss function L all When convergence occurs, the training stops, and the optimal facial expression feature extraction network model M is obtained. p , used for subsequent active learning. In this example, batch_size is set to 1024.

[0084] Step 4: Active learning, its main structure is as follows Figure 3 As shown:

[0085] Step 4.1: Divide the face dataset X into a small-scale labeled dataset

[0086] and a large unlabeled dataset D L The last position initial_points in the data set is the starting point for distinguishing whether there is labeled data or not. In this example, initial_points is set to 10% of the data set.

[0087] Step 4.2, minimum confidence method:

[0088] Set the number of learning rounds to 7. In each round of active learning, the model M p From the unlabeled dataset D U Select the batch size of the sample in U Each sample in All feature values ​​are respectively classified by Softmax classifier Determine the expression category and get D U Corresponding expression category prediction probability According to the highest confidence level P max =max(P U ) Sort all unlabeled samples from small to large, and select the sample with the highest confidence and the lowest ranking as the sample selected for active learning, and record it as in represents the selected sample with the smallest k-th highest confidence in the batch. new Add to the labeled dataset D L In the unlabeled dataset D UThese samples are removed. In this embodiment, the model output feature map is input into the global average pooling layer AavgPool, and finally the activation function softmax gives the predicted category probability value. Active learning can be used to select those facial expression samples with the highest current uncertainty of the model for further annotation and training. In this way, the model can handle complex and uncertain samples more effectively, improving the performance and robustness of the overall model.

[0089] Step 4.3: Use the updated labeled dataset D L Retrain or fine-tune model M P , generate a new model M P+1 The performance of the model on the validation set is evaluated until the budget is reached or 7 rounds are completed. In this embodiment, the Adam optimizer is used, the initial learning rate is 1e-3, and the decay method is to decay to 0.1 times the original every 100 times. After 500 training iterations, the trained model is obtained.

[0090] In order to verify the effectiveness of the method of the present invention, the existing active learning methods on FER13, RAF-DB, and KDEF were selected for comparative verification. Table 1 shows the comparison of expression recognition accuracy with various active learning methods.

[0091]

[0092] Table 1 Comparison with other algorithms

[0093] Compared with related technologies, the expression recognition method integrating uncertainty estimation and active learning provided by the present invention has the following beneficial effects:

[0094] The present invention is divided into two main stages: first, the model is self-supervised pre-trained through contrastive learning and uncertainty estimation to enhance feature extraction capabilities; then, the model is fine-tuned using the knowledge gained in the pre-training stage. This method makes full use of limited data labels, helps alleviate the dependence on large amounts of labeled data and the cold start problem that exists in active learning of small data sets, thereby significantly improving recognition accuracy.

[0095] The above descriptions are merely embodiments of the present invention and are not intended to limit the patent scope of the present invention. Any equivalent structure or equivalent process transformation made using the contents of the present invention specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present invention.

Claims

1. A facial expression recognition method integrating uncertainty estimation and active learning, characterized in that: The steps include: Step 1: Construct the first stage: compare the pre-training learning framework. Step 1 includes sub-steps: Step 1.1, data sampling: select a data set and obtain a face image data set by randomly sampling face images , Indicates Individual face images, and save the label values ​​corresponding to their faces separately as , Indicates The label value corresponding to the individual face image; Step 1.2: For each data set Each face in the image is resized, pixel values ​​are normalized, and then random transformations are applied to obtain data augmented images. , ,in and Respectively represent random data augmentation views of samples and take these two views as positive sample pairs; Step 2: Build SimCLR: The pre-trained expression recognition network is composed of a backbone network, a global average pooling layer, a projection head, and a dynamic re-labeling module. Step 2 includes the following sub-steps: Step 2.1, the backbone network is a ResNet18 network, and includes a first convolution block, a second convolution block, a third convolution block, a fourth convolution block, a fifth convolution block, and a sixth fully connected block in sequence; The convolution kernel of the first convolution block is , the second convolution block consists of a residual block, the third convolution block consists of a residual block; the fourth convolution block consists of two residual blocks; the fifth convolution block consists of two residual blocks; the sixth fully connected block contains global average pooling and a fully connected layer; each residual block contains two The convolution kernel of Step 2.2: , are respectively input into the backbone network for processing, and after being processed by five convolution blocks in sequence, the first Zhang enhanced graph feature map , , the global average pooling layer compresses the spatial dimension of each feature map to a single value, resulting in two fixed-length feature vectors ; Step 2.3: Compare the loss and train the training to calculate the positive sample pairs cosine similarity; select negative samples from the current batch; the feature vector of each view is a negative sample of the feature vector of other views; Using formula (1), for each positive sample pair, maximize its similarity and minimize its similarity with all negative sample pairs. (1) Formula (1) The samples are and positive samples The embedding representation of is the embedding representation of negative samples, is the temperature parameter, is an indicator function, when 1 when it is, otherwise 0; is the positive sample set; Step 3: Construct a re-labeling module based on uncertainty estimation, which mainly includes a self-attention weight module, a logarithmic weighted cross entropy loss, regularized sorting, and a dynamic threshold re-labeling. Step 3 includes the following sub-steps: Step 3.1, calculate the self-attention weight: After back propagation into the Softmax classifier, the feature weights of the fully connected FC layer are extracted , and with the sample Perform feature weighting and obtain the feature weight corresponding to each image sample through formula (2); (2) In formula (2), It is The feature weight of each sample, is the weight coefficient of the fully connected layer, is the Sigmod function; Step 3.2: Calculate the log-weighted cross entropy loss using formula (3) (3) in is the weight of the jth classifier; Step 3.3, regularized sorting: First, sort the attention weights in descending order, and then divide them into high standard deviation attention weight groups proportionally and low standard deviation attention weights two groups; then use rank regularization of formulas (4) and (5) to ensure that the standard deviation attention weight of the high-importance group is slightly higher than the standard deviation attention weight of the low-importance group; (4) In formula (4), In order to control the marginal value of the proportion of the total number of low-importance groups, during training, the comprehensive loss function is defined as: (5) Formula (5), where γ is the trade-off ratio; Step 3.4, Dynamic Threshold Relabeling: Uncertain samples usually show lower importance weights, i.e. , the re-labeling module will The sample data in the Softmax probability method is used to evaluate samples with lower importance weights; the highest predicted probability and the probability assigned to the original label are compared using formulas (6) and (7); when the former exceeds the latter, the sample is reassigned a new pseudo-label; the operation of the dynamic threshold relabeling process is as follows: (6) （7） In formula (6), is the threshold value, is the value with the largest predicted probability, Represents the probability of a given label value; and are the index of the original given label and the index of the maximum predicted value, respectively; In formula (7), increase and decrease Indicates the magnitude of each adjustment; Step 3.5: Based on the overall loss function Train the contrast supervised pre-trained network. When the number of training iterations reaches the set number or the overall loss function When convergence occurs, the training stops, thus obtaining the optimal facial expression feature extraction network model. , used for subsequent active learning; Step 4: Active learning, wherein step 4 includes sub-steps: Step 4.1: Face dataset Divide into a small-scale labeled dataset , and a large unlabeled dataset ; The last position in the dataset initial_points To distinguish the starting point with or without labeled data; Step 4.2, minimum confidence method: Set the number of learning rounds to 7. In each round of active learning, the model From unlabeled datasets Select the batch size of the samples in Each sample in All feature values ​​are respectively classified by Softmax classifier Determine the expression category and get Corresponding expression category prediction probability , according to the highest confidence Sort all unlabeled samples from small to large, and select the sample with the highest confidence and the lowest ranking as the sample selected for active learning, and record it as ,in Indicates The selected samples with the smallest highest confidence in the batch; the newly labeled data Add to labeled dataset In the dataset, no annotations have been found. Remove these samples from the Step 4.3: Use the updated labeled dataset Retrain or fine-tune the model , generate a new model ; Evaluate the performance of the model on the validation set until the budget is reached or 7 epochs are completed.

Images