A pseudo-label based line of sight estimation method

Abstract

The present application relates to a kind of pseudo-label-based line-of-sight estimation method, comprising the following steps: obtaining face image;The face image is input into trained line-of-sight evaluation model to predict and obtain predicted line-of-sight;Wherein, the line-of-sight evaluation model is trained when training, a small amount of labeled real training set is used to train the model, and pre-training model is obtained, pseudo-label is stamped on unlabeled data according to the pre-training model with increased random inactivation layer, the uncertainty of the pseudo-label is calculated, and the data corresponding to the pseudo-label with the uncertainty less than threshold value is reserved, the data corresponding to the reserved pseudo-label and real training set are used to retrain the model, and the model is iterated until training is completed, and the line-of-sight evaluation model is obtained.The present application greatly reduces the dependence of line-of-sight estimation on artificial label.

Description

Technical Field

[0001] This invention relates to the field of gaze estimation technology, and in particular to a gaze estimation method based on pseudo-labels. Background Technology

[0002] Gaze estimation methods can be divided into model-based methods and appearance-based methods. Model-based methods calculate key points from human eye or face images and combine them with a 3D human eye model to predict gaze. Appearance-based methods directly extract features from human eye or face images to predict gaze. Model-based methods have high accuracy but rely on specialized equipment such as infrared cameras and are usually limited to indoor scenes. Appearance-based methods do not rely on specialized equipment and are applicable to both indoor and outdoor scenes, but their accuracy is relatively lower.

[0003] Thanks to the development of deep learning, appearance-based deep learning methods have achieved a significant improvement in accuracy, surpassing traditional machine learning methods. However, deep learning relies on large-scale datasets. Existing gaze estimation datasets have limited scene variety and a relatively small number of images, which restricts further improvement in algorithm performance. How to generate more labeled data on existing datasets has become a key research focus.

[0004] The intuitive idea is to assign pseudo-labels to unlabeled data, thus obtaining a large amount of labeled data at a relatively low cost. Ghosh et al. first detected eye keypoints and then used these keypoints in conjunction with a 3D model of the human eye to estimate pseudo-labels for gaze. This method heavily relies on eye keypoint detection algorithms, and since different people have different eye physiological structures, using the same human eye model cannot take into account individual differences. Kothari et al. designed pseudo-labels for eye gaze by utilizing the characteristic that people look at each other during conversation, but these methods depend on specific scenarios and cannot guarantee that the conversation participants are always looking at each other.

[0005] Cheng et al. were the first to introduce the Vision Transformer (ViT) into the field of gaze estimation. They first used a residual network to extract feature maps from facial images, then used these feature maps as input to the transformer, keeping other structures unchanged from ViT, and finally outputting the gaze. Although this method achieved state-of-the-art results on common gaze estimation datasets, the ViT model has a large number of parameters, and its performance is poor when trained directly on a small dataset. It requires pre-training with a large amount of labeled data, which significantly increases the computational cost of the model and limits the scalability of the algorithm.

[0006] The multi-head attention mechanism plays a crucial role in the Transformer. Cai et al. discarded other complex structures in ViT and only retained the multi-head attention part. They first used a residual network to extract features from the left and right eye images and the facial image. Then, they treated these three features as a sequence of length three and input them into the multi-head attention network. Finally, they concatenated the features and used a fully connected layer and the ReLU activation function to output the gaze. Compared with Cheng et al.'s method, this method is relatively simple in model structure and does not require large datasets for pre-training. However, the model requires additional left and right eye image inputs, which increases the additional data preprocessing work. Moreover, if the eyes are occluded and eye images cannot be collected, the model's performance may be significantly affected. Summary of the Invention

[0007] This invention provides a gaze estimation method based on pseudo-labels, which solves the problem of datasets with simple scenes and relatively few images.

[0008] The technical solution adopted by this invention to solve its technical problem is: to provide a gaze estimation method based on pseudo-labels, comprising the following steps:

[0009] Acquire facial images;

[0010] The facial image is input into a trained gaze assessment model for prediction. During training, the gaze assessment model is trained using a small amount of labeled real training data to obtain a pre-trained model. Based on this pre-trained model with an added random deactivation layer, pseudo-labels are applied to unlabeled data. The uncertainty value of the pseudo-labels is calculated, and data corresponding to pseudo-labels with uncertainty values ​​less than a threshold are retained. The retained pseudo-label data and the real training set are then used to retrain the model. This process is iterated until training is complete, resulting in the gaze assessment model.

[0011] The process of adding pseudo-labels to unlabeled data using the pre-trained model with an added random deactivation layer specifically involves inputting the same unlabeled data multiple times into the pre-trained model with the random deactivation layer enabled to obtain multiple pseudo-labels.

[0012] The calculation of the uncertainty value of the pseudo-label specifically involves: calculating the mean of multiple pseudo-labels and using the mean as the pseudo-label of the unlabeled data; calculating the standard deviation of the multiple pseudo-labels and using the standard deviation as the uncertainty value of the pseudo-label.

[0013] The loss function of the model is: in, It is the model's prediction result on the real training set images, y train These are the real labels from the actual training set. It is the model's prediction result for pseudo-labeled data, y pseudo These are the retained pseudo-labels, and α represents the weight information of the pseudo-label data.

[0014] The gaze assessment model includes: a backbone network for extracting feature representations from the facial image; a fully connected layer for regressing predicted gazes from the feature representations; and a random deactivation layer positioned between the backbone network and the fully connected layer when pseudo-labeling unlabeled data.

[0015] Beneficial effects

[0016] By employing the above-described technical solution, this invention has the following advantages and positive effects compared to existing technologies: This invention trains a model using a small amount of labeled data to obtain a pre-trained model. Then, the pre-trained model is used to assign pseudo-labels to a large amount of unlabeled data. Next, the uncertainty of the pseudo-labels is calculated, and a threshold is set to remove pseudo-labeled data with uncertainty greater than the threshold. Only data with smaller pseudo-labels are retained and added to the real training set. The model is then retrained using the real training set and the selected pseudo-labeled data. This process is iterated until training is complete. This invention can achieve excellent performance on multiple gaze estimation datasets without relying on manual labels, greatly reducing the dependence of gaze estimation on manual labels. Attached Figure Description

[0017] Figure 1 This is a flowchart of model training in an embodiment of the present invention;

[0018] Figure 2 This is a schematic diagram of the model structure in an embodiment of the present invention. Detailed Implementation

[0019] The present invention will be further illustrated below with reference to specific embodiments. It should be understood that these embodiments are for illustrative purposes only and are not intended to limit the scope of the invention. Furthermore, it should be understood that after reading the teachings of this invention, those skilled in the art can make various alterations or modifications to the invention, and these equivalent forms also fall within the scope defined by the appended claims.

[0020] The embodiments of the present invention relate to a pseudo-label-based gaze estimation method, comprising the following steps: acquiring a facial image; inputting the facial image into a trained gaze evaluation model to predict the gaze.

[0021] like Figure 1As shown, during training, the gaze assessment model is trained using a small amount of labeled real training data to obtain a pre-trained model. Pseudo-labels are then applied to unlabeled data based on the pre-trained model with added random deactivation layers. The uncertainty value of the pseudo-labels is calculated, and data corresponding to pseudo-labels with uncertainty values ​​less than a threshold are retained. The retained pseudo-label data and the real training set are then used to retrain the model. This process is iterated until training is complete, resulting in the gaze assessment model.

[0022] In this embodiment, the standard deviation of the predicted values ​​is used as the uncertainty of the pseudo-labels. When assigning pseudo-labels to unlabeled data based on the pre-trained model, this embodiment uses a random deactivation layer. This ensures that for the same sample, multiple inputs into the pre-trained model yield different predicted values. For example, if the same facial image is input into the pre-trained model M times, due to the presence of the random deactivation layer, the pre-trained model will randomly set some parameters to zero each time it is tested, resulting in different parameters for each test and thus different prediction results. This yields M different pseudo-labels. Let's assume that the pseudo-label predicted each time is... The average of the M prediction results is This value can be used as a pseudo-label for the unlabeled data, and the standard deviation is:

[0023]

[0024] Let's assume the label data (Data Label) is D. L Unlabeled data (Data Unlabel) is D UL There are Q unlabeled images, and the total number of iterations is N. The standard deviation threshold is μ. Therefore, the training set for each iteration is...

[0025]

[0026]

[0027] If the input image is clear and has little occlusion, and the model has learned a good representation of the image, then even if the model is slightly different each time due to the random discarding of some parameters, the predicted line-of-sight values ​​should be very close or even the same. Conversely, if the predicted values ​​are very close each time, it means that the model is robust to the current prediction result and the current prediction result is reliable.

[0028] Obviously, the similarity of predicted values ​​can be judged by calculating the standard deviation of multiple prediction results. If the standard deviation of multiple prediction results is small, it means that the prediction results are very close, and the current prediction result is reliable, making it a reliable pseudo-label. If the standard deviation of multiple prediction results is large, it means that the prediction results differ significantly, and the current result is relatively unreliable, so the current pseudo-label should be discarded. By setting an appropriate threshold, data corresponding to pseudo-labels with high confidence can be selected as a supplement to the real training set. Adding the data corresponding to pseudo-labels with high confidence to the real training set yields a new training set. The model is then retrained using this new training set. Considering that pseudo-label data may contain some error, a coefficient can be set for the loss function to reduce the impact of incorrect labels, as shown in the following formula:

[0029]

[0030] It is the model's prediction result on the real training set images, y train These are the real labels from the actual training set. It is the model's prediction result for pseudo-labeled data, y pseudo The pseudo-labels are retained, and α is the weight information of the pseudo-label data. It can be set to a number less than 1.0, such as 0.5, which means increasing the weight of the true label loss function and reducing the impact of pseudo-label prediction errors.

[0031] Given the significantly increased training set size compared to the previous round, the model trained this time is expected to perform better. It can predict relatively stable results even for facial images with poor quality, select more pseudo-label data, and retrain the model using the selected pseudo-labels and real labels. This iterative process continues until no more pseudo-label data can be selected or a pre-set number of iterations is reached, at which point the iteration optimization stops. Finally, the trained model is output as a gaze evaluation model.

[0032] It is easy to see that this invention trains a model using a small amount of labeled data to obtain a pre-trained model. Then, the pre-trained model is used to assign pseudo-labels to a large amount of unlabeled data. Next, the uncertainty of the pseudo-labels is calculated, and a threshold is set to remove pseudo-labeled data with uncertainty greater than the threshold. Only data with smaller pseudo-labels are retained and added to the real training set. The model is then retrained using the real training set and the selected pseudo-labeled data. This process is iterated until training is complete. This invention can achieve excellent performance on multiple gaze estimation datasets without relying on manual labels, greatly reducing the dependence of gaze estimation on manual labels.

[0033] The network structure of the gaze assessment model used in this embodiment is as follows: Figure 2As shown, it includes: a backbone network for extracting feature representations from the facial image; a fully connected layer for regressing predicted gaze from the feature representations; and a random deactivation layer positioned between the backbone network and the fully connected layer when pseudo-labeling unlabeled data is applied.

Claims

1. A pseudo-label based line-of-sight estimation method, characterized in that, The method comprises the following steps: obtaining a face image; inputting the face image into a trained gaze evaluation model for prediction to obtain a predicted gaze; wherein, during training, the gaze evaluation model is trained using a small amount of labeled real training set to obtain a pre-trained model, pseudo labels are added to unlabeled data according to the pre-trained model with a random inactivation layer, the uncertainty value of the pseudo labels is calculated, and the data corresponding to the pseudo labels with an uncertainty value less than a threshold value is retained, the retained data corresponding to the pseudo labels and the real training set are used to retrain the model, and the iteration is repeated until the training is completed to obtain the gaze evaluation model; the loss function of the gaze evaluation model is: wherein, is the prediction result of the model on the real training set images, y train is the real label of the real training set, is the prediction result of the model on the pseudo-labeled data, y pseudo is the reserved pseudo-label, and a is the weight information of the pseudo-labeled data.

2. The pseudo-label based line-of-sight estimation method of claim 1, wherein, The pseudo labels are added to the unlabeled data according to the pre-trained model with a random inactivation layer, specifically: the same unlabeled data is inputted multiple times into the pre-trained model with an opened random inactivation layer to obtain multiple pseudo labels.

3. The pseudo-label-based line of sight estimation method of claim 2, wherein, The uncertainty value of the pseudo labels is calculated, specifically: the mean of multiple pseudo labels is calculated, and the mean is taken as the gaze pseudo label of the unlabeled data; the standard deviation of multiple pseudo labels is calculated, and the standard deviation is taken as the uncertainty value of the pseudo label.

4. The pseudo-label-based line of sight estimation method of claim 1, wherein, The gaze evaluation model comprises: a backbone network for extracting feature representation from the face image; a fully connected layer for regressing predicted gaze from the feature representation; when pseudo labels are added to unlabeled data, the random inactivation layer is arranged between the backbone network and the fully connected layer.

Images