Line-of-sight direction estimation method and system based on facial image

Abstract

The invention provides a sight line direction estimation method based on a facial image. The method comprises the following steps: constructing and training a sight line estimation model and a portrait enhancement model by source domain data; selecting training data from a target domain, and training the sight line estimation model in an unsupervised learning mode; and processing a target image of the target domain through the portrait enhancement model and the sight line estimation model to obtain a sight line direction estimation result of the target image. The invention also provides a sight line direction estimation system based on the face image and a data processing device for realizing sight line direction estimation based on the face image.

Description

Technical Field

[0001] The present invention belongs to the technical field of computer vision, and in particular relates to a line of sight estimation method and system based on facial images. Background Art

[0002] Facial gaze direction estimation refers to estimating the user's gaze direction or viewpoint position based on the face image captured by the camera. This technology is the core technology that uses computers to perceive and understand human visual attention, and has broad application prospects in human-computer interaction, emotional computing, medical diagnosis and other fields. Traditional gaze and viewpoint estimation technologies usually rely on expensive additional equipment, which makes it difficult to promote and apply them on a large scale. In recent years, with the widespread application of deep learning technology in the field of computer vision, the method of using only monocular cameras for gaze has developed rapidly.

[0003] Existing methods for line of sight estimation based on facial images rely on supervised data and machine learning methods, such as China's national patents "Line of sight estimation method, device, electronic device and computer-readable storage medium" (publication number: CN113470114A), "A line of sight estimation method based on appearance" (publication number: CN113807251A) and "An adaptive line of sight estimation method, system, electronic device and storage medium" (publication number: CN115862095A). However, the line of sight datasets currently used to train line of sight estimation models can only cover a limited number of scenarios. Due to differences in users, image quality, background environment and lighting, the line of sight estimation model trained in some scenarios may not perform well in other scenarios. The training data domain is usually called the source domain, and the new application scenario of the model is called the target domain. In order to improve the line of sight estimation accuracy of the line of sight estimation model in the target scene, the Chinese national patent "Adaptive line of sight estimation method and device" (publication number: CN111353506A) proposes to collect samples with line of sight annotations of the target user through calibration, and then use the annotated samples to adapt the trained line of sight estimation model to the target user. However, the step of obtaining the user's line of sight annotations proposed by it requires the cooperation of the user and takes a lot of time. In order to improve the accuracy of the line of sight estimation model for the target domain data by using unlabeled target domain data, the Chinese national patent "Line of sight estimation method, device and storage medium" (publication number: CN114863200A) proposes a method of adversarial training to optimize the line of sight estimation model. However, this method requires the use of source domain data and target domain data to train the model at the same time. However, in many real line of sight estimation application scenarios, the source domain data and the target domain data are usually not available to the user at the same time. First, for the purpose of protecting the privacy of data contributors, the face data used for source domain model training may be difficult to obtain at other stages other than the training stage. Secondly, reprocessing the source domain data during the adaptation and testing phases in the target domain may consume more computing resources and computing time, which is also unacceptable in real-time line of sight estimation tasks. Summary of the invention

[0004] In view of the above problems, the present invention proposes a gaze direction estimation method based on facial images, comprising: constructing and training a gaze estimation model and a portrait enhancement model with source domain data; selecting training data from a target domain and training the gaze estimation model in an unsupervised learning manner; and passing a target image of the target domain through the portrait enhancement model and the gaze estimation model to obtain a gaze direction estimation result of the target image.

[0005] The gaze direction estimation method of the present invention, wherein the steps of training the gaze estimation model and the portrait enhancement model specifically include: using a facial image set in a source domain Train the line of sight estimation model θ is the model parameter; the facial image I h Perform a resolution reduction operation to obtain a low-quality image I l , with I l Through this portrait enhancement model Obtain enhanced image Take I h and Train for the training set φ is the model parameter; the facial image I s pass Obtain enhanced image pass Get The line of sight estimation result based on and I s The true sight direction label g s ,optimization The facial image I m pass Obtain enhanced image Take I m The true sight direction label g m and The line of sight estimation result train And get the intermediate model at each stage of training Building a line of sight estimation model group

[0006] The method for estimating the line of sight direction of the present invention, wherein the step of training the line of sight estimation model in an unsupervised learning manner comprises: selecting a facial image I t and its corresponding enhanced image Supervised training with variance minimization and pseudo-labeling is the face image set of the target domain; the objective function of minimizing variance Indicates all The average line of sight estimation result in training In the Tth iteration, a time-averaged sight estimation model is generated Model parameters Then the pseudo label of the T+1th iteration is Objective function of pseudo-label supervision The overall objective function for training the line of sight estimation model is: γ is the weight hyperparameter of the balanced loss; Model parameters The average value of θ ★ , as the final line of sight estimation model The model parameters.

[0007] The sight line direction estimation method of the present invention is for a target facial image I o ,pass Get enhanced image and through Get I o The line of sight direction estimation result

[0008] The present invention also proposes a gaze direction estimation system based on facial images, comprising: a source domain model training module, used to construct and train a gaze estimation model and a portrait enhancement model with source domain data; a target domain model adaptation module, used to select training images from a target domain and train the gaze estimation model in an unsupervised learning manner; and a gaze direction estimation module, used to pass a target image of the target domain through the portrait enhancement model and the gaze estimation model to obtain a gaze direction estimation result of the target image.

[0009] The gaze direction estimation system of the present invention, wherein the source domain model training module comprises: a gaze estimation model pre-training module for using a facial image set of the source domain Train the line of sight estimation model θ is the model parameter; the portrait enhancement model pre-training module is used to transform the facial image I h Perform a resolution reduction operation to obtain a low-quality image I l , with I l Through this portrait enhancement model Obtain enhanced image Take I h and Train for the training set φ is the model parameter; the portrait enhancement model adjustment module is used to transform the facial image I s pass Obtain enhanced image pass Get The line of sight estimation result based on and I s The real sight direction label gs is optimized The gaze estimation model adjustment module is used to transform the facial image I m pass Obtain enhanced image Take I m The true sight direction label g m and The line of sight estimation result train And get the intermediate model at each stage of training Building a line of sight estimation model group

[0010] The sight direction estimation system of the present invention, wherein the target domain model adaptation module comprises: a variance minimization training module for selecting a facial image I t and its corresponding enhanced image Training with variance minimization is the face image set of the target domain; the objective function of minimizing variance Indicates all The average line of sight estimation result of ; pseudo label supervision training module is used to train In the Tth iteration, a time-averaged sight estimation model is generated Model parameters Then the pseudo label of the T+1th iteration is Objective function of pseudo-label supervision The target domain adaptation optimization module is used to optimize the overall objective function Train the line of sight estimation model and use all Model parameters The average value of θ ★ , as the final line of sight estimation model The model parameters are γ, and γ is the weight hyperparameter of the balance loss.

[0011] The sight line direction estimation system of the present invention, wherein the sight line direction estimation module specifically comprises: for the target facial image I o ,pass Get enhanced image and through Get I o The line of sight direction estimation result

[0012] The present invention also proposes a computer-readable storage medium storing computer-executable instructions, characterized in that when the computer-executable instructions are executed, the above-mentioned method for estimating the gaze direction based on facial images is implemented.

[0013] The present invention also proposes a data processing device, comprising the computer-readable storage medium as described above, when the processor of the data processing device calls and executes the computer-executable instructions in the computer-readable storage medium, it realizes the estimation of the gaze direction based on the facial image. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] Figure 1 Schematic diagram of the framework of the sight line direction estimation system of the present invention.

[0015] Figure 2 It is a flow chart of the line of sight direction estimation method of the present invention.

[0016] Figure 3 It is a flow chart of training the image enhancement model and the sight line estimation model of the present invention.

[0017] Figure 4 It is a schematic diagram of a data processing device of the present invention. DETAILED DESCRIPTION

[0018] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings. It should be understood that the specific implementation method described herein is only used to explain the present invention and is not used to limit the present invention.

[0019] The present invention aims to solve the problem that the general gaze estimation model performs poorly in the new target domain scene when there are large differences between the source domain and the target domain (such as users, image quality, background environment and lighting). To this end, the present invention proposes a gaze direction estimation method and system based on facial images, which can improve the estimation accuracy of the gaze model on the target scene by using a small batch of data in the target scene.

[0020] When studying the line of sight estimation method and application system, the inventor found that the data in the actual use scenario (target domain) is different from the data when training the model (source domain), and the target domain and source domain data are difficult to obtain at the same time, but are usually obtained by the algorithm in the two stages of model development and model application respectively. The prior art does not consider how to adapt the line of sight estimation model to the target domain, that is, to improve the line of sight estimation accuracy of the model in the target domain, when the source domain data and the target domain data cannot be obtained at the same time. After research, the inventor found that this defect can be solved by enhancing the portrait and reducing the variance of the line of sight estimation results of the source domain model group on the target domain samples.

[0021] The method for estimating the line of sight direction based on facial images of the present invention includes a portrait enhancement module that keeps the line of sight unchanged. The module obtains a portrait enhancement model through a series of joint training steps for a portrait enhancement network and a line of sight estimation network. The joint training includes four steps: pre-training the line of sight estimation model, pre-training the portrait enhancement network, fine-tuning the portrait enhancement model, and fine-tuning the line of sight estimation model. The portrait enhancement model can enhance the details related to the line of sight in the target domain face image, and can ensure that the line of sight direction corresponding to the portrait before and after enhancement remains unchanged. After applying the portrait enhancement model to the target domain image, the line of sight estimation result of the source domain model on the enhanced target domain image can be improved.

[0022] In addition, a gaze direction estimation model integration method based on multi-model parameter averaging is proposed. The parameters of a group of gaze estimation models with the same architecture but different parameters are averaged, and the averaged parameters are used as the parameters of the final gaze estimation model. Compared with a single gaze estimation model, the use of an average parameter model can improve the gaze estimation results of the model in the target scene.

[0023] Third, a target domain model adaptation method based on reducing the prediction variance of the line of sight estimation model group on the target sample is proposed. A target sample can be input into different line of sight models in the line of sight estimation model group to obtain a set of different line of sight estimation values. The variance of the line of sight estimation values ​​is used as a loss function, and the loss function is used to optimize the entire line of sight estimation model group. By reducing the variance of the line of sight estimation results of the line of sight estimation model group on the target sample through the optimization method, the line of sight estimation error of the models in the model group on the target sample can be significantly reduced.

[0024] Fourth, a line of sight estimation model training system that integrates portrait enhancement, target domain model adaptation, and multi-model integration is proposed. First, in the source domain model training stage, a portrait enhancement module that can ensure the unchanged line of sight and a set of line of sight estimation model groups are trained, and the portrait enhancement module is used to enhance the target domain face image. The variance of the line of sight estimation results of the line of sight estimation model group on the enhanced target sample is reduced through optimization methods. Finally, the average parameter model of the optimized line of sight estimation model group and the portrait enhancement model are used to estimate the line of sight direction of the target scene. In this way, a model with high line of sight accuracy in the target domain can be trained using labeled source domain data and unlabeled target domain data.

[0025] For the problem of eye gaze direction estimation based on facial images, the present invention first defines the training set source domain as in and Respectively represent the facial image of the i-th sample and its corresponding true gaze direction label, N s is the number of samples in the source domain. Usually, samples in the source domain are collected in a controlled environment, because the true value of the line of sight direction can be easily obtained in a controlled environment. represents the target domain data, that is, the application scenario of the sight line estimation method of the present invention. t is the number of samples in the target domain.

[0026] The goal of the present invention is to estimate the sight line direction of the target domain when the source domain data and the target domain data cannot be obtained at the same time. That is, the present invention uses the source domain training set to train a source domain model, and then uses the unlabeled target domain data to adapt the source domain model to the target domain, so as to improve the accuracy of the sight line estimation model in the target domain.

[0027] Figure 1 The specific framework of the line of sight estimation system proposed in the present invention is demonstrated. The framework includes three stages, namely, the source domain model training stage, the passive domain target domain adaptation stage and the target domain sample testing stage. In the source domain model training stage, a portrait enhancement model that keeps the line of sight unchanged and a group of line of sight estimation models are jointly trained. A portrait enhancement model is used to enhance the details of the input face image and improve the quality of the input image, so that the enhanced sample can be more conducive to the line of sight estimation, and a group of line of sight estimation models are used to adapt the model to the target domain. In the target domain adaptation stage of the passive domain, the group of line of sight estimation models is updated by two mechanisms: variance minimization and pseudo-label supervision. The variance minimization mechanism can reduce the model uncertainty of the line of sight estimation model group, while the pseudo-label supervision mechanism ensures the ability of the model group to accurately estimate the line of sight direction. In the target domain testing stage, the average value of the parameters of the line of sight estimation model group updated in the previous stage is used to obtain a final line of sight estimation model, and the model is used to estimate the line of sight direction of the enhanced target domain image. As shown in FIG. Figure 2 As shown, the line of sight direction estimation method of the present invention specifically includes the following steps:

[0028] Step S1: Source domain model training

[0029] In the source domain model training stage, the present invention jointly trains a portrait enhancement model capable of maintaining the gaze direction and a group of gaze estimation models for the target domain adaptation stage. Figure 3 The specific steps of training the image enhancement model and the line of sight estimation model of the present invention are shown. First, a line of sight estimation model is trained using source domain data. Second, a portrait enhancement model is pre-trained using a set of high-quality images. Then, the portrait enhancement model is fine-tuned by adding line of sight consistency constraints to labeled source domain images and unlabeled high-quality images. Finally, the line of sight estimation model is fine-tuned using the enhanced source domain data.

[0030] Step S11: Pre-training sight line estimation model

[0031] In the embodiment of the present invention, ResNet18 is used as the line of sight estimation model The present invention uses labeled source domain data Optimize (train) the gaze estimation model by reducing the predicted value of the gaze estimation model and the true value of the gaze direction:

[0032]

[0033] in and are the estimated value and true value of the line of sight of the original training image, respectively.

[0034] The line of sight estimation model of the present invention may also be other feature extraction networks, such as AlexNet network, HRNet network, etc., but the present invention is not limited thereto.

[0035] Step S12: Pre-training portrait enhancement model

[0036] The present invention uses a universal image super-resolution network Real-ESRGAN (here can be any image enhancement model) as a portrait enhancement model Next, the present invention uses a high-definition portrait dataset to The pre-training process. Given a high-quality portrait I h The present invention first degrades the image through an image degradation model, and then degrades the image I l Input into the portrait enhancement model to get the enhanced image The present invention ensures that the image generated by the network is consistent with the loss and adversarial network. and the original image I h They are consistent at both the pixel and semantic levels and cannot be distinguished by a trained discriminator.

[0037] The portrait enhancement model of the present invention may also be other image enhancement models, such as a SwinIR model, a CodeFormer model, etc., but the present invention is not limited thereto.

[0038] Step S13: Fine-tuning the portrait enhancement model

[0039] Although the pre-trained portrait enhancement model can recover image details from low-quality face images, the model may also modify the appearance of the eyes, resulting in a different gaze direction after enhancement than before enhancement. In order to ensure that the gaze direction before and after enhancement is not changed, the present invention fine-tunes the portrait enhancement model by requiring the estimated gaze direction after enhancement and the true gaze direction of the image before enhancement. For an image I in the source domain, s Use the portrait enhancement model to enhance it into Then use the pre-trained line of sight estimation model The present invention requires the line of sight estimation result This is achieved by optimizing the following line of sight estimation loss l g To get:

[0040]

[0041] As in pre-training, the present invention still uses the HD image dataset to optimize the reconstruction loss and adversarial loss of the portrait enhancement model in the fine-tuning stage. In addition, the present invention also ensures the line of sight consistency of the portrait enhancement model on the HD image dataset, that is, it requires the original high-quality image I h and enhanced low-quality images\ have the same line of sight estimation value. To achieve this goal, the present invention minimizes the following line of sight consistency loss l gc :

[0042]

[0043] Step S14: Fine-tuning the sight line estimation model

[0044] After fine-tuning the portrait enhancement model, the present invention needs to enable the sight line estimation model to accurately estimate the sight line direction of the enhanced portrait, so the present invention uses the enhanced source domain image to fine-tune the sight line estimation model. First, the present invention fixes the parameters of the portrait enhancement model, and then the present invention updates the enhanced image through the following optimization objectives:

[0045]

[0046] in is the facial image I m Source domain image enhanced by the portrait enhancement model.

[0047] In the source domain model training stage, the present invention obtains a portrait enhancement model that can maintain the gaze direction and a gaze estimation model that can predict the gaze of the enhanced image. In the final stage of fine-tuning the gaze estimation model, the present invention obtains a series of gaze estimation models with the same structure but different parameters from different stages of the fine-tuning process (a possible example: taking models with different iterations in the fine-tuning process as different models), and uses them in the subsequent passive domain target domain adaptation stage.

[0048] Step S2: Passive domain to target domain adaptation

[0049] In the target domain adaptation stage, the present invention can be applied in scenarios where only unlabeled target domain data and trained source domain models can be obtained, but source domain data cannot be used. In order to adapt the trained sight estimation model to the unlabeled target domain, the present invention needs to update the parameters of the sight estimation model group through an unsupervised objective function. The unsupervised objective function proposed in the present invention includes minimizing the parameters of the sight estimation model group and using pseudo-label supervision to retain the ability of the model group to perform sight estimation. Given a target domain image I t The present invention first improves the image quality of the portrait enhancement model trained above, and obtains an enhanced image. Then, The original sight estimation model group is optimized by minimizing the variance and using pseudo-label supervision. Next, the invention introduces these two mechanisms for unsupervised target domain adaptation in detail, and introduces the optimization objective (i.e., the training loss function) of the final target domain adaptation stage.

[0050] Step S21: Minimize variance

[0051] The present invention records the trained sight line estimation model group as in is the kth model in the model group, and its model parameters are θ k , and K is the number of models in the model group. The present invention updates the parameters by minimizing the model uncertainty of the target domain samples. The formalized goal is:

[0052]

[0053] in Represents the kth line of sight estimation model In the enhanced target domain image The line of sight estimation result on is the average estimation result of K models.

[0054] Step S22: Pseudo-label supervision

[0055] Only using variance minimization to optimize the model group may cause the model group to output the same result for all samples, that is, the model degenerates to a trivial solution. In order to retain the ability of the line of sight estimation model group to perform line of sight estimation, the present invention introduces line of sight pseudo-labels to supervise the line of sight estimation results of the model group in the target domain. Considering that directly using the output of the line of sight estimation model of the variance minimization branch as a pseudo-label may accumulate errors, the present invention uses the temporal average model of the line of sight estimation model to generate pseudo-labels.

[0056] In the Tth iteration of the training in the target domain adaptation stage, the present invention performs the following operations on each sight estimation model: Keep a time-averaged version of the model And use this set of time series average models to generate pseudo labels.

[0057] At the Tth iteration, the time series average parameter Updated by:

[0058]

[0059] Then, the present invention uses the time series average model of K T iterations After enhancement, the target domain image The average of the outputs is used as the pseudo label for line of sight estimation in the next iteration:

[0060]

[0061] in It is the sight line estimation result of the time series average model with the kth iteration number T.

[0062] In order to ensure that the model can output reliable line of sight estimation results, the present invention requires that the output of the line of sight estimation model in the target domain adaptation stage should not deviate too far, and its optimization goal is:

[0063]

[0064] in is the kth model in the model group in the image The line of sight estimation results on .

[0065] Step S23: Optimization target of target domain adaptation

[0066] In the unsupervised target domain adaptation stage in the source domain, the present invention minimizes the uncertainty and pseudo-label supervision mechanism on a small number of target domain samples. vm and l pl The weighted sum of , so that it can adapt to the target domain. Formally, the goal is:

[0067]

[0068] in is a subset of the target domain samples, and γ is a weight hyperparameter used to balance the two losses so that they are roughly in the same order of magnitude.

[0069] Step S3: Target domain image estimation

[0070] After completing the target domain adaptation of the gaze estimation model group, the present invention uses the adapted gaze estimation model group to estimate the gaze direction of the target domain image. o The present invention first inputs it into the portrait enhancement model obtained in step S1 to obtain an enhanced facial image Then Input to an average gaze estimation model The parameters of the average line of sight estimation model are the parameters of the adapted line of sight estimation model group The average value of the parameter.

[0071] Figure 4 Schematic diagram of the data processing device of the present invention. Figure 4 As shown, an embodiment of the present invention further provides a computer-readable storage medium and a data processing device. The computer-readable storage medium of the present invention stores computer-executable instructions, and when the computer-executable instructions are executed by the processor of the data processing device, the line of sight direction estimation based on the facial image is realized. A person of ordinary skill in the art can understand that all or part of the steps in the above method can be completed by instructing related hardware (such as a processor, FPGA, ASIC, etc.) through a program, and the program can be stored in a readable storage medium, such as a read-only memory, a disk or an optical disk, etc. All or part of the steps in the above embodiment can also be implemented using one or more integrated circuits. Accordingly, each module in the above embodiment can be implemented in the form of hardware, such as implementing its corresponding functions through an integrated circuit, or can be implemented in the form of a software function module, such as implementing its corresponding functions through a processor executing a program / instruction stored in a memory. The embodiment of the present invention is not limited to any specific form of combination of hardware and software.

[0072] In general, the training method and system of the gaze estimation model based on facial images proposed in the present invention can improve the gaze estimation accuracy of the gaze estimation model in the target scene with the help of a small batch of unlabeled data in the target scene, when the unlabeled target domain samples, portrait enhancement model and source domain gaze estimation pre-training model are known.

[0073] The above implementation modes are only used to illustrate the present invention, but not to limit the present invention. Ordinary technicians in the relevant technical field can make various changes and modifications without departing from the spirit and scope of the present invention. Therefore, all equivalent technical solutions also belong to the scope of the present invention. The patent protection scope of the present invention should be defined by the claims.

Claims

1. A method for estimating gaze direction based on facial images, characterized in that: include: Build and train gaze estimation and portrait enhancement models using source domain data; Selecting training data from the target domain and training the line of sight estimation model in an unsupervised learning manner; The target image in the target domain is passed through the portrait enhancement model and the sight line estimation model to obtain a sight line direction estimation result of the target image.

2. The method for estimating the line of sight direction according to claim 1, wherein: The steps of training the sight estimation model and the portrait enhancement model specifically include: The facial image set in the source domain Train the line of sight estimation model θ is the model parameter; The facial image I h Perform a resolution reduction operation to obtain a low-quality image I l , with I l Through this portrait enhancement model Get enhanced image Take I h and Train for the training set φ is the model parameter; The facial image I s pass Get enhanced image pass Get The line of sight estimation result based on and I s The true sight direction label g s ,optimization The facial image I m pass Get enhanced image Take I m The true sight direction label g m and The line of sight estimation result train And get the intermediate model at each stage of training Building a line of sight estimation model group 3. The method for estimating the line of sight direction according to claim 2, wherein: The steps to train the line of sight estimation model in an unsupervised learning manner include: Select facial image I t and its corresponding enhanced image Supervised training with variance minimization and pseudo-labeling is the face image set of the target domain; The objective function of minimizing variance Indicates all The average line of sight estimation result of In training In the Tth iteration, a time-averaged sight estimation model is generated Model parameters Then the pseudo label of the T+1th iteration is Objective function of pseudo-label supervision The overall objective function for training the line of sight estimation model is: γ is the weight hyperparameter of the balance loss; With all Model parameters The average value of θ ★ , as the final line of sight estimation model The model parameters.

4. The method for estimating the line of sight direction according to claim 3, characterized in that: For the target face image I o ,pass Get enhanced image and through Get I o The line of sight direction estimation result 5. A system for estimating gaze direction based on facial images, characterized in that: include: A source domain model training module, which is used to build and train a gaze estimation model and a portrait enhancement model using source domain data; A target domain model adaptation module, used to select training images from the target domain and train the line of sight estimation model in an unsupervised learning manner; The sight line direction estimation module is used to obtain a sight line direction estimation result of the target image in the target domain through the portrait enhancement model and the sight line estimation model.

6. The sight line direction estimation system according to claim 5, characterized in that: The source domain model training module includes: Gaze estimation model pre-training module for face image sets in the source domain Train the line of sight estimation model θ is the model parameter; The portrait enhancement model pre-training module is used to transform the facial image I h Perform a resolution reduction operation to obtain a low-quality image I l , with I l Through this portrait enhancement model Get enhanced image Take I h and Train for the training set φ is the model parameter; The portrait enhancement model adjustment module is used to adjust the facial image I s pass Get enhanced image pass Get The line of sight estimation result based on and I s The true sight direction label g s ,optimization The gaze estimation model adjustment module is used to transform the facial image I m pass Get enhanced image Take I m The true sight direction label g m and The line of sight estimation result train And get the intermediate model at each stage of training Building a line of sight estimation model group 7. The sight line direction estimation system according to claim 6, characterized in that: The target domain model adaptation module includes: Variance minimization training module for selecting facial images I t and its corresponding enhanced image Training with variance minimization is the face image set of the target domain; the objective function of minimizing variance Indicates all The average line of sight estimation result of Pseudo-label supervised training module is used to train In the Tth iteration, a time-averaged sight estimation model is generated Model parameters Then the pseudo label of the T+1th iteration is Objective function of pseudo-label supervision The target domain adaptation optimization module is used to optimize the overall objective function Train the line of sight estimation model and use all Model parameters The average value of θ ★ , as the final line of sight estimation model The model parameters are γ, and γ is the weight hyperparameter of the balance loss.

8. The sight line direction estimation system according to claim 7, characterized in that: The sight direction estimation module specifically includes: for the target facial image I o ,pass Get enhanced image and through Get I o The line of sight direction estimation result 9. A computer-readable storage medium storing computer-executable instructions, characterized in that: When the computer executable instructions are executed, the gaze direction estimation method based on facial images as described in any one of claims 1 to 4 is implemented.

10. A data processing device, comprising the computer-readable storage medium as claimed in claim 9, wherein when a processor of the data processing device retrieves and executes computer-executable instructions in the computer-readable storage medium, gaze direction estimation based on facial images is implemented.

Images