A line of sight estimation method and system

Abstract

The application relates to the technical field of image recognition, and discloses a gaze estimation method and system, which uses a personalized feature extraction model to perform personalized feature extraction on left and right eye images of a target person, obtains a corresponding feature vector W1, also uses the feature extraction model to perform feature extraction on a face image of the target person, obtains a corresponding feature vector W2, removes face features irrelevant to the gaze and retains features relevant to the gaze, splices and fuses the feature vectors W1 and W2, inputs the fused feature vector W3 into an MLP network, and finally obtains a horizontal direction angle G yaw and a vertical direction angle G pitch , obtains a three-dimensional gaze direction vector according to the horizontal direction angle and the vertical direction angle, and finally performs gaze estimation according to the three-dimensional gaze direction vector, so that the estimation accuracy of a gaze estimation model is improved.

Description

Technical Field

[0001] This invention relates to the field of image recognition technology, and in particular to a gaze estimation method and system. Background Technology

[0002] Gaze is a manifestation of human attention and interest, containing information such as inner state and behavioral intentions. Furthermore, human gaze possesses characteristics unmatched by other forms of information, including directness, naturalness, and bidirectionality. Therefore, research on gaze estimation has garnered significant interest. Building upon facial recognition, we can further utilize gaze estimation techniques to calculate the area of ​​focus a user is paying attention to on the screen. The applications of gaze estimation technology are vast, including smart home appliances, advertising research, intelligent computing, and virtual games. Existing gaze estimation methods can be broadly categorized into model-based and appearance-based methods. Model-based methods, however, have high requirements for hardware and experimental environments, making them unsuitable for complex and ever-changing real-world application scenarios, and resulting in low recognition rates and generalization abilities.

[0003] Existing technology discloses a gaze estimation method and apparatus. The method includes acquiring a target image, the target image presenting a facial image of a user to be tested; performing facial feature detection on the target image to obtain facial feature information of the user to be tested; inputting the target image and the facial feature information into a machine learning model for gaze estimation; and obtaining the gaze estimation result output by the machine learning model. Existing technology only uses facial features for gaze recognition, while ignoring eye features with personalized information, resulting in low accuracy of gaze estimation; furthermore, the human images used to train the model are limited, leading to low accuracy of gaze estimation when recognizing new human figures and low generalization ability of the model. Summary of the Invention

[0004] The purpose of this invention is to provide a gaze estimation method and system based on facial features and personalized eye features.

[0005] To achieve the above objectives, the present invention provides a line-of-sight estimation method, which includes the following steps: S1: Obtain the target person image, and crop the target person image into the target person's eye image and the target person's face image to be estimated; S2: Input the eye image of the target person to be estimated into the personalized feature extraction model, and output a feature vector with personalized features. ; S3: Input the facial image of the target person to be estimated into the feature extraction model, and output a feature vector containing feature information closely related to gaze. ; S4: Transfer the feature vector and the feature vector The fusion process yields a feature vector containing personalized characteristics and features closely related to gaze. ; S5: Transfer the feature vector Input to MLP network to obtain horizontal orientation angle and vertical direction angle ; S6: Based on the horizontal direction angle and vertical direction angle The three-dimensional gaze direction vector is obtained, and the three-dimensional gaze direction vector is determined by the following formula: in( , , These constitute the three-dimensional line-of-sight direction vector. Line-of-sight estimation is performed based on the three-dimensional line-of-sight direction vector to obtain the estimation result.

[0006] Furthermore, the personalized feature extraction model described in step S2 is determined in the following manner: S2.1: Construct a personalized feature extraction model; S2.2: Obtain training dataset A, which contains a large number of high-resolution eye images of different people with different head poses, and group the training dataset A by the person's name; S2.3: Randomly select a main eye image Img1 of a person from the training dataset A, and then, according to the person's name in the main eye image Img1, select an auxiliary eye image Img2 in the group to which it belongs, which has the same head posture as the main eye image Img1 but a different gaze direction. Then, select an auxiliary eye image Img3 in the grouped images of other people's eyes, which has a different head posture but the same gaze direction as the main eye image Img1. S2.4: The main eye image Img1, the auxiliary eye image Img2, and the auxiliary eye portrait 3 are cropped into corresponding left and right eye images, and then the corresponding left and right eye images are processed by an asymmetric network to extract personalized features, thereby obtaining corresponding 12-dimensional feature vectors V1, V2, and V3. S2.5: Process the feature vectors V1, V2, and V3 using a loss function to correct for gaze deviation. The loss function is determined as follows: in This is a personalized feature vector for the main image of the eye. This is a personalized feature vector representing the same person in the main image but from different viewing angles. To generate personalized feature vectors from different people at the same gaze angle as the main image. The loss function represents the optimization of individual personalized information. Loss functions for optimizing personalized information for different individuals The loss function is optimized for total personalized information; S2.6: Repeat steps S2.1 to S2.6, using the previous loss function for each training iteration. Optimize the next This leads to the final personalized feature extraction model.

[0007] Furthermore, the training dataset A mentioned in step S2.2 is obtained in the following manner: Obtain the ETH-XGaze and Gaze360 training datasets. The ETH-XGaze training dataset contains 1 million high-resolution images of 110 people with different head poses, and the Gaze360 training dataset contains a large number of motion images of 238 people in different environments. Cropping the eyes from the images in the ETH-XGaze and Gaze360 training datasets yields training dataset A.

[0008] Furthermore, the asymmetric networks described in step S2.4 include: Base-CNN, FC1, and FC2. The Base-CNN consists of six convolutional layers connected in sequence, with the input of the first convolutional layer being the left-eye image. Or the right eye image, the output of the last convolutional layer is connected to FC1; FC1 is a fully connected layer with an output of 500. The input of FC1 is connected to the output of the last convolutional layer in Base-CNN, and the output of FC1 is connected to the input of FC2. The FC2 is a fully connected layer with a 12-dimensional output. The input of the FC2 is connected to the output of the FC1, and the output of the FC2 is a 12-dimensional feature vector.

[0009] The feature extraction model in step S3 is determined as follows: S3.1: Construct a feature extraction model; S3.2: Obtain training dataset B, which contains a large number of high-resolution facial images of different people with different head poses, and group the training dataset B according to the people's names; S3.3: Extract a first face image with the same name as the main eye image 1 from the training dataset B, and input the first face image into the feature extraction network Backbone to obtain a corresponding feature vector V4; S3.4: Input the feature vector V4 into the SA-Module to inversely recover the second face image; S3.5: Optimize V4 using the total loss function, which is determined as follows: in For the generated line-of-sight angle, The viewing angle of the label data. The first face image. This is the second face image generated in reverse. Let be the loss function for gaze regression. To counteract the loss function of reverse graph generation, The loss function is optimized for the total personalized features. These three loss functions are then weighted according to different weights to obtain the total loss function of the feature extraction model. ,in These are weight hyperparameters; S3.6: Repeat steps S3.1 to S3.5, each time using the previous total loss function to optimize the next feature vector V4, and finally obtain the feature extraction model.

[0010] The training dataset B mentioned in step 3.2 is obtained in the following way: Obtain the ETH-XGaze and Gaze360 training datasets. The ETH-XGaze training dataset contains 1 million high-resolution images of 110 people with different head poses, and the Gaze360 training dataset contains a large number of motion images of 238 people in different environments. Cropping faces from images in the ETH-XGaze and Gaze360 training datasets yields training dataset B.

[0011] Furthermore, the feature extraction network Backbone described in step S3.3 is composed of ResNet50.

[0012] Further, as described in step S3.5 Weight hyperparameters, where The value range is 1.7 to 2.3. Take a value of 0.7~1.3. Take a value of 1.3 to 1.8.

[0013] Furthermore, the weight hyperparameter Set to 2.0, weight hyperparameter Set to 1.0, weight hyperparameter Take 1.5.

[0014] To achieve the above objectives, the present invention also provides a line-of-sight estimation system, comprising: Acquisition module: Used to acquire images of the target person's eyes and face; Personalized feature extraction module: This module takes the eye image of the target person to be estimated as input into the personalized feature extraction model and outputs a feature vector with personalized features. ; Feature extraction module: This module takes the facial image of the target person to be estimated as input into the feature extraction model and outputs a feature vector containing feature information closely related to gaze. ; Fusion module: used to fuse the feature vector and the feature vector The fusion process yields a feature vector containing personalized characteristics and features closely related to gaze. ; Recognition module: used to process the feature vector Input to MLP network to obtain horizontal orientation angle and vertical direction angle According to the horizontal direction angle and vertical direction angle The three-dimensional gaze direction vector is obtained, and the three-dimensional gaze direction vector is determined by the following formula: in( , , These constitute the three-dimensional line-of-sight direction vector. Line-of-sight estimation is performed based on the three-dimensional line-of-sight direction vector to obtain the estimation result.

[0015] Compared with existing technologies, its advantages are as follows: This invention uses a personalized feature extraction model to extract personalized features from the left and right eye images of the target person, obtaining corresponding feature vectors W1. It also uses a feature extraction model to extract features from the face image of the target person, obtaining corresponding feature vectors W2. Facial features unrelated to gaze are removed while features related to gaze are retained. Finally, feature vectors W1 and W2 are concatenated and fused, and the fused feature vector W3 is input into an MLP network to obtain the horizontal direction angle G. yaw and vertical direction angle G pitch The invention can obtain a three-dimensional gaze direction vector based on the horizontal and vertical angles, and then perform gaze estimation based on the three-dimensional gaze direction vector, thereby improving the estimation accuracy of the gaze estimation model. The invention also adopts a cross-domain training model method. Since the environmental and human information collected by different datasets varies greatly, the invention uses the large datasets ETH-XGaze and Gaze360 to train a personalized feature extraction model and a feature extraction model, thereby improving the model's generalization ability in different environments and further improving the estimation accuracy of the gaze estimation model. Attached Figure Description

[0016] Figure 1 This is a flowchart of the line-of-sight estimation method according to an embodiment of the present invention; Figure 2 This is a structural block diagram of the line-of-sight estimation system according to an embodiment of the present invention; Figure 3 This is the personalized feature extraction network of the gaze estimation method in this embodiment of the invention; Figure 4 This is the feature extraction network of the gaze estimation method in this embodiment of the invention; Figure 5 This describes the network structure used in the line-of-sight estimation method of this invention. Detailed Implementation

[0017] The specific embodiments of the present invention will be described in further detail below with reference to the accompanying drawings and examples. The following examples are for illustrative purposes only and are not intended to limit the scope of the invention.

[0018] Example 1: like Figure 1 As shown, a preferred embodiment of the present invention provides a line-of-sight estimation method, which includes the following steps: S1: Obtain the target person image, and crop the target person image into the target person's eye image and the target person's face image to be estimated; S2: Input the eye image of the target person to be estimated into the personalized feature extraction model, and output a feature vector with personalized features. ; S3: Input the facial image of the target person to be estimated into the feature extraction model, and output a feature vector containing feature information closely related to gaze. ; S4: Transfer the feature vector and the feature vector The fusion process yields a feature vector containing personalized characteristics and features closely related to gaze. ; S5: Transfer the feature vector Input to MLP network to obtain horizontal orientation angle and vertical direction angle ; S6: Based on the horizontal direction angle and vertical direction angle The three-dimensional gaze direction vector is obtained, and the three-dimensional gaze direction vector is determined by the following formula: in( , , These constitute the three-dimensional line-of-sight direction vector. Line-of-sight estimation is performed based on the three-dimensional line-of-sight direction vector to obtain the estimation result.

[0019] This embodiment uses a personalized feature extraction model to extract personalized features from the left and right eye images of the target person, obtaining the corresponding feature vector W1. It also uses a feature extraction model to extract features from the face image of the target person, obtaining the corresponding feature vector W2. Facial features unrelated to gaze are removed while features related to gaze are retained. Finally, feature vectors W1 and W2 are concatenated and fused, and the fused feature vector W3 is input into an MLP network to obtain the horizontal direction angle G. yaw and vertical direction angle G pitch The invention can obtain a three-dimensional gaze direction vector based on the horizontal and vertical angles, and then perform gaze estimation based on the three-dimensional gaze direction vector, thereby improving the estimation accuracy of the gaze estimation model. The invention also adopts a cross-domain training model method. Since the environmental and human information collected by different datasets varies greatly, the invention uses the large datasets ETH-XGaze and Gaze360 to train a personalized feature extraction model and a feature extraction model, thereby improving the model's generalization ability in different environments and further improving the estimation accuracy of the gaze estimation model.

[0020] Example 2: like Figure 1 As shown, a preferred embodiment of the present invention provides a line-of-sight estimation method, which includes the following steps: S1: Obtain the target person image, and crop the target person image into the target person's eye image and the target person's face image to be estimated; In this embodiment, Eyediap and MPIIGaze are used as validation datasets. The validation datasets contain a large number of high-resolution images of different people in different motion postures. An image of a person is randomly selected from the validation dataset, and this image is cropped into an eye image and a face image. The images are then input into the corresponding model to obtain the gaze estimation result, and the gaze recognition accuracy is high.

[0021] S2: Input the eye image of the target person to be estimated into the personalized feature extraction model, and output a feature vector with personalized features. ; In this embodiment, the personalized feature extraction model is determined in the following way: Obtain the ETH-XGaze and Gaze360 training datasets. The ETH-XGaze training dataset contains 1 million high-resolution images of 110 people with different head poses, and the Gaze360 training dataset contains a large number of motion images of 238 people in different environments. Cropping the eyes from the images in the ETH-XGaze and Gaze360 training datasets yields training dataset A; S2.1: Construct a personalized feature extraction model; S2.2: Obtain training dataset A, which contains a large number of high-resolution eye images of different people with different head poses, and group the training dataset A by the person's name; S2.3: Randomly select a main eye image Img1 of a person from the training dataset A, and then, according to the person's name in the main eye image Img1, select an auxiliary eye image Img2 in the group to which it belongs, which has the same head posture as the main eye image Img1 but a different gaze direction. Then, select an auxiliary eye image Img3 in the grouped images of other people's eyes, which has a different head posture but the same gaze direction as the main eye image Img1. S2.4: The main eye image Img1, the auxiliary eye image Img2, and the auxiliary eye portrait 3 are cropped into corresponding left and right eye images, and then the corresponding left and right eye images are processed by an asymmetric network to extract personalized features, thereby obtaining corresponding 12-dimensional feature vectors V1, V2, and V3. S2.5: Process the feature vectors V1, V2, and V3 using a loss function to correct for gaze deviation. The loss function is determined as follows: in This is a personalized feature vector for the main image of the eye. This is a personalized feature vector representing the same person in the main image but from different viewing angles. To generate personalized feature vectors from different people at the same gaze angle as the main image. The loss function represents the optimization of individual personalized information. Loss functions for optimizing personalized information for different individuals The loss function is optimized for total personalized information; S2.6: Repeat steps S2.1 to S2.6, using the previous loss function for each training iteration. Optimize the next This leads to the final personalized feature extraction model.

[0022] S3: Input the facial image of the target person to be estimated into the feature extraction model, and output a feature vector containing feature information closely related to gaze. ; In this embodiment, the feature extraction model is determined in the following way: Obtain the ETH-XGaze and Gaze360 training datasets. The ETH-XGaze training dataset contains 1 million high-resolution images of 110 people with different head poses, and the Gaze360 training dataset contains a large number of motion images of 238 people in different environments. Cropping faces from images in the ETH-XGaze and Gaze360 training datasets yields training dataset B. S3.1: Construct a feature extraction model; S3.2: Obtain training dataset B, which contains a large number of high-resolution facial images of different people with different head poses, and group the training dataset B according to the people's names; S3.3: Extract a first face image with the same name as the main eye image 1 from the training dataset B, and input the first face image into the feature extraction network Backbone to obtain a corresponding feature vector V4; S3.4: Input the feature vector V4 into the SA-Module to inversely recover the second face image; S3.5: Optimize V4 using the total loss function, which is determined as follows: in For the generated line-of-sight angle, The viewing angle of the label data. The first face image. This is the second face image generated in reverse. Let be the loss function for gaze regression. To counteract the loss function of reverse graph generation, The loss function is optimized for the total personalized features. These three loss functions are then weighted according to different weights to obtain the total loss function of the feature extraction model. ,in These are weight hyperparameters; S3.6: Repeat steps S3.1 to S3.5, each time using the previous total loss function to optimize the next feature vector V4, and finally obtain the feature extraction model.

[0023] S4: Transfer the feature vector and the feature vector The fusion process yields a feature vector containing personalized characteristics and features closely related to gaze. ; In this embodiment, the feature vector It contains feature information closely related to the line of sight, while feature information unrelated to the line of sight is removed. Therefore, through the feature vector... and the feature vector Feature vectors obtained by fusion It contains features related to line of sight, which can improve the results of subsequent line of sight estimation.

[0024] S5: Transfer the feature vector Input to MLP network to obtain horizontal orientation angle and vertical direction angle ; In this embodiment, the MLP network is a multilayer neural network, namely a multilayer perceptron, which is a type of artificial neural network with a directional structure.

[0025] S6: Based on the horizontal direction angle and vertical direction angle The three-dimensional gaze direction vector is obtained, and the three-dimensional gaze direction vector is determined by the following formula: in( , , These constitute the three-dimensional line-of-sight direction vector. Line-of-sight estimation is performed based on the three-dimensional line-of-sight direction vector to obtain the estimation result.

[0026] In this embodiment, the horizontal direction angle is used. and vertical direction angle The three-dimensional gaze direction vector is calculated and obtained. Based on the three-dimensional direction vector, gaze estimation is performed, which improves the robustness and high accuracy of the gaze estimation model. Through 100 rounds of training experiments, it is shown that the accuracy is improved by 9.1% compared with the existing cross-environment gaze estimation methods.

[0027] Example 3: like Figure 2 As shown, a preferred embodiment of the present invention provides a line-of-sight estimation system, comprising: Acquisition module: Used to acquire images of the target person's eyes and face; Personalized feature extraction module: This module takes the eye image of the target person to be identified, inputs it into the personalized feature extraction model, and outputs a feature vector with personalized features. ; Feature extraction module: Used to input the facial image of the target person to be identified into the feature extraction model, and output a feature vector containing feature information closely related to gaze. ; Fusion module: used to fuse the feature vector and the feature vector The fusion process yields a feature vector containing personalized characteristics and features closely related to gaze. ; Recognition module: used to process the feature vector Input to MLP network to obtain horizontal orientation angle and vertical direction angle According to the horizontal direction angle and vertical direction angle The three-dimensional gaze direction vector is obtained, and the three-dimensional gaze direction vector is determined by the following formula: in( , , These constitute the three-dimensional line-of-sight direction vector. Line-of-sight estimation is performed based on the three-dimensional line-of-sight direction vector to obtain the estimation result.

[0028] This embodiment uses a personalized feature extraction model to extract personalized features from the left and right eye images of the target person, obtaining the corresponding feature vector W1. It also uses a feature extraction model to extract features from the face image of the target person, obtaining the corresponding feature vector W2. Facial features unrelated to gaze are removed while features related to gaze are retained. Finally, feature vectors W1 and W2 are concatenated and fused, and the fused feature vector W3 is input into an MLP network to obtain the horizontal direction angle G. yaw and vertical direction angle G pitch The invention can obtain a three-dimensional gaze direction vector based on the horizontal and vertical angles, and then perform gaze estimation based on the three-dimensional gaze direction vector, thereby improving the estimation accuracy of the gaze estimation model. The invention also adopts a cross-domain training model method. Since the environmental and human information collected by different datasets varies greatly, the invention uses the large datasets ETH-XGaze and Gaze360 to train a personalized feature extraction model and a feature extraction model, thereby improving the model's generalization ability in different environments and further improving the estimation accuracy of the gaze estimation model.

[0029] In summary, this embodiment of the invention uses a personalized feature extraction model to extract personalized features from the left and right eye images of the target person, obtaining corresponding feature vectors W1. It also uses a feature extraction model to extract features from the face image of the target person, obtaining corresponding feature vector W2. Facial features unrelated to gaze are removed while features related to gaze are retained. Finally, feature vectors W1 and W2 are concatenated and fused, and the fused feature vector W3 is input into an MLP network to ultimately obtain the horizontal direction angle G. yaw and vertical direction angle G pitch The invention can obtain a three-dimensional gaze direction vector based on the horizontal and vertical angles, and then perform gaze estimation based on the three-dimensional gaze direction vector, thereby improving the estimation accuracy of the gaze estimation model. The invention also adopts a cross-domain training model method. Since the environmental and human information collected by different datasets varies greatly, the invention uses the large datasets ETH-XGaze and Gaze360 to train a personalized feature extraction model and a feature extraction model, thereby improving the model's generalization ability in different environments and further improving the estimation accuracy of the gaze estimation model.

[0030] The above description is only a preferred embodiment of the present invention. It should be noted that for those skilled in the art, several improvements and substitutions can be made without departing from the technical principles of the present invention, and these improvements and substitutions should also be considered within the scope of protection of the present invention.

Claims

1. A line-of-sight estimation method, characterized in that, The method includes the following steps: S1: Obtain the target person image, and crop the target person image into the target person's eye image and the target person's face image to be estimated; S2: Input the eye image of the target person to be estimated into the personalized feature extraction model, and output a feature vector with personalized features. ; S3: Input the facial image of the target person to be estimated into the feature extraction model, and output a feature vector containing feature information closely related to gaze. ; S4: Transfer the feature vector and the feature vector The fusion process yields a feature vector containing personalized characteristics and features closely related to gaze. ; S5: Transfer the feature vector Input to MLP network to obtain horizontal orientation angle and vertical direction angle ; S6: Based on the horizontal direction angle and vertical direction angle The three-dimensional gaze direction vector is obtained, and the three-dimensional gaze direction vector is determined by the following formula: in( , , These constitute the three-dimensional line-of-sight direction vector. Line-of-sight estimation is performed based on the three-dimensional line-of-sight direction vector to obtain the estimation result. The personalized feature extraction model is determined in the following way: S2.1: Construct a personalized feature extraction model; S2.2: Obtain training dataset A, which contains a large number of high-resolution eye images of different people with different head poses, and group the training dataset A by the person's name; S2.3: Randomly select a main eye image Img1 of a person from the training dataset A, and then, according to the person's name in the main eye image Img1, select an auxiliary eye image Img2 in the group to which it belongs, which has the same head posture as the main eye image Img1 but a different gaze direction. Then, select an auxiliary eye image Img3 in the grouped images of other people's eyes, which has a different head posture but the same gaze direction as the main eye image Img1. S2.4: The main eye image Img1, auxiliary eye image Img2, and auxiliary eye image Img3 are cropped into corresponding left and right eye images, respectively. Then, the corresponding left and right eye images are processed by an asymmetric network to extract personalized features, thereby obtaining corresponding 12-dimensional feature vectors V1, V2, and V3. S2.5: Process the feature vectors V1, V2, and V3 using a loss function to correct for gaze deviation. The loss function is determined as follows: in This is the personalized feature vector of the main eye image. This is a personalized feature vector representing the same person in the main image but from different viewing angles. To generate personalized feature vectors from different people at the same gaze angle as the main image. The loss function represents the optimization of individual personalized information. Loss functions for optimizing personalized information for different individuals The loss function is optimized for total personalized information; S2.6: Repeat steps S2.1 to S2.6, using the previous loss function for each training iteration. Optimize the next This leads to the final personalized feature extraction model.

2. The line-of-sight estimation method according to claim 1, characterized in that, The training dataset A mentioned in step S2.2 is obtained in the following way: Obtain the ETH-XGaze and Gaze360 training datasets. The ETH-XGaze training dataset contains 1 million high-resolution images of 110 people with different head poses, and the Gaze360 training dataset contains a large number of motion images of 238 people in different environments. Cropping the eyes from the images in the ETH-XGaze and Gaze360 training datasets yields training dataset A.

3. The line-of-sight estimation method according to claim 2, characterized in that, The asymmetric networks mentioned in step S2.4 include: Base-CNN, FC1, and FC2. The Base-CNN consists of six convolutional layers connected in sequence, with the input of the first convolutional layer being the left-eye image. Or the right eye image, the output of the last convolutional layer is connected to FC1; FC1 is a fully connected layer with an output of 500. The input of FC1 is connected to the output of the last convolutional layer in Base-CNN, and the output of FC1 is connected to the input of FC2. The FC2 is a fully connected layer with a 12-dimensional output. The input of the FC2 is connected to the output of the FC1, and the output of the FC2 is a 12-dimensional feature vector.

4. The line-of-sight estimation method according to claim 2, characterized in that, The feature extraction model in step S3 is determined as follows: S3.1: Construct a feature extraction model; S3.2: Obtain training dataset B, which contains a large number of high-resolution facial images of different people with different head poses, and group the training dataset B according to the people's names; S3.3: Extract a first face image with the same name as the main eye image Img1 from the training dataset B, and input the first face image into the feature extraction network Backbone to obtain a corresponding feature vector V4; S3.4: Input the feature vector V4 into the SA-Module to inversely recover the second face image; S3.5: Optimize V4 using the total loss function, which is determined as follows: in For the generated line-of-sight angle, The viewing angle of the label data. The first face image. The second face image generated in reverse engineering. Let be the loss function for gaze regression. To counteract the loss function of reverse graph generation, The loss function is optimized for the total personalized features. These three loss functions are then weighted according to different weights to obtain the total loss function of the feature extraction model. ,in These are weight hyperparameters; S3.6: Repeat steps S3.1 to S3.5, each time using the previous total loss function to optimize the next feature vector V4, and finally obtain the feature extraction model.

5. The line-of-sight estimation method according to claim 4, characterized in that, The training dataset B mentioned in step 3.2 is obtained in the following way: Obtain the ETH-XGaze and Gaze360 training datasets. The ETH-XGaze training dataset contains 1 million high-resolution images of 110 people with different head poses, and the Gaze360 training dataset contains a large number of motion images of 238 people in different environments. Cropping faces from images in the ETH-XGaze and Gaze360 training datasets yields training dataset B.

6. The line-of-sight estimation method according to claim 5, characterized in that, The feature extraction network Backbone described in step S3.3 is composed of ResNet50.

7. The line-of-sight estimation method according to claim 5, characterized in that, The steps described in step S3.5 Weight hyperparameters, where The value ranges from 1.7 to 2.

3. Take a value of 0.7~1.

3. Take a value of 1.3 to 1.

8.

8. The line-of-sight estimation method according to claim 7, characterized in that, The weight hyperparameter Set to 2.0, weight hyperparameter Set to 1.0, weight hyperparameter Take 1.

5.

9. A gaze estimation system, characterized in that, The system is based on the method according to any one of claims 1 to 8, and the system comprises: Acquisition module: Used to acquire images of the target person's eyes and face; Personalized Feature Extraction Module: This module takes the target person's eye image as input into the personalized feature extraction model and outputs a feature vector with personalized features. ; Feature extraction module: This module takes the facial image of the target person as input to the feature extraction model and outputs a feature vector containing feature information closely related to gaze. ; Fusion module: used to fuse the feature vector and the feature vector The fusion process yields a feature vector containing personalized characteristics and features closely related to gaze. ; Recognition module: used to process the feature vector Input to MLP network to obtain horizontal orientation angle and vertical direction angle According to the horizontal direction angle and vertical direction angle The three-dimensional gaze direction vector is obtained, and the three-dimensional gaze direction vector is determined by the following formula: in( , , These constitute the three-dimensional line-of-sight direction vector. Line-of-sight estimation is performed based on the three-dimensional line-of-sight direction vector to obtain the estimation result.

Images