Line of sight estimation method and device

Abstract

This invention provides a gaze estimation method and apparatus, relating to the field of autonomous driving technology. The method includes: acquiring a face image and eye images within the face image; inputting the face image and eye images into a preset gaze estimation model to obtain a first gaze and a second gaze output by the gaze estimation model; and determining a gaze estimation result based on the first gaze and the second gaze. The gaze estimation model is trained using face sample images and eye sample images as input, and sample gazes corresponding to the face sample images and eye sample images as labels. This invention not only uses eye images to estimate gaze but also combines face images and eye images to estimate gaze. Due to the assistance of facial features, it solves the problem of difficulty in accurately estimating gaze due to the influence of the surrounding lighting environment, achieving a more accurate estimation result.

Description

Technical Field

[0001] This invention relates to the field of autonomous driving technology, and in particular to a line-of-sight estimation method and apparatus. Background Technology

[0002] Gaze estimation has long been considered an important research topic due to its various promising real-world applications, such as gaze-based interfaces, center-point rendering in virtual reality, behavior analysis, and human-computer interaction. In Driver Monitor Systems (DMS), gaze estimation plays a crucial role in determining whether a driver's attention is distracted, and gaze is typically characterized by visual angles, specifically pitch (vertical angle) and yaw (horizontal angle).

[0003] Currently, in the DMS system, a gaze estimation model is trained using eye images. This model outputs the gaze based solely on eye features, i.e., the corresponding pitch and offset angles. However, in the cockpit, due to complex environments such as lighting or issues with camera quality, the lighting environment around the driver's eyes often changes suddenly, such as when sunlight shines directly around the eyes, causing the camera to fail to capture clear eye images. Therefore, it is difficult to accurately estimate the driver's gaze. Summary of the Invention

[0004] This invention provides a method and apparatus for estimating line of sight, which solves the problem in the prior art that when estimating line of sight solely by human eye, it is difficult to accurately estimate line of sight due to the influence of the surrounding light environment.

[0005] This invention provides a line-of-sight estimation method, comprising:

[0006] Obtain a face image and the eye image within the face image;

[0007] The face image and eye image are input into a preset gaze estimation model to obtain the first gaze and the second gaze output by the gaze estimation model;

[0008] Determine the line of sight estimation result based on the first line of sight and the second line of sight.

[0009] The gaze estimation model is trained using face sample images and eye sample images as input, and the gaze samples corresponding to the face sample images and eye sample images as labels.

[0010] The gaze estimation model is used to extract facial features and eye features from the face image and eye image respectively, and to estimate the first gaze and the second gaze based on two fused features composed of the facial features and eye features respectively.

[0011] According to a gaze estimation method provided by the present invention, the gaze estimation model includes: a first estimation sub-model and a second estimation sub-model, wherein inputting a face image and an eye image into a preset gaze estimation model to obtain a first gaze and a second gaze output by the gaze estimation model includes:

[0012] The face image is input into the first estimation sub-model to extract face features, and the eye image is input into the second estimation sub-model to extract eye features;

[0013] The first estimation sub-model acquires the eye features and fuses them with the face features to form a first fused feature; the second estimation sub-model acquires the face features and fuses them with the eye features to form a second fused feature.

[0014] The first estimation sub-model outputs the first line of sight based on the first fusion feature, and the second estimation sub-model outputs the second line of sight based on the second fusion feature.

[0015] According to a gaze estimation method provided by the present invention, the first fusion feature is: fusing facial features and eye features through an addition operation.

[0016] According to a gaze estimation method provided by the present invention, the second fusion feature is: fusing facial features and eye features through a multiplication operation.

[0017] According to a gaze estimation method provided by the present invention, before inputting a face image and an eye image into a preset gaze estimation model, the method further includes: training the gaze estimation model, the training process of which is as follows:

[0018] The face sample image and the eye sample image are input into the first estimation sub-model and the second estimation sub-model, respectively and simultaneously;

[0019] The first and second estimation sub-models extract face sample features and eye sample features, respectively;

[0020] The first estimation sub-model acquires the eye sample features and fuses them with the face sample features to form a first sample fusion feature; the second estimation sub-model acquires the face sample features and fuses them with the eye sample features to form a second sample fusion feature.

[0021] Based on the first sample fusion feature and the second sample fusion feature, the first estimation sub-model and the second estimation sub-model respectively output the first estimation result and the second estimation result;

[0022] Using the sample line of sight as a label, the first estimation result is substituted into a preset first loss function, and the second estimation result is substituted into a preset second loss function. Training is completed when both loss functions converge.

[0023] According to the line-of-sight estimation method provided by the present invention, both the first loss function and the second loss function are mean square loss functions:

[0024] ;

[0025] in, n Let i be the number of face sample images and i be the number of eye sample images, i = 1, 2, ... n , target For the first estimation sub-model, given the pitch and offset angles of the sample line of sight, pred For the first estimation result, the pitch angle and offset angle of the line of sight are given. For the second estimation sub-model, pred For the pitch and offset angles of the line of sight in the second estimation result, when pred close to target When the loss value is less than a predetermined threshold, the mean square loss function is considered to have converged, and training is complete.

[0026] According to a line-of-sight estimation method provided by the present invention, a first loss function and a second loss function are summed. When each loss function... pred close to target When the sum of the losses of the two loss functions is less than a predetermined threshold, the mean square loss function is considered to have converged, and training is complete.

[0027] According to a line-of-sight estimation method provided by the present invention, determining the line-of-sight estimation result based on a first line of sight and a second line of sight includes:

[0028] Calculate the average of the first line of sight and the second line of sight, and determine the average value as the line of sight estimation result.

[0029] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the program, implements the gaze estimation method as described above.

[0030] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the line-of-sight estimation method as described above.

[0031] The gaze estimation method and apparatus provided by this invention extract facial features and eye features from face images and eye images respectively through a preset gaze estimation model. The first gaze and the second gaze are estimated based on two fused features composed of the facial and eye features. The final gaze estimation result is then determined by the first and second gazes. That is, this invention does not only use eye images to estimate gaze, but combines face images and eye images to estimate gaze. Due to the assistance of facial features, even when the eye area is subject to external interference, a more reasonable gaze angle can be estimated based on the added face image. This solves the problem of difficulty in accurately estimating gaze due to the influence of the surrounding light environment, achieving a more accurate prediction effect. Attached Figure Description

[0032] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0033] Figure 1 This is a flowchart illustrating the line-of-sight estimation method provided by the present invention;

[0034] Figure 2 This is a schematic diagram of the line-of-sight estimation model principle in the line-of-sight estimation method provided by the present invention;

[0035] Figure 3 This is a flowchart of the line-of-sight estimation model operation in the line-of-sight estimation method provided by the present invention;

[0036] Figure 4 This is a schematic diagram of the line-of-sight estimation device provided by the present invention;

[0037] Figure 5 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0038] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0039] The line-of-sight estimation method of this invention, as shown in the embodiments, is as follows: Figure 1 As shown, it includes:

[0040] Step S110: Acquire a face image and an image of the eyes within the face image. In this step, a face image captured by a camera or other camera can be acquired. An infrared camera is preferred as it facilitates image acquisition in low-light environments. In the field of autonomous driving, a camera captures an image of the driver's head area within the cockpit. In this step, after acquiring the image of the driver's head area, a face image is identified based on this image, and the eye images within that face are also identified.

[0041] Step S120: Input the face image and the eye image within the face image into a preset gaze estimation model to obtain a first gaze and a second gaze output by the gaze estimation model. The gaze estimation model is trained using face sample images and eye sample images as input, and corresponding gaze samples as labels. The gaze estimation model is used to extract facial features and eye features from the face image and eye image respectively, and estimates the first gaze and the second gaze based on two fused features composed of the facial features and eye features. Specifically, the first gaze is represented by a first pitch angle and a first offset angle, and the second gaze is represented by a second pitch angle and a second offset angle.

[0042] Step S130: Determine the line of sight estimation result based on the first line of sight and the second line of sight to obtain the pitch angle and offset angle of the estimated line of sight.

[0043] The gaze estimation method provided in this invention extracts facial features and eye features from face and eye images respectively using a preset gaze estimation model. It then estimates a first gaze and a second gaze based on two fused features composed of these facial and eye features. Finally, the first and second gazes are used to determine the final gaze estimation result. This invention not only uses eye images to estimate gaze but also combines face and eye images. Due to the assistance of facial features, even when the eye area is subject to external interference, a more reasonable gaze angle can be estimated based on the added face image. This solves the problem of difficulty in accurately estimating gaze due to the influence of the surrounding lighting environment, achieving a more accurate gaze estimation effect. Experiments have shown that using the gaze estimation method of this embodiment significantly reduces false detections and improves accuracy by 9%.

[0044] like Figure 2 As shown, the gaze estimation model in this embodiment includes a first estimation sub-model and a second estimation sub-model. The two sub-models have the same model structure and run simultaneously during training and application. The first estimation sub-model is used to estimate a first gaze based on a face image, and the second estimation sub-model is used to estimate a second gaze based on an eye image. Based on the first and second estimation sub-models, the process of step S120 is as follows: Figure 3As shown, it includes:

[0045] Step S310: Input the face image into the first estimation sub-model to extract face features, and input the eye image into the second estimation sub-model to extract eye features. Specifically, the face image and eye image can be extracted by the convolutional layers in the two sub-models respectively.

[0046] Step S320: The first estimation sub-model acquires the eye features and fuses them with the face features to form a first fused feature. Specifically, the eye features extracted by the second estimation sub-model and the face features extracted by the first estimation sub-model are fused according to a first predetermined method. The first estimation sub-model then performs subsequent calculations based on the first fused feature. The second estimation sub-model acquires the face features and fuses them with the eye features to form a second fused feature. Specifically, the face features extracted by the first estimation sub-model and the eye features extracted by the second estimation sub-model are fused according to a second predetermined method. The second estimation sub-model then performs subsequent calculations based on the second fused feature.

[0047] Step S330: The first estimation sub-model outputs the first line of sight based on the first fusion feature, and the second estimation sub-model outputs the second line of sight based on the second fusion feature.

[0048] In this embodiment, the gaze estimation model is designed as two gaze estimation sub-models, which can not only extract the facial features of the face image and the eye features of the eye image at the same time, but also facilitate feature fusion after feature extraction. After feature fusion, the two estimation sub-models output the first gaze and the second gaze respectively. The two gaze estimation sub-models can be convolutional neural network (CNN) models.

[0049] In this embodiment, the first fusion feature is: fusing face features (feature_face) and eye features (feature_eye) through an addition operation, i.e., feature_face + feature_eye. In the first fusion feature, face features are used as global features, and eye features are used as local features. Face features themselves include eye features. By using an addition operation, eye features extracted from the eye image are added to the face features, thereby strengthening the eye features without destroying other feature information in the face, making the first gaze output based on the first fusion feature more accurate.

[0050] In this embodiment, the second fusion feature is: fusing face features (feature_face) and eye features (feature_eye) through a multiplication operation, i.e., feature_face × feature_eye. In the second fusion feature, face features are used as global features and eye features are used as local features. The multiplication operation is used to multiply the face features by the eye features extracted from the eye image. Thus, without destroying the eye features in the face features, the remaining features in the face features other than the eyes are multiplied by the eye features extracted from the eye image. This is equivalent to adding the weight of the eye features to the remaining features in the face features, making the second gaze output according to the second fusion feature more accurate.

[0051] Since the first and second lines of sight are more accurate, the line-of-sight estimation results determined based on the first and second lines of sight are also more accurate.

[0052] In this embodiment, before step S120, the method further includes: training the gaze estimation model, and the training process is as follows:

[0053] The face sample image and the eye sample image are input into the first estimation sub-model and the second estimation sub-model, respectively and simultaneously. Of course, the eye sample image is extracted from the corresponding face sample image.

[0054] The first and second estimation sub-models extract face sample features and eye sample features, respectively.

[0055] The first estimation sub-model acquires the eye sample features and fuses them with the face sample features to form a first sample fusion feature. The second estimation sub-model acquires the face sample features and fuses them with the eye sample features to form a second sample fusion feature.

[0056] Based on the first sample fusion feature and the second sample fusion feature, the first estimation result and the second estimation result are obtained respectively.

[0057] Using the sample line of sight as a label, the first estimation result is substituted into a preset first loss function, and the second estimation result is substituted into a preset second loss function. Training ends when both loss functions converge.

[0058] Both loss functions are mean squared loss (MSE):

[0059] ;

[0060] in, n This refers to the number of face sample images and the number of eye sample images. i=1,2,…, n , target For the first estimation sub-model, given the pitch and offset angles of the sample line of sight (i.e., the label), pred For the first estimation result, the pitch angle and offset angle of the line of sight are given. For the second estimation sub-model, pred These represent the pitch and offset angles of the line of sight in the second estimation result. During training, when... pred close to target When the loss value is less than a predetermined threshold, the MSE loss function is considered convergent. The predetermined threshold can be set according to the actual situation. Convergence of the MSE loss function indicates that training is complete. Furthermore, since the two estimating sub-models have essentially the same structure and use the same two loss functions, they have the same convergence trend. To ensure that both estimating sub-models complete training simultaneously, the two loss functions are added. When the sum of the two loss values ​​converges, the training of both estimating sub-models is complete, that is, the training of the entire gaze estimation model is complete. Specifically, when the sum of the two loss functions converges, the training of the entire gaze estimation model is complete. pred close to target When the sum of the losses of the two loss functions is less than a predetermined threshold, the mean square loss function is considered to have converged, and training is complete.

[0061] In this embodiment, step S130 includes: calculating the average value of the first line of sight and the second line of sight, and determining the average value as the line of sight estimation result. Of course, a weighted average or other methods can also be used to determine the final line of sight estimation result. Using the average value as the line of sight estimation result not only reduces the computational load but also yields a line of sight estimation result with high accuracy obtained through experiments.

[0062] The line-of-sight estimation device provided by the present invention is described below. The line-of-sight estimation device described below can be referred to in correspondence with the line-of-sight estimation method described above.

[0063] like Figure 4 As shown, the line-of-sight estimation device provided by the present invention includes:

[0064] The face image acquisition module 410 is used to acquire face images and eye images within the face images.

[0065] The model running module 420 is used to input the face image and eye image into a preset gaze estimation model to obtain the first gaze and the second gaze output by the gaze estimation model.

[0066] The line-of-sight determination module 430 is used to determine the line-of-sight estimation result based on the first line of sight and the second line of sight.

[0067] The gaze estimation model is trained using face sample images and eye sample images as input, and the gaze samples corresponding to the face sample images and eye sample images as labels.

[0068] The gaze estimation model is used to extract facial features and eye features from the face image and eye image respectively, and to estimate the first gaze and the second gaze based on two fused features composed of the facial features and eye features respectively.

[0069] The gaze estimation device provided by this invention extracts facial features and eye features from face images and eye images respectively through a preset gaze estimation model. It estimates the first gaze and the second gaze based on two fused features composed of the facial features and eye features. Finally, it determines the estimated gaze result by combining the first gaze and the second gaze. That is, this invention does not only use eye images to estimate gaze, but combines face images and eye images to estimate gaze. With the assistance of facial features, it solves the problem that it is difficult to accurately estimate gaze due to the influence of the surrounding light environment, and achieves a more accurate estimation effect.

[0070] Optionally, the line-of-sight estimation model includes: a first estimation sub-model and a second estimation sub-model, and the model running module 420 includes:

[0071] The feature extraction module is used to input the face image into the first estimation sub-model to extract face features, and to input the eye image into the second estimation sub-model to extract eye features.

[0072] The feature fusion module is used for the first estimation sub-model to obtain the eye features and fuse them with the face features to form a first fused feature, and for the second estimation sub-model to obtain the face features and fuse them with the eye features to form a second fused feature.

[0073] A gaze output module is used for the first estimation sub-model to output the first gaze based on the first fusion feature, and the second estimation sub-model to output the second gaze based on the second fusion feature.

[0074] Optionally, the gaze estimation device of the present invention further includes: a model training module, used to input the face sample image and the eye sample image into a first estimation sub-model and a second estimation sub-model respectively and simultaneously; the first estimation sub-model and the second estimation sub-model extract face sample features and eye sample features respectively; the first estimation sub-model obtains the eye sample features and fuses them with the face sample features to form a first sample fusion feature, the second estimation sub-model obtains the face sample features and fuses them with the eye sample features to form a second sample fusion feature; based on the first sample fusion feature and the second sample fusion feature, the first estimation sub-model and the second estimation sub-model respectively output a first estimation result and a second estimation result; using the sample gaze as a label, along with the first estimation result, it is input into a preset first loss function, using the sample gaze as a label, along with the second estimation result, it is input into a preset second loss function, and training is completed when both loss functions converge.

[0075] Optionally, both the first loss function and the second loss function are mean square loss functions:

[0076] ;

[0077] in, n Let i be the number of face sample images and i be the number of eye sample images, i = 1, 2, ... n , target For the first estimation sub-model, given the pitch and offset angles of the sample line of sight, pred For the first estimation result, the pitch angle and offset angle of the line of sight are given. For the second estimation sub-model, pred For the pitch and offset angles of the line of sight in the second estimation result, when pred close to target When the loss value is less than a predetermined threshold, the mean square loss function is considered to have converged, and training is complete.

[0078] Furthermore, summing the first loss function and the second loss function, when each loss function, pred close to target When the sum of the losses of the two loss functions is less than a predetermined threshold, the mean square loss function is considered to have converged, and training is complete.

[0079] The gaze estimation method and apparatus of the present invention are not limited to use in the field of autonomous driving to determine the driver's gaze, but can also be used in other practical application fields that require the determination of human gaze, such as: gaze-based human-computer interaction, which can determine whether the gaze is on a button on the display screen, or whether the gaze is on a certain part of the display screen, and can magnify the image at that point so that the observer can see it more clearly.

[0080] Figure 5 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 5 As shown, the electronic device may include: a processor 510, a communication interface 520, a memory 530, and a communication bus 540, wherein the processor 510, the communication interface 520, and the memory 530 communicate with each other through the communication bus 540. The processor 510 can call logical instructions in the memory 530, including a line-of-sight estimation method, which includes:

[0081] Obtain a face image and the eye image within the face image.

[0082] The face image and eye image are input into a preset gaze estimation model to obtain the first gaze and the second gaze output by the gaze estimation model.

[0083] The line of sight estimation result is determined based on the first line of sight and the second line of sight.

[0084] The gaze estimation model is trained using face sample images and eye sample images as input, and the gaze samples corresponding to the face sample images and eye sample images as labels.

[0085] The gaze estimation model is used to extract facial features and eye features from the face image and eye image respectively, and to estimate the first gaze and the second gaze based on two fused features composed of the facial features and eye features respectively.

[0086] Furthermore, the logical instructions in the aforementioned memory 530 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0087] On the other hand, the present invention also provides a computer program product, the computer program product comprising a computer program that can be stored on a non-transitory computer-readable storage medium, wherein when the computer program is executed by a processor, the computer is capable of executing the gaze estimation method provided by the above methods, the method comprising:

[0088] Obtain a face image and the eye image within the face image.

[0089] The face image and eye image are input into a preset gaze estimation model to obtain the first gaze and the second gaze output by the gaze estimation model.

[0090] The line of sight estimation result is determined based on the first line of sight and the second line of sight.

[0091] The gaze estimation model is trained using face sample images and eye sample images as input, and the gaze samples corresponding to the face sample images and eye sample images as labels.

[0092] The gaze estimation model is used to extract facial features and eye features from the face image and eye image respectively, and to estimate the first gaze and the second gaze based on two fused features composed of the facial features and eye features respectively.

[0093] In another aspect, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to perform the line-of-sight estimation methods provided by the methods described above, the method comprising:

[0094] Obtain a face image and the eye image within the face image.

[0095] The face image and eye image are input into a preset gaze estimation model to obtain the first gaze and the second gaze output by the gaze estimation model.

[0096] The line of sight estimation result is determined based on the first line of sight and the second line of sight.

[0097] The gaze estimation model is trained using face sample images and eye sample images as input, and the gaze samples corresponding to the face sample images and eye sample images as labels.

[0098] The gaze estimation model is used to extract facial features and eye features from the face image and eye image respectively, and to estimate the first gaze and the second gaze based on two fused features composed of the facial features and eye features respectively.

[0099] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0100] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0101] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A line-of-sight estimation method, characterized in that, include: Obtain a face image and the eye image within the face image; The face image and eye image are input into a preset gaze estimation model to obtain the first gaze and the second gaze output by the gaze estimation model; Determine the line of sight estimation result based on the first line of sight and the second line of sight. The gaze estimation model is trained using face sample images and eye sample images as input, and the gaze samples corresponding to the face sample images and eye sample images as labels. The gaze estimation model is used to extract facial features and eye features from the face image and eye image respectively, and to estimate the first gaze and the second gaze based on two fusion features composed of the facial features and eye features respectively; The gaze estimation model includes a first estimation sub-model and a second estimation sub-model. The face image and eye image are input into the preset gaze estimation model to obtain the first and second gazes output by the gaze estimation model, including: The face image is input into the first estimation sub-model to extract face features, and the eye image is input into the second estimation sub-model to extract eye features; The first estimation sub-model acquires the eye features and fuses them with the face features in a first predetermined manner to form a first fused feature; the second estimation sub-model acquires the face features and fuses them with the eye features in a second predetermined manner to form a second fused feature. The first estimation sub-model outputs the first line of sight based on the first fusion feature, and the second estimation sub-model outputs the second line of sight based on the second fusion feature.

2. The line-of-sight estimation method according to claim 1, characterized in that, The first fusion feature is: fusing facial features and eye features through an addition operation.

3. The line-of-sight estimation method according to claim 1, characterized in that, The second fusion feature is: fusing facial features and eye features through a multiplication operation.

4. The line-of-sight estimation method according to claim 1, characterized in that, Before inputting the face image and eye image into the preset gaze estimation model, the process further includes: training the gaze estimation model, the training process of which is as follows: The face sample image and the eye sample image are input into the first estimation sub-model and the second estimation sub-model, respectively and simultaneously; The first and second estimation sub-models extract face sample features and eye sample features, respectively; The first estimation sub-model acquires the eye sample features and fuses them with the face sample features to form a first sample fusion feature; the second estimation sub-model acquires the face sample features and fuses them with the eye sample features to form a second sample fusion feature. Based on the first sample fusion feature and the second sample fusion feature, the first estimation sub-model and the second estimation sub-model respectively output the first estimation result and the second estimation result; Using the sample line of sight as a label, the first estimation result is substituted into a preset first loss function, and the second estimation result is substituted into a preset second loss function. Training is completed when both loss functions converge.

5. The line-of-sight estimation method according to claim 4, characterized in that, Both the first and second loss functions are mean square loss functions: ； in, n This refers to the number of face sample images and the number of eye sample images. i =1,2,…, n , target For the first estimation sub-model, given the pitch and offset angles of the sample line of sight, pred For the first estimation result, the pitch angle and offset angle of the line of sight are given. For the second estimation sub-model, pred For the pitch and offset angles of the line of sight in the second estimation result, when pred close to target When the loss value is less than a predetermined threshold, the mean square loss function is considered to have converged, and training is complete.

6. The line-of-sight estimation method according to claim 5, characterized in that, Summing the first and second loss functions, when each loss function... pred close to target When the sum of the losses of the two loss functions is less than a predetermined threshold, the mean square loss function is considered to have converged, and training is complete.

7. The line-of-sight estimation method according to any one of claims 1 to 6, characterized in that, The step of determining the line-of-sight estimation result based on the first line of sight and the second line of sight includes: Calculate the average value of the first line of sight and the second line of sight, and determine the average value as the line of sight estimation result.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the line-of-sight estimation method as described in any one of claims 1 to 7.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the line-of-sight estimation method as described in any one of claims 1 to 7.

Images