3D sight line estimation method and system based on eyeball model

Abstract

The invention discloses a 3D sight line estimation method and system based on an eyeball model, and the method comprises the steps: obtaining an internal reference matrix of a current camera and an ellipse parameter of a pupil contour, and constructing a multi-frame ellipse set which comprises the center coordinates of an ellipse, the lengths of long and short axes, and a rotation angle; according to the short axis direction of each ellipse in the ellipse set, calculating a projection point of an eyeball center on an image plane; on the basis of the camera internal reference matrix and the eyeball center projection point, constructing an eyeball model corresponding to each ellipse in the ellipse set, and obtaining model features and a line-of-sight direction; and based on the eyeball model feature of each ellipse in the ellipse set, determining whether the eyeball model feature of the current frame is outlier, and if so, marking the current frame as an abnormal frame. According to the method, efficient fitting of the eyeball model and accurate estimation of the gazing direction are realized through few-sample calculation and sparse sampling strategies.

Description

Technical Field

[0001] This invention belongs to the field of 3D gaze estimation technology, and particularly relates to a 3D gaze estimation method and system based on an eye model. Background Technology

[0002] 3D gaze estimation is a computer vision technique that maps two-dimensional eye features to three-dimensional gaze vectors. It is widely used in virtual reality, augmented reality, driving behavior analysis, psychological research, and intelligent interactive devices. Eye-model-based 3D gaze estimation methods construct a geometric model of the eye that conforms to human anatomy to physically model the gaze direction. This approach offers advantages such as strong interpretability and generalization ability, making it one of the mainstream 3D gaze estimation methods. However, current eye-model-based 3D gaze estimation methods share a common problem: sliding error. This is because fitting the eye model incurs high computational costs and cannot be updated in real time. Consequently, when the camera slides relative to the eye, the eye model cannot be corrected immediately, resulting in significant sliding errors.

[0003] Existing technologies include: 3D gaze estimation methods based on eye models: estimating gaze direction by fitting an eye model; this method utilizes eye or facial image features for modeling and possesses good interpretability and personalized adaptability; 3D gaze estimation methods based on corneal reflection: estimating the corneal center position using corneal glints from infrared light sources or geometric inference, thereby achieving 3D pupil localization; and 3D gaze estimation methods based on deep learning: decoupling invariant features (such as eye radius and position) from variable features (such as gaze direction, pupil, and iris) through neural networks, directly obtaining 3D gaze estimation results from eye images. However, the computational cost of fitting the eye model is high, making it difficult to meet the needs of real-time applications. The corneal reflection-based method relies on infrared light source arrays and has poor robustness in natural lighting environments. The deep learning-based method has high model training costs and limited generalization ability to different environmental conditions and shooting angles. Therefore, there is an urgent need to propose a 3D gaze estimation method and system based on eye models. Summary of the Invention

[0004] To address the aforementioned technical problems, this invention proposes a 3D gaze estimation method and system based on an eye model. Through few-sample computation and sparse sampling strategies, it achieves efficient fitting of the eye model and accurate estimation of the gaze direction.

[0005] To achieve the above objectives, this invention provides a 3D gaze estimation method based on an eyeball model, comprising:

[0006] Obtain the current camera intrinsic parameter matrix and the ellipse parameters of the pupil contour, and construct a multi-frame ellipse set, wherein the ellipse set includes the center coordinates, major and minor axis lengths, and rotation angles of the ellipse;

[0007] Calculate the projection point of the eyeball center onto the image plane based on the direction of the minor axis of each ellipse in the set of ellipses;

[0008] Based on the camera intrinsic parameter matrix and the eyeball center projection point, construct the eyeball model corresponding to each ellipse in the ellipse set to obtain model features and gaze direction;

[0009] Based on the eye model features of each ellipse in the ellipse set, it is determined whether the eye model features of the current frame are outliers. If they are outliers, the current frame is marked as an abnormal frame.

[0010] On the other hand, to achieve the above objectives, the present invention also provides a 3D gaze estimation system based on an eyeball model, comprising:

[0011] The first acquisition module is used to acquire the current camera intrinsic parameter matrix and the ellipse parameters of the pupil contour, and construct a multi-frame ellipse set, wherein the ellipse set includes the center coordinates, major and minor axis lengths and rotation angles of the ellipse;

[0012] The second fitting module is used to calculate the projection point of the eyeball center on the image plane based on the minor axis direction of each ellipse in the set of ellipses.

[0013] The third fitting module is used to construct an eye model corresponding to each ellipse in the ellipse set based on the camera intrinsic parameter matrix and the projection point of the eyeball center, and to obtain the model features and the gaze direction;

[0014] The fourth determination module is used to determine whether the eye model features of the current frame are outliers based on the eye model features of each ellipse in the ellipse set. If they are outliers, the current frame is marked as an abnormal frame.

[0015] Technical advantages of this invention: This invention discloses a 3D gaze estimation method and system based on an eye model, which has high gaze estimation accuracy (error not exceeding 1°), fast model fitting speed (time not exceeding 0.05ms), and a sliding detection mechanism based on fitting error to filter out bad frames caused by camera sliding, thereby improving system stability and the reliability of estimation results. Attached Figure Description

[0016] The accompanying drawings, which form part of this application, are used to provide a further understanding of this application. The illustrative embodiments and descriptions of this application are used to explain this application and do not constitute an undue limitation of this application. In the drawings:

[0017] Figure 1This is a flowchart illustrating the 3D gaze estimation method based on an eyeball model according to an embodiment of the present invention.

[0018] Figure 2 This is a schematic diagram of the structure of the 3D gaze estimation system based on an eyeball model according to an embodiment of the present invention;

[0019] Figure 3 This is a schematic diagram illustrating the acquisition of the projection point of the eyeball's center onto the image plane according to an embodiment of the present invention;

[0020] Figure 4 This is a schematic diagram of the spatial geometric relationship of the eyeball model according to an embodiment of the present invention;

[0021] Figure 5 This is a schematic diagram of the polar angle and azimuth angle in the polar coordinate system of the eyeball according to an embodiment of the present invention. Detailed Implementation

[0022] It should be noted that, unless otherwise specified, the embodiments and features described in this application can be combined with each other. This application will now be described in detail with reference to the accompanying drawings and embodiments.

[0023] It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases the steps shown or described may be executed in a different order than that shown here.

[0024] like Figure 1 As shown, this embodiment provides a 3D gaze estimation method based on an eyeball model, including:

[0025] Obtain the current camera intrinsic parameter matrix and the ellipse parameters of the pupil contour, and construct a multi-frame ellipse set, wherein the ellipse set includes the center coordinates, major and minor axis lengths, and rotation angles of the ellipse;

[0026] Calculate the projection point of the eyeball center onto the image plane based on the direction of the minor axis of each ellipse in the set of ellipses;

[0027] Based on the camera intrinsic parameter matrix and the eyeball center projection point, construct the eyeball model corresponding to each ellipse in the ellipse set to obtain model features and gaze direction;

[0028] Based on the eye model features of each ellipse in the ellipse set, it is determined whether the eye model features of the current frame are outliers. If they are outliers, the current frame is marked as an abnormal frame.

[0029] Furthermore, the projection point of the eyeball center onto the image plane is obtained by calculating the minimum distance to the line containing the minor axis of each ellipse in the ellipse set using the least squares method.

[0030] Furthermore, the eyeball model corresponding to each ellipse is obtained, including the following steps:

[0031] Based on the camera intrinsic parameter matrix, the incident vectors from the camera optical center to the projection point of the eyeball center on the image plane and the center of the ellipse are obtained respectively.

[0032] The angle formed by the center of the eyeball, the optical center of the camera, and the center of the pupil is obtained based on the incident vector;

[0033] Based on the ratio of the major and minor axes of the ellipse, obtain the angle formed by the center of the pupil, the center of the eyeball, and the optical center of the camera;

[0034] Based on the sine theorem, the ratio of the eyeball center to the camera optical center and the ratio of the pupil center to the eyeball center are obtained, which are the eyeball model features corresponding to the ellipse.

[0035] Furthermore, the eye model of the set includes the projection point of the eyeball center on the image plane, the rotation matrix from the spherical coordinate system to the camera coordinate system, and the average value of the ratio after removing outliers.

[0036] Furthermore, the gaze direction and inference error of the current frame are obtained, including the following steps:

[0037] Based on the camera intrinsic parameter matrix, the incident vectors from the camera optical center to the projection point of the eyeball center on the image plane and the center of the ellipse are obtained respectively.

[0038] The angle formed by the eyeball center, the camera optical center, and the pupil center, and the azimuth angle of the pupil center in the spherical coordinate system are obtained based on the incident vector.

[0039] Based on the ratio of the major and minor axes of the ellipse, obtain the polar angle of the pupil center in the spherical coordinate system;

[0040] Based on the law of sine, the ratio of the eyeball center to the camera optical center and the ratio of the pupil center to the eyeball center are obtained;

[0041] Calculate the error value between the ratio and the eye model. If the error value is higher than a set threshold, mark the current frame as an abnormal frame.

[0042] Based on the rotation matrix, the gaze direction of the current frame is obtained.

[0043] Furthermore, the error value is calculated as ε=|γ-γ p | / γ, where γ p γ is the ratio obtained by ellipse fitting in the current frame, and γ is the ratio of the eyeball model.

[0044] Furthermore, after fitting the eye model and obtaining the gaze direction of the current frame, the 3D gaze estimation method includes:

[0045] Obtain the display page, create a display window for the display page, and display the eye model, gaze direction, and fitting error through the display window.

[0046] Specifically, the implementation process of this embodiment includes:

[0047] Application process of 3D gaze estimation:

[0048] 1. First Reasoning Module

[0049] 1.1 Obtaining the camera intrinsic parameter matrix

[0050] The camera intrinsic parameter matrix K is usually represented as:

[0051]

[0052] Among them, (f x ,f y (c) represents focal length multiplied by pixel density (usually in pixels), x ,c y () represents the coordinates of the image center;

[0053] The camera intrinsic parameter matrix can be obtained from the image sensor's product manual or calibrated using OpenCV.

[0054] 1.2 Obtaining the Ellipse Parameters of the Pupil Contour

[0055] A two-dimensional pupil detection method is applied to an eye image dataset to obtain the ellipse parameters of the pupil contour. The results are saved in JSON format. The detection results for each frame include the following: frame index, whether a pupil is detected, coordinates of the ellipse center, lengths of the major and minor axes, and rotation angle of the ellipse.

[0056] 1.3 Constructing a multi-frame ellipse set

[0057] Starting from the current frame, ellipses are selected and added to the set by moving forward at a certain step size. This sparse sampling strategy can significantly reduce computational overhead while effectively avoiding excessive similarity in the geometric positions of the selected ellipses, thus improving the stability of model fitting.

[0058] like Figure 3 The second fitting module shown

[0059] On the image pixel plane, construct the line containing the minor axis of each ellipse in the set of ellipses;

[0060] The point where the sum of all straight-line distances is minimized is the projection point E′ of the eyeball's center onto the image plane, obtained by using the least squares method.

[0061] like Figure 4 The third fitting module shown

[0062] 3.1 Constructing an eyeball model

[0063] For any pixel (u, v) on the image plane, its incident space vector can be obtained through the camera intrinsic parameter matrix:

[0064]

[0065] Given the projection point E′ of the eyeball's center onto the image plane and the projection point P′ of the pupil's center onto the image plane (the center of the ellipse), the included angle can be obtained.

[0066] Based on the ratio of the major and minor axes of the ellipse, the angle θ2 formed by the pupil center P, the eyeball center E, and the camera optical center O can be obtained as cos -1 (b / a);

[0067] The characteristics of the eyeball model can be obtained using the law of sine:

[0068]

[0069] 4.2 Calculate the direction of sight

[0070] In the spherical coordinate system of the eye model, the polar angle θ and the azimuth angle can be used. To describe the positional relationship between the center of the pupil and the center of the eyeball, where θ = cos -1 (b / a),

[0071] The line-of-sight direction is represented by a unit vector in spherical coordinates:

[0072]

[0073] Since the z-axis of the spherical coordinate system is the incident vector d of the eyeball's center, a rotation matrix can be constructed as follows:

[0074]

[0075] in,

[0076] The final line of sight direction is represented by a vector in the camera coordinate system:

[0077] v global =R·v local .

[0078] like Figure 5 The fourth determination module shown

[0079] 4.1 Calculate the fitting error

[0080] Obtain the lower quartile Q1 and the upper quartile Q3 of the eye model features in the ellipse set, and calculate the interquartile range IQR = Q3 - Q1;

[0081] If the eye model feature γ of the current frame is greater than Q3 + IQR or γ < Q1 - IQR, it is determined as an outlier.

[0082] Alternative 1: 3D gaze estimation method based on corneal reflection

[0083] Use a near-eye camera and two infrared light sources that are collinear, and utilize their same geometric plane characteristics for 3D gaze estimation.

[0084] Alternative 2: 3D gaze estimation method based on deep learning

[0085] A spatial decoupling network with a double-branch decoding architecture is proposed to decouple invariant features (such as eye radius, position) and variable features (such as gaze direction, pupil, iris) in the latent space.

[0086] As Figure 2 shown, in this embodiment, a 3D gaze estimation system based on an eye model is further provided, including:

[0087] A first acquisition module, configured to acquire the current camera internal parameter matrix and the elliptical parameters of the pupil contour, and construct a multi-frame ellipse set, where the ellipse set includes the center coordinates, the major and minor axis lengths, and the rotation angle of the ellipse;

[0088] A second fitting module, configured to calculate the projection point of the eye center on the image plane according to the minor axis direction of each ellipse in the ellipse set;

[0089] A third fitting module, configured to construct an eye model corresponding to each ellipse in the ellipse set based on the camera internal parameter matrix and the projection point of the eye center, and obtain the model features and the gaze direction;

[0090] A fourth determination module, configured to determine whether the eye model features of the current frame are outliers based on the eye model features of each ellipse in the ellipse set. If they are outliers, mark the current frame as an abnormal frame.

[0091] Specifically, the input of the first acquisition module is the current camera internal parameter matrix and the elliptical parameters of the pupil contour, and the output is a multi-frame ellipse set;

[0092] The input of the second fitting module is the multi-frame ellipse set, and the output is the projection point of the eye center on the image plane;

[0093] The input of the third fitting module is the multi-frame ellipse set and the projection point of the eye center, and the output is the eye model features and the gaze direction;

[0094] The fourth inference module takes eyeball model features as input and outputs whether it is an outlier.

[0095] The above are merely preferred embodiments of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A 3D gaze estimation method based on an eyeball model, characterized in that, include: Obtain the current camera intrinsic parameter matrix and the ellipse parameters of the pupil contour, and construct a multi-frame ellipse set, wherein the ellipse set includes the center coordinates, major and minor axis lengths, and rotation angles of the ellipse; Calculate the projection point of the eyeball center onto the image plane based on the direction of the minor axis of each ellipse in the set of ellipses; Based on the camera intrinsic parameter matrix and the eyeball center projection point, construct the eyeball model corresponding to each ellipse in the ellipse set to obtain model features and gaze direction; Based on the eye model features of each ellipse in the ellipse set, it is determined whether the eye model features of the current frame are outliers. If they are outliers, the current frame is marked as an abnormal frame.

2. The 3D gaze estimation method based on an eyeball model as described in claim 1, characterized in that, The process of constructing a multi-frame ellipse set includes: Starting from the current frame, obtain the ellipse parameters of the image frame forward according to the set step size and sampling number.

3. The 3D gaze estimation method based on an eyeball model as described in claim 1, characterized in that, The process of calculating the projection point of the eyeball's center onto the image plane includes: Construct the line containing the minor axis of each ellipse in the set of ellipses; The point with the smallest sum of all the straight-line distances is found using the least squares method and is taken as the projection point of the eyeball center on the image plane.

4. The 3D gaze estimation method based on an eyeball model as described in claim 1, characterized in that, The process of constructing an eye model includes: Based on the camera intrinsic parameter matrix, calculate the incident vector from the camera optical center to the projection point of the eyeball center and the center of the ellipse; Calculate the angle formed by the center of the eyeball, the optical center of the camera, and the center of the pupil based on the incident vector; Calculate the angle formed by the pupil center, the eyeball center, and the camera optical center based on the ratio of the major and minor axes of the ellipse; The ratio of the distance between the center of the eyeball and the optical center of the camera, and the ratio of the distance between the center of the pupil and the center of the eyeball, are calculated based on the sine theorem and used as features of the eyeball model.

5. The 3D gaze estimation method based on an eyeball model as described in claim 4, characterized in that, The process of calculating the direction of the line of sight includes: Calculate the polar angle and azimuth angle in the eye coordinate system; Based on the incident vector from the camera optical center to the projection point of the eyeball center, calculate the transformation matrix from the eye coordinate system to the camera coordinate system. Based on the transformation matrix, the vector representation of the viewing direction in the camera coordinate system is calculated.

6. The 3D gaze estimation method based on an eyeball model as described in claim 1, characterized in that, The process of determining whether the eye model features of the current frame are outliers includes: Obtain the lower and upper quartiles of the eyeball model features in the elliptic set, and calculate the interquartile range; If the eye model feature of the current frame is greater than the upper quartile plus the interquartile range or less than the lower quartile minus the interquartile range, it is determined to be an outlier.

7. The 3D gaze estimation method based on an eyeball model as described in claim 1, characterized in that, The method further includes: acquiring a display page, creating a display window for the display page, and displaying the eye model, gaze direction, and fitting error through the display window.

8. A system for estimating 3D gaze based on an eye model according to any one of claims 1-7, characterized in that, include: The first acquisition module is used to acquire the current camera intrinsic parameter matrix and the ellipse parameters of the pupil contour, and construct a multi-frame ellipse set, wherein the ellipse set includes the center coordinates, major and minor axis lengths and rotation angles of the ellipse; The second fitting module is used to calculate the projection point of the eyeball center on the image plane based on the minor axis direction of each ellipse in the set of ellipses. The third fitting module is used to construct an eye model corresponding to each ellipse in the ellipse set based on the camera intrinsic parameter matrix and the projection point of the eyeball center, and to obtain the model features and the gaze direction; The fourth determination module is used to determine whether the eye model features of the current frame are outliers based on the eye model features of each ellipse in the ellipse set. If they are outliers, the current frame is marked as an abnormal frame.

Images