Eye movement tracking and positioning method based on main and auxiliary eyes
By establishing a simplified 3D eyeball model and visual axis mapping model, combined with binocular fusion technology, outputting a unique gaze point, solving the problem of low gaze accuracy in the existing technology, and achieving more efficient binocular eye tracking and positioning.
Patent Information
- Application Number
- CN202411091016.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-09
- Publication Date
- 2024-11-15
AI Technical Summary
Existing binocular eye tracking technology based on video only uses the average value of left and right eye gaze points as the final gaze point, resulting in insufficient gaze accuracy.
通过确定主视眼,建立简化3D眼球模型,建立光轴-视轴转换模型,结合用户标定建立视轴与屏幕注视点的映射模型,采集新眼图时求解视轴和注视点,采用双目 Fusion outputs the only gaze point, making full use of the analysis and positioning advantages of the main eye.
It effectively improves the accuracy of binocular eye movement tracking and positioning, making eye movement interaction more accurate and efficient, and is suitable for desktop, head-mounted, and glasses-style eye movement tracking systems.
Smart Images

Figure CN118968604A_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the technical field of video-based eye tracking, and in particular relates to an eye tracking and positioning method based on a primary eye and a secondary eye. Background Art
[0002] The existing video-based eye tracking technology is a two-dimensional platform eye image collected in real time by the left and right eye cameras. The two-dimensional plane eye features such as pupil, iris, Purkinje spot, and eye corner point are extracted by image processing-based methods. By establishing a mapping relationship between the two-dimensional eye features and the screen (virtual AR screen or computer screen), or establishing a three-dimensional eyeball model, when new two-dimensional plane eye features are collected, the left and right eye gaze points on the screen can be calculated according to the mapping relationship or the three-dimensional eyeball model. Due to the existence of errors, the left and right eye gaze points will hardly overlap. When performing binocular eye tracking and positioning, the existing method generally uses the average of the left and right eye gaze points as the final gaze point.
[0003] From a physiological point of view, human eyes are divided into the main eye and the auxiliary eye. The main eye, also called the fixation eye or the dominant eye, is responsible for the clear part of the field of vision and observes more detailed things. It is the eye that the brain habitually uses to analyze and locate objects, and is in a dominant position. The auxiliary eye is mainly responsible for the blurred part, and is more about perceiving the environment and observing the light and dark, and is in a subordinate position. Everyone has a dominant eye, which may be the left eye or the right eye, and the human brain habitually uses it for imaging analysis and locating objects. To further improve the accuracy of binocular eye tracking and positioning, the difference between the main and auxiliary eyes can be fully considered when using video-based eye tracking technology. Therefore, it is necessary to propose and design an eye tracking and positioning method based on the main and auxiliary eyes. Summary of the invention
[0004] Purpose of the Invention
[0005] In order to solve the problem that the existing video-based binocular eye tracking technology simply uses the average of the left and right eye gaze points as the final gaze point, resulting in insufficient gaze accuracy, the present invention provides an eye tracking and positioning method based on the main and auxiliary eyes.
[0006] Inventing technical solutions
[0007] An eye tracking and positioning method based on primary and secondary eyes comprises the following steps:
[0008] (1) Determine the dominant eye;
[0009] (2) Positioning the pupil ellipses of the left and right eyes based on the eye images collected in real time by the left and right eye cameras;
[0010] (3) Building a simplified 3D eyeball model based on pupil ellipse positioning information;
[0011] (4) Establishing an optical axis-visual axis conversion model, the two-dimensional pupil ellipse is reversely projected into a three-dimensional pupil circle, the line connecting the center of the eyeball and the center of the three-dimensional pupil circle is the optical axis, and then the visual axis is obtained based on the optical axis;
[0012] (5) Establish a mapping model between the visual axis and the screen gaze point through user calibration, and solve the mapping model parameters;
[0013] (6) When a new eye image is collected, the visual axis is solved by simplifying the 3D eyeball model, and then the left and right eye gaze points are solved through the mapping model;
[0014] (7) Binocular fusion, outputting a unique gaze point.
[0015] Preferably, the step of establishing a simplified 3D eyeball model includes:
[0016] 1) Solving the three-dimensional pupil circle, the three-dimensional pupil circle includes the normal vector of the three-dimensional pupil circle plane and the center of the pupil circle;
[0017] 2) Find the intersection of the normal vectors of a set number of pupil circles, which is the center of the eyeball in three-dimensional space;
[0018] 3) Based on the above three-dimensional pupil circle and the three-dimensional eyeball center, multiple eyeball radii are calculated, and the average value is taken as the final eyeball radius.
[0019] Preferably, in step 1), based on the pupil ellipse positioning information, a method of full perspective projection based on a pinhole camera model is used to solve the pupil circle in three-dimensional space.
[0020] Preferably, the number of pupil circles in step 2) is 10 to 100.
[0021] Preferably, in step (4), the optical axis is converted to obtain the visual axis according to the Kappa angle between the optical axis and the visual axis.
[0022] Preferably, the specific method of step (7) is: when the distance D between the main eye's gaze point (xz, yz) and the auxiliary eye's gaze point (xf, yf) is less than or equal to the threshold Dth1, the midpoint of the main and auxiliary eye's gaze points is taken as the only gaze point; when the distance D between the main eye's gaze point and the auxiliary eye's gaze point is greater than the threshold Dth1 and less than or equal to the threshold Dth2, the fused gaze point is close to the main viewpoint, and the only gaze point ((K-1)*xz / K+1*xf / K, (K-1)*yz / K+1*yf / K) is output, K=nD / Dth1, 2≤n≤20 and n is an integer; when the distance D between the main eye's gaze point and the auxiliary eye's gaze point is greater than the threshold Dth2, the main eye's gaze point (xz, yz) is taken as the only gaze point.
[0023] Preferably, the method of pupil ellipse positioning includes the processes of preprocessing, pupil edge segment screening, pupil edge segment combination reconstruction, and two-dimensional pupil ellipse confidence solution, and the pupil ellipse positioning information with the highest confidence is obtained.
[0024] Preferably, in step 2), the eye center in three-dimensional space is solved by least square method.
[0025] Preferably, the thresholds Dth1 and Dth2 are determined by setting the pixel distance or physical distance on the screen corresponding to the angle error in the line of sight direction as D 0 , 0.2°≤а≤2°, 0.5D 0 ≤Dth1<1.5D 0 <Dth2<n D 0 , 2≤n≤20 and n is an integer.
[0026] Advantages of the invention: The invention makes full use of the advantages of the dominant eye in analyzing and locating objects, and effectively improves the core technical indicator of the accuracy of binocular eye tracking and positioning based on video without changing the hardware, making eye movement interaction more accurate and efficient. The method can be promoted and applied to desktop, head-mounted, and glasses-type eye tracking systems, and has good application prospects. BRIEF DESCRIPTION OF THE DRAWINGS
[0027] Figure 1 It is a flow chart of an eye movement tracking and positioning method based on primary and auxiliary eyes of the present invention. DETAILED DESCRIPTION
[0028] The present invention is achieved through the following technical solutions.
[0029] An eye tracking and positioning method based on primary and secondary eyes comprises the following steps:
[0030] (1) Determine whether the dominant eye is L (left eye) or R (right eye).
[0031] (2) The left and right eye pupil ellipses are located based on the eye images collected by the left and right eye cameras in real time.
[0032] Among them, the left and right eye pupil ellipse positioning includes preprocessing, pupil edge segment screening, pupil edge segment combination reconstruction, two-dimensional pupil ellipse confidence solution, and the pupil ellipse positioning information with the highest confidence.
[0033] (3) A simplified 3D eyeball model is established based on a certain amount of pupil ellipse positioning information. In this embodiment, 30 valid pupil ellipse positioning information are used.
[0034] The steps to create a simplified 3D eyeball model include:
[0035] 1) Solving the pupil circle in three dimensions based on inverse projection. Based on the pupil ellipse positioning information, the pupil circle in three dimensions is solved by using the full perspective projection method based on the pinhole camera model, including the normal vector of the pupil circle plane in three dimensions and the center of the pupil circle;
[0036] 2) Solving the eyeball center based on the pupil circle in three-dimensional space. Based on a certain number of pupil circles in three-dimensional space, a certain algorithm is used to find the intersection of the normal vectors of these pupil circles, and the intersection is the eyeball center in three-dimensional space. Preferably, the number of pupil circles in three-dimensional space is 10 to 100. In this embodiment, the certain algorithm is the least squares method.
[0037] 3) Calculating the eyeball radius: Based on the above certain number of three-dimensional pupil circles and eyeball centers, multiple eyeball radii are calculated, and the average value is taken as the final eyeball radius.
[0038] 4) Construct a simplified 3D eyeball model. Construct a simplified 3D eyeball model based on the eyeball center and eyeball radius.
[0039] (4) Establish a conversion relationship model between the optical axis and the visual axis. Based on the reverse projection of the two-dimensional pupil ellipse into a three-dimensional pupil circle, the line connecting the center of the eyeball and the center of the three-dimensional pupil circle is the optical axis at this moment. Considering the Kappa angle between the optical axis and the visual axis, the visual axis can be obtained after conversion.
[0040] (5) User calibration: establish a mapping model between the visual axis and the screen gaze point, and solve the mapping model parameters.
[0041] (6) When a new eye image is collected, the visual axis is solved through the eyeball model, and then the left and right eye gaze points are solved through the mapping model.
[0042] (7) Binocular fusion, output a unique gaze point. Set the distance on the screen (pixel distance or physical distance) corresponding to the angle error of the line of sight direction а (0.2°≤а≤2°) to D 0 , set two distance thresholds Dth1 and Dth2, and 0.5D 0 ≤Dth1<1.5D 0 <Dth2<n D 0, 2≤n≤20 and n is an integer. When the distance between the main eye's gaze point (xz, yz) and the auxiliary eye's gaze point (xf, yf) is less than or equal to the threshold Dth1, the midpoint of the main and auxiliary eye's gaze points ((xz+xf) / 2, (yz+yf) / 2) is taken as the only gaze point. When the distance D between the main eye's and the auxiliary eye's gaze points is greater than the threshold Dth1 and less than or equal to Dth2, take the ratio K=nD / Dth1 (2≤n≤20 and n is an integer), then the fused gaze point is close to the main viewpoint, and the only gaze point ((K-1)*xz / K+1*xf / K, (K-1)*yz / K+1*yf / K) is output; when the distance between the main eye's and the auxiliary eye's gaze points is greater than the threshold Dth2, the main eye's gaze point (xz, yz) is taken as the only gaze point.
[0043] The above embodiments are only for illustrating the technical concept and features of the present invention, and their purpose is to enable technicians familiar with the technical field to understand the content of the present invention and implement it accordingly, and they cannot be used to limit the protection scope of the present invention. Any equivalent transformation or modification made according to the spirit and essence of the present invention should be included in the protection scope of the present invention. The technology, shape, and structure parts not described in detail in the present invention are all well-known technologies.
Claims
1. An eye tracking and positioning method based on primary and secondary eyes, It is characterized in that The steps include: (1) Determine the dominant eye; (2) Positioning the pupil ellipses of the left and right eyes based on the eye images collected in real time by the left and right eye cameras; (3) Building a simplified 3D eyeball model based on pupil ellipse positioning information; (4) Establishing an optical axis-visual axis conversion model, the two-dimensional pupil ellipse is reversely projected into a three-dimensional pupil circle, the line connecting the center of the eyeball and the center of the three-dimensional pupil circle is the optical axis, and then the visual axis is obtained based on the optical axis; (5) Establish a mapping model between the visual axis and the screen gaze point through user calibration, and solve the mapping model parameters; (6) When a new eye image is collected, the visual axis is solved by simplifying the 3D eyeball model, and then the left and right eye gaze points are solved through the mapping model; (7) Binocular fusion, outputting a unique gaze point.
2. The eye tracking and positioning method based on the main and auxiliary eyes as claimed in claim 1, It is characterized in that The steps to create a simplified 3D eyeball model include: 1) Solving the three-dimensional pupil circle, the three-dimensional pupil circle includes the normal vector of the three-dimensional pupil circle plane and the center of the pupil circle; 2) Find the intersection of the normal vectors of a set number of pupil circles, which is the center of the eyeball in three-dimensional space; 3) Based on the above three-dimensional pupil circle and the three-dimensional eyeball center, multiple eyeball radii are calculated, and the average value is taken as the final eyeball radius.
3. The eye tracking and positioning method based on the main and auxiliary eyes as claimed in claim 2, It is characterized in that In step 1), based on the pupil ellipse positioning information, the pupil circle in three-dimensional space is solved by using a full perspective projection method based on a pinhole camera model.
4. The eye tracking and positioning method based on the main and auxiliary eyes as claimed in claim 2, It is characterized in that In step 2), the number of pupil circles is 10 to 100.
5. The eye tracking and positioning method based on the main and auxiliary eyes as claimed in claim 1, It is characterized in that In step (4), the optical axis is converted to obtain the visual axis according to the Kappa angle between the optical axis and the visual axis.
6. The eye tracking and positioning method based on the main and auxiliary eyes as claimed in claim 1, It is characterized in that The specific method of step (7) is as follows: when the distance D between the main eye's gaze point (xz, yz) and the auxiliary eye's gaze point (xf, yf) is less than or equal to the threshold Dth1, the midpoint of the main and auxiliary eye's gaze points is taken as the only gaze point; when the distance D between the main eye's gaze point and the auxiliary eye's gaze point is greater than the threshold Dth1 and less than or equal to the threshold Dth2, the fused gaze point is close to the main viewpoint, and the only gaze point ((K-1)*xz / K+1*xf / K, (K-1)*yz / K+1*yf / K) is output, K=nD / Dth1, 2≤n≤20 and n is an integer; when the distance D between the main eye's gaze point and the auxiliary eye's gaze point is greater than the threshold Dth2, the main eye's gaze point (xz, yz) is taken as the only gaze point.
7. The eye tracking and positioning method based on the main and auxiliary eyes as claimed in claim 1, It is characterized in that The pupil ellipse positioning method includes preprocessing, pupil edge segment screening, pupil edge segment combination reconstruction, two-dimensional pupil ellipse confidence solution process, and the pupil ellipse positioning information with the highest confidence.
8. The eye tracking and positioning method based on the main and auxiliary eyes as claimed in claim 2, It is characterized in that In step 2), the eye center in three-dimensional space is solved by the least square method.
9. The eye tracking and positioning method based on the main and auxiliary eyes as claimed in claim 6, It is characterized in that The method for determining the thresholds Dth1 and Dth2 is as follows: set the pixel distance on the screen or the physical distance corresponding to the angle error in the line of sight direction as D 0 , 0.2°≤а≤2°, 0.5D 0 ≤Dth1<1.5D 0 <Dth2<n D 0 , 2≤n≤20 and n is an integer.
Citation Information
Patent Citations
System for treating visual dysfunction by adopting ocular dominance regulation
CN103239347A
Human eye tracking method and human eye tracking device
CN108427503A
Recognition method and device for gaze behavior in three-dimensional space and storage medium
CN111309144A
Method for determining fixation point of two eyes and electronic equipment
CN114581514A
Near-to-eye multi-camera eye movement tracking device and method independent of corneal reflection
CN117876426A
