Three-dimensional model line-of-sight direction detection method and terminal

Abstract

The application discloses a three-dimensional model visual line direction detection method and a terminal. The method comprises the following steps: acquiring basic information, then solving a corneal curvature radius based on the basic information; updating eye diagram information, and taking the corneal curvature radius as a constant to solve a corneal curvature center; taking the corneal curvature radius and the corneal curvature center as constants, solving a distance from the corneal curvature center to a pupil center based on the basic information and a pupil edge point coordinate; finally, calculating a Kappa angle of an eyeball; determining a primary and secondary eye, performing visual line fusion through the primary and secondary eye, and obtaining a final visual line direction. The binocular three-dimensional eyeball model is divided into two monocular eyeball models, and the solving parameters between each model only need to solve the human eye parameters related to the visual line, without mutual constraint of the parameters between the binoculars.

Description

Technical Field

[0001] This invention belongs to the field of gaze tracking technology, and in particular relates to a method and terminal for detecting gaze direction in a three-dimensional model. Background Technology

[0002] Eye-tracking technology tracks the movement of human eyes and uses corresponding gaze mapping algorithms to estimate the observer's visual path, obtaining information about the target area or gaze point of interest. Accurate gaze estimation is a crucial step, especially for professionals in the automotive and aviation industries, such as car and airplane pilots. Precise gaze estimation helps pilots perform screen control, meeting the needs of precise, fast, and natural human-computer interaction in screen-controlled environments. How to accurately detect the direction and point of a worker's gaze is a significant research area for human-computer interaction and driver screen control.

[0003] Currently, methods for line-of-sight estimation can be mainly divided into two categories: estimation methods based on two-dimensional mapping and estimation methods based on three-dimensional models.

[0004] Two-dimensional mapping-based methods primarily involve constructing a mapping model between the gaze point or gaze direction and user-calibrated or detected two-dimensional gaze parameters to estimate the gaze. These parameters include the corner of the eye, pupil position, eye center, and corneal curvature. While simple, this method suffers from low accuracy, susceptibility to interference, complex system configuration, and poor real-time performance. Solving for more complex polynomial coefficients leads to a more cumbersome user calibration process, which does not meet the requirements of free human-computer interaction. Three-dimensional model-based methods primarily determine eye features such as the eye center and radius by fitting a three-dimensional eye model, and then estimate the gaze by combining the geometric relationships between these features.

[0005] Methods based on 3D models rely on realistic models of the human eye, requiring the solution of numerous realistic eye parameters, such as corneal curvature and refractive index, and the distance between the center of the corneal reflected light spot and the pupil center. While complex to process, these methods offer high accuracy in gaze estimation, good real-time performance, and are less susceptible to interference, thus possessing broader application prospects. Building an eye model based on realistic human eye parameters requires solving a series of gaze-related eye parameters, such as the corneal radius of curvature R, the corneal center of curvature C, the pupil center P, the distance K from the corneal center of curvature C to the pupil center P, and the Kappa angle between the visual axis and the optical axis. Given the complexity of human eye parameters, each parameter influences the direction and point of gaze. Therefore, establishing a realistic 3D model of the human eye is the most common and effective method for improving the accuracy of gaze estimation.

[0006] While the binocular 3D eye model offers high accuracy, the solution process is complex, introducing too many facial and eye feature parameters and failing to consider the differences between the two eyes. The calculations are complex and the calibration is cumbersome, so the accuracy urgently needs to be improved. Summary of the Invention

[0007] Purpose of the invention

[0008] To address the challenges of complex solutions for existing 3D eyeball models, the excessive number of parameters to consider, and the lack of integration of primary and secondary eye gaze directions, this invention provides a method for detecting the gaze direction of a 3D model.

[0009] Invention Technology Solutions

[0010] A method for detecting the gaze direction of a 3D model includes the following steps:

[0011] Step S1: Obtain basic information, including camera optical center coordinates, coordinates of two light sources, and coordinates of two corneal reflection points corresponding to the two light sources in monocular vision;

[0012] Step S2: Calculate the corneal curvature radius based on the aforementioned basic information;

[0013] Step S3: Update the eye diagram information and use the corneal curvature radius as a constant to solve for the corneal curvature center;

[0014] Step S4: Using the corneal curvature radius and corneal curvature center as constants, calculate the distance from the corneal curvature center to the pupil center based on the basic information and the pupil edge point coordinates;

[0015] Step S5: Calculate the Kappa angle of the eyeball; determine the dominant and secondary eyes, and fuse the gaze using the dominant and secondary eyes to obtain the final gaze direction.

[0016] Preferably, the method for solving the corneal curvature radius in step S2 includes the following steps:

[0017] Step S21: By setting unknown coefficients, determine the two corneal reflection points q l1 and q l2 Corneal curvature center C l Indicate:

[0018] q l1 =O l +k l1 (u l1 -O l );

[0019] q l2 =O l +k l2 (u l2 -Ol );

[0020]

[0021] In the formula, k l1 k l2 and k l3 O represents the unknown coefficient. l U represents the coordinates of the camera's optical center. l1 and u l2 C represents the corneal reflection point in the three-dimensional world coordinate system obtained from the eye diagram. l Represents the coordinates of the corneal curvature center. The line of intersection representing the reflection planes of two light sources;

[0022] Step S22: Obtain the first reflected ray through the principle of reflection, and at the same time obtain the second reflected ray based on the corneal reflection point and the optical center; solve the unknown coefficients by using the principle that the first and second reflected rays are equal, and obtain the three-dimensional world coordinates of the two corneal reflection points and the corneal curvature center.

[0023] Step S23: Use the average radius obtained from the two light sources as the corneal curvature radius R. lc :

[0024] R lc =(||q l1 -C l ||+||q l2 -C l ||) / 2.

[0025] Preferably, step S22 includes:

[0026] Step S221: Based on the corneal curvature center C l The three-dimensional world coordinates q of the two corneal reflection points in the eyeball l1 q l2 The positional relationship yields the relevant unknown coefficient k. l3 Relationship:

[0027]

[0028] Step S222: Based on camera optical center coordinates O l and corneal curvature center C l Based on the positional relationship and the fact that the first and second reflected rays are equal, we can obtain:

[0029]

[0030] In the formula, i = 1, 2, l li Indicates the coordinates of the light source, O lIndicates the coordinates of the camera's optical center;

[0031] Step S223: Solve the system of equations using the bisection method, and based on the constraints of the three-dimensional world coordinates of the two corneal reflection points and the distance between the corneal curvature centers, obtain the value with the minimum distance error as the solution to the system of equations. The expression for the constraint relationship is as follows:

[0032] ||q l1 -C l ||=||q l2 -C l ||.

[0033] Preferably, step S3 includes the following steps: during the continuous movement of the eyeball, the eye map information is updated to obtain two updated corneal reflection points; based on the principle that the incident angle is equal to the reflection angle, and using the corneal curvature radius calculated in step S2 as a constant for calculation, and based on the principle that the distances from the three-dimensional world coordinates of the two corneal reflection points to the corneal curvature center are equal, the corneal curvature center is obtained by solving using the Newton iteration method.

[0034] Preferably, the method for obtaining the coordinates of the pupil edge point in step S4 includes:

[0035] Step S41: Represent the reflection point of the pupil edge on the cornea using the camera's optical center coordinates and the pupil edge point in the eye diagram:

[0036] r l1 =O l +k lr (v l1 -O l )

[0037] In the formula, r li O represents the reflection point of the pupil's edge on the cornea. l Represents the camera's optical center coordinates, v l1 k represents the pupil edge point in the eye diagram. lr Indicates the unknown coefficient;

[0038] Step S42: Solve for the unknown coefficient k by taking the distance between the corneal reflection point and the center of corneal curvature as equal to the corneal radius of curvature. lr :

[0039] ||r l1 -C l1 ||=R l ;

[0040] Step S43: Based on the principle of solving for the unit vector of the refracted ray and the unit vector of the normal in the refraction of light, the unit vector of the refracted ray is obtained.

[0041] Step S44: Calculate the three-dimensional coordinates of the pupil edge point in the eyeball:

[0042]

[0043] In the formula, k li Indicates the unknown coefficient;

[0044] Step S45: Based on the principle that the distance from the pupil edge point to the pupil center is equal, the selected multiple pupil edge points are fitted and solved. The least squares method is used to obtain the fitted circle, and the three-dimensional coordinates of the pupil edge point in the eyeball are obtained.

[0045] Preferably, the method for solving the distance from the corneal curvature center to the pupil center is as follows: obtain the pupil center by fitting a circle, and take the distance between the pupil center and the corneal curvature center at this moment as the K value.

[0046] Preferably, the method for calculating the Kappa angle of the eyeball includes the following steps:

[0047] Step S511: Calculate the unit vector of the optical axis based on the three-dimensional world coordinates of the corneal curvature center and the pupil center. Then, based on the observation point coordinates T and the corneal curvature center, the visual axis unit vector is calculated.

[0048] Step S512: Based on the optical axis unit vector and the line-of-sight unit vector obtained when observing the specified observation point, solve for the horizontal and vertical components of the optical axis and the horizontal and vertical components of the line-of-sight and optical axis, thus calibrating the Kappa angle.

[0049] Preferably, the method for determining the dominant and secondary eyes in step S5 includes the following steps: observing with both eyes separately, and then observing with both eyes, wherein the eye with the smaller distance deviation between monocular observation and binocular observation is the dominant eye.

[0050] Preferably, when the distance d between the lines of sight of the left and right eyes is... lr When the distance is less than or equal to the distance threshold d, the gaze point is the fusion of the gaze points of the left and right eyes. When the distance between the gaze points of the left and right eyes is d... lr When the distance is greater than the distance threshold d, the point where the dominant eye's line of sight falls is taken as the final point of sight.

[0051] Preferably, the method can be implemented using computer software.

[0052] A terminal includes a memory and a processor; the memory is used to store a computer program and a method for detecting the line-of-sight direction of a 3D model; the processor is used to execute the computer program and the method for detecting the line-of-sight direction of a 3D model.

[0053] Advantages of this invention:

[0054] 1) The present invention uses a binocular three-dimensional eye model with primary and secondary eye fusion, which can effectively estimate the gaze point. The dominant eye, also called the fixating eye or dominant eye, is responsible for the clear part of the field of vision and observes more detailed things because of its higher resolution. The secondary eye is the opposite, mainly responsible for the blurred part, such as observing the background and perceiving brightness and darkness, so it is highly sensitive to color. Therefore, the idea of ​​primary and secondary eye fusion can greatly improve the accuracy of gaze estimation.

[0055] 2) This invention divides the binocular 3D eye model into two monocular eye models. The solution parameters for each model only need to be solved for human eye parameters related to the line of sight, without needing to constrain the parameters between the two eyes. Dividing the binocular model into two monocular eye models can effectively avoid situations where the difference between the user's two eyes is too large. By using monocular calculations instead of constraining the parameters between the two eyes, this problem is effectively solved.

[0056] 3) When calculating monocular eyeball parameters, this invention does not require the introduction of additional facial feature parameters and human eye feature parameters for solving. Instead, it uses the positional relationship between the camera optical center coordinates and the corneal curvature center, as well as the three-dimensional world coordinates of two corneal reflection points and the distance between the corneal curvature centers to obtain the corresponding parameter range. Then, it uses the bisection method to solve for the values ​​of each parameter, which can reduce the number of parameters to be solved and the parameters to be calibrated, and quickly solve for individualized three-dimensional parameters of the human eye. Attached Figure Description

[0057] Figure 1 This is a schematic diagram of the method flow according to an embodiment of the present invention.

[0058] Figure 2 This is a schematic diagram of the eye diagram transformation relationship according to an embodiment of the present invention.

[0059] Figure 3 This is a schematic diagram showing the position coordinate relationship of monocular feature points in an embodiment of the present invention.

[0060] Figure 4 This is a schematic diagram of the coordinate relationship of a monocular camera in a three-dimensional world coordinate system according to an embodiment of the present invention.

[0061] Figure 5 This is a schematic diagram illustrating the relationship between the visual axis and the optical axis in an embodiment of the present invention.

[0062] Figure 6 This is a schematic diagram of the gaze fusion of two monocular three-dimensional eyeball models according to an embodiment of the present invention. Detailed Implementation

[0063] The present invention is achieved through the following technical solution.

[0064] like Figure 1 As shown, this embodiment of the invention provides a method for detecting the gaze direction of a three-dimensional model, including the following steps:

[0065] Step S1: Obtain basic information, which includes the camera optical center coordinates, the coordinates of the two light sources, and the coordinates of the two corneal reflection points corresponding to the two light sources in monocular vision.

[0066] In this embodiment of the invention, the two-dimensional pixel coordinates of the corneal reflective spot center, pupil center, and pupil boundary extracted from eye movement features are converted into three-dimensional world coordinates through the pixel coordinate and world coordinate conversion relationship in the system settings. The conversion relationship diagram is shown below. Figure 2 As shown, where X w Y w Z w The coordinate system is the world coordinate system, X c Y c Z c The coordinate system is the camera coordinate system, the XY coordinate system is the image physical coordinate system, and the UV coordinate system is the image pixel coordinate system.

[0067] Specifically, the Zhang Zhengyou calibration method is used to calibrate the camera using a checkerboard calibration board image, obtaining the camera's extrinsic and intrinsic parameter matrices. Based on the camera imaging model and the camera's extrinsic and intrinsic parameters, the transformation relationship from world coordinates to pixel coordinates is derived. Due to the uncertainty of depth information, the three-dimensional coordinates corresponding to pixels are not unique. Since the information used in subsequent solutions is a unit vector, the depth information of each pixel can be uniformly set to 1 for subsequent calculations, and the unit vector used for calculations is unaffected by the depth information. Thus, the two-dimensional parameter coordinates in the eye diagram are successfully converted into three-dimensional world coordinates for subsequent calculations, which is the basic information in this embodiment of the invention.

[0068] Step S2: Calculate the corneal curvature radius based on the basic information.

[0069] In this embodiment of the invention, the individualized three-dimensional pupil parameter, corneal curvature radius, is solved based on the camera's optical center coordinates, the coordinates of the two light sources, and the three-dimensional world coordinates of the two corneal reflection points converted from the eye diagram. The position coordinate relationship of each point in a single eye is as follows: Figure 3 As shown. Where C l q is the center point of corneal curvature. l1 and q l2 For the corneal reflex point of the eyeball, u l1 and u l2 For the corneal reflection point of the image, l l1 and l l2 O is the coordinate point of the light source. l Let R be the coordinates of the camera's optical center. l The radius of curvature of the cornea is denoted as .

[0070] In practical operation, based on the principle that the parametric coordinates of the eyeball in three-dimensional world coordinates are collinear with the three-dimensional coordinates of the camera optical center and the eye diagram, the coordinates of the camera optical center O are known. l The coordinates of the two light sources l l1 and l l2 And the corneal reflection point u in the three-dimensional world coordinate system obtained by transforming the eye image. l1 and u l2 The three-dimensional world coordinates q of the two corneal reflection points in the eyeball can be determined. l1 and q l2 , where k l1 and k l2 Unknown:

[0071] q l1 =O l +k l1 (u l1 -O l );

[0072] q l2 =O l +k l2 (u l2 -O l );

[0073] Furthermore, based on the properties of the law of reflection, the intersection line of the two light source planes can be obtained. Based on the intersection line of the plane containing the two light sources passing through the optical center coordinate O of the camera. l and corneal curvature center C l From the principle, we can know that:

[0074]

[0075] Where k l3 It is an unknown.

[0076] When solving for unknown coefficients, based on the principle that the unit vectors of the incident ray and the normal ray are known in the reflection of light, the reflected ray can be obtained. Alternatively, the reflected ray can be directly obtained from the corneal reflection point and the optical center. For monocular vision, based on the principle that the unit vectors of the reflected rays obtained from both methods are equal, solving only the corneal curvature radius of the monocular vision can yield the three-dimensional world coordinates of the two corneal reflection points and the corneal curvature center.

[0077] In the specific calculation, it is based on the corneal curvature center C. l The three-dimensional world coordinates q of the two corneal reflection points in the eyeball l1 and q l2 From the positional relationship, we can obtain information about k. l3 Relationship:

[0078]

[0079] Based on the camera optical center coordinates O l and corneal curvature center C l Based on the positional relationship and the fact that the unit vectors of the reflected rays are equal in both solution principles, we can obtain:

[0080]

[0081] In the formula, l li Represents the coordinates of the light source, C l O represents the coordinates of the corneal curvature center. l Indicates the coordinates of the camera's optical center;

[0082] Then based on With k l3 The constraint relationship thus reduces k l3 The solution range is determined, and then the bisection method is used to solve the above equations.

[0083] The constraints are based on the three-dimensional world coordinates of the two corneal reflection points and the distance between the centers of corneal curvature:

[0084] ||q l1 -C l ||=||q l2 -C l ||;

[0085] Finally, the value with the minimum distance error is obtained as the solution to the system of equations. Based on the three-dimensional world coordinates of the two corneal reflection points and the corneal curvature center obtained from the solution, the average radius obtained from the two light sources can be used as the corneal curvature radius. Taking the left eye as an example, the expression is as follows:

[0086] R lc =(||q l1 -C l ||+||q l2 -C l ||) / 2;

[0087] R lc This represents the radius of curvature of the cornea in the left eye.

[0088] Step S3: Update the eye diagram information and use the corneal curvature radius as a constant to solve for the corneal curvature center.

[0089] Step S2 yields the initial corneal curvature center and radius. However, due to continuous updates to the eye diagram, the position of the corneal curvature center also updates, requiring continuous calculation. Since the radius remains constant, this invention treats the radius as a known quantity when calculating the corneal curvature center.

[0090] Specifically, the corneal reflection point u in the three-dimensional world coordinate system obtained from the updated eye diagram. l1 u l2 This allows us to obtain the continuously updated three-dimensional world coordinates q of two corneal reflection points within the eyeball. l1 and q l2 Then, based on the principle that the incident angle equals the reflection angle, and using the corneal curvature radius obtained from the above solution as a constant for calculation, the three-dimensional world coordinates q of the two corneal reflection points in the eyeball are then used. l1 and q l2 to the corneal curvature center C l The distance between them is the radius of corneal curvature, and they are equal. The center of corneal curvature C is obtained by solving using Newton's iterative method. l The unknown k in l3 The corneal curvature center is ultimately obtained for subsequent line of sight estimation.

[0091] Step S4: Using the corneal curvature radius and corneal curvature center as constants, calculate the distance from the corneal curvature center to the pupil center based on the basic information and the pupil edge point coordinates; where v l1 r is the pupil edge point in the eye diagram. li The plane of the arc is the corneal plane, r li p is the reflection point of the pupil edge on the cornea. li Point at the edge of the pupil. P is the unit vector between the edge of the pupil and its corresponding corneal reflector. l C is the center point of the pupil. l Let K be the center point of corneal curvature, and the distance between them be K.

[0092] Specifically, based on the principle that the parametric coordinates of the eyeball in three-dimensional world coordinates are collinear with the three-dimensional coordinates of the camera optical center and the eye diagram, the coordinates of the camera optical center O are known. l The coordinates of the two light sources l l1 and l l2 And the pupil edge point v in the three-dimensional world coordinate system obtained by transforming the eye diagram. l1 Here, we take one of them as an example to illustrate the three-dimensional world coordinates of the reflection point of the pupil boundary in the cornea, where k lr Unknown:

[0093] r l1 =O l +k lr (v l1 -O l );

[0094] Based on the known quantities prior to this step, including the corneal radius of curvature, the center of corneal curvature, and the corneal reflector r...l1 The distance to the center of corneal curvature is the radius of corneal curvature. Therefore:

[0095] ||r l1 -C l1 ||=R l ;

[0096] Solving the system of equations consisting of the two relations above, we can obtain k. lr The value of .

[0097] Based on the principle of determining the unit vector of the refracted ray given the unit vector of the incident ray and the unit vector of the normal in the refraction of light, the refractive index is typically 1.3375, which is the combined effective refractive index of the aqueous humor and cornea. This allows us to obtain the unit vector of the refracted ray.

[0098] Therefore, the three-dimensional coordinates of the pupil edge point in the eyeball can be obtained:

[0099]

[0100] Where k li It is an unknown.

[0101] Next, based on the principle that the distance from the pupil edge points to the pupil center is equal, the selected pupil edge points are fitted to the pupil. The least squares method is used to obtain the fitted circle, and the center of the circle is taken as the pupil center. The distance between the pupil center and the corneal curvature center at this moment is taken as the K value. The unknown coefficient k can be obtained through the fitting operation. li The three-dimensional coordinates of the pupil edge point can then be obtained.

[0102] When solving for the pupil center, the method is similar to that used to solve for the pupil edge points. Specifically, the corneal curvature radius and the distance K from the corneal curvature center to the pupil center are solved once and used as constants. Based on the camera optical center coordinates, the coordinates of the two light sources, and the continuously updated three-dimensional world coordinates of the two corneal reflection points converted from the eye diagram, as well as the three-dimensional world coordinates of the pupil center, the three-dimensional pupil parameters corneal curvature center and pupil center are solved.

[0103] Based on the pupil center v in the 3D world coordinate system obtained by eye diagram transformation lp We can determine the three-dimensional world coordinates of the reflection point of the pupil center in the cornea, where k lrp Unknown:

[0104] r lp =O l +k lrp (v lp -O l );

[0105] Based on the known quantities prior to this step, including the corneal radius of curvature, the center of corneal curvature, and the corneal reflector r... lp The distance to the center of corneal curvature is the radius of corneal curvature. Therefore:

[0106] ||r lp -C l1 ||=R l ;

[0107] Solving the system of equations consisting of the two relations above, we can obtain r. lp The value of .

[0108] Based on the principle of determining the unit vector of the refracted ray from the known unit vectors of the incident ray and the normal ray in light reflection, where the refractive index is typically 1.3375 (the effective refractive index of the aqueous humor and cornea combined), the unit vector of the refracted ray can be obtained.

[0109] Therefore, the three-dimensional coordinates of the center of the pupil's edge point in the eyeball can be obtained:

[0110]

[0111] Where k lp It is an unknown.

[0112] Based on the known three-dimensional pupil parameters, the constant distance K from the corneal curvature center to the pupil center, and the corneal curvature center obtained at this moment, we can obtain:

[0113] ||P l -C l1 ||=K

[0114] Solving this formula quickly yields the three-dimensional world coordinates of the pupil center for subsequent calculations.

[0115] Step S5: Determine the parameters of the two eyeballs through steps S1 to S4, calculate the Kappa angle of the eyeballs; determine the primary and secondary eyes, and perform visual fusion through the primary and secondary eyes to obtain the final visual direction.

[0116] Specifically, based on the solution of the three-dimensional pupil parameters at this moment—the corneal curvature center and the pupil center—the individualized three-dimensional pupil parameter kappa angle is calculated. The relationship between the visual axis and the optical axis is as follows: Figure 5 As shown. Where X w Y w Z w The coordinate system is the world coordinate system, X c Y c Z c The coordinate system is the camera coordinate system, with point T being the observation point, and C... l P is the center point of corneal curvature. lLet θ be the center point of the pupil, the line connecting the center point of corneal curvature and the center point of the pupil be the optical axis, and let θ be the sum of the two angles. Let α and β be the horizontal and vertical components of the optical axis, respectively, and let α and β be the horizontal and vertical kappa angles that need to be solved.

[0117] During the solution process, the optical axis unit vector is calculated based on the three-dimensional world coordinates of the corneal curvature center and the pupil center obtained from the solution. Then, based on the observation point coordinates T and the corneal curvature center, the visual axis unit vector is calculated. Then, based on the optical axis unit vector and the line-of-sight unit vector obtained when observing a specified observation point, the horizontal and vertical components of the optical axis and the horizontal and vertical components of the line-of-sight and optical axis can be solved, thus calibrating the Kappa angle.

[0118] In this embodiment of the invention, the determination of the dominant and secondary eyes includes the following steps: observation is performed using both eyes, and then observation is performed using both eyes, wherein the eye with the smaller distance deviation between monocular and binocular observations is the dominant eye.

[0119] Next, the primary and secondary eyes are fused. Based on the above process, the corneal curvature radius, the distance from the corneal curvature center to the pupil center, and the Kappa angle are calculated as constant parameters of the human eye for both eyes, and two eyeball models are established respectively. Based on the two eyeball models, the left eye's line of sight point g is continuously updated and calculated when the user observes other target points. l and the point of gaze of the right eye g r .

[0120] Based on conventional methods used by those skilled in the art to solve for the points of interest, the unit vector of the optical axis is obtained. The optical axis is then corrected based on the calibrated Kappa angle to obtain the unit vector of the visual axis. The point of interest is then calculated based on the distance L between the target screen and the eyeball.

[0121] g l =C l +k gl v l

[0122] Where v l This is the corrected unit vector of the view axis. And k gl To calculate the distance relationships required for the points of interest:

[0123]

[0124] Where angles α and β are the horizontal and vertical angles of the kappa angle that need to be solved, and L is the distance between the target screen and the eyeball.

[0125] Then, based on the gaze points obtained from the left and right eye models, the gaze points of the primary and secondary eyes are fused. In this embodiment of the invention, the left eye is used as the primary visual eye to illustrate the formula:

[0126]

[0127] When the distance d between the points of focus of the left and right eyes lr When the distance is less than the distance threshold d, the point of gaze is the fusion of the points of gaze from the left and right eyes. When the distance between the points of gaze from the left and right eyes is d... lr When the distance is greater than the distance threshold d, the point of view of the dominant eye is g. l As the final point of focus.

[0128] Based on the coordinates of the corneal curvature centers of the left and right eyes and the final landing point, the final line of sight of the left and right eyes can be determined.

[0129] This method can be implemented using a computer program. This invention also provides a terminal, including a memory and a processor; the memory stores the computer program and the 3D model gaze direction detection method; the processor executes the computer program and the 3D model gaze direction detection method to implement the above-described method.

[0130] The technical features of the above embodiments can be combined arbitrarily. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described; only preferred embodiments of the present invention are illustrated. The descriptions are relatively specific and detailed, but they should not be construed as limiting the scope of the present invention. As long as the combination of these technical features does not contradict each other, it should be considered within the scope of this specification.

[0131] It should be noted that those skilled in the art can make various modifications and improvements without departing from the inventive concept, and these all fall within the scope of protection of this invention. Therefore, the scope of protection of this patent should be determined by the appended claims.

Claims

1. A method of detecting a line-of-sight direction of a three-dimensional model, characterized by, Includes the following steps: Step S1: Obtain basic information, including camera optical center coordinates, coordinates of two light sources, and coordinates of two corneal reflection points corresponding to the two light sources in monocular vision; Step S2: Calculate the corneal curvature radius based on the aforementioned basic information; Step S3: Update the eye diagram information and use the corneal curvature radius as a constant to solve for the corneal curvature center; Step S4: Using the corneal curvature radius and corneal curvature center as constants, calculate the distance from the corneal curvature center to the pupil center based on the basic information and the pupil edge point coordinates; Step S5: Determine the parameters of the two eyeballs through steps S1 to S4, and calculate the Kappa angle of the eyeballs; Determine the primary and secondary eyes, and fuse the gaze using the primary and secondary eyes to obtain the final gaze direction; The method for determining the dominant and secondary eyes in step S5 includes the following steps: observing with both eyes separately, then observing with both eyes, wherein the eye with the smaller distance deviation between monocular and binocular observations is the dominant eye; when the distance between the line of sight points of the left and right eyes... When the distance is less than or equal to the distance threshold d, the gaze point is the fusion of the gaze points of the left and right eyes. When the distance is greater than the distance threshold d, the point where the dominant eye's line of sight falls is taken as the final point of sight.

2. The method of claim 1, wherein, The method for determining the corneal curvature radius in step S2 includes the following steps: Step S21: Representing two corneal reflection points and , corneal curvature center by setting unknown coefficients. ； ； ； wherein , and denote unknown coefficients, denote camera optical center coordinates, and denote the corneal reflection points in the three-dimensional world coordinate system obtained by the conversion of the eye diagram, denote the corneal curvature center coordinates, denote the intersection line of the two light source reflection planes; Step S22: Obtain the first reflected ray through the principle of reflection, and at the same time obtain the second reflected ray based on the corneal reflection point and the optical center; solve the unknown coefficients by using the principle that the first and second reflected rays are equal, and obtain the three-dimensional world coordinates of the two corneal reflection points and the corneal curvature center. Step S23: average the radii obtained from the two light sources to obtain the corneal curvature radius : 。 3. The method of claim 2, wherein Step S22 includes: Step S221: based on the corneal curvature center and the position relationship between the three-dimensional world coordinates of the two corneal reflection points in the eyeball, the relationship formula about the unknown coefficients is obtained ​​ ； Step S222: based on the position relationship between the camera optical center coordinate and the corneal curvature center , and the first reflected light and the second reflected light are equal to obtain: where i = 1, 2, denotes the light source coordinate, denotes the camera optical center coordinate; Step S223: Solve the system of equations using the bisection method, and based on the constraints of the three-dimensional world coordinates of the two corneal reflection points and the distance between the corneal curvature centers, obtain the value with the minimum distance error as the solution to the system of equations. The expression for the constraint relationship is as follows: 。 4. The method for detecting the gaze direction of a three-dimensional model according to claim 1, characterized in that: Step S3 includes the following steps: During the continuous movement of the eyeball, the eye map information is updated to obtain two updated corneal reflection points; Based on the principle that the incident angle is equal to the reflection angle, and using the corneal curvature radius calculated in step S2 as a constant for calculation, and based on the principle that the distances from the three-dimensional world coordinates of the two corneal reflection points to the corneal curvature center are equal, the corneal curvature center is obtained by solving the problem using the Newton iteration method.

5. The method for detecting the gaze direction of a three-dimensional model as described in claim 1, characterized in that, The method for obtaining the coordinates of the pupil edge points in step S4 includes: Step S41: Represent the reflection point of the pupil edge on the cornea using the camera's optical center coordinates and the pupil edge point in the eye diagram: In the formula, This indicates the reflection point of the pupil's edge on the cornea. Indicates the coordinates of the camera's optical center. This represents the pupil edge point in the eye diagram. Indicates the unknown coefficient; Step S42: Solve for the unknown coefficients by finding that the distance between the corneal reflection point and the center of corneal curvature is equal to the corneal radius of curvature. : ； Step S43: Based on the principle of solving for the unit vector of the refracted ray and the unit vector of the normal in the refraction of light, the unit vector of the refracted ray is obtained. ; Step S44: Calculate the three-dimensional coordinates of the pupil edge point in the eyeball: ； In the formula, Indicates the unknown coefficient; Step S45: Based on the principle that the distance from the pupil edge point to the pupil center is equal, the selected multiple pupil edge points are fitted and solved. The least squares method is used to obtain the fitted circle, and the three-dimensional coordinates of the pupil edge point in the eyeball are obtained.

6. The method for detecting the gaze direction of a three-dimensional model as described in claim 1, characterized in that, The method to calculate the distance from the corneal curvature center to the pupil center is as follows: obtain the pupil center by fitting a circle, and take the distance between the pupil center and the corneal curvature center at this moment as the K value.

7. The method for detecting the gaze direction of a three-dimensional model as described in claim 1, characterized in that, The method for calculating the Kappa angle of the eyeball includes the following steps: Step S511: Calculate the unit vector of the optical axis based on the three-dimensional world coordinates of the corneal curvature center and the pupil center. Then, based on the observation point coordinates T and the corneal curvature center, the visual axis unit vector is calculated. ; Step S512: Based on the optical axis unit vector and the line-of-sight unit vector obtained when observing the specified observation point, solve for the horizontal and vertical components of the optical axis and the horizontal and vertical components of the line-of-sight and optical axis, thus calibrating the Kappa angle.

8. A terminal based on the three-dimensional model gaze direction detection method as described in any one of claims 1 to 7, characterized in that, It includes a memory and a processor; the memory is used to store the computer program and the 3D model line-of-sight direction detection method; the processor is used to execute the computer program and the 3D model line-of-sight direction detection method.

Images