Method and device for automatically calibrating Kappa angle of eyeball

Abstract

The invention discloses a method and device for automatically calibrating a Kappa angle of an eyeball, and the method comprises the steps: collecting the watching parameters of the eyeball of a user through a single camera and double infrared light sources which are disposed in a head-mounted device; calculating the corneal curvature center and the optical axis of the eyeballs of the user according to the fixation parameters; calculating a Kappa angle according to the cornea curvature center and the optical axis; wherein the Kappa angle is an included angle between a visual axis and an optical axis of eyeballs of the user; calculating the visual axis of the eyeballs of the user according to the optical axis and the Kappa angle; according to the method, the process of explicit calibration of a traditional sight tracking method is omitted, calibration of the Kappa angle can be implicitly completed in the tracking stage, and accurate, efficient and automatic tracking of the sight of the user is achieved.

Description

Technical Field

[0001] The present invention relates to the field of visual tracking, and in particular to a method and a device for automatically calibrating the Kappa angle of an eyeball. Background Art

[0002] In the field of visual tracking, the gaze tracking method uses existing electronic, mechanical, optical and other methods to detect the direction of the human eye's gaze or the point of gaze. It is widely used in cognitive process research, assisted driving, psychology and psychiatry research, marketing and advertising, virtual reality, human-computer interaction and other fields. In addition, in recent years, VR technology has made great progress, VR and AR wearable devices are constantly updated, providing users with a variety of immersive experiences, and gaze tracking technology has gradually become an important part of VR technology. The gaze tracking method can reduce the parallax between the real and virtual worlds, and can provide users with a more realistic and natural VR application experience. However, it is difficult to cope with the free movement of the head and the drastic changes in lighting conditions, and the calibration procedure is cumbersome and inconvenient for the subjects, and it is easy to make mistakes when the subjects do not see the expected place due to fatigue or lack of concentration.

[0003] Therefore, how to automatically track the user's line of sight has become a technical problem that technical personnel in this field urgently need to solve. Summary of the invention

[0004] The purpose of the present invention is to provide a method and device for automatically calibrating the Kappa angle of an eyeball, which can automatically track the user's line of sight accurately and efficiently.

[0005] According to one aspect of the present invention, a method for automatically calibrating the Kappa angle of an eyeball is provided, the method comprising: Using a single camera and dual infrared light sources provided in the head mounted device, the gaze parameters of the user's eyes are collected; Calculating the corneal curvature center and the optical axis of the user's eyeball according to the gaze parameter; Calculating the Kappa angle according to the corneal curvature center and the optical axis; wherein the Kappa angle is the angle between the visual axis of the user's eyeball and the optical axis; The visual axis of the user's eyeball is calculated according to the optical axis and the Kappa angle.

[0006] Optionally, calculating the center of corneal curvature of the user's eyeball according to the gaze parameter includes: Create a three-dimensional coordinate system and set the optical center of the single camera The point is set as the origin of the three-dimensional coordinate system of the system; In the three-dimensional coordinate system of the system, according to the dual infrared light sources and The coordinate position of the dual infrared light source is calculated based on the parameters of the single camera. and Two imaging points presented on the imaging surface of the single camera and The coordinate position of the dual infrared light source and Through two reflection points on the outer surface of the cornea of ​​the user's eyeball and After reflection, and at the same time through the optical center After the point, it intersects with the imaging surface at two imaging points and ; According to the optical center of the single camera in the three-dimensional coordinates of the system The coordinate position of the point, the dual infrared light source and The coordinate position and two imaging points and The coordinate position of the user is used to calculate the corneal radius of the user Corneal curvature center The coordinate position of a point in the three-dimensional coordinate system of the system.

[0007] Optionally, the optical center of the single camera in the three-dimensional coordinates of the system The coordinate position of the point, the dual infrared light source and The coordinate position and two imaging points and The coordinate position of the user is used to calculate the corneal radius of the user Corneal curvature center The coordinate position of a point in the three-dimensional coordinate system of the system includes: The user's corneal radius is calculated by the following formula: Corneal curvature center The coordinate position of the point in the three-dimensional coordinate system of the system: ; ; ; ; ; Among them, ,at this time, , and h represents the center of corneal curvature Point to the optical center of the single camera The modulus of the point distance.

[0008] Optionally, calculating the optical axis of the user's eyeball according to the gaze parameter includes: The pupil center of the user is calculated by the following formula: The coordinate position of the point in the three-dimensional coordinate system, and according to the pupil center Corneal curvature center Point calculates the optical axis of the user's eyeball: ; ; ; ; ; The light passes through the outer surface of the cornea of ​​the user's eyeball. After refraction, the point reaches the outer edge of the pupil of the user's eyeball point, and and are the incident angle and the outgoing angle during the refraction process.

[0009] and are the refractive index of light in the air and the refractive index of light inside the cornea; The point is any point on the edge of the pupil of the user's eyeball, and any point on the edge of the pupil to the center of the pupil The distance between the points is the same, j∈[1,200]

[0010] is the unit vector in the incident direction during refraction; for Point and The modulus of the point distance.

[0011] Optionally, calculating the Kappa angle according to the corneal curvature center and the optical axis includes: The left eye rotation axis and the left eye rotation angle are calculated according to the left eye optical axis collected for the i-th time and the left eye optical axis collected for the (i+1)th time, and the left eye rotation matrix is ​​calculated according to the left eye rotation axis and the left eye rotation angle. ; The right eye rotation axis and the right eye rotation angle are calculated according to the right eye optical axis collected for the i-th time and the right eye optical axis collected for the (i+1)th time, and the right eye rotation matrix is ​​calculated according to the right eye rotation axis and the right eye rotation angle. ; Establish the left eyeball coordinate system of the user; wherein the center of the left eyeball corneal curvature is Set as the origin of the left eyeball coordinate system, and set the visual axis direction of the left eye collected for the i-th time as the left eyeball coordinate system axis; Establish the user's right eyeball coordinate system; wherein the center of the right eyeball corneal curvature is Set as the origin of the right eyeball coordinate system, and set the right eye visual axis direction collected for the i-th time as the right eyeball coordinate system axis; The Kappa angle of the user's left eyeball and the Kappa angle of the right eyeball are calculated jointly according to the following formula: ; ; ; ; ; in, and Respectively represent the horizontal component and vertical component of the Kappa angle of the user's left eyeball; Respectively represent the horizontal component and vertical component of the Kappa angle of the user's right eyeball; represents the unit vector in the direction of the visual axis collected for the i-th time in the left eyeball and the right eyeball of the user, ; represents the normal of the plane formed by the two visual axes in the left eyeball and the right eyeball of the user during the (i+1)th acquisition; represents the unit vector in the direction of the visual axis acquired for the (i+1)th time in the left eyeball of the user; represents the unit vector in the direction of the visual axis acquired for the (i+1)th time in the right eyeball of the user.

[0012] Optionally, after calculating the Kappa angle according to the corneal curvature center and the optical axis, the method further includes: Step 1: Based on the N groups of initial Kappa angles calculated from the N collected data, the 0th generation population is formed. { }, and set the 0th generation population as the target population; among them, individual is the Kappa angle of the left eye calculated based on the i-th collected data And the Kappa angle of the right eye The four-dimensional vector formed; ; Step 2: input each individual in the target population into a preset objective function in turn, and set the individual corresponding to the minimum value of the objective function as the target individual; Wherein, the objective function is: ; is the distance between the two visual axes in the i-th acquisition data, and ; Step 3: Determine whether the preset stop iteration rule is reached, if yes, execute step 5, if no, execute step 4; wherein the preset stop iteration rule is: the minimum value of the objective function is less than the first preset threshold, or the number of generations of the target population reaches the second preset threshold; Step 4: Update the target population according to the preset differential evolution algorithm to obtain the t-th generation population, set the t-th generation population as the target population, and re-execute step 2; Step 5: Form the final Kappa angle of the user's eyeball according to the target individuals in the latest generation population.

[0013] Optionally, the preset stop iteration rule includes: The boundary vector is obtained according to the following formula : ; in, ; ,

[0014] , , Is randomly selected from Individuals selected from the generation population; Mutation Operator is a real constant factor; The intermediate vector is formed according to the following formula : ; in,

[0015] is the crossover probability

[0016] for A random real number; for A random integer; According to the following formula, the individuals in the t-th generation population are formed : .

[0017] In order to achieve the above object, the present invention also provides a device for automatically calibrating the Kappa angle of an eyeball, the device comprising: A collection module, used to collect the gaze parameters of the user's eyes using a single camera and dual infrared light sources provided in the head mounted device; An optical axis module, used to calculate the corneal curvature center and the optical axis of the user's eyeball according to the gaze parameter; An angle module, used to calculate the Kappa angle according to the corneal curvature center and the optical axis; wherein the Kappa angle is the angle between the visual axis of the user's eyeball and the optical axis; A visual axis module is used to calculate the visual axis of the user's eyeball based on the optical axis and the Kappa angle.

[0018] In order to achieve the above-mentioned purpose, the present invention also provides a computer device, which specifically includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, and when the processor executes the computer program, the steps of the method for automatically calibrating the Kappa angle of the eyeball introduced above are implemented.

[0019] In order to achieve the above-mentioned object, the present invention also provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the steps of the method for automatically calibrating the Kappa angle of the eyeball introduced above are implemented.

[0020] The method and device for automatically calibrating the Kappa angle of the eye provided by the present invention collect the gaze parameters of the user's eye by utilizing a single camera and dual infrared light sources arranged in the head-mounted device; calculate the corneal curvature center and the optical axis of the user's eye according to the gaze parameters; calculate the Kappa angle according to the corneal curvature center and the optical axis; calculate the visual axis of the user's eye according to the optical axis and the Kappa angle, and the user does not need to complete the gaze of the calibration point at a specific time, thereby improving the consistency of sight tracking and user experience, and omitting the explicit calibration process of the traditional sight tracking method, and can implicitly complete the calibration of the Kappa angle in the tracking stage, thereby realizing accurate and efficient automatic tracking of the user's sight. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Various other advantages and benefits will become apparent to those of ordinary skill in the art by reading the detailed description of the preferred embodiments below. The accompanying drawings are only for the purpose of illustrating the preferred embodiments and are not to be considered as limiting the present invention. Moreover, the same reference symbols are used throughout the accompanying drawings to represent the same components. In the accompanying drawings: Figure 1 A schematic diagram of an optional flow chart of the method for automatically calibrating the eyeball Kappa angle provided in Example 1; Figure 2 A schematic diagram of the reflection of the dual infrared light sources provided in Example 1 on the outer surface of the cornea; Figure 3 A schematic diagram of the refraction of light on the outer surface of the cornea provided in Example 1; Figure 4 A schematic diagram of a user observing a screen provided in Embodiment 1; Figure 5 A schematic diagram of the eyeball coordinate system provided in Example 1; Figure 6 A schematic diagram of the movement of the rotating shaft provided in Example 1; Figure 7 A schematic diagram of a three-dimensional eyeball model provided in Example 1; Figure 8 A flow chart of line of sight estimation provided in Example 1; Fig. 9 A schematic diagram of the sight point provided in Example 1; Fig.10 A schematic diagram of an optional structure of the device for automatically calibrating the Kappa angle of the eye provided in the second embodiment; Fig.11 This is a schematic diagram of an optional hardware structure of a computer device provided in Example 3. DETAILED DESCRIPTION

[0022] In order to make the purpose, technical scheme and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0023] Embodiment 1 The embodiment of the present invention provides a method for automatically calibrating the eyeball Kappa angle, which is applied to a head-mounted device, such as Figure 1 As shown, the method specifically comprises the following steps: S101: Collect gaze parameters of the user's eyes using a single camera and dual infrared light sources provided in the head mounted device.

[0024] In this embodiment, a user wears a head-mounted three-dimensional eye tracking device, which has multiple light sources, a display screen, and a single camera. When the user wears the device and looks at an object, the user's eye gaze parameters are collected by detecting the human eye image; wherein the gaze parameters at least include the position information of two infrared light sources, the position information of two imaging points on the imaging surface of the single camera, and the position information of the gaze point on the display screen.

[0025] S102: Calculate the corneal curvature center and the optical axis of the user's eyeball according to the gaze parameters.

[0026] The imaging rule of the infrared light source reflected on the outer surface of the cornea and refracted at the edge of the pupil in a single camera is used to calculate the corneal radius, corneal curvature center and pupil center of the user's eyeball. The optical axis is formed by the straight line connecting the corneal curvature center and the pupil center.

[0027] Specifically, calculating the corneal curvature center of the user's eyeball according to the gaze parameter in step S102 includes: Step A1: Create a three-dimensional coordinate system and place the optical center of the single camera The point is set as the origin of the three-dimensional coordinate system of the system; Step A2: In the three-dimensional coordinate system of the system, according to the dual infrared light sources and The coordinate position of the dual infrared light source is calculated based on the parameters of the single camera. and Two imaging points presented on the imaging surface of the single camera and The coordinate position of the dual infrared light source and Through two reflection points on the outer surface of the cornea of ​​the user's eyeball and After reflection, and at the same time through the optical center After the point, it intersects with the imaging surface at two imaging points and ; Step A3: According to the optical center of the single camera in the three-dimensional coordinates of the system The coordinate position of the point, the dual infrared light source and The coordinate position and two imaging points and The coordinate position of the user is used to calculate the corneal radius of the user Corneal curvature center The coordinate position of a point in the three-dimensional coordinate system of the system.

[0028] In this embodiment, if Figure 2 As shown, dual infrared light sources and When reflected by the outer surface of the cornea, the optical center of the single camera The point is set as the origin of the system's three-dimensional coordinate system, the infrared light source Passing through the reflection point on the outer surface of the cornea After reflection, the optical center of the single camera The imaging point where the point falls on the imaging surface of a single camera ; Infrared light source Passing through the reflection point on the outer surface of the cornea After reflection, the optical center of the single camera The imaging point where the point falls on the imaging surface of a single camera . Based on dual infrared light sources and The coordinate position of the single camera The coordinate position of the point and the two imaging points and The user's corneal radius is calculated from the coordinate position Corneal curvature center The coordinate position of a point in the system's three-dimensional coordinate system.

[0029] Further, the corneal radius of the user is calculated Corneal curvature center The coordinate position of a point in the three-dimensional coordinate system of the system includes: The user's corneal radius is calculated by the following formula: Corneal curvature center The coordinate position of the point in the three-dimensional coordinate system of the system: According to the law of reflection of light, the incident angle is equal to the outgoing angle. When two infrared light sources are reflected on the outer surface of the cornea, the incident light, the outgoing light and the unit vector on the straight line where the normal line is located have the following relationship: ; ; In the reflection of light, the incident light, the normal line and the outgoing light are coplanar, and two sets of reflections will form a reflection plane. Both reflection planes contain the , On the intersection of the two reflection planes, It can be expressed as the cross product of two plane normals: ; The expression can be simplified to , so the center of the cornea It can be expressed as: ; h represents the center of corneal curvature Point to the optical center of the single camera The modulus of the point distance; Corneal curvature center Points and Reflection Points and The distances between them are all corneal radius values: ; The cornea can be equivalent to a sphere, and the coordinate equation of the sphere is as follows: ; Combining the above equations (1)-(6), we can get the user's corneal radius: Corneal curvature center The coordinate position of a point in the three-dimensional coordinate system of the system.

[0030] Specifically, the step of calculating the optical axis of the user's eyeball according to the gaze parameter in step S102 includes: like Figure 3 As shown, The pupil is the edge point on the outer surface of the cornea after refraction. The pupil is approximated as a circular plane inside the cornea. Due to the different propagation speeds of light inside the eyeball and in the outside air, the pupil plane undergoes a refraction on the corneal surface before forming an image in the camera. The refractive indices of light in the air and inside the cornea are and , the incident angle and the outgoing angle in the refraction process are and , they satisfy the following relationship: (7); The incident light, normal and outgoing light in the refraction process are coplanar, and the pupil edge point The refraction geometric model satisfies the following equation: ; In particular, and is a known quantity, Take 1, Take 1.34, and assume that the unit vector in the incident direction during refraction is: (9); but ,in for Point and Because the pupil is approximated as a circular plane, the distance from the edge of the pupil to the center The distances between the points are the same, so: ;in, , Point is any point on the edge of the pupil of the user's eyeball, and the distance from any point on the edge of the pupil to the center of the pupil is the same, a, j ∈ [1, 200]

[0031] The optical axis of the eye passes through the center of corneal curvature Point and pupil center point, and is perpendicular to the pupil plane, so the vector The vectors on the pupil plane are perpendicular to each other: ; Combining the above equations (7)-(11), the user's pupil center can be calculated The coordinate position of the point in the three-dimensional coordinate system, and according to the pupil center Corneal curvature center The optical axis is calculated from the point.

[0032] S103: Calculate the Kappa angle according to the corneal curvature center and the optical axis; wherein the Kappa angle is the angle between the visual axis of the user's eyeball and the optical axis.

[0033] In this embodiment, if Figure 4 When the user observes the screen, the intersection of the left eye visual axis and the right eye visual axis with the screen coincides. The Kappa angle of the user's left eye and the Kappa angle of the right eye are calculated according to the corneal curvature center and the optical axis of the user's left eye and right eye respectively; the Kappa angle is the angle between the visual axis of the user's eye and the optical axis.

[0034] Specifically, the step S103 of calculating the Kappa angle according to the corneal curvature center and the optical axis includes: Step B1: Calculate the left eye rotation axis and the left eye rotation angle according to the left eye optical axis collected for the i-th time and the left eye optical axis collected for the (i+1)th time, and calculate the left eye rotation matrix according to the left eye rotation axis and the left eye rotation angle. ; Step B2: Calculate the right eye rotation axis and the right eye rotation angle according to the right eye optical axis collected for the i-th time and the right eye optical axis collected for the (i+1)th time, and calculate the right eye rotation matrix according to the right eye rotation axis and the right eye rotation angle. ; Step B3: Establish the coordinate system of the user's left eyeball; wherein the center of the left eyeball corneal curvature is Set as the origin of the left eyeball coordinate system, and set the visual axis direction of the left eye collected for the i-th time as the left eyeball coordinate system axis; Step B4: Establish the coordinate system of the user's right eyeball; wherein the center of the corneal curvature of the right eyeball is Set as the origin of the right eyeball coordinate system, and set the right eye visual axis direction collected for the i-th time as the right eyeball coordinate system axis; Step B5: Calculate the Kappa angle of the user's left eyeball and the Kappa angle of the right eyeball according to the following formula: In this embodiment, if Figure 5 As shown in the figure, the eyeball coordinate system is the center of the cornea Defined as the origin of the coordinate system, the visual axis direction is defined as Axis. At the initial gaze position axis, axis, Axis and world coordinate system axis, axis, When the gaze direction changes, the eye movement conforms to Listing's law, and the eyeball coordinate system rotates according to Listing's law after the gaze direction changes.

[0035] In the eyeball coordinate system, the Kappa angle is calculated using the horizontal component and the vertical component Therefore, the unit vector in the direction of the optical axis in the eyeball coordinate system is It can be expressed as: ; At the initial eye position, the world coordinate system is parallel to the eye coordinate system, so at this time It is also the unit vector in the direction of the optical axis in the world coordinate system. The visual axis vector at different gaze positions is obtained through the transformation matrix Transformed from the initial gaze position: ;in , Represents the unit vector in the visual axis direction at the current gaze position and initial gaze position of the user's left or right eye, respectively. .

[0036] like Figure 6 As shown in , from Listing's law, the movement of the eyeball from the initial gaze direction to any other gaze direction can be described by a rotation around the normal of the plane formed by the visual axis at the initial and final gaze positions. Translate to the world coordinate system origin ,at this time exist On axis. and are the unit vectors in the direction of the optical axis at the initial and final positions, respectively. From the initial gaze position l to the gaze position 2, the rotation axis of the eye movement yes and Because exist axis, so it can be determined exist plane. And because the eyeball moves within a certain angle range, we know exist The component in the direction is not zero, so we can set . Can also be seen as Around Rotated, so: ; Assumptions and is a known quantity. and Angle between is the angle of eyeball rotation. If the rotation axis and rotation angle are known, then arrive The rotation matrix It can be obtained by Rodriguez formula: ; When calculating the Kappa angle of the user's left eye and Kappa angle with the user's right eyeball When the subject is looking naturally, the visual axes of the left and right eyes intersect at one point. Using the optical axis and corneal curvature center collected at each gaze moment, the rotation matrix of the left and right eyeballs is calculated by the above formula , the rotation matrix of the left and right eyeballs Respectively expressed as and , and then find the unit vector in the visual axis direction of the user's left eyeball collected for the (i+1)th time And the unit vector in the visual axis direction collected for the (i+1)th time in the user's right eyeball .

[0037] (16); (17); At any time of fixation, the visual axes of the left and right eyes intersect at a point in space, and the two visual axes form a plane. The normal of this plane can be expressed as : (18); Construct a vector from the center of the left cornea to the center of the right cornea: (19); Since the visual axis passes through the center of the cornea, On the plane formed by the visual axes of the left and right eyes, Normal to the plane vertical: (20); in, and Respectively represent the horizontal component and vertical component of the Kappa angle of the user's left eyeball; Respectively represent the horizontal component and vertical component of the Kappa angle of the user's right eyeball; represents the unit vector in the direction of the visual axis collected for the i-th time in the left eyeball and the right eyeball of the user, ; represents the normal of the plane formed by the two visual axes in the left eyeball and the right eyeball of the user during the (i+1)th acquisition; By combining the above equations (16)-(20), the Kappa angle of the user's left eye and the Kappa angle of the right eye under ideal conditions are calculated.

[0038] Specifically, after calculating the Kappa angle according to the corneal curvature center and the optical axis in step S103, the method further includes: Step C1: Based on the N groups of initial Kappa angles calculated from the N collected data, the 0th generation population is formed. { }, and set the 0th generation population as the target population; among them, individual is the Kappa angle of the left eye calculated based on the i-th collected data And the Kappa angle of the right eye The four-dimensional vector formed; ; Step C2: inputting each individual in the target population into a preset objective function in turn, and setting the individual corresponding to the minimum value of the objective function as the target individual; Wherein, the objective function is: ; is the distance between the two visual axes in the i-th acquisition data, and ; Step C3: Determine whether the preset stop iteration rule is reached, if yes, execute step C5, if no, execute step C4; wherein the preset stop iteration rule is: the minimum value of the objective function is less than the first preset threshold, or the number of generations of the target population reaches the second preset threshold; Step C4: updating the target population according to a preset differential evolution algorithm to obtain a t-th generation population, setting the t-th generation population as the target population, and re-executing step C2; Step C5: forming the final Kappa angle of the user's eyeball according to the target individuals in the latest generation population.

[0039] Furthermore, the preset stop iteration rule includes: Step D1: The variation of each individual is achieved through a differential strategy, and the differential strategy used is as follows: ; in, ; ,

[0040] , , Is randomly selected from Individuals selected from the generation population; mutation operator is a real constant factor; if Beyond the boundary, then Take the boundary value.

[0041] Step D2: and Perform crossover operation to generate new vector : ; in, is the crossover probability for A random real number; for A random integer; Step D3: According to the following formula, form the individuals in the t-th generation population : .

[0042] According to the above formulas (21)-(25), after completing the above evolutionary operation, the value of the objective function in the population is iterated again to find the best individual in this generation. The algorithm ends after completing the maximum number of evolutionary generations or finding an individual whose objective function value is less than the set value.

[0043] S104: Calculate the visual axis of the user's eyeball according to the optical axis and the Kappa angle.

[0044] In this embodiment, after obtaining the Kappa angle of the user's left eye and the Kappa angle of the right eye, the visual axis of the left eye is calculated based on the optical axis and the Kappa angle of the user's left eye, and the visual axis of the right eye is calculated based on the optical axis and the Kappa angle of the user's right eye, thereby estimating the user's line of sight direction in real time.

[0045] like Figure 7 As shown in the figure, the symmetry axis of the eyeball, i.e. the optical axis, passes through the center of the eyeball, the center of the corneal curvature and the center of the pupil. The outer surface of the cornea can be regarded as a spherical section with the optical axis as the symmetry axis, and the eyeball is regarded as a curved surface formed by the intersection of two large and small spheres. Assuming that the center of the corneal curvature of the eyeball coincides with the node of the eyeball (i.e. the optical center of the eyeball, which is located behind the lens in the simplified model of the eyeball), the visual axis (line of sight) is defined as a straight line connecting the fovea and the center of the corneal curvature. The angle between the optical axis and the visual axis is also called the kappa angle. The size and direction of this angle vary from person to person and need to be obtained through a calibration program. As shown in Figure 8 As shown, the system single camera in the head-mounted device collects face images in real time, and processes the real-time image of the human eye on the face image to extract features; according to the internal parameters of the single camera and the dual infrared light source points, combined with the internal structure parameters of the eyeball, the corneal curvature center, corneal radius and pupil center of the eyeball are calculated to construct a real-time eyeball coordinate system. According to the multiple sets of eyeball data parameters collected in real time, the differential evolution algorithm is used to continuously optimize the kappa angle of the user's left and right eyeballs. After obtaining the optimal kappa angle, the visual axis of the user's left and right eyeballs is calculated according to the optical axis and the kappa angle to determine the user's real-time gaze direction.

[0046] The following is a specific test example of calculating the kappa angle according to the above method: The subjects were away Screen In this case, design evenly distributed Coordinate grid points. Generate the virtual subject's gaze 25 sets of binocular corneal center coordinates and optical axis direction vector coordinates at the coordinate grid points, and add an amplitude of The noise of the optical axis direction vector is added with an amplitude of The estimated sight point is as follows Fig. 9As shown in the figure, it can be found that compared with the actual sight point, the estimated sight point has a significant collective rightward deviation. This is because the calculated value of the kappa angle maintains the consistency of the error direction of the left and right eyes in both the vertical and horizontal components, which leads to the consistency of the deviation direction of the sight point estimated by the kappa angle calculation value. The average deviation distance between the 25 groups of sight points and the actual sight point is , this accuracy can meet some simple eye tracking system applications.

[0047] In order to verify the feasibility of the automatic calibration algorithm, an actual system experiment was also conducted. A glasses-type eye tracker was used in the experiment. The eye tracker was equipped with one infrared camera and eight infrared light sources for the left and right eyes respectively. When the corneal parameters were actually calculated, only two of the infrared light sources were used for each eye. The eye tracker is also equipped with a front camera for detecting the object observed by the user. The world coordinate system is the camera coordinate system of the front camera. The subject wore the eye tracker and sat upright about 50 cm in front of the 15.6-inch laptop screen. The eye tracker was connected to the laptop via a USB interface. Run the program, collect the eye image and perform image processing, extract the coordinates of the light spot on the outer surface of the cornea and the coordinates of the pupil edge point, obtain the coordinates of the center of corneal curvature and the center of the pupil, and reconstruct the optical axis direction vector. Finally, the automatic calibration method of the differential evolution method was used to obtain the values ​​of the kappa angle of the subject's eyes as shown in Table 1: Table 1

[0048] In the experiment, circular patterns with known radius and screen coordinates were fixed at the four corners of the computer screen. The spatial coordinates of the centers of the four circular patterns can be obtained by taking pictures with the front camera and performing spatial circular detection, and then the screen coordinates can be determined. The true line of sight direction can be determined and the line of sight landing point can be calculated by the known coordinates of the gaze point and the center coordinates of the corneal curvature. The line of sight landing point calculation error of the automatic user calibration algorithm of the differential evolution method is 2.9°.

[0049] In this embodiment, the gaze parameters of the user's eyes are collected by utilizing a single camera and dual infrared light sources provided in the head-mounted device; the center of corneal curvature and the optical axis of the user's eyes are calculated according to the gaze parameters; the Kappa angle is calculated according to the center of corneal curvature and the optical axis; the visual axis of the user's eyes is calculated according to the optical axis and the Kappa angle; the user does not need to actively cooperate during user calibration, and the user is not required to complete the gaze of the calibration point at a specific time, thereby improving the consistency of the line of sight tracking process, omitting the explicit calibration process of the traditional line of sight tracking method, and implicitly completing the calibration of the Kappa angle in the tracking stage, thereby achieving accurate and efficient automatic tracking of the user's line of sight, and using the differential evolution algorithm to greatly reduce the error of line of sight estimation.

[0050] Embodiment 2 The embodiment of the present invention provides a device for constructing a terrain model, such as Fig.10 As shown, the device specifically includes the following components: The acquisition module 1001 is used to acquire the gaze parameters of the user's eyes by using a single camera and dual infrared light sources provided in the head mounted device; An optical axis module 1002, used to calculate the corneal curvature center and the optical axis of the user's eyeball according to the gaze parameter; An angle module 1003, used to calculate the Kappa angle according to the corneal curvature center and the optical axis; wherein the Kappa angle is the angle between the visual axis of the user's eyeball and the optical axis; The visual axis module 1004 is used to calculate the visual axis of the user's eyeball according to the optical axis and the Kappa angle.

[0051] Specifically, the optical axis module 1002 is used for: Create a three-dimensional coordinate system and set the optical center of the single camera point is set as the origin of the three-dimensional coordinate system of the system; in the three-dimensional coordinate system of the system, according to the dual infrared light source and The coordinate position of the dual infrared light source is calculated based on the parameters of the single camera. and Two imaging points presented on the imaging surface of the single camera and The coordinate position of the dual infrared light source and Through two reflection points on the outer surface of the cornea of ​​the user's eyeball and After reflection, and at the same time through the optical center After the point, it intersects with the imaging surface at two imaging points and ; According to the optical center of the single camera in the three-dimensional coordinates of the system The coordinate position of the point, the dual infrared light source and The coordinate position and two imaging points and The coordinate position of the user is used to calculate the corneal radius of the user Corneal curvature center The coordinate position of a point in the three-dimensional coordinate system of the system.

[0052] Furthermore, the optical axis module 1002 is also used for: The user's corneal radius is calculated by the following formula: Corneal curvature center The coordinate position of the point in the three-dimensional coordinate system of the system: ; ; ; ; ; Among them, ,at this time, , and h represents the center of corneal curvature Point to the optical center of the single camera The modulus of the point distance.

[0053] Furthermore, the optical axis module 1002 is also used for: The pupil center of the user is calculated by the following formula: The coordinate position of the point in the three-dimensional coordinate system, and according to the pupil center Corneal curvature center Point calculates the optical axis of the user's eyeball: ; ; ; ; ; The light passes through the outer surface of the cornea of ​​the user's eyeball. After refraction, the point reaches the outer edge of the pupil of the user's eyeball. point, and and are the incident angle and the outgoing angle during the refraction process. and are the refractive index of light in the air and the refractive index of light inside the cornea; The point is any point on the edge of the pupil of the user's eyeball, and any point on the edge of the pupil to the center of the pupil The distance between points is the same, j∈[1, 200] is the unit vector in the incident direction during refraction; for Point and The modulus of the point distance.

[0054] Specifically, the angle module 1003 is used to: The left eye rotation axis and the left eye rotation angle are calculated according to the left eye optical axis collected for the i-th time and the left eye optical axis collected for the (i+1)th time, and the left eye rotation matrix is ​​calculated according to the left eye rotation axis and the left eye rotation angle. ; Calculate the right eye rotation axis and the right eye rotation angle according to the right eye optical axis collected for the i-th time and the right eye optical axis collected for the (i+1)th time, and calculate the right eye rotation matrix according to the right eye rotation axis and the right eye rotation angle ; Establish the left eyeball coordinate system of the user; wherein the center of the left eyeball corneal curvature Set as the origin of the left eyeball coordinate system, and set the visual axis direction of the left eye collected for the i-th time as the left eyeball coordinate system Axis; establish the user's right eyeball coordinate system; wherein the center of the right eyeball corneal curvature Set as the origin of the right eyeball coordinate system, and set the right eye visual axis direction collected for the i-th time as the right eyeball coordinate system Axis; the Kappa angle of the left eyeball and the Kappa angle of the right eyeball of the user are calculated according to the following formula: ; ; ; ; ; in, and Respectively represent the horizontal component and vertical component of the Kappa angle of the user's left eyeball; Respectively represent the horizontal component and vertical component of the Kappa angle of the user's right eyeball; represents the unit vector in the direction of the visual axis collected for the i-th time in the left eyeball and the right eyeball of the user, ; represents the normal of the plane formed by the two visual axes in the left eyeball and the right eyeball of the user during the (i+1)th acquisition; represents the unit vector in the direction of the visual axis acquired for the (i+1)th time in the left eyeball of the user; represents the unit vector in the direction of the visual axis acquired for the (i+1)th time in the right eyeball of the user.

[0055] Specifically, the device further includes an error module, which is used to: Step 1: Based on the N groups of initial Kappa angles calculated from the N collected data, the 0th generation population is formed. { }, and set the 0th generation population as the target population; among them, individual is the Kappa angle of the left eye calculated based on the i-th collected data And the Kappa angle of the right eye The four-dimensional vector formed; ; Step 2: input each individual in the target population into a preset objective function in turn, and set the individual corresponding to the minimum value of the objective function as the target individual; Wherein, the objective function is: ; is the distance between the two visual axes in the i-th acquisition data, and ; Step 3: Determine whether the preset stop iteration rule is reached, if yes, execute step 5, if no, execute step 4; wherein the preset stop iteration rule is: the minimum value of the objective function is less than the first preset threshold, or the number of generations of the target population reaches the second preset threshold; Step 4: Update the target population according to the preset differential evolution algorithm to obtain the t-th generation population, set the t-th generation population as the target population, and re-execute step 2; Step 5: Form the final Kappa angle of the user's eyeball according to the target individuals in the latest generation population.

[0056] Furthermore, the error module is also used for: The boundary vector is obtained according to the following formula : ; in, ; ,

[0057] , , Is randomly selected from Individuals selected from the generation population; Mutation Operator is a real constant factor; The intermediate vector is formed according to the following formula : ; in, is the crossover probability for A random real number; for A random integer; According to the following formula, the individuals in the t-th generation population are formed : .

[0058] Embodiment 3 This embodiment also provides a computer device, such as a smart phone, tablet computer, laptop computer, desktop computer, rack server, blade server, tower server or cabinet server (including an independent server or a server cluster composed of multiple servers) that can execute programs. Fig.11 As shown, the computer device 110 of this embodiment includes at least but not limited to: a memory 1101 and a processor 1102 that can be interconnected through a system bus. It should be noted that Fig.11 Only computer device 110 is shown with components 1101 - 1102 , but it should be understood that implementing all of the components shown is not a requirement, and more or fewer components may alternatively be implemented.

[0059] In this embodiment, the memory 1101 (i.e., readable storage medium) includes flash memory, hard disk, multimedia card, card-type memory (e.g., SD or DX memory, etc.), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, magnetic disk, optical disk, etc. In some embodiments, the memory 1101 may be an internal storage unit of the computer device 110, such as a hard disk or memory of the computer device 110. In other embodiments, the memory 1101 may also be an external storage device of the computer device 110, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. equipped on the computer device 110. Of course, the memory 1101 may also include both the internal storage unit of the computer device 110 and its external storage device. In this embodiment, the memory 1101 is generally used to store the operating system and various application software installed on the computer device 110. In addition, the memory 1101 can also be used to temporarily store various types of data that have been output or are to be output.

[0060] In some embodiments, the processor 1102 may be a central processing unit (CPU), a controller, a microcontroller, a microprocessor, or other data processing chips. The processor 1102 is generally used to control the overall operation of the computer device 110 .

[0061] Specifically, in this embodiment, the processor 1102 is used to execute the program of the method for automatically calibrating the eyeball Kappa angle stored in the memory 1101, and the program of the method for automatically calibrating the eyeball Kappa angle implements the following steps when being executed: Using a single camera and dual infrared light sources provided in the head mounted device, the gaze parameters of the user's eyes are collected; Calculating the corneal curvature center and the optical axis of the user's eyeball according to the gaze parameter; Calculating the Kappa angle according to the corneal curvature center and the optical axis; wherein the Kappa angle is the angle between the visual axis of the user's eyeball and the optical axis; The visual axis of the user's eyeball is calculated according to the optical axis and the Kappa angle.

[0062] The specific implementation process of the above method steps can be found in Example 1, and this embodiment will not be repeated here.

[0063] Embodiment 4 This embodiment also provides a computer-readable storage medium, such as a flash memory, a hard disk, a multimedia card, a card-type memory (for example, an SD or DX memory, etc.), a random access memory (RAM), a static random access memory (SRAM), a read-only memory (ROM), an electrically erasable programmable read-only memory (EEPROM), a programmable read-only memory (PROM), a magnetic memory, a disk, an optical disk, a server, an App application store, etc., on which a computer program is stored. When the computer program is executed by a processor, the following method steps are implemented: Using a single camera and dual infrared light sources provided in the head mounted device, the gaze parameters of the user's eyes are collected; Calculating the corneal curvature center and the optical axis of the user's eyeball according to the gaze parameter; Calculating the Kappa angle according to the corneal curvature center and the optical axis; wherein the Kappa angle is the angle between the visual axis of the user's eyeball and the optical axis; The visual axis of the user's eyeball is calculated according to the optical axis and the Kappa angle.

[0064] The specific implementation process of the above method steps can be found in Example 1, and this embodiment will not be repeated here.

[0065] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or device including the element.

[0066] The serial numbers of the above embodiments of the present invention are only for description and do not represent the advantages or disadvantages of the embodiments.

[0067] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method.

[0068] The above are only preferred embodiments of the present invention, and are not intended to limit the patent scope of the present invention. Any equivalent structure or equivalent process transformation made using the contents of the present invention specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present invention.

Claims

1. A method for automatically calibrating the eyeball Kappa angle, characterized in that: Applied to a head mounted device, the method comprises: Using a single camera and dual infrared light sources provided in the head mounted device, the gaze parameters of the user's eyes are collected; Calculating the corneal curvature center and the optical axis of the user's eyeball according to the gaze parameter; Calculating the Kappa angle according to the corneal curvature center and the optical axis; wherein the Kappa angle is the angle between the visual axis of the user's eyeball and the optical axis; The visual axis of the user's eyeball is calculated according to the optical axis and the Kappa angle.

2. The method for automatically calibrating the eyeball Kappa angle according to claim 1, characterized in that: Calculating the corneal curvature center of the user's eyeball according to the gaze parameter includes: Create a three-dimensional coordinate system and set the optical center of the single camera The point is set as the origin of the three-dimensional coordinate system of the system; In the three-dimensional coordinate system of the system, according to the dual infrared light sources and The coordinate position of the dual infrared light source is calculated based on the parameters of the single camera. and Two imaging points presented on the imaging surface of the single camera and The coordinate position of the dual infrared light source and Through two reflection points on the outer surface of the cornea of ​​the user's eyeball and After reflection, and at the same time through the optical center After the point, it intersects with the imaging surface at two imaging points and ; According to the optical center of the single camera in the three-dimensional coordinates of the system The coordinate position of the point, the dual infrared light source and The coordinate position and two imaging points and The coordinate position of the user is used to calculate the corneal radius of the user Corneal curvature center The coordinate position of a point in the three-dimensional coordinate system of the system.

3. The method for automatically calibrating the eyeball Kappa angle according to claim 2, characterized in that: According to the optical center of the single camera in the three-dimensional coordinates of the system The coordinate position of the point, the dual infrared light source and The coordinate position and two imaging points and The coordinate position of the user is used to calculate the corneal radius of the user Corneal curvature center The coordinate position of a point in the three-dimensional coordinate system of the system includes: The user's corneal radius is calculated by the following formula: Corneal curvature center The coordinate position of the point in the three-dimensional coordinate system of the system: ； ； ； ； ； Among them, ,at this time, , and h represents the center of corneal curvature Point to the optical center of the single camera The modulus of the point distance.

4. The method for automatically calibrating the eyeball Kappa angle according to claim 3, characterized in that: Calculating the optical axis of the user's eyeball according to the gaze parameter includes: The pupil center of the user is calculated by the following formula: The coordinate position of the point in the three-dimensional coordinate system, and according to the pupil center Corneal curvature center Point calculates the optical axis of the user's eyeball: ； ； ； ； ； The light passes through the outer surface of the cornea of ​​the user's eyeball. After refraction, the point reaches the outer edge of the pupil of the user's eyeball point, and and are the incident angle and the outgoing angle during the refraction process. and are the refractive index of light in the air and the refractive index of light inside the cornea; The point is any point on the edge of the pupil of the user's eyeball, and any point on the edge of the pupil to the center of the pupil The distance between the points is the same, j∈[1,200] is the unit vector in the incident direction during refraction; for Point and The modulus of the point distance.

5. The method for automatically calibrating the eyeball Kappa angle according to claim 4, characterized in that: Calculating the Kappa angle according to the corneal curvature center and the optical axis includes: The left eye rotation axis and the left eye rotation angle are calculated according to the left eye optical axis collected for the i-th time and the left eye optical axis collected for the (i+1)th time, and the left eye rotation matrix is ​​calculated according to the left eye rotation axis and the left eye rotation angle. ; The right eye rotation axis and the right eye rotation angle are calculated according to the right eye optical axis collected for the i-th time and the right eye optical axis collected for the (i+1)th time, and the right eye rotation matrix is ​​calculated according to the right eye rotation axis and the right eye rotation angle. ; Establish the left eyeball coordinate system of the user; wherein the center of the left eyeball corneal curvature is Set as the origin of the left eyeball coordinate system, and set the visual axis direction of the left eye collected for the i-th time as the left eyeball coordinate system axis; Establish the user's right eyeball coordinate system; wherein the center of the right eyeball corneal curvature is Set as the origin of the right eyeball coordinate system, and set the right eye visual axis direction collected for the i-th time as the right eyeball coordinate system axis; The Kappa angle of the user's left eyeball and the Kappa angle of the right eyeball are calculated jointly according to the following formula: ； ； ； ； ； in, and Respectively represent the horizontal component and vertical component of the Kappa angle of the user's left eyeball; Respectively represent the horizontal component and vertical component of the Kappa angle of the user's right eyeball; represents the unit vector in the direction of the visual axis acquired for the i-th time in the left eye and the right eye of the user, and ; represents the normal of the plane formed by the two visual axes in the left eyeball and the right eyeball of the user during the (i+1)th acquisition; represents the unit vector in the direction of the visual axis acquired for the (i+1)th time in the left eyeball of the user; represents the unit vector in the direction of the visual axis acquired for the (i+1)th time in the right eyeball of the user.

6. The method for automatically calibrating the eyeball Kappa angle according to claim 5, characterized in that: After calculating the Kappa angle according to the corneal curvature center and the optical axis, the method further includes: Step 1: Based on the N groups of initial Kappa angles calculated from the N collected data, the 0th generation population is formed. { }, and set the 0th generation population as the target population; among them, individual is the Kappa angle of the left eye calculated based on the i-th collected data And the Kappa angle of the right eye The four-dimensional vector formed; ; Step 2: input each individual in the target population into a preset objective function in turn, and set the individual corresponding to the minimum value of the objective function as the target individual; Wherein, the objective function is: ; is the distance between the two visual axes in the i-th acquisition data, and ; Step 3: Determine whether the preset stop iteration rule is reached, if yes, execute step 5, if no, execute step 4; wherein the preset stop iteration rule is: the minimum value of the objective function is less than the first preset threshold, or the number of generations of the target population reaches the second preset threshold; Step 4: Update the target population according to the preset differential evolution algorithm to obtain the t-th generation population, set the t-th generation population as the target population, and re-execute step 2; Step 5: Form the final Kappa angle of the user's eyeball according to the target individuals in the latest generation population.

7. The method for automatically calibrating the eyeball Kappa angle according to claim 6, characterized in that: The preset stop iteration rule includes: The boundary vector is obtained according to the following formula : ； in, ; ， , , is randomly selected from the first ) Individuals selected from the generation population; Mutation Operator is a real constant factor; The intermediate vector is formed according to the following formula : ； in, is the crossover probability for A random real number; for A random integer; According to the following formula, the individuals in the t-th generation population are formed : 。 8. A device for automatically calibrating the Kappa angle of an eyeball, characterized in that: Applied to a head mounted device, the device comprises: A collection module, used to collect the gaze parameters of the user's eyes using a single camera and dual infrared light sources provided in the head mounted device; An optical axis module, used to calculate the corneal curvature center and the optical axis of the user's eyeball according to the gaze parameter; An angle module, used to calculate the Kappa angle according to the corneal curvature center and the optical axis; wherein the Kappa angle is the angle between the visual axis of the user's eyeball and the optical axis; A visual axis module is used to calculate the visual axis of the user's eyeball based on the optical axis and the Kappa angle.

9. A computer device, comprising: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the method according to any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images