Eye Tracking Processing Method, Device, System, Computer Device, and Storage Medium

Abstract

The present invention provides a method, device, system, computer device and storage medium for gaze tracking. The method includes: obtaining current eye images of a target to be tracked to obtain two target eye images; performing pupil detection and segmentation on the target eye images to obtain pupil region images, and determining pupil image coordinates and light source spot coordinates; calculating pupil spatial coordinates according to the optical centers of two cameras and the pupil image coordinates; calculating corneal center spatial coordinates according to the optical centers of two cameras, the spatial coordinates of two light sources relative to each camera, and the light source spot coordinates; connecting the straight line between the pupil spatial coordinates and the corneal center spatial coordinates to obtain a target optical axis; determining the kappa angle of the target to be tracked, where the kappa angle is a fixed angle between the human eye optical axis and the visual axis; and determining the current gaze direction of the target to be tracked according to the kappa angle and the target optical axis. Through the present invention, gaze tracking can be achieved based on dual cameras and dual light sources.

Description

Technical Field

[0001] The present invention relates to the field of gaze tracking processing technology, and in particular to a gaze tracking processing method, device, system, computer equipment and storage medium. Background Art

[0002] Visual information accounts for 80% to 90% of all information humans acquire. A wealth of relevant information can be extracted from the direction of a person's gaze. Gaze tracking systems use eye movement characteristics to estimate a person's current gaze direction or gaze point location. Gaze tracking technology can be applied in fields such as psychology, medical research, and human-computer interaction. The principles of gaze tracking technology can be categorized into two-dimensional and three-dimensional gaze tracking.

[0003] Two-dimensional gaze tracking systems typically consist of a single camera and a single light source, using a second-order polynomial to represent the direct mapping from the pupil spot vector to the gaze point. Three-dimensional gaze tracking systems typically involve a single camera and multiple light sources, or multiple cameras and multiple light sources. In contrast, single-camera, multiple light source, and multi-camera systems can detect the three-dimensional gaze direction within the system's camera coordinate system.

[0004] Two-dimensional gaze tracking generally uses a monocular camera. Monocular imaging cannot obtain the three-dimensional position information of an object. Therefore, this type of system can only calculate the gaze landing point on the screen through a mapping model between the image's gaze feature parameters and the landing points on the screen. It cannot obtain the gaze's true spatial direction in the system's camera coordinate system. Therefore, this method is greatly limited in terms of natural head movement.

[0005] Three-dimensional gaze tracking can estimate the spatial direction of the gaze within the system's camera coordinate system using the eyeball's structure and imaging model. Therefore, this type of system can obtain the three-dimensional gaze direction under natural head movement and is unaffected by the user's head movement. A multi-camera, multi-light source system—that is, a system comprising three or more light sources and three or more cameras—can obtain three-dimensional feature information about the eyeball through stereo vision, eliminating the need to determine excessive eyeball structural parameters and simplifying the user calibration process. Three-dimensional gaze tracking technology offers significant advantages over two-dimensional gaze tracking technology and represents the future development direction of gaze tracking technology.

[0006] Generally, three-dimensional gaze tracking methods use a multi-camera and multi-light source system, and the relative coordinates of the light source and camera usually need to be determined during the calculation process. Therefore, errors are easily generated when calibrating the relative positions of the light source and camera, which has a certain impact on the accuracy of the gaze tracking results.

[0007] Therefore, how to improve the accuracy of three-dimensional gaze tracking methods has become a technical problem that needs to be solved urgently in this field. Summary of the Invention

[0008] The object of the present invention is to provide a method, device, system, computer device and storage medium for gaze tracking processing, so as to solve the technical problems in the prior art.

[0009] On the one hand, to achieve the above object, the present invention provides a method for gaze tracking processing.

[0010] The method for gaze tracking processing includes: acquiring current eye images of a target to be tracked to obtain two target eye images, where the two target eye images are respectively acquired by two cameras, and each target eye image has light source spots formed by reflection of two light sources in the eyes; performing pupil detection and segmentation on the target eye images to obtain pupil region images; determining pupil image coordinates and light source spot coordinates in the pupil region images; calculating pupil spatial coordinates based on the optical centers of the two cameras and the pupil image coordinates; calculating corneal center spatial coordinates based on the optical centers of the two cameras, the spatial coordinates of the two light sources relative to each camera, and the light source spot coordinates; connecting the straight line between the pupil spatial coordinates and the corneal center spatial coordinates to obtain the target optical axis; determining the kappa angle of the target to be tracked, where the kappa angle is a fixed angle between the human eye optical axis and the visual axis; and determining the current gaze direction of the target to be tracked based on the kappa angle and the target optical axis.

[0011] Further, the two target eye images include a first image acquired by a first camera and a second image acquired by a second camera. The step of calculating the corneal center spatial coordinates based on the optical centers of the two cameras, the spatial coordinates of the two light sources relative to each camera, and the light source spot coordinates includes: determining the connection line between the optical center of the first camera and the corneal center based on the optical center of the first camera, the spatial coordinates of the two light sources relative to the first camera, and the light source spot coordinates in the first image to obtain a first line; determining the connection line between the optical center of the second camera and the corneal center based on the optical center of the second camera, the spatial coordinates of the two light sources relative to the second camera, and the light source spot coordinates in the second image to obtain a second line; and calculating the intersection point of the first line and the second line to obtain the corneal center spatial coordinates.

[0012] Further, the two light sources include a first light source and a second light source. The step of determining the connection line between the optical center of the first camera and the corneal center based on the optical center of the first camera, the spatial coordinates of the two light sources relative to the first camera, and the light source spot coordinates in the first image to obtain a first line includes: forming a first plane by using the optical center of the first camera, the spatial coordinates of the first light source relative to the first camera, and the light source spot coordinates of the first light source in the first image; forming a second plane by using the optical center of the first camera, the spatial coordinates of the second light source relative to the first camera, and the light source spot coordinates of the second light source in the first image; and calculating the intersection line of the first plane and the second plane to obtain the connection line between the optical center of the first camera and the corneal center.

[0013] Further, the steps for determining the kappa angle of the target to be tracked include: collecting an eye image of the target to be tracked when it is gazing at a predetermined point to obtain a detected eye image; determining the optical axis and the spatial coordinates of the corneal center corresponding to the detected eye image; connecting the predetermined point and the spatial coordinates of the corneal center corresponding to the detected eye image to obtain the visual axis; and calculating the angle between the optical axis and the visual axis corresponding to the detected eye image to obtain the kappa angle.

[0014] Further, the following steps are adopted to calculate the spatial coordinates of the light source relative to the camera: placing the small ball at the first position and the second position within the imaging range of the camera respectively for shooting to obtain two small ball images, where each small ball image includes the small ball and the small ball light spot formed by the small ball reflecting the light emitted by the light source to the camera; for each small ball image, respectively adopting the following steps to calculate a plane equation to obtain two plane equations: determining the image coordinates of the ball center and the small ball light spot coordinates in the small ball image; connecting the image coordinates of the ball center and the optical center of the camera to obtain the third line; determining the spatial coordinates of the ball center of the small ball according to the third line, the position of the small ball and the radius of the small ball; connecting the small ball light spot coordinates and the optical center of the camera to obtain the fourth line; determining the spatial coordinates of the reflection point of the light ray emitted by the light source on the small ball according to the fourth line and the position of the small ball; determining the plane equation according to the spatial coordinates of the reflection point, the optical center of the camera and the spatial coordinates of the ball center; calculating the intersection line of the corresponding planes according to the two plane equations to obtain the light source line where the light source is located; and determining the spatial coordinates of the light source relative to the camera on the light source line.

[0015] Further, the steps for determining the spatial coordinates of the light source relative to the camera on the light source line include: measuring the actual distances between the light source and the small ball and between the small ball and the camera; determining the initial length of iteration according to the actual distances, and iterating the positions of the light source and the small ball according to a predetermined iteration step size; calculating whether the incident angle is equal to the reflection angle every time of iteration; if the incident angle is equal to the reflection angle, determining the position of the light source as a feasible solution; when the number of iterations meets the iteration threshold, performing weighted averaging on multiple feasible solutions to obtain the spatial coordinates of the light source relative to the camera.

[0016] On the other hand, to achieve the above object, the present invention provides a line-of-sight tracking processing device.

[0017] The line-of-sight tracking processing device includes: a first acquisition module, configured to acquire current eye images of a target to be tracked, obtaining two target eye images, where the two target eye images are respectively acquired by two cameras, and each target eye image has light source spots formed by reflection of two light sources in the eyes; a processing module, configured to perform pupil detection and segmentation on the target eye images to obtain pupil region images; a first determination module, configured to determine pupil image coordinates and light source spot coordinates in the pupil region images; a first calculation module, configured to calculate pupil spatial coordinates based on the optical centers of the two cameras and the pupil image coordinates; a second calculation module, configured to calculate corneal center spatial coordinates based on the optical centers of the two cameras, the spatial coordinates of the two light sources relative to each camera, and the light source spot coordinates; a connection module, configured to connect the straight line of the pupil spatial coordinates and the corneal center spatial coordinates to obtain a target optical axis; a second determination module, configured to determine the kappa angle of the target to be tracked, where the kappa angle is a fixed angle between the human eye optical axis and the visual axis; and a third determination module, configured to determine the current line-of-sight direction of the target to be tracked based on the kappa angle and the target optical axis.

[0018] In another aspect, to achieve the above object, the present invention provides a line-of-sight tracking processing system.

[0019] The line-of-sight tracking processing system includes a first camera, a second camera, two light sources, and a processing device, where the processing device is configured to execute any one of the line-of-sight tracking processing methods provided by the present invention.

[0020] In another aspect, to achieve the above object, the present invention further provides a computer device, including a memory, a processor, and a computer program stored on the memory and executable on the processor, where the processor implements the steps of the above method when executing the computer program.

[0021] In another aspect, to achieve the above object, the present invention further provides a computer-readable storage medium, on which a computer program is stored, and the computer program implements the steps of the above method when executed by a processor.

[0022] The line-of-sight tracking processing method, device, system, computer device, and storage medium provided by the present invention use a dual-camera and dual-light-source tracking system. The light source spots formed by the reflection of the light source by the human eye are captured by the cameras, and an eye image including the human eye pupil and the light source spots is obtained, that is, two target eye images captured by two cameras. Then, pupil detection and segmentation are performed on the target eye images to obtain pupil region images. Further, pupil image coordinates and light source spot coordinates are determined in the pupil region images. Then, pupil spatial coordinates are calculated based on the optical centers of the two cameras and the pupil image coordinates, and corneal center spatial coordinates are calculated based on the optical centers of the two cameras, the spatial coordinates of the two light sources relative to each camera, and the light source spot coordinates. The straight line connecting the pupil spatial coordinates and the corneal center spatial coordinates is the target optical axis. After obtaining the optical axis, based on the fixed angle between the human eye optical axis and the visual axis, that is, the kappa angle, the current line-of-sight direction of the target to be tracked can be finally determined. Through the present invention, a tracking system implemented based on a dual-camera and dual-light-source uses the image of the pupil formed in the camera to calculate the pupil coordinates in space, uses the coordinates of the light spot in the camera to calculate the corneal center position, obtains the connection line between the pupil and the cornea, that is, the optical axis, and finally determines the fixed angle kappa angle of the human body, so as to obtain the line-of-sight direction, realizing line-of-sight tracking of a dual-camera and dual-light-source system. Compared with a multi-camera and multi-light-source line-of-sight tracking system, it can reduce the introduction of errors and improve the accuracy of realizing tracking. Description of the Drawings

[0023] By reading the following detailed description of the preferred embodiments, various other advantages and benefits will become clear to those of ordinary skill in the art. The drawings are only for the purpose of showing the preferred embodiments and are not considered to be a limitation of the present invention. Moreover, throughout the drawings, the same reference numerals are used to represent the same components. In the drawings:

[0024] Figure 1 is a flowchart of the line-of-sight tracking processing method provided in Embodiment 1 of the present invention;

[0025] Figure 2 is a schematic diagram for calculating the pupil spatial coordinates provided in an embodiment of the present invention;

[0026] Figure 3 is a schematic diagram for calculating the connection line between the optical center and the corneal center provided in an embodiment of the present invention;

[0027] Figure 4 is a schematic diagram of the kappa angle provided in an embodiment of the present invention;

[0028] Figure 5 is a schematic diagram for calculating the plane equation provided in an embodiment of the present invention;

[0029] Figure 6 is a schematic diagram of the small ball position iteration provided in an embodiment of the present invention;

[0030] Figure 7 Schematic diagram of light source position iteration provided by an embodiment of the present invention;

[0031] Figure 8 Block diagram of the line-of-sight tracking processing device provided by the second embodiment of the present invention;

[0032] Figure 9 Block diagram of the line-of-sight tracking processing system provided by the third embodiment of the present invention;

[0033] Figure 10 Hardware structure diagram of the computer device provided by the third embodiment of the present invention. Detailed implementation manners

[0034] In order to make the objectives, technical solutions and advantages of the present invention clearer, the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts shall fall within the protection scope of the present invention.

[0035] Embodiment 1

[0036] The embodiment of the present invention provides a line-of-sight tracking processing method, which realizes a line-of-sight tracking processing method based on dual cameras and dual light sources. Through this method, dual cameras and dual light sources are set for line-of-sight tracking, and the line-of-sight direction can be calculated by using human eye parameters through the dual-camera and dual-light-source device. Compared with the method of line-of-sight tracking based on multi-camera and multi-light-source devices, on the basis of realizing three-dimensional line-of-sight tracking, the use of cameras and light sources is reduced, which is beneficial to reducing the error generated when calibrating the relative positions of the light source and the camera, and is beneficial to improving the accuracy of the three-dimensional line-of-sight tracking method. Specifically, Figure 1 Flowchart of the line-of-sight tracking processing method provided by the first embodiment of the present invention, as Figure 1 shown, the line-of-sight tracking processing method provided by this embodiment includes the following steps S101 to S108.

[0037] Step S101: Obtain the current human eye images of the target to be tracked, and obtain two target human eye images.

[0038] Among them, dual cameras and dual light sources are set for line-of-sight tracking. Optionally, based on an infrared dual-camera and dual-light-source device, in a scenario where human eye line-of-sight tracking is required, according to the set image acquisition frequency, the dual-camera and dual-light-source device acquires human eye images, and the obtained images are defined as target human eye images, that is, two target human eye images are obtained and acquired by two cameras respectively. Each target human eye image has light source spots formed by the reflection of two light sources in the human eye.

[0039] Step S102: performing pupil detection and segmentation on the target human eye image to obtain a pupil area image.

[0040] Step S103: Determine the pupil image coordinates and the light source spot coordinates in the pupil area image.

[0041] Optionally, the trained neural network model is first used to detect and segment the pupil of the target eye image, obtaining an image of the pupil region, also known as the pupil region image. The pupil center coordinates in the image are also obtained, defined as the pupil image coordinates. The pupil region image is then binarized using a threshold, and the light spot contour is detected on the region image to obtain the coordinates of the light spot formed by the light source reflected from the eye, defined as the light spot coordinates.

[0042] Step S104: Calculate the pupil space coordinates based on the optical centers of the two cameras and the pupil image coordinates.

[0043] Optionally, the two target human eye images include a first image captured by a first camera and a second image captured by a second camera, and the above step S104 specifically includes:

[0044] Step S1041: Connecting the pupil image coordinates in the first image and the optical center of the first camera to obtain a connecting line, which is defined as the fifth line.

[0045] Step S1042: Connecting the pupil image coordinates in the second image and the optical center of the second camera to obtain a connecting line, which is defined as the sixth line.

[0046] Step S1043: Calculate the intersection of the fifth line and the sixth line to obtain the pupil space coordinates.

[0047] Figure 2 The calculation diagram of pupil space coordinates provided by the embodiment of the present invention is as follows: Figure 2 As shown, P c is the pupil in space, calibrated by the pupil space coordinates, O1 and O2 are the optical centers of the two cameras, and their parameters are known. 1img and P 2img It is the imaging point of the pupil center on the imaging plane, which is calibrated by the pupil image coordinates. 1img The straight line from the optical center O1 passes through the pupil P in space c , through P 2img The straight line with the optical center O2 also passes through the pupil P c , therefore, P 1img and O1 to get the fifth line, and P 2img The sixth line obtained by connecting with O2 passes through the pupil P c , the pupil space coordinates can be obtained by calculating the intersection of the fifth line and the sixth line.

[0048] Step S105: Calculate the spatial coordinates of the corneal center based on the optical centers of the two cameras, the spatial coordinates of the two light sources relative to each camera, and the coordinates of the light source spots.

[0049] Optionally, in one embodiment, the above step S105 includes:

[0050] Step S1051: Determine the connection line between the optical center of the first camera and the corneal center based on the optical center of the first camera, the spatial coordinates of the two light sources relative to the first camera, and the coordinates of the light source spots in the first image, to obtain the first line;

[0051] Step S1052: Determine the connection line between the optical center of the second camera and the corneal center based on the optical center of the second camera, the spatial coordinates of the two light sources relative to the second camera, and the coordinates of the light source spots in the second image, to obtain the second line;

[0052] Step S1053: Calculate the intersection point of the first line and the second line to obtain the spatial coordinates of the corneal center.

[0053] Specifically, use the optical center O1 of the first camera and the two spots in the first image to solve the direction of the corneal center C, that is, the first line. Similarly, use the optical center O2 of the second camera and the two spots in the second image to also solve the direction of the corneal center C, that is, the second line. Both lines pass through the corneal center C, and calculating the intersection point of the two lines can obtain the corneal center C.

[0054] Among them, the two light sources include a first light source and a second light source. Further, the above step S1051 includes:

[0055] Step S10511: Use the optical center of the first camera, the spatial coordinates of the first light source relative to the first camera, and the coordinates of the spot of the first light source in the first image to form a first plane;

[0056] Step S10512: Use the optical center of the first camera, the spatial coordinates of the second light source relative to the first camera, and the coordinates of the spot of the second light source in the first image to form a second plane;

[0057] Step S10513: Calculate the intersection line of the first plane and the second plane to obtain the connection line between the optical center of the first camera and the corneal center, that is, the first line.

[0058] The above step S1052 includes:

[0059] Step S10521: Use the optical center of the second camera, the spatial coordinates of the first light source relative to the second camera, and the coordinates of the spot of the first light source in the second image to form a third plane;

[0060] Step S10522: A fourth plane is formed by using the optical center of the second camera, the spatial coordinates of the second light source relative to the second camera, and the spot coordinates of the second light source in the second image.

[0061] Step S10523: Calculate the intersection line of the third plane and the fourth plane to obtain the line connecting the optical center of the second camera and the corneal center, that is, the second line.

[0062] The processes of calculating the first line and the second line are basically the same. Here, only the calculation of any one of the two lines is taken as an example for the description in the specification. Figure 3 It is a schematic diagram for calculating the line connecting the optical center and the corneal center provided by the embodiment of the present invention. As Figure 3 shown, the camera optical center O, the first light source L1, and the spot coordinates G 1img formed by the first light source form a plane OL1G 1img . Similarly, the camera optical center O, the second light source L2, and the spot coordinates G 2img formed by the second light source form another plane OL2G 2img . Specifically, on the corneal surface, the light rays emitted by the light source enter the camera after being reflected by the cornea. The corneal surface can be regarded as a spherical surface. According to the principle of reflection, the incident light, the reflected light, and the normal line are in a plane. That is, at point G1, the incident light l 1、 , the reflected light r1, and the normal line n1 are in a plane. At point G2, the incident light l 2、 , the reflected light r2, and the normal line n2 are in a plane. Since the corneal surface is a spherical surface, the normal line passes through the corneal curvature center, abbreviated as the corneal center C. Therefore, the incident light, the reflected light, and the corneal center are in a plane. That is, the incident light l 1、 , the reflected light r1, and the corneal center C are in a plane OL1G 1img . The incident light l 2、 , the reflected light r2, and the corneal center C are in a plane OL2G 2img . Therefore, both the plane OL1G 1img and the plane OL2G 2img pass through the corneal center C. The intersection of the two planes can obtain the OC line, which is the line connecting the camera optical center and the corneal center.

[0063] Step S106: Connect the straight line of the pupil spatial coordinates and the corneal center spatial coordinates to obtain the target optical axis.

[0064] Continue to refer to Figure 3 , the direction defined by the pupil spatial coordinates P c and the corneal center spatial coordinates C, that is, the target optical axis.

[0065] Step S107: Determine the kappa angle of the target to be tracked.

[0066] Among them, the kappa angle is the fixed angle between the optical axis and the visual axis of the human eye.

[0067] Specifically, Figure 4 is a schematic diagram of the kappa angle provided by an embodiment of the present invention. As Figure 4 shown, the human eye model includes the vitreous body, the lens, the retina, and the cornea. There is a fixed included angle between the optical axis and the visual axis, the kappa angle. Therefore, to know the line-of-sight direction, the parameter of the kappa angle also needs to be known. Among them, any kappa angle measurement method in the prior art can be used, such as the perimeter method, etc. The kappa angle corresponding to the characteristics of the user (including age, gender, height, etc.) can also be preset according to empirical parameters, and then the corresponding kappa angle can be obtained according to the specific situation of the target to be tracked.

[0068] Optionally, in one embodiment, the above step S107 includes:

[0069] Step S1071: Collect a human eye image when the target to be tracked gazes at a predetermined point to obtain a detected human eye image.

[0070] Step S1072: Determine the spatial coordinates of the optical axis and the corneal center corresponding to the detected human eye image.

[0071] Step S1073: Connect the predetermined point and the spatial coordinates of the corneal center corresponding to the detected human eye image to obtain the visual axis.

[0072] Step S1074: Calculate the included angle between the optical axis and the visual axis corresponding to the detected human eye image to obtain the kappa angle.

[0073] Specifically, a certain determined predetermined point is set. When the target to be tracked gazes at this predetermined point, human eye images are collected by two cameras respectively. At this time, the collected human eye images are named detected human eye images. For the detected human eye images, according to the above steps S102 to S106, the steps of processing and calculating the target human eye images are performed to obtain the spatial coordinates of the optical axis and the corneal center corresponding to the detected human eye images. Then, connect the known predetermined point and the spatial coordinates of the corneal center corresponding to the detected human eye images to obtain the visual axis. Finally, calculate the included angle between the optical axis and the visual axis corresponding to the detected human eye images, and the kappa angle can be obtained.

[0074] Step S108: Determine the current line-of-sight direction of the target to be tracked according to the kappa angle and the target optical axis.

[0075] After obtaining the kappa angle, when the human eye gazes at other directions, the optical axis is calculated according to the above steps. Using the optical axis and the kappa angle, the line-of-sight direction of the human eye can be estimated.

[0076] In the line-of-sight tracking processing method provided in this embodiment, the tracking system uses dual cameras and dual light sources. The light source spots formed by the reflection of the light source by the human eye will be captured by the cameras, and a human eye image including the human eye pupil and the light source spots will be obtained, that is, two target human eye images captured by two cameras. Then, pupil detection and segmentation are performed on the target human eye images to obtain pupil region images. Further, pupil image coordinates and light source spot coordinates are determined in the pupil region images. Then, the pupil spatial coordinates are calculated based on the optical centers of the two cameras and the pupil image coordinates, and the corneal center spatial coordinates are calculated based on the optical centers of the two cameras, the spatial coordinates of the two light sources relative to each camera, and the light source spot coordinates. The straight line connecting the pupil spatial coordinates and the corneal center spatial coordinates is the target optical axis. After obtaining the optical axis, based on the fixed angle between the human eye optical axis and the visual axis, that is, the kappa angle, the current line-of-sight direction of the target to be tracked can be finally determined. By using the line-of-sight tracking processing method provided in this embodiment, a tracking system implemented based on dual cameras and dual light sources calculates the pupil coordinates in space by using the image of the pupil formed in the camera, calculates the corneal center position by using the coordinates of the light spot in the camera, obtains the connection line between the pupil and the cornea, that is, the optical axis, and finally determines the fixed angle kappa angle of the human body, and then the line-of-sight direction can be obtained, realizing the line-of-sight tracking of the dual-camera and dual-light-source system. Compared with the line-of-sight tracking system of multiple cameras and multiple light sources, it can reduce the introduction of errors and improve the accuracy of realizing tracking.

[0077] Optionally, in one embodiment, in the above step S105, the following steps are used to calculate the spatial coordinates of the light source relative to the camera: The small balls are placed at the first position and the second position within the imaging range of the camera respectively for shooting, and two small ball images are obtained, where the small ball images include the small balls and the small ball spots formed by the small balls reflecting the light rays emitted by the light source to the camera; for each small ball image, the following steps are used to calculate a plane equation respectively, and two plane equations are obtained: Determine the center-of-sphere image coordinates and the small ball spot coordinates in the small ball image; connect the center-of-sphere image coordinates and the optical center of the camera to obtain a third line; determine the spatial coordinates of the center of the small ball according to the third line, the position of the small ball and the radius of the small ball; connect the small ball spot coordinates and the optical center of the camera to obtain a fourth line; determine the spatial coordinates of the reflection point of the light ray emitted by the light source on the small ball according to the fourth line and the position of the small ball; determine the plane equation according to the spatial coordinates of the reflection point, the optical center of the camera and the spatial coordinates of the center of the sphere; calculate the intersection line of the corresponding planes according to the two plane equations to obtain the light source line where the light source is located; and determine the spatial coordinates of the light source relative to the camera on the light source line.

[0078] Since the dual cameras are required to capture images of the human eye and cannot directly capture the light source, the spatial coordinates of the light source relative to the cameras cannot be directly obtained through stereo imaging. In this embodiment, a small ball with a smooth and reflective surface is placed within the imaging range of the cameras. The small ball can reflect the light emitted by the light source into the cameras, forming small ball light spots. Based on the positions of the small ball light spots and the actual position of the small ball in the images, the positional relationship of the light source relative to the cameras can be calculated. Specifically, when calculating the spatial coordinates of a light source relative to a camera, the small ball is placed at a position within the imaging range of the camera and photographed first to obtain a small ball image, which includes the small ball and the small ball light spot formed by the small ball reflecting the light emitted by the light source into the camera. Figure 5 Schematic diagram for calculating the plane equation provided by the embodiment of the present invention, as Figure 5 shown, determine the image coordinates C of the center of the ball img and the small ball light spot coordinates G img in the small ball image, and connect the image coordinates C of the center of the ball img and the optical center O of the camera to obtain the third line OC img . According to the third line OC img , the position of the small ball and the radius of the small ball, determine the spatial coordinates C of the center of the ball of the small ball; connect the small ball light spot coordinates G img and the optical center O of the camera to obtain the fourth line OG img ; according to the fourth line OG img and the position of the small ball, determine the spatial coordinates G of the reflection point of the light ray emitted by the light source L on the small ball; determine a plane equation based on the spatial coordinates G of the reflection point, the optical center O of the camera and the spatial coordinates C of the center of the ball.

[0079] Similarly, place the small ball at another position within the imaging range of the camera and photograph it again to obtain another small ball image. According to the above process, the spatial coordinates of another reflection point, the optical center of the camera and the spatial coordinates of another center of the ball can be obtained to determine another plane equation.

[0080] The two plane equations calibrate two planes, and the two planes intersect at a straight line including the optical center O of the camera and the light source L. In this embodiment, this line is named the light source line. It can be determined that the light source L is located on this light source line, and finally the position of the light source is determined on the light source line, that is, the spatial coordinates of the light source relative to the camera are determined. It should be noted that in practical applications, the small ball can be placed at multiple positions for photographing and calculation, and finally the average value can be taken.

[0081] Further optionally, the step of determining the spatial coordinates of the light source relative to the camera on the light source line includes: measuring the actual distances between the light source and the small ball, and between the small ball and the camera; determining the initial length of the iteration according to the actual distances, and iterating the positions of the light source and the small ball according to a predetermined iteration step length; calculating whether the incident angle is equal to the reflection angle each time of iteration; if the incident angle is equal to the reflection angle, determining the position of the light source as a feasible solution; when the number of iterations meets the iteration threshold, performing weighted average on multiple feasible solutions to obtain the spatial coordinates of the light source relative to the camera.

[0082] Specifically, for the position of the center of the ball and any position of the light source, they are always on the light source line. At the same time, the position of the light source, the position of the center of the ball, and the optical center of the camera must conform to the law of physical reflection, that is, the incident angle is equal to the reflection angle. On this basis, the actual distances between the light source and the small ball, and between the small ball and the camera can be measured first, and a length slightly less than the two actual distances is taken as the initial length. According to the predetermined iteration step length and the number of iterations, the positions of the center of the ball and the light source are iterated. During the iteration process, it is calculated whether the corresponding coordinates conform to the law of physical reflection, that is, whether the incident angle is equal to the reflection angle each time of iteration. When the difference between the incident angle and the reflection angle is within the set threshold range, the light source coordinates are a feasible solution. The position of the small ball is changed multiple times, multiple feasible solutions are calculated and obtained, and weighted average is performed on the multiple feasible solutions to fit the final light source coordinates, that is, the spatial coordinates of the light source relative to the camera. Figure 6 FIG. is a schematic diagram of the iteration of the position of the small ball provided by an embodiment of the present invention. Figure 7 FIG. is a schematic diagram of the iteration of the position of the light source provided by an embodiment of the present invention. As Figure 6 and Figure 7 shown, the small ball iterates at different positions C1, C2, and C3. Each time a position is iterated, it is calculated whether the incident angle is equal to the reflection angle when the light source is at different positions L1', L1", and L1'''. When the incident angle is equal to the reflection angle, it can be determined that this position is a feasible solution. After multiple iterations, multiple feasible solutions are obtained when the number of iterations is met, and the final light source coordinates are fitted by weighted average.

[0083] Embodiment 2

[0084] Corresponding to the above Embodiment 1, Embodiment 2 of the present invention provides a line-of-sight tracking processing device. The corresponding technical feature details and corresponding technical effects can be referred to the above Embodiment 1 and will not be elaborated herein. Figure 8 FIG. is a block diagram of the line-of-sight tracking processing device provided by Embodiment 2 of the present invention. As Figure 8 shown, the line-of-sight tracking processing device includes: a first acquisition module 201, a processing module 202, a first determination module 203, a first calculation module 204, a second calculation module 205, a connection module 206, a second determination module 207, and a third determination module 208.

[0085] The first acquisition module 201 is configured to acquire the current eye images of the target to be tracked, and obtain two target eye images, where the two target eye images are respectively acquired by two cameras, and each of the target eye images has light source spots formed by the reflection of two light sources in the eyes; the processing module 202 is configured to perform pupil detection and segmentation on the target eye images to obtain pupil region images; the first determination module 203 is configured to determine pupil image coordinates and light source spot coordinates in the pupil region images; the first calculation module 204 is configured to calculate pupil spatial coordinates according to the optical centers of the two cameras and the pupil image coordinates; the second calculation module 205 is configured to calculate corneal center spatial coordinates according to the optical centers of the two cameras, the spatial coordinates of the two light sources relative to each camera, and the light source spot coordinates; the connection module 206 is configured to connect the straight line between the pupil spatial coordinates and the corneal center spatial coordinates to obtain a target optical axis; the second determination module 207 is configured to determine the kappa angle of the target to be tracked, where the kappa angle is a fixed angle between the human eye optical axis and the visual axis; and the third determination module 208 is configured to determine the current line of sight direction of the target to be tracked according to the kappa angle and the target optical axis.

[0086] Optionally, in an embodiment, the two target eye images include a first image acquired by a first camera and a second image acquired by a second camera. The second calculation module 205 includes: a first determination unit configured to determine the connection line between the optical center of the first camera and the corneal center according to the optical center of the first camera, the spatial coordinates of the two light sources relative to the first camera, and the light source spot coordinates in the first image, to obtain a first line; a second determination unit configured to determine the connection line between the optical center of the second camera and the corneal center according to the optical center of the second camera, the spatial coordinates of the two light sources relative to the second camera, and the light source spot coordinates in the second image, to obtain a second line; a first calculation unit configured to calculate the intersection point of the first line and the second line to obtain the corneal center spatial coordinates.

[0087] Optionally, in an embodiment, the two light sources include a first light source and a second light source. The first determination unit specifically performs the following steps: forming a first plane by using the optical center of the first camera, the spatial coordinates of the first light source relative to the first camera, and the light source spot coordinates of the first light source in the first image; forming a second plane by using the optical center of the first camera, the spatial coordinates of the second light source relative to the first camera, and the light source spot coordinates of the second light source in the first image; calculating the intersection line of the first plane and the second plane to obtain the connection line between the optical center of the first camera and the corneal center.

[0088] Optionally, in one embodiment, the second determination module 207 includes: an acquisition unit configured to acquire an eye image of the target to be tracked when looking at a predetermined point to obtain a detected eye image; a third determination unit configured to determine the optical axis and the corneal center spatial coordinates corresponding to the detected eye image; a connection unit configured to connect the predetermined point and the corneal center spatial coordinates corresponding to the detected eye image to obtain a visual axis; and a second calculation unit configured to calculate an angle between the optical axis corresponding to the detected eye image and the visual axis to obtain the kappa angle.

[0089] Optionally, in one embodiment, the second calculation module 205 calculates the spatial coordinates of the light source relative to the camera by the following steps: placing the small ball at the first position and the second position within the imaging range of the camera respectively for shooting to obtain two small ball images, where each small ball image includes the small ball and a small ball light spot formed by the small ball reflecting the light emitted by the light source to the camera; for each small ball image, calculating a plane equation respectively by the following steps to obtain two plane equations: determining the center image coordinates and the small ball light spot coordinates in the small ball image; connecting the center image coordinates and the optical center of the camera to obtain a third line; determining the spatial coordinates of the center of the small ball according to the third line, the position of the small ball and the radius of the small ball; connecting the small ball light spot coordinates and the optical center of the camera to obtain a fourth line; determining the spatial coordinates of the reflection point of the light emitted by the light source on the small ball according to the fourth line and the position of the small ball; determining the plane equation according to the spatial coordinates of the reflection point, the optical center of the camera and the spatial coordinates of the center of the small ball; calculating the intersection line of the corresponding planes according to the two plane equations to obtain the light source line where the light source is located; and determining the spatial coordinates of the light source relative to the camera on the light source line.

[0090] Optionally, in one embodiment, the step of determining the spatial coordinates of the light source relative to the camera on the light source line includes: measuring the actual distances between the light source and the small ball and between the small ball and the camera; determining an initial length of iteration according to the actual distances, and iterating the positions of the light source and the small ball according to a predetermined iteration step size; calculating whether the incident angle is equal to the reflection angle every time of iteration; if the incident angle is equal to the reflection angle, determining the position of the light source as a feasible solution; when the number of iterations meets an iteration threshold, performing weighted averaging on multiple feasible solutions to obtain the spatial coordinates of the light source relative to the camera.

[0091] Embodiment III

[0092] Embodiment III of the present invention provides a line-of-sight tracking processing system. Figure 9 For the block diagram of the line-of-sight tracking processing system provided in Embodiment III of the present invention, as Figure 9As shown, the line-of-sight tracking processing system includes a first camera 301, a second camera 302, two light sources 303, and a processing device 304. The processing device 304 is used for any line-of-sight tracking processing method provided in the first embodiment above. Corresponding technical feature details and corresponding technical effects can be referred to the first embodiment above, and will not be elaborated in this embodiment.

[0093] Embodiment 4

[0094] This embodiment also provides a computer device, such as a smart phone, a tablet computer, a notebook computer, a desktop computer, a rack server, a blade server, a tower server, or a cabinet server (including an independent server, or a server cluster composed of multiple servers) that can execute programs. As Figure 3 shown, the computer device 01 of this embodiment at least includes, but is not limited to: a memory 012 and a processor 011 that can communicate with each other through a system bus, as Figure 10 shown. It should be noted that Figure 10 only the computer device 01 with components memory 012 and processor 011 is shown, but it should be understood that it is not required to implement all the shown components, and more or fewer components can be implemented alternatively.

[0095] In this embodiment, the memory 012 (i.e., the readable storage medium) includes flash memory, a hard disk, a multimedia card, a card-type memory (e.g., SD or DX memory, etc.), a random access memory (RAM), a static random access memory (SRAM), a read-only memory (ROM), an electrically erasable programmable read-only memory (EEPROM), a programmable read-only memory (PROM), a magnetic memory, a magnetic disk, an optical disk, etc. In some embodiments, the memory 012 can be an internal storage unit of the computer device 01, such as the hard disk or memory of the computer device 01. In other embodiments, the memory 012 can also be an external storage device of the computer device 01, such as a plug-in hard disk equipped on the computer device 01, a Smart Media Card (SMC), a Secure Digital (SD) card, a Flash Card, etc. Of course, the memory 012 can also include both the internal storage unit and the external storage device of the computer device 01. In this embodiment, the memory 012 is generally used to store the operating system and various application software installed on the computer device 01, such as the program code of the line-of-sight tracking processing device in Embodiment 2. In addition, the memory 012 can also be used to temporarily store various data that have been output or will be output.

[0096] In some embodiments, the processor 011 may be a Central Processing Unit (CPU), a controller, a microcontroller, a microprocessor, or other data processing chips. The processor 011 is generally used to control the overall operation of the computer device 01. In this embodiment, the processor 011 is used to run the program code stored in the memory 012 or process data, such as the gaze tracking processing method, etc.

[0097] Embodiment Five

[0098] This embodiment also provides a computer-readable storage medium, such as a flash memory, a hard disk, a multimedia card, a card-type memory (e.g., SD or DX memory, etc.), a Random Access Memory (RAM), a Static Random Access Memory (SRAM), a Read-Only Memory (ROM), an Electrically Erasable Programmable Read-Only Memory (EEPROM), a Programmable Read-Only Memory (PROM), a magnetic memory, a magnetic disk, an optical disk, a server, an App application mall, etc. A computer program is stored thereon, and when the program is executed by a processor, corresponding functions are implemented. The computer-readable storage medium of this embodiment is used to store the gaze tracking processing device, and when executed by a processor, the gaze tracking processing method of Embodiment One is implemented.

[0099] It should be noted that in this article, the term "including", "comprising", or any other variation thereof is intended to cover a non-exclusive inclusion, such that a process, method, article, or device including a series of elements not only includes those elements but also includes other elements not expressly listed, or further includes elements inherent to such process, method, article, or device. Without further limitations, an element defined by the statement "including a..." does not exclude the existence of additional identical elements in the process, method, article, or device including that element.

[0100] The serial numbers of the above embodiments of the present invention are only for description and do not represent the superiority or inferiority of the embodiments.

[0101] Through the description of the above embodiments, those skilled in the art can clearly understand that the above-described embodiment methods can be implemented by means of software plus a necessary general hardware platform. Of course, they can also be implemented by hardware, but in many cases, the former is a better implementation method.

[0102] The above are only the preferred embodiments of the present invention, and do not limit the patent scope of the present invention accordingly. Any equivalent structure or equivalent process transformation made by using the specification and drawings of the present invention, or directly or indirectly applied to other related technical fields, shall be equally included in the patent protection scope of the present invention.

Claims

1. A method for gaze tracking processing, characterized in that, Comprising: Obtaining current eye images of a target to be tracked to obtain two target eye images, wherein the two target eye images are respectively acquired by two cameras, and each of the target eye images has light source spots formed by reflection of two light sources in the eyes; Performing pupil detection and segmentation on the target eye images to obtain pupil region images; Determining pupil image coordinates and light source spot coordinates in the pupil region images; Calculating pupil spatial coordinates according to the optical centers of the two cameras and the pupil image coordinates; Calculating corneal center spatial coordinates according to the optical centers of the two cameras, the spatial coordinates of the two light sources relative to each camera, and the light source spot coordinates; Connecting a straight line between the pupil spatial coordinates and the corneal center spatial coordinates to obtain a target optical axis; Determining the kappa angle of the target to be tracked, wherein the kappa angle is a fixed angle between the human eye optical axis and the visual axis; and Determining the current line of sight direction of the target to be tracked according to the kappa angle and the target optical axis.

2. The line-of-sight tracking processing method according to claim 1, wherein The two target eye images include a first image acquired by a first camera and a second image acquired by a second camera. The step of calculating corneal center spatial coordinates according to the optical centers of the two cameras, the spatial coordinates of the two light sources relative to each camera, and the light source spot coordinates includes: Determining a connection line between the optical center of the first camera and the corneal center according to the optical center of the first camera, the spatial coordinates of the two light sources relative to the first camera, and the light source spot coordinates in the first image to obtain a first line; Determining a connection line between the optical center of the second camera and the corneal center according to the optical center of the second camera, the spatial coordinates of the two light sources relative to the second camera, and the light source spot coordinates in the second image to obtain a second line; Calculating the intersection point of the first line and the second line to obtain the corneal center spatial coordinates.

3. The method for gaze tracking processing according to claim 2, characterized in that, The two light sources include a first light source and a second light source. The step of determining a connection line between the optical center of the first camera and the corneal center according to the optical center of the first camera, the spatial coordinates of the two light sources relative to the first camera, and the light source spot coordinates in the first image to obtain a first line includes: Using the optical center of the first camera, the spatial coordinates of the first light source relative to the first camera, and the light source spot coordinates of the first light source in the first image to form a first plane; Using the optical center of the first camera, the spatial coordinates of the second light source relative to the first camera, and the light source spot coordinates of the second light source in the first image to form a second plane; Calculating the intersection line of the first plane and the second plane to obtain the connection line between the optical center of the first camera and the corneal center.

4. The line-of-sight tracking processing method according to claim 1, characterized in that The step of determining the kappa angle of the target to be tracked includes: Collecting an eye image of the target to be tracked when looking at a predetermined point to obtain a detected eye image; Determining the optical axis and corneal center spatial coordinates corresponding to the detected eye image; Connecting the predetermined point and the corneal center spatial coordinates corresponding to the detected eye image to obtain a visual axis; and Calculating the included angle between the optical axis corresponding to the detected eye image and the visual axis to obtain the kappa angle.

5. The line-of-sight tracking processing method according to claim 1, wherein The following steps are used to calculate the spatial coordinates of the light source relative to the camera: The small ball is placed at the first position and the second position within the imaging range of the camera respectively for shooting, and two small ball images are obtained. Wherein, the small ball images include the small ball and the small ball spot formed by the light source reflecting the light to the camera; For each of the small ball images, the following steps are respectively used to calculate a plane equation, and two plane equations are obtained: Determine the center image coordinates and the small ball spot coordinates in the small ball image; Connect the center image coordinates and the optical center of the camera to obtain the third line; Determine the spatial coordinates of the center of the small ball according to the third line, the position of the small ball and the radius of the small ball; Connect the small ball spot coordinates and the optical center of the camera to obtain the fourth line; Determine the spatial coordinates of the reflection point of the light ray emitted by the light source on the small ball according to the fourth line and the position of the small ball; Determine the plane equation according to the spatial coordinates of the reflection point, the optical center of the camera and the spatial coordinates of the center of the ball; Calculate the intersection line of the corresponding planes according to the two plane equations to obtain the light source line where the light source is located; and Determine the spatial coordinates of the light source relative to the camera on the light source line.

6. The line-of-sight tracking processing method according to claim 5, wherein The steps of determining the spatial coordinates of the light source relative to the camera on the light source line include: Measure the actual distances between the light source and the small ball and between the small ball and the camera; Determine the initial length of the iteration according to the actual distances, and iterate the positions of the light source and the small ball according to a predetermined iteration step size; Each time an iteration is performed, calculate whether the incident angle is equal to the reflection angle; If the incident angle is equal to the reflection angle, determine the position of the light source as a feasible solution; When the number of iterations meets the iteration threshold, perform a weighted average on multiple feasible solutions to obtain the spatial coordinates of the light source relative to the camera.

7. A line-of-sight tracking processing device, characterized in that Include: The first acquisition module is used to acquire the current human eye images of the target to be tracked, and obtain two target human eye images. Wherein, the two target human eye images are respectively acquired by two cameras, and each target human eye image has light source spots formed by the reflection of two light sources in the human eye; The processing module is used to perform pupil detection and segmentation on the target human eye images to obtain pupil region images; The first determination module is used to determine the pupil image coordinates and the light source spot coordinates in the pupil region images; The first calculation module is used to calculate the pupil spatial coordinates according to the optical centers of the two cameras and the pupil image coordinates; The second calculation module is used to calculate the corneal center spatial coordinates according to the optical centers of the two cameras, the spatial coordinates of the two light sources relative to each camera, and the light source spot coordinates; The connection module is used to connect the straight line of the pupil spatial coordinates and the corneal center spatial coordinates to obtain the target optical axis; The second determination module is used to determine the kappa angle of the target to be tracked, where the kappa angle is a fixed angle between the human eye optical axis and the visual axis; and The third determination module is used to determine the current line-of-sight direction of the target to be tracked according to the kappa angle and the target optical axis.

8. A line-of-sight tracking processing system, characterized in that It includes a first camera, a second camera, two light sources, and a processing device. Among them, the processing device is used to execute the line-of-sight tracking processing method described in any one of claims 1 to 6.

9. A computer device, comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, When the processor executes the computer program, it realizes the steps of the method described in any one of claims 1 to 6.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by the processor, it realizes the steps of the method described in any one of claims 1 to 6.

Images