Virtual light source information determination method, device and storage medium for line-of-sight tracking

Abstract

The application discloses a virtual light source information determination method and device for line-of-sight tracking, equipment and a storage medium, the method is applied to a line-of-sight estimation system, the line-of-sight estimation system comprises a real light source and a plurality of cameras, and the method comprises the following steps: determining the position of a preset virtual light source in a three-dimensional space; determining a plane that passes through the principal point of the camera and is parallel to a first straight line that passes through the real light source and the virtual light source as a virtual camera plane of the camera; mapping a real light spot image point formed by the real light source on a real image plane of the camera to the virtual camera plane to obtain a first mapping point corresponding to the real light source on the virtual camera plane; based on the first mapping point, a second mapping point corresponding to the virtual light source on the virtual camera plane is calculated; and the second mapping point is reversely mapped to the real image plane to obtain a virtual light spot image point formed by the virtual light source on the real image plane.

Description

Cross-references to related applications

[0001] This application claims priority to Chinese Patent Application No. CN202610157899.0, filed on February 4, 2026, entitled "Method, Apparatus, Device and Storage Medium for Determining Light Source Information of Virtual Light Source", the entire contents of which are incorporated herein by reference. Technical Field

[0002] This invention relates to the field of gaze estimation technology, and in particular to a method, apparatus and storage medium for determining virtual light source information for gaze tracking. Background Technology

[0003] In the field of line-of-sight estimation based on 3D geometric models, different hardware configurations have their own characteristics in terms of implementation principles and performance, while also facing their own inherent technical challenges.

[0004] Currently, mainstream technical solutions have formed different technical paths based on the combination of the number of cameras and light sources. Among them, multi-camera multi-light source systems can provide the richest observational information and theoretically achieve the highest estimation accuracy, but this advantage comes with high hardware costs, system complexity, and deployment difficulties. In contrast, to achieve a balance between cost and performance, researchers have developed a series of other hardware configuration schemes. For example, to reduce hardware costs while ensuring a certain level of accuracy, researchers have explored multi-camera single-light source systems with fewer light sources. This system retains the stereoscopic vision advantage of multi-camera systems to obtain accurate three-dimensional coordinates of the pupil center, but uses only one light source, thus generating only one light spot. In this case, due to the lack of a second light spot to directly constrain the shape of the corneal curvature, the system must know or assume a set of fixed eye parameters in advance, such as the corneal radius of curvature r, the distance k from the pupil center to the corneal center, and the refractive index of the aqueous humor. Using this prior knowledge and a single light spot, the approximate location of the corneal curvature center can be indirectly calculated, thereby constructing the optical axis.

[0005] In summary, existing technologies face a core objective limitation on the path to low-cost, high-precision line-of-sight estimation: given a fixed hardware configuration, the amount of independent observation information available for computation is limited. This is particularly true in potentially viable architectures like multi-camera single-light-source systems, where the single physical light source becomes an information bottleneck. Therefore, exploring new methods to improve the utilization rate of effective information without altering the physical hardware, achieving true low cost, high precision, and high robustness, is a crucial direction for overcoming current limitations.

[0006] There is currently no effective solution to the above-mentioned defects. Summary of the Invention

[0007] The purpose of this invention is to provide a method, apparatus and storage medium for determining virtual light source information for gaze tracking, which can solve the problems of limited accuracy and poor universality caused by the reliance on inaccurate general eye parameters due to the lack of information in existing gaze estimation systems based on multiple cameras and a single light source.

[0008] According to one aspect of the present invention, a method for determining virtual light source information for gaze tracking is provided, applied to a gaze estimation system, the gaze estimation system including a real light source and multiple cameras, the method comprising: Obtain the position of the pre-set virtual light source in three-dimensional space; The plane that passes through the principal point of the camera and is parallel to the first straight line passing through the real light source and the virtual light source is determined as the virtual camera plane of the camera. The real light spot image point formed by the real light source on the real image plane of the camera is mapped onto the virtual camera plane to obtain the first mapping point corresponding to the real light source on the virtual camera plane; wherein, the real light spot image point is the image of the light spot reflected by the real light source onto the target cornea on the real image plane; Based on the first mapping point, calculate the second mapping point corresponding to the virtual light source on the virtual camera plane; The second mapping point is reverse-mapped onto the real image plane to obtain the virtual light spot image point formed by the virtual light source on the real image plane.

[0009] Optionally, mapping the real light spot image points formed by the real light source on the real image plane of the camera onto the virtual camera plane to obtain the first mapping point corresponding to the real light source on the virtual camera plane includes: Determine the second straight line passing through the optical center of the camera and the actual light spot image point; Calculate the intersection point of the second straight line and the virtual camera plane, and use it as the first mapping point.

[0010] Optionally, calculating the second mapping point corresponding to the virtual light source on the virtual camera plane based on the first mapping point includes: Determine the direction vector of the first straight line as the target direction vector; On the virtual camera plane, a third straight line is determined that passes through the first mapping point and has the direction vector of the target direction vector; wherein the second mapping point is located on the third straight line; A plane passing through the optical center of the camera, the corneal curvature center of the target cornea, and the virtual light source is defined as the virtual light source plane; wherein, the second mapping point is located on the virtual light source plane; The second mapping point is calculated based on the third straight line and the virtual light source plane.

[0011] Optionally, calculating the second mapping point based on the third straight line and the virtual light source plane includes: Calculate the intersection point of the third straight line and the virtual light source plane, and use it as the second mapping point; or Calculate the intersection line between the virtual light source plane and the virtual camera plane, and calculate the intersection point between this intersection line and the third straight line, which is used as the second mapping point.

[0012] Optionally, the step of reverse mapping the second mapping point to the real image plane to obtain the virtual light spot image point formed by the virtual light source on the real image plane includes: Determine the fourth straight line passing through the optical center of the camera and the second mapping point; The intersection point of the fourth straight line and the real image plane is calculated and used as the virtual light spot image point.

[0013] Optionally, after mapping the second mapping point in reverse onto the real image plane to obtain the virtual light spot image point formed by the virtual light source on the real image plane, the method further includes: Based on the camera's preset lens distortion coefficient, the virtual light spot image points are subjected to distortion correction.

[0014] To achieve the above objectives, the present invention further provides a virtual light source information determination device for gaze tracking, applied to a gaze estimation system. The gaze estimation system includes a real light source, multiple cameras, and the virtual light source information determination device. The device includes: The acquisition module is used to acquire the position of a pre-set virtual light source in three-dimensional space; The determination module is used to determine a plane that passes through the principal point of the camera and is parallel to a first straight line passing through the real light source and the virtual light source, as the virtual camera plane of the camera; The first mapping module is used to map the real light spot image point formed by the real light source on the real image plane of the camera onto the virtual camera plane to obtain the first mapping point corresponding to the real light source on the virtual camera plane; wherein, the real light spot image point is the image of the light spot reflected by the real light source onto the target cornea on the real image plane; The calculation module is used to calculate the second mapping point corresponding to the virtual light source on the virtual camera plane based on the first mapping point; The second mapping module is used to reverse map the second mapping point to the real image plane to obtain the virtual light spot image point formed by the virtual light source on the real image plane.

[0015] Optionally, the first mapping module is specifically used for: Determine the second straight line passing through the optical center of the camera and the actual light spot image point; Calculate the intersection point of the second straight line and the virtual camera plane, and use it as the first mapping point.

[0016] To achieve the above objectives, the present invention also provides a computer device, the computer device comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the virtual light source information determination method for eye tracking described above.

[0017] To achieve the above objectives, the present invention also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is used to implement the steps of the virtual light source information determination method for eye-tracking described above.

[0018] The present invention provides a method, apparatus, device, and storage medium for determining virtual light source information for eye tracking. This invention solves the problems of limited accuracy and poor universality in existing eye estimation systems based on multi-camera single light sources, which rely on inaccurate general eye parameters due to a lack of information. The invention proposes a method that generates equivalent multi-light source observation information through algorithms without adding physical light sources, thereby increasing the effective information content of the system. This completely eliminates the dependence on prior parameters and fundamentally improves the accuracy and robustness of eye estimation, ultimately achieving high-performance eye tracking at low cost. Attached Figure Description

[0019] Various other advantages and benefits will become apparent to those skilled in the art upon reading the following detailed description of preferred embodiments. The accompanying drawings are for illustrative purposes only and are not intended to limit the invention. Furthermore, the same reference numerals denote the same parts throughout the drawings. In the drawings: Figure 1 A flowchart of the virtual light source information determination method for eye-tracking provided in Embodiment 1; Figure 2 This is a schematic diagram of the virtual light source information determination scheme for eye tracking provided in Embodiment 1; Figure 3 An architecture diagram of the line-of-sight estimation system provided in Example 1; Figure 4This is a block diagram of the virtual light source information determination device for eye tracking provided in Embodiment 2; Figure 5 This is a block diagram of a computer device suitable for implementing a virtual light source information determination method for eye-tracking, as provided in Embodiment 3. Detailed Implementation

[0020] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without inventive effort are within the scope of protection of this invention.

[0021] Gaze estimation technology infers a user's gaze point or gaze direction by analyzing eye images or head posture information. It can be widely used in scenarios such as driver fatigue detection, user attention analysis, medical auxiliary diagnosis, and interactive display systems.

[0022] Currently, gaze estimation methods based on 3D geometric models have become a mainstream research direction due to their clear physical meaning and high accuracy potential. The core of this method lies in utilizing features such as the pupil center captured by the camera and the light spot formed on the corneal surface by the light source, combined with an eye optical model, to geometrically calculate and invert the spatial pose and gaze direction of the eye. Its hardware foundation consists of a system composed of a camera and a light source. Based on the complexity of the system configuration, existing technologies can be mainly divided into: multi-camera multi-light source systems, multi-camera single-light source systems, single-camera multi-light source systems, and single-camera single-light source systems. Multi-camera multi-light source systems can provide the richest observation information and theoretically achieve the highest estimation accuracy, but this advantage comes with higher hardware costs, system complexity, and deployment difficulties. In contrast, to achieve a balance between cost and performance, researchers have developed a series of other hardware configuration schemes. The core common problem of these schemes is how to ensure the accuracy and robustness of the estimation under limited hardware resources.

[0023] To overcome the challenges posed by insufficient information, existing technologies primarily employ two optimization approaches. The first is to improve the completeness of hardware configuration, i.e., by using more and better-laid-out cameras and light sources; however, this directly contradicts the initial goal of cost reduction. The second approach involves compensation and optimization at the software algorithm level, such as by introducing user calibration processes to compensate for model errors.

[0024] In summary, existing technologies face a core objective limitation on the path to low-cost, high-precision line-of-sight estimation: given a fixed hardware configuration, the amount of independent observation information available for computation is limited. This is particularly true in potentially viable architectures like multi-camera single-light-source systems, where the single physical light source becomes an information bottleneck. Therefore, exploring new methods to improve the utilization rate of effective information without altering the physical hardware, achieving true low cost, high precision, and high robustness, is a crucial direction for overcoming current limitations.

[0025] This invention addresses multi-camera single-light source systems based on traditional gaze estimation methods by proposing a method to infer a series of virtual light sources from existing light sources. This method aims to overcome the inherent limitations of single-light source systems due to insufficient observation dimensions. The invention posits that the bright spots formed by light reflections on the cornea contain implicit information about the three-dimensional structure and relative position of the eyeball. The method of this invention mines and utilizes this information by calculating "where the reflection point should be if the light source were elsewhere" to synthesize new observation data and apply it to the calculation of gaze direction, thereby improving the accuracy and robustness of gaze estimation. Specific implementation methods are described in the following embodiments.

[0026] Example 1 Embodiment 1 of this invention provides a method for determining virtual light source information for gaze tracking, applied to a gaze estimation system, which includes a real light source and multiple cameras. The method can set at least one virtual light source. For each virtual light source, the virtual light spot image point formed by the virtual light source on the real image plane of each camera can be calculated. For example, if there are M virtual light sources and N cameras, then for each virtual light source, N virtual light spot image points need to be calculated; for M virtual light sources, M×N virtual light spot image points need to be calculated. The calculation method for each virtual light spot image point is the same. This invention uses the calculation of a virtual light spot image point formed by a certain virtual light source on the real image plane of a certain camera as an example to explain the specific calculation method. Figure 1 As shown, the method for determining virtual light source information for gaze tracking may include steps S1 to S5, wherein: Step S1: Obtain the position of the pre-set virtual light source in three-dimensional space.

[0027] When calculating the light source information of a virtual light source, a virtual light source is pre-set in three-dimensional space, and its position in that space is determined. Based on actual testing, placing the virtual light source further away from the real light source yields better results. The light source information for each virtual light source includes its position and the virtual light spot image point formed by the virtual light source on the real image plane of each camera. The virtual light spot image point is the image formed on the real image plane of the camera by the light spot reflected from the virtual light source onto the target cornea.

[0028] After setting up the virtual light source, it is necessary to calculate the light source information of the virtual light source, and then proceed to step S1. Obtaining the position of the virtual light source in three-dimensional space serves two purposes: firstly, to collect the light source information of the virtual light source, and secondly, to ensure that the location of the virtual light source is clearly known when it is used in subsequent steps.

[0029] Step S2: Determine a plane that passes through the principal point of the camera and is parallel to the first straight line passing through the real light source and the virtual light source, and use it as the virtual camera plane of the camera.

[0030] For each camera, a virtual camera plane is set up. The function of this plane is to project the relative positions of the real light source and the virtual light source in space onto the positional relationship of the first and second mapping points on the virtual camera plane, thereby calculating the position of the virtual light spot image point in the real image plane. Therefore, this virtual camera plane must pass through the principal image point of the camera and be parallel to the line connecting the real light source and the virtual light source.

[0031] Step S3: Map the real light spot image point formed by the real light source on the real image plane of the camera onto the virtual camera plane to obtain the first mapping point corresponding to the real light source on the virtual camera plane; wherein, the real light spot image point is the image of the light spot reflected by the real light source onto the target cornea on the real image plane.

[0032] Specifically, the real light spot image point is mapped onto the virtual camera plane along a straight line passing through the real light spot image point and the optical center of the camera, resulting in a mapped point, which is used as the first mapped point. This first mapped point serves as a reference point to transform the relative positional relationship between the real and virtual light sources in space onto the virtual camera plane, thereby calculating the position of the second mapped point reflected by the virtual light source on the virtual camera plane. Here, the optical center of the camera is the optical center point of the camera, a known parameter in the calibrated camera model.

[0033] This step establishes a bridge between real observation data and the virtual computational plane through the first mapping point. This first mapping point encapsulates the geometric relationship between the real light source, the target corneal reflection point, and the camera imaging center, and transforms this relationship into an expression on a virtual camera plane parallel to the virtual light source's orientation. This provides the necessary starting conditions and spatial reference for the next step of inferring and calculating the image point position corresponding to the virtual light source within the same planar system.

[0034] Optionally, mapping the real light spot image points formed by the real light source on the real image plane of the camera onto the virtual camera plane to obtain the first mapping point corresponding to the real light source on the virtual camera plane includes: Determine the second straight line passing through the optical center of the camera and the actual light spot image point; Calculate the intersection point of the second straight line and the virtual camera plane, and use it as the first mapping point.

[0035] Specifically, each image point on the image plane corresponds to a unique ray originating from the camera's optical center, passing through that image point, and extending into three-dimensional space. This process is called "back projection." The three-dimensional coordinates of the camera's optical center and the two-dimensional pixel coordinates of the actual light spot image points detected from the image are known. The camera's optical center coordinates are derived from system calibration parameters, while the actual light spot image point coordinates are obtained through image processing algorithms. The two-dimensional pixel coordinates of the actual light spot image points, combined with camera intrinsic parameters (such as focal length and principal point), are back projected into three-dimensional space using the camera calibration model. This determines a straight line passing through the camera's optical center and the corresponding three-dimensional direction point of that image point; this straight line is the second straight line.

[0036] This step establishes a stable reference point with physical geometric constraints for the entire virtual light source algorithm by accurately mapping real light spot images onto the virtual camera plane. This simplifies the complex spatial reflection relationship into geometric operations within the plane, significantly improving the accuracy of subsequent calculations and the robustness of the overall system while ensuring the physical authenticity of the synthesized data.

[0037] Step S4: Based on the first mapping point, calculate the second mapping point corresponding to the virtual light source on the virtual camera plane.

[0038] Specifically, step S4 includes: Determine the direction vector of the first straight line as the target direction vector; A plane passing through the optical center of the camera, the corneal curvature center of the target cornea, and the virtual light source is defined as the virtual light source plane; wherein, the second mapping point is located on the virtual light source plane; The second mapping point is calculated based on the first mapping point, the target direction vector, and the virtual light source plane.

[0039] The first mapping point is the mapping of the complete optical path of the real light source onto the virtual camera plane, providing a definite two-dimensional spatial coordinate origin and reference for calculating the image points of the virtual light source. The target direction vector ensures that the relative orientation between the image points of the real and virtual light sources on the virtual camera plane is strictly consistent with their actual relative orientation in three-dimensional space, thus maintaining the topological authenticity of the observation layout. A plane is defined by the center of curvature of the target cornea as the virtual light source plane. This virtual light source plane passes through the optical center of the camera, the center of curvature of the target cornea, and the virtual light source. Geometrically, this virtual light source plane is the plane formed by the line connecting the center of curvature of the cornea and the camera, and the position of the virtual light source, used for subsequent intersection point calculations. The position of the center of curvature of the target cornea can be calculated using existing methods, which will not be elaborated here.

[0040] This step generates a point with clear physical meaning—the second mapping point—at the algorithmic level. This point is essentially the theoretical imaging position of the virtual light source on the virtual camera plane under the current user's specific eye state. Furthermore, by transforming limited single-light source physical observations into rich virtual observation data containing multi-light source geometric constraints, this step fundamentally enhances the constraints and universality of the gaze estimation model, while significantly reducing its dependence on general prior eye parameters.

[0041] Optionally, calculating the second mapping point corresponding to the virtual light source on the virtual camera plane based on the first mapping point includes: Determine the direction vector of the first straight line as the target direction vector; On the virtual camera plane, a third straight line is determined that passes through the first mapping point and has the direction vector of the target direction vector; wherein the second mapping point is located on the third straight line; A plane passing through the optical center of the camera, the corneal curvature center of the target cornea, and the virtual light source is defined as the virtual light source plane; wherein, the second mapping point is located on the virtual light source plane; The second mapping point is calculated based on the third straight line and the virtual light source plane.

[0042] Calculate the direction vector of the line connecting the virtual light source and the real light source in the actual space, and use it as the target direction vector. After the calculation is completed, map this positional relationship onto the virtual camera plane to obtain the relative positional relationship of the two light spots (i.e., the first mapping point and the second mapping point).

[0043] The target direction vector defines the relative arrangement of the real and virtual light sources in three-dimensional space. This constraint requires that the relative orientation between the second and first mapping points of the virtual light source on the virtual camera plane must be consistent with the actual relative orientation of the two light sources in three-dimensional space. This ensures the consistency of the topological relationship between the virtual observation and the real system.

[0044] Optionally, calculating the second mapping point based on the third straight line and the virtual light source plane includes: Calculate the intersection point of the third straight line and the virtual light source plane, and use it as the second mapping point; or Calculate the intersection line between the virtual light source plane and the virtual camera plane, and calculate the intersection point between this intersection line and the third straight line, which is used as the second mapping point.

[0045] Since the second mapping point exists both on the virtual light source plane and on the virtual camera plane along a straight line with the target direction vector as its direction vector and passing through the first mapping point, the position of the second mapping point on the virtual camera plane can be calculated by finding the intersection of the virtual light source plane and this straight line. Alternatively, the position of the second mapping point can be calculated by finding the intersection of the virtual light source plane and the virtual camera plane, where the second mapping point lies, and then finding the intersection of this intersection line with the third straight line.

[0046] Step S5: The second mapping point is reverse-mapped onto the real image plane to obtain the virtual light spot image point formed by the virtual light source on the real image plane.

[0047] Any point on the virtual camera plane can be considered a virtual image point originating from a ray of light in three-dimensional space. To find the corresponding position of that point on the real image plane, it is necessary to trace the ray backward and find out where it intersects with the real image plane.

[0048] Optionally, the step of reverse mapping the second mapping point to the real image plane to obtain the virtual light spot image point formed by the virtual light source on the real image plane includes: Determine the fourth straight line passing through the optical center of the camera and the second mapping point; The intersection point of the fourth straight line and the real image plane is calculated and used as the virtual light spot image point.

[0049] Specifically, a spatial ray is established starting from the camera's optical center and passing through a calculation point on the virtual camera plane to connect the virtual computational domain and the real imaging domain. In three-dimensional space, the connection point between the camera's optical center and the second mapping point uniquely determines a spatial straight line, namely the fourth straight line. Geometrically, this line represents a virtual ray path: assuming there exists a three-dimensional point whose emitted light, after reflection by the eye, passes through the camera's optical center and is imaged precisely at the second mapping point on the virtual camera plane. The fourth straight line is the set of all rays that meet this condition. By calculating the intersection of this ray path with the real image plane, the theoretical values ​​on the virtual camera plane can be accurately projected onto the pixel coordinate system used in the actual image acquisition, ensuring that the generated virtual spot images and the real spot images detected directly from the image are on the same coordinate reference and physical scale.

[0050] This invention achieves a complete technical loop, starting from a real two-dimensional observation of a single physical light source, and through a series of derivations based on geometric optics and ocular physiological parameters, ultimately generating an equivalent two-dimensional observation of another virtual light source. This loop demonstrates the feasibility of enhancing the system's observation capabilities through algorithms without increasing hardware.

[0051] Optionally, after mapping the second mapping point in reverse onto the real image plane to obtain the virtual light spot image point formed by the virtual light source on the real image plane, the method further includes: Based on the camera's preset lens distortion coefficient, the virtual light spot image points are subjected to distortion correction.

[0052] Specifically, if camera distortion coefficients are provided, the coordinates of virtual spot images can be corrected for distortion, thus more accurately reflecting the imaging characteristics of the actual camera. Lens distortion is a systematic error between the actual camera lens and the ideal pinhole camera model, mainly including radial distortion and tangential distortion. Distortion correction can significantly improve the accuracy of virtual spot positioning, especially when using wide-angle or fisheye lenses, where the distortion effect in the edge areas is more pronounced. The Brown-Conrady model can be used to combine radial and tangential distortion to form a complete distortion correction model.

[0053] This invention proposes an innovative gaze-tracking technology, the core of which lies in generating virtual light sources and their corresponding corneal reflection points through algorithms. Compared with existing technologies, this solution provides additional geometric constraint information for the gaze estimation process without changing the hardware configuration, which helps improve the system's information utilization efficiency. In practice, this solution introduces observation information equivalent to multiple light sources within a multi-camera, single-light-source system framework, providing an innovative technical path to improve the accuracy and stability of gaze estimation. From an implementation perspective, this method does not involve hardware modifications; its effectiveness mainly depends on the quality of the algorithm implementation and the optimization level of the image feature extraction and gaze estimation modules. Furthermore, theoretically, by adding effective geometric constraints, the system's dependence on prior eye physiological parameters can be reduced, which significantly reduces model adaptation errors caused by individual differences when used by different users. Simultaneously, since the number of actual light sources in the system is small, the interference of light spots formed by reflections on the cornea on the extraction of features such as pupil contours is reduced, ensuring the quality of front-end image feature extraction and significantly enhancing the system's accuracy and robustness. Finally, this scheme also provides a certain degree of flexibility in system design. By adjusting the number and layout of virtual light sources at the algorithm level, a balance can be achieved between computational cost and expected accuracy.

[0054] The following is based on Figure 2 The implementation process of the present invention will be illustrated using an example.

[0055] like Figure 2 As shown, the actual light source L is the real light source, and C is the center of corneal curvature. The radius of the cornea is given, and the image plane is the camera's true image plane. For the real light source L reflected to Figure 2 The light spot on the cornea shown For the virtual light source V to reflect to Figure 2 The light spot on the cornea shown It is an unknown quantity; For real light spot images, As the first mapping point, For the second mapping point, This refers to virtual light spot images. The objective of this invention is to calculate... ,according to It can be calculated in reverse. .

[0056] Specifically, calculate the first mapping point of the real light source on the virtual camera plane. : The real light spot image points on the known image Connect the camera optical center O and The intersection of the line containing this line segment and the virtual camera plane is the desired point. The position of the virtual light source is determined by using the first mapping point as a reference point to transform the relative positions of the real and virtual light sources in space onto the virtual camera plane. This is then used to calculate the position of the second mapping point of the reflected light spot from the virtual light source on the virtual camera plane. Specifically, the direction vector of the line connecting the real and virtual light sources in the actual space is calculated. After the calculation, this positional relationship is mapped onto the virtual camera plane to obtain the relative positional relationship between the first and second mapping points on the virtual camera plane. Since the second mapping point exists both on the virtual light source plane and on the line with the calculated direction vector as its direction vector and passing through the first mapping point (this line is on the virtual camera plane), the intersection of the virtual light source plane and this line can be calculated to determine the position of the second mapping point of the virtual light source on the virtual camera plane. Finally, analogous to the process of transforming real light spot images from the real image plane to the virtual camera plane, the virtual light spot images on the virtual camera plane are projected back to the real image plane: connecting the camera optical center O and... The intersection of the line segment and the real image plane is the location of the light spot image formed by the reflection of the virtual light source, which is also the virtual light spot image point. This completes the overall calculation of how the virtual light source reflects off the cornea to form a light spot.

[0057] In the above calculation, the virtual camera plane is constructed to facilitate coordinate transformation with the real image plane. This plane passes through the principal point of the image and is parallel to the line connecting the two light sources. In this way, it is easy to transform the positions of the two light sources in space into the positional relationship between the two light spots on the image plane.

[0058] The present invention proposes a virtual light source design method for a multi-camera single-light source line-of-sight estimation system. By setting up a virtual light source on the basis of the original real light source, the limitation of the original system having few light sources is solved from the algorithm level. This method improves the accuracy and stability of line-of-sight estimation while reducing system complexity.

[0059] The architecture of the entire line-of-sight estimation system is as follows: Figure 3 As shown, the invention mainly comprises five parts, with the generation of virtual light sources being the core innovative module. Based on the geometric optics principle of corneal reflection, this invention utilizes actual light sources and acquired eye feature information to generate multiple equivalent virtual light sources and their corresponding reflected light spots through mirror transformation and projection calculations. This provides more geometric constraint information for gaze estimation, thereby improving the accuracy of gaze estimation. Specifically, the corneal curvature center is calculated based on the position of the actual light source in the system. Then, based on the position of the corneal curvature center, multiple virtual light sources and their corresponding corneal reflection points are generated through mirror transformation and geometric optics principles. The main steps include key parameter acquisition, virtual light spot position calculation, and consideration of lens distortion.

[0060] Example 2 Embodiment 2 of the present invention provides a virtual light source information determination device for gaze tracking, applied to a gaze estimation system, wherein the gaze estimation system includes a real light source, multiple cameras, and the virtual light source information determination device. Figure 4 As shown, the virtual light source information determination device 40 for eye tracking specifically includes the following components: The acquisition module 401 is used to acquire the position of a pre-set virtual light source in three-dimensional space; The determining module 402 is used to determine a plane that passes through the principal point of the camera and is parallel to the first straight line passing through the real light source and the virtual light source, as the virtual camera plane of the camera. The first mapping module 403 is used to map the real light spot image point formed by the real light source on the real image plane of the camera onto the virtual camera plane to obtain the first mapping point corresponding to the real light source on the virtual camera plane; wherein, the real light spot image point is the image of the light spot reflected by the real light source onto the target cornea on the real image plane; Calculation module 404 is used to calculate, based on the first mapping point, the second mapping point corresponding to the virtual light source on the virtual camera plane; The second mapping module 405 is used to reverse map the second mapping point to the real image plane to obtain the virtual light spot image point formed by the virtual light source on the real image plane.

[0061] Optionally, the first mapping module is specifically used for: Determine the second straight line passing through the optical center of the camera and the actual light spot image point; Calculate the intersection point of the second straight line and the virtual camera plane, and use it as the first mapping point.

[0062] Optionally, the computing module is specifically used for: Determine the direction vector of the first straight line as the target direction vector; On the virtual camera plane, a third straight line is determined that passes through the first mapping point and has the direction vector of the target direction vector; wherein the second mapping point is located on the third straight line; A plane passing through the optical center of the camera, the corneal curvature center of the target cornea, and the virtual light source is defined as the virtual light source plane; wherein, the second mapping point is located on the virtual light source plane; The second mapping point is calculated based on the third straight line and the virtual light source plane.

[0063] Optionally, when performing the step of calculating the second mapping point based on the third straight line and the virtual light source plane, the calculation module is specifically used for: Calculate the intersection point of the third straight line and the virtual light source plane, and use it as the second mapping point; or Calculate the intersection line between the virtual light source plane and the virtual camera plane, and calculate the intersection point between this intersection line and the third straight line, which is used as the second mapping point.

[0064] Optionally, the second mapping module is specifically used for: Determine the fourth straight line passing through the optical center of the camera and the second mapping point; The intersection point of the fourth straight line and the real image plane is calculated and used as the virtual light spot image point.

[0065] Optionally, the virtual light source information determination device for eye tracking further includes: The correction module is used to perform distortion correction on the virtual light spot image point based on the preset lens distortion coefficient of the camera after the second mapping point is reverse mapped to the real image plane to obtain the virtual light spot image point formed by the virtual light source on the real image plane.

[0066] Example 3 This embodiment also provides a computer device, such as a smartphone, tablet computer, laptop computer, desktop computer, rack server, blade server, tower server, or cabinet server (including a standalone server or a server cluster composed of multiple servers), etc., capable of executing programs. Figure 5 As shown, the computer device 50 in this embodiment includes, but is not limited to, a memory 501 and a processor 502 that are communicatively connected to each other via a system bus. It should be noted that... Figure 5 Only a computer device 50 with components 501-502 is shown; however, it should be understood that it is not required to implement all of the components shown, and more or fewer components may be implemented instead.

[0067] In this embodiment, the memory 501 (i.e., the readable storage medium) includes flash memory, hard disk, multimedia card, card-type memory (e.g., SD or DX memory), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, disk, optical disk, etc. In some embodiments, the memory 501 may be an internal storage unit of the computer device 50, such as the hard disk or memory of the computer device 50. In other embodiments, the memory 501 may also be an external storage device of the computer device 50, such as a plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, etc., equipped on the computer device 50. Of course, the memory 501 may include both the internal storage unit and the external storage device of the computer device 50. In this embodiment, the memory 501 is typically used to store the operating system and various application software installed on the computer device 50. In addition, the memory 501 may also be used to temporarily store various types of data that have been output or will be output.

[0068] In some embodiments, processor 502 may be a central processing unit (CPU), controller, microcontroller, microprocessor, or other data processing chip. This processor 502 is typically used to control the overall operation of computer device 50.

[0069] Specifically, in this embodiment, the processor 502 is used to execute the program for determining virtual light source information stored in the memory 501. When the program for determining virtual light source information for eye tracking is executed, it performs the following steps: Obtain the position of the pre-set virtual light source in three-dimensional space; The plane that passes through the principal point of the camera and is parallel to the first straight line passing through the real light source and the virtual light source is determined as the virtual camera plane of the camera. The real light spot image point formed by the real light source on the real image plane of the camera is mapped onto the virtual camera plane to obtain the first mapping point corresponding to the real light source on the virtual camera plane; wherein, the real light spot image point is the image of the light spot reflected by the real light source onto the target cornea on the real image plane; Based on the first mapping point, calculate the second mapping point corresponding to the virtual light source on the virtual camera plane; The second mapping point is reverse-mapped onto the real image plane to obtain the virtual light spot image point formed by the virtual light source on the real image plane.

[0070] For a detailed description of the above method steps, please refer to Example 1. This example will not be repeated here.

[0071] Example 4 This fourth embodiment also provides a computer-readable storage medium, such as flash memory, hard disk, multimedia card, card-type memory (e.g., SD or DX memory), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, disk, optical disk, server, App application store, etc., which stores a computer program. When the computer program is executed by a processor, it implements the following method steps: Obtain the position of the pre-set virtual light source in three-dimensional space; The plane that passes through the principal point of the camera and is parallel to the first straight line passing through the real light source and the virtual light source is determined as the virtual camera plane of the camera. The real light spot image point formed by the real light source on the real image plane of the camera is mapped onto the virtual camera plane to obtain the first mapping point corresponding to the real light source on the virtual camera plane; wherein, the real light spot image point is the image of the light spot reflected by the real light source onto the target cornea on the real image plane; Based on the first mapping point, calculate the second mapping point corresponding to the virtual light source on the virtual camera plane; The second mapping point is reverse-mapped onto the real image plane to obtain the virtual light spot image point formed by the virtual light source on the real image plane.

[0072] For a detailed description of the above method steps, please refer to Example 1. This example will not be repeated here.

[0073] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0074] The sequence numbers of the above embodiments of the present invention are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0075] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method.

[0076] The above are merely preferred embodiments of the present invention and do not limit the scope of the patent. Any equivalent structural or procedural transformations made based on the description and drawings of the present invention, or direct or indirect applications in other related technical fields, are similarly included within the scope of patent protection of the present invention.

Claims

1. A method for determining virtual light source information for gaze tracking, characterized in that, Applied to a gaze estimation system, the gaze estimation system comprising a real light source and multiple cameras, the method includes: Obtain the position of the pre-set virtual light source in three-dimensional space; The plane that passes through the principal point of the camera and is parallel to the first straight line passing through the real light source and the virtual light source is determined as the virtual camera plane of the camera. The real light spot image point formed by the real light source on the real image plane of the camera is mapped onto the virtual camera plane to obtain the first mapping point corresponding to the real light source on the virtual camera plane; wherein, the real light spot image point is the image of the light spot reflected by the real light source onto the target cornea on the real image plane; Based on the first mapping point, calculate the second mapping point corresponding to the virtual light source on the virtual camera plane; The second mapping point is reverse-mapped onto the real image plane to obtain the virtual light spot image point formed by the virtual light source on the real image plane.

2. The method for determining virtual light source information for gaze tracking according to claim 1, characterized in that, The step of mapping the real light spot image points formed by the real light source on the real image plane of the camera onto the virtual camera plane to obtain the first mapped point corresponding to the real light source on the virtual camera plane includes: Determine the second straight line passing through the optical center of the camera and the actual light spot image point; Calculate the intersection point of the second straight line and the virtual camera plane, and use it as the first mapping point.

3. The method for determining virtual light source information for gaze tracking according to claim 1, characterized in that, The step of calculating the second mapping point of the virtual light source on the virtual camera plane based on the first mapping point includes: Determine the direction vector of the first straight line as the target direction vector; On the virtual camera plane, a third straight line is determined that passes through the first mapping point and has the direction vector of the target direction vector; wherein the second mapping point is located on the third straight line; A plane passing through the optical center of the camera, the corneal curvature center of the target cornea, and the virtual light source is defined as the virtual light source plane; wherein, the second mapping point is located on the virtual light source plane; The second mapping point is calculated based on the third straight line and the virtual light source plane.

4. The method for determining virtual light source information for gaze tracking according to claim 3, characterized in that, The calculation of the second mapping point based on the third straight line and the virtual light source plane includes: Calculate the intersection point of the third straight line and the virtual light source plane, and use it as the second mapping point; or Calculate the intersection line between the virtual light source plane and the virtual camera plane, and calculate the intersection point between this intersection line and the third straight line, which is used as the second mapping point.

5. The method for determining virtual light source information for gaze tracking according to claim 1, characterized in that, The step of reverse mapping the second mapping point to the real image plane to obtain the virtual light spot image point formed by the virtual light source on the real image plane includes: Determine the fourth straight line passing through the optical center of the camera and the second mapping point; The intersection point of the fourth straight line and the real image plane is calculated and used as the virtual light spot image point.

6. The method for determining virtual light source information for gaze tracking according to claim 1, characterized in that, After mapping the second mapping point in reverse onto the real image plane to obtain the virtual light spot image point formed by the virtual light source on the real image plane, the method further includes: Based on the camera's preset lens distortion coefficient, the virtual light spot image points are subjected to distortion correction.

7. A device for determining virtual light source information for gaze tracking, characterized in that, This is applied to a gaze estimation system, which includes a real light source, multiple cameras, and a virtual light source information determination device, the device comprising: The acquisition module is used to acquire the position of a pre-set virtual light source in three-dimensional space; The determination module is used to determine a plane that passes through the principal point of the camera and is parallel to a first straight line passing through the real light source and the virtual light source, as the virtual camera plane of the camera; The first mapping module is used to map the real light spot image point formed by the real light source on the real image plane of the camera onto the virtual camera plane to obtain the first mapping point corresponding to the real light source on the virtual camera plane; wherein, the real light spot image point is the image of the light spot reflected by the real light source onto the target cornea on the real image plane; The calculation module is used to calculate the second mapping point corresponding to the virtual light source on the virtual camera plane based on the first mapping point; The second mapping module is used to reverse map the second mapping point to the real image plane to obtain the virtual light spot image point formed by the virtual light source on the real image plane.

8. The virtual light source information determination device for gaze tracking according to claim 7, characterized in that, The first mapping module is specifically used for: Determine the second straight line passing through the optical center of the camera and the actual light spot image point; Calculate the intersection point of the second straight line and the virtual camera plane, and use it as the first mapping point.

9. A computer device, the computer device comprising: A memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that the processor executes the computer program to implement the steps of the method according to any one of claims 1 to 6.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it is used to implement the steps of the method according to any one of claims 1 to 6.

Images