Event camera based eye tracking method, device, medium and program product

Abstract

The application provides an event camera-based eyeball tracking method, device, medium and program product. The method comprises the following steps: controlling a plurality of discrete light emitting units of a light source assembly to respectively emit light to eyeballs of a user according to preset light emitting timing data, and controlling an event camera to collect corresponding light spot event data of the eyeballs of the user; obtaining three-dimensional space coordinates of a light spot actually formed by the eyeballs of the user in a camera coordinate system based on the light spot event data; fitting a sclera sphere and a cornea sphere corresponding to the eyeballs of the user according to the three-dimensional space coordinates; and determining an optical axis direction in combination with the center coordinates of the sclera sphere and the cornea sphere. Through the implementation of the application scheme, the event camera is introduced into 3D line-of-sight tracking detection, which can provide hardware support and prerequisite conditions for breaking through the algorithm complexity bottleneck of the current technology, can meet the high-speed eyeball tracking demand, and can reduce the cost of the algorithm platform.

Description

Technical Field

[0001] This application relates to the field of eye-tracking technology, and in particular to an eye-tracking method, device, medium, and program product based on an event camera. Background Technology

[0002] Eye tracking is a technology that uses mechanical, electronic, and optical methods to obtain the user's current gaze direction. In recent years, various eye tracking analysis methods have been developed, including current recording, electromagnetic coil methods, reflection recording methods, double Pulcim imaging, and corneal reflection methods. With the continuous deepening and development of eye-tracking technology research, its application fields are becoming increasingly wide-ranging, and it is currently widely used in human-computer interaction, virtual reality, vehicle-assisted driving, human factors analysis, and psychological analysis, among other fields.

[0003] Camera-based 3D gaze estimation is a relatively mature eye-tracking method. It typically utilizes geometric imaging relationships such as light source reflection and pupil refraction, including the vectors of the corneal center and optical axis, and the transformation relationship between the optical axis and the visual axis, to solve for the spatial 3D information related to the gaze and estimate the 3D gaze. However, this method involves solving a system of multivariate high-order nonlinear equations, resulting in slow computation speed and high computing platform costs, thus hindering low-cost, high-speed eye tracking. Summary of the Invention

[0004] This application provides an eye-tracking method, device, medium, and program product based on an event camera, which can at least solve the problems of high computing platform cost and low eye-tracking speed of eye-tracking methods provided in related technologies.

[0005] The first aspect of this application provides an eye-tracking method based on an event camera, comprising: controlling multiple discrete light-emitting units of a light source assembly to emit light to a user's eyeball according to preset light emission timing data; and controlling an event camera to collect corresponding light spot event data of the user's eyeball; obtaining the three-dimensional spatial coordinates of the actual light spot formed by the user's eyeball in the camera coordinate system based on the light spot event data; fitting the scleral sphere and corneal sphere corresponding to the user's eyeball according to the three-dimensional spatial coordinates; and determining the optical axis direction by combining the center coordinates of the scleral sphere and the corneal sphere.

[0006] The second aspect of this application provides an eye-tracking device, including: a light source component, an event camera, a memory, and a processor; the multiple discrete light-emitting units of the light source component are used to emit light to the user's eyeballs according to preset light emission timing data; the event camera is used to collect corresponding light spot event data of the user's eyeballs; the processor is used to execute a computer program stored in the memory, and when the processor executes the computer program, it implements the steps of the eye-tracking method based on the event camera provided in the first aspect of this application.

[0007] The third aspect of this application provides a computer-readable storage medium storing a computer program thereon. When the computer program is executed by a processor, it implements the steps of the eye-tracking method based on an event camera provided in the first aspect of this application.

[0008] The fourth aspect of this application provides a computer program product that, when run on a computer, implements the steps of the eye-tracking method based on an event camera provided in the first aspect of this application.

[0009] As can be seen from the above, the eye-tracking method, device, medium, and program products based on event cameras provided in this application involve controlling multiple discrete light-emitting units of the light source component to emit light towards the user's eyeball according to preset light emission timing data, and controlling the event camera to collect corresponding light spot event data of the user's eyeball; obtaining the three-dimensional spatial coordinates of the actual light spot formed by the user's eyeball in the camera coordinate system based on the light spot event data; fitting the scleral sphere and corneal sphere corresponding to the user's eyeball based on the three-dimensional spatial coordinates; and determining the optical axis direction by combining the center coordinates of the scleral sphere and corneal sphere. By implementing this application, introducing an event camera into 3D gaze tracking detection can provide hardware support and prerequisites for overcoming the algorithmic complexity bottleneck of current technologies, meet the needs of high-speed eye tracking, and reduce the cost of computing platforms. Attached Figure Description

[0010] Figure 1 This is a schematic diagram of the structure of an eye-tracking device provided in one embodiment of this application;

[0011] Figure 2 This is a schematic diagram of the structure of an X-type light source assembly provided in one embodiment of this application;

[0012] Figure 3 A schematic diagram illustrating the illumination and imaging structure of an eye-tracking device according to an embodiment of this application;

[0013] Figure 4 This is a schematic diagram of a lighting structure provided in one embodiment of this application;

[0014] Figure 5This is a schematic diagram of the basic process of an eye-tracking method provided in an embodiment of this application;

[0015] Figure 6 A schematic diagram of the light emission timing of a binary encoding method provided in an embodiment of this application;

[0016] Figure 7 A schematic diagram illustrating an eye-tracking principle according to an embodiment of this application;

[0017] Figure 8 A detailed flowchart illustrating an eye-tracking method provided in an embodiment of this application;

[0018] Figure 9 This is a schematic diagram of the program modules of an eye-tracking device provided in an embodiment of this application. Detailed Implementation

[0019] To make the inventive objectives, features, and advantages of this application more apparent and understandable, the technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0020] In the description of the embodiments of this application, it should be understood that some terms indicating orientation or positional relationship may refer to the orientation or positional relationship shown in the relevant drawings, only for the convenience of describing the embodiments of this application and simplifying the description, and not to indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation, and therefore should not be construed as a limitation of the present invention.

[0021] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of the embodiments of this application, "multiple" means two or more, unless otherwise explicitly specified.

[0022] The following will describe in detail, with reference to the accompanying drawings, an eye-tracking method, device, medium, and program product based on an event camera according to embodiments of this application.

[0023] To address the issues of high computing platform costs and low eye-tracking speeds in existing eye-tracking methods, this application provides an embodiment of an event camera-based eye-tracking method applied to an eye-tracking device. In practical applications, this eye-tracking device can be a head-mounted display worn on the user's head. It should be noted that the eye-tracking device in this embodiment can be augmented reality (AR) glasses / helmets, mixed reality (MR) glasses / helmets, or other electronic devices used to combine digital content with the real world.

[0024] like Figure 1 The diagram shows a schematic of an eye-tracking device according to an embodiment of this application. The eye-tracking device mainly includes: a light source assembly 101, an event camera 102, a memory 103, and a processor 104. The number of processors 104 can be one or more. The memory 103 stores a computer program 105 that can run on the processor 104. The memory 103 and the processor 104 are communicatively connected. When the processor 104 executes the computer program 105, it implements the following flow of an eye-tracking method based on an event camera: First, it controls multiple discrete light-emitting units of the light source assembly 101 to emit light to the user's eyeball according to preset light emission timing data, and controls the event camera 102 to collect corresponding light spot event data from the user's eyeball; then, it obtains the three-dimensional spatial coordinates of the actual light spot formed by the user's eyeball in the camera coordinate system based on the light spot event data; next, it fits the scleral sphere and corneal sphere corresponding to the user's eyeball based on the three-dimensional spatial coordinates; finally, it determines the optical axis direction by combining the center coordinates of the scleral sphere and corneal sphere.

[0025] In this embodiment, the light source assembly 101 is preferably implemented as an X-shaped light source assembly. In other embodiments, it can also be implemented as a three-shaped or square-shaped assembly, such as... Figure 2 The diagram shown is a structural schematic of an X-type light source assembly provided in this embodiment. In this embodiment, the X-type light source assembly 101 may include a first linear light source assembly 1011 and a second linear light source assembly 1012, which are inclined and have opposite inclination directions. Both the first linear light source assembly 1011 and the second linear light source assembly 1012 are provided with multiple discrete light-emitting units. Figure 2 Each point in the diagram represents a discrete light-emitting unit, or an independent light-emitting unit. Through the X-type light source assembly 101 of this embodiment, an elongated light spot corresponding to the first linear light source assembly 1011 and the second linear light source assembly 1012 can be formed on the surface of the user's eyeball, covering the sclera and cornea respectively. The two elongated light spots form an X-type light spot.

[0026] In an optional embodiment of this invention, the light-emitting unit of the X-type light source assembly 101 can be implemented using an LED lamp or a laser lamp. The first linear light source assembly 1011 and the second linear light source assembly 1012 can be arranged using two separate light source chips, or they can be arranged on the same light source chip (i.e., as shown in the image). Figure 2 (The implementation shown) In addition, the light-emitting unit can be a circular light-emitting unit or a square light-emitting unit.

[0027] In this embodiment, the Event-based Vision Sensor (EVS) configured in the event camera 102 is a novel sensor that simulates the human retina. It responds to pixel pulses caused by brightness changes due to motion. That is, the event camera 102 asynchronously records the brightness changes on the pixels. When the brightness change exceeds a certain threshold, it outputs an event including coordinates (x, y), timestamp (t), and event polarity (p, with values ​​of +1 and -1, representing increases and decreases in brightness, respectively). Each event is represented in the form e = (t, x, y, p). Therefore, it can capture the brightness changes (i.e., light intensity changes) of the scene at a very high frame rate, record events at specific times and specific locations in the image, forming an event stream instead of a frame stream. This can solve the problems of information redundancy, large data storage, and large real-time processing volume of traditional cameras. In this embodiment, EVS has the characteristics of low average power consumption of 20mW, high sampling rate of 106Hz and dynamic range of 140dB [59,60], which can effectively reduce sensor power consumption, prevent eye tracking failure caused by drastic changes in ambient light, prevent high-speed scanning from causing image blurring and decreased eye tracking accuracy, output high-speed edge feature information, and effectively reduce the computing power required for image feature extraction.

[0028] like Figure 3 The diagram shown is a schematic representation of the illumination and imaging structure of an eye-tracking device provided in this embodiment. Figure 4 The diagram shown is a schematic diagram of a lighting structure provided in this embodiment. In an optional embodiment, a first lens (such as...) is provided on the light-emitting side of the X-type light source assembly 101. Figure 3 and 4 The event camera 102 has a second lens (such as a Tx Lens) on its light-incident side. Figure 4 In this embodiment, the X-shaped light source assembly 101 is offset and positioned on the side of the central axis of the first lens away from the event camera 102 (Rx Lens). It should be understood that in this embodiment... Figure 3 and Figure 4 The image shows "Eyes" as a diagram of the user's eyeball, and "101" is an enlarged schematic diagram of the X-shaped light source component.

[0029] In this embodiment, it is necessary to consider imaging measurement of the eyeball at an ultra-close distance. Therefore, the positions of the light source and the projection lens (i.e., the first lens) need to be offset so that the center of the X-shaped light spot basically coincides with the center of the eyeball, thereby achieving alignment between the center of the light source and the center of the eyeball.

[0030] It should be noted that the memory 103 in this embodiment can be a high-speed random access memory (RAM) or a non-volatile memory, such as a disk storage device. The memory 103 is used to store executable program code, and the processor 104 is coupled to the memory 103.

[0031] One embodiment of this application also provides a computer-readable storage medium, which may be disposed in the aforementioned eye-tracking device. The computer-readable storage medium may be as described above. Figure 1 The memory in the illustrated embodiment.

[0032] In this embodiment, a computer program is stored on the computer-readable storage medium. When executed by a processor, the computer program can implement the aforementioned eye-tracking method based on an event camera. Furthermore, the computer-readable storage medium can also be a USB flash drive, a portable hard drive, a read-only memory (ROM), RAM, a magnetic disk, or an optical disk, or any other medium capable of storing program code.

[0033] An embodiment of this application also provides a computer program product containing instructions that, when run on a computer or processor, can implement the aforementioned process of the eye-tracking method based on an event camera.

[0034] like Figure 5 This is a basic flowchart of an event camera-based eye-tracking method provided in an embodiment of this application. The eye-tracking method can be... Figure 1 The eye-tracking device performs the following steps:

[0035] Step 501: Control the multiple discrete light-emitting units of the light source component to emit light to the user's eyeball according to the preset light emission timing data, and control the event camera to collect the corresponding light spot event data of the user's eyeball.

[0036] Specifically, this embodiment exemplifies the use of the aforementioned X-type light source component as the light source component. The X-type light source component and the event camera are placed close together. The emission timing of the X-type light source component can be pre-encoded. The encoded emission timing data is used to control the discrete light-emitting units to switch on and off at different times. Different discrete light-emitting units have different brightness and darkness timings, peak power, duty cycles, and intervals between multiple lighting sequences. Encoding methods include pseudo-random and one-dimensional Gray code.

[0037] As mentioned above, the X-type light source assembly in this embodiment may include a first linear light source assembly and a second linear light source assembly, such as... Figure 6 This is a schematic diagram of the light emission timing of a binary encoding method provided in this embodiment. In the figure, L 11 L 12 ...represents discrete light-emitting units at different positions or numbered on one of the linear light source components, t pluse T represents the duration for which a single discrete light-emitting unit is lit, and is generally greater than or equal to the EVS frame rate. cycle This represents one emission control cycle of a linear light source component.

[0038] In practical applications, when a light source emits light into a user's eyeball, a light spot is formed on the user's eyeball. The event camera responds to the changes in brightness of the user's eyeball and collects the corresponding light spot event data.

[0039] Step 502: Obtain the three-dimensional spatial coordinates of the actual light spot formed by the user's eyeball in the camera coordinate system based on the light spot event data.

[0040] In an optional implementation of this embodiment, firstly, based on the light spot event data and the emission timing data, a coordinate correspondence between the light spot captured by the camera and the discrete emission unit is established; then, based on the coordinate correspondence, the three-dimensional spatial coordinates of the light spot actually formed by the user's eyeball in the camera coordinate system are obtained.

[0041] Continuing with the aforementioned example of an X-type light source assembly that can include a first linear light source assembly and a second linear light source assembly, the coordinate positions of the discrete light-emitting units on the first linear light source assembly are... and the second linear light source assembly Pre-calibrated, and based on the spot event data collected by the event camera and the emission timing data of the X-shaped light source component, the spot events corresponding to each discrete emission unit can be identified, thereby determining the coordinate position of the spot captured by the camera. (or The coordinate position of the discrete light-emitting unit (or The serial numbers are matched one-to-one.

[0042] Next, in this embodiment, the initial extrinsic and initial intrinsic parameters of the first linear light source component, the second linear light source component, and the event camera are obtained respectively. The extrinsic parameters include a rotation matrix and a translation vector. Based on the initial extrinsic parameters, the relative extrinsic parameters are calculated with the optical center of the event camera as the origin of the world coordinate system. Based on the coordinate correspondence, the relative extrinsic parameters, the initial intrinsic parameters, and the translation vector in the initial extrinsic parameters of the event camera, the three-dimensional spatial coordinates of the X-shaped light spot actually formed by the user's eyeball in the camera coordinate system are calculated.

[0043] like Figure 7 The diagram shown is a schematic representation of an eye-tracking principle provided in this embodiment, featuring a separate X-shaped light source assembly. Figure 7 Images (a) and (b) show the imaging optical paths corresponding to the first linear light source assembly Laser1 and the second linear light source assembly Laser2, respectively. Eyes represents the user's eyeball, and EVS represents the event camera. Figure 7 The image shows the front view and top view of the user's eye. In this embodiment, existing calibration methods can be used to calibrate the intrinsic and extrinsic parameters of the event camera and the X-shaped light source assembly, obtaining the extrinsic parameter rotation matrix of the event camera. Extrinsic translation vector and internal reference The external parameter rotation matrix of the first linear light source component Extrinsic translation vector and internal reference Rotation matrix of the second linear light source assembly Translation vector and internal reference As the initial extrinsic and initial intrinsic parameter matrices.

[0044] Then, in this embodiment, the initial extrinsic parameter matrix is ​​substituted into the preset relative extrinsic parameter matrix calculation formula to calculate the optical center O of the event camera. C The relative extrinsic parameter matrix is ​​the origin of the world coordinate system; the formula for calculating the relative extrinsic parameter matrix is ​​as follows.

[0045] Represented as:

[0046]

[0047]

[0048]

[0049]

[0050] in, Tx1-Rx and T Tx1-Rx Let R represent the relative rotation matrix and relative translation vector of the first linear light source component relative to the event camera, respectively. Tx2-Rxand T Tx2-Rx Let R represent the relative rotation matrix and relative translation vector of the second linear light source component relative to the event camera, respectively. Rx and T Rx Let R represent the initial rotation matrix and initial translation vector of the event camera, respectively. Tx1 and T Tx1 Let R represent the initial rotation matrix and initial translation vector of the first linear light source component, respectively. Tx2 and T Tx2 Let T represent the initial rotation matrix and initial translation vector of the second linear light source component, respectively, and T denote the transpose operation.

[0051] Finally, the coordinate correspondence, relative extrinsic matrix, initial intrinsic matrix, and translation vector from the event camera's initial extrinsic matrix are substituted into the preset coordinate calculation formula to calculate the three-dimensional spatial coordinates of the X-shaped light spot actually formed by the user's eyeball in the camera coordinate system; the coordinate calculation formula is expressed as:

[0052] C 1i =A Rx [R Tx1-Rx *P 1i +T Rx ]

[0053] L 1i =A Tx1 [R Tx1-Rx *P 1i +T Tx1-Rx ]

[0054] C 2i =A Rx [R Tx2-Rx *P 2i +T Ex ]

[0055] L 2i =A Tx2 [R Tx2-Rx *P 2i +T Tx2-Rx ];

[0056] Among them, L 1i L represents the coordinates of the i-th discrete light-emitting unit on the first linear light source assembly. 2i C represents the coordinates of the i-th discrete light-emitting unit on the second linear light source assembly. 1i C represents the coordinates of the camera-captured light spot corresponding to the i-th discrete light-emitting unit on the first linear light source assembly. 2i A represents the coordinates of the camera-captured light spot corresponding to the i-th discrete light-emitting unit on the second linear light source assembly. Rx Let A represent the initial intrinsic parameter matrix of the event camera. Tx1Let A represent the initial intrinsic parameter matrix of the first linear light source component. Tx2 P represents the initial intrinsic parameter matrix of the second linear light source component. 1i P represents the three-dimensional spatial coordinates of the light spot actually formed on the user's eyeball, corresponding to the i-th discrete light-emitting unit on the first linear light source component. 2i This represents the three-dimensional spatial coordinates of the light spot actually formed on the user's eyeball, corresponding to the i-th discrete light-emitting unit on the second linear light source component.

[0057] This embodiment adjusts C based on the coordinates of the transmitting point and the receiving point. 2i -L 2i Or C 1i -L 1i After solving the above system of equations with pre-calibrated intrinsic and extrinsic parameters, the three-dimensional spatial coordinates of the speckle covering the user's eyeball surface corresponding to the emitted light rays from the two linear light source components in the camera coordinate system are obtained. and

[0058] Step 503: Fit the scleral sphere and corneal sphere corresponding to the user's eyeball based on the three-dimensional spatial coordinates.

[0059] In this embodiment, a preset spherical fitting equation can be used to fit the three-dimensional spatial coordinates corresponding to the first linear light source component and the second linear light source component, respectively, to obtain the scleral sphere and corneal sphere corresponding to the user's eyeball; the spherical fitting equation is expressed as:

[0060]

[0061]

[0062]

[0063]

[0064] in, O sclera =(a sclera b sclera c sclera ), O sclera Let r represent the coordinates of the center of the sclera sphere. sclera O represents the radius of the sclera sphere. sornea =(a cornea b cornea c cornea ), O cornea The coordinates of the center of the corneal sphere, r cornea This represents the radius of the corneal sphere.

[0065] Step 504: Determine the direction of the optical axis by combining the coordinates of the center of the sclera and the cornea.

[0066] In this embodiment, the optical axis direction ultimately determined by the eye-tracking device can be expressed as: (a sclera ,b sclera ,c sclera ) T →(a cornea ,b cornea ,c cornea ) T That is, the direction of the line connecting the center coordinates of the sclera to the center coordinates of the cornea is the optical axis direction.

[0067] Based on the above technical solutions in the embodiments of this application, EVS is introduced into 3D gaze tracking measurement, and X-shaped light source illumination covering the iris and cornea of ​​the eyeball surface is proposed in combination with eyeball characteristics. The illumination light is discretely time-series encoded to adapt to the high frame rate of EVS and the time sequence distinction between multiple light sources. Furthermore, it is combined with a linear gaze tracking solution algorithm, which has fast calculation speed and low computing power consumption, and can effectively overcome the bottlenecks in computing speed and computing power platform cost faced by current eye tracking technology.

[0068] Figure 8 The method described herein is a refined eye-tracking method based on an event camera, applied to an eye-tracking device including an X-shaped light source assembly and an event camera. The X-shaped light source assembly includes a first linear light source assembly and a second linear light source assembly, both tilted and in opposite directions. The specific implementation process of this eye-tracking method includes the following steps:

[0069] Step 801: Control the multiple discrete light-emitting units of the X-type light source assembly to emit light to the user's eyeball according to the preset light emission timing data, and control the event camera to collect the corresponding light spot event data of the user's eyeball;

[0070] Step 802: Based on the light spot event data and the emission time sequence data, establish the coordinate correspondence between the light spot captured by the camera and the discrete emission unit;

[0071] Step 803: Obtain the initial extrinsic and initial intrinsic parameter matrices of the first linear light source component, the second linear light source component, and the event camera, respectively;

[0072] Step 804: Calculate the relative extrinsic matrix with the optical center of the event camera as the origin of the world coordinate system based on the initial extrinsic matrix;

[0073] Step 805: Based on the coordinate correspondence, relative extrinsic matrix, initial intrinsic matrix, and translation vector in the initial extrinsic matrix of the event camera, calculate the three-dimensional spatial coordinates of the X-shaped light spot actually formed by the user's eyeball in the camera coordinate system.

[0074] Step 806: Use a preset spherical fitting equation to fit the three-dimensional spatial coordinates corresponding to the first linear light source component and the second linear light source component respectively, to obtain the scleral sphere and corneal sphere corresponding to the user's eyeball;

[0075] Step 807: Determine the direction of the optical axis by combining the coordinates of the center of the sclera and the cornea.

[0076] It should be understood that the sequence number of each step in this embodiment does not imply the order in which the steps are executed. The execution order of each step should be determined by its function and internal logic, and should not constitute a unique limitation on the implementation process of this application embodiment.

[0077] Figure 9 An eye-tracking device based on an event camera is provided as an embodiment of this application. This eye-tracking device can be used to implement the eye-tracking method based on an event camera in the foregoing embodiments. The eye-tracking device mainly includes:

[0078] The control module 901 is used to control multiple discrete light-emitting units of the light source component to emit light to the user's eyeball according to preset light emission timing data, and to control the event camera to collect corresponding light spot event data of the user's eyeball;

[0079] The acquisition module 902 is used to acquire the three-dimensional spatial coordinates of the actual light spot formed by the user's eyeball in the camera coordinate system based on the light spot event data;

[0080] The fitting module 903 is used to fit the scleral sphere and corneal sphere corresponding to the user's eyeball based on three-dimensional spatial coordinates.

[0081] The determination module 904 is used to determine the direction of the optical axis by combining the center coordinates of the sclera and cornea.

[0082] In some embodiments of this example, the acquisition module is specifically used to: establish a coordinate correspondence between the light spot captured by the camera and the discrete light-emitting unit based on the light spot event data and the light emission timing data; and obtain the three-dimensional spatial coordinates of the light spot actually formed by the user's eyeball in the camera coordinate system based on the coordinate correspondence.

[0083] In some embodiments of this example, the light source component is an X-shaped light source component; correspondingly, when the acquisition module performs the function of obtaining the three-dimensional spatial coordinates of the light spot actually formed by the user's eyeball in the camera coordinate system based on the coordinate correspondence relationship, it is specifically used to: obtain the three-dimensional spatial coordinates of the X-shaped light spot actually formed by the user's eyeball in the camera coordinate system based on the coordinate correspondence relationship.

[0084] In some embodiments of this example, the X-type light source assembly includes a first linear light source assembly and a second linear light source assembly that are tilted and have opposite tilting directions. Both the first linear light source assembly and the second linear light source assembly are provided with multiple discrete light-emitting units.

[0085] Accordingly, when the acquisition module performs the function of obtaining the three-dimensional spatial coordinates of the X-shaped light spot actually formed by the user's eyeball in the camera coordinate system based on the coordinate correspondence, it is specifically used to: obtain the initial extrinsic parameter matrix and the initial intrinsic parameter matrix of the first linear light source component, the second linear light source component, and the event camera respectively; wherein, the extrinsic parameter matrix includes a rotation matrix and a translation vector; calculate the relative extrinsic parameter matrix with the optical center of the event camera as the origin of the world coordinate system based on the initial extrinsic parameter matrix; and calculate the three-dimensional spatial coordinates of the X-shaped light spot actually formed by the user's eyeball in the camera coordinate system based on the coordinate correspondence, the relative extrinsic parameter matrix, the initial intrinsic parameter matrix, and the translation vector in the initial extrinsic parameter matrix of the event camera.

[0086] In some embodiments of this example, the fitting module is specifically used to: fit the three-dimensional spatial coordinates corresponding to the first linear light source component and the second linear light source component respectively using a preset spherical fitting equation to obtain the scleral sphere and corneal sphere corresponding to the user's eyeball.

[0087] It should be noted that the eye-tracking methods in the foregoing embodiments can all be implemented based on the eye-tracking device provided in this embodiment. Those skilled in the art can clearly understand that, for the sake of convenience and brevity, the specific working process of the eye-tracking device described in this embodiment can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0088] Based on the technical solution of the above embodiments of this application, multiple discrete light-emitting units of the control light source component emit light to the user's eyeball according to preset light emission timing data, and control the event camera to collect corresponding light spot event data of the user's eyeball; obtain the three-dimensional spatial coordinates of the actual light spot formed by the user's eyeball in the camera coordinate system based on the light spot event data; fit the scleral sphere and corneal sphere corresponding to the user's eyeball according to the three-dimensional spatial coordinates; and determine the optical axis direction by combining the center coordinates of the scleral sphere and corneal sphere. By implementing the solution of this application, the introduction of the event camera into 3D gaze tracking detection can provide hardware support and prerequisites for overcoming the algorithmic complexity bottleneck of the current technology, meet the needs of high-speed eye tracking, and reduce the cost of computing platforms.

[0089] It should be noted that the apparatuses and methods disclosed in the several embodiments provided in this application can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative. For instance, the division of modules is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple modules or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or modules may be electrical, mechanical, or other forms.

[0090] The modules described as separate components may or may not be physically separate. Similarly, the components shown as modules may or may not be physical modules; they may be located in one place or distributed across multiple network modules. Some or all of the modules can be selected to achieve the purpose of this embodiment, depending on actual needs.

[0091] Furthermore, the functional modules in the various embodiments of this application can be integrated into one processing module, or each module can exist physically separately, or two or more modules can be integrated into one module. The integrated modules described above can be implemented in hardware or as software functional modules.

[0092] If the integrated module is implemented as a software functional module and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a readable storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned readable storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, ROM, RAM, magnetic disks, or optical disks.

[0093] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that this application is not limited to the described order of actions, as some steps may be performed in other orders or simultaneously according to this application. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions and modules involved are not necessarily essential to this application.

[0094] In the above embodiments, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0095] The above is a description of the eye-tracking method, device, medium, and program product based on an event camera provided in this application. For those skilled in the art, based on the ideas of the embodiments of this application, there will be changes in the specific implementation methods and application scope. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. An eye-tracking method based on an event camera, characterized in that, Applied to event cameras, including: The system controls multiple discrete light-emitting units of the light source assembly to emit light to the user's eyeball according to preset light emission timing data, and controls the event camera to collect corresponding light spot event data of the user's eyeball; Based on the light spot event data and the emission time sequence data, a coordinate correspondence between the light spot captured by the camera and the discrete emission unit is established; Based on the coordinate correspondence, obtain the three-dimensional spatial coordinates of the light spot actually formed by the user's eyeball in the camera coordinate system; The three-dimensional spatial coordinates corresponding to the light source component are fitted using a preset spherical fitting equation to obtain the scleral sphere and corneal sphere corresponding to the user's eyeball; The optical axis direction is determined by combining the coordinates of the center of the sclera and the cornea.

2. The eye-tracking method according to claim 1, characterized in that, The light source component is an X-type light source component; the step of obtaining the three-dimensional spatial coordinates of the light spot actually formed by the user's eyeball in the camera coordinate system based on the coordinate correspondence includes: Based on the coordinate correspondence, obtain the three-dimensional spatial coordinates of the X-shaped light spot actually formed by the user's eyeball in the camera coordinate system.

3. The eye-tracking method according to claim 2, characterized in that, The X-shaped light source assembly includes a first linear light source assembly and a second linear light source assembly that are tilted and have opposite tilting directions. Both the first linear light source assembly and the second linear light source assembly are provided with a plurality of discrete light-emitting units. The step of obtaining the three-dimensional spatial coordinates of the X-shaped light spot actually formed by the user's eyeball in the camera coordinate system based on the coordinate correspondence includes: The initial extrinsic and intrinsic parameter matrices of the first linear light source component, the second linear light source component, and the event camera are obtained respectively; wherein, the extrinsic parameter matrix includes a rotation matrix and a translation vector; Calculate the relative extrinsic matrix with the optical center of the event camera as the origin of the world coordinate system based on the initial extrinsic matrix; Based on the coordinate correspondence, the relative extrinsic matrix, the initial intrinsic matrix, and the translation vector in the initial extrinsic matrix of the event camera, the three-dimensional spatial coordinates of the X-shaped light spot actually formed by the user's eyeball in the camera coordinate system are calculated.

4. The eye-tracking method according to claim 3, characterized in that, The step of calculating the relative extrinsic matrix based on the initial extrinsic matrix with the optical center of the event camera as the origin of the world coordinate system includes: Substitute the initial extrinsic matrix into the preset relative extrinsic matrix calculation formula to calculate the relative extrinsic matrix with the optical center of the event camera as the origin of the world coordinate system; the relative extrinsic matrix calculation formula is expressed as follows: ； in, and Let represent the relative rotation matrix and relative translation vector of the first linear light source component relative to the event camera, respectively. and These represent the relative rotation matrix and relative translation vector of the second linear light source component relative to the event camera, respectively. and Let these represent the initial rotation matrix and initial translation vector of the event camera, respectively. and Let represent the initial rotation matrix and initial translation vector of the first linear light source component, respectively. and These represent the initial rotation matrix and initial translation vector of the second linear light source component, respectively. This indicates the transpose operation.

5. The eye-tracking method according to claim 4, characterized in that, The step of calculating the three-dimensional spatial coordinates of the X-shaped light spot actually formed by the user's eyeball in the camera coordinate system based on the coordinate correspondence, the relative extrinsic matrix, the initial intrinsic matrix, and the translation vector in the initial extrinsic matrix of the event camera includes: Substituting the coordinate correspondence, the relative extrinsic matrix, the initial intrinsic matrix, and the translation vector in the initial extrinsic matrix of the event camera into the preset coordinate calculation formula, the three-dimensional spatial coordinates of the X-shaped light spot actually formed by the user's eyeball in the camera coordinate system are calculated; the coordinate calculation formula is expressed as: ； in, Indicates the first linear light source assembly on the first The coordinates of the discrete light-emitting units, Indicates the second linear light source assembly. The coordinates of the discrete light-emitting units, This indicates the first linear light source assembly. The camera captures the coordinates of the light spots for each of the discrete light-emitting units. This indicates the first corresponding linear light source assembly. The camera captures the coordinates of the light spots for each of the discrete light-emitting units. This represents the initial intrinsic parameter matrix of the event camera. This represents the initial intrinsic parameter matrix of the first linear light source component. This represents the initial intrinsic parameter matrix of the second linear light source component. This indicates that the actual image formed on the user's eyeball is different from the image on the first linear light source component. The three-dimensional spatial coordinates of the light spot corresponding to each of the discrete light-emitting units. This indicates that the actual image formed on the user's eyeball is different from the image on the second linear light source component. The three-dimensional spatial coordinates of the light spot corresponding to each discrete light-emitting unit.

6. The eye-tracking method according to claim 5, characterized in that, The step of fitting the scleral sphere and corneal sphere corresponding to the user's eyeball based on the three-dimensional spatial coordinates includes: The three-dimensional spatial coordinates corresponding to the first linear light source component and the second linear light source component are respectively fitted using a preset spherical fitting equation to obtain the scleral sphere and corneal sphere corresponding to the user's eyeball; the spherical fitting equation is expressed as: ； in, , , , Indicates the coordinates of the center of the sclera sphere. This represents the radius of the scleral sphere. , Indicates the coordinates of the center of the corneal sphere. This represents the radius of the corneal sphere.

7. An eye-tracking device, characterized in that, Includes light source components, event camera, memory, and processor; The multiple discrete light-emitting units of the light source assembly are used to emit light to the user's eyeballs according to preset light emission timing data; The event camera is used to collect light spot event data corresponding to the user's eyeball; The processor is used to execute computer programs stored in the memory; When the processor executes the computer program, it implements the steps of the eye-tracking method based on an event camera as described in any one of claims 1 to 6.

8. The eye-tracking device according to claim 7, characterized in that, The light source assembly is an X-type light source assembly.

9. The eye-tracking device according to claim 8, characterized in that, The X-shaped light source assembly includes a first linear light source assembly and a second linear light source assembly that are tilted and have opposite tilting directions. Both the first linear light source assembly and the second linear light source assembly are provided with a plurality of discrete light-emitting units.

10. The eye-tracking device according to claim 8, characterized in that, The X-shaped light source assembly has a first lens on its light-emitting side, and the event camera has a second lens on its light-receiving side.

11. The eye-tracking device according to claim 10, characterized in that, The X-shaped light source assembly is offset on the side of the central axis of the first lens away from the event camera.

12. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the steps of the eye-tracking method based on an event camera as described in any one of claims 1 to 6.

13. A computer program product, characterized in that, When the computer program product is run on a computer, it implements the steps of the eye-tracking method based on an event camera as described in any one of claims 1 to 6.

Images