Eye tracking method, system, readable storage medium and electronic device

Abstract

An eye tracking method, comprising: loading personalized anterior segment optical surface parameters of a user; projecting a light beam to the user's eye; detecting the light beam reflected by the anterior surface of the cornea to generate a sensor signal; determining the current pose of the eye based on the parameters and the sensor signal through geometric solving or deep learning. The application uses the pre-stored personalized anterior segment optical surface parameters of the user as a geometric reference, performs a visual axis calibration of the relative eye pose once at the first use and records it, and does not need to perform gaze point calibration in subsequent use. When the device is worn and slides, the geometric reference remains unchanged, and the tracking accuracy can still be maintained. The scheme can be integrated into a wearable electronic device to complete real-time eye tracking with low power consumption and high robustness.

Description

Technical Field

[0001] This application relates to the field of eye-tracking technology, and in particular to an eye-tracking system, method, storage medium, and wearable device based on an anterior segment optical surface geometry model. Background Technology

[0002] Eye tracking is a key technology in fields such as human-computer interaction, virtual reality, and medical diagnosis. In the field of eye tracking technology, the pupil-corneal reflection (PCCR) method is one of the most widely used approaches. This method estimates the gaze direction by detecting the relative position of the pupil center and the corneal reflection spot, offering advantages such as simplicity and fast response. However, this method typically requires the user to fixate on multiple calibration points before use to establish a mapping relationship between image features and fixation points. This calibration process is cumbersome; furthermore, this method relies on a stable device-eye geometry, and once the device wearing status changes, the mapping relationship fails, requiring recalibration, which severely impacts the user experience. While this method indirectly compensates for individual differences in corneal curvature and Kappa angle during calibration, this is not its primary purpose, and this compensation effect fails as the device wearing status changes. Pose estimation methods based on pupil or iris contours are another common approach. These methods detect the projected ellipse of the pupil or iris contour in the image and assume it to be circular in space, using the perspective projection geometry of a circle to infer eye pose. However, this method is quite sensitive to errors in the projection ellipse. Physiological parameters such as elastic deformation and refractive distortion of the pupil outline, gradient blurring of the iris outline, and approximate pupil diameter can all produce large errors in the projection ellipse.

[0003] To simplify system architecture and reduce power consumption, existing solutions attempt to replace traditional image sensors with different types of optical sensors, such as position-sensitive detectors (PSDs) or angle sensors (e.g., photodiode arrays). These solutions determine eye movement parameters by detecting the direction or position of corneal reflected light and mapping it to an ocular geometry model. While these solutions achieve sensor miniaturization and low power consumption at the hardware level, their system architecture remains essentially a "real-time measurement-experience mapping" model, still relying on user calibration to compensate for individual differences, and requiring recalibration after device movement. Other research attempts to estimate gaze using phase deflection techniques through high-density ocular surface 3D reconstruction. This involves illuminating the eye with an active light source, analyzing the deformation of the reflection patterns on the cornea and sclera, reconstructing the 3D morphology of the ocular surface through integration, and then calculating the approximate positions of the corneal and scleral curvature centers, using the line connecting them as the optical axis. The resulting surface is a relative surface shape rather than an absolute height surface, and its absolute depth involves an unknown integral constant term translation. The relative surface itself does not possess an independent absolute height reference, making it difficult to directly apply to low-power eye-tracking schemes that rely solely on sparse reflection point detection. The latter requires absolute height information that can serve as a fixed geometric reference to constrain attitude calculation. Therefore, current technology still lacks a solution that can achieve both high-precision, low-power eye tracking while fundamentally resisting device slippage and eliminating the need for repeated calibration. Summary of the Invention

[0004] This application addresses the shortcomings of the prior art by providing an eye-tracking system, method, storage medium, and electronic device based on an anterior segment optical surface geometric model. By loading pre-stored user-personalized anterior segment optical surface parameters (including absolute height information), and employing at least one of multiple sensing schemes (including position-sensitive detectors, event cameras, image sensors, or sparse brightness sensor arrays) during the real-time tracking phase, it combines the pre-stored anterior segment optical surface parameters with geometric calculations or deep learning inference to achieve high-precision, low-power, and anti-slip eye tracking. The main steps of the eye tracking in this application are as follows: Figure 1 As shown, the process includes loading the anterior segment optical surface parameters S11, obtaining the eyeball pose S12, obtaining the eyeball frame rotation center S13, obtaining the eye optical axis S14, obtaining the visual axis calibration parameters S15, and calculating the eye visual axis S16.

[0005] To achieve the above objectives, this application provides the following technical solution:

[0006] An eye-tracking system based on an anterior segment optical surface geometry model includes:

[0007] Storage unit, configured to store the user's personalized eye-spot optical surface parameters;

[0008] The light source unit is configured to project a light beam toward the user's eyes;

[0009] An optical sensing module is configured to detect a light beam reflected by the eye and generate a sensor signal containing information related to the light reflected from the anterior surface of the cornea;

[0010] The processing unit is connected to the storage unit and the optical sensing module respectively, and is configured to determine the current posture of the eyeball through geometric calculation based on the optical surface parameters of the anterior segment and the sensor signal.

[0011] Optionally, the light source unit includes multiple point light sources, which are configured to be lit sequentially in a time-division multiplexing manner, and the processing unit distinguishes the reflected signals from different point light sources according to the timing code.

[0012] Optionally, the light source unit is configured to generate at least two directional beams with known directions, the propagation direction of each directional beam is pre-calibrated and stored in the system, and the processing unit distinguishes the reflected signals from different directional beams according to temporal coding or spatial coding.

[0013] Optionally, the optical sensing module is at least one of a position-sensitive detector, an event camera, or an image sensor.

[0014] Optionally, the processing unit is configured to: determine the direction vector of the reflected ray based on the sensor signal; intersect the reflected ray with the surface model described by the anterior segment optical surface parameters to determine the reflection point on the corneal surface; establish geometric constraints between the incident ray, the reflected ray, and the surface normal according to the law of reflection; and solve for the eyeball posture parameters that satisfy the geometric constraints through an optimization algorithm, wherein the eyeball posture includes the eyeball rotation center and rotation angle.

[0015] Optionally, the processing unit is further configured to: predict the position of the theoretical reflected light spot of each beam on the sensor imaging plane based on the currently estimated eye pose during the tracking process; and when the deviation between the actual detected position and the theoretical predicted position exceeds a preset threshold, determine the light spot as abnormal and remove it.

[0016] In another embodiment, the optical sensing module includes multiple brightness sensors. The multiple emitters are sequentially and individually illuminated in a time-division multiplexing manner. During the illumination of each emitter, all brightness sensors simultaneously collect the reflected light intensity to form an intensity matrix. The processing unit inputs the personalized anterior segment optical surface parameters and the intensity matrix into a pre-trained deep learning model, and determines the current pose of the eyeball based on the model's output.

[0017] Optionally, the system further includes a first polarizer disposed in front of each emitter and a second polarizer disposed in front of each brightness sensor, wherein the transmission axis of the first polarizer and the second polarizer are parallel to each other to suppress diffuse reflection interference.

[0018] Optionally, the light source unit is driven by pulse width modulation (PWM), and the processing unit dynamically adjusts the PWM duty cycle according to the signal strength fed back by the brightness sensor, so that the reflected light intensity is kept within a preset linear working range to avoid sensor saturation.

[0019] Optionally, the processing unit inserts a time slice in each measurement cycle during which all emitters are turned off, acquires ambient light intensity and dark current, and performs background subtraction on the intensity signal.

[0020] Optionally, the number of the plurality of light emitters The number of the plurality of brightness sensors satisfy , ,and and The product is not less than 36.

[0021] Optionally, the deep learning model is at least one of a fully connected neural network, a graph neural network, or a Transformer network structure, wherein the Transformer network is a sequence modeling network based on a self-attention mechanism. The deep learning model is obtained through offline training, with training data generated using Latin hypercube sampling to cover a reasonable range of different corneal parameters and simulate Fresnel reflection characteristics, sensor noise characteristics, and ambient light interference. When used online, the model directly outputs eye pose based on the input personalized anterior segment optical surface parameters and the real-time acquired intensity matrix.

[0022] Optionally, the processing unit is further configured to: guide the user to gaze at at least two preset gaze points with known spatial coordinates; record the optical axis during gazing; calculate personalized visual axis calibration parameters for the user based on the deviation between the optical axis and the preset gaze points; and determine the visual axis based on the optical axis and the visual axis calibration parameters during real-time tracking.

[0023] Optionally, the anterior segment optical surface parameters include parameters of the anterior corneal surface and / or the anterior scleral surface, the parameters including at least one of curvature distribution, aspheric coefficient, Zernike coefficient, or B-spline control points.

[0024] This application also provides an eye-tracking method based on an anterior segment optical surface geometry model, comprising: loading user-personalized anterior segment optical surface parameters pre-stored in a storage unit; projecting a light beam onto the user's eye; detecting the light beam reflected by the eye and generating a sensor signal, the sensor signal containing information related to the light reflected from the anterior surface of the cornea; and determining the current posture of the eyeball through geometric calculation based on the anterior segment optical surface parameters and the sensor signal.

[0025] Optionally, the step of projecting the beam includes: sequentially illuminating multiple point light sources in a time-division multiplexing manner, and distinguishing the reflected signals from different point light sources according to the timing code.

[0026] Optionally, the step of projecting the beam includes: generating at least two directional beams with different propagation directions, each propagation direction being pre-calibrated, and the geometric solution simultaneously solving for the rotation and translation of the eyeball.

[0027] Optionally, this application uses pre-stored personalized anterior segment optical surface parameters, combined with corneal reflected light intensity distribution information detected by a brightness sensor array, to directly map intensity patterns to eye pose using a pre-trained deep learning model, thereby achieving low-power, anti-slip eye tracking.

[0028] This application also discloses a readable storage medium having a computer program stored thereon that, when executed by a processor, implements the method described in any of the above-mentioned embodiments.

[0029] This application also discloses an electronic device including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the method described in any of the above-mentioned embodiments.

[0030] Compared with the prior art, this application has the following beneficial effects:

[0031] This application decouples "high-precision modeling" from "real-time tracking" by loading pre-stored user-personalized anterior segment optical surface geometry parameters. During the tracking phase, real-time 3D reconstruction or intensive image processing is unnecessary; only sparse reflected light signals need to be processed to determine eye pose, significantly reducing system power consumption and computational complexity. Simultaneously, it automatically compensates for device slippage, eliminating the need for frequent recalibration and fundamentally solving the inherent problems of traditional PCCR methods being sensitive to individual differences and susceptible to interference. This application does not limit the method of acquiring offline parameters and can employ any one or more combinations of existing technologies such as OCT, Placido disk, Scheimpflug imaging, and phase measurement deflection, exhibiting good compatibility and scalability.

[0032] This application supports multiple optical sensing schemes. When using position-sensitive detectors, event cameras, or image sensors, time-division multiplexing of multiple light sources and geometric calculations, combined with pre-stored personalized anterior segment optical surface geometry models, simplifies the signal processing process, eliminates the need for repeated gaze point calibration, and resists device slippage. When using a brightness sensor array, polarization differential illumination and deep learning inference similarly eliminate the need for repeated gaze point calibration, resist device slippage, and lower hardware costs. All schemes share the same personalized anterior segment optical surface geometry model architecture, allowing for flexible selection or combination based on application scenarios. Attached Figure Description

[0033] Figure 1 This is a flowchart of the eye-tracking method of this application;

[0034] Figure 2 This is a block diagram of the eye-tracking system in the first embodiment of this application;

[0035] Figure 3 This is a schematic diagram of the optical path and timing in the second embodiment of this application;

[0036] Figure 4 This is a flowchart of the eye-tracking method in the second embodiment of this application;

[0037] Figure 5 This is a schematic diagram of spatial relationships in the second embodiment of this application;

[0038] Figure 6 This is a perspective projection diagram of the corneal reflective point in the second embodiment of this application;

[0039] Figure 7 This is a flowchart illustrating the process of obtaining eye pose and visual axis in the third embodiment of this application;

[0040] Figure 8 This is a schematic diagram of the line-of-sight calibration in the third embodiment of this application;

[0041] Figure 9 This is a block diagram of the eye-tracking system in the fourth embodiment of this application;

[0042] Figure 10 This is a schematic diagram of the optical path and timing in the fifth embodiment of this application;

[0043] Figure 11 This is a flowchart of the eye-tracking method in the fifth embodiment of this application;

[0044] Figure 12 This is a schematic diagram of the structure of the electronic device in the first and fourth embodiments of this application. Detailed Implementation

[0045] The embodiments of this application are described in detail below. Examples of these embodiments are shown in the accompanying drawings, wherein the same or similar symbols denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain this application, and should not be construed as limiting this application.

[0046] These and other aspects of the embodiments of this application will become clear from the following description and accompanying drawings. In these descriptions and drawings, some specific implementations of the embodiments of this application are specifically disclosed to illustrate some ways of carrying out prototypes of the embodiments of this application; however, it should be understood that the scope of the embodiments of this application is not limited thereto. Rather, the embodiments of this application include all variations, modifications, and equivalents falling within the spirit and scope of the appended claims.

[0047] Example 1: Time-division multiplexing-based PSD multi-spot eye-tracking system

[0048] (I) System Overview

[0049] This embodiment provides a time-division multiplexing-based PSD eye-tracking system, which is particularly suitable for scenarios that require distinguishing the reflected light spots of multiple point light sources, such as sparse dot matrix eye tracking based on a pre-stored corneal anterior surface model.

[0050] The core of this embodiment lies in sequentially illuminating multiple point light sources, allowing a single PSD to receive corneal reflections from different light sources at different times, thus solving the problem that the PSD cannot simultaneously distinguish multiple light spots. Combined with a pre-stored corneal model and a short-term eye pose smoothing assumption, high-precision, low-power, and anti-slip eye tracking is achieved.

[0051] (II) Hardware Components

[0052] like Figure 2 As shown, the system in this embodiment includes:

[0053] A point light source array 210, consisting of One (e.g.) The infrared LEDs are arranged in a ring around the camera.

[0054] An optical sensing module 220 includes a single two-dimensional PSD (position-sensitive detector) configured to receive the corresponding corneal reflected light spot during each light source illumination period and output an analog current signal related to the centroid position of the light spot.

[0055] A storage unit 230 is configured to store a pre-acquired personalized corneal anterior surface parameter model of the user;

[0056] A processing unit 240 includes: a light source driving unit 241, a signal processing unit 242, and a geometry calculation unit 243;

[0057] A light source driving unit 241 is configured to control the lighting sequence of each light emitter in the light source unit 210. The light emitters are lit sequentially. There are point light sources, and the lighting time of each light source is... A protective interval is provided between adjacent light sources;

[0058] A signal processing unit 242 includes analog front-end circuitry, an ADC, and a demultiplexing module, configured to distribute the signal output from the PSD to the corresponding light source channel by time slices;

[0059] An attitude calculation unit 243 is configured to determine the current attitude of the eyeball through geometric calculation based on the results output by the signal processing unit 242 and the personalized anterior segment optical surface geometric parameters in the storage unit 230.

[0060] (III) Time-division multiplexing light source control strategy

[0061] In order for a single PSD to be able to distinguish reflected light spots from different light sources, this embodiment uses time-division multiplexing to control the point light source array.

[0062] Figure 3 This diagram illustrates the optical path and timing of the time-division multiplexing control signal for the eye-tracking system provided in this embodiment. The user's eye includes the eyeball 101, cornea 102, and iris 103. The dashed box represents the head-mounted frame 200. The light source unit 210, optical sensing module 220, storage unit 230, and processing unit 240 are integrated on the head-mounted frame 200. The timing waveform of the time-division multiplexing control signal is shown at the bottom of the diagram.

[0063] During each measurement cycle, the light source driving unit 241 is lit sequentially. There are 10 point light sources, each with a lighting duration of 10 seconds. Up to 100 A protective interval (e.g., 5) is provided between adjacent light sources. This is to avoid signal crosstalk.

[0064] Let the first The sampled value of the light spot position of each light source is , Because the response speed of a PSD can reach the microsecond level, this sampling rate is much higher than the frequency of eye movement (the frequency of eye movement is usually below 100 Hz), therefore, the data collected in each measurement cycle... Each spot position can be considered to correspond to the same eyeball posture.

[0065] (iv) Abnormal spot detection and removal

[0066] Due to obstruction or irregular reflection from the eyelids, eyelashes, and corneal-scleral transition areas, some reflected light spots from the light source may not be effectively received by the PSD or may produce abnormal positional output. Therefore, this embodiment includes an anomaly detection mechanism:

[0067] (1) Prediction based on corneal model and current posture

[0068] Let the current time be The estimated eye pose is (Including rotation angle and translation vector). Based on the pre-stored corneal anterior surface parameter model, the theoretical position of the reflected light spot corresponding to each point light source on the PSD photosensitive surface can be predicted. .

[0069] (2) Criteria for determining anomalies

[0070] For the actual sampled spot position If any of the following conditions are met, it is determined to be an abnormal light spot and is removed in the attitude calculation:

[0071] ,in The preset positional deviation threshold (e.g., corresponding to a 0.5° line-of-sight deviation);

[0072] The light spot signal intensity is lower than the preset threshold (indicating that it is blocked by the eyelid or eyelashes).

[0073] The temporal variation in the position of the light spot exceeds the maximum angular velocity allowed by physics.

[0074] (3) Handling after an anomaly

[0075] After removing abnormal light spots, if the number of effective light spots is greater than or equal to the minimum number required for solving the problem, the attitude is calculated. If the number of effective light spots is insufficient, the displacement component in the attitude at the previous moment is used, and only the rotation component in the eyeball attitude is calculated. Alternatively, the attitude at the previous moment is used as the current result, or a Kalman filter is used for prediction.

[0076] (v) Anti-slip properties

[0077] In this embodiment, the point light source array, PSD, and user head are fixedly connected (e.g., integrated into an eyeglass frame or VR headset). When the device experiences a slight slip, the positions of the reflected light spots from all the light sources on the PSD will shift as a whole.

[0078] Since this embodiment optimizes the eyeball pose calculation by pre-stored corneal model and measured reflective point direction, the overall offset is automatically incorporated into the rotation matrix. Translation vector Therefore, this embodiment has a natural ability to compensate for device slippage, eliminating the need for additional slip sensors.

[0079] (vi) Beneficial effects

[0080] This embodiment uses time-division multiplexing to control multiple point light sources, enabling a single PSD to systematically detect corneal reflection spots from each light source, thus solving the inherent problem that PSDs cannot simultaneously distinguish multiple spots. By combining a pre-stored corneal model, a short-time smoothing assumption, and an abnormal spot detection mechanism, the following beneficial effects are achieved:

[0081] Low power consumption: PSD is an analog device, which does not require high frame rate image acquisition and transmission, and its overall power consumption is much lower than that of image sensor-based solutions;

[0082] High precision: PSD's submicron-level positional resolution, combined with a pre-stored corneal model, enables eye-tracking accuracy better than 0.1°;

[0083] Anti-slip: The systematic deviation caused by slip does not affect the solution of eye pose, and therefore does not affect the estimation of line of sight;

[0084] High robustness: Automatically removes abnormal light spots caused by occlusions such as eyelids and eyelashes through model prediction and anomaly detection. Simple system: A single PSD can distinguish multiple point light sources, avoiding the complexity of multi-sensor calibration and synchronization.

[0085] With the above configuration, this embodiment, when performing eye tracking, does not require high-power real-time 3D reconstruction or full-frame image processing. It only needs to process sparse light spot signals and combine them with a pre-stored high-precision anterior segment optical surface geometry model to achieve high-precision, low-power, and anti-slip eye tracking. When the system is integrated into a head-mounted device, due to the use of a geometric solution method based on a pre-stored model, it has a natural ability to compensate for minor slippage that occurs during device use.

[0086] To obtain personalized anterior face optical surface parameters for users, this scheme can employ various offline measurement devices. As an example, optical coherence tomography (OCT) can be used for offline measurement. Paper 1 (Direct cornealelevation measurements using multiple delay en face optical coherencetomography, J. Biomed. Opt., vol. 13, no. 5, p. 054054, Sep. 2008) reports an en face OCT corneal topography method based on multiple delay elements (MDEs). This method introduces multiple delay elements into the OCT reference arm, enabling a single en face scan to simultaneously acquire information on multiple axial positions. Combined with a high-precision calibration sphere, a mapping relationship between pixel coordinates and actual spatial coordinates is established, thereby directly extracting the three-dimensional coordinates of multiple points on the anterior corneal surface to generate an absolute elevation topographic map. The anterior corneal surface parameters obtained by this method can be used as the personalized anterior face optical surface parameters of this application, pre-stored in the storage unit for subsequent real-time eye tracking.

[0087] Those skilled in the art should understand that the above-described OCT scheme is merely an example, and the offline measurement device of this application is not limited thereto. Any device capable of acquiring absolute elevation data of the anterior segment optical surface, including but not limited to Placido disk corneal topography, Scheimpflug imaging system, structured light projection system, laser scanning ranging system, etc., can be used as the offline measurement device of this application and all fall within the protection scope of this application. The core of this application lies in using a pre-stored personalized anterior segment optical surface geometric model for real-time eye tracking, rather than limiting the method of acquiring offline parameters.

[0088] Example 2: PSD-based eye-tracking method

[0089] (I) Method Overview

[0090] This embodiment provides an eye-tracking method based on a position-sensitive detector (PSD). Utilizing known personalized anterior segment optical surface parameters, multiple point light sources are sequentially illuminated in a time-division multiplexing manner. The position signal of the light spot output by the PSD is detected, and the eye pose is determined in real time by combining geometric calculations. The PSD-based eye-tracking method provided in this embodiment includes... Figure 4 The steps are shown.

[0091] (II) Methods and Procedures

[0092] Step S21: Load the optical surface parameters of the anterior segment

[0093] The system loads the user's personalized anterior corneal optical surface parameters, which have been pre-acquired and stored in the storage unit. These anterior corneal optical surface parameters include, but are not limited to, the curvature distribution of the anterior corneal surface, aspheric coefficient, Zernike coefficient, or B-spline control points.

[0094] Step S22: Point Light Source Timing Control

[0095] Multiple point light sources are controlled to illuminate sequentially and individually using time-division multiplexing. Assume the system has... A point light source, a complete measurement cycle includes The system consists of several consecutive time slices, each corresponding to the lighting period of a point light source. The lighting sequence of each point light source is encoded to distinguish reflected signals from different point light sources.

[0096] Optionally, a guard interval can be set between adjacent time slices to avoid signal crosstalk.

[0097] Step S23: PSD signal acquisition

[0098] During each point light source illumination period, the PSD simultaneously outputs analog current signals along the X and Y axes corresponding to the centroid position of the corneal reflective spot. After amplification, filtering, and analog-to-digital conversion by the analog front-end circuit, the centroid coordinates of the spot for that time slice are obtained. .

[0099] After a complete measurement cycle, the following results are obtained: The centroid coordinates of the light spots constitute the observation data for the current frame.

[0100] Optionally, the PSD signal can be sampled multiple times and averaged within each time slice to reduce random noise.

[0101] Step S24: Extraction of light spot position

[0102] Based on the PSD position calculation formula, the acquired analog current signal is converted into the centroid coordinates of the light spot:

[0103] ,

[0104] in , , , The output current of the corresponding electrode. , The distance between the electrodes is denoted as .

[0105] Step S25: Attitude geometry calculation

[0106] Figure 5This is a schematic diagram of the three-dimensional eye-tracking pose calculation provided in this embodiment, showing the geometric model of the eye in the camera coordinate system. The diagram includes: a camera coordinate system 221, seven point light sources 211, seven reflection points corresponding to the corneal surface 102, and an iris contour 103. The point light sources are represented by seven different shapes "+○×□※◇▽". The reflection points are located on the anterior surface of the cornea, and each reflection point and its corresponding point light source satisfy the law of reflection constraints. The eye pose is solved by optimizing the distance error between the incident light rays and the point light sources at each reflection point. The camera coordinate system 221 is... Figure 2 The camera coordinate system corresponding to the optical sensing unit 220.

[0107] Figure 6 This embodiment provides a perspective projection diagram on a virtual image plane, illustrating the projection trajectory of the iris contour and the projection distribution of reflection points under nine different eye poses. Each pose corresponds to seven reflection point projections 222 and one iris contour projection 223. By comparing the projection distributions of the nine poses, the influence of eye pose changes on the iris contour and reflection point projections can be intuitively reflected, providing a visual basis for the convergence and accuracy of the pose calculation algorithm.

[0108] Figure 5 and Figure 6 Together, they constitute a visual explanation of the attitude calculation method in this embodiment. Figure 5 This shows the geometric model of the eyeball and the distribution of reflection points in the camera coordinate system in three-dimensional space. Figure 6 The projection changes of the iris contour and reflection points onto the image plane under various poses are shown.

[0109] Based on the spatial coordinates of each point light source and the current frame The eye pose is determined through geometric calculation by combining the centroid coordinates of individual light spots with pre-stored personalized anterior segment optical surface parameters.

[0110] Determine the direction of the reflected light based on the position coordinates of each light spot;

[0111] The intersection of the reflected light rays with the surface model described by the optical surface parameters of the anterior segment is used to determine the reflection point on the corneal surface;

[0112] Establish geometric constraints between the incident ray, the reflected ray, and the surface normal based on the law of reflection;

[0113] The eye pose parameters that satisfy the geometric constraints are solved by optimizing the algorithm (minimizing the distance error between the incident ray and the point light source or minimizing the direction error between the incident ray and the directional light source).

[0114] Step S26: Determine the visual axis based on eye pose

[0115] The optical axis of the eye is determined based on the geometric features of the anterior segment optical surface and the eye posture, and then the visual axis of the eye is determined using personalized visual axis calibration parameters.

[0116] Optionally, the method further includes a time filtering step (e.g., Kalman filtering) to smooth the output pose of consecutive frames.

[0117] Example 3: A method for solving eye pose and determining visual axis based on a parametric corneal surface model.

[0118] Specifically, this embodiment provides a method for solving the eyeball pose in step S25 and a method for determining the visual axis in step S26 based on a parametric corneal surface model. The flowchart of the method is shown below. Figure 7 As shown, corresponding Figure 1 Steps S11 to S16 in the process.

[0119] Step S11: Load the anterior segment optical surface parameters. Load the anterior corneal surface... Represented as an explicit function ,in The surface parameter vector is obtained in advance, and the surface parameters include corneal vertex curvature, aspheric coefficient, and higher-order deformation parameters. This surface function... It is continuous and first-order differentiable within its domain.

[0120] Optionally, the initial expression for the optical surface in front of the eye can be a point cloud, a polynomial, etc. For point clouds, they can be pre-fitted as polynomials for easier computation.

[0121] For the surface any point on Its gradient vector is derived from the partial derivatives. and Given the unit normal vector of this point. Obtained through gradient calculation: .

[0122] Step S12: Obtain eye pose. Define the eye pose vector. Including the rotation vector of the eyeball frame Translation vector .

[0123] Let the parametric equation of the reflected ray be: ,in As the starting point of the light , Let be the unit direction vector of the reflected ray. Let be the distance parameter. Assume the centroid coordinates of the light spot detected by the PSD are . The system's equivalent focal length is Then the unit direction vector of the reflected ray It is calculated by the following formula: .

[0124] The reflected light and the curved surface The intersection points are solved by solving a simultaneous equation. and Received, among which Numerical iterative methods (such as the Newton-Raphson method) are used to solve for the intersection parameters. Thus, the coordinates of the intersection point are obtained. .

[0125] According to the law of reflection, the reflected ray and the intersection point... Calculate the incident ray from the normal at the point.

[0126] Calculate each one The distance between an incident ray and its corresponding point source. As an error term.

[0127] The error function is defined as ,in It is the estimated initial attitude. It is attitude error.

[0128] Alternatively, for a directional light source, the angle between the direction of the incident light and the direction of the directional light source can be used as an error.

[0129] Minimize the error function using a nonlinear optimization algorithm (such as the Levenberg-Marquardt algorithm):

[0130]

[0131] Obtain the optimal pose .

[0132] Step S13: Obtain the rotation center of the eye frame. Defining the anterior segment optical surface requires specifying a local coordinate system. The anterior segment optical surface may contain multiple parts. If the local coordinate systems of the different parts of the anterior segment optical surface are different, the transformation relationship between the local coordinate systems needs to be known. Each part of the anterior segment optical surface rotates around the rotation center of the eye frame.

[0133] Optionally, the origin of the local coordinate system of the loaded anterior segment optical surface is the rotation center of the eye frame. In this case, when there are enough reflected light spots, the eye pose can be solved in one frame.

[0134] Optionally, the origin of the local coordinate system of the anterior segment optical surface is not the rotation center of the eyeball frame; for example, the origin of the local coordinate system is the corneal curvature center, the iris center, or any arbitrary point. In this case, multiple frames are required to solve for the eyeball pose.

[0135] S131: Wait for the user to move their eyes to different postures and record the corresponding sensor data.

[0136] S132: Solve the eye pose frame by frame based on sensor data.

[0137] S133: Calculate the spatial transformation between the local coordinate systems of the eye pose in each frame to obtain a common rotation center, which is then used as the rotation center of the eye frame. After obtaining the rotation center of the eye frame, the local coordinate system of the anterior segment optical surface can be normalized to have the rotation center of the eye frame as the origin, and the expression of the anterior segment optical surface can be modified accordingly.

[0138] Step S14: Obtain the orientation of the optical axis in the local coordinate system from the loaded anterior segment optical surface parameters.

[0139] Step S141: If the loaded anterior segment optical surface parameters do not include the orientation of the optical axis in the local coordinate system, then determine the eye's optical axis based on the surface's geometric features or specified rules. And based on the optimal orientation... This transforms the optical axis in the local coordinate system to the optical axis in the global coordinate system.

[0140] Step S15: After obtaining the optical axis, it is calibrated using the user's personalized visual axis calibration parameters to further obtain the visual axis.

[0141] Specifically, using a personalized visual axis calibration parameter quadruple This indicates the deviation between the visual axis and the optical axis. Among them, It is the relative pitch angle between the line of sight and the optical axis. It is the relative yaw angle between the line of sight and the optical axis. It is the length of the common perpendicular segment between the optical axis and the visual axis. It is the distance from the origin to the foot of the perpendicular line between the optical axis and the visual axis on the optical axis.

[0142] In a local coordinate system with the optical axis as the reference direction, the line of sight... .

[0143] The unit direction vector of the common perpendicular of the visual axis and the optical axis The intersection of the common perpendicular and the optical axis The intersection of the perpendicular and the line of sight Thus, the line-of-sight parameter equations are obtained. , .

[0144] Optionally, the personalized visual axis calibration parameters can be included in the loaded anterior segment optical surface parameters, and the visual axis parameter equation can be calculated.

[0145] Step S151: Optionally, the personalized visual axis calibration parameters are not included in the loaded anterior segment optical surface parameters. This application determines the personalized visual axis calibration parameters through an initial optical axis deviation calibration process. .

[0146] Specifically, such as Figure 8 As shown, during the initial optical axis visual axis deviation calibration process, the user gazes at the screen 300. 310 preset fixation points Taking the eyeball frame coordinate system as the local coordinate system 111, and the x-axis of the local coordinate system as the optical axis 112, It is the unit direction vector of the optical axis. The viewing axis 113 deviates to a certain extent from the optical axis.

[0147] For the One gaze point The rotation matrix of the detected eye pose Translation vector .

[0148] All gaze coordinates transformed to the local coordinate system 111 They are collinear. Alternatively, PCA can be used to fit multiple gaze points in the local coordinate system 111. The visual axis is obtained as 113.

[0149] The point on the visual axis 113 can be represented as: , .

[0150] The direction of the line of sight can be determined by any two points. ,use Component calculation of pitch angle Yaw angle .

[0151] The unit direction vector of the common perpendicular of the visual axis and the optical axis .

[0152] The distance from the line of sight to the optical axis is the length of the common perpendicular segment. .

[0153] Establish about The system of equations: Let the intersection of the common perpendicular and the optical axis be... The intersection of the common perpendicular and the line of sight. , ,in It is a point on the line of sight. And .

[0154] Solving the system of equations yields the following results. .

[0155] Optionally, the personalized visual axis calibration parameter calibration only needs to be performed once, and is loaded simultaneously with the registered anterior segment optical surface information during subsequent eye tracking.

[0156] Step S16: In the local coordinate system 111, based on the visual axis 112 and the personalized visual axis calibration parameter quadruple... Calculate the parametric equation of the line along the optical axis 113. , .

[0157] This application achieves a complete three-dimensional spatial transformation from the optical axis to the visual axis by calibrating the visual axis calibration parameter set (including pitch deviation, yaw deviation, common perpendicular segment length, and optical axis foot position), thus solving the problem of line-of-sight estimation error caused by neglecting spatial offset when only considering angular deviation in the prior art.

[0158] Through the above steps, this embodiment achieves low-power, high-precision eye pose solving based on a pre-stored parameterized corneal surface model, which can be further used for gaze tracking.

[0159] Example 4: Eye-tracking system based on a sparse brightness sensor using known anterior segment optical surface parameters

[0160] (I) System Composition

[0161] This embodiment provides an eye-tracking system based on sparse brightness sensors with known anterior corneal optical surface parameters. By utilizing the specular reflection characteristics of the anterior corneal surface, the system senses corneal posture through a sparse brightness sensor array, achieving low-cost eye-tracking without the need for repeated fixation point calibration. Figure 9 This is a schematic diagram of the structure of an eye-tracking system provided in an embodiment of the brightness sensor of this application. The system includes:

[0162] Light source unit 910: includes A light emitter (such as an LED or VCSEL) is configured to project a beam of light toward the user's eyes;

[0163] Optical sensing module 920: includes A luminance sensor (such as a photodiode or a reverse-biased LED) is configured to detect the intensity of a light beam reflected from the anterior surface of the cornea;

[0164] Storage unit 930: configured to store the user's personalized eye-end optical surface parameters;

[0165] Processing unit 940: It is connected to the light source unit 910, the optical sensing module 920 and the storage unit 930 respectively, and realizes functions such as light source control, signal preprocessing and deep learning inference;

[0166] The light source driving unit 941 is configured to control the sequential lighting of each emitter in the light source unit 910 in a time-division multiplexing manner. Pulse width modulation (PWM) is used to drive the emitters to avoid sensor saturation.

[0167] The signal processing unit 942 is configured to receive and process the intensity signal output by the optical sensing module 920, and to retain the corneal specular reflection (effective signal) and suppress the diffuse reflection (noise) of the skin, eyelids and other tissues by using measures such as polarization filtering and differential illumination.

[0168] The deep learning inference unit 943 is configured to determine the current pose of the eyeball using a pre-trained deep learning model based on the preprocessing results output by the signal processing unit 942 and the personalized anterior segment optical surface geometric parameters in the storage unit 930.

[0169] Optionally, the light source unit and the optical sensing module are integrated into a wearable frame (such as an eyeglass frame or a VR headset), and each light emitter and brightness sensor has a known relative spatial position.

[0170] Optionally, the personalized anterior segment optical surface parameters are obtained through an offline measurement device, which includes, but is not limited to, an optical coherence tomography (OCT) device, a Placido disc corneal topography instrument, a Scheimpflug imaging system, or a structured light projection system.

[0171] (II) Physical Principles

[0172] The core physical principle of this embodiment lies in the fact that the anterior surface of the cornea acts as a convex mirror, and its specular reflection intensity varies with the incident angle, a change described by Fresnel's equations. For unpolarized light, the reflectivity can be approximated as: ,in Angle of incidence The reflectivity is approximately 2.5% when incident perpendicularly. This is the proportionality coefficient.

[0173] When the eye's posture changes, the incident angle of each light source... This change leads to variations in the intensity of reflected light received by each brightness sensor. The corresponding changes occur. Therefore, the intensity matrix is ​​composed of all sensor measurements. Eyeball posture Functions: ,in These are pre-stored personalized anterior segment optical surface parameters. Due to the monotonicity of Fresnel reflectivity and the different incident angle ranges corresponding to different light sources and sensor pairs, there is a one-to-one mapping relationship between the intensity matrix and the eye pose, making it possible to deduce the eye pose from the intensity matrix.

[0174] (III) Illuminator Control and Sensor Data Acquisition

[0175] In this embodiment, the multiple emitters are individually lit sequentially according to a preset time sequence, with only one emitter emitting light at any given time, while the others remain off. During the illumination of each emitter, all brightness sensors simultaneously sample and record the current reflected light intensity. Figure 10 This embodiment provides an optical path schematic diagram and a timing diagram for time-division multiplexing. The dashed box 900 represents the head-mounted frame, on which the light source unit 910, optical sensing module 920, storage unit 930, and processing unit 940 are integrated. Figure 10 As shown, multiple emitters 910 (e.g., LED1, LED2) project light beams onto the corneal surface, which are then reflected by the anterior corneal surface and received by multiple luminance sensors 920 (e.g., PD1, PD2). Each emitter is lit sequentially in a time-division multiplexing manner, with only one emitter in the luminous state at any given time, and all luminance sensors sampling synchronously.

[0176] Figure 10 The timing waveform diagram shows the lighting sequence of each emitter 910 within one measurement cycle: the first emitter lights up in the first time slice, the second emitter lights up in the second time slice, and a guard interval is provided between each time slice. All brightness sensors 920 synchronously collect the reflected light intensity within each time slice to form an intensity matrix.

[0177] Optionally, the brightness sensor is a reverse-biased LED. The same LED projects a light beam onto the eye during the light-emitting phase and detects the intensity of reflected light during the light-sensing phase, thus realizing time-division multiplexing of the light-emitting and light-sensing functions.

[0178] Suppose the system has One light source and One brightness sensor, a complete measurement cycle includes The data is collected in consecutive time slices, each time slice corresponding to the lighting period of one emitter. After the data collection is completed, a... intensity matrix , of which elements Indicates the first When the first light-emitting device is lit, the first... The intensity of reflected light measured by a brightness sensor.

[0179] Optionally, the processing unit dynamically adjusts the length of each time slice and the lighting sequence of the emitters based on the real-time signal quality to adapt to different eye movement speeds and environmental conditions. Optionally, the sensor is sampled multiple times within each time slice and averaged to reduce random noise.

[0180] (iv) Diffuse reflection suppression and signal enhancement

[0181] Since the signal received by the brightness sensor 920 simultaneously includes corneal specular reflection (effective signal) and diffuse reflection (noise) from tissues such as skin and eyelids, this embodiment employs the following techniques to suppress diffuse reflection:

[0182] (1) Polarization filtering

[0183] A first polarizer (with a transmission axis of 0°) is placed in front of the light source, and a second polarizer (with a transmission axis parallel to the first polarizer) is placed in front of the brightness sensor. The corneal specular reflection maintains the polarization state of the incident light, which can be efficiently transmitted through parallel polarization configuration; the polarization state of diffuse reflection light is randomized, and only about 50% can pass through the analysis polarizer, thereby achieving about 50% suppression of diffuse reflection.

[0184] (2) Differential lighting

[0185] Each emitter was sequentially encoded and lit in turn. The corneal specular reflection exhibits directional sensitivity with varying incident angles, while diffuse reflection intensity remains relatively stable. The differences in sensor readings when different emitters were lit were calculated. It can effectively counteract global diffuse background components.

[0186] (3) Ambient light reduction

[0187] Insert time slices with all emitters off in each measurement cycle to collect ambient light intensity. and dark current Background subtraction is performed on each sensor reading: .

[0188] (v) Avoid sensor saturation

[0189] To prevent excessively strong specular reflection signals from saturating the sensor, this embodiment employs pulse width modulation (PWM) to drive the emitter. The processing unit dynamically adjusts the duty cycle of the PWM signal based on the signal strength fed back from the brightness sensor, ensuring that the intensity of the reflected light signal remains within the preset linear operating range of the brightness sensor.

[0190] Optionally, the frequency of the PWM signal is from 200Hz to 20kHz. During the system initialization phase, the saturation threshold for the current user is determined by scanning the PWM duty cycle, and the normal operating duty cycle is set below the saturation threshold.

[0191] (vi) Deep learning models

[0192] This embodiment uses a deep learning model to learn the mapping from the intensity matrix to eye pose. To meet the anti-slip requirements, the model is constructed using a conditional mapping approach, with input including the user's personalized anterior segment optical surface parameters.

[0193] Model input: ,in For pre-stored anterior segment optical surface parameters (such as Zernike coefficients or B-spline control points). This is the vectorized representation of the intensity matrix.

[0194] Model output: Eye pose parameters This includes rotation angles (such as Euler angles or quaternions) and translation vectors.

[0195] Optionally, the model is at least one of a fully connected neural network, a graph neural network, or a Transformer.

[0196] Offline Training: The model is obtained through offline training, with training data generated using a synthetic method. During synthetic data generation, firstly, simulations of virtual users in different postures are used to capture data, covering the changes in the eye's rotation center position in three-dimensional space, as well as the range of pitch and yaw angles, enabling the deep learning model to learn continuous mapping relationships in posture space. Secondly, Latin hypercube sampling is used to uniformly sample the individual corneal parameter space, covering reasonable ranges for parameters such as radius of curvature, aspheric coefficient, and astigmatism, to account for individual differences among users. Finally, a noise model is used to simulate sensor response characteristics to reduce the difference between synthetic and real data.

[0197] Online inference: When the user uses the system, pre-stored personalized anterior segment optical surface parameters are loaded, real-time intensity matrices are collected, and the eyeball pose is directly output as input to the model.

[0198] (vii) Differences from existing technologies

[0199] In some schemes, corneal specular reflection is considered interference and discarded through gating or other methods to obtain diffuse reflection signals from the iris or other tissues. In contrast, in this application, the corneal specular reflection signal is considered a carrier of corneal posture information and needs to be retained and utilized. This embodiment enhances the specular reflection signal at the source and suppresses diffuse reflection interference through polarization filtering, differential illumination, and PWM dimming.

[0200] Compared to image sensor-based eye-tracking solutions, this embodiment only requires processing Each scalar intensity value reduces the amount of data by 3-4 orders of magnitude, significantly reducing power consumption and computational complexity. Furthermore, the sensor output does not contain identifiable eye images, thus naturally possessing privacy protection characteristics.

[0201] (viii) Beneficial effects

[0202] Extremely low power consumption: The brightness sensor consumes far less power than the image sensor or PSD, and the time-sharing working mode further reduces power consumption;

[0203] High precision: Through personalized corneal parameters and deep learning models, tracking accuracy comparable to high-end eye trackers can be achieved;

[0204] Ease of use: Only a one-time initial calibration is required to check the deviation between the visual axis and the optical axis; subsequent use does not require frequent calibration.

[0205] Strong robustness: Polarization filtering and differential illumination effectively suppress ambient light and diffuse reflection interference;

[0206] Privacy protection: The sensor only outputs intensity scalars and does not contain images of identifiable biometric features;

[0207] Low cost: The cost of brightness sensors and LEDs is far lower than that of optical lenses and image sensors, making them suitable for large-scale consumer deployments.

[0208] (ix) Postscript

[0209] Those skilled in the art should understand that the number of emitters, sensors, light source layout, PWM parameters, model structure, etc., described in the above embodiments can be adjusted according to specific application requirements. Any modifications, equivalent substitutions, and improvements that do not depart from the spirit and scope of this application should be included within the protection scope of this application.

[0210] Example 5: An Eye-Tracking Method Based on Brightness Sensor Array and Deep Learning

[0211] (I) Method Overview

[0212] This embodiment provides an eye-tracking method based on a brightness sensor array and deep learning. Utilizing known personalized anterior segment optical surface parameters, it controls multiple emitters to illuminate sequentially via time-division multiplexing, collecting the reflected light intensity from each brightness sensor, and combining this with an offline-trained deep learning model to calculate eye pose in real time. Unlike Embodiment 2, this embodiment does not rely on geometric calculations but instead employs a data-driven approach, directly mapping eye pose from intensity patterns. Figure 11 This is a flowchart illustrating the brightness sensor eye-tracking method provided in this application embodiment. The method includes an offline training phase and an online inference phase. The offline training phase includes steps T31 to T32, and the online inference phase includes steps S31 to S36. The offline training phase establishes a mapping from the geometric parameters and intensity matrix of the anterior segment optical surface to the eye pose through synthetic data generation and model training. The online inference phase achieves real-time eye tracking through loading pre-stored parameters, timing control, signal acquisition, preprocessing, deep learning inference, and pose output.

[0213] (II) Offline Training Phase

[0214] The core of this embodiment lies in establishing a mapping model from "anterior segment optical surface parameters + intensity matrix" to "eye pose" through offline training.

[0215] Step T31: Generation of synthetic training data

[0216] Latin hypercube sampling is used to uniformly sample individual corneal parameters, covering a reasonable range of parameters such as radius of curvature R (7.0-9.0 mm), aspheric coefficient Q (-0.5-0), astigmatism C (0-3.0 D), and Kappa angle K (0°-8°), generating... Corneal parameters of a virtual user .

[0217] For each virtual user, simulated in the simulation environment Each light emitter is lit sequentially using a time-division multiplexing method. Each brightness sensor simultaneously collects the intensity of reflected light to obtain an intensity matrix. The intensity of reflected light is calculated using Fresnel's formula: ,in Angle of incidence This represents the reflectivity when incident perpendicularly (approximately 2.5%). This is the proportionality coefficient.

[0218] Noise models are added to the synthetic data to simulate sensor noise (Gaussian noise, Poisson noise), ambient light interference, and emitter intensity fluctuations, thereby improving the model's ability to generalize to real-world environments.

[0219] Step T32: Model Training

[0220] Construct a deep learning model, the input of which is The output is eye pose parameters. The model structure can use a fully connected neural network, for example:

[0221] Input layer: dimension

[0222] Hidden layer 1: 256-dimensional, ReLU activation, Dropout (0.2)

[0223] Hidden layer 2: 128-dimensional, ReLU activation, Dropout (0.2)

[0224] Hidden layer 3: 64-dimensional, ReLU-activated output layer: 6-dimensional, linear activation

[0225] The training loss function is the mean squared error (MSE):

[0226]

[0227] in Weighting coefficients to balance rotational and translational losses.

[0228] Optionally, the deep learning model may also use a graph neural network (GNN) to encode the spatial topological relationship between the emitter and the sensor into the model, or use a Transformer to capture the coupling relationship between different light source-sensor pairs.

[0229] (III) Online Reasoning Stage

[0230] Step S31: Load parameters

[0231] Load the user's personalized anterior segment optical surface parameters (For example, offline measurements via OCT, Placido disk, or Scheimpflug imaging, and pre-stored in the storage unit).

[0232] Step S32: Light emitter timing control

[0233] Controlled by time-division multiplexing Each emitter is lit individually in sequence. Each measurement cycle contains... Each time slice corresponds to a lighting period of a single light emitter, and the lighting sequence of each light emitter is encoded.

[0234] Optionally, each sensor may be sampled multiple times within each time slice and averaged to reduce random noise.

[0235] Step S33: Intensity Matrix Acquisition

[0236] During the period when each light emitter is lit, all Multiple brightness sensors simultaneously sample and record the current intensity of reflected light. After one complete measurement cycle, the following data is obtained: Intensity matrix , of which elements Indicates the first When the first light-emitting device is lit, the first... The intensity of reflected light measured by a brightness sensor.

[0237] Step S34: Signal Preprocessing

[0238] Preprocess the acquired intensity matrix:

[0239] Background subtraction: Ambient light intensity is captured using a time slice when all emitters are off. and dark current Each sensor reading is calibrated: .

[0240] Optionally, a parallel polarizer configuration (with polarizers in front of the emitter and in front of the sensor, and the transmission axes parallel) can be used to suppress diffuse reflection interference.

[0241] Optionally, the PWM duty cycle of the light emitter can be dynamically adjusted to keep the intensity of the reflected light within the linear operating range of the sensor based on the real-time feedback of the sensor signal strength, thus avoiding saturation.

[0242] Step S35: Deep Learning Inference

[0243] Pre-stored personalized anterior segment optical surface parameters and the preprocessed intensity matrix Input to a pre-trained deep learning model:

[0244] The model directly outputs the current eye pose. , including rotation vector (or quaternions / Euler angles) and translation vectors .

[0245] Step S36: Determine the visual axis based on eye posture

[0246] The optical axis of the eye is determined based on the geometric features of the anterior segment optical surface and the eyeball posture, and then the visual axis is determined using visual axis calibration parameters.

[0247] Optionally, the output pose of consecutive frames can be temporally filtered (e.g., Kalman filtering or exponential smoothing) to suppress random noise.

[0248] To overcome the problems of high power consumption, large size, and decreased accuracy due to device slippage that are common in existing eye-tracking technologies, this application proposes a lightweight and anti-slip eye-tracking scheme based on a pre-stored corneal model.

[0249] The core invention lies in utilizing pre-acquired and stored personalized anterior segment optical surface parameters to replace the traditional method of relying on real-time detection by an image sensor and gaze point calibration each time the user uses the device. These parameters serve as a geometric reference, allowing eye pose calculation during real-time tracking to be performed only by detecting sparse reflection points, eliminating the need for high-power real-time image processing by an image sensor. Furthermore, this geometric reference maintains geometric invariance even when the device slips during wear, achieving eye tracking that eliminates the need for repeated gaze point calibration and resists slippage. This application also allows for a one-time visual axis calibration during initial use to determine the deviation between the optical axis and the visual axis, further improving the accuracy of gaze estimation. This calibration only needs to be performed once and does not need to be repeated in subsequent uses.

[0250] Therefore, this application transforms eye tracking from the traditional "image processing" paradigm to a "geometric matching" or "sparse inference" paradigm. Unlike existing low-power solutions that still rely on user calibration and are difficult to resist device slippage, this application, based on a personalized geometric benchmark, fundamentally achieves slippage-resistant eye tracking without the need for repeated calibration; at the same time, since it only needs to process sparse reflection point information, it also has low power consumption characteristics.

[0251] This application also proposes an electronic device, please refer to... Figure 12 The diagram shows the electronic devices described in the first and fourth embodiments of this application, including a processor 10, a memory 20, and a computer program 30 stored in the memory and executable on the processor. When the processor 10 executes the computer program 30, it implements the eye-tracking method as described above. The only difference between the first and fourth embodiments is the specific flow corresponding to the computer program executed by the processor; their hardware architectures are the same.

[0252] The electronic device may be, but is not limited to, a virtual reality headset, a computer, a server, or other devices. In some embodiments, the processor 10 may be a Central Processing Unit (CPU), a controller, a microcontroller, a microprocessor, or other data processing chip, used to run program code stored in the memory 20 or process data.

[0253] The memory 20 includes at least one type of readable storage medium, such as flash memory, hard disk, multimedia card, card-type memory (e.g., SDP or dual memory), magnetic memory, magnetic disk, optical disk, etc. In some embodiments, the memory 20 can be an internal storage unit of an electronic device, such as a hard disk. In other embodiments, the memory 20 can be an external storage device of the electronic device, such as a plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, etc. Furthermore, the memory 20 can include both internal and external storage units of the electronic device. The memory 20 can be used not only to store application software and various types of data installed on the electronic device, but also to temporarily store data that has been output or will be output.

[0254] Optionally, the electronic device may further include a user interface, a network interface, a communication bus, etc. The user interface may include a display, an input unit such as a keyboard, and optionally, a standard wired interface or a wireless interface. Optionally, in some embodiments, the display may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, or an OLED (Organic Light-Emitting Diode) touchscreen, etc. The display may also be appropriately referred to as a screen or display unit, used to display information processed in the electronic device and to display a visual user interface. The network interface may optionally include a standard wired interface or a wireless interface (such as a Wi-Fi interface), typically used to establish communication connections between the device and other electronic devices. The communication bus is used to enable communication between these components.

[0255] It should be pointed out that, Figure 12 The structure shown does not constitute a limitation on the electronic device. In other embodiments, the electronic device may include fewer or more components than shown, or combine certain components, or have different component arrangements.

[0256] This application also proposes a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the eye-tracking method described above.

[0257] Those skilled in the art will understand that the logic and / or steps represented in the flowchart or otherwise described herein are as follows. For example, a sequence of executable instructions for implementing logical functions can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system or apparatus (such as a computer-based system, a processor-included system, or other system that can fetch and execute instructions from or in conjunction with such an instruction execution system or apparatus). For the purposes of this specification, "computer-readable medium" can mean any device that can contain, store, communicate, propagate, or transmit programs for use by or in conjunction with an instruction execution system or apparatus.

[0258] More specific examples of computer-readable media (a non-exhaustive list) include: electrical connectors (electronic devices) with one or more wires, portable computer disk drives (magnetic devices), random access memory (RAM), read-only memory (ROM), erasable and editable read-only memory (EPROM or flash memory), fiber optic devices, and portable optical disc read-only memory (CDROM). Additionally, computer-readable media can even be paper or other suitable media on which the program can be printed, as this can be achieved, for example, by optical scanning of the paper or other medium. The selected program is then edited, interpreted, or otherwise processed as necessary to obtain it electronically, and then stored in computer memory.

[0259] It should be understood that various parts of this application can be implemented using hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented using software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0260] In the description of this specification, references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., refer to specific features, structures, materials, or characteristics described in connection with that embodiment or example, which are included in at least one embodiment or example of this application. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0261] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this patent application should be determined by the appended claims.

Claims

1. An eye-tracking system based on an anterior segment optical surface geometry model, characterized in that, include: The storage unit is configured to store the user's personalized front surface parameters of the anterior end optical surface; The light source unit is configured to project a light beam onto the optical surface in front of the user's eye. An optical sensing module is configured to detect a light beam reflected from the anterior segment optical surface of the cornea and generate a sensor signal containing information related to the light reflected from the anterior surface of the cornea. The processing unit is connected to the storage unit and the optical sensing module respectively, and is configured to determine the current posture of the eyeball through geometric calculation based on the optical surface parameters of the anterior segment and the sensor signal.

2. The system according to claim 1, characterized in that, The optical sensing module is at least one of a position-sensitive detector, an event camera, or an image sensor.

3. The system according to claim 1, characterized in that, The processing unit is configured as follows: The direction vector of the reflected light is determined based on the sensor signal; The intersection of the reflected ray with the surface model described by the optical surface parameters of the anterior segment is used to determine the reflection point on the corneal surface; Establish geometric constraints between the incident ray, the reflected ray, and the surface normal based on the law of reflection; The eye pose parameters that satisfy the geometric constraints are solved by an optimization algorithm. The eye pose includes the eye rotation center and rotation angle.

4. The system according to claim 1, characterized in that, The processing unit is further configured to: predict the position of the theoretical reflected light spot of each beam on the sensor imaging plane based on the currently estimated eye pose during the tracking process; and when the deviation between the actual detected position and the theoretical predicted position exceeds a preset threshold, determine the light spot as abnormal and remove it.

5. The system according to claim 1, characterized in that, The processing unit is further configured to: Guide the user to gaze at at least two preset gaze points with known spatial coordinates; Record the optical axis during gaze; Calculate the user's personalized visual axis calibration parameters based on the deviation between the optical axis direction and the preset gaze point; The personalized line-of-sight calibration parameters include: pitch angle deviation, yaw angle deviation, length of the common perpendicular segment between the optical axis and the line-of-sight, and the distance from the foot of the common perpendicular segment on the optical axis to the starting point of the optical axis. In real-time tracking, the visual axis is calculated based on the optical axis direction and the personalized visual axis calibration parameters.

6. The system according to claim 1, characterized in that, The anterior segment optical surface parameters include parameters of the anterior corneal surface and / or the anterior scleral surface, and the parameters include at least one of curvature distribution, aspheric coefficient, Zernike coefficient, or B-spline control points.

7. An eye-tracking method based on an anterior segment optical surface geometry model, characterized in that, include: Load the user-personalized front surface parameters of the anterior optical surface that are pre-stored in the storage unit; Projecting a beam of light onto the optical surface in front of the user's eye; The light beam reflected from the anterior segment optical surface of the cornea is detected, and a sensor signal is generated, the sensor signal containing information related to the light reflected from the anterior surface of the cornea; Based on the optical surface parameters of the anterior segment and the sensor signals, the current posture of the eyeball is determined through geometric calculation.

8. An eye-tracking system based on an anterior segment optical surface model, characterized in that, include: Storage unit, configured to store the user's personalized eye-spot optical surface parameters; The light source unit includes multiple emitters configured to project a light beam toward the user's eyes; An optical sensing module, including multiple brightness sensors, is configured to detect the intensity of a light beam reflected from the anterior surface of the cornea and generate an intensity signal; The processing unit, connected to the storage unit, the light source unit, and the optical sensing module respectively, is configured as follows: The multiple light emitters are controlled to be lit sequentially and individually in a time-division multiplexing manner; During the illumination of each emitter, intensity signals from all brightness sensors are simultaneously acquired to form an intensity matrix; The personalized anterior segment optical surface parameters and the intensity matrix are input into a pre-trained deep learning model; The current pose of the eyeball is determined based on the output of the deep learning model.

9. The system according to claim 8, characterized in that, Also includes: The first polarizer disposed in front of each light emitter has a first direction for its light transmission axis; The second polarizer, which is placed in front of each brightness sensor, has its transmission axis parallel to the first direction.

10. The system according to claim 8, characterized in that, The processing unit is also configured to: insert a time slice in each measurement cycle during which all emitters are turned off, acquire ambient light intensity and dark current, and perform background subtraction on the intensity signal.

11. The system according to claim 8, characterized in that, The light source unit is driven by pulse width modulation (PWM), and the processing unit dynamically adjusts the PWM duty cycle according to the signal strength fed back by the brightness sensor, so that the intensity of reflected light is kept within a preset linear working range.

12. The system according to claim 8, characterized in that, The deep learning model is at least one of a fully connected neural network, a graph neural network, or a Transformer, and is obtained through offline training; the training data is generated using Latin hypercube sampling, covering a reasonable range of different corneal parameters.

13. The system according to claim 8, characterized in that, The number M of the plurality of light emitters and the number N of the plurality of brightness sensors satisfy M ≥ 3, N ≥ 6, and the product of M and N is not less than 36.

14. An eye-tracking method based on an anterior segment optical surface geometry model, characterized in that, include: Load the user-personalized anterior segment optical surface parameters pre-stored in the storage unit; Multiple emitters are lit sequentially using a time-division multiplexing method. During the lighting period of each emitter, intensity signals from multiple brightness sensors are collected simultaneously to form an intensity matrix. The personalized anterior segment optical surface parameters and the intensity matrix are input into a pre-trained deep learning model; The current pose of the eyeball is determined based on the output of the deep learning model.

15. The method according to claim 14, characterized in that, The offline training of the deep learning model includes: Latin hypercube sampling was used to generate synthetic training data covering different corneal parameters; When generating training data, the eye's pose parameters are made to traverse a preset range of values, including the rotation center position, pitch angle, and yaw angle. Simulate Fresnel reflection characteristics, sensor noise characteristics, and ambient light interference in synthetic data; The deep learning model is trained using the personalized anterior segment optical surface parameters and intensity matrix as input and the eye pose as a label.

16. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by a processor, it implements the steps of the method according to any one of claims 7, 14 to 15.

17. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the method as described in any one of claims 7, 14 to 15.

Images