Eye tracking method, electronic device, and computer-readable storage medium

Abstract

The application relates to the technical field of eye movement tracking, and discloses an eye movement tracking method, an electronic device and a computer readable storage medium. The method comprises the following steps: acquiring a plurality of eye ball images obtained through continuous acquisition; performing intra-frame parallel processing on the eye ball images to obtain corresponding gaze point coordinate parameters, and time windows of various processing steps in the intra-frame parallel processing at least partially overlap; and generating a continuous eye movement trajectory data stream according to the gaze point coordinate parameters. The embodiment of the application can improve the real-time performance of eye movement tracking and reduce end-to-end delay.

Description

Technical Field

[0001] This application relates to the field of eye-tracking technology, and in particular to an eye-tracking method, electronic device, and computer-readable storage medium. Background Technology

[0002] Eye-tracking technology estimates a user's gaze direction by analyzing eye images and is a key technology in fields such as virtual reality interaction, driver fatigue detection, and cognitive psychology research. With the continuous increase in camera frame rates, the demand for real-time performance in high-dynamic-range eye-tracking analysis is rising. However, existing eye-tracking algorithms generally employ a serial pipeline architecture, sequentially performing pupil detection, eye model parameter estimation, and gaze point estimation for each frame. When the camera frame rate reaches 1000Hz, the single-frame processing time window is compressed to 1ms. Existing serial architectures struggle to complete calculations within this time constraint, resulting in a usable frame rate limited by the algorithm's processing speed, failing to meet the end-to-end latency requirements of ultra-high frame rate real-time tracking. Summary of the Invention

[0003] The purpose of this application is to provide an eye-tracking method, electronic device, and computer-readable storage medium that can improve the real-time performance of eye-tracking and reduce end-to-end latency.

[0004] This application provides an eye-tracking method, including: Acquire multiple frames of eye images obtained through continuous acquisition; The eye image is processed in intra-frame parallel to obtain the corresponding gaze point coordinate parameters; the time windows of each processing step in the intra-frame parallel processing overlap at least partially. A continuous eye movement trajectory data stream is generated based on the gaze point coordinate parameters.

[0005] In some embodiments, the intra-frame parallel processing of the eye image includes: Pupil detection is performed on the eye image of the current frame, and model parameters are predicted on the eye image of the current frame in parallel based on the model parameters of the eye image of the previous frame to obtain approximate model parameters; During pupil detection, based on the approximate model parameters, the eye image of the current frame is initially estimated for model parameters and initial for fixation point, to obtain the initially estimated model parameters and initial for fixation point coordinate parameters. When pupil detection is completed, the model parameters of the current frame eye image are determined based on the degree of deviation between the initial estimated model parameters and the pupil contour parameters of the current frame eye image obtained by pupil detection. In parallel, the gaze point of the current frame eye image is re-estimated based on the pupil contour parameters of the current frame eye image to obtain the re-estimated gaze point coordinate parameters. The gaze coordinate parameters of the current frame's eye image are determined based on the degree of deviation between the initially estimated gaze coordinate parameters and the re-estimated gaze coordinate parameters.

[0006] In some embodiments, the model parameters of the eye image are cached in one dual cache area, and the pupil contour parameters of the eye image are cached in another dual cache area. The dual cache area includes a first cache sub-area and a second cache sub-area, and the first cache sub-area and the second cache sub-area are used alternately as write targets and read sources.

[0007] In some embodiments, determining the model parameters of the eye image of the current frame based on the degree of deviation between the initially estimated model parameters and the pupil contour parameters of the eye image of the current frame obtained by pupil detection includes: Map the initial model parameters to the corresponding initial pupil contour parameters; When the deviation between the initially estimated pupil contour parameters and the pupil contour parameters of the current frame's eye image exceeds a preset contour parameter deviation threshold, the initially estimated model parameters are fine-tuned based on the pupil contour parameters of the current frame's eye image to obtain the model parameters of the current frame's eye image; otherwise, the initially estimated model parameters are used as the model parameters of the current frame's eye image.

[0008] In some embodiments, determining the gaze point coordinate parameters of the current frame's eye image based on the degree of deviation between the initially estimated gaze point coordinate parameters and the re-estimated gaze point coordinate parameters includes: When the deviation between the initially estimated fixation point coordinate parameters and the re-estimated fixation point coordinate parameters exceeds a preset coordinate parameter deviation threshold, the re-estimated fixation point coordinate parameters are used as the fixation point coordinate parameters of the eye image of the current frame; otherwise, the initially estimated fixation point coordinate parameters are used as the fixation point coordinate parameters of the eye image of the current frame.

[0009] In some embodiments, the pupil detection processing includes coarse localization processing and fine segmentation processing. Within a preset time window, when the average duration of pupil detection is greater than the average combined duration of the model parameter prediction and the initial model parameter estimation, the coarse localization processing and the fine segmentation processing are executed in parallel.

[0010] In some embodiments, prior to performing intra-frame parallel processing on the eye image, the method further includes: According to the acquisition order of the eye images, the eye images are stored in a cache queue; The eye image is retrieved from the cache queue and distributed to the corresponding parallel processing unit for intra-frame parallel processing.

[0011] In some embodiments, the eye-tracking method further includes: Detect the processor's operating load; When the operating load exceeds a preset load threshold, the number of parallel processing units running is reduced, and / or the frequency of distributing the eye images is reduced.

[0012] This application also provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the eye-tracking method described above.

[0013] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described eye-tracking method.

[0014] The beneficial effects of this application are as follows: When extracting the gaze coordinate parameters of an eye image, intra-frame parallel processing is used to ensure that the time windows of each processing step in the process of extracting gaze coordinate parameters overlap at least partially in scenarios where there is data dependency between some processing steps. This significantly shortens the overall processing time of a single frame of eye image, meets the extremely short time window required by high-speed cameras, improves the real-time performance of eye tracking, and reduces end-to-end latency. Attached Figure Description

[0015] Figure 1 This is a diagram illustrating the application environment of the eye-tracking method provided in the embodiments of this application.

[0016] Figure 2 This is a flowchart of the eye-tracking method provided in the embodiments of this application.

[0017] Figure 3 This is a flowchart of a method for intra-frame parallel processing of eyeball images provided in an embodiment of this application.

[0018] Figure 4 This is a schematic diagram of the dual-buffer structure provided in an embodiment of this application.

[0019] Figure 5 This is a schematic diagram of the structure of the eye-tracking device provided in the embodiments of this application.

[0020] Figure 6 This is a schematic diagram of the hardware structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0021] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0022] It should be noted that although functional modules are divided in the device schematic diagram and a logical order is shown in the flowchart, in some cases, the steps shown may be performed in a different order than the module division in the device or the order in the flowchart. The terms "first," "second," etc., in the specification, claims, and drawings are used to distinguish similar objects and are not used to describe a specific order or sequence.

[0023] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit the application. Furthermore, the information, data, and signals involved in the embodiments of this application are all authorized by the relevant parties or have been fully authorized by the parties, and the collection, use, and processing of related data comply with relevant laws, regulations, and standards.

[0024] In existing eye-tracking technologies, the use of high-frame-rate cameras compresses the single-frame processing time window to the millisecond level. The existing serial pipeline architecture, which sequentially performs pupil detection, eye model parameter estimation, and gaze point estimation for each frame, cannot complete all computational steps within a 1ms time window. This results in the actual usable frame rate being limited by the algorithm's processing speed rather than the hardware's acquisition capabilities. Furthermore, existing parallel processing schemes only achieve inter-frame parallelism, i.e., allocating different frames to multiple processing threads, but do not consider the potential parallelism between processing modules within a single frame. Especially in scenarios where pupil detection and model parameter estimation have data dependencies, inter-frame parallelism cannot effectively reduce single-frame latency, making it difficult to meet the end-to-end latency requirements of ultra-high frame-rate real-time tracking. For example, in virtual reality headsets, when a user rapidly rotates their head, the camera continuously acquires eye images at a rate of 1000Hz. Due to the serial processing architecture, the pupil detection operation in the current frame must be completely completed before model parameter estimation and fixation point estimation can be started. When the eye movement is intense and the pupil detection time increases, the processing time of a single frame exceeds the 1ms time window, causing the eye movement trajectory data stream to be interrupted or delayed.

[0025] If the aforementioned processing latency issue within a single frame is not resolved, the eye-tracking system will be unable to maintain a continuous and stable output of eye movement trajectory data in highly dynamic scenarios. Consequently, gaze drift may occur in virtual reality interactions, key eye movement features may be missed in fatigue driving detection applications, and the accuracy of capturing high-frequency eye movement events in cognitive psychology research will decrease, thus affecting the system's technical reliability in critical scenarios such as real-time interaction, safety monitoring, and experimental analysis.

[0026] Based on this, embodiments of this application provide an eye-tracking method, an electronic device, and a computer-readable storage medium. By employing intra-frame parallel processing technology, the time windows of each processing step partially overlap, optimizing the processing flow in high frame rate scenarios, improving the real-time performance of eye tracking, and reducing end-to-end latency.

[0027] Figure 1 This diagram illustrates the application environment of the eye-tracking method provided in this embodiment. (See attached diagram.) Figure 1 This method is applied to an eye-tracking system. The system includes a camera 101 and an eye-tracking device 102. The camera 101 and the eye-tracking device 102 are communicatively connected. The camera 101 is used to continuously acquire multiple frames of eye images. The eye-tracking device 102 is used to acquire the continuously acquired multiple frames of eye images, perform intra-frame parallel processing on the eye images to obtain corresponding gaze point coordinate parameters, and generate a continuous eye movement trajectory data stream based on the gaze point coordinate parameters. The time windows of each processing step in the intra-frame parallel processing at least partially overlap.

[0028] See Figure 2 In one embodiment, an eye-tracking method is provided, wherein the subject executing the method is the aforementioned eye-tracking device, including but not limited to steps S201 to S203.

[0029] Step S201: Acquire multiple frames of eye images obtained through continuous acquisition.

[0030] Eye images are digital images that reflect the shape and movement characteristics of a user's eyes, continuously captured by optical sensors (such as cameras). Multiple frames of eye images form an eye image sequence.

[0031] There are several ways to continuously acquire multiple frames of eye images. For example, acquisition can be manually triggered, with an operator starting or stopping the camera to continuously capture eye images. Another method is timed acquisition, where the camera is triggered at preset fixed time intervals (e.g., 100 frames per second) to acquire multiple frames of eye images. Event-driven acquisition can also be used, where the camera is triggered to continuously acquire eye images when a specific event (e.g., blinking or head movement) is detected. The eye-tracking device then acquires the acquired multiple frames of eye images from the camera.

[0032] Step S202: Perform intra-frame parallel processing on the eye image to obtain the corresponding gaze point coordinate parameters.

[0033] The time windows of each processing step in intra-frame parallel processing overlap at least partially.

[0034] Intra-frame parallel processing refers to a computational strategy that executes multiple processing steps or tasks within a single frame of an eye image simultaneously or partially simultaneously. Unlike existing serial processing, intra-frame parallel processing aims to reduce the overall processing latency of a single frame and improve processing efficiency through concurrent execution.

[0035] The gaze point coordinate parameter refers to the two-dimensional or three-dimensional coordinate data used to represent the user's gaze position at a specific moment after being processed by the eye-tracking algorithm.

[0036] Intra-parallel processing of eye images can be achieved by dividing the processing task of a single frame into several large, relatively independent subtasks (e.g., image preprocessing, feature extraction, parameter calculation) and attempting to execute these subtasks simultaneously on different processor cores. For example, one core might handle image denoising while another handles edge detection. This coarse-grained parallelism may lead to waiting due to data dependencies between modules, preventing full parallelism. As another approach, a computationally intensive step in single-frame image processing (e.g., matrix operations, filtering) can be decomposed into multiple smaller subtasks, and these subtasks can be executed simultaneously using the parallel computing capabilities of multi-core processors or graphics processing units (GPUs). For example, pixel value calculations can be performed simultaneously on different regions of the image. This fine-grained parallelism may require complex synchronization mechanisms, and not all processing steps can be effectively subdivided. Furthermore, the processing steps of a single frame can be organized into a pipeline, where intermediate results are immediately passed to the next step after the previous step completes part of its task, without waiting for the previous step to finish completely. For example, once a region of an image has completed preprocessing, its result is immediately sent to the feature extraction module for further processing, while the remaining regions of the image are still undergoing preprocessing. Based on this, by ensuring that different processing steps overlap in time—that is, that the time windows at least partially overlap—processor utilization can be maximized. For instance, when pupil detection is halfway through, model parameter prediction has already begun to use some of the available data for calculation.

[0037] Step S203: Generate a continuous eye movement trajectory data stream based on the gaze point coordinate parameters.

[0038] Eye movement data stream refers to a data sequence formed by arranging a series of continuous gaze point coordinate parameters in chronological order, which can dynamically reflect the user's eye movement path and gaze pattern.

[0039] After obtaining the gaze coordinate parameters for each frame, they need to be integrated into a complete eye movement trajectory. One approach is direct stitching, simply connecting the gaze coordinate parameters obtained from each frame in chronological order to form the original eye movement trajectory data stream. As an improvement, the gaze coordinate parameters for each frame can be low-pass filtered or averaged before stitching to reduce noise, and then the eye movement trajectory data stream can be generated. Furthermore, when the gaze coordinate parameters for some frames are missing or inaccurate, linear interpolation or spline interpolation can be performed using valid data from preceding and following frames to fill in the missing data, thereby generating a more continuous eye movement trajectory data stream.

[0040] In summary, the eye-tracking method provided in this application, when extracting the gaze coordinate parameters of an eye image, ensures that the time windows of each processing step in the process of extracting the gaze coordinate parameters overlap at least partially in scenarios where there are data dependencies between some processing steps. This significantly shortens the overall processing time of a single frame of eye image, meets the extremely short time window required by high-speed cameras, improves the real-time performance of eye tracking, and reduces end-to-end latency.

[0041] See Figure 3 In one embodiment, the method for intra-frame parallel processing of eye images includes, but is not limited to, steps S301 to S303.

[0042] Step S301: Perform pupil detection on the eye image of the current frame, and in parallel, predict the model parameters of the eye image of the current frame based on the model parameters of the eye image of the previous frame to obtain approximate model parameters.

[0043] In step S302, during pupil detection, the model parameters and gaze point are initially estimated for the current frame's eye image based on the approximate model parameters, resulting in the initially estimated model parameters and gaze point coordinate parameters.

[0044] Step S303: When pupil detection is completed, the model parameters of the current frame eye image are determined based on the degree of deviation between the initially estimated model parameters and the pupil contour parameters of the current frame eye image obtained by pupil detection. In parallel, the gaze point of the current frame eye image is re-estimated based on the pupil contour parameters of the current frame eye image to obtain the re-estimated gaze point coordinate parameters.

[0045] Step S304: Determine the gaze point coordinate parameters of the current frame's eye image based on the degree of deviation between the initially estimated gaze point coordinate parameters and the re-estimated gaze point coordinate parameters.

[0046] Pupil detection refers to the process of identifying and locating the pupil region in an eye image. Its purpose is to provide a precise geometric basis for subsequent eye model parameter estimation and fixation point calculation. Pupil detection can be implemented using various image processing algorithms, such as edge detection, region growing, or thresholding methods. It can also utilize deep learning models for semantic segmentation to obtain pixel-level pupil region details.

[0047] Model parameter prediction refers to using the eye model parameters determined in previous frames to estimate the eye model parameters in the current frame through a prediction algorithm, thereby obtaining a preliminary approximate model parameter. Prediction algorithms can include Kalman filtering, linear regression analysis, or prediction models based on time series analysis.

[0048] Preliminary model parameter estimation and gaze point estimation refer to the rapid estimation of preliminary eye model parameters and gaze point coordinates during the parallel stage before pupil detection processing is fully completed. This estimation utilizes the obtained approximate model parameters and combines them with partial image information from the current frame. This preliminary estimation provides a fast, initial result, laying the foundation for subsequent refined processing and thus reducing overall processing time. Preliminary estimation can be based on simplified geometric model fitting, such as ellipse fitting, or can be performed quickly using a pre-trained lightweight neural network model.

[0049] Fixation recalculation refers to the process of recalculating more accurate fixation coordinates based on precise pupil contour parameters obtained from pupil detection and combined with a geometric model of the eye. This recalculation provides a high-precision fixation coordinate parameter as a benchmark for comparison with the initial fixation coordinate parameter. Recalculation can be performed using ray tracing principles, more complex eye movement models, or vector calculation methods based on the pupil center and corneal reflector (if available).

[0050] This application's solution achieves a balance between efficiency and accuracy by organizing the various steps of intra-frame parallel processing. First, the time-consuming pupil detection operation is performed in parallel with model parameter prediction. Model parameter prediction quickly generates approximate model parameters for the current frame using the eye model parameters from the previous frame. While pupil detection is still in progress, these approximate model parameters are used for initial model parameter estimation and fixation point estimation, thus obtaining preliminary fixation point coordinate parameters at an early stage. Once pupil detection is complete and accurate pupil contour parameters are obtained, two parallel operations are immediately performed using these parameters: first, to determine more accurate eye model parameters based on the deviation between the initially estimated model parameters and the pupil contour parameters; second, to re-estimate the fixation point based on the pupil contour parameters, obtaining re-estimated fixation point coordinate parameters. Finally, by comparing the deviation between the initially estimated and re-estimated fixation point coordinate parameters, the final fixation point coordinate parameters are selected or corrected. This multi-stage, parallel, and mutually verified processing flow utilizes the processor's idle time to superimpose time-consuming operations with rapid prediction and initial estimation operations, and performs rapid corrections after obtaining accurate data, thereby improving the accuracy of gaze point coordinate parameters while ensuring real-time performance.

[0051] The following example illustrates the technical advantages of intra-frame parallel processing over serial schemes.

[0052] For a single frame of an eye image, the total time consumed by the serial scheme is: T A +T B +T C =1.0ms, Among them, T A The pupil detection time is approximately 0.4 ms, T B The duration for estimating the eyeball model parameters is approximately 0.3 ms, T C The estimated fixation duration is approximately 0.3 ms.

[0053] For a single frame of an eye image, the intra-frame parallel processing flow is as follows: 0ms~0.1ms: Pupil detection and model parameter prediction start simultaneously and run in parallel. Initial fixation point estimation is completed in about 0.1ms. 0.1ms~0.4ms: Initial model parameter estimation and initial fixation point estimation are initiated, and pupil detection, initial model parameter estimation, and initial fixation point estimation run in parallel; 0.4ms: Pupil detection, initial model parameter estimation, and initial fixation point estimation are completed; 0.4ms~0.7ms: The model parameters of the eye image are determined, which is completed in about 0.45ms. The gaze point re-estimation is started in parallel, which is completed in about 0.7ms. 0.7ms~0.75ms: Determine the gaze coordinate parameters of the eye image.

[0054] Therefore, the total time required for intra-frame parallel processing of a single eye image is: max(T A ,T P +T B’ ,T P +T c’ )+T C’’ +T C’’’ , Among them, T P For the prediction duration of model parameters, T B’ For the initial estimation of model parameters, T C’ For the initial estimation of fixation duration, T C’’ For fixation re-estimation duration, T C’’’ Correct the fixation time.

[0055] In typical scenarios, T A T P +T B’ and T P +T c’ Approximately 0.4ms, T C’’ Approximately 0.3ms, T C’’’ The latency is approximately 0.05ms, with a total delay of approximately 0.75ms, which is significantly better than the 1.0ms latency of the serial solution.

[0056] In some embodiments, model parameters of an eye image are cached in one dual cache area, and pupil contour parameters of the eye image are cached in another dual cache area. The dual cache area includes a first cache sub-area and a second cache sub-area, and the first cache sub-area and the second cache sub-area are used alternately as write targets and read sources.

[0057] Dual buffers can be a high-speed random access memory (RAM) region, such as the L1 and L2 caches inside the processor, or an external DDR memory region, for fast read and write operations. Alternatively, dual buffers can be FIFO (First-In-First-Out) buffers specifically designed for image processing or real-time data streaming, or a circular buffer to ensure data continuity and efficient updates.

[0058] The alternation of the first and second cache sub-areas as write targets and read sources means that the first and second cache sub-areas alternately switch to active states, with the data provider writing data to the currently inactive cache sub-area and the data user reading data from the currently active cache sub-area.

[0059] In the above-mentioned eye-tracking method, in order to efficiently process the continuous stream of eye images and ensure the real-time updating and accuracy of model parameters and pupil contour parameters, the embodiments of this application configure corresponding dual buffers for caching and updating the model parameters and pupil contour parameters of the eye images, respectively.

[0060] See Figure 4 The dual buffer comprises a first buffer sub-region B1 and a second buffer sub-region B2. The states of both sub-regions B1 and B2 are identified by an atomic flag. The entity writing data to either sub-region B1 or B2 is the data provider, and the entity reading data from either sub-region B1 or B2 is the data user. For the dual buffer storing model parameters of eye images, during intra-frame parallel processing of the current frame's eye image, if sub-region B1 is active and sub-region B2 is inactive, the module implementing model parameter prediction, acting as the data user, reads the model parameters of the previous frame's eye image from sub-region B1. Upon completion of model parameter prediction, the module acting as the data provider writes approximate model parameters into sub-region B2 and updates the atomic flag, making sub-region B1... While the second buffer sub-region B2 is in an inactive state, during pupil detection, the modules implementing model parameter estimation and gaze point estimation, as data users, read approximate model parameters from the second buffer sub-region B2. After determining the model parameters of the eye image in the current frame, the module implementing model parameter estimation, as a data provider, writes the model parameters of the eye image in the current frame into the first buffer sub-region B1 and updates the atomic flag bit flag, making the first buffer sub-region B1 in an active state and the second buffer sub-region B2 in an inactive state. For a dual-buffer system that caches pupil contour parameters of eye images, when performing intra-frame parallel processing on the eye image of the current frame, if the first buffer sub-region B1 is active and the second buffer sub-region B2 is inactive, when pupil detection is completed, the module implementing pupil detection, as the data provider, writes the pupil contour parameters into the second buffer sub-region B2 and updates the atomic flag bit flag, so that the first buffer sub-region B1 is inactive and the second buffer sub-region B2 is active. The modules implementing model parameter estimation and the modules implementing gaze point estimation, as data users, read the pupil contour parameters from the second buffer sub-region B2.

[0061] In some embodiments, determining the model parameters of the eye image in the current frame based on the degree of deviation between the initially estimated model parameters and the pupil contour parameters of the eye image in the current frame obtained by pupil detection includes: mapping the initially estimated model parameters to the corresponding initially estimated pupil contour parameters; when the deviation between the initially estimated pupil contour parameters and the pupil contour parameters of the eye image in the current frame exceeds a preset contour parameter deviation threshold, fine-tuning the initially estimated model parameters based on the pupil contour parameters of the eye image in the current frame to obtain the model parameters of the eye image in the current frame; otherwise, using the initially estimated model parameters as the model parameters of the eye image in the current frame.

[0062] In eye tracking, the initially estimated model parameters are first converted into corresponding initially estimated pupil contour parameters on the image plane. Then, these initially estimated pupil contour parameters are compared with the actual pupil contour parameters extracted from the current frame's eye image using a pupil detection algorithm, and the deviation between the two is calculated. If the deviation is small, it indicates that the initially estimated model parameters are close enough to reality, and in this case, the initially estimated model parameters can be directly used as the final model parameters for the current frame, avoiding unnecessary computational overhead. If the deviation exceeds the contour parameter deviation threshold, it indicates a significant difference between the initially estimated model parameters and the actual detection results, and direct use may lead to errors in subsequent gaze point calculations. In this case, the initially estimated model parameters are fine-tuned based on the pupil contour parameters of the actually detected eye image of the current frame. This fine-tuning process aims to bring the model parameters closer to the actual observed values, obtaining more accurate model parameters for the current frame's eye image. Through this mechanism, the proposed solution can intelligently determine when more refined model parameter optimization is needed, avoiding redundant calculations when the model parameters are already sufficiently accurate, while ensuring timely correction when there are significant errors in the initially estimated model parameters. This makes the process of determining model parameters both efficient and accurate, providing a solid foundation for subsequent fixation point calculations.

[0063] In some embodiments, determining the gaze point coordinate parameters of the current frame's eye image based on the degree of deviation between the initially estimated gaze point coordinate parameters and the re-estimated gaze point coordinate parameters includes: when the deviation between the initially estimated gaze point coordinate parameters and the re-estimated gaze point coordinate parameters exceeds a preset coordinate parameter deviation threshold, using the re-estimated gaze point coordinate parameters as the gaze point coordinate parameters of the current frame's eye image; otherwise, using the initially estimated gaze point coordinate parameters as the gaze point coordinate parameters of the current frame's eye image.

[0064] The aforementioned eye-tracking methods generate initial and re-estimated gaze coordinate parameters when performing intra-frame parallel processing on eye images. To make the optimal choice between the two, this application introduces an intelligent decision-making mechanism. Specifically, it first calculates the deviation between the initial and re-estimated gaze coordinate parameters. This deviation quantifies the consistency between the two estimation results. Subsequently, it compares this deviation with a preset coordinate parameter deviation threshold. If the deviation exceeds the threshold, it indicates that the initial gaze coordinate parameters may have a large error, and the more accurate re-estimated gaze coordinate parameters are selected as the final gaze coordinate parameters for the current frame of the eye image. Conversely, if the deviation does not exceed the threshold, it indicates that the initial gaze coordinate parameters are accurate enough, and the initial gaze coordinate parameters are directly adopted as the final result. In this way, this application's solution can flexibly select more reliable gaze coordinate parameters based on the actual estimation accuracy, thereby ensuring both accuracy and processing efficiency.

[0065] In some embodiments, pupil detection processing includes coarse localization processing and fine segmentation processing. Within a preset time window, when the average duration of pupil detection is greater than the average combined duration of model parameter prediction and initial model parameter estimation, coarse localization processing and fine segmentation processing are performed in parallel.

[0066] In the aforementioned eye-tracking method, intra-frame parallel processing of eye images aims to maximize processing efficiency. When it is consistently observed that the average duration of pupil detection is significantly longer than the combined average duration of model parameter prediction and initial model parameter estimation within a preset time window, it indicates that pupil detection has become the main time-consuming step in the current processing flow. At this point, the parallel execution mode within pupil detection is triggered, i.e., coarse localization and fine segmentation are performed simultaneously. Coarse localization quickly provides approximate pupil location information, while fine segmentation, based on this preliminary information, performs high-precision contour extraction within a smaller area. The two processes overlap in time, thus significantly shortening the overall completion time of pupil detection. This dynamically adjusted parallel strategy ensures that the pupil detection process itself does not slow down the progress of the entire intra-frame parallel processing, allowing eye images to be processed at a higher frame rate, thereby guaranteeing the real-time performance and smoothness of the continuous eye movement trajectory data stream.

[0067] In some embodiments, before performing intra-parallel processing on the eye images, the eye tracking method further includes: storing the eye images in a buffer queue according to the acquisition order of the eye images; retrieving the eye images from the buffer queue and distributing them to the corresponding parallel processing units for intra-parallel processing on the eye images in the parallel processing units.

[0068] A buffer queue is a first-in, first-out (FIFO) data structure used to temporarily store continuously acquired eye images. Its main function is to act as a buffer between the image acquisition module and the image processing module, addressing potential data rate mismatches and ensuring smooth data transmission. Buffer queues can be implemented in memory using various methods such as circular buffers, linked lists, or arrays.

[0069] A parallel processing unit (PCU) is a computational entity that performs intra-frame parallel processing tasks for eye images. It can be a dedicated hardware accelerator (e.g., a field-programmable gate array (FPGA) or application-specific integrated circuit (ASIC)), a processing core in a multi-core processor (e.g., a CPU core or a stream processor in a graphics processing unit (GPU), or an independent processing thread or process running on a general-purpose processor. Its function is to perform complex computational tasks such as pupil detection, model parameter prediction, and gaze point estimation.

[0070] This application's solution optimizes the process of eye image acquisition and intra-frame parallel processing by introducing an image caching and distribution mechanism. For details, please refer to... Figure 5 After eye images are acquired sequentially, they are first stored in a buffer queue according to their acquisition order. This buffer queue acts as an intermediate buffer, effectively decoupling the image acquisition speed from the subsequent image processing speed, avoiding data loss or processing delays caused by excessively fast acquisition speeds or temporary lags in processing speeds. Subsequently, the frame-level task scheduler retrieves eye images from this buffer queue and distributes them to the corresponding parallel processing units. This distribution mechanism ensures that each parallel processing unit receives the eye images to be processed in a timely manner, and that these images are processed in the original acquisition order. Finally, each parallel processing unit inputs the obtained gaze point coordinate parameters into the result fusion module, which generates an eye movement trajectory data stream. In this way, even under conditions of fluctuating image acquisition rates or uneven loads on parallel processing units, the orderly and continuous supply of eye images can be guaranteed, thus providing a stable and reliable data source for subsequent intra-frame parallel processing, ultimately contributing to the generation of a more continuous and accurate eye movement trajectory data stream.

[0071] In some embodiments, the eye-tracking method further includes: detecting the processor's operating load; and when the operating load exceeds a preset load threshold, reducing the number of parallel processing units in operation and / or reducing the frequency of distributing eye images.

[0072] Detecting processor load refers to real-time monitoring of the workload borne by a processor (such as a CPU, GPU, or DSP) while executing tasks. This can be achieved in several ways, such as periodically reading performance metrics like processor utilization, task queue length, memory usage, or I / O operation frequency through the operating system's application programming interface (API). Another approach is to integrate a hardware monitoring module within the processing unit, which can output the processor's workload or the number of tasks completed per unit time in real time, thus indirectly reflecting the load. A preset load threshold is a pre-defined critical value used to determine whether the processor load is too high. This threshold can be determined based on system design requirements, hardware performance test results, and expected system response time. For example, a certain percentage of CPU utilization (such as 85% or 90%) can be set as the threshold, or the maximum allowable length of the processing unit's task queue can be set as the threshold.

[0073] This application's solution achieves intelligent management of system resources by real-time detection of processor load and dynamic adjustment of the number of parallel processing units and / or the frequency of eye image distribution based on the load. When the processor load exceeds a preset load threshold, it indicates that system resources may be nearing saturation or overload. At this point, reducing the number of parallel processing units directly reduces the consumption of computing resources, allowing the remaining active processing units to operate more efficiently and avoiding resource contention. Simultaneously, or as a supplementary measure, reducing the frequency of eye image distribution reduces the amount of raw data entering the processing pipeline, thereby alleviating the immediate workload of the parallel processing units and preventing cache queue overflow and data processing delays. This adaptive load management mechanism enables the eye-tracking system to maintain stable operation and acceptable performance levels even under conditions of limited resources or fluctuating processing demands, effectively avoiding performance degradation or system crashes caused by system overload and ensuring stable output of continuous eye-tracking trajectory data streams.

[0074] This application also provides an electronic device. Figure 6 This is a schematic diagram of the hardware structure of the electronic device provided in an embodiment of this application. For example... Figure 6 As shown, the electronic device in this embodiment mainly includes a processor 601 and a memory 602. The memory 602 can be configured to store a program for executing the eye-tracking method of the above-described method embodiments, and the processor 601 can be configured to execute the program in the memory 602. This program includes, but is not limited to, a program for executing the eye-tracking method of the above-described method embodiments. For ease of explanation, only the parts related to the embodiments of this application are shown. For specific technical details not disclosed, please refer to the method section of the embodiments of this application.

[0075] In some embodiments, the electronic device may include multiple processors 601 and multiple memories 602. The program executing the eye-tracking method of the above-described method embodiments can be divided into multiple subroutines, each of which can be loaded and run by a processor 601 to perform different steps of the eye-tracking method of the above-described method embodiments. Specifically, each subroutine can be stored in a different memory 602, and each processor 601 can be configured to execute programs in one or more memories 602 to jointly implement the eye-tracking method of the above-described method embodiments; that is, each processor 601 executes different steps of the eye-tracking method of the above-described method embodiments to jointly implement the eye-tracking method of the above-described method embodiments.

[0076] The aforementioned multiple processors 601 can be processors deployed on the same device. For example, the aforementioned electronic device can be a high-performance device composed of multiple processors, and the aforementioned multiple processors 601 can be processors configured on that high-performance device. Alternatively, the aforementioned multiple processors 601 can also be processors deployed on different devices. For example, the aforementioned electronic device can be a server cluster, and the aforementioned multiple processors 601 can be processors on different servers within the server cluster.

[0077] This application also provides a computer-readable storage medium. In one embodiment of the computer-readable storage medium according to this application, the computer-readable storage medium can be configured to store a program that performs the eye-tracking method of the above-described method embodiments. This program can be loaded and run by a processor to implement the above-described eye-tracking method. For ease of explanation, only the parts related to the embodiments of this application are shown; for specific technical details not disclosed, please refer to the method section of the embodiments of this application. The computer-readable storage medium can be a memory formed by various electronic devices. Optionally, in the embodiments of this application, the computer-readable storage medium is a non-transitory computer-readable storage medium.

[0078] The eye-tracking method, electronic device, and computer-readable storage medium provided in this application, when extracting gaze coordinate parameters of an eye image, ensure that the time windows of each processing step in the process of extracting gaze coordinate parameters overlap at least partially in scenarios where there are data dependencies between some processing steps. This significantly shortens the overall processing time of a single frame of eye image, meets the extremely short time window required by high-speed cameras, improves the real-time performance of eye tracking, and reduces end-to-end latency.

[0079] Exemplary embodiments of this disclosure have been specifically shown and described above. It should be understood that this disclosure is not limited to the detailed structures, arrangements, or implementations described herein; rather, this disclosure is intended to cover various modifications and equivalent arrangements contained within the spirit and scope of the appended claims.

Claims

1. An eye-tracking method, characterized in that, include: Acquire multiple frames of eye images obtained through continuous acquisition; The eye image is processed intra-frame in parallel to obtain the corresponding gaze point coordinate parameters; The time windows of each processing step in the intra-frame parallel processing overlap at least partially; Based on the gaze point coordinate parameters, a continuous eye movement trajectory data stream is generated; The intra-frame parallel processing of the eyeball image includes: Pupil detection is performed on the eye image of the current frame, and model parameters are predicted on the eye image of the current frame in parallel based on the model parameters of the eye image of the previous frame to obtain approximate model parameters; During pupil detection, based on the approximate model parameters, the eye image of the current frame is initially estimated for model parameters and initial for fixation point, to obtain the initially estimated model parameters and initial for fixation point coordinate parameters. When pupil detection is completed, the model parameters of the current frame eye image are determined based on the degree of deviation between the initial estimated model parameters and the pupil contour parameters of the current frame eye image obtained by pupil detection. In parallel, the gaze point of the current frame eye image is re-estimated based on the pupil contour parameters of the current frame eye image to obtain the re-estimated gaze point coordinate parameters. The gaze coordinate parameters of the current frame's eye image are determined based on the degree of deviation between the initially estimated gaze coordinate parameters and the re-estimated gaze coordinate parameters.

2. The eye-tracking method according to claim 1, characterized in that, The model parameters of the eye image are cached in one dual cache area, and the pupil contour parameters of the eye image are cached in another dual cache area. The dual cache area includes a first cache sub-area and a second cache sub-area, and the first cache sub-area and the second cache sub-area are used alternately as write targets and read sources.

3. The eye-tracking method according to claim 1, characterized in that, The step of determining the model parameters of the eye image in the current frame based on the deviation between the initially estimated model parameters and the pupil contour parameters of the eye image in the current frame obtained by pupil detection includes: Map the initial model parameters to the corresponding initial pupil contour parameters; When the deviation between the initially estimated pupil contour parameters and the pupil contour parameters of the current frame's eye image exceeds a preset contour parameter deviation threshold, the initially estimated model parameters are fine-tuned based on the pupil contour parameters of the current frame's eye image to obtain the model parameters of the current frame's eye image; otherwise, the initially estimated model parameters are used as the model parameters of the current frame's eye image.

4. The eye-tracking method according to claim 1, characterized in that, Determining the gaze point coordinate parameters of the current frame's eye image based on the deviation between the initially estimated gaze point coordinate parameters and the re-estimated gaze point coordinate parameters includes: When the deviation between the initially estimated fixation point coordinate parameters and the re-estimated fixation point coordinate parameters exceeds a preset coordinate parameter deviation threshold, the re-estimated fixation point coordinate parameters are used as the fixation point coordinate parameters of the eye image of the current frame; otherwise, the initially estimated fixation point coordinate parameters are used as the fixation point coordinate parameters of the eye image of the current frame.

5. The eye-tracking method according to claim 1, characterized in that, The pupil detection process includes coarse localization and fine segmentation. Within a preset time window, when the average duration of pupil detection is greater than the average combined duration of the model parameter prediction and the initial model parameter estimation, the coarse localization and fine segmentation processes are executed in parallel.

6. The eye-tracking method according to claim 1, characterized in that, Before performing intra-frame parallel processing on the eyeball image, the method further includes: According to the acquisition order of the eye images, the eye images are stored in a cache queue; The eye image is retrieved from the cache queue and distributed to the corresponding parallel processing unit for intra-frame parallel processing.

7. The eye-tracking method according to claim 6, characterized in that, Also includes: Detect the processor's operating load; When the operating load exceeds a preset load threshold, the number of parallel processing units running is reduced, and / or the frequency of distributing the eye images is reduced.

8. An electronic device, characterized in that, The electronic device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the eye-tracking method according to any one of claims 1 to 7.

9. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the eye-tracking method according to any one of claims 1 to 7.

Images