Eye-tracking model training methods, eye-tracking methods, devices and systems

Abstract

This disclosure provides an eye-tracking model training method, an eye-tracking method, an apparatus, and a system. The eye-tracking model training method includes: receiving multiple frames of scene images acquired by a scene camera at a first preset frequency; receiving multiple events acquired by an event camera at a second preset frequency; processing the nth frame of the scene image to obtain the gaze coordinates of the user's gaze target in the nth frame of the scene image, which are used as the nth gaze coordinate label; determining the nth time window based on the nth frame of the scene image; extracting events whose timestamps are located within the nth time window from the multiple events to obtain the nth event set; processing the nth event set to obtain the nth voxel tensor; processing the nth voxel tensor using an eye-tracking model to obtain the nth gaze coordinate prediction result; and training the eye-tracking model based on the N gaze coordinate prediction results and the N gaze coordinate labels.

Description

Technical Field

[0001] This disclosure relates to the field of computer vision technology, and in particular to an eye-tracking model training method, eye-tracking method, device and system. Background Technology

[0002] Eye-tracking technology, as a key interaction technology for human-computer interaction, is evolving towards higher temporal resolution and lower system latency. Currently, eye-tracking technology typically analyzes image frames captured by ordinary cameras.

[0003] However, the human eye moves extremely fast during saccades. Traditional frame-based image acquisition techniques, limited by the exposure time and sampling frequency of ordinary cameras, suffer from motion blur when capturing saccades. Furthermore, the range of human eye movement is very small compared to the imaging range of an image frame; therefore, traditional frame-based image acquisition techniques also face the problem of excessive data redundancy.

[0004] Neuromorphic vision sensors, also known as event cameras, are a type of biomimetic vision sensor inspired by the working principles of the biological retina. Unlike conventional cameras that use full-frame exposure, event cameras employ an asynchronous triggering mechanism, outputting event data only when the pixel's photosensitive unit detects a change in light intensity exceeding a specific threshold. With their microsecond-level response time, extremely high dynamic range (HDR), and data sparsity, event cameras are gradually becoming core sensor components in high-performance eye-tracking systems. Summary of the Invention

[0005] The inventors noted that in eye-tracking technology based on event cameras, it is difficult to obtain accurate labels for the collected event data, resulting in poor training performance of the model and low accuracy of the model's prediction results.

[0006] Accordingly, this disclosure provides an eye-tracking model training method that can improve the prediction accuracy of gaze point coordinates.

[0007] According to a first aspect of the present disclosure, an eye-tracking model training method is provided, comprising: receiving multiple frames of scene images acquired by a scene camera at a first preset frequency; receiving multiple events acquired by an event camera at a second preset frequency, wherein the second preset frequency is greater than the first preset frequency; processing the nth frame of the scene image to obtain the gaze coordinates of a user's gaze target in the nth frame of the scene image, and using them as the nth gaze coordinate label, wherein... N represents the total number of scene images; based on the nth scene image, the nth time window is determined; events with timestamps within the nth time window are extracted from multiple events to obtain the nth event set; the nth event set is processed to obtain the nth voxel tensor; the nth voxel tensor is processed using an eye-tracking model to obtain the nth gaze coordinate prediction result; based on the N gaze coordinate prediction results and N gaze coordinate labels, the eye-tracking model is trained.

[0008] In some embodiments, the eye-tracking model includes a spatial feature extraction module, a temporal modeling module, and a regression module. Processing the nth voxel tensor using the eye-tracking model to obtain the nth gaze coordinate prediction result includes: extracting features from the nth voxel tensor using the spatial feature extraction module to obtain the spatial structure features of the nth voxel tensor; extracting features from the spatial structure features using the temporal modeling module to obtain the temporal features of the nth voxel tensor; and mapping the temporal features using the regression module to obtain the nth gaze coordinate prediction result.

[0009] In some embodiments, the spatial feature extraction module includes a multi-layer convolutional neural network; the temporal modeling module includes a short-term temporal modeling module and a global temporal modeling module, wherein the short-term temporal modeling module includes multiple gated recurrent units, and the global temporal modeling module includes a Transformer encoder.

[0010] In some embodiments, training an eye-tracking model based on N gaze coordinate prediction results and N gaze coordinate labels includes: determining a loss value based on the nth gaze coordinate prediction result and the nth gaze coordinate label; and training the eye-tracking model using the loss value.

[0011] In some embodiments, the nth gaze coordinate prediction result includes M gaze coordinate prediction values, where the mth gaze coordinate prediction value is the prediction value of the eye-tracking model at the mth time step. M represents the total number of time steps. Based on the prediction result of the nth gaze coordinate and the label of the nth gaze coordinate, the loss value is determined as follows: the first sub-loss value is determined based on the prediction value of the Mth gaze coordinate in the prediction result of the nth gaze coordinate and the label of the nth gaze coordinate; the second sub-loss value is determined based on all M prediction values ​​of the nth gaze coordinate and the label of the nth gaze coordinate; and the loss value is determined using the first sub-loss value and the second sub-loss value.

[0012] In some embodiments, determining the loss value includes: calculating a weighted sum of the first sub-loss value and the second sub-loss value to obtain the loss value.

[0013] In some embodiments, processing the nth event set to obtain the nth voxel tensor includes: determining whether the nth event set is a valid event set; if the nth event set is a valid event set, processing the nth event set to obtain the nth voxel tensor; if the nth event set is not a valid event set, deleting the nth event set and the nth gaze coordinate label.

[0014] In some embodiments, determining whether the nth event set is a valid event set includes: determining whether the nth event set is a valid event set based on at least one of the total number of events in the nth event set, the effective event pixel coverage, the event concentration, and the positive and negative polarity ratio.

[0015] In some embodiments, determining whether the nth event set is a valid event set based on the total number of events in the nth event set includes: determining whether the total number of events in the nth event set is within a first preset range; if the total number of events is within the first preset range, then the nth event set is a valid event set; if the total number of events is not within the first preset range, then the nth event set is not a valid event set.

[0016] In some embodiments, determining whether the nth event set is a valid event set based on the effective event pixel coverage rate of the nth event set includes: determining the number of events occurring for each pixel within a preset target area based on the pixel coordinates of each event in the nth event set; identifying pixels within the preset target area whose number of events exceeds a threshold as active pixels; calculating the ratio of the number of active pixels within the preset target area to the total number of pixels within the preset target area to obtain the effective event pixel coverage rate of the nth event set; if the effective event pixel coverage rate is within a second preset range, then the nth event set is a valid event set; if the effective event pixel coverage rate is not within the second preset range, then the nth event set is not a valid event set.

[0017] In some embodiments, determining whether the nth event set is a valid event set based on the event concentration of the nth event set includes: setting multiple statistical regions within the event plane; determining the number of events occurring in each statistical region based on the pixel coordinates of each event in the nth event set; calculating the maximum value of the ratio of the number of events occurring in each statistical region to the total number of events in the nth event set to obtain the event concentration of the nth event set; if the event concentration is within a third preset range, then the nth event set is a valid event set; if the event concentration is not within the third preset range, then the nth event set is not a valid event set.

[0018] In some embodiments, determining whether the nth event set is a valid event set based on the positive-to-negative polarity ratio of the nth event set includes: calculating the ratio of the number of positive events to the number of negative events in the nth event set to obtain the positive-to-negative polarity ratio of the nth event set; if the positive-to-negative polarity ratio is within a fourth preset range, then the nth event set is a valid event set; if the positive-to-negative polarity ratio is not within the fourth preset range, then the nth event set is not a valid event set.

[0019] In some embodiments, processing the nth event set to obtain the nth voxel tensor further includes: dividing the nth time window into multiple time sub-windows; dividing the nth event set into multiple primary event subsets based on the multiple time sub-windows; dividing each primary event subset into two secondary event subsets based on the polarity of each event; accumulating all events in each secondary event subset to obtain the accumulation result of each secondary event subset; and using the accumulation result of all secondary event subsets to construct the nth voxel tensor.

[0020] In some embodiments, constructing the nth voxel tensor using the cumulative results of all subsets of secondary events includes: performing monotonically compressed processing on the cumulative results of each subset of secondary events to obtain the compressed cumulative results of each subset of secondary events; and constructing the nth voxel tensor using the compressed cumulative results of all subsets of secondary events.

[0021] In some embodiments, processing the nth frame scene image to obtain the gaze coordinates of the gaze target in the nth frame scene image includes: performing target detection on the nth frame scene image to obtain the bounding box of the gaze target; and determining the gaze coordinates based on the bounding box of the gaze target.

[0022] In some embodiments, determining the gaze coordinates based on the bounding box of the gaze target includes: calculating the coordinates of the center of the bounding box of the gaze target to obtain the gaze coordinates.

[0023] In some embodiments, determining the gaze coordinates based on the bounding box of the gaze target further includes: calculating the coordinates of the center of the bounding box of the gaze target to obtain the gaze coordinates to be processed in the nth frame scene image; and mapping the gaze coordinates to be processed to the display screen coordinate system using a pre-calibrated homography matrix to obtain the gaze coordinates.

[0024] In some embodiments, training an eye-tracking model based on N gaze coordinate prediction results and N gaze coordinate labels includes: selecting multiple valid labels from the N gaze coordinate labels; and training the eye-tracking model using the multiple valid labels and the gaze coordinate prediction results corresponding to each valid label.

[0025] In some embodiments, selecting multiple valid labels from N gaze coordinate labels includes: selecting a predetermined number of consecutive gaze coordinate labels from the N gaze coordinate labels; determining the variation range of the predetermined number of gaze coordinate labels; and if the variation range is within a fifth preset range, then the predetermined number of gaze coordinate labels are taken as valid labels.

[0026] In some embodiments, determining the nth time window based on the nth frame scene image includes: calculating the difference between the timestamp of the nth frame scene image and a first duration to obtain the start time of the nth time window; and calculating the sum of the timestamp of the nth frame scene image and a second duration to obtain the end time of the nth time window.

[0027] In some embodiments, receiving multiple events collected by the event camera at a second preset frequency includes: reading event stream data sent by the event camera in a block-based reading manner; retrieving multiple events to be processed from the event stream data according to a preset frame header and a preset length; and verifying each event to be processed in order to extract the events that have been successfully verified from the multiple events to be processed, thereby obtaining multiple events.

[0028] According to a second aspect of the present disclosure, an eye-tracking model training apparatus is provided, comprising: a memory; and a processor coupled to the memory, the processor being configured to execute instructions stored in the memory to implement the eye-tracking model training method as described in any of the above embodiments.

[0029] According to a third aspect of the present disclosure, an eye-tracking method is provided, comprising: receiving multiple events collected by an event camera at a preset frequency; processing the multiple events to obtain a voxel tensor; and processing the voxel tensor using an eye-tracking model to obtain a gaze coordinate prediction result, wherein the eye-tracking model is trained using the eye-tracking model training method involved in any of the above embodiments.

[0030] According to a fourth aspect of the present disclosure, an eye-tracking control device is provided, comprising: a memory; and a processor coupled to the memory, the processor being configured to execute instructions stored in the memory to implement an eye-tracking method as described in any of the foregoing embodiments.

[0031] According to a fifth aspect of the present disclosure, an eye-tracking system is provided, comprising: an eye-tracking control device according to any of the above embodiments; an event camera configured to acquire multiple events at a preset frequency; and an event transmission device configured to send the multiple events acquired by the event camera to the eye-tracking control device.

[0032] In some embodiments, the event transmission device includes an event encapsulation module, a circular buffer module, a linear peeping module, a burst transmission module, and a callback module. The event encapsulation module is configured to encapsulate each of a plurality of events into an event frame; the circular buffer module is configured to write each event frame into a circular buffer area using a circular buffer mechanism; the linear peeping module is configured to read event stream data from the circular buffer area according to a read pointer of the circular buffer area; the burst transmission module is configured to send the event stream data in blocks to the eye-tracking control device using a burst transmission mechanism; and the callback module is configured to advance the read pointer after the event stream data transmission is completed.

[0033] According to a sixth aspect of the present disclosure, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores computer instructions that, when executed by a processor, implement the method as described in any of the above embodiments.

[0034] According to a seventh aspect of the present disclosure, a computer program product is provided, including computer instructions, wherein when executed by a processor, the computer instructions implement the method as described in any of the above embodiments.

[0035] Other features and advantages of this disclosure will become clear from the following detailed description of exemplary embodiments with reference to the accompanying drawings. Attached Figure Description

[0036] To more clearly illustrate the technical solutions in the embodiments of this disclosure or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this disclosure. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0037] Figure 1 This is a schematic diagram of the structure of an eye-tracking system according to an embodiment of the present disclosure.

[0038] Figure 2 This is a schematic diagram of the structure of an event transmission device according to an embodiment of the present disclosure.

[0039] Figure 3 This is a schematic diagram of the structure of an event frame according to an embodiment of the present disclosure.

[0040] Figure 4 This is a flowchart illustrating an eye-tracking model training method according to an embodiment of the present disclosure.

[0041] Figure 5 This is a flowchart illustrating an event stream data processing method according to an embodiment of the present disclosure.

[0042] Figure 6This is a schematic diagram of the voxelization encoding process according to an embodiment of the present disclosure.

[0043] Figure 7 This is a schematic diagram of the gaze coordinate prediction process according to an embodiment of the present disclosure.

[0044] Figure 8 This is a schematic flowchart of an eye-tracking method according to an embodiment of the present disclosure.

[0045] Figure 9 This is a schematic diagram of the structure of an electronic device according to an embodiment of the present disclosure.

[0046] Figure 10 This is a schematic diagram of the structure of an eye-tracking system according to another embodiment of the present disclosure. Detailed Implementation

[0047] The technical solutions of the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this disclosure, and not all embodiments. The following description of at least one exemplary embodiment is merely illustrative and is in no way intended to limit this disclosure or its application or use. All other embodiments obtained by those skilled in the art based on the embodiments of this disclosure without creative effort are within the scope of protection of this disclosure.

[0048] Unless otherwise specifically stated, the relative arrangement, numerical expressions, and values ​​of the components and steps set forth in these embodiments do not limit the scope of this disclosure.

[0049] At the same time, it should be understood that, for ease of description, the dimensions of the various parts shown in the accompanying drawings are not drawn according to actual scale.

[0050] Techniques, methods, and equipment known to those skilled in the art may not be discussed in detail, but where appropriate, such techniques, methods, and equipment should be considered part of the specification.

[0051] In all examples shown and discussed herein, any specific values ​​should be interpreted as merely exemplary and not as limitations. Therefore, other examples of exemplary embodiments may have different values.

[0052] It should be noted that similar labels and letters in the following figures indicate similar items; therefore, once an item is defined in one figure, it does not need to be discussed further in subsequent figures.

[0053] In related technologies, eye-tracking technology based on event cameras typically includes steps such as event data transmission, data representation extraction, and label generation.

[0054] In the event data transmission step, communication device protocol standards are typically followed, and a transmission link is constructed using a universal serial bus interface to transmit the high-speed asynchronous event stream generated by the event camera to the back-end computing platform in real time. This process involves the underlying firmware encapsulating continuously triggered discrete events into data packets with specific frame structures, and using techniques such as direct memory access to move the data to a transmission buffer to maintain the continuity of the data stream.

[0055] In the data representation and extraction step, since the event data generated by the event camera is in the form of unstructured sparse point cloud, it needs to be transformed into a form suitable for convolutional neural network processing. For example, event data within a very short time window can be stacked and discretized in the spatiotemporal dimension to construct a voxel grid or cumulative map containing time information. The model can then extract the velocity and direction features of eye movement from the voxel grid or cumulative map.

[0056] In the label generation step, to train the eye-tracking model, labels corresponding to the event data need to be generated. For example, the gaze coordinates output by a traditional eye-tracking device can be used as labels. However, the sampling frequency of a traditional eye-tracking device is much lower than that of an event camera, resulting in a data asynchrony between the gaze coordinate labels and the event data. To solve this problem, precise alignment of the clocks of different sensors can be achieved through hardware trigger signals, and mathematical methods such as linear interpolation can be used to calculate the labels corresponding to the high-frequency event data based on the low-frequency gaze coordinates output by the traditional eye-tracking device.

[0057] Based on the above analysis, the relevant event camera-based eye-tracking technology has the following drawbacks.

[0058] (1) Poor stability of event data transmission. The universal serial bus transmission mechanism suffers from throughput bottlenecks and uncontrollable jitter in high-speed scenarios. Existing technologies lack efficient flow control and double buffering mechanisms for bursty data streams. When the human eye performs rapid scanning movements, the data rate generated by the event camera will surge instantly. The underlying driver is prone to frequent busy states due to its inability to process large amounts of event data in a timely manner, and the central processing unit consumes too much resources in polling and data copying. This will lead to severe jitter in data transmission. In full-speed mode with limited bandwidth, it will also lead to an excessively high proportion of short packets or even silent packet loss, directly disrupting the continuity of event data and introducing noise that is difficult to repair at the algorithm layer.

[0059] (2) Lack of data verification mechanism. Existing technologies typically use compact binary stream formats to transmit events, lacking macroblock or frame structures similar to video streams. During data transmission, the lack of frame-level verification mechanisms and self-healing capabilities makes the data parsing link extremely fragile. In complex electromagnetic interference environments or high-load transmissions, once byte misalignment or a small amount of data loss occurs, the decoder at the receiving end cannot detect the error due to the lack of frame header synchronization markers and frame-level verification mechanisms, and will continue to parse subsequent data according to the erroneous offset. This avalanche effect can cause parsing to fall into a long-term error state, rendering the entire sample data invalid. The system lacks the ability to automatically resynchronize and has poor robustness.

[0060] (3) The label generation strategy based on linear interpolation introduces significant noise. The saccade motion of the human eye is an explosive ballistic motion with extremely high acceleration, and its trajectory exhibits a non-linear shape on the time axis. Existing technologies use simple linear interpolation to fill the time gap between low-frequency traditional eye-tracking devices and high-frequency event streams, which leads to a huge phase deviation and spatial error between the generated gaze point label and the actual eye position. This noisy labeling with artifacts forces the model to fit incorrect motion patterns, making model training difficult to converge and limiting the upper limit of the model's accuracy on high-frequency eye regression tasks.

[0061] (4) Data representation methods in related technologies neglect key spatiotemporal information and are computationally inefficient. Simple cumulative graphs eliminate the temporal and polarity information of events through integration, degrading high-dimensional dynamics into static textures, making it difficult for the model to capture fine-grained features such as velocity and acceleration. While voxel meshes retain the time dimension, they generate a large amount of sparse zero-value data, resulting in excessive computation and memory usage, which is not conducive to efficient real-time inference on resource-constrained edge embedded devices.

[0062] Accordingly, this disclosure provides an eye-tracking model training method that can improve the prediction accuracy of gaze point coordinates.

[0063] Figure 1 This is a schematic diagram of the structure of an eye-tracking system according to an embodiment of this disclosure. Figure 1 As shown, the eye-tracking system includes an event acquisition device 11, an event transmission device 12, a scene image acquisition device 13, a sample processing device 14, and a deep learning device 15.

[0064] Event acquisition device 11 is configured to acquire event stream data. The event stream data captures the user's eye movements. Event acquisition device 11 may include an event camera. For example, event acquisition device 11 includes a GENX320 event camera and its lens assembly.

[0065] The event transmission device 12 is configured to transmit the event stream data acquired by the event acquisition device 11 to the sample processing device 14. For example, the event transmission device 12 includes a microcontroller and a USB (Universal Serial Bus) communication interface. The microcontroller can be an MCU (Microcontroller Unit) platform with USB Device Full-Speed ​​peripherals and sufficient RAM (Random Access Memory). For example, the microcontroller can be an STM32F746G-DISCO development board.

[0066] The microcontroller in event transmission device 12 and the event camera in event acquisition device 11 can be electrically connected via interfaces such as SPI (Serial Peripheral Interface), I2C (Inter-Integrated Circuit), or GPIO (General-Purpose Input / Output). The microcontroller can transmit the event stream data output from the event camera to the sample processing device 14, or it can decode the output signal of the event camera to obtain the event stream data and transmit it to the sample processing device 14. The microcontroller in event transmission device 12 and the sample processing device 14 can be connected via USB Communication Device Class (CDC) to form a stable event stream data transmission link. Alternatively, connections can be established via USB High-Speed, Windows USB drivers, Bulk Transfer, Human Interface Device (HID), Peripheral Component Interconnect Express (PCIe), Ethernet User Datagram Protocol (UDP), Ethernet Transmission Control Protocol (TCP), or wireless links.

[0067] For example, event transmission device 12 such as Figure 2 As shown. To achieve high throughput, low packet loss, and recoverable synchronization of event stream data transmission, the event transmission device 12 can employ a ring buffer and burst transmission mechanism to transmit event stream data.

[0068] Figure 2 This is a schematic diagram of the structure of an event transmission device according to an embodiment of this disclosure. Figure 2 As shown, the event transmission device 12 includes an event encapsulation module 21, a ring buffer module 22, a linear peeping module 23, a burst sending module 24, and a callback module 25.

[0069] The event encapsulation module 21 is configured to encapsulate each event captured by the event camera into a binary event frame of a preset length according to a preset frame structure.

[0070] Figure 3 This is a schematic diagram of the structure of an event frame according to an embodiment of this disclosure. Figure 3 As shown, each event frame is 12 bytes long, and each event frame includes a frame header field 31, a first coordinate field 32, a second coordinate field 33, a timestamp field 34, a polarity field 35, and a check field 36.

[0071] For example, frame header field 31 is 0x55 and 0xAA. First coordinate field 32 is the x-coordinate of the event, and second coordinate field 33 is the y-coordinate of the event, both in 16-bit little-endian format and 2 bytes long. Timestamp field 34 is the MCU timestamp t_us, in microseconds or other preset units, in 32-bit little-endian format and 4 bytes long. Polarity field 35 is the event polarity pol, for example, pol = 1 indicates positive polarity, pol = 0 indicates negative polarity, and is 1 byte long. Checksum field 36 is used to check the event frame, for example, performing XOR8 check (8-bit XOR check) or CRC8 check (8-bit Cyclic Redundancy Check) on the payload bytes. Event frames can use... Figure 3 The structure shown can also be other structures.

[0072] The circular buffer module 22 is configured to use a circular buffer mechanism to write each event frame to the circular buffer area. The size of the circular buffer area is a power of 2 bytes (e.g., 128KB). The valid data range is identified by read and write pointers. The write end is located in the task state or acquisition thread, and the read end is located in the USB transmission completion callback state, forming a single producer and single consumer model.

[0073] The linear peep module 23 is used to acquire a continuous segment of data without crossing the tail of the annular buffer.

[0074] The burst transmission module 24 aligns the continuous data to the common packet length of USB full-speed bulk transmission (USB FS Bulk) (e.g., 64 bytes) and submits it in segments. In one scheduling, it continuously submits a preset number of TransmitPacket calls (e.g., 8 to 16 times) to form burst packets, thereby reducing the frequency of USB device busy (USBD_BUSY) and the proportion of short packets.

[0075] In the CDC_TransmitCplt callback, the callback module 25 advances the read pointer with the actual transmission length and triggers burst transmission scheduling again, so that the data to be sent in the ring buffer can be continuously consumed, thereby maintaining high and stable throughput even when the system load fluctuates.

[0076] It should be noted that event stream data can be sent using a circular buffer, as well as double buffering, segmented linked list buffering, DMA (Direct Memory Access) descriptor chaining, or RTOS (Real-Time Operating System) message queues. 64-byte aligned burst sending is possible, along with adaptive packet length, timed flushing of short packets, host-based flow control, or dynamic burst depth adjustment based on queue watermarks. The full buffer strategy can be replaced with configurable strategies such as blocking and waiting, dropping, downsampling, or only dropping low-priority events.

[0077] The event transmission device described in the above embodiments can ensure that event stream data is transmitted to the sample processing device in a continuous, sequential, and low-packet-loss manner, providing highly complete input for subsequent sample construction, model training, and inference.

[0078] Scene image acquisition device 13 is configured to acquire scene images. The scene images include the user's gaze target so that sample processing device 14 can use the scene images to generate gaze coordinate labels. Scene image acquisition device 13 can be a camera. For example, scene image acquisition device 13 can include a USB / UVC (USB Video Class) scene camera. Scene image acquisition device 13 is electrically connected to sample processing device 14.

[0079] The sample processing device 14 is configured to generate gaze coordinate labels using scene images, temporally align event stream data and gaze coordinate labels, and construct training samples. For example, the gaze coordinate labels are the pixel coordinates of the center of the gaze target in the scene image.

[0080] In some embodiments, the sample processing device 14 includes an event reading module, a verification module, a timestamp alignment module, a caching module, a tag generation module, a timing alignment module, a voxelization encoding module, and a sample storage module.

[0081] The event reading module is used to read event stream data from the USB CDC virtual serial port in a block-by-block manner.

[0082] The verification module is used to search the event stream data according to a preset frame header and preset length to obtain multiple events to be processed. Each event to be processed is verified in order to extract the successfully verified events from the multiple events to obtain multiple events. For the retrieved events to be processed, if the verification is successful, they are parsed as valid events; if the verification fails, the search continues to slide in the event stream data to restore synchronization, thereby avoiding data misalignment that would cause a large number of false events to enter the subsequent process.

[0083] The timestamp alignment module is used to append a monotonic clock timestamp (e.g., host_ns) to each parsed event, while simultaneously retaining the timestamp field in the event frame (e.g., the MCU timestamp t_us) for subsequent dual-clock domain consistency checks or drift compensation. For example, the monotonic clock timestamp host_ns is used to record the time when the data packet arrives at the sample processing device 14, and the MCU timestamp t_us is used to record the time when the event frame is generated.

[0084] For example, a linear mapping model is established with the monotonic clock timestamp `host_ns` as the dependent variable and the MCU timestamp `t_us` as the independent variable. The linear mapping model can obtain the mapped timestamp based on the MCU timestamp `t_us`. The frequency deviation factor `a` is tracked and corrected in real time using the Exponential Moving Average (EMA) algorithm. This factor reflects the physical frequency difference between the two hardware crystal oscillators, thereby eliminating long-term cumulative time errors caused by temperature changes or crystal oscillator temperature drift. The system's quality is monitored in real time by calculating the residual distribution between the mapped timestamp and the actual arrival timestamp of the data packet (i.e., the monotonic clock timestamp `host_ns`). When the residual exceeds a preset confidence interval (threshold value), it is considered that the transmission link is congested or the clock is out of sync, triggering a system reset mechanism.

[0085] The caching module stores events in ascending order of the monotonic clock timestamp `host_ns` and supports window queries based on binary search for quick retrieval of the event set within any given time window. Each event includes a first coordinate, a second coordinate, a polarity, a first timestamp, and a second timestamp. For example, the first coordinate is the x-coordinate, the second coordinate is the y-coordinate, the polarity `pol` is either positive or negative, the first timestamp is the monotonic clock timestamp `host_ns`, and the second timestamp is the MCU timestamp `t_us`.

[0086] The caching module can also be used to prune stored event data according to a preset retention period or maximum cache size to ensure long-term system operation and improve system memory stability. For example, the preset retention period is 5 seconds.

[0087] The label generation module is used to generate gaze coordinate labels required for supervised learning using scene images.

[0088] The timing alignment module is used to align event stream data and gaze coordinate labels in a timely manner. The timing alignment module can call the window query interface of the caching module to apply quality gating to the event set based on the event set within the time window corresponding to the gaze coordinate label.

[0089] The voxelization encoding module is used to voxelize and encode the event set to obtain voxel tensors.

[0090] The sample storage module is used to construct training samples using voxel tensors and corresponding gaze coordinate labels. It also stores the voxel tensors, corresponding gaze coordinate labels, and metadata of the training samples. For example, compressed npz files are used to store voxel tensors and gaze coordinate labels, while JSON files are used to store metadata. The metadata includes at least one of the following: scene image size (scene_size), event camera resolution (eye_event_size), window start and end times, number of time bins, first duration, second duration, number of events, target center coordinates, whether homography mapping was performed, and the save time. The metadata may also include other data involved in the sample construction process. The sample storage module can generate a CSV-formatted index file to record the file path and key statistical indicators for each sample, facilitating subsequent training set partitioning and quality tracking. Key statistical indicators include at least one of the following: total number of events, time window start and end times, window duration, temporal deviation, target center coordinates, supervision label coordinates, number of event edge pixels, scene cropping image sharpness, overexposure ratio, underexposure ratio, and event space configuration parameters.

[0091] The deep learning device 15 is configured to train an eye-tracking model using training samples and to predict the user's gaze coordinates in real time using the eye-tracking model. The deep learning device 15 can be integrated with the sample processing device 14 on the same device. Alternatively, the deep learning device 15 can be a standalone GPU (Graphics Processing Unit) server, connected to the sample processing device 14 via a network, or obtain training samples constructed by the sample processing device 14 through offline data files to enable the training and inference process of the eye-tracking model.

[0092] In some embodiments, the deep learning device 15 includes a spatial feature extraction module, a temporal modeling module, and a regression module. The spatial feature extraction module is used to extract spatial structural features, the temporal modeling module is used to extract temporal features, and the regression module is used to map the temporal features to gaze coordinate prediction results.

[0093] Figure 4 This is a schematic flowchart of an eye-tracking model training method according to an embodiment of the present disclosure. In some embodiments, the following eye-tracking model training method is performed by an eye-tracking model training device, or by a sample processing device 14 and a deep learning device 15, including steps 41-48.

[0094] In step 41, multiple frames of scene images acquired by the scene camera at a first preset frequency are received.

[0095] In step 42, multiple events are received from the event camera at a second preset frequency. The second preset frequency is greater than the first preset frequency.

[0096] The user gazes at a target, and a scene camera captures scene images containing the gazed target at a first preset frequency. The gazed target can be a three-dimensional object or a two-dimensional object on the display screen. An event camera captures events at a second preset frequency to capture the user's eye movements. The coordinates in the event represent the coordinates of pixels whose brightness has changed, and the polarity of the event indicates whether the brightness has increased or decreased. The second preset frequency can be much higher than the first preset frequency. The event camera transmits the event stream data to an eye-tracking model training device, which processes the event stream data to obtain multiple events.

[0097] In some embodiments, the method for processing event stream data includes: reading event stream data sent by an event camera in a block-by-block manner; retrieving multiple events to be processed from the event stream data based on a preset frame header and a preset length; validating each event to be processed in order to extract successfully validated events from the multiple events to obtain multiple events.

[0098] Figure 5This is a flowchart illustrating an event stream data processing method according to an embodiment of this disclosure. Figure 5 As shown, the event stream data processing method includes steps 51-56.

[0099] In step 51, event stream data is received.

[0100] In step 52, a frame header is searched in the event stream data. For example, the frame header is 0x55, 0xAA.

[0101] In step 53, the frame header and the bytes following the frame header are read according to the preset length to obtain the event to be processed.

[0102] In step 54, the event to be processed is validated.

[0103] If the verification passes, proceed to step 55.

[0104] If the verification fails, proceed to step 56.

[0105] In step 55, the events to be processed are output. If necessary, step 52 can be repeated in subsequent event stream data to obtain multiple events from the event stream data.

[0106] In step 56, slide one byte backward in the event stream data and repeat step 52.

[0107] It should be noted that the verification method can use XOR, CRC8, CRC16, or CRC32. Sequence number and timestamp fields can also be added to the event frame to enhance packet loss detection and sequence recovery capabilities. The event frame structure can be replaced with a fixed-length frame containing a length field, a TLV (Tag-Length-Value) structure, or compressed encoding (such as coordinate differential or time differential) can be introduced to further reduce bandwidth requirements.

[0108] The event stream data parsing method described in the above embodiments can improve the accuracy of event stream data parsing.

[0109] In step 43, the nth frame scene image is processed to obtain the gaze coordinates of the user's gaze target in the nth frame scene image, which are used as the nth gaze coordinate label. N is the total number of frames in the scene image.

[0110] In some embodiments, object detection is performed on the nth frame of the scene image to obtain the bounding box of the gaze target. The gaze coordinates are then determined based on the bounding box of the gaze target. For example, the YOLO algorithm is used to perform object detection on the nth frame of the scene image.

[0111] In some embodiments, the coordinates of the center of the bounding box of the gaze target are calculated to obtain the gaze coordinates.

[0112] When the object being gazed upon is a three-dimensional object, the center coordinates of the bounding box of the object being gazed upon can be directly calculated and used as the gaze coordinates.

[0113] In some embodiments, the coordinates of the center of the bounding box of the gaze target are calculated to obtain the gaze coordinates to be processed in the nth frame of the scene image. Using a pre-calibrated homography matrix, the gaze coordinates to be processed are mapped to the display screen coordinate system to obtain the gaze coordinates.

[0114] For cases where the gaze target is a two-dimensional object on the display screen—that is, the scene image captures the display screen's view, and the gaze target is a two-dimensional object within that view—a pre-calibrated homography matrix is ​​used to map the center coordinates of the gaze target's bounding box to the display screen's coordinate system. This adapts to the application scenario where the actual gaze point is defined on the screen.

[0115] It should be noted that if multiple gaze targets are detected in the nth frame of the scene image, or if no gaze target is detected, the nth frame of the scene image is marked as an invalid scene image and skipped. Furthermore, stability can be determined for multiple consecutive gaze coordinate labels; stable labels are used as valid labels to construct training samples, while unstable labels are discarded.

[0116] In some embodiments, gaze coordinate labels in pixel coordinate form are saved. gaze coordinate labels in normalized coordinate form You can also choose to output gaze coordinate labels in pixel coordinates or normalized coordinates.

[0117] It should be noted that screen calibration mapping, eye-screen geometric mapping, external gaze point generator, attitude-assisted inference based on inertial measurement unit (IMU) or multi-camera triangulation processing can also be performed on scene images to obtain the corresponding gaze coordinate labels.

[0118] In step 44, the nth time window is determined based on the scene image of the nth frame.

[0119] In some embodiments, the difference between the timestamp of the nth frame scene image and the first duration is calculated to obtain the start time of the nth time window. The sum of the timestamp of the nth frame scene image and the second duration is calculated to obtain the end time of the nth time window.

[0120] For example, the timestamp host_ns of the scene image in the nth frame can be used as the determining moment of the nth gaze label. Calculate the timestamp host_ns of the scene image in frame n and the first duration. The difference is used to obtain the start time of the nth time window. The timestamp host_ns of the scene image in the nth frame is calculated and then compared with the second duration. The sum of the values ​​yields the end time of the nth time window, which is then the nth time window is... First duration Second duration It is a configurable millisecond-level parameter, for example, the first duration. It can be configured to 60ms, the second duration. It can be configured to 20ms.

[0121] It should be noted that the time window can be a fixed window, a sliding window, or an event-adaptive window (e.g., collecting a specified number of events before ending the window). A hybrid strategy can also be used to determine the time window to adapt to different event densities.

[0122] In step 45, events whose timestamps fall within the nth time window are extracted from multiple events to obtain the nth event set.

[0123] In step 46, the nth event set is processed to obtain the nth voxel tensor.

[0124] In some embodiments, it is determined whether the nth event set is a valid event set. If the nth event set is a valid event set, it is processed to obtain the nth voxel tensor. If the nth event set is not a valid event set, the nth event set and the nth gaze coordinate label are deleted.

[0125] It should be noted that quality gating can be applied to the event set to improve the quality of the voxel tensor, thereby improving the model training effect.

[0126] In some embodiments, it is determined whether the nth event set is a valid event set based on at least one of the total number of events in the nth event set, the effective event pixel coverage, the event concentration, and the positive and negative polarity ratio.

[0127] For example, the total number of events in an event set, the effective event pixel coverage, the event concentration, and the positive / negative polarity ratio can be used as quality gating indicators. Among these, the effective event pixel coverage, event concentration, and positive / negative polarity ratio are used to constrain the spatial distribution of events within the event set. Furthermore, the validity of the nth event set can be determined based on the target confidence, target velocity, target acceleration, event density stability, event polarity ratio stability, or quality scores and thresholds based on statistical characteristics.

[0128] In some embodiments, determining whether the nth event set is a valid event set based on the total number of events in the nth event set includes: determining whether the total number of events in the nth event set is within a first preset range; if the total number of events is within the first preset range, then the nth event set is a valid event set; if the total number of events is not within the first preset range, then the nth event set is not a valid event set.

[0129] For example, the first preset range is determined based on the minimum number of events (min_events) and the maximum number of events (max_events). If the total number of events in the event set is less than the minimum number of events (min_events), the event set is considered to contain insufficient information, which may lead to missing profiles; therefore, the event set is not a valid event set. If the total number of events in the event set is greater than the maximum number of events (max_events), the event set is considered to be oversaturated, which may lead to noise bursts; therefore, the event set is not a valid event set.

[0130] In some embodiments, determining whether the nth event set is a valid event set based on the effective event pixel coverage rate of the nth event set includes: determining the number of events occurring for each pixel within a preset target area based on the pixel coordinates of each event in the nth event set; identifying pixels within the preset target area whose number of events exceeds a threshold as active pixels; calculating the ratio of the number of active pixels within the preset target area to the total number of pixels within the preset target area to obtain the effective event pixel coverage rate of the nth event set; if the effective event pixel coverage rate is within a second preset range, then the nth event set is a valid event set; if the effective event pixel coverage rate is not within the second preset range, then the nth event set is not a valid event set.

[0131] For example, suppose the threshold for the number of events is... The total number of pixels in the preset target area is The number of active pixels in the preset target area is The effective event pixel coverage P is shown in the following formula (1).

[0132] (1)

[0133] Effective event pixel coverage is used to assess whether the spatial distribution of events is too sparse or too concentrated. The second preset range can be determined based on the minimum and maximum coverage.

[0134] In some embodiments, determining whether the nth event set is a valid event set based on the event concentration of the nth event set includes: setting multiple statistical regions within the event plane; determining the number of events occurring in each statistical region based on the pixel coordinates of each event in the nth event set; calculating the maximum value of the ratio of the number of events occurring in each statistical region to the total number of events in the nth event set to obtain the event concentration of the nth event set; if the event concentration is within a third preset range, then the nth event set is a valid event set; if the event concentration is not within the third preset range, then the nth event set is not a valid event set.

[0135] For example, suppose the total number of events in the nth event set is R statistical regions are set within the event plane, and the number of events occurring in the r-th statistical region is... , Then the event concentration of the nth event set As shown in the following formula (2).

[0136] (2)

[0137] Event concentration is used to assess whether events in an event set are excessively clustered in a few local areas. A higher event concentration indicates that events are more concentrated in a local area. The third preset range can be determined based on a preset concentration threshold. When the event concentration of the nth event set exceeds the preset concentration threshold, it can be considered that the nth event set contains local noise bursts, reflective interference, or abnormal clustering, which is detrimental to generating stable and effective voxel tensors; therefore, the nth event set is not a valid event set.

[0138] In some embodiments, determining whether the nth event set is a valid event set based on the positive-to-negative polarity ratio of the nth event set includes: calculating the ratio of the number of positive events to the number of negative events in the nth event set to obtain the positive-to-negative polarity ratio of the nth event set; if the positive-to-negative polarity ratio is within a fourth preset range, then the nth event set is a valid event set; if the positive-to-negative polarity ratio is not within the fourth preset range, then the nth event set is not a valid event set.

[0139] For example, a positive event indicates an increase in the brightness of the corresponding pixel. A negative event indicates a decrease in the brightness of the corresponding pixel. Positive polarity can be represented by +1 and negative polarity by -1, or positive polarity by 1 and negative polarity by 0. The fourth preset range can be determined based on the minimum and maximum scale values.

[0140] The method described in the above embodiments aligns the event set with the gaze coordinate labels in time, thereby ensuring that the generated voxel tensor is also aligned with the gaze coordinate labels in time. Quality gating of the event set can improve the quality of the voxel tensor, solve problems such as voxel tensor blurring and unclear contours, and thus improve the training effect of the model.

[0141] In some embodiments, the method for voxelizing and encoding an event set to generate a voxel tensor includes the following steps S11-S15.

[0142] In step S11, the nth time window is divided into multiple time sub-windows.

[0143] For example, suppose the nth time window is It is divided into 5 time sub-windows , , , and ,in A time sub-window can also be called a time bin.

[0144] In step S12, based on multiple time sub-windows, the nth event set is divided into multiple primary event subsets.

[0145] For example, based on the event timestamp host_ns or timestamp t_us and the above 5 time sub-windows, the nth event set is divided into 5 primary event subsets, thereby realizing time binning of the event set.

[0146] In step S13, each primary event subset is divided into two secondary event subsets based on the polarity of each event.

[0147] For example, for each primary event subset, it is divided into two secondary event subsets based on whether the event is positive or negative. Dividing the above 5 primary event subsets results in a total of 10 secondary event subsets, each corresponding to one channel of the voxel tensor.

[0148] In step S14, all events in each sub-event subset are accumulated to obtain the accumulated result of each sub-event subset.

[0149] For example, suppose the resolution of the event camera is Then the cumulative result of each subset of secondary events is The matrix represents the event camera's pixel coordinates. Each element in the matrix corresponds to a pixel of the event camera, and the initial value of each element can be 0. For each subset of secondary events, each event increments the value of the matrix element corresponding to the event's pixel coordinates by one, thus obtaining the cumulative result for that subset of secondary events. The event camera's resolution... For example is .

[0150] In step S15, the cumulative result of all subsets of secondary events is used to construct the nth voxel tensor.

[0151] In some embodiments, the cumulative result of each subset of secondary events is monotonically compressed to obtain the compressed cumulative result of each subset of secondary events. The compressed cumulative results of all subsets of secondary events are used to construct the nth voxel tensor.

[0152] Monotonic compression can be logarithmic compression. For example, the cumulative result for each subset of secondary events is... For a matrix, where each element has a value of count, after performing logarithmic compression, the values ​​of the elements will be... As shown in the following formula (3).

[0153] (3)

[0154] By performing monotonic compression on the accumulated results of a subset of secondary events, the dynamic range can be controlled, thereby improving the stability of model training.

[0155] For example, suppose the size of the accumulated result of the subset of secondary events is If the number of time bins is B, meaning the number of primary event subsets is B and the number of secondary event subsets is 2B, then the size of the resulting voxel tensor is... The voxel tensor has 2B channels. The voxel tensor can be rearranged into a sequential input for temporal networks to learn. The size of the rearranged voxel tensor X is... .

[0156] Figure 6 This is a schematic diagram illustrating the voxelization encoding process of one embodiment of this disclosure. Figure 6 As shown, the nth time window is divided into B timeboxes, including timebox bin 1 to timebox bin B. Based on these B timeboxes, the nth event set 61 is divided into multiple primary event subsets 62. Based on event polarity, the multiple primary event subsets 62 are further divided into multiple secondary event subsets, including multiple positive polarity secondary event subsets 63 and multiple negative polarity secondary event subsets 64. The cumulative results of all secondary event subsets constitute a voxel tensor 65.

[0157] By using the methods described in the above embodiments, the event set is voxelized and encoded, preserving three key types of information—spatial contour, temporal evolution, and polarity direction—without increasing transmission bandwidth. This enables the model to distinguish changes in eye movement direction, polarity changes caused by eyelid movement, and differences in event distribution across different time periods, thereby improving the accuracy and robustness of fixation point regression.

[0158] It's worth noting that event sets can be used to construct a time surface, or event frame stacking, point cloud projection, or learned encoding (e.g., using a small network to process events to obtain event embedding vectors and then aggregating these vectors) to build the model input. Multi-scale time bins and non-uniform time bins (e.g., finer time bins are formed as the distance from the label's timestamp decreases) can also be used for time binning. When constructing the model input, temporal weight decay and polarity weights can also be introduced.

[0159] In step 47, the nth voxel tensor is processed using an eye-tracking model to obtain the prediction result of the nth gaze coordinate.

[0160] In some embodiments, the eye-tracking model includes a spatial feature extraction module, a temporal modeling module, and a regression module.

[0161] In some embodiments, the processing of voxel tensors by the eye-tracking model includes the following steps S21-S23.

[0162] In step S21, the spatial feature extraction module is used to extract features from the nth voxel tensor to obtain the spatial structure features of the nth voxel tensor.

[0163] In some embodiments, the spatial feature extraction module includes a multi-layer convolutional neural network.

[0164] The spatial feature extraction module is used to learn morphological information such as eye edges, pupil / iris contours, eyelid boundaries, and reflective areas.

[0165] In step S22, the temporal modeling module is used to extract features from the spatial structure features to obtain the temporal features of the nth voxel tensor.

[0166] In some embodiments, the timing modeling module includes a short-term timing modeling module and a global timing modeling module. The short-term timing modeling module includes multiple gated recurrent units (GRUs). The global timing modeling module includes a Transformer encoder.

[0167] Gated recurrent units can model short-term dynamics and suppress random noise. Transformer encoders can be equipped with relative position bias. Transformer encoders can capture more complex cross-timebox dependencies and bidirectional dynamic trends through relative position attention, thereby enhancing the representation of temporal patterns such as saccades, microsaccades, and blink perturbations.

[0168] In step S23, the temporal features are mapped using the regression module to obtain the prediction result of the nth gaze coordinate.

[0169] Figure 7 This is a schematic diagram illustrating the gaze coordinate prediction process according to an embodiment of this disclosure. Figure 7 As shown, the method for processing voxel tensors using an eye-tracking model to obtain gaze coordinate prediction results includes steps 71-75.

[0170] In step 71, input the voxel tensor. For example, input the nth voxel tensor according to the time dimension. Divide into B slices Each slice corresponds to a bipolar event frame in a timebox.

[0171] In step 72, spatial features are extracted. The spatial feature extraction module includes a convolutional layer 721, a normalization layer 722, an activation layer 723, a pooling layer 724, a convolutional layer 725, a normalization layer 726, an activation layer 727, and a pooling layer 728. For example, convolutional layers 721 and 725 may include a... The convolution kernels, normalization layers 722 and 726 can use batch normalization, activation layers 723 and 727 can use the ReLU function as the activation function, and pooling layer 724 can use... Max pooling is performed on the window, and global average pooling can be used for pooling layer 728.

[0172] For example, using a multi-layer two-dimensional convolutional network in the spatial feature extraction module to extract multiple slices Each slice in the image is processed to extract its spatial structure features, resulting in... .

[0173] In step 73, short-term time series modeling is performed. The short-term time series modeling module includes GRU 731, GRU 732, GRU 733 to GRU 73M, a total of M GRUs. The short-term time series modeling module is used to model spatial structural features. The process is performed, and the hidden states at M time steps are output. . Initially hidden state, This is the hidden state at the first time step. For the hidden state at the second time step, similarly, This represents the hidden state at the Mth time step.

[0174] In step 74, global temporal modeling is performed. The global temporal modeling module includes a multi-head self-attention layer 741, a normalization layer 742, a feed-forward network 743, and a normalization layer 744. The multi-head self-attention layer 741 can employ a multi-head self-attention mechanism, while the normalization layers 742 and 744 can use a combination of residual connections and normalization (Add & Normal). The global temporal modeling module is used to process the hidden states at M time steps. Processing is performed to extract the temporal features of the voxel tensor.

[0175] In step 75, gaze coordinate regression is performed. The regression module includes a fully connected layer 751 and a linear output layer 752. The linear output layer 752 is used to output two-dimensional gaze coordinates. The regression module is used to map temporal features to obtain the nth gaze coordinate prediction result.

[0176] It should be noted that the spatial feature extraction module can also include a lightweight convolutional neural network (CNN) or separable convolutions, and can also incorporate a multi-scale feature pyramid. The short-term temporal modeling module can also include a Long Short-Term Memory (LSTM), a Temporal Convolutional Network (TCN), or a Convolutional Gated Recurrent Unit (ConvGRU). The global temporal modeling module can also include a convolutionally enhanced Transformer (Conformer), a Perceiver, or a linear attention structure.

[0177] In step 48, an eye-tracking model is trained based on the N gaze coordinate prediction results and N gaze coordinate labels.

[0178] In some embodiments, multiple valid labels are selected from N gaze coordinate labels. The eye-tracking model is trained using the multiple valid labels and the gaze coordinate prediction results corresponding to each valid label.

[0179] It's important to note here that, to reduce label jitter, label stability can be assessed. Stable labels are used as valid labels for model training, while unstable labels are discarded.

[0180] In some embodiments, a predetermined number of consecutive gaze coordinate labels are selected from N gaze coordinate labels. The variation range of the predetermined number of gaze coordinate labels is determined. If the variation range is within a fifth preset range, the predetermined number of gaze coordinate labels are considered valid labels.

[0181] For example, the predetermined number can be 2 to 5. Multiple consecutive gaze coordinate labels are selected, and the statistical dispersion of these labels relative to their average coordinates is calculated as the variation range of the gaze coordinate labels. The fifth preset range can be determined based on the pixel threshold `stable_pixel_th`. If the variation range is less than the pixel threshold, the multiple gaze coordinate labels are considered stable and are thus considered valid labels.

[0182] It should be noted that the stability of the labels can be determined after generating labels from scene images, or it can be determined before model training.

[0183] In some embodiments, a loss value is determined based on the prediction result of the nth gaze coordinate and the label of the nth gaze coordinate. The eye-tracking model is then trained using the loss value.

[0184] In some embodiments, the nth gaze coordinate prediction result includes M gaze coordinate prediction values, where the mth gaze coordinate prediction value is the prediction value of the eye-tracking model at the mth time step. M represents the total number of time steps.

[0185] It should be noted that the gaze coordinate prediction results output by the eye-tracking model can include predictions from M time steps, which can be used to calculate the loss value during the model training phase; or it can include only the prediction value from the last time step, so as to output accurate gaze coordinates during the real-time inference phase.

[0186] In some embodiments, the method for determining the loss value includes the following steps S31-S33.

[0187] In step S31, the first sub-loss value is determined based on the predicted value of the Mth gaze coordinate in the prediction result of the nth gaze coordinate and the label of the nth gaze coordinate.

[0188] In step S32, the second sub-loss value is determined based on all M predicted gaze coordinates and the nth gaze coordinate label in the nth gaze coordinate prediction result.

[0189] In step S33, the loss value is determined using the first sub-loss value and the second sub-loss value.

[0190] In some embodiments, a weighted sum of the first sub-loss value and the second sub-loss value is calculated to obtain the loss value.

[0191] It should be noted that by using temporal consistency constraints to calculate the loss value, for the sequence output, the first sub-loss value is calculated using the output of the last time step and the label, and the second sub-loss value is calculated using the output of all time steps and the label. Then, the first and second sub-loss values ​​are weighted. This can solve the problems of insufficient intermediate temporal learning caused by supervising the output of the last time step only, or the convergence instability caused by strong constraints on the whole sequence. This allows the model to still output stable gaze coordinate prediction results under the conditions of sparse events, fluctuations, or local noise perturbations.

[0192] It should also be noted that sequence consistency regularization (e.g., temporal smoothing constraints or adjacent time-bin output consistency constraints) can be used to calculate the loss value. Eye-tracking models can also be used to perform multi-task learning, such as simultaneously predicting gaze coordinates and eye movement states, such as saccades or blinks. The uncertainty of the eye-tracking model's output can also be modeled, for example, by modeling the mean and variance of the output to evaluate the stability of the model's predictions. Furthermore, samples with higher prediction difficulty can be identified, and sample quality weights can be introduced to improve the model's robustness. The output of an eye-tracking model can also be a gaze vector, gaze region classification, or gaze point trajectory prediction.

[0193] The eye-tracking model training method described in the above embodiments can improve the prediction accuracy of gaze point coordinates.

[0194] Figure 8 This is a schematic flowchart of an eye-tracking method according to an embodiment of the present disclosure. In some embodiments, the following eye-tracking method is executed by an eye-tracking control device, or by a sample processing device 14 and a deep learning device 15, including steps 81-83.

[0195] In step 81, multiple events collected by the event camera at a preset frequency are received.

[0196] For example, a specified number of events collected in real time, or multiple events within a specified time period, can be used as the set of events collected at the current moment.

[0197] In step 82, multiple events are processed to obtain voxel tensors.

[0198] For example, the set of events collected at the current moment can be voxelized and encoded to obtain a voxel tensor.

[0199] In step 83, the voxel tensor is processed using an eye-tracking model to obtain the gaze coordinate prediction result. The eye-tracking model utilizes... Figure 4 The eye-tracking model is trained using the eye-tracking model training method described in any of the embodiments.

[0200] It should be noted that during the real-time inference phase, the gaze coordinate prediction results output by the eye-tracking model can include only the prediction values ​​of the last time step in order to output accurate gaze coordinates.

[0201] The eye-tracking method described in the above embodiments can improve the prediction accuracy of gaze coordinates and ensure high-frequency gaze coordinate output.

[0202] Figure 9 This is a schematic diagram of the structure of an electronic device according to an embodiment of this disclosure. Figure 9 As shown, the electronic device 90 includes a memory 91, a processor 92, and a bus 93 that connects different system components.

[0203] The memory 91 may include, for example, system memory, non-volatile storage media, etc. System memory may store, for example, an operating system, application programs, a boot loader, and other programs. System memory may include volatile storage media, such as random access memory (RAM) and / or cache memory. Non-volatile storage media may store, for example, instructions for at least one eye-tracking model training method or a corresponding embodiment of an eye-tracking method being executed. Non-volatile storage media include, but are not limited to, disk storage, optical storage, flash memory, etc.

[0204] The processor 92 can be implemented using a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete hardware components such as discrete gates or transistors. Accordingly, the method in any of the above embodiments can be implemented by instructions in the central processing unit (CPU) running memory to execute the corresponding steps, or by dedicated circuitry to execute the corresponding steps.

[0205] For example, processor 92 is configured for memory-based instruction execution implementation such as Figures 4 to 7 The eye-tracking model training method involved in any of the embodiments. In this case, the electronic device 90 can serve as an eye-tracking model training device.

[0206] For example, processor 92 is configured for memory-based instruction execution implementation such as Figure 8 The eye-tracking method involved in any of the embodiments. In this case, the electronic device 90 can serve as an eye-tracking control device.

[0207] Bus 93 can use any of the various bus architectures. For example, bus architectures include, but are not limited to, the Industry Standard Architecture (ISA) bus, the Micro Channel Architecture (MCA) bus, and the Peripheral Component Interconnect (PCI) bus.

[0208] The interfaces 94, 95, and 96 of electronic device 90, as well as memory 91 and processor 92, can be connected via bus 93. Input / output interface 94 provides a connection interface for input / output devices such as monitors, mice, and keyboards. Network interface 95 provides a connection interface for various networked devices. Storage interface 96 provides a connection interface for external storage devices such as floppy disks, USB flash drives, and SD cards.

[0209] Figure 10 This is a schematic diagram of the structure of an eye-tracking system according to another embodiment of this disclosure. Figure 10 As shown, the eye-tracking system 100 includes an eye-tracking control device 101, an event camera 102, and an event transmission device 103.

[0210] The eye-tracking control device 101 can be as follows: Figure 9 The eye-tracking control device shown.

[0211] The event camera 102 is configured to acquire multiple events at a preset frequency.

[0212] The event transmission device 103 is configured to send multiple events captured by the event camera 102 to the eye-tracking control device 101.

[0213] In some embodiments, the event transmission device 103 includes an event encapsulation module, a ring buffer module, a linear peeping module, a burst transmission module, and a callback module. The event transmission device 103 may be as follows: Figure 2 The event transmission device shown.

[0214] The event encapsulation module is configured to encapsulate each of a plurality of events into an event frame.

[0215] The circular buffer module is configured to use a circular buffer mechanism to write each event frame to the circular buffer area.

[0216] The linear peep module is configured to read event stream data from the circular buffer region based on the read pointer of the circular buffer region.

[0217] The burst transmission module is configured to use a burst transmission mechanism to send event stream data in blocks to the eye-tracking control device 101.

[0218] The callback module is configured to advance the read pointer after the event stream data has been sent.

[0219] The eye-tracking system described in the above embodiments, combined with a circular buffer and burst transmission mechanism, can reduce the busy frequency of USB devices and reduce packet loss when users perform rapid saccades and generate a large amount of event stream data.

[0220] Various aspects of this disclosure are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus, and computer program products according to embodiments of this disclosure. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations thereof, can be implemented by computer-readable program instructions.

[0221] These computer-readable program instructions are provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable device to produce a machine, such that execution of the instructions by the processor produces means for implementing the functions specified in one or more boxes of the flowchart and / or block diagram.

[0222] These computer-readable program instructions may also be stored in a computer-readable storage medium. These instructions cause a computer to work in a particular manner to produce an article of manufacture, including instructions that implement the functions specified in one or more boxes in a flowchart and / or block diagram.

[0223] This disclosure may take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects.

[0224] This disclosure also provides a computer-readable storage medium storing computer instructions that, when executed by a processor, implement... Figures 4 to 8 The method involved in any of the embodiments.

[0225] This disclosure also provides a computer program product, including computer instructions, wherein the computer instructions, when executed by a processor, implement as follows: Figures 4 to 8 The method involved in any of the embodiments.

[0226] In some embodiments, the functional units described above may be implemented as general-purpose processors, programmable logic controllers (PLCs), digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, or any suitable combination thereof for performing the functions described herein.

[0227] By implementing the methods described in the above embodiments, the following beneficial effects can be obtained.

[0228] (1) By using a ring buffer zero-copy, 64-byte alignment and burst transmission mechanism, the effective throughput and stability can be improved under the condition of full-speed communication of USB devices, the proportion of busy and short packets of USB devices can be reduced, event loss and delay jitter can be reduced, and the integrity of event sequence can be ensured from the source.

[0229] (2) Fixed-length frame header and XOR (or CRC) check mechanism enable the system to have fast resynchronization and error removal capabilities, improving the robustness of the transmission link.

[0230] (3) The event window truncation and stability gating sampling strategy based on label time makes the samples aligned with the labels in time and the quality controllable, reducing noise labeling and improving the convergence and accuracy limit of model training.

[0231] (4) The voxel encoding method of temporal binning and polarity separation preserves the key spatiotemporal structure of events, enabling deep learning networks to learn fine-grained spatial texture changes and temporal dynamic features related to eye movements, thereby improving the accuracy and robustness of fixation point regression.

[0232] (5) By combining spatial feature extraction and temporal modeling, the model can capture local short-term motion and global temporal dependence at the same time. Furthermore, a sequence loss combination strategy is introduced to jointly constrain the regression accuracy of the last time step and the consistency of the whole sequence, thereby significantly reducing prediction jitter and improving regression stability and noise resistance.

[0233] The methods or apparatuses involved in the above embodiments provided in this disclosure can be applied to fields such as Augmented Reality (AR) / Virtual Reality (VR) interactive gaze control, vehicle driver monitoring and distraction detection, medical rehabilitation training and visual function assessment, industrial human factors analysis, barrier-free interaction, smart home gaze control, and edge smart terminals for ultra-low latency interaction.

[0234] Those skilled in the art will understand that all or part of the steps of the above embodiments can be implemented by hardware or by a program instructing related hardware. The program can be stored in a computer-readable storage medium, such as a read-only memory, a disk, or an optical disk.

[0235] The description in this disclosure is provided for illustrative and descriptive purposes only and is not intended to be exhaustive or to limit the disclosure to its forms. Many modifications and variations will be apparent to those skilled in the art. The embodiments were chosen and described in order to better illustrate the principles and practical application of this disclosure and to enable those skilled in the art to understand this disclosure and to design various embodiments with various modifications suitable for a particular purpose.

Claims

1. A method for training an eye-tracking model, comprising: Receive multiple frames of scene images captured by the scene camera at a first preset frequency; Receive multiple events collected by the event camera at a second preset frequency, wherein the second preset frequency is greater than the first preset frequency; The nth frame scene image is processed to obtain the gaze coordinates of the user's gaze target in the nth frame scene image, which are used as the nth gaze coordinate label, where N is the total number of frames in the scene image; Based on the nth frame scene image, determine the nth time window; Extract the events whose timestamps fall within the nth time window from the multiple events to obtain the nth event set; The nth event set is processed to obtain the nth voxel tensor; The nth voxel tensor is processed using an eye-tracking model to obtain the prediction result of the nth gaze coordinate; The eye-tracking model is trained based on N gaze coordinate prediction results and N gaze coordinate labels.

2. The eye-tracking model training method according to claim 1, wherein, The eye-tracking model includes a spatial feature extraction module, a temporal modeling module, and a regression module. The process of using the eye-tracking model to process the nth voxel tensor to obtain the nth gaze coordinate prediction result includes: The spatial feature extraction module is used to extract features from the nth voxel tensor to obtain the spatial structure features of the nth voxel tensor. The temporal modeling module is used to extract features from the spatial structure features to obtain the temporal features of the nth voxel tensor. The regression module is used to map the temporal features to obtain the prediction result of the nth gaze coordinate.

3. The eye-tracking model training method according to claim 2, wherein, The spatial feature extraction module includes a multi-layer convolutional neural network; The timing modeling module includes a short-term timing modeling module and a global timing modeling module. The short-term timing modeling module includes multiple gated loop units, and the global timing modeling module includes a Transformer encoder.

4. The eye-tracking model training method according to claim 1, wherein, The step of training the eye-tracking model based on N gaze coordinate prediction results and N gaze coordinate labels includes: The loss value is determined based on the prediction result of the nth gaze coordinate and the label of the nth gaze coordinate; The eye-tracking model is trained using the loss value.

5. The eye-tracking model training method according to claim 4, wherein, The nth gaze coordinate prediction result includes M gaze coordinate prediction values, wherein the mth gaze coordinate prediction value is the prediction value of the eye-tracking model at the mth time step. M represents the total number of time steps. The determination of the loss value based on the prediction result of the nth gaze coordinate and the label of the nth gaze coordinate includes: Based on the predicted value of the Mth gaze coordinate in the predicted result of the nth gaze coordinate and the label of the nth gaze coordinate, determine the first sub-loss value; The second sub-loss value is determined based on all M predicted gaze coordinates in the nth gaze coordinate prediction result and the nth gaze coordinate label; The loss value is determined using the first sub-loss value and the second sub-loss value.

6. The eye-tracking model training method according to claim 5, wherein, Determining the loss value includes: The weighted sum of the first sub-loss value and the second sub-loss value is calculated to obtain the loss value.

7. The eye-tracking model training method according to claim 1, wherein, The process of processing the nth event set to obtain the nth voxel tensor includes: Determine whether the nth event set is a valid event set; If the nth event set is the valid event set, the nth event set is processed to obtain the nth voxel tensor; If the nth event set is not a valid event set, delete the nth event set and the nth gaze coordinate label.

8. The eye-tracking model training method according to claim 7, wherein, The determination of whether the nth event set is a valid event set includes: Whether the nth event set is a valid event set is determined based on at least one of the following: total number of events, effective event pixel coverage, event concentration, and positive / negative polarity ratio.

9. The eye-tracking model training method according to claim 8, wherein, Determining whether the nth event set is a valid event set based on the total number of events in the nth event set includes: Determine whether the total number of events in the nth event set is within a first preset range; If the total number of events is within the first preset range, then the nth event set is the valid event set; If the total number of events is not within the first preset range, then the nth event set is not the valid event set.

10. The eye-tracking model training method according to claim 8, wherein, Determining whether the nth event set is a valid event set based on the effective event pixel coverage rate of the nth event set includes: Based on the pixel coordinates of each event in the nth event set, determine the number of events occurring for each pixel within the preset target area; Pixels within the preset target area whose number of events exceeds the event occurrence threshold are designated as active pixels; Calculate the ratio of the number of active pixels in the preset target area to the total number of pixels in the preset target area to obtain the effective event pixel coverage rate of the nth event set; If the effective event pixel coverage rate is within a second preset range, then the nth event set is the effective event set; If the effective event pixel coverage is not within the second preset range, then the nth event set is not the effective event set.

11. The eye-tracking model training method according to claim 8, wherein, Determining whether the nth event set is a valid event set based on the event concentration of the nth event set includes: Set up multiple statistical regions within the event plane; The number of events occurring in each statistical region is determined based on the pixel coordinates of each event in the nth event set. The maximum value of the ratio of the number of events occurring in each statistical region to the total number of events in the nth event set is calculated to obtain the event concentration of the nth event set; If the event concentration is within a third preset range, then the nth event set is the valid event set; If the event concentration is not within the third preset range, then the nth event set is not the valid event set.

12. The eye-tracking model training method according to claim 8, wherein, Determining whether the nth event set is a valid event set based on the ratio of positive to negative polarity includes: Calculate the ratio of the number of positive events to the number of negative events in the nth event set to obtain the positive-to-negative polarity ratio of the nth event set; If the ratio of positive to negative polarity is within a fourth preset range, then the nth event set is the valid event set; If the positive and negative polarity ratio is not within the fourth preset range, then the nth event set is not the valid event set.

13. The eye-tracking model training method according to claim 1, wherein, The process of processing the nth event set to obtain the nth voxel tensor further includes: The nth time window is divided into multiple time sub-windows; Based on the multiple time sub-windows, the nth event set is divided into multiple primary event subsets; Based on the polarity of each event, each primary event subset is divided into two secondary event subsets; Accumulate all events in each sub-event subset to obtain the cumulative result for each sub-event subset; The nth voxel tensor is constructed by accumulating the results of all subsets of secondary events.

14. The eye-tracking model training method according to claim 13, wherein, The process of constructing the nth voxel tensor using the cumulative results of all subsets of secondary events includes: The cumulative result of each subset of secondary events is monotonically compressed to obtain the compressed cumulative result of each subset of secondary events. The nth voxel tensor is constructed by using the compressed cumulative result of all subsets of secondary events.

15. The eye-tracking model training method according to claim 1, wherein, The process of processing the nth frame scene image to obtain the gaze coordinates of the user's gaze target in the nth frame scene image includes: Target detection is performed on the nth frame scene image to obtain the bounding box of the gaze target; The gaze coordinates are determined based on the bounding box of the gaze target.

16. The eye-tracking model training method according to claim 15, wherein, Determining the gaze coordinates based on the bounding box of the gaze target includes: The coordinates of the center of the bounding box of the gaze target are calculated to obtain the gaze coordinates.

17. The eye-tracking model training method according to claim 15, wherein, The step of determining the gaze coordinates based on the bounding box of the gaze target further includes: Calculate the coordinates of the center of the bounding box of the gaze target to obtain the gaze coordinates to be processed in the nth frame scene image; Using a pre-calibrated homography matrix, the gaze coordinates to be processed are mapped to the display screen coordinate system to obtain the gaze coordinates.

18. The eye-tracking model training method according to claim 1, wherein, The step of training the eye-tracking model based on N gaze coordinate prediction results and N gaze coordinate labels includes: Select multiple valid labels from the N gaze coordinate labels; The eye-tracking model is trained using the multiple valid labels and the gaze coordinate prediction results corresponding to each valid label.

19. The eye-tracking model training method according to claim 18, wherein, The step of selecting multiple valid labels from the N gaze coordinate labels includes: Select a predetermined number of consecutive gaze coordinate labels from the N gaze coordinate labels; Determine the variation range of the predetermined number of gaze coordinate labels; If the change range is within the fifth preset range, then the predetermined number of gaze coordinate labels will be used as valid labels.

20. The eye-tracking model training method according to claim 1, wherein, Determining the nth time window based on the nth frame scene image includes: Calculate the difference between the timestamp of the nth frame scene image and the first duration to obtain the start time of the nth time window; The sum of the timestamp of the nth frame scene image and the second duration is calculated to obtain the end time of the nth time window.

21. The eye-tracking model training method according to any one of claims 1-20, wherein, The events collected by the receiving event camera at a second preset frequency include: The event stream data sent by the event camera is read in a block-by-block manner; Based on the preset frame header and preset length, multiple events to be processed are retrieved from the event stream data. Each pending event is validated in order to extract the successfully validated events from the plurality of pending events, thus obtaining the plurality of events.

22. An eye-tracking model training device, comprising: Memory; A processor, coupled to a memory, is configured to implement the eye-tracking model training method as described in any one of claims 1-21, based on the memory-stored instruction execution.

23. An eye-tracking method, comprising: Receive multiple events captured by the event camera at a preset frequency; The multiple events are processed to obtain voxel tensors; The voxel tensor is processed using an eye-tracking model to obtain a gaze coordinate prediction result, wherein the eye-tracking model is trained using the eye-tracking model training method according to any one of claims 1-21.

24. An eye-tracking control device, comprising: Memory; A processor, coupled to a memory, is configured to implement the eye-tracking method of claim 23 based on instructions stored in the memory.

25. An eye-tracking system, comprising: The eye-tracking control device as described in claim 24; An event camera is configured to capture multiple events at a preset frequency; An event transmission device is configured to send multiple events captured by the event camera to the eye-tracking control device.

26. The eye-tracking system according to claim 25, wherein, The event transmission device includes an event encapsulation module, a ring buffer module, a linear peeping module, a burst transmission module, and a callback module. The event encapsulation module is configured to encapsulate each of the plurality of events into an event frame; The circular buffer module is configured to use a circular buffer mechanism to write each event frame into the circular buffer area. The linear peeping module is configured to read event stream data from the annular buffer region according to the read pointer of the annular buffer region; The burst transmission module is configured to use a burst transmission mechanism to send the event stream data in blocks to the eye-tracking control device; The callback module is configured to advance the read pointer after the event stream data has been sent.

27. A computer-readable storage medium, wherein, The computer-readable storage medium stores computer instructions that, when executed by a processor, implement the method as described in any one of claims 1-21 and 23.

28. A computer program product comprising computer instructions, wherein the computer instructions, when executed by a processor, implement the method as described in any one of claims 1-21, 23.

Images