Eye movement data transmission method, computer storage medium and 3D display equipment

Abstract

The invention provides an eye movement data transmission method, a computer readable storage medium and 3D display equipment, belongs to the technical field of display, and can solve the problem of poor real-time performance of existing 3D display equipment. The method disclosed by the invention comprises the following steps: providing a storage unit which at least comprises a data buffer area; sequentially processing each frame of original eye movement data in the data frame sequence to generate data nodes in one-to-one correspondence with each frame of original eye movement data; sequentially writing each data node into a data buffer area of a storage unit through a writing unit according to a first mapping relation between the writing unit and the storage unit; reading a readable data node when the readable data node exists in the data buffer area through at least one reading unit according to a second mapping relation between the reading unit and the storage unit; and based on the readable data node, generating first eye movement data corresponding to the readable data node, and uploading the first eye movement data to the display control module, so that the display control module updates the display content based on the first eye movement data.

Description

Technical Field

[0001] This disclosure belongs to the field of display technology, specifically relating to a method for transmitting eye-tracking data, a computer storage medium, and a 3D display device. Background Technology

[0002] In recent years, glasses-free 3D display technology has been widely used in consumer electronics, education and training, healthcare, and cultural entertainment as a novel display technology. In short, the purpose of glasses-free 3D display technology is to allow viewers to experience stereoscopic visual effects without wearing 3D glasses. Currently, glasses-free 3D display technology typically involves adding a beam splitter to the display panel, providing different images to the viewer's left and right eyes respectively. When the viewer's brain receives the two different two-dimensional images, it automatically fuses them, ultimately synthesizing a picture with depth, spatial relationships, and a 3D stereoscopic effect.

[0003] Eye tracking technology is a key technology for achieving high-quality glasses-free 3D displays. It refers to the technology of detecting and analyzing human eye movements to achieve real-time positioning and tracking of the eye's position. In glasses-free 3D display systems, eye tracking technology is used to obtain the viewer's perspective information, thereby enabling accurate rendering and display of 3D images. Specifically, glasses-free 3D display systems can adjust the image's perspective and focal length in real time based on eye tracking, allowing viewers to see high-quality images from different positions; alternatively, eye tracking can enable real-time interaction between the viewer and the glasses-free 3D display device, allowing the viewer to click on corresponding locations on the screen based on their gaze, thus achieving interaction with the device.

[0004] However, existing eye tracking technologies suffer from low eye tracking frame rates and high data transmission latency, resulting in poor real-time performance and responsiveness of glasses-free 3D display devices. In addition, the lack of dynamic eye tracking compensation mechanisms in related technologies leads to 3D ghosting issues caused by untimely eye tracking when the viewer moves. Summary of the Invention

[0005] This invention aims to at least solve one of the technical problems existing in the prior art. On one hand, it provides a method for transmitting eye-tracking data, applied to a 3D display device. The method includes: providing a storage unit; sequentially processing each frame of raw eye-tracking data in a data frame sequence to generate data nodes corresponding one-to-one with each frame of raw eye-tracking data; sequentially writing each data node into the data buffer of the storage unit according to a first mapping relationship between the writing unit and the storage unit through a writing unit; reading the readable data node when it exists in the data buffer according to a second mapping relationship between the reading unit and the storage unit through at least one reading unit; generating first eye-tracking data corresponding to the readable data node based on the readable data node, and uploading it to the display control module of the 3D display device so that the display control module can update the display content of the 3D display device based on the first eye-tracking data.

[0006] In some optional embodiments, the storage unit further includes an encryption key area; for any frame of raw eye-tracking data, processing the frame of raw eye-tracking data to generate a data node corresponding to the frame of raw eye-tracking data includes: obtaining encrypted raw eye-tracking data based on the frame of raw eye-tracking data and the key obtained from the encryption key area; generating a corresponding checksum based on the encrypted raw eye-tracking data to generate a data node corresponding to the frame of raw eye-tracking data; the data node includes at least: the encrypted raw eye-tracking data and the checksum.

[0007] In some optional embodiments, the storage unit further includes a metadata area; for any data node corresponding to the original eye-tracking data of any frame, writing the data node into the data buffer includes: obtaining the write position of the data node corresponding to the original eye-tracking data of the frame in the data buffer according to the write pointer of the metadata area; writing the data node corresponding to the original eye-tracking data of the frame into the write position in the data buffer indicated by the write pointer of the metadata area; updating the write pointer of the metadata area and the current counter node to complete the writing of the data node corresponding to the original eye-tracking data of the frame; the current counter node represents the number of readable data nodes.

[0008] In some optional embodiments, reading the readable data node includes: obtaining the position of the readable data node in the data buffer according to the read pointer corresponding to the reading unit in the metadata area; and reading the readable data node from the position of the readable data node in the data buffer.

[0009] In some optional embodiments, generating the first eye-tracking data corresponding to the readable data node includes: verifying the readable data node according to the verification code; after the verification is successful, decrypting the readable data node according to the key in the encryption key area to generate decrypted data, and using the decrypted data as the first eye-tracking data.

[0010] In some optional embodiments, while writing the data node to the data buffer of the storage unit, the method further includes: sending a notification signal to the reading unit; the notification signal indicating that the readable data node exists in the data buffer.

[0011] In some optional embodiments, the storage unit further includes a metadata area; before reading the readable data node, the method further includes: periodically querying the metadata area to determine whether the current count node in the metadata area is greater than 0, and when the current count node is greater than 0, determining that there is a readable data node in the data buffer.

[0012] In some optional embodiments, the storage unit further includes a metadata area and an encryption key area; before the reading unit reads the readable data node, the method further includes: determining whether the reading unit has read permission based on the identifier permission list in the metadata area and the identifier of the reading unit; if the reading unit has read permission, synchronizing the key in the encryption key area to the eye-tracking data generation unit corresponding to the reading unit so that it can generate the first eye-tracking data; if the reading unit does not have read permission, generating a security alarm.

[0013] In some optional embodiments, the data frame sequence includes N frames of raw eye-tracking data, where N is an integer not less than 1; m frames of the raw eye-tracking data in the N frames are measured eye-tracking data, and (Nm) frames of the raw eye-tracking data are predicted eye-tracking data; m is an integer not less than 1 and not greater than N; before processing each frame of raw eye-tracking data sequentially, the method further includes: generating the data frame sequence; generating the data frame sequence includes: acquiring m frames of measured eye-tracking data; using a pre-trained interpolation algorithm, calculating (Nm) frames of predicted eye-tracking data based on the m frames of measured eye-tracking data; arranging the m frames of measured eye-tracking data and (Nm) frames of predicted eye-tracking data according to the time sequence corresponding to each frame to generate the data frame sequence.

[0014] In some optional embodiments, the pre-trained interpolation algorithm includes one or more of the following: linear interpolation algorithm, quadratic interpolation algorithm, trigonometric interpolation algorithm, Newton interpolation algorithm, and Gaussian interpolation algorithm.

[0015] In a second aspect, this disclosure provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the eye-tracking data transmission method described in any of the embodiments of the first aspect above.

[0016] Thirdly, this disclosure provides a 3D display device, comprising: a processor, a computer-readable storage medium, and a display control module; the computer-readable storage medium stores a computer program, and the processor is configured to implement the eye-tracking data transmission method according to any one of claims 1-10 when executing the computer program, to generate a first data frame sequence based on each frame of original eye-tracking data in the data frame sequence, and upload it to the display control module; the first data frame sequence includes multiple frames of first eye-tracking data corresponding one-to-one with the original eye-tracking data; for each frame of first eye-tracking data, the display control module is configured to calculate the corresponding disparity adjustment coefficient based on the frame of first eye-tracking data, and update the display content once based on the disparity adjustment coefficient.

[0017] In some optional embodiments, the data frame sequence includes N frames of raw eye-tracking data, where N is an integer not less than 1; m frames of the raw eye-tracking data in the N frames are measured eye-tracking data, and (Nm) frames of the raw eye-tracking data are predicted eye-tracking data; m is an integer not less than 1 and not greater than N; the 3D display device further includes an eye-tracking device configured to acquire m frames of binocular images at a preset frequency within a time period corresponding to a data frame sequence, and calculate m frames of measured eye-tracking data based on the m frames of binocular images. Attached Figure Description

[0018] Figure 1 This is a schematic diagram of the structure of a naked-eye 3D display device based on a slit grating.

[0019] Figure 2 This is a schematic diagram of a naked-eye 3D display device based on a lenticular lens array.

[0020] Figure 3 This is a schematic diagram of an exemplary modular structure for a glasses-free 3D display device.

[0021] Figure 4 This is a flowchart illustrating the eye-tracking data transmission method provided in this disclosure.

[0022] Figure 5 This is a schematic diagram of the regional structure of a storage unit.

[0023] Figure 6 This is a schematic diagram of the modular structure of the eye-tracking data transmission device provided in this disclosure.

[0024] Figure 7This is a schematic diagram of the modular structure of the 3D display device provided in this disclosure. Detailed Implementation

[0025] To enable those skilled in the art to better understand the technical solution of the present invention, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0026] Unless otherwise defined, the technical or scientific terms used in this disclosure shall have the ordinary meaning understood by one of ordinary skill in the art to which this disclosure pertains. The terms “first,” “second,” and similar terms used in this disclosure do not indicate any order, quantity, or importance, but are merely used to distinguish different components. Similarly, the terms “an,” “a,” or “the,” and similar terms do not indicate a quantity limitation, but rather indicate the presence of at least one. The terms “including,” “comprising,” or “containing,” and similar terms mean that the element or object preceding the word encompasses the elements or objects listed following the word and their equivalents, without excluding other elements or objects. The terms “connected,” “linked,” or similar terms are not limited to physical or mechanical connections, but can include electrical connections, whether direct or indirect. The terms “upper,” “lower,” “left,” and “right,” etc., are used only to indicate relative positional relationships, and these relative positional relationships may change accordingly when the absolute position of the described objects changes.

[0027] As used herein, “parallel” and “perpendicular” include the described situation and situations that are similar to the described situation, within an acceptable range of deviation, which is determined by those skilled in the art taking into account the measurement under discussion and the error associated with the measurement of a particular quantity (i.e., the limitations of the measurement system). For example, “parallel” includes absolute parallelism and approximate parallelism, where an acceptable range of deviation for approximate parallelism may be, for example, within 5°; “perpendicular” includes absolute perpendicularity and approximate perpendicularity, where an acceptable range of deviation for approximate perpendicularity may also be, for example, within 5°.

[0028] It should be understood that when a layer or element is referred to as being on another layer or substrate, it can mean that the layer or element is directly on the other layer or substrate, or that there is an intermediate layer between the layer or element and the other layer or substrate.

[0029] In this article, "electrical connection" includes the situation where constituent elements are connected together by a component that has a certain electrical function. There are no particular restrictions on the "component that has a certain electrical function" as long as it enables the transmission and reception of electrical signals between the connected constituent elements. Examples of "components that have a certain electrical function" include not only electrodes and wiring, but also switching elements such as transistors, resistors, inductors, capacitors, and other components with various functions.

[0030] This document describes exemplary embodiments with reference to sectional views and / or plan views, which are idealized exemplary drawings. In the drawings, the thickness of layers and regions is enlarged for clarity. Therefore, variations in shape relative to the drawings are contemplated due to, for example, manufacturing techniques and / or tolerances. Therefore, exemplary embodiments should not be construed as limited to the shapes of the regions shown herein, but rather include shape deviations due to, for example, manufacturing processes. Thus, the regions shown in the drawings are schematic in nature, and their shapes are not intended to show the actual shapes of the regions of the device, nor are they intended to limit the scope of the exemplary embodiments.

[0031] In summary, current technologies for achieving glasses-free 3D displays generally include: multi-viewpoint stereoscopic display technology, volumetric 3D display technology, holographic 3D display technology, and light field display technology. Among these, multi-viewpoint stereoscopic display technology is the most widely used and mature glasses-free 3D display technology in commercial applications. Simply put, glasses-free 3D display devices based on multi-viewpoint stereoscopic display technology mainly consist of a display panel and a light-controlling device located on the light-emitting side of the display panel. The display panel is mainly used to display images from multiple perspectives of the 3D picture, while the light-controlling device is used to control the direction of light emitted by each pixel of the display panel to construct different viewpoints. This allows the viewer's two eyes to correspond to different viewpoints, viewing two images with parallax, thus forming a 3D effect.

[0032] Commonly used light control devices include slit gratings or cylindrical lens arrays.

[0033] Figure 1 This is a schematic diagram of a glasses-free 3D display device based on a slit grating. Figure 1 As shown, the naked-eye 3D display device mainly includes a display panel 1 and a slit grating 2. The display panel 1 includes multiple pixels, which can be divided into multiple groups according to the number of viewpoints. Commonly, when the viewpoints include a first viewpoint VP1 and a second viewpoint VP2, the multiple pixels can be correspondingly divided into a first pixel group and a second pixel group. The first pixel group includes multiple first pixels PX1 corresponding to the first viewpoint VP1, and the second pixel group includes multiple second pixels PX2 corresponding to the second viewpoint VP2. The slit grating 2 includes alternately arranged light-blocking areas and light-transmitting areas. The light-transmitting areas allow light to pass through, while the light-blocking areas physically block the light. After the light emitted from the display panel 1 passes through the slit grating 2, the light emitted from each first pixel PX1 is directed to the first viewpoint VP1, and the light emitted from each second pixel PX2 is directed to the second viewpoint VP2. Ultimately, this results in a parallax between the images received by the two viewpoints. The brain then merges the two images with parallax to produce a stereoscopic 3D effect.

[0034] Figure 2This is a schematic diagram of a glasses-free 3D display device based on a lenticular lens array. Figure 2 As shown, the naked-eye 3D display device mainly includes a display panel 1 and a lenticular lens array 3. The display panel 1 includes multiple pixels, which can be divided into multiple groups according to the number of viewpoints, such as... Figure 3 As shown, when the viewpoint includes a first viewpoint VP1 and a second viewpoint VP2, multiple pixels can be correspondingly divided into a first pixel group and a second pixel group. The first pixel group includes multiple first pixels PX1 corresponding to the first viewpoint VP1, and the second pixel group includes multiple second pixels PX2 corresponding to the second viewpoint VP2. The lenticular lens array 3 includes multiple lenses used to deflect the light emitted from the display panel 1, thereby focusing the light emitted from the first pixel PX1 onto the first viewpoint VP1 and focusing the light emitted from the second pixel PX2 onto the second viewpoint VP2. This allows the first viewpoint VP1 and the second viewpoint VP2 to receive light emitted from different pixels, that is, to receive different images. After receiving two images with parallax, the brain automatically fuses them to generate a 3D image.

[0035] Figure 3 This is a schematic diagram of an exemplary modular structure for a glasses-free 3D display device. Figure 3 As shown, in addition to the display panel 1 and light control devices (such as the slit grating or lenticular lens array lamp mentioned above), the naked-eye 3D display device also includes an eye-tracking module 4, a control module 5, and an adjustment module 6. The eye-tracking module 4 includes an acquisition unit and a processing unit integrated around the display panel 1. The acquisition unit is configured to acquire images of the human eye at a certain sampling frequency. The processing unit is configured to locate the binocular regions based on the acquired images, extract feature points from the binocular regions, and analyze the changes in the feature points in consecutive image frames to infer the direction and speed of eye movement. Finally, through a specific mapping function, it calculates the coordinates of the eyes in the coordinate system of the display panel 1 and the focal point of the gaze (referred to as eye-tracking data). The control module 5 is configured to determine, based on the eye-tracking data, whether the currently displayed content can allow the viewer to see the correct 3D image. If the determination result is negative, it calculates the parallax coefficient suitable for the current coordinates based on the binocular coordinates to generate a suitable image for display. Furthermore, while updating the display content, the control module 5 generates an adjustment signal, which causes the adjustment module 6 to respond to the adjustment signal and adjust the light control device to ensure that the light emitted by the display panel 1 is directed to a specific viewpoint; for example, adjusting the tilt angle or setting position of the slit grating or the columnar lens array to change the direction of light deflection and accurately guide it to the human eye.

[0036] The naked-eye 3D display devices in related technologies have the following technical problems.

[0037] In practice, the acquisition unit often uses a camera with a specific acquisition frequency. The time it takes for the eye-tracking module 4 to acquire one frame of eye-tracking data is the time interval between two consecutive images acquired by the camera plus the time to calculate the eye-tracking data based on the images. However, because the camera frame rate used in current glasses-free 3D display devices is relatively low (e.g., 60 fPs), the time interval between two consecutive images acquired is relatively long (16.7ms for 60 fPs). This results in a low tracking frame rate for the eye-tracking module 4. Since the screen refresh rate of glasses-free 3D display devices is usually 240Hz, that is, refreshed once every 4.17ms, when the eye-tracking frame rate is too low, it cannot provide eye-tracking data for screen refresh in a timely manner, which can easily lead to the screen not refreshing in time, causing viewers to experience dizziness or stuttering.

[0038] In related technologies, eye-tracking module 4 transmits eye-tracking data to control module 5 via a socket. However, sockets are a network-layer communication method that requires binding to an IP address and port. This presents redundancy in local module communication scenarios and lacks a built-in client authentication mechanism, making it vulnerable to unauthorized modules illegally obtaining eye-tracking data and posing a risk of data leakage. Furthermore, although local sockets (such as Unix domain sockets) can reduce transmission overhead, in actual tests, the transmission latency of a single frame of eye-tracking data can still reach 3ms-4ms. Therefore, in high frame rate scenarios (such as 120fPs), with a processing window of only 8ms, the 3ms-4ms transmission latency severely compresses the subsequent computation and processing time of control module 5.

[0039] Furthermore, traditional eye-tracking data transmission methods typically output only one channel of eye-tracking data at a time. When a glasses-free 3D display device includes multiple 3D applications, such as 3D rendering applications and data analysis applications, these applications need to share eye-tracking data. If traditional transmission methods are used, the data needs to be distributed to different 3D applications at different times. This is not only time-consuming but also prone to data usage conflicts between different applications.

[0040] In summary, due to factors such as low eye-tracking frame rate and data transmission delay, naked-eye 3D display devices suffer from poor real-time performance and untimely response.

[0041] In order to solve at least one of the above-mentioned technical problems, on the one hand, this disclosure provides a method for transmitting eye-tracking data, which is applied to a 3D display device.

[0042] Figure 4 This is a flowchart illustrating the eye-tracking data transmission method provided in this disclosure. Figure 4 As shown, the flow of the eye-tracking data transmission method includes:

[0043] Step S1: Provide a storage unit 103.

[0044] The storage unit 103 disclosed herein is a shared memory region created using the Windows system API CreatFileMapping function. When creating the shared memory region, its capacity and name can be specified. For example, the memory capacity of storage unit 103 can be specified as 1Mb. This 1Mb storage unit 103 can hold approximately 10 seconds of data for a data frame sequence comprising 120 frames of raw eye-tracking data, with each frame containing approximately 100 bytes of raw eye-tracking data (one data frame sequence corresponds to 1 second of time).

[0045] At the same time, the storage unit 103 can also be divided into regions. Figure 5 This is a schematic diagram of the regional structure of a storage unit. For example... Figure 5 As shown, storage unit 103 is divided into metadata area, data buffer and encryption key area.

[0046] The metadata area stores a series of management variables, such as: head pointer, tail pointer, current count node, maximum count node, data format version, and an identifier permission list. Specifically, head indicates the position in the data buffer that the writing unit can write a data node to; tail indicates the position in the data buffer that the reading unit can read a data node from; count represents the current number of data nodes, indicating whether there are readable data nodes (a value greater than 0 indicates the presence of readable data nodes); and max_count represents the maximum number of nodes the data buffer can hold, used to prevent data overflow. The data format version refers to the format version number of the currently stored data node, ensuring that the device parses the data node only if the format version number matches, avoiding parsing errors due to version inconsistencies. The identifier permission list stores the identifiers of each reading unit allowed to read the data nodes in storage unit 103 and their corresponding permissions (e.g., "read-only," "readable and processable"), preventing unauthorized reading units from unauthorizedly reading data nodes and causing data leakage, thus ensuring user privacy and security. Understandably, when there are multiple reading units (i.e., multiple 3D applications need to utilize eye-tracking data), there are multiple tail pointers in the metadata area, each corresponding to a reading unit. Additionally, the metadata area may include a digital signature to prevent malicious processes from tampering with the pointers or modifying the identifier permission list. This digital signature is periodically updated by the writing unit 102. When reading a data node, the reading unit 104 needs to verify the digital signature. If the verification passes, it indicates that the metadata area originates from the legitimate owner of the digital signature and that the data in the metadata area has not been tampered with. Then, it reads the data node according to the position indicated by the pointer in the metadata area.

[0047] The data buffer is a circular stack structure containing multiple data nodes. Each data node corresponds to one frame of raw eye-tracking data. When writing data nodes to this circular stack structure, a "write-overwrite" strategy is followed. That is, when the current counting node `count` is greater than or equal to the maximum counting node `max_count`, the new data node overwrites the oldest data node pointed to by the read pointer `tail`, ensuring real-time data transmission. Each data node includes [timestamp (8 bytes), data length (4 bytes), physical quantity type (2 bytes), encrypted raw eye-tracking data (length depends on the actual data), and checksum (4 bytes)]. The timestamp refers to the moment of the raw eye-tracking data; the data length refers to the length of the encrypted raw eye-tracking data; the physical quantity type refers to the specific physical quantity of the raw eye-tracking data, such as pupil coordinates and motion vectors; the checksum is a CRC32 checksum, used to verify the data after the device reads the data node, ensuring error-free data transmission.

[0048] The encryption key area is used to store the symmetric encryption key, enabling the device to encrypt the raw eye-tracking data using the key, and simultaneously decrypt the encrypted raw eye-tracking data using the key. This symmetric encryption key can be set to be automatically updated periodically to improve data security. For example, taking the storage unit 103 described above, which can hold 10 seconds of data, the update frequency of the symmetric encryption key in the encryption key area can be set to once every 10 seconds. After the update, the old key is retained for an additional 5 seconds for subsequent effective decryption by the data node being read.

[0049] Then, the Windows system's MapViewOfFile function (memory mapping function) is called to map the aforementioned storage unit 103 to the virtual memory of the write unit 102. At the same time, the aforementioned storage unit 103 is also mapped to the virtual memory of the read unit 104. That is, a first mapping relationship is established between the storage unit 103 and the write unit 102, and a second mapping relationship is established between the storage unit 103 and the read unit 104, so that the write unit 102 and the read unit 104 can operate on the storage unit 103 by operating on their own virtual memory.

[0050] Step S2: Process the raw eye-tracking data of each frame in the data frame sequence in sequence to generate data nodes that correspond one-to-one with the raw eye-tracking data of each frame.

[0051] Specifically, for any frame of raw eye-tracking data, the raw eye-tracking data of that frame is processed to generate a data node corresponding to that frame of raw eye-tracking data, including:

[0052] Step S21: Based on the original eye-tracking data of the frame and the key obtained from the encryption key area, the encrypted original eye-tracking data is obtained.

[0053] Step S22: Generate a corresponding checksum based on the encrypted raw eye-tracking data to generate a data node corresponding to the frame of raw eye-tracking data. Simultaneously, obtain the timestamp and physical quantity type based on the time corresponding to the raw eye-tracking data and its associated physical quantity; obtain the data length based on the data length of the encrypted raw eye-tracking data, thus obtaining a complete data node. As described above, the data node includes: timestamp, data length, physical quantity type, encrypted raw eye-tracking data, and checksum. For example, the above encryption of the raw eye-tracking data can be implemented using the AES-128 encryption algorithm, where the 128-bit key refers to the symmetric encryption key stored in the encryption key area of ​​storage unit 103.

[0054] Step S3: Through the writing unit 102, according to the first mapping relationship between the writing unit 102 and the storage unit 103, each data node is sequentially written into the data buffer of the storage unit.

[0055] Specifically, for any data node corresponding to any frame of raw eye-tracking data, the data node is written into the data buffer, including:

[0056] Step S31: Based on the write pointer head of the metadata area, obtain the write position of the data node corresponding to the original eye-tracking data of the frame in the data buffer.

[0057] Step S32: Write the data node corresponding to the original eye-tracking data of the frame to the write position in the data buffer indicated by the write pointer head in the metadata area.

[0058] Step S32: Update the write pointer and current counter node of the metadata area to complete the writing of the data nodes corresponding to the original eye-tracking data of this frame. The current counter node represents the number of readable data nodes.

[0059] As described above, the current counter node represents the number of readable data nodes. For example, updating the write pointer and the current counter node in the metadata area is implemented using an atomic operation (InterlockedConpareExchange), thereby avoiding data corruption caused by contention between the write unit 102 and the read unit 104. Traditional locking mechanisms cause the other unit to be suspended (entering a sleep state) when either the write unit 102 or the read unit 104 operates on the storage unit 103, waiting for the currently operating unit to complete its operation before it can execute its own. This results in queuing and waiting for both write and read processes, leading to transmission delays. Atomic operations allow the write unit 102 and the read unit 104 to perform operations on the storage unit 103 simultaneously without blocking. Therefore, compared to traditional locking mechanisms, atomic operations can avoid deadlocks and reduce latency.

[0060] Step S4: Using at least one reading unit 104, when a readable data node exists in the data buffer according to the second mapping relationship between the reading unit and the storage unit, read the readable data node.

[0061] Specifically, reading readable data nodes includes:

[0062] Step S41: Obtain the position of the readable data node in the data buffer based on the read pointer tail corresponding to the read unit in the metadata area.

[0063] Step S42: Read the readable data node from its position in the data buffer.

[0064] Step S5: The eye-tracking data generation unit 105 corresponding to the reading unit 104 generates the first eye-tracking data corresponding to the readable data node based on the readable data node, and uploads it to the display control module so that the display control module can update the display content based on the first eye-tracking data.

[0065] Specifically, the first eye-tracking data corresponding to this readable data node is generated, including:

[0066] Step S51: Verify the readable data nodes according to the check code.

[0067] Step S52: After successful verification, decrypt the readable data node using the key in the encryption key area to generate decrypted data, which is then used as the first eye-tracking data. If verification fails, discard the data node to prevent dirty data from being transmitted to the 3D application.

[0068] In related technologies, eye-tracking data transmission often uses Socket. As a network-layer-based simultaneous method, it requires binding to IP addresses and ports, resulting in redundant design in local communication scenarios. Moreover, it lacks a built-in client authentication mechanism, which can easily lead to unauthorized reading units illegally obtaining eye-tracking data, posing a risk of data leakage. In addition, actual tests have shown that the Socket transmission method requires multiple large data copies, resulting in a relatively long latency of about 3ms-4ms for transmitting a single frame of eye-tracking data. For high frame rate display devices, the processing time for each frame is required to be very short. For example, a 120Hz display device needs to refresh the display content every 8ms. When the data transmission latency reaches 3-4ms, it means that the data processing and calculation time of the display device is only about 4ms, which severely compresses the subsequent calculation time, resulting in poor real-time performance of the display device.

[0069] The eye-tracking data transmission method disclosed herein utilizes a storage unit where writing and reading units can operate in parallel. This allows the writing unit to process the raw eye-tracking data according to specific encryption rules and write it into the storage unit based on its mapping relationship with the storage unit. Simultaneously, the reading unit can read the eye-tracking data from the storage unit based on its mapping relationship. Compared to traditional socket transmission methods, this approach eliminates the overhead of multiple large data copies and, due to zero-copy, avoids excessively long transmission links, thus significantly improving communication speed. Actual testing shows that the transmission latency for a single frame of raw eye-tracking data using the transmission device disclosed herein can be reduced to 0.2ms-0.5ms. Furthermore, since the storage unit stores an identifier permission list, the identifier of the reading unit can be verified, preventing unauthorized reading units or 3D applications from unauthorizedly reading eye-tracking data, thereby improving user privacy and data security. Furthermore, the metadata area employs a digital signature mechanism, and each piece of raw eye-tracking data is encrypted with a key before being written. These two methods respectively prevent the data in the metadata area and the data buffer from being tampered with or stolen, further enhancing data security and meeting the high privacy protection requirements in some scenarios. Moreover, the transmission method disclosed herein supports multiple reading units / 3D applications reading data nodes simultaneously. Each reading unit reads data from the corresponding location according to its own read pointer tail, making it suitable for complex architecture scenarios involving multiple 3D applications, such as scenarios including 3D rendering, eye-tracking data analysis, and other 3D applications.

[0070] Furthermore, in some examples, the transmission method also includes:

[0071] Step SS1: While writing the data node into the data buffer of the storage unit, a notification signal is sent to the read unit; the notification signal indicates that there is a readable data node in the data buffer.

[0072] In specific implementation, the function of the notification unit can be implemented using the kernel event (CreatEvent) function in the Windows system. After the writing unit 102 writes the data node, the notification unit can trigger the EVENT_DATA_AVAILABLE event, and the reading unit 104 can read after waiting for the EVENT_DATA_AVAILABLE event.

[0073] Alternatively, in other examples, the transmission method also includes:

[0074] Step SA1: Before reading readable data nodes, periodically query the metadata area to determine if the current counter node count in the metadata area is greater than 0. If the current counter node count is greater than 0, it is determined that there are readable data nodes in the data buffer.

[0075] In other words, a polling mechanism can be used to query the current counting node at a certain frequency to determine whether there are readable data nodes.

[0076] Both steps SS1 and SSA are methods to determine whether readable data nodes exist; either one can be chosen. However, compared to step SSA, step SS1 avoids the resource consumption caused by polling because it does not require periodic queries.

[0077] In some examples, the transmission method further includes the following steps prior to step SSA:

[0078] Step SA0: Based on the identification number permission list in the metadata area and the identification number of the reading unit, determine whether the reading unit has reading permission. If the reading unit has reading permission, synchronize the key in the encryption key area to the eye-tracking data generation unit corresponding to the reading unit so that it can generate the first eye-tracking data. If the reading unit does not have reading permission, generate a security alarm.

[0079] In simple terms, the identifier permission list of the metadata area can be modified based on the identifier of the accessible reading unit 104. Before reading a data node, it is necessary to determine whether the identifier of the reading unit 104 is in the identifier permission list. If it is, the reading unit 104 is an authorized reading unit; otherwise, it is an unauthorized reading unit. For example, generating a security alert can be achieved using the Windows system's memory protection function (PAGE_READWRITE SECURITY_DESCRIPTOR). When this function is called, it will issue an ERROR_ACCES_DENIDE error, which can be used as a security alert. This can prevent unauthorized applications from obtaining eye-tracking data, thereby ensuring user privacy and data security.

[0080] In related technologies, raw eye-tracking data is obtained through collection and calculation using eye-tracking devices. These devices typically include a camera and a data processing module. The camera captures images of both eyes at a specific frequency, while the data processing module calculates raw eye-tracking data based on these images. However, cameras typically have image capture frame rates of 60Hz or 90Hz, meaning they can only generate 60 or 90 frames of raw eye-tracking data per second. In contrast, common 3D applications often use refresh rates of 120Hz or 240Hz, requiring content updates 120 or 240 times per second. This results in a situation where some refresh moments in 3D applications lack available eye-tracking data, preventing effective content updates and causing a stuttering or lag-like experience for the user.

[0081] To address the aforementioned issues, the method provided in this disclosure obtains the data frame sequence as follows: based on several frames of measured eye-tracking data calculated by an eye-tracking device, several frames of predicted eye-tracking data are generated, ensuring that the number of frames of the original eye-tracking data in a data frame sequence matches the refresh rate of the 3D application. Taking a camera with a 90Hz acquisition frame rate as an example, it can acquire 90 frames of measured eye-tracking data per second. Therefore, based on these 90 frames of measured eye-tracking data, 150 frames of predicted eye-tracking data can be calculated. The 90 frames of measured eye-tracking data and the 150 frames of predicted eye-tracking data are then arranged in chronological order to form the data frame sequence.

[0082] Specifically, in some examples, the process prior to step S1 includes:

[0083] Step S01: Generate a data frame sequence. The data frame sequence includes N frames of raw eye-tracking data, where N is an integer not less than 1; m frames of raw eye-tracking data in the N frames are measured eye-tracking data, and (Nm) frames of raw eye-tracking data are predicted eye-tracking data; m is an integer not less than 1 and not greater than N.

[0084] Specifically, generating a data frame sequence includes:

[0085] Step S001: Obtain m frames of measured eye movement data.

[0086] The m-frame measured eye movement data can be collected and calculated by an eye-tracking device, which typically includes a camera and a data processing module. The camera is used to collect binocular images at a specific frequency, and the data processing module is used to calculate the raw eye movement data based on the binocular images.

[0087] Step S002: Using a pre-trained interpolation algorithm, calculate the predicted eye movement data for (Nm) frames based on the measured eye movement data of the m frames.

[0088] Examples of pre-trained interpolation algorithms include one or more of the following: linear interpolation algorithm, quadratic interpolation algorithm, trigonometric interpolation algorithm, Newton interpolation algorithm, and Gaussian interpolation algorithm.

[0089] During training, the parameters of the interpolation algorithm can be dynamically adjusted based on the deviation between the predicted eye-tracking data every 100ms and the measured eye-tracking data at that moment. This improves the accuracy of the interpolation algorithm and prevents the prediction error from accumulating in subsequent calculations, thus affecting the calculation results of 3D applications. For example, the training termination condition can be set as follows: the deviation between the predicted eye-tracking data every 100ms and the measured eye-tracking data at that moment is less than a preset value. For instance, the error between the predicted pupil coordinates and the measured pupil coordinates must be less than 0.5° for the interpolation algorithm training to be considered complete.

[0090] It should be noted that this disclosure does not impose any restrictions on the interpolation algorithm for calculating predicted eye-tracking data, or on the rules for inserting predicted eye-tracking data. In some examples, a fixed number of predicted eye-tracking data can be uniformly inserted between any two adjacent measured eye-tracking data points; in other examples, the number of predicted eye-tracking data points between two adjacent measured eye-tracking data points is not fixed. For example, the change between the (i-1)th and the ith measured eye-tracking data points is small, so the number of predicted eye-tracking data points inserted between them can be small, while the change between the ith and the (i+1)th measured eye-tracking data points is drastic, so the number of predicted eye-tracking data points inserted between them can be large, where i is an integer greater than or equal to 2 and less than or equal to N-1. Furthermore, continuing with the "240Hz" frame rate example above, when the refresh rate of a 3D application is 240Hz, it means that it needs to refresh once every 4.17ms, that is, it needs to acquire eye-tracking data once every 4.17ms. However, due to the randomness of insertion, the time of the 240 data points in the data frame sequence is not one every 4.17ms. At this time, the time of each data node in the storage unit 102 is also not one every 4.17ms. In this case, the 3D application can use the data node closest to the current time. For example, if the current time is 4.17ms, and the storage unit 102 has data nodes at times of 4ms, 5ms, 6.2ms, etc., then the 3D application can choose the data node corresponding to the 4ms time as the basis for its current refresh. Furthermore, it should be noted that if the current time is 4.17ms, and the storage unit 102 contains data nodes at times such as 3ms, 4ms, and 5ms, the data node at 3ms is the actual eye-tracking data, while the data node at 4ms is the predicted eye-tracking data. In this case, the 3D application can choose the actual eye-tracking data as the basis for the current refresh instead of selecting the data with the closest time according to the time priority principle, thus improving the accuracy of the refresh.

[0091] Step S003: Arrange the m-frame measured eye movement data and (Nm)-frame predicted eye movement data according to the time sequence corresponding to each frame to generate a data frame sequence.

[0092] The eye-tracking data transmission method disclosed herein, based on the refresh frame rate of a 3D application, generates multiple frames of predicted eye-tracking data from several frames of measured eye-tracking data as supplementary data to the measured eye-tracking data. The measured eye-tracking data and the predicted eye-tracking data are then fused together to form a complete data frame sequence, ensuring that the number of frames in the data frame sequence matches the refresh frame rate of the 3D application. The advantages are twofold: firstly, it allows the 3D application to acquire the corresponding eye-tracking data at each refresh, thereby improving the timeliness and real-time response of the 3D application to human eye movements; secondly, it eliminates the need for high frame rate and high performance cameras, thus reducing hardware costs.

[0093] Based on the same inventive concept, in a second aspect, this disclosure provides an eye-tracking data transmission device.

[0094] Figure 6 This is a schematic diagram of the eye-tracking data transmission device provided in this disclosure. Figure 6 As shown, the eye-tracking data transmission device 100 mainly includes: a storage unit 103, a raw data processing unit 101, a writing unit 102, at least one reading unit 104, and an eye-tracking data generation unit 105 corresponding to the reading unit 104.

[0095] The storage unit 103 disclosed herein is a shared memory region created using the Windows system API CreatFileMapping function. When creating the shared memory region, its capacity and name can be specified. For example, the memory capacity of storage unit 103 can be specified as 1Mb. This 1Mb storage unit 103 can hold approximately 10 seconds of data for a data frame sequence comprising 120 frames of raw eye-tracking data, with each frame containing approximately 100 bytes of raw eye-tracking data (one data frame sequence corresponds to 1 second of time).

[0096] At the same time, the storage unit 103 can also be divided into regions, such as Figure 4 As shown, storage unit 103 is divided into metadata area, data buffer and encryption key area.

[0097] The metadata area stores a series of management variables, such as: head pointer, tail pointer, current count node, maximum count node, data format version, and an identifier permission list. Specifically, head indicates the position in the data buffer that the writing unit can write a data node to; tail indicates the position in the data buffer that the reading unit can read a data node from; count represents the current number of data nodes, indicating whether there are readable data nodes (a value greater than 0 indicates the presence of readable data nodes); and max_count represents the maximum number of nodes the data buffer can hold, used to prevent data overflow. The data format version refers to the format version number of the currently stored data node, ensuring that the device parses the data node only if the format version number matches, avoiding parsing errors due to version inconsistencies. The identifier permission list stores the identifiers of each reading unit allowed to read the data nodes in storage unit 103 and their corresponding permissions (e.g., "read-only," "readable and processable"), preventing unauthorized reading units from unauthorizedly reading data nodes and causing data leakage, thus ensuring user privacy and security. Understandably, when there are multiple reading units (i.e., multiple 3D applications need to utilize eye-tracking data), there are multiple tail pointers in the metadata area, each corresponding to a reading unit. Additionally, the metadata area may include a digital signature to prevent malicious processes from tampering with the pointers or modifying the identifier permission list. This digital signature is periodically updated by the writing unit 102. When reading a data node, the reading unit 104 needs to verify the digital signature. If the verification passes, it indicates that the metadata area originates from the legitimate owner of the digital signature and that the data in the metadata area has not been tampered with. Then, it reads the data node according to the position indicated by the pointer in the metadata area.

[0098] The data buffer is a circular stack structure containing multiple data nodes. Each data node corresponds to one frame of raw eye-tracking data. When writing data nodes to this circular stack structure, a "write-overwrite" strategy is followed. That is, when the current counting node `count` is greater than or equal to the maximum counting node `max_count`, the new data node overwrites the oldest data node pointed to by the read pointer `tail`, ensuring real-time data transmission. Each data node includes [timestamp (8 bytes), data length (4 bytes), physical quantity type (2 bytes), encrypted raw eye-tracking data (length depends on the actual data), and checksum (4 bytes)]. The timestamp refers to the moment of the raw eye-tracking data; the data length refers to the length of the encrypted raw eye-tracking data; the physical quantity type refers to the specific physical quantity of the raw eye-tracking data, such as pupil coordinates and motion vectors; the checksum is a CRC32 checksum, used to verify the data after the device reads the data node, ensuring error-free data transmission.

[0099] The encryption key area is used to store the symmetric encryption key, enabling the device to encrypt the raw eye-tracking data using the key, and simultaneously decrypt the encrypted raw eye-tracking data using the key. This symmetric encryption key can be set to be automatically updated periodically to improve data security. For example, taking the storage unit 103 described above, which can hold 10 seconds of data, the update frequency of the symmetric encryption key in the encryption key area can be set to once every 10 seconds. After the update, the old key is retained for an additional 5 seconds for subsequent effective decryption by the data node being read.

[0100] Then, the Windows system's MapViewOfFile function (memory mapping function) is called to map the aforementioned storage unit 103 to the virtual memory of the write unit 102. At the same time, the aforementioned storage unit 103 is also mapped to the virtual memory of the read unit 104. That is, a first mapping relationship is established between the storage unit 103 and the write unit 102, and a second mapping relationship is established between the storage unit 103 and the read unit 104, so that the write unit 102 and the read unit 104 can operate on the storage unit 103 by operating on their own virtual memory.

[0101] The raw data processing unit 101 is configured to process each frame of raw eye-tracking data in the data frame sequence sequentially to generate data nodes that correspond one-to-one with each frame of raw eye-tracking data.

[0102] Specifically, for any frame of raw eye-tracking data, the raw data processing unit is configured as follows:

[0103] Based on the original eye-tracking data of the frame and the key obtained from the encryption key area, encrypted original eye-tracking data is obtained. A corresponding checksum is generated based on the encrypted original eye-tracking data to generate the data node corresponding to the original eye-tracking data of that frame. Simultaneously, based on the time corresponding to the original eye-tracking data and its associated physical quantity, a timestamp and physical quantity type are obtained; based on the data length of the encrypted original eye-tracking data, the data length is obtained. Thus, a complete data node is obtained, which, as described above, includes: timestamp, data length, physical quantity type, encrypted original eye-tracking data, and checksum. For example, the above encryption of the original eye-tracking data can be implemented using the AES-128 encryption algorithm, where the 128-bit key refers to the symmetric encryption key stored in the encryption key area of ​​storage unit 103.

[0104] The write unit 102 is configured to write each data node sequentially into the data buffer according to the first mapping relationship between the write unit 102 and the storage unit 101.

[0105] Specifically, for any frame of raw eye-tracking data, the writing unit 102 is configured as follows:

[0106] Based on the write pointer `head` in the metadata area, obtain the write position of the data node corresponding to the original eye-tracking data of this frame in the data buffer. Write the data node corresponding to the original eye-tracking data of this frame to the write position in the data buffer indicated by the write pointer `head` in the metadata area. Update the write pointer and the current counter node in the metadata area to complete the writing of the data node corresponding to the original eye-tracking data of this frame. The current counter node, as described above, represents the number of readable data nodes. For example, updating the write pointer of the metadata area and the current counter node is implemented using an atomic operation (InterlockedConpareExchange), thereby avoiding data corruption caused by the competition between the write unit 102 and the read unit 104. In traditional locking mechanisms, when either the write unit 102 or the read unit 104 operates on the storage unit 103, the other will be suspended, i.e., put into a sleep state, waiting for the operation to be completed before it can be executed. This causes the write and read processes to queue and wait, resulting in transmission delay. Atomic operation allows the write unit 102 and the read unit 104 to perform operations on the storage unit 103 simultaneously without blocking. Therefore, compared with traditional locking mechanisms, atomic operation can avoid deadlock and reduce latency.

[0107] The read unit 104 is configured to read a readable data node when a readable data node exists in the data buffer, based on a second mapping relationship between the read unit 104 and the storage unit 101.

[0108] Specifically, for any readable data node, the read unit 104 is configured as follows:

[0109] Based on the read pointer `tail` corresponding to the read unit 104 in the metadata area, obtain the position of the readable data node in the data buffer. Read the readable data node from its position in the data buffer.

[0110] The eye-tracking data generation unit 105, corresponding to the reading unit 104, is configured to generate first eye-tracking data corresponding to the readable data node based on the readable data node, and upload it to the display control module so that the display control module can update the display content based on the first eye-tracking data.

[0111] Specifically, for any readable data node, the eye-tracking data generation unit 105 is configured as follows:

[0112] The readable data node is verified using a checksum. If the verification passes, the readable data node is decrypted using the key in the encryption key area to generate decrypted data, which is then used as the first eye-tracking data. If the verification fails, the data node is discarded.

[0113] In related technologies, eye-tracking data transmission often uses Socket. As a network-layer-based simultaneous method, it requires binding to IP addresses and ports, resulting in redundant design in local communication scenarios. Moreover, it lacks a built-in client authentication mechanism, which can easily lead to unauthorized reading units illegally obtaining eye-tracking data, posing a risk of data leakage. In addition, actual tests have shown that the Socket transmission method requires multiple large data copies, resulting in a relatively long latency of about 3ms-4ms for transmitting a single frame of eye-tracking data. For high frame rate display devices, the processing time for each frame is required to be very short. For example, a 120Hz display device needs to refresh the display content every 8ms. When the data transmission latency reaches 3-4ms, it means that the data processing and calculation time of the display device is only about 4ms, which severely compresses the subsequent calculation time, resulting in poor real-time performance of the display device.

[0114] The eye-tracking data transmission device disclosed herein, by designing a storage unit in which the writing and reading units can operate in parallel, allows the writing unit to process the raw eye-tracking data according to specific encryption rules and write it into the storage unit based on its mapping relationship with the storage unit. Simultaneously, the reading unit can read the eye-tracking data from the storage unit based on its mapping relationship. Compared to the traditional Socket transmission method, this method eliminates the overhead of multiple large data copies and, due to zero copying, avoids excessively long transmission links, thus significantly improving communication speed. Actual testing shows that the transmission latency for a single frame of raw eye-tracking data using this device can be reduced to 0.2ms-0.5ms. Furthermore, because the storage unit stores an identifier permission list, the identifier of the reading unit can be verified, thus preventing unauthorized reading units or 3D applications from unauthorizedly reading eye-tracking data, improving user privacy and data security. Furthermore, the metadata area employs a digital signature mechanism, and each piece of raw eye-tracking data is encrypted with a key before being written. These two methods respectively prevent the data in the metadata area and the data buffer from being tampered with or stolen, further enhancing data security and meeting the high privacy protection requirements in some scenarios. Moreover, the transmission device disclosed herein supports multiple reading units / 3D applications reading data nodes simultaneously. Each reading unit reads data from the corresponding location according to its own read pointer tail, making it suitable for complex architecture scenarios involving multiple 3D applications, such as scenarios including 3D rendering, eye-tracking data analysis, and other 3D applications.

[0115] Furthermore, in some embodiments, the eye-tracking data transmission device 100 also includes a notification unit. This notification unit is configured to send a notification signal to the reading unit 104 when a data node corresponding to a frame of raw eye-tracking data is written to the data buffer. This notification signal indicates that a readable data node exists in the data buffer. For example, in a specific implementation, the function of this notification unit can be implemented using the kernel event function in the Windows system. This notification unit can trigger an EVENT_DATA_AVAILABLE event after the writing unit 102 writes the data node, and the reading unit 104 can wait for the EVENT_DATA_AVAILABLE event before performing the reading.

[0116] Alternatively, in some embodiments, the eye-tracking data transmission device 100 further includes a query unit. This query unit is configured to periodically query the metadata area to determine whether the current counting node's count in the metadata area is greater than 0. If the current counting node's count is greater than 0, it is determined that a readable data node exists in the data buffer. In other words, the query unit employs a polling mechanism, querying the current counting node at a certain frequency to determine whether a readable data node exists.

[0117] Both the notification unit and the query unit mentioned above can be used to determine whether readable data nodes exist; in specific implementation, only one needs to be selected. However, compared to the query unit, the notification unit does not require periodic queries, thus avoiding the resource consumption caused by polling.

[0118] Furthermore, in some embodiments, the eye-tracking data transmission device also includes a verification unit. This verification unit is configured to determine whether the reading unit 104 has read permissions based on the identifier permission list in the metadata area and the identifier of the reading unit 104 before the reading unit 104 reads the readable data node. If the reading unit has read permissions, the verification unit synchronizes the key in the encryption key area to the eye-tracking data generation unit 105 corresponding to the reading unit, so that it can generate first eye-tracking data; if the reading unit does not have read permissions, it generates a security alarm.

[0119] In simple terms, this verification unit can modify the identifier permission list of the metadata area based on the identifier of the accessible reading unit 104. Before the reading unit 104 reads a data node, the verification unit needs to determine whether the identifier of the reading unit 104 is in the identifier permission list. If it is, the reading unit 104 is an authorized reading unit; otherwise, it is an unauthorized reading unit. For example, the specific implementation of generating a security alert can utilize the Windows system's memory protection function (PAGE_READWRITE SECURITY_DESCRIPTOR). When this function is called, it will issue an ERROR_ACCES_DENIDE error, which can be used as a security alert. This can prevent unauthorized applications from obtaining eye-tracking data, thereby ensuring user privacy and data security.

[0120] In related technologies, raw eye-tracking data is obtained through collection and calculation using eye-tracking devices. These devices typically include a camera and a data processing module. The camera captures images of both eyes at a specific frequency, while the data processing module calculates raw eye-tracking data based on these images. However, cameras typically have image capture frame rates of 60Hz or 90Hz, meaning they can only generate 60 or 90 frames of raw eye-tracking data per second. In contrast, common 3D applications often use refresh rates of 120Hz or 240Hz, requiring content updates 120 or 240 times per second. This results in a situation where some refresh moments in 3D applications lack available eye-tracking data, preventing effective content updates and causing a stuttering or lag-like experience for the user.

[0121] To address the aforementioned issues, the eye-tracking data transmission apparatus provided in this disclosure further includes a data frame sequence generation unit.

[0122] In summary, the data frame sequence generation unit can generate several frames of predicted eye movement data based on several frames of measured eye movement data calculated by the eye-tracking device, so that the number of frames of original eye movement data in a data frame sequence matches the refresh rate of the 3D application. Taking the aforementioned camera with a 90Hz acquisition frame rate as an example, it can acquire 90 frames of measured eye movement data per second. The data frame sequence generation unit can calculate 150 frames of predicted eye movement data based on these 90 frames of measured eye movement data, and arrange the 90 frames of measured eye movement data and the 150 frames of predicted eye movement data in chronological order as a data frame sequence.

[0123] Specifically, the data frame sequence includes N frames of raw eye-tracking data, where N is an integer not less than 1; m frames of raw eye-tracking data within the N frames are measured eye-tracking data, and (Nm) frames of raw eye-tracking data are predicted eye-tracking data; where m is an integer not less than 1 and not greater than N. The aforementioned data frame sequence generation unit is configured to acquire m frames of measured eye-tracking data; use a pre-trained interpolation algorithm to calculate (Nm) frames of predicted eye-tracking data based on the m frames of measured eye-tracking data; and arrange the m frames of measured eye-tracking data and the (Nm) frames of predicted eye-tracking data according to the time sequence corresponding to each frame to generate the data frame sequence.

[0124] Examples of pre-trained interpolation algorithms include one or more of the following: linear interpolation algorithm, quadratic interpolation algorithm, trigonometric interpolation algorithm, Newton interpolation algorithm, and Gaussian interpolation algorithm.

[0125] During training, the parameters of the interpolation algorithm can be dynamically adjusted based on the deviation between the predicted eye-tracking data every 100ms and the measured eye-tracking data at that moment. This improves the accuracy of the interpolation algorithm and prevents the prediction error from accumulating in subsequent calculations, thus affecting the calculation results of 3D applications. For example, the training termination condition can be set as follows: the deviation between the predicted eye-tracking data every 100ms and the measured eye-tracking data at that moment is less than a preset value. For instance, the error between the predicted pupil coordinates and the measured pupil coordinates must be less than 0.5° for the interpolation algorithm training to be considered complete.

[0126] It should be noted that this disclosure does not impose any restrictions on the interpolation algorithm for calculating predicted eye-tracking data, or on the rules for inserting predicted eye-tracking data. In some examples, a fixed number of predicted eye-tracking data can be uniformly inserted between any two adjacent measured eye-tracking data points; in other examples, the number of predicted eye-tracking data points between two adjacent measured eye-tracking data points is not fixed. For example, the change between the (i-1)th and the ith measured eye-tracking data points is small, so the number of predicted eye-tracking data points inserted between them can be small, while the change between the ith and the (i+1)th measured eye-tracking data points is drastic, so the number of predicted eye-tracking data points inserted between them can be large, where i is an integer greater than or equal to 2 and less than or equal to N-1. Furthermore, continuing with the "240Hz" frame rate example above, when the refresh rate of a 3D application is 240Hz, it means that it needs to refresh once every 4.17ms, that is, it needs to acquire eye-tracking data once every 4.17ms. However, due to the randomness of insertion, the time of the 240 data points in the data frame sequence is not one every 4.17ms. At this time, the time of each data node in the storage unit 102 is also not one every 4.17ms. In this case, the 3D application can use the data node closest to the current time. For example, if the current time is 4.17ms, and the storage unit 102 has data nodes at times of 4ms, 5ms, 6.2ms, etc., then the 3D application can choose the data node corresponding to the 4ms time as the basis for its current refresh. Furthermore, it should be noted that if the current time is 4.17ms, and the storage unit 102 contains data nodes at times such as 3ms, 4ms, and 5ms, the data node at 3ms is the actual eye-tracking data, while the data node at 4ms is the predicted eye-tracking data. In this case, the 3D application can choose the actual eye-tracking data as the basis for the current refresh instead of selecting the data with the closest time according to the time priority principle, thus improving the accuracy of the refresh.

[0127] The eye-tracking data transmission device provided in this disclosure, by setting up a data frame sequence generation unit, generates multiple frames of predicted eye-tracking data based on the refresh frame rate of the 3D application and several frames of measured eye-tracking data, as supplementary data to the measured eye-tracking data, and fuses the measured eye-tracking data and the predicted eye-tracking data into a complete data frame sequence, so that the number of frames in the data frame sequence matches the refresh frame rate of the 3D application. The effects are: on the one hand, it can enable the 3D application to obtain the corresponding eye-tracking data at each refresh time, thereby improving the timeliness and real-time response of the 3D application to human eye movement; on the other hand, it eliminates the need to use a high frame rate and high performance camera, thereby reducing hardware costs.

[0128] Based on the same inventive concept, in a third aspect, this disclosure provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the eye-tracking data transmission method described in any of the embodiments of the first aspect above.

[0129] Based on the same inventive concept, in a fourth aspect, this disclosure provides a 3D display device.

[0130] Figure 7 This is a schematic diagram of the modular structure of the display device provided in this disclosure. Figure 7 As shown, the display device includes: a computer-readable storage medium 100, a processor 400, a display control module 200, and an eye-tracking device 300. The computer-readable storage medium 100 stores a computer program, and the processor 400 is configured to implement the eye-tracking data transmission method described in any embodiment of the first aspect when executing the computer program, to generate a first data frame sequence based on each frame of original eye-tracking data in the data frame sequence, and upload it to the display control module 200. The first eye-tracking data frame sequence includes multiple frames of first eye-tracking data that correspond one-to-one with the original eye-tracking data; the specific implementation process can be referred to above and will not be repeated here. For each frame of first eye-tracking data, the display control module 200 is configured to calculate its corresponding disparity adjustment coefficient based on that frame of first eye-tracking data, and update the display content once based on the disparity adjustment coefficient.

[0131] It is understandable that the principle behind achieving naked-eye 3D is that after both eyes receive images with parallax, the brain automatically fuses these images to generate a 3D image. Therefore, before each refresh, the display control module 200 needs to determine the parallax adjustment coefficient based on the first eye movement data (such as the pupil coordinates of both eyes), calculate the parallax of the binocular images based on the parallax adjustment coefficient and the current pixel depth value, and update the binocular images to generate the best 3D image suitable for the current human eye coordinates.

[0132] In some examples, the data frame sequence includes N frames of raw eye-tracking data, where N is an integer not less than 1; m frames of raw eye-tracking data within the N frames are measured eye-tracking data, and (Nm) frames of raw eye-tracking data are predicted eye-tracking data; where m is an integer not less than 1 and not greater than N. The eye-tracking device 300 is configured to acquire m frames of binocular images at a preset frequency within a time period (e.g., 1 second) corresponding to a data frame sequence, and calculate m frames of measured eye-tracking data based on the m frames of binocular images.

[0133] Specifically, the eye-tracking device 300 includes an image acquisition unit and an eye-tracking data calculation unit. The image acquisition unit is used to acquire images of both eyes; the eye-tracking data calculation unit is used to perform feature recognition and detection based on the acquired images of both eyes, and to calculate the measured eye-tracking data (including pupil coordinates, motion vectors, etc.).

[0134] As a concrete example, the image acquisition unit can employ a camera with a specific frequency. For instance, high-specification cameras with frame rates of 90Hz, 120Hz, and above are preferred to reduce the acquisition interval between adjacent frames. Taking 90Hz as an example, the acquisition time interval between two adjacent images is approximately 11ms, while a 60Hz camera requires 16.7ms to acquire two adjacent frames. Therefore, high frame rate cameras can improve data acquisition speed and efficiency, making them suitable for scenarios with rapid head movements. Preferably, the image acquisition device can employ a global shutter camera, which can avoid image blurring in fast eye-movement scenarios using a rolling shutter, ensuring the validity of each frame of data.

[0135] In related technologies, after acquiring binocular images, data storage, memory copying, and I / O parsing are performed before processing the binocular images. This results in a delay in the transmission of images to the eye-tracking data processing unit. To further reduce the delay caused by data acquisition and transmission, this disclosure allows for direct acquisition of binocular images via a USB / Type-C direct connection to a computer after acquisition.

[0136] Furthermore, to further improve the speed of eye-tracking data acquisition, the eye-tracking data processing unit provided in this disclosure downsamples the resolution of the binocular images after receiving them, for example, reducing the resolution from 720p to 360p, thereby reducing the computational load and increasing the processing speed. In addition, when the binocular images are RGB images, the eye-tracking data processing unit converts the RGB images into single-channel grayscale images, eliminating redundant calculations for color channels. All of these operations can significantly shorten the processing time of the eye-tracking data processing unit for a single frame image, avoiding a reduction in the eye-tracking efficiency and real-time performance of the display device due to computational delays.

[0137] It is understood that the above embodiments are merely exemplary implementations used to illustrate the principles of the present invention, and the present invention is not limited thereto. For those skilled in the art, various modifications and improvements can be made without departing from the spirit and essence of the present invention, and these modifications and improvements are also considered to be within the scope of protection of the present invention.

Claims

1. A method for transmitting eye movement data, applied to a 3D display device, comprising the steps of: The method comprises: providing a storage unit; the storage unit at least comprises a data buffer; processing each frame of raw eye movement data in a sequence of data frames in turn to generate a data node corresponding to each frame of raw eye movement data; through a write unit, according to the first mapping relationship between the write unit and the storage unit, writing each data node into the data buffer of the storage unit in turn; through at least one reading unit, according to the second mapping relationship between the reading unit and the storage unit, when there is a readable data node in the data buffer, reading the readable data node; based on the readable data node, generating the first eye movement data corresponding to the readable data node, and uploading to the display control module of the 3D display device, so that the display control module updates the display content of the 3D display device based on the first eye movement data.

2. The method of claim 1, wherein, The storage unit further comprises an encryption key area; for any frame of raw eye movement data, processing the frame of raw eye movement data to generate a data node corresponding to the frame of raw eye movement data, comprising: based on the frame of raw eye movement data and the key obtained from the encryption key area, obtaining the encrypted raw eye movement data; according to the encrypted raw eye movement data, generating a corresponding check code to generate the data node corresponding to the frame of raw eye movement data; the data node at least comprises the encrypted raw eye movement data and the check code.

3. The method of claim 2, wherein, The storage unit further comprises a metadata area; for any data node corresponding to the frame of raw eye movement data, writing the data node into the data buffer, comprising: according to the write pointer of the metadata area, obtaining the writing position of the data node corresponding to the frame of raw eye movement data in the data buffer; writing the data node corresponding to the frame of raw eye movement data into the writing position in the data buffer indicated by the write pointer of the metadata area; updating the write pointer of the metadata area and the current count node to complete the writing of the data node corresponding to the frame of raw eye movement data; the current count node represents the number of readable data nodes.

4. The method of claim 3, wherein, The reading of the readable data node comprises: according to the read pointer corresponding to the reading unit in the metadata area, obtaining the position of the readable data node in the data buffer; reading the readable data node from the position of the readable data node in the data buffer.

5. The method of claim 4, wherein, The generation of the first eye movement data corresponding to the readable data node comprises: verifying the readable data node according to the check code; after verification, decrypting the readable data node according to the key of the encryption key area to generate decryption data, and taking the decryption data as the first eye movement data.

6. The method according to any one of claims 1-5, characterized in that, When writing the data node into the data buffer of the storage unit, it also comprises: sending a notification signal to the reading unit; the notification signal indicates that there is a readable data node in the data buffer.

7. The method according to any one of claims 1-5, characterized in that, The storage unit further comprises a metadata area; before the reading of the readable data node, it also comprises: Periodically query the metadata area, determine whether the current count node in the metadata area is greater than 0, and when the current count node is greater than 0, determine that there is readable data node in the data buffer area.

8. The method according to any one of claims 1-5, characterized in that, The storage unit further comprises a metadata area and an encryption key area. Before the reading unit reads the readable data node, further comprising: According to the identification number permission list in the metadata area and the identification number of the reading unit, it is judged whether the reading unit has reading permission, and when the reading unit has reading permission, the key of the encryption key area is synchronized to the eye movement data generation unit corresponding to the reading unit, so that it generates the first eye movement data; when the reading unit does not have reading permission, a security alarm prompt is generated.

9. The method according to any one of claims 1-5, characterized in that, The data frame sequence comprises N frames of original eye movement data, N is an integer not less than 1; m frames of the original eye movement data in N frames are measured eye movement data, and (N-m) frames of the original eye movement data are predicted eye movement data; m is an integer not less than 1 and not greater than N; Before each frame of original eye movement data in the data frame sequence is processed in turn, further comprising: Generating the data frame sequence; The generating the data frame sequence comprises: Obtaining m frames of the measured eye movement data; Using a pre-trained interpolation algorithm, (N-m) frames of the predicted eye movement data are calculated according to the m frames of the measured eye movement data; m frames of the measured eye movement data and (N-m) frames of the predicted eye movement data are arranged in order according to the time of each frame to generate the data frame sequence.

10. The method of claim 9, wherein, The pre-trained interpolation algorithm comprises: One or more of linear interpolation algorithm, quadratic interpolation algorithm, triangular interpolation algorithm, Newton interpolation algorithm and Gaussian interpolation algorithm.

11. A computer readable storage medium, characterized in that, A computer program is stored thereon, and the computer program is executed by a processor to implement the eye movement data transmission method of any one of claims 1-10.

12. A 3D display device, characterized by Comprise: A processor, a computer readable storage medium, and a display control module; The computer readable storage medium stores a computer program, and the processor is configured to implement the eye movement data transmission method of any one of claims 1-10 when executing the computer program to generate a first data frame sequence according to each frame of original eye movement data in the data frame sequence and upload to the display control module; the first data frame sequence comprises a plurality of frames of first eye movement data corresponding to the original eye movement data one by one; For each frame of the first eye movement data, the display control module is configured to calculate the parallax adjustment coefficient corresponding to the frame according to the first eye movement data, and update the display content once according to the parallax adjustment coefficient.

13. The 3D display device of claim 12, wherein, The data frame sequence comprises N frames of original eye movement data, N is an integer not less than 1; m frames of the original eye movement data in N frames are measured eye movement data, and (N-m) frames of the original eye movement data are predicted eye movement data; m is an integer not less than 1 and not greater than N; The 3D display device further comprises: An eye movement tracking device is configured to collect m frames of binocular images at a preset frequency within a time corresponding to a sequence of data frames, and calculate m frames of measured eye movement data based on the m frames of binocular images.

Images