Eye movement data transmission method and electronic device

Abstract

The application discloses an eye movement data transmission method and an electronic device, relates to the technical field of human-computer interaction, and comprises the following steps: after acquiring eye movement sensing data, if the eye movement sensing data meets a predetermined condition, generating structured eye movement event data, the structured eye movement event data comprising first indication information representing an eye movement event type, and transmitting the structured eye movement event data to an upper layer of an operating system.

Description

Technical Field

[0001] This application relates to the field of eye-tracking technology, and more particularly to an eye-tracking data transmission method and electronic device. Background Technology

[0002] Currently, eye-tracking technology, by capturing the user's eye movement trajectory and gaze point, has been widely used in applications (APPs) in various fields of electronic devices. Typically, raw eye movement data (such as eye image data) is directly transmitted to each APP that requires eye-tracking interaction, and each APP processes it itself to achieve eye-tracking interaction.

[0003] However, the current method of repeatedly transmitting raw eye-tracking data to various apps for secondary processing wastes system computing resources, transmission bandwidth, and power consumption. Different app developers have different processing logics, resulting in inconsistent or even contradictory eye-tracking interaction experiences for users using different apps. Furthermore, when multiple apps simultaneously call eye-tracking algorithms and associated hardware, it is easy to cause contention and conflicts, leading to data loss, response failures, or even application crashes, which seriously affects the reliability of interaction and system stability. Summary of the Invention

[0004] In view of the above problems, this application provides the following solution:

[0005] The first aspect of this application provides a method for transmitting eye-tracking data, comprising:

[0006] Acquire eye-tracking perception data;

[0007] If the eye movement perception data meets predetermined conditions, structured eye movement event data is generated, and the structured eye movement event data includes first indication information representing the type of eye movement event;

[0008] The structured eye-tracking event data is transmitted to the upper layers of the operating system.

[0009] In one possible implementation, if the eye-tracking perception data meets predetermined conditions, structured eye-tracking event data is generated, including:

[0010] The eye movement perception data is analyzed to determine changes in eye movement state;

[0011] If the eye movement state change meets the predetermined conditions for any eye movement event type, the corresponding structured eye movement event data is generated;

[0012] The predetermined conditions are determined based on at least one of a time dimension threshold and a spatial dimension threshold.

[0013] In one possible implementation, the structured eye-tracking event data includes data corresponding to at least one of the following eye-tracking event types:

[0014] Eye-closing events, eye-opening events, gaze events, blinking events, continuous blinking events, and gaze beyond device boundaries events.

[0015] In one possible implementation, the eye movement state change includes at least one of the following:

[0016] The transition between the first state and the second state;

[0017] The state transition pair formed by the first state and the second state;

[0018] Changes in the coordinates of the focal point of the line of sight;

[0019] The first state and the second state are two different states, one with eyes open and the other with eyes closed.

[0020] The state transition pair includes a first state transition and a second state transition that is adjacent to it and has the opposite direction.

[0021] In one possible implementation, the structured eye-tracking event data further includes second indication information, which represents an attribute of the eye-tracking event type, the attribute including at least one of the following:

[0022] Timestamps of eye-tracking events;

[0023] Coordinates of the focal point of the gaze during an eye-tracking event;

[0024] User identifier that generated the eye-tracking event;

[0025] The eye source type that generates the eye movement event, wherein the eye source type includes the left eye, the right eye, or both eyes.

[0026] In one possible implementation, if the eye-tracking perception data meets predetermined conditions, structured eye-tracking event data is generated, including:

[0027] Detect the eye movement state from the first state to the second state;

[0028] Determine the duration for which the second state is maintained;

[0029] If the duration exceeds the first time threshold, corresponding structured eye movement event data is generated. The first indication information in the eye movement event data is used to indicate the closed eye event or the open eye event corresponding to the second state.

[0030] In one possible implementation, if the eye-tracking perception data meets predetermined conditions, structured eye-tracking event data is generated, including:

[0031] Detect at least one pair of state transitions occurring during eye movement;

[0032] Obtain the state interval duration between the first state transition and the second state transition in each state transition pair;

[0033] If the state interval of a state transition pair occurs within the valid time interval, corresponding structured eye movement event data is generated, and the first indication information in the eye movement event data is used to indicate the blink event;

[0034] If the state interval of a preset number of consecutive state transitions occurring within a preset time window is within the effective time interval, corresponding structured eye movement event data is generated. The first indication information in the eye movement event data is used to indicate consecutive blinking events, and the second indication information includes the number of consecutive blinks.

[0035] In one possible implementation, if the eye-tracking perception data meets predetermined conditions, structured eye-tracking event data is generated, including any of the following:

[0036] If the coordinate movement distance of the gaze focus is less than the spatial distance threshold, and the duration of the state is greater than the second time threshold, corresponding structured eye movement event data is generated. The first indication information in the eye movement event data is used to indicate the gaze event, and the second indication information includes the coordinates of the gaze focus when the gaze event occurs. The second indication information is determined based on multiple gaze coordinates collected within the generation time window of the gaze event.

[0037] If the coordinates of the gaze focus exceed the device boundary range and the duration of this state is greater than a third time threshold, corresponding structured eye movement event data is generated. The first indication information in the eye movement event data is used to indicate the gaze exceeding the device boundary event, and the second indication information includes the direction of excess or the amount of excess offset.

[0038] In one possible implementation, the transmission of the structured eye-tracking event data to the upper layer of the operating system includes any of the following methods:

[0039] The structured eye-tracking event data is stored in the operating system's event pool for the upper layers of the operating system to read.

[0040] The structured eye-tracking event data is pushed directly to the upper layer of the operating system through an internal communication process.

[0041] A second aspect of this application provides an electronic device, comprising:

[0042] At least one visual sensor is used to collect eye-tracking perception data;

[0043] A processing device, coupled to the at least one vision sensor, is configured to:

[0044] Acquire the eye-tracking perception data;

[0045] If the eye movement perception data meets predetermined conditions, structured eye movement event data is generated, and the structured eye movement event data includes first indication information representing the type of eye movement event;

[0046] The structured eye-tracking event data is transmitted to the upper layers of the operating system. Attached Figure Description

[0047] The above and other features, advantages, and aspects of the embodiments of this disclosure will become more apparent from the accompanying drawings and the following detailed description. Throughout the drawings, the same or similar reference numerals denote the same or similar elements. It should be understood that the drawings are schematic, and the originals and elements are not necessarily drawn to scale.

[0048] Figure 1 This is a flowchart illustrating an eye-tracking data transmission method proposed in Embodiment 1 of the application.

[0049] Figure 2 This is a flowchart illustrating an eye-tracking data transmission method proposed in Embodiment 2 of the application.

[0050] Figure 3 This is a flowchart illustrating an eye-tracking data transmission method proposed in Embodiment 3 of the application.

[0051] Figure 4 This is a flowchart illustrating an eye-tracking data transmission method proposed in Embodiment 4 of the application.

[0052] Figure 5 This is a flowchart illustrating an eye-tracking data transmission method proposed in Embodiment 5 of the application.

[0053] Figure 6 This is a schematic diagram of the structure of an eye-tracking data transmission device proposed in the embodiments of the application;

[0054] Figure 7 This is a schematic diagram of the hardware structure of an electronic device applicable to the eye-tracking data transmission method proposed in this application;

[0055] Figure 8 This is a schematic diagram of the software architecture of an electronic device applicable to the eye-tracking data transmission method proposed in this application. Detailed Implementation

[0056] The embodiments of this application are described below with reference to the accompanying drawings. The terminology used in the implementation section of this application is only for explaining specific embodiments and is not intended to limit the application. Those skilled in the art will recognize that, with technological advancements and the emergence of new scenarios, the technical solutions provided in the embodiments of this application are also applicable to similar technical problems.

[0057] The terms "first," "second," etc., used in the context of this application and the aforementioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such terms are interchangeable where appropriate; this is merely a way of distinguishing objects with the same attributes in the embodiments of this application. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion, so that a process, method, system, product, or apparatus that comprises a series of units is not necessarily limited to those units, but may include other units not explicitly listed or inherent to those processes, methods, products, or apparatuses.

[0058] Reference Figure 1 This is a flowchart illustrating an eye-tracking data transmission method proposed in Embodiment 1 of this application. The method proposed in this application can be executed by the underlying operating system of an electronic device (such as a mobile phone, tablet, VR / AR device, smart home device, smart car, or other user terminal, etc.). This underlying system can be any software / hardware / firmware layer located above or below the operating system layer, such as the hardware layer, driver layer, system kernel layer, or hardware abstraction layer. This addresses various technical problems caused by directly transmitting raw eye-tracking data to the application layer. Figure 1 As shown, the eye-tracking data transmission method proposed in this embodiment may include:

[0059] Step S11: Acquire eye-tracking perception data;

[0060] The eye-tracking perception data can be perception data containing the human eye region collected by at least one visual sensor (i.e., an eye-tracking sensor), such as a front-facing camera, infrared camera, depth camera, or dedicated eye-tracking module. This data can include, but is not limited to, at least one of the following: eye image data, depth data, infrared data, gaze point coordinate data, pupil position data, and eyelid opening / closing. Therefore, the eye-tracking perception data obtained in this application can be raw perception data collected by the visual sensor, image data obtained after preprocessing the raw eye image (such as noise reduction, face detection, and eye region cropping), or feature data extracted from the raw eye image (such as gaze point coordinate data, pupil position data, etc.). This application does not limit the acquisition method and content of the eye-tracking perception data and can determine it based on the hardware configuration of the electronic device and application requirements.

[0061] Step S12: If the eye movement perception data meets the predetermined conditions, generate structured eye movement event data, which includes first indication information representing the type of eye movement event.

[0062] Step S13: Transmit the structured eye-tracking event data to the upper layer of the operating system.

[0063] Eye-tracking data acquired directly is typically generated continuously at a high frame rate, requiring significant bandwidth and computing power for processing. To address this, this application avoids multiple applications (APPs) repeatedly calling hardware interfaces and fetching eye-tracking data, saving data transmission bandwidth at the source and eliminating the risk of conflicts caused by multiple applications competing for hardware resources. This application pre-configures several eye-tracking event triggering conditions (i.e., predetermined conditions). Before transmitting eye-tracking data to the upper layer of the operating system, the eye-tracking data is analyzed. Only when the predetermined conditions are met will structured eye-tracking event data be generated and transmitted, instead of continuously transmitting each frame of raw eye-tracking data. If the predetermined conditions are not met, eye-tracking data can continue to be monitored without generating event data. This conditional triggering mechanism reduces the data reporting frequency by several orders of magnitude, significantly saving system bandwidth and power consumption.

[0064] The upper layer of the operating system can be the application layer where various apps installed on the electronic device reside, or a service layer acting as middleware, such as a dedicated service module for managing eye-tracking events, which provides the application layer with the eye-tracking event data or processing results required by the apps. This application allows for different transmission methods for the structured eye-tracking event data to different upper layers, providing each app with a clear, stable, and unified data (structured eye-tracking event data) interface. This eliminates the need for upper layers to concern themselves with changes to the underlying hardware and algorithms; they only need to process the predefined eye-tracking events, thus improving development efficiency and system compatibility.

[0065] The eye-tracking event data generated by this application can be organized according to a predefined data structure (transmission protocol), which at least includes first indication information representing the type of eye-tracking event. The first indication information can be, but is not limited to, binary instruction codes, such as 0x1 indicating closed eyes, 0x2 indicating open eyes, 0x3 indicating fixation, and 0x4 indicating blinking. The term "structured" indicates that the event data generated by this application has a predefined, fixed format, such as a structured event data packet (e.g., a binary packet). This packet can be designed to contain fields corresponding to relevant event information. This design makes the event data a self-describing, standardized information unit, facilitating rapid parsing by upper layers. Furthermore, this lightweight structured data significantly reduces data transmission volume and saves data transmission bandwidth compared to a large amount of raw eye-tracking perception data.

[0066] This application decouples underlying event generation from upper-layer event consumption. It centralizes the complex, resource-intensive eye-tracking algorithm (the logic for processing eye-motion perception data) at the system's bottom layer, executing it once. The standardized results can then be simultaneously accessed by multiple upper-layer apps. This eliminates the need for each app to occupy its own visual sensor and algorithm resources. Even when multiple applications simultaneously request eye-tracking functionality, issues such as camera contention or algorithm repetitive execution leading to crashes and data loss do not occur, resolving the problem of concurrent resource conflicts among multiple applications. Furthermore, the upper layer of the operating system receives and parses structured eye-motion event data encapsulated in the same data structure. There is no need to integrate and run independently developed eye-motion recognition algorithms, define event judgment logic, or process raw data, reducing application development costs, saving significant repetitive computation overhead, and ensuring consistent user experience across applications. Specifically, it guarantees the consistency of interaction logic such as "what constitutes a valid blink" and "what constitutes a stable gaze" across the entire system, resolving the problem of fragmented user experience.

[0067] Moreover, structured event data is small in size and quick to parse, allowing applications to obtain eye-tracking events with extremely low latency. This makes it suitable for scenarios with high real-time requirements, such as game control, fatigue detection, and accessibility interaction, thus improving system real-time performance and response speed. Furthermore, because the event data is a predefined data structure, it supports flexible event expansion. This means that as technology advances, new event types (such as saccade events and tracking events) or other fields (such as pupil diameter and fixation duration) can be easily added without affecting the compatibility of existing application events with known data types.

[0068] Therefore, this application fundamentally solves the problems of fragmentation / inconsistency in eye-tracking data processing, resource waste, and competition / call conflicts that exist in the traditional method of directly transmitting the original eye-tracking perception data to the upper-layer application for secondary processing, through the technical means of condition triggering, structured encapsulation, and system-level shared transmission. This improves system interaction efficiency, ensures interaction consistency, and enhances system stability, providing reliable underlying technical support for the popularization of eye-tracking interaction technology in consumer electronic devices.

[0069] Reference Figure 2 This is a flowchart illustrating an eye-tracking data transmission method proposed in Embodiment 2 of this application. This embodiment provides a detailed description of a possible implementation method for intelligently and systematically generating structured event data with predetermined semantics from raw eye-tracking perception data. Figure 2 As shown, this implementation method may include, but is not limited to:

[0070] Step S21: Analyze the eye movement perception data to determine changes in eye movement state;

[0071] In this embodiment, by analyzing eye-motion perception data from visual sensors in real time, the basic state units of the human eye can be extracted, and the physical changes that occur in the human eye in a specific dimension and can be interpreted by the machine can be determined. That is, continuous, high-dimensional eye-motion perception data is reduced in dimension and abstracted into discrete, low-dimensional, computable state sequences (such as "eye open-eye closed-eye open", "eye focus coordinates move from A to B", etc.). Optionally, eye-motion state changes can be extracted from eye-motion perception data through a lightweight eye-motion recognition model or other computer vision processing models. This application does not limit the implementation method of step S21.

[0072] In one possible implementation, this application can extract eye movement state changes from eye movement perception data in one or more dimensions, including but not limited to changes in eyelid state, changes in gaze focus coordinates, fixation point stability, and blinking (eyelid state transition) behavior. Eyelid state changes can be determined by analyzing eye images or infrared emission intensity to determine whether the eye is currently open or closed, and the time point of the state transition is recorded. For example, a change from "open eye state" in the previous frame to "closed eye state" in the current frame constitutes a closed eye state transition; conversely, an open eye state transition occurs. Blinking behavior can refer to a detected complete sequence such as open-closed-open, recording the duration of eye closure (state interval duration) to distinguish prolonged eye closure behavior.

[0073] Optionally, this application can use the pupil-corneal reflex method or a deep learning model to calculate the two-dimensional / three-dimensional coordinates of the current gaze focus in the screen or spatial coordinate system, and track the coordinate movement distance, trajectory / direction, and speed between adjacent frames to determine the change in gaze focus coordinates. For gaze stability, the dispersion of the gaze focus within a continuous time window (such as movement radius, standard deviation, etc.) can be calculated to determine whether the user is gazing at a certain area. If the gaze focus coordinates are within that area within the continuous time window, it indicates that the human eye is in a gaze state.

[0074] Therefore, the analysis results of the above dimensions are represented in the form of changes in eye movement state. For example: "switching from open eyes to closed eyes" (state transition), "the closed eye state lasted for 500ms" (state duration); "the gaze focus moved by 50 pixels" (coordinate change); "three open-closed-open eye sequences were completed within 1 second" (alternating sequence / multiple adjacent state transition pairs), etc. This application does not limit the content and method of representing changes in eye movement state.

[0075] Step S22: If the change in eye movement state meets the predetermined conditions for any eye movement event type, generate the corresponding structured eye movement event data; the predetermined conditions are determined based on at least one of the time dimension threshold and the spatial dimension threshold.

[0076] To accurately and promptly identify eye-tracking events, this application maintains a predefined event rule base. For each type of eye-tracking event (such as at least one of the following: eye-closing event, eye-opening event, fixation event, blinking event (which is just a single blink), continuous blinking event, and gaze beyond device boundary event), one or more "predefined conditions" can be associated. These conditions are typically defined based on time-dimensional thresholds and / or spatial-dimensional thresholds to identify the currently generated eye-tracking event type by matching it with the aforementioned determined eye-tracking state changes. Therefore, by predefining multiple eye-tracking event types and uniformly identifying them at the underlying level, this application facilitates the transmission of structured event data with clear semantics, reduces the complexity of upper-layer processing, and allows upper-layer applications to subscribe to specific eye-tracking event types on demand, avoiding the receipt of irrelevant eye-tracking event data. Multiple apps can also collaboratively adjust their respective states based on the same eye-tracking event (such as pausing navigation interaction or increasing the sensitivity of voice assistants), achieving system-level user experience consistency. This application does not restrict the content of the at least one predefined condition associated with different eye-tracking event types or the method for determining it.

[0077] In one possible implementation, the aforementioned time dimension threshold can be a threshold regarding duration. Predetermined conditions based on this threshold can be used to identify gaze events, as well as sustained eye-closing and eye-opening events. For example, if an "eye-closing (or eye-opening) state" is continuously maintained for more than 800ms (a first time threshold, i.e., the duration threshold / minimum of the closed / open state, but not limited to this value; it can be configured / adjusted based on scenario needs, experience, or user-defined settings), it will be identified as an "eye-closing (or eye-opening) event" to distinguish between brief blinks and intentional eye-closing (or eye-opening), thereby generating structured eye-closing (or eye-opening) event data using the recognition results. If it is shorter than 400ms (a blinking state threshold, but not limited to this value; it can be dynamically adjusted as needed), it may be judged by subsequent logic as part of a "blinking event" and will not be identified as an "eye-closing (or eye-opening) event," thus avoiding unrecognized events.

[0078] Optionally, the time dimension threshold can also be a threshold for the time interval of state transitions. Predefined conditions based on this threshold can be used to identify blink events and consecutive blink events. For example, a "valid blink" is defined as a complete sequence (one adjacent state transition pair) of transitioning from an "open eye state" to a "closed eye state" and then back to an "open eye state," with the duration of the "closed eye state" in between 400ms and 800ms. This allows for the generation of structured blink event data. Here, 400ms and 800ms are key time thresholds to filter out invalid blinks (such as micro-blinks shorter than 400ms or eye closures longer than 800ms). These thresholds can be flexibly adjusted according to actual conditions, and this application does not impose any restrictions on their numerical values.

[0079] Depending on the needs, the predetermined conditions can also be determined in conjunction with a counting dimension threshold, such as the number of eye movement events (or repetition frequency). In this way, predetermined conditions for identifying continuous blinking events can be determined based on the state switching time interval threshold and the eye movement event occurrence threshold (preset count threshold). In this case, based on the identification of a blink event according to the time interval threshold described above, it is further possible to identify whether the cumulative number of valid blinks within a preset time window is greater than the preset count threshold. If so, it can be considered a continuous blinking event. Combining the identification results, structured continuous blinking event data is generated.

[0080] In one possible implementation, the aforementioned spatial dimension threshold can be a threshold relating to the coordinate position of the gaze focus and the radius of movement / stability. Predetermined conditions based on this threshold can be used to identify gaze events. For example, within a time window of T milliseconds, if the distance (or standard deviation of coordinates) the gaze focus moves is less than a threshold of R pixels, the gaze is considered to be in a "stable" state and can be identified as a "gaze event" (such as a user focusing on a certain area), thus generating structured gaze event data. Optionally, the spatial dimension threshold can also be a threshold relating to the range of the device boundary (the display boundary of the electronic device) (or a relative relationship with the boundary range). Specifically determined predetermined conditions can be used to identify gaze events exceeding the device boundary (also known as gaze events exceeding the display boundary, which can be used for edge scrolling or page turning, etc.). In this case, the predetermined condition can be that the gaze focus coordinates (such as two-dimensional coordinates (x, y) in the screen / pixel coordinate system) exceed the boundary of the screen display area. The threshold value can be determined based on the display size of the electronic device, and this application does not impose any limitations on this.

[0081] Based on the above analysis, this application can compare the changes in eye movement state obtained from real-time analysis of eye movement perception data with predetermined conditions associated with various types of eye movement events. For example, if it detects a "transition from an open state to a closed state" and "the closed state lasted for 850ms," it determines that the predetermined conditions for a closed-eye event are met; if it detects that "the radius of movement of the gaze focus is always less than 10 pixels within 300ms," it determines that the predetermined conditions for a fixation event are met. Only when the changes in eye movement state completely match all the predetermined conditions associated with a certain type of eye movement event will the subsequent event generation process be triggered; otherwise, monitoring continues without generating eye movement event data. For example, a blink event requires a state interval duration between 400ms and 800ms. If a closed-eye state is detected to last only 200ms, it is determined to be an invalid blink event, and no blink event data is generated in this case.

[0082] Therefore, this application, by introducing quantifiable spatiotemporal thresholds and combining them with counting thresholds as needed, achieves precise and repeatable semantic definition of eye-movement actions / behaviors. This transforms subjective and vague behaviors such as "glance" and "blink" into objective, programmable, and standardized machine judgments, ensuring the consistency and accuracy of eye-movement event generation. This application does not limit the recognition methods or associated predetermined conditions for various eye-movement events.

[0083] In conjunction with the description of the predetermined conditions associated with various types of eye movement events above, if the first state and the second state are two different states in the open-eye state and the closed-eye state, respectively, the eye movement state changes extracted from the eye movement perception data in this application may include, but are not limited to, at least one of the following: the jump between the first state and the second state (such as the jump from the open-eye state to the closed-eye state, and the jump from the closed-eye state to the open-eye state), the state jump pair formed by the first state and the second state (the state jump pair includes the first state jump and the second state jump adjacent to it and in the opposite direction, that is, the above-mentioned alternating sequence, such as the eye movement behavior when a single / continuous blinking event occurs), the coordinate change of the gaze focus, etc., and may also include the continuous statistical count of the state jump pair as needed.

[0084] Under the condition of meeting the predetermined conditions as described above, but not limited to the method described above, the analysis results / recognition results of the eye movement perception data are encapsulated into eye movement event data that conforms to a predefined data structure (i.e., transmission protocol). For example, corresponding identifiers (indication information) are filled into the predefined fields contained in the data structure. For example, first indication information indicating the type of eye movement event is filled into the event type field. This application does not limit the content of the first indication information. It can be a binary code or other format code / identifier uniquely configured for each eye movement event type, and can be determined based on the similarity of the data structure.

[0085] In summary, this embodiment transforms raw eye movement changes (such as coordinate movement and eyelid closure) into semantically meaningful eye movement events (such as "gaze event," "eye-closing event," and "blink event") through a refined process of "analyzing eye movement state changes → matching predetermined conditions → generating structured events." The upper layer can directly trigger subsequent interaction logic based on the type of eye movement event, eliminating the need for the application to write complex state machines or classifiers to identify eye movement events. For example, a game application can execute in-game skill release upon receiving an "eye-closing event," without needing to know whether the user's left or right eye was closed for how long, thus improving the timeliness and reliability of subsequent application interactions.

[0086] The predefined conditions are determined based on preset multi-dimensional thresholds (such as spatial, temporal, sequential / statistical, etc.), effectively filtering out invalid or interfering eye movement signals. For example, brief, unconscious eye closure (such as blinking) will not trigger an eye closure event; only prolonged, intentional eye closure by the user will be recognized. Slight eye tremors will not trigger a fixation event; only stable fixation exceeding a certain duration will be reported. This significantly improves the accuracy of event recognition and reduces the risk of false triggers due to data noise. The predefined conditions in this application support personalized or scenario-based threshold configuration, allowing users or application developers to dynamically adjust these thresholds (e.g., setting different eye closure duration thresholds for the elderly or children), thereby adapting to different user groups and usage scenarios. When it is necessary to add new eye movement event types (such as at least one "salivation event" or "tracking event"), new eye movement state change patterns and their associated predefined conditions can be defined in the system without modifying the upper-layer interface. The event type field in the structured data can be flexibly expanded, ensuring the system's variability.

[0087] Furthermore, this application generates events only when eye movement changes meet predetermined conditions, avoiding the uploading of raw eye movement perception data for each frame. For example, a gaze event is reported only once after the user's gaze stabilizes, rather than sending coordinates for each frame; an eye-closing event is reported only once after the user actually closes their eyes and the timeout period expires, rather than continuously reporting the closed-eye state. This reduces the reporting frequency, decreasing system bandwidth and power consumption. Therefore, this embodiment, through refined eye movement change analysis and condition matching mechanisms, achieves accurate, efficient, and semantic generation of eye movement events, providing ready-to-use event data for the upper layers of the operating system, significantly improving the practicality and user experience of eye-tracking interaction.

[0088] In some embodiments, particularly in interactive applications based on gaze events, consecutive blink events, and / or gaze beyond device boundaries events, in addition to the eye movement event type, certain necessary attributes are required to complete subsequent interactive operations, such as the gaze focus coordinates when the eye movement event occurs, and the repetition frequency / number of consecutive occurrences of the eye movement event. Depending on the needs, general attributes such as the occurrence timestamp, user ID, and eye source type (e.g., left eye, right eye, or both eyes) may also be required. This generates a complete, self-contained structured eye movement event data set, which no longer contains the original eye movement perception data (e.g., original eye images or their intermediate feature data), but instead includes event semantics (e.g., type identifier) ​​and necessary contextual attributes. This results in structured eye movement event data transmitted to the upper layer of the operating system with high information density, meaning that all redundant eye movement perception data is eliminated, retaining only the core semantics and key attributes, greatly improving data transmission and processing efficiency, and reducing resource consumption during data transmission.

[0089] Therefore, the structured eye-tracking event data generated by this application may include, in addition to the first indication information, second indication information. This second indication information can be used to represent attributes indicating the type of eye-tracking event. These attributes may include, but are not limited to, at least one of the following: the timestamp of the eye-tracking event; the coordinates of the gaze focus at the time of the eye-tracking event; the user identifier that generated the eye-tracking event; and the type of eye source that generated the eye-tracking event. This application does not limit the method for identifying these attributes. Based on the type of eye-tracking event, at least one corresponding attribute or the second indication information representing that attribute can be extracted from current and historical eye-tracking perception data and / or changes in eye-tracking state, and then populated into the corresponding attribute field in a predefined data structure.

[0090] Based on the above analysis, the timestamp of eye-tracking events can record the exact moment when eye-tracking events are recognized and generated by the system with high precision (e.g., millisecond level). This not only enables time sequence alignment and causal relationship determination—providing precise time sequence markers for multiple eye-tracking events (e.g., two consecutive gazes, or a gaze followed by a blink)—the application layer can use this to determine the combination relationship of eye-tracking events (e.g., "blink within 300ms after gazing at point A"), achieving accurate recognition of complex interaction commands; it also provides system developers with tools to measure the end-to-end latency "from the occurrence of user action to the generation of eye-tracking events," serving as a key indicator for performance analysis and debugging. Furthermore, it can achieve deduplication and anti-shaking of eye-tracking events in complex scenarios, eliminating duplicate or invalid eye-tracking events and ensuring that the upper layer receives a stable and reliable stream of eye-tracking events.

[0091] The attribute of the gaze focus coordinates during eye-tracking events enables precise spatial orientation interactions, such as "gaze-to-select" and "gaze control." The application layer can analyze these coordinates to pinpoint which button, text, or object the user is "looking at" on the interface, triggering corresponding actions (such as highlighting, clicking, or zooming). It also supports dynamic interactions: for "gaze events," the coordinates represent a stable gaze area; for "saccade events" or "trajectory events," a series of eye-tracking events with coordinates and timestamps can reconstruct the user's gaze path, used to analyze reading patterns, regions of interest, etc.

[0092] The user identifier attribute that generates an eye-tracking event uniquely identifies the individual user who triggered the event in a multi-user scenario. It supports multi-user eye-tracking interaction: in scenarios where multiple users share devices (such as conference tablets, smart TVs, and in-vehicle infotainment systems), this attribute is crucial for distinguishing the intentions of different users. The system can route eye-tracking events from different users to their corresponding personalized contexts or accounts based on the user identifier, achieving "one-person-one-interaction" and avoiding interference between user operations. Furthermore, this attribute enables personalized experience adaptation: combined with the user identifier, the system can access different users' exclusive eye-tracking calibration parameters, interaction preference settings, or access controls to provide a customized eye-tracking interaction experience; the implementation method is not detailed in this application.

[0093] The attribute specifying the eye source type of the eye-tracking event indicates whether the event is generated by the user's left eye, right eye, or a combined action of both eyes. It supports differentiated monocular / binocular interactions, enabling richer semantic interactions for application design. For example, a "left eye blink" could represent "confirm," and a "right eye blink" "cancel"; or "both eyes simultaneously focusing" could trigger high-security operations or multi-window switching, significantly expanding the coding space and flexibility of interaction commands. Furthermore, when a user's single eye is obstructed, fatigued, or exhibits visual acuity differences, the system can intelligently choose to rely on data from one eye or comprehensively assess binocular information, thereby improving the reliability and usability of the eye-tracking system in practical use—enhancing robustness and fault tolerance. Additionally, it supports advanced physiological and behavioral analysis, providing foundational data for professional fields (such as medical diagnosis, psychological research, driver status monitoring, etc.) to analyze binocular coordination, dominant eye, and other information.

[0094] Optionally, for different attributes as described above, the corresponding categories of the second indication information can be different. The second indication information can be the same as or different from the category of the first indication information. This application does not impose any restrictions on this, and it can be determined based on the attribute content. It can also be determined in conjunction with a predefined data structure / transmission protocol. For example, the transmission protocol can be expressed as a fixed-length binary bitstream format (160-bit data structure as shown in Table 1 below) to improve parsing efficiency and save bandwidth. In this case, the fixed number of binary bits in the eye-tracking event data can record the first indication information and each of the second indication information. It can fill the recognition result data or the converted indication information into the corresponding binary bits according to the content required to be recorded by each binary bit defined by the transmission protocol to form eye-tracking event data that conforms to the binary bitstream protocol.

[0095] For example, the first indication information can be the instruction code (instruction code) corresponding to each eye movement event type, indicating that the description of the corresponding eye movement event type is filled into the corresponding binary bit (BIT) in the binary bit stream protocol. For example, 0x1 represents an eye-closing event, 0x2 represents an eye-opening event, 0x3 represents a fixation event, 0x4 represents a blink event (which is only a valid single blink), 0x5 represents a continuous blink event, and 0x6 represents a gaze beyond the device boundary event, etc. Similarly, the second indication information representing the eye source type attribute can also be an instruction code, such as 0x0 representing both eyes, 0x1 representing the left eye, 0x2 representing the right eye, etc.; the second indication information representing the user identifier (each face is configured with a unique ID number) can also be an instruction code, such as 0x0-0xff representing different user identifiers, etc. For other attributes listed above, such as the occurrence timestamp, gaze focus coordinates (X coordinates and Y coordinates, which can be represented by pixel coordinates), blink count, direction beyond the device boundary, etc., the instruction code often cannot directly represent the complete attribute content. For these, the corresponding numerical value can be directly identified as the second indication information representing the corresponding attribute.

[0096]

[0097] Table 1

[0098] Taking the transmission protocol shown in Table 1 above as an example, it is a 160-bit binary bitstream protocol, including a 32-bit protocol header. As shown in Table 1, bits 0-31 can be filled with instruction codes. Bits 0-3 are filled with instruction codes to represent user identification (multi-user ID instruction codes), bits 4-5 are filled with instruction codes to represent eye source type, and bits 6-31 are filled with instruction codes to represent eye movement event type (event instruction codes). Secondary indication information representing other attributes can be used as event data and filled into the corresponding binary bits of the remaining 128 bits, including but not limited to the structural order defined by the transmission protocol shown in Table 1.

[0099] In some embodiments, the structured eye-tracking event data of this application can also be obtained using a key-value pair-based data exchange format, such as JSON (JavaScript Object Notation, a lightweight data exchange format) or XML (eXtensible Markup Language). The first indication information representing the eye-tracking event type is used as the key of the key-value pair, and the second indication information representing at least one associated attribute is used as the value of the key-value pair to obtain structured eye-tracking event data. Of course, this application can also use a transmission protocol based on shared memory C / C++ structures or other data formats to organize the identified eye-tracking event types and their associated at least one attribute into structured data according to predefined rules, thus generating structured eye-tracking event data. This application does not limit the data structure it represents.

[0100] It should be understood that the predefined transmission protocol in this application needs to be adapted to the operating system architecture and meet real-time requirements. This application does not restrict the implementation method of how to generate and transmit structured eye-tracking event data to the upper layer of the operating system (such as the application layer or eye-tracking event service layer). Regardless of the transmission protocol format, this application analyzes the raw eye-tracking perception data uniformly at the bottom layer, centralizes rule judgment, and standardizes / structures data encapsulation. The upper layer of the operating system (such as various apps) obtains eye-tracking information through the same standardized data structure, which greatly simplifies application development and ensures the consistency of cross-application interaction logic and data. Application developers no longer need to process the raw eye-tracking perception data to extract these attributes on their own, saving the app's computing resources and data transmission bandwidth.

[0101] The second indicator information, representing attributes related to eye-tracking events, is a crucial component of structured eye-tracking event data. This transforms eye-tracking event data from simply "what happened" (eye-tracking event type) into a complete contextual profile encompassing "when, where, who, and which eye" it occurred. This enables the upper layers of the operating system to make intelligent decisions based on rich context. The various attributes represented by the second indicator information imbue eye-tracking events with rich contextual information through different dimensions. Specifically, through combinations such as time, space, user, and eye type, a limited number of basic event types (gaze, blink, etc.) can derive almost an infinite number of specific and meaningful interactive commands, satisfying various needs from simple operations to complex controls. This greatly enriches the semantics of interaction, facilitating the unlocking of more complex, precise, and reliable interactive applications. Furthermore, based on the above analysis, these attributes enable the operating system or middleware to perform functions such as event filtering, priority scheduling, and user management based on more comprehensive information. This improves the performance, security, and intelligence level of the entire eye-tracking interaction system, laying the foundation for system-level optimization and advanced functions.

[0102] Based on the above description of the various predefined eye-tracking event types and their associated predetermined conditions, the following will describe in detail the identification process of the eye-tracking events listed above and the generation process of their structured data, but it is not limited to the eye-tracking event types and their structured data generation methods listed in the embodiments below.

[0103] Reference Figure 3 This is a flowchart illustrating an eye-tracking data transmission method proposed in Embodiment 3 of this application. This embodiment describes the process of recognizing eye-closing and eye-opening events and generating structured data for them. Figure 3 As shown, the eye-tracking data transmission method proposed in this embodiment may include:

[0104] Step S31: Based on eye movement perception data, detect the transition of eye movement state from the first state to the second state; wherein the first state and the second state are two different states in the open-eye state and the closed-eye state, respectively.

[0105] In this embodiment, eyelid (or eyelid) closure data can be analyzed (e.g., through a state classifier) ​​to continuously monitor whether the eye movement state of the current frame is open or closed. This can be determined by comparing the eyelid closure degree with the corresponding closure degree threshold for the eye movement state. For example, 100% indicates fully open eyes, and 0% indicates fully closed eyes. The closure degree threshold for each eye movement state can be flexibly configured according to actual needs. For example, an eyelid opening degree greater than 80% indicates an open eye state, and an eyelid opening degree less than 20% indicates a closed eye state. Optionally, this application can also combine aperture coordinates to assist in determining the eye movement state. For example, determining that the pupil is occluded indicates a closed eye state, and that the pupil is not occluded / or even a relatively complete pupil is identified indicates an open eye state. This application does not limit the classification and detection method of eye movement state.

[0106] Furthermore, this application can also record a state transition when it determines that the eye movement state of the current frame is different from that of the previous frame. For example, by analyzing eye movement perception data through a state machine model, it can detect whether the eye movement state transitions from an open-eye state to a closed-eye state (where the first state refers to the open-eye state and the second state refers to the closed-eye state), or from a closed-eye state to an open-eye state (where the first state refers to the closed-eye state and the second state refers to the open-eye state). Combining the above analysis of eye movement states, it can detect continuous changes in eyelid closure to determine whether a state transition has occurred and the direction of the transition. For example, if eye movement perception data indicates that the eyelid opening / closing degree decreases from 90% to 15%, it records a state transition from an open-eye state to a closed-eye state, but is not limited to this detection method.

[0107] As can be seen, the steps in this application achieve real-time, discrete capture of the user's most basic eye movements (opening and closing eyes), converting continuous visual input into clear, time-precise "state switching" signals, providing a clear starting point for subsequent intent determination. Optionally, in the detection of eye movement state transitions, to avoid false transitions caused by noise, the system can be configured with a jitter reduction mechanism. For example, it can require the new state to be maintained for at least a short time (e.g., 50ms) before confirming the transition as valid. For this purpose, an edge-triggered method can be used, where the state is determined to have switched only when the eyelid closure degree exceeds a preset threshold for several consecutive frames.

[0108] Step S32: Determine the duration of maintaining the second state;

[0109] Once a state transition is confirmed, the timing begins at that moment, while continuing to monitor eye movement, to record the duration the eye movement remains in the second state. Timing continues until another state transition occurs (e.g., from closed eyes back to open eyes) or the maximum timing limit is reached. The time at that moment is the timing end, and the time difference between that end and the timing start point is the duration. For closed eye events: timing begins after the transition from open to closed eyes and continues until the closed eye state ends (i.e., transitions back to open eyes) or the preset maximum closed eye duration (e.g., 10 seconds). For open eye events: timing begins after the transition from closed to open eyes and continues until the open eye state ends or the maximum duration is reached.

[0110] As can be seen, this application introduces quantitative measurement of the time dimension, accurately measuring the physical length of time a user maintains a new state after a state transition, serving as a key distinction between the core intent of "brief passing by" and "intentional retention." For example, this application can differentiate between a rapid, unconscious eyelid twitch (extremely short duration) and an intentional eye-closing action indicating "confirmation" or "start" (longer duration) based on duration. The measurement accuracy of this duration is typically in the millisecond range, achievable using a system clock or a high-precision timer.

[0111] Step S33: If the duration is greater than the first time threshold, generate corresponding structured eye movement event data. The first indication information in the eye movement event data is used to indicate the closed eye event or open eye event corresponding to the second state.

[0112] Following the above analysis, this application pre-configures a first time threshold (e.g., 800ms), which can be set based on human-computer interaction research and experimental data. This threshold is used to define the duration of "effective, interactive, continuous states" (such as conscious behaviors like actively closing eyes to rest or actively opening eyes to recover) versus "ineffective eye physiological tremors or momentary movements" (such as unconscious, brief behaviors like natural blinking). This threshold can be optimized through extensive user testing (e.g., set or adjusted in the system device interface) to balance "response speed" and "false touch rate." It can also be combined with other filtering conditions to avoid misjudgments. Alternatively, it can be preset in the system firmware. This application does not restrict the value of the first time threshold or its configuration method. It can be dynamically adjusted according to different users (e.g., the elderly, children) or different application scenarios (e.g., games, office work), enhancing the system's adaptability and user experience.

[0113] In this way, through the state transition detection described above, capturing the corresponding eye movements (such as open / closed eyes) in the second state (such as open / closed eyes) immediately triggers the extraction process. Continuously capturing the second state and identifying it as the corresponding eye movement event type if the duration exceeds the first time threshold is also considered. In other words, the actual measured duration of the transition from the first state to the second state and the duration of the transition in the second state is compared with the first time threshold to automatically filter out false state-holding signals that are shorter than the threshold due to blinking, changes in light, or momentary misjudgments by the algorithm. This significantly improves the accuracy and noise resistance of eye movement event recognition and establishes a unified and repeatable technical standard for the entire system for "what kind of closed / open eyes count as a valid interaction event," fundamentally solving the problem of fragmented experience caused by different application judgment rules.

[0114] Specifically, if the second state is a closed-eye state and its duration exceeds the first time threshold, it indicates that the eye movement event causing this state transition is a closed-eye event; if the second state is an open-eye state and its duration exceeds the first time threshold, it indicates that the eye movement event causing this state transition is an open-eye event. The code corresponding one-to-one with the identified eye movement event type is determined as the first indication information, and the second indication information representing the corresponding attributes, such as the timestamp of the eye movement event, user identifier, and eye source type, is filled in as needed. This gives the generated structured eye movement event data clear semantics, and the upper layer of the operating system does not need to determine the closed-eye duration or state switching logic itself. It can directly perform the corresponding operation based on the eye movement event type, simplifying application development.

[0115] Furthermore, after receiving eye-tracking event data, the application can trigger explicit interaction logic. For example, parsing the eye-closing event (referring to prolonged / continuous eye closure) can serve as a signal to trigger "automatic scrolling start" in a reader app; in a game app, it can serve as a signal to trigger "enter aiming mode" or other special skills. Of course, the app can also execute interactive commands such as exiting the application, turning off the screen, or putting the app into standby mode when it parses an eye-closing event. It can also be used for fatigue detection, such as detecting multiple prolonged eye closures in a short period of time (e.g., 5 eye-closing events within 1 minute, each lasting more than 800ms), determining that the user may be fatigued, and triggering a reminder, etc. There are no restrictions on this. Similarly, parsing the eye-opening event (referring to prolonged / continuous eye open) can serve as a clear "wake-up" or "ready" signal when the user recovers from resting with their eyes closed (i.e., transitioning from a closed-eye state to an open-eye state), which can be used to trigger the interface to light up or resume interaction, etc. There are no restrictions on this.

[0116] In practical applications, "eye-closing events" and "eye-opening events" can be used in pairs to form a state switching switch (similar to "press" and "release" on a mouse). This can be used to implement complex interactions that require continuous state maintenance, such as "press and drag" and "start / stop recording." The implementation process will not be detailed in this application. As can be seen, this application transforms physiological signals that are prone to noise into stable, high signal-to-noise ratio structured events through rigorous temporal logic. This not only enhances the robustness and usability of eye-tracking interaction but also ensures the consistency and predictability of cross-application interaction behavior through standardized thresholds and event definitions. Furthermore, it generates corresponding structured event data only after a state transition and a sustained timeout, rather than reporting eyelid closure data for each frame, significantly reducing data transmission volume and upper-layer processing burden, and lowering the consumption of transmission and computing resources.

[0117] Reference Figure 4 This is a flowchart illustrating an eye-tracking data transmission method proposed in Embodiment 4 of this application. This embodiment describes the generation process of blink events and continuous blink events, such as... Figure 4 As shown, the eye-tracking data transmission method proposed in this embodiment may include:

[0118] Step S41: Based on eye movement perception data, detect at least one state transition pair in the eye movement state;

[0119] Based on the above descriptions of the open-eye and closed-eye states, each state transition pair in this embodiment can include a first state transition and a second state transition adjacent to it but in the opposite direction. That is, it includes two consecutive state transitions in opposite directions. The first and second state transitions are two distinct state transitions within the open-eye action (from closed-eye to open-eye) and the closed-eye action (from open-eye to closed-eye), respectively. Therefore, a complete blinking behavior is represented in terms of eye movement as: open-eye → closed-eye → open-eye, which can be considered a complete and reversible eye movement cycle.

[0120] This application can identify the moment of a state transition in real time by detecting changes in eyelid closure or electrooculography (EOG) signals. Specifically, it continuously monitors the state transition sequence, and each time a transition is detected, it checks whether its direction is opposite to the previous transition. If opposite, these two transitions are bound together as a "state transition pair," representing a potential complete blink. If the directions are the same (e.g., two consecutive "open eye → closed eye"), it may indicate the beginning of a prolonged eye closure and does not constitute a valid blink pair. To avoid noise interference, a de-jitter mechanism can be used: a transition is only confirmed after the eyelid closure exceeds a certain threshold and stabilizes for several consecutive frames. The implementation process can be referred to the relevant description of the eye-opening / closing event generation process above.

[0121] As can be seen, this application does not view each state change in isolation, but extracts action units with specific causal and logical relationships (one closing and one opening) from the time sequence, providing the correct analysis object for subsequent judgment of whether the action is a "effective blink", and avoiding the erroneous inclusion of eyelid tremors or prolonged eye closure in blink analysis.

[0122] Step S42: Obtain the state interval duration between the first and second state transitions in each state transition pair;

[0123] The state interval duration refers to the time difference between the moment the first state transition (e.g., "eyes open state → eyes closed state") occurs and the moment the second state transition (e.g., "eyes closed state → eyes open state") occurs. This application can start a timer after detecting the first state transition and stop the timer when the second state transition is detected, recording this time interval to obtain the corresponding state interval duration. If a high-precision system clock is used, the time can be accurate to the millisecond level.

[0124] As can be seen, this application quantifies the core features of the blinking action, transforming the vague "blinking" into a precisely calculable physical quantity (state interval duration) to reflect the duration of the user's closed-eye state. This serves as a core indicator for judging the validity of blinking and lays the foundation for subsequent objective judgment of blinking events.

[0125] Step S43: Determine the number of consecutive state transition pairs whose state interval duration falls within the effective time interval within the preset time window;

[0126] Step S44: Determine whether the consecutive quantity has reached the preset quantity. If not, proceed to step S45; if yes, proceed to step S46.

[0127] Step S45: Generate corresponding structured eye movement event data, wherein the first indication information in the eye movement event data is used to indicate blinking events;

[0128] Step S46: Generate corresponding structured eye movement event data. The first indication information in the eye movement event data is used to indicate continuous blinking events, and the second indication information includes the number of continuous blinks.

[0129] In this embodiment, an effective time interval can be pre-configured to distinguish between normal blinking and ineffective actions (such as unconscious micro-blinking or prolonged eye closure). This interval has a lower threshold (e.g., 400ms) and an upper threshold (e.g., 800ms). The lower threshold indicates that eye closure below this value is considered unconscious micro-blinking (such as brief eye closure due to changes in light or dry eyes), and does not trigger a blinking event. The upper threshold indicates that eye closure above this value is considered active eye closure (such as rest or thinking), and is not considered normal blinking, nor does it trigger a blinking event. The configuration method for these two thresholds is similar to the configuration method for the first time threshold described above, and this application does not impose any limitations on this.

[0130] Thus, this application only identifies a blink action / behavior as a valid blink event if the duration of the acquired state interval is not less than the lower threshold (e.g., ≥400ms) and not greater than the upper threshold (e.g., ≤800ms). Conversely, if the duration of the state interval of a detected state transition pair is less than the lower threshold or greater than the upper threshold, it is filtered as an invalid blink action and will not trigger the blink event generation process. This filters out basic eye-opening / closing actions that trigger the extraction / end process, as well as some unconscious micro-blinks and other invalid blink actions, thereby improving the accuracy of blink event recognition and reducing the false trigger rate.

[0131] In addition, to distinguish between single blink events and continuous blink events, this application sets a preset time window w (e.g., 2 seconds) and defines the minimum number of valid blinks that must occur consecutively within this window w for continuous blink events, such as a preset number of 2 or 3 (preset blink count). These values ​​can be configured according to actual conditions or user-defined, and their specific size is not limited. Thus, within the window W, the number of consecutive valid blink events (valid state transition pairs) identified by the methods described in steps S41 and S43 is continuously monitored. By determining whether this consecutive number reaches the preset number, it distinguishes between a single valid blink event and a continuous blink event occurring within the window W. The preset time window W can be a sliding window (starting from each blink) or a fixed window (starting from the first blink), etc.

[0132] As can be seen, this embodiment performs aggregate analysis on discrete single events in the time dimension. If a state transition pair occurs within window W and the state interval is within the valid time interval, and the number of consecutive state transition pairs is less than a preset number, such as 3, even if two blinking actions are detected consecutively within W, it is identified as a single valid blinking event, not a continuous blinking event. Only when the number of consecutive state transition pairs (referring to valid blinking actions) is greater than or equal to the preset number is it identified as a continuous blinking event. For example, if three or more blinking actions are detected consecutively within W, it is identified as a continuous blinking event.

[0133] In this method, if the state interval of a state transition pair occurring within W is within a valid time interval, the first indication information in the corresponding structured eye movement event data (blink event data) generated at this time can be used to indicate the blink event, such as a pre-configured unique code for a single blink event. Depending on the needs, a second indication information may also be included to represent at least one attribute of the blink event (such as occurrence timestamp, user identifier, eye source type, etc.). If the preset number of consecutive state transitions occurring within W all fall within the valid time interval for their respective state intervals, the generated structured eye-tracking event data can include first and second indication information. The first indication information indicates the eye-tracking event type of continuous blinking, while the second indication information includes the number of consecutive blinks. It may also include information representing other attributes of the blinking event, enabling the eye-tracking event data to express complex semantics. This not only conveys the fact that "continuous blinking occurred" but also precisely indicates "how many times it was blinked." This allows applications to achieve differentiated interactions, such as "double-clicking" to select items and "triple-clicking" to select all. Furthermore, the second indication information contains rich interactive dimensions, greatly expanding the expressive power of the interaction. Users can input different commands by controlling the number of blinks, making eye-tracking interaction no longer limited to simple binary triggers.

[0134] For example, if a user blinks naturally once, with the eyes closed for 500ms (between 400ms and 800ms), and the state transition interval of the detected state transition pair is 500ms, it is considered a valid blink. Blink event data can be generated and sent to the upper-layer application (which can be used for taking photos, turning pages, etc.). If the user blinks rapidly three times within 2.5 seconds, with each blink lasting 450ms, 520ms, and 480ms respectively, all within the valid time range, and three adjacent state transition pairs are detected within a preset time window (e.g., 3 seconds) using the above method, the cumulative number of valid blinks (consecutive counts) reaches 3. This reaches the preset number, generating continuous blink event data containing 3 consecutive blinks. This data is sent to the upper-layer application and can be used to trigger shortcuts, special game skills, etc. If a user blinks rapidly and unconsciously due to dust getting into their eyes, and the duration of the closed eyes is 150ms (less than the lower threshold of 400ms), a state transition pair is detected. However, since the state interval is 150ms, which is not within the valid time range, no event is generated.

[0135] In summary, this application's embodiments construct a multi-layered, highly reliable blink interaction recognition method through refined judgment of "state transition pairs, interval duration ranges, and time window counting." This method confines blink events to specific time intervals, avoiding misjudging behaviors such as fatigue-induced eye closing or thinking-induced eye closing as blinks. It reduces unnecessary responses from upper-layer applications, lowers the false alarm rate, and thus accurately identifies single valid blinks, avoiding various interferences such as excluding unconscious micro-blinks and distinguishing between active eye closing (prolonged eye closing) and ordinary blinks. This ensures the semantic purity of blink events, improving the reliability of eye-tracking interaction. Furthermore, it can intelligently identify continuous blink patterns representing clear counting intentions. By accumulating valid blink counts within a time window, it can accurately identify continuous blinking behavior and report the blink count, enabling upper-layer applications to perform rich and convenient operations. This solves the core challenge of reliably converting the ambiguous and noisy physiological signal of "blinking" into stable interactive commands with different counting semantics, providing key technical support for achieving refined, natural, and efficient eye-tracking interaction.

[0136] The system's adaptability is enhanced because the upper and lower limits of the effective time interval, the length of the time window, and the threshold for consecutive blinks can all be dynamically adjusted according to user habits or application scenarios. The generated "blink events" and "consecutive blink events" have clear semantics, eliminating the need for upper-layer applications to manually determine the duration of eye closure or count the number of blinks. They can directly execute the corresponding logic based on the eye movement event type and the accompanying blink count, simplifying application development. Furthermore, eye source type attributes can be added to distinguish between left-eye blinks, right-eye blinks, or simultaneous blinks of both eyes, further enriching interactive commands (e.g., left-eye blink for confirmation, right-eye blink for cancellation).

[0137] It should be noted that in some embodiments, the effective time interval can also be an open interval or a half-open interval (e.g., greater than the lower threshold and less than or equal to the upper threshold). Furthermore, the determination of continuous blinking events can adopt a cumulative mode: triggering occurs as soon as the number of effective blinks within the sliding time window reaches the threshold, and the window moves with each new blink without waiting for the window to end. For scenarios requiring differentiation of "continuous blinking speed," a blinking frequency condition (e.g., blinks per second) can be added; the implementation process is not detailed in this application.

[0138] Reference Figure 5 This is a flowchart illustrating an eye-tracking data transmission method proposed in Embodiment 5 of this application. This embodiment describes the generation process of gaze events, such as... Figure 5 As shown, the eye-tracking data transmission method proposed in this embodiment may include:

[0139] Step S51: Obtain the coordinate movement distance of the focal point;

[0140] In this embodiment, the two-dimensional / three-dimensional coordinate sequence of the gaze focus on the screen or in space can be acquired in real time. Each time a new coordinate of the gaze focus is acquired, it is compared with the previous stable coordinate or the coordinate of the starting point of the time window, and the corresponding coordinate movement distance is calculated. This calculation can be performed using, but is not limited to, Euclidean distance. Optionally, to detect stability, the maximum span or standard deviation of all coordinate points within the current time window can also be calculated as the coordinate movement distance, etc., to serve as an indicator of the degree of "wobbling" or "moving" of the gaze focus within that time window. This application does not limit the implementation method of step S51.

[0141] As can be seen, this embodiment achieves a quantitative assessment of the spatial stability of the gaze. The smaller the obtained coordinate movement distance, the more concentrated and stable the user's gaze is in space, and the more likely they are in a "fixed gaze" state. This provides a spatial dimension for judging actions such as rapid scanning and unconscious wandering.

[0142] Step S52: If the distance the coordinate moves is less than the spatial distance threshold, record the duration of the state.

[0143] Step S53: If the duration of the hold is greater than the second time threshold, generate corresponding structured eye movement event data. The first indication information in the eye movement event data is used to indicate the gaze event, and the second indication information includes the coordinates of the gaze focus when the gaze event occurs.

[0144] Following the above analysis, this application introduces constraints in both spatial and temporal dimensions for the identification of gaze events, which must be met simultaneously to be considered a "gaze event." For the spatial stability condition, a "spatial distance threshold" is pre-configured, defining the maximum allowable range of eye movement considered "stable gaze," such as 5% of the screen width or 30 pixels. This threshold can also be determined by considering factors such as device resolution and the accuracy of the eye-tracking algorithm; this application does not impose a limit on the value of the threshold. If the calculated coordinate movement distance is less than this spatial distance threshold, it indicates that the gaze focus is relatively stable in space and has not shifted significantly.

[0145] However, a single judgment is insufficient to guarantee that the user is looking, as the gaze may flicker briefly and then quickly shift away. Therefore, this application also introduces a duration condition, pre-configuring a second time threshold, which defines the minimum duration required to be considered "intentional gaze," used to distinguish between brief gaze pauses and genuine attention, such as 200ms-300ms. The second time threshold can be determined to balance response speed and accuracy as needed. Too short a threshold may easily misjudge short pauses during saccades as gaze, while too long a threshold will lead to a delay in the response to gaze events. This application does not limit the size of the threshold.

[0146] Based on this, this application can monitor the coordinate movement distance of the gaze focus in real time. Once the coordinate movement distance is less than the spatial distance threshold, the system starts timing (i.e., this is the starting point for the duration of the gaze) and records the duration of the stable state. When the duration of the stable state exceeds a second time threshold, the timing is reset to zero, and it is considered that the user is indeed looking at a certain area, which is identified as a gaze event. In this way, by using the dual thresholds of "spatial stability" and "temporal duration," the interactive intent of the "gaze event" is strictly defined, ensuring that the event is only triggered when the user truly and continuously focuses their visual attention on a certain area. This eliminates interference such as passing by the gaze line or brief drift, achieving high-precision recognition. Moreover, these dual conditions are highly consistent with the natural pattern of human eye observation (first moving quickly to the target, then stabilizing for processing), so that the generated "gaze event" can accurately reflect the user's cognitive focus.

[0147] After identifying a gaze event using the method described above, a structured data object can be created, and the first indication information can be set as the code corresponding to the gaze event type, such as 0x3. The key attribute of the gaze focus coordinates at the time of the identified gaze event is filled into the corresponding data object as the second indication information. At this time, the coordinates can be determined based on multiple gaze coordinates collected within the time window of the gaze event (usually the time window that meets the conditions). These coordinates are often not a single coordinate at the instant the conditions are met. They can be the average of all coordinates within the window (mean filtering) or the median (outlier removal), or a weighted average (the closer to the center of the window, the higher the weight) to effectively smooth out the small jitter of the coordinate data and obtain more stable coordinate values ​​that better represent the real gaze area. This provides high-precision, low-noise directional information to the upper layer, making the "gaze as interaction" experience more accurate and natural, avoiding interaction jitter caused by physiological tremors of the gaze. The APP can also directly use this coordinate to realize rich spatial directional interactions such as button highlighting, content preview, and focus switching (switching eye focus in multiple windows, automatically switching to the window the user is looking at, without manual clicking).

[0148] It is understandable that, depending on actual needs, when generating the second indication information in the structured eye-tracking event data that indicates a gaze event, the corresponding attribute fields in the data structure can be filled with other attributes such as timestamps (usually the moment when the condition is first met), user IDs, eye source types, or their unique codes. For example, if 15 gaze focus coordinates are collected within a stable window of 300ms, and the arithmetic mean of these coordinates is calculated as the final gaze point coordinates (the gaze focus coordinates when the gaze event occurs), the generated eye-tracking event data can be: [Eye-tracking event type=0x3][Timestamp=2026-04-15 10:30:45.123][Gaze focus coordinates=(320.5, 240.2)][User ID=0x1][Eye source type=0x0].

[0149] In summary, since eye-tracking data typically contains physiological micro-saccades (amplitude approximately 0.2°-1°) and sensor noise, directly reporting real-time coordinates can lead to frequent jumps in the gaze point position. This embodiment determines the gaze point coordinates by statistically analyzing multiple coordinates within a time window, effectively filtering out gaze jitter and enabling the upper layer to obtain a stable and smooth gaze position. This is suitable for fine-grained interactions such as cursor control and target selection, and also supports subsequent interaction expansion, such as implementing various gaze-based interface operations. Moreover, considering that simple spatial stability (small movement distance) may be caused by the user's gaze pausing at a certain position without actually focusing (such as gaze wandering during thinking), this application ensures that the user is consciously focusing on a certain area by adding a time duration requirement (maintaining a duration greater than a second time threshold), reducing false triggers, improving gaze recognition accuracy, and reporting a gaze event only once after the stable duration condition is met, rather than reporting coordinates for every frame, thus reducing the event reporting frequency and significantly reducing data transmission volume and upper-layer processing burden.

[0150] In some embodiments, during the analysis of real-time acquisition of gaze focus coordinates (which can be represented by (x_t, y_t)), it is also possible to detect whether they exceed the device screen or a preset effective boundary range (collectively referred to as the device boundary range), and combine this with the duration of the out-of-bounds state to determine whether the user intends to perform boundary interactions (such as edge scrolling, page turning, etc.), while reporting the out-of-bounds direction or out-of-bounds offset, so that the upper layer can implement rich edge control functions. The device boundary range can be a rectangular area, such as x∈[0,W], y∈[0,H] in a screen coordinate system with the lower left corner of the device screen as the origin, where W is the width and H is the height. The device boundary range can also be customized according to application requirements (e.g., only the left and right edges are valid, the top and bottom edges are invalid), and the effective gaze focus coordinate range can be determined accordingly: [X_min,X_max] and [Y_min,Y_max], where the subscript min represents the minimum value of the corresponding coordinate, and the subscript max represents the maximum value of the corresponding coordinate.

[0151] This allows for real-time determination of whether the current line-of-sight focus coordinates exceed the device boundary. The line-of-sight focus is considered to be outside the boundary if any of the following conditions are met: x < 0 (or x_t < X_min, exceeding the left boundary), x > W (or x_t > X_max, exceeding the right boundary), y < 0 (or y_t < Y_min, exceeding the upper boundary), y > H (or y_t > Y_max, exceeding the lower boundary). The degree of exceeding the boundary can be represented by an offset: for example, the left boundary offset equals the absolute value of x, the right boundary offset equals the absolute value of the difference between x and W, the upper boundary offset equals the absolute value of y, and the right boundary offset equals the absolute value of the difference between x and H. The direction of exceeding the boundary is recorded as left, right, upper, or lower boundary, which can be indicated by a positive or negative sign.

[0152] Similar to the gaze events described above, to prevent false triggers caused by the gaze quickly crossing the boundary or tracking noise, this application also introduces a third time threshold. When the gaze focus exceeds the boundary, the system starts timing and records the duration of the out-of-bounds state. During this process, once the gaze returns to the boundary, the timer is reset to zero. Only when the duration of the out-of-bounds state exceeds the third time threshold is a valid "gaze beyond device boundary event" considered to have occurred. This avoids unnecessary boundary events when the user's gaze briefly crosses the boundary (e.g., quickly glancing at the edge and then returning). Only behavior of continuously looking out of the screen will be captured, that is, the coordinates of the gaze focus exceed the device boundary range, and the duration of this state is greater than the third time threshold, generating corresponding structured eye-tracking event data.

[0153] In this application, the first indication information in the structured eye-tracking event data indicating a "viewpoint exceeding device boundary event" represents this type of eye-tracking event. This information can be represented by binary instruction codes. Furthermore, different instruction codes can be predefined for different types of viewpoint exceeding device boundary events, such as Y-axis exceeding boundary, X-axis exceeding boundary, and both X and Y exceeding boundary, to precisely determine which boundary has been exceeded. Alternatively, the direction of exceeding the boundary (e.g., left, right, up, down) can be used as the second indication information. Simultaneously, the offset exceeding the boundary (e.g., the number of pixels exceeding the left boundary) can also be recorded as the second indication information, allowing the upper layer to perform different interactive operations based on the direction and distance of exceeding the boundary. The second indication information may also include other attributes or their unique codes as needed.

[0154] For example, when a user's gaze moves beyond the right edge of the screen and lingers for 150ms, the generated eye-tracking event data is: [Eye-tracking event type=0x6][Occurrence timestamp=2026-03-15 10:30:45.123][Out-of-direction=Right][Out-of-direction offset=25px], etc. This data is sent to the upper layer of the operating system, which uses the "out-of-direction" attribute to understand where the user is looking, thereby triggering the corresponding navigation operation. For example, "continuously looking to the right side of the screen" can be mapped to "turning the page to the right," and "continuously looking to the bottom of the screen" can be mapped to "scrolling down." The "out-of-direction offset" attribute makes it possible to simulate "accelerated scrolling" or "controlled scrolling speed." The larger the offset, the faster the scrolling speed, achieving a continuous control effect similar to sliding the edge of a joystick or touchpad. Therefore, this application can transform the space outside the physical boundaries of the screen into an effective interactive input area, greatly expanding the interactive range and providing an innovative means to achieve more natural and expansive interface navigation and control.

[0155] In summary, the "gaze" and "over-boundary" event generation mechanisms described in this application collectively construct the system's deep understanding of user visual attention space management. "Gaze events" enable precise capture and stable reporting of focus within the interface, forming the foundation for refined operations; "over-boundary events" enable intelligent recognition of intentions to move out of the current view, crucial for wide-range navigation and view switching. Based on the above analysis, over-boundary events can be used to implement functions similar to touchscreen edge swiping. For example, when the gaze exceeds the left edge of the screen, a "back" operation is triggered; exceeding the right edge triggers a "forward" operation; exceeding the top / bottom edges triggers page scrolling or volume adjustment. This provides a new dimension of interaction for contactless operations. The reported over-boundary offset can be used to simulate scrolling speed or operation intensity; a larger over-boundary offset results in faster page scrolling, while a smaller offset results in slower scrolling, thus enhancing the naturalness of the interaction. Taking racing games as an example, an over-boundary event can trigger a left turn, an over-boundary event can trigger a right turn, and the over-boundary offset can control the turning radius.

[0156] This application prevents unnecessary actions from being triggered when a user's gaze inadvertently briefly crosses the boundary (e.g., when the eye scans across an edge). Only when the user intentionally keeps their gaze outside the boundary for a certain period is it considered a valid instruction. Furthermore, the event is reported only after the user has exceeded the boundary and the timeout period has elapsed, rather than continuously reporting coordinates near the boundary, thus reducing data volume and upper-layer processing burden. Moreover, the aforementioned boundary range, third time threshold, and whether direction detection is enabled can all be dynamically configured by the system or application to adapt to different devices and interaction scenarios (such as mobile phones, tablets, VR headsets, etc.).

[0157] In some other embodiments, in addition to the eye movement event data generation method described in the above embodiments for each type of eye movement event, the generation of structured event data proposed in this application can also be achieved by combining multiple spatiotemporally related and identified eye movement events according to preset event combination rules to generate a composite eye movement event data with a new and complex semantics, thereby realizing a higher level of interactive capabilities, further reducing the logical burden of the application, and improving interactive efficiency.

[0158] The pre-configured event combination rules can be used to define which basic eye-tracking events (such as eye-closing events, eye-opening events, fixation events, blinking events, continuous blinking events, and gaze beyond device boundaries events described above) can be combined into a composite eye-tracking event when they meet specific spatiotemporal relationships. The event type of the composite eye-tracking event is jointly determined by the types of the basic eye-tracking events it composes and their relationships. In addition to the first indication information (composite event type), its structured data may also include a second indication information (such as fixation point coordinates, number of blinks, and excess offset). This application does not limit the content of the event combination rules or their configuration methods.

[0159] Optionally, the event combination rules may include, but are not limited to, at least one of the following elements: eye movement event type combination (e.g., "gaze event and monocular blink event combination", "gaze beyond boundary event and gaze event combination", "gaze event and eye-opening event combination"), spatiotemporal constraints (e.g., the time difference between the occurrence of the constituent events is less than a preset synchronization threshold, the order of occurrence is fixed, or they occur in the same spatial area (e.g., the gaze point coordinates are close to the boundary position)), optional attribute inheritance or calculation (the attributes of the composite event can be inherited from the constituent events (e.g., gaze point coordinates) or calculated according to rules (e.g., the end of burst fire marker)). Based on this, triggering conditions for triggering the composite eye movement event generation process are generated. Eye movement perception data is analyzed in real time. Once the analysis result meets any triggering condition, the corresponding composite eye movement event generation process is triggered, generating the corresponding structured eye movement event data.

[0160] For example, during document reading or web browsing, if a user gazes at a keyword and then blinks with one eye (left or right), the system highlights the keyword. Following the method described in the above embodiments, a "gaze event" is detected, the gaze point coordinates (x, y) are obtained, and the keyword in the corresponding text area is determined through the application context. Within a preset time window (e.g., 500ms) after the gaze event, the system detects a blink event (left or right) and obtains the eye source type (left / right). If the time difference between the two events is less than a synchronization threshold (e.g., 200ms), and the gaze point is located within a valid text area, a combination rule is triggered to generate structured eye-tracking event data containing first and second indication information. The first indication information may include an instruction code representing a "keyword highlighting instruction"; the second indication information may include the gaze point coordinates (keyword location), eye source type (left or right eye, which can be used to distinguish highlight color or action), and optional keyword text content (parsed by the application layer), etc. In this way, users can quickly mark key content simply by looking at words of interest and blinking, without any manual or voice operation. This is especially suitable for scenarios where hands are occupied or where there are no obstacles to operation, greatly improving the efficiency of information acquisition and processing.

[0161] In scenarios such as when reading ebooks or browsing long web pages, if a user's gaze moves beyond the bottom edge of the screen and lingers near it, and the system automatically scrolls down, the method described in the above embodiment detects a gaze exceeding the boundary event, with the direction of the exceedance being "down," and records the exceedance offset (e.g., exceeding the bottom boundary by 10 pixels). Simultaneously or shortly before this, the system detects a gaze event, with the gaze point coordinates located in the area near the bottom edge of the screen (e.g., the y-coordinate is greater than the difference between the screen height and the boundary area height threshold, such as 50 pixels). To avoid accidental page turning caused by simply looking out of the boundary (e.g., a user quickly scans to the bottom and immediately looks back), it is required to simultaneously gaze at the bottom border area to confirm the user's intention. Therefore, a combination rule is triggered only when the time difference between these two events is less than a preset threshold (e.g., 300ms) and the duration of looking out of the boundary is greater than a third time threshold. This generates structured eye-tracking event data containing first indication information representing a "page-down command" and second indication information containing attributes such as the offset (which can be used to control the page-turning speed or amplitude) and gaze coordinates (used to confirm the target area). This enables intuitive boundary-triggered navigation. Users only need to move their gaze out of the bottom of the screen and pause briefly, and the system interprets this as a page-turning intention, simulating the operation of a finger sliding on the edge of a touchscreen. This provides a smooth, zero-learning eye-tracking navigation method for reading long documents and browsing lists.

[0162] In a first-person shooter game, a user looks at an enemy and closes their eyes to initiate burst fire. When it's time to stop firing, the user looks at the same target and opens their eyes, ending the burst fire. Following the method described in the above embodiment, currently in a "burst fire in progress" state (previously triggered by a combination of a gaze event and an eye-closing event), the system detects a gaze event, and the gaze point coordinates match the current target location (e.g., distance less than a threshold). During the gaze event, the system detects an eye-opening event (switching from an eye-closed state to an eye-opening state, with the eye-opening duration exceeding a first time threshold). If the gaze event and the eye-opening event overlap in time, or if the eye-opening occurs within a time window after the gaze begins (e.g., 1 second), a combination rule is triggered, generating first indication information indicating a "burst fire stop command," and second indication information including / representing attributes such as gaze point coordinates (target location) and eye source type (left eye / right eye, which can be used to distinguish different weapons or actions). This enables a complete and sophisticated eye-tracking control chain for "aiming-firing-stopping" in the game: "gazing" is used for continuous aiming, "long-term eye closure" serves as the trigger for "starting to fire" (another rule), and "opening the eyes" serves as the trigger for "stopping to fire." This composite event avoids the application having to maintain its own shooting state machine and event associations, simplifying game development and enhancing the game's immersion and accessibility.

[0163] It should be noted that the conditions for triggering the generation of structured eye-tracking event data include, but are not limited to, the single basic eye-tracking event or a combination of at least two basic eye-tracking events listed above meeting the trigger conditions. These conditions can be dynamically configured or adjusted based on actual circumstances. In other words, new combination rules can be added at any time (e.g., "gazing + continuous blinking" to start screen recording) without modifying the existing application; the application only needs to subscribe to the corresponding composite event type. This event combination method allows the upper layer to directly receive semantically clear composite events (such as "keyword highlighting," "page down," and "stop continuous blinking") without having to monitor event sequences, maintain state machines, or calculate spatiotemporal relationships. This significantly reduces development costs and enables the expression of richer interactive intentions, making eye-tracking interaction no longer limited to simple cursor control or single clicks, but enabling functions similar to combination shortcut keys in a graphical interface. Both basic eye-tracking events and composite events still use a structured data format (including event type and attributes), sharing the same transmission channel and event pool as basic events, without requiring new interfaces or modifications to the application architecture.

[0164] The structured eye-tracking event data generated can be transmitted to the upper layers of the operating system in ways including, but not limited to, at least one of the following two methods. Method 1: The structured eye-tracking event data can be stored in the operating system's event pool for the upper layers of the operating system to read. In this method, a shared memory area called the event pool can be maintained within the operating system's kernel space (kernel / driver layer) or within a trusted system service process (such as the eye-tracking event service in the system service layer). This pool can be implemented based on data structures such as shared memory, ring buffers, message queues, or databases, and features configurable capacity (a preset maximum number of events to prevent unlimited growth), thread safety (e.g., supporting multi-threaded concurrent read / write through mutexes, read-write locks, or lock-free queues), and event lifecycle management (e.g., events can be retained for a period of time (e.g., 5 seconds) after being read for other applications, and automatically reclaimed after the timeout; or a "single consumption" model can be adopted, where events are deleted immediately after being read).

[0165] Thus, whenever the system generates a structured eye-tracking event (such as an eye-closing event, gaze event, or composite event), the eye-tracking event service serializes it into a standard format such as binary or JSON according to a predefined data structure and writes it to the tail of the event pool. This write operation should guarantee atomicity, ensuring that the event is not partially written. Simultaneously, the event pool can attach metadata such as a timestamp and event ID to each event, facilitating querying and filtering by upper layers. Optional upper layers of the operating system can obtain structured eye-tracking event data from the event pool through methods such as reading interfaces, active queries, or event subscriptions.

[0166] The reading interface method refers to the operating system providing a unified API interface for upper layers to query or listen to the event pool to obtain all new events after a certain sequence number. The active query method, as a polling approach, allows upper-layer components (such as games, readers, system navigation bars, etc.) to periodically call the system-provided query interface, passing in the event type of interest (e.g., querying only gaze events) and the maximum number of responses. Based on this, matching event data is actively retrieved from the event pool and returned. The query interface can support filtering by time range, event type, eye source, etc. The event subscription (i.e., event callback) method allows application-layer applications to register subscriptions for specific event types with the system. When a new event is added to the event pool, it checks if the event type matches any subscription; if a match is found, the application is asynchronously notified via a callback function, and the eye-tracking event data is passed to the application. This subscription mode avoids CPU idle time caused by polling, is suitable for scenarios with high real-time requirements, and reduces unnecessary polling overhead.

[0167] As can be seen, the event pool in this embodiment acts as an intermediate buffer, allowing underlying event generation and upper-layer event processing to proceed independently without blocking each other. The upper layer does not need to know how the underlying events are generated (e.g., from the hardware layer, driver layer, or kernel layer); it only needs to read standardized data from the event pool. Furthermore, upper-layer applications are allowed to query missed historical events or perform event replay and analysis during startup or recovery. By granting the upper layer control—allowing it to process events according to its own pace and capabilities—application overload caused by event surges is avoided, enhancing the system's resilience and controllability. Even if the upper-layer application is temporarily busy or crashes, events will not be lost. Support for concurrent access by multiple applications is also provided: multiple applications can simultaneously read events from the event pool without competing for underlying sensor or algorithm resources. Each application can subscribe to different event types as needed, and the system automatically distributes events. This method, based on query conditions or subscription types, ensures that applications only receive the events they need, reducing unnecessary cross-process communication and data parsing.

[0168] Method two involves directly pushing structured eye-tracking event data to the upper layers of the operating system via an internal communication process. In practical applications, a lightweight inter-process communication (IPC) mechanism provided by the operating system (such as message queues, shared memory, or other communication mechanisms) can be used to establish an internal communication channel between the underlying or intermediate eye-tracking event service and the upper layers. During system initialization, the eye-tracking event service starts a listening port or registers a service name. When upper-layer components start, they initiate a connection to the eye-tracking event service and register callback interfaces, indicating that they are ready to receive specific types of events. A push subscription table can be maintained to record the process ID, callback address or message queue ID, and subscribed event types for each application connection.

[0169] In this way, whenever the system's underlying layer generates a structured eye-tracking event, the underlying layer or the eye-tracking event service acting as middleware immediately iterates through the push subscription table and sends the eye-tracking event data directly to all upper-layer components (such as applications) that have subscribed to that eye-tracking event type through the established communication channel. The push method can be synchronous (waiting for the application to complete processing before returning) or asynchronous (returning immediately after sending, without waiting for a response). To avoid affecting the real-time performance of event generation, asynchronous non-blocking sending is usually used, and there are no restrictions on this.

[0170] Therefore, in this embodiment, eye-tracking events do not require queuing and polling of an event pool from generation to push to the upper layer. The latency is typically in the millisecond range, making it suitable for scenarios with extremely high real-time requirements, such as games and driving monitoring. This eliminates the need for write, storage, and query operations in the event pool, reducing system overhead such as memory copying and CPU usage. Furthermore, this on-demand push method ensures that upper-layer applications only receive the event types they have subscribed to, avoiding interference from irrelevant events. New applications only need to connect to the service and register to seamlessly receive eye-tracking events without modifying the underlying layer.

[0171] Optionally, in certain application scenarios, this application can also adopt both of the above methods simultaneously. For example, high-priority applications can use push notifications, while ordinary applications can use an event pool. The system can manage this automatically, for example, by downgrading to the event pool when the push notification queue is backed up, to better meet the eye-tracking interaction needs of multiple applications running simultaneously with different requirements. It is understandable that the event pool can serve as a backup storage in case of push notification failure: if an application fails to process push notification events in a timely manner due to process blocking or abnormalities, the system can temporarily store the events in the event pool, and retrieve the lost events through the query interface after the application recovers.

[0172] In summary, this application, through the two distribution mechanisms described above (and potentially other mechanisms as needed), constitutes a flexible, robust, and high-performance eye-tracking event distribution framework, balancing low latency and high reliability. It can intelligently select or combine the most suitable distribution strategies based on different application requirements and system loads, eliminating the need for redundant development of underlying communication logic. This ensures that structured eye-tracking event data reaches where it is needed most effectively, whether it's a system service requiring broadcast notifications or a foreground game application demanding millisecond-level response times. This comprehensively guarantees the availability and superior performance of the method in various complex application scenarios.

[0173] Reference Figure 6 This is a schematic diagram of the structure of an eye-tracking data transmission device proposed in an embodiment of this application, as shown below. Figure 6 As shown, the eye-tracking data transmission device may include:

[0174] Module 61 is used to acquire eye-tracking perception data;

[0175] The generation module 62 is configured to generate structured eye movement event data if the eye movement perception data meets predetermined conditions, wherein the structured eye movement event data includes first indication information representing the type of eye movement event;

[0176] The transmission module 63 is used to transmit the structured eye-tracking event data to the upper layer of the operating system.

[0177] Optionally, the generation module 62 may include:

[0178] The first determining unit is used to analyze the eye movement perception data and determine changes in eye movement state;

[0179] The first generation unit is configured to generate corresponding structured eye movement event data if the eye movement state change satisfies a predetermined condition for any eye movement event type; wherein the predetermined condition is determined based on at least one of a time dimension threshold and a spatial dimension threshold.

[0180] Optionally, structured eye-tracking event data includes data corresponding to at least one of the following eye-tracking event types: eye-closing events, eye-opening events, fixation events, blinking events, continuous blinking events, and gaze beyond device boundaries events.

[0181] The aforementioned eye movement state changes may include at least one of the following: a jump between a first state and a second state; a state jump pair consisting of the first state and the second state; a change in the coordinates of the gaze focus; wherein the first state and the second state are two different states in the open-eye state and the closed-eye state, respectively; the state jump pair includes a first state jump and a second state jump adjacent to it and in the opposite direction.

[0182] Optionally, the structured eye-tracking event data may further include second indication information, which is used to represent the attributes of the eye-tracking event type. The attributes include at least one of the following: the timestamp of the eye-tracking event; the coordinates of the gaze focus when the eye-tracking event occurred; the user identifier that generated the eye-tracking event; and the eye source type that generated the eye-tracking event, which includes the left eye, right eye, or both eyes.

[0183] Based on the above analysis, the generation module 62 may include:

[0184] The first detection unit is used to detect the eye movement state transitioning from a first state to a second state;

[0185] The second determining unit is used to determine the duration of maintaining the second state;

[0186] The second generation unit is used to generate corresponding structured eye movement event data if the duration is greater than the first time threshold. The first indication information in the eye movement event data is used to indicate the closed eye event or the open eye event corresponding to the second state.

[0187] Optionally, the above-mentioned generation module 62 may include:

[0188] The second detection unit is used to detect at least one pair of state transitions occurring during eye movement.

[0189] The first acquisition unit is used to acquire the state interval duration between the first state transition and the second state transition in each state transition pair;

[0190] The third generation unit is used to generate corresponding structured eye movement event data if the state interval of a state transition pair is within the effective time interval, wherein the first indication information in the eye movement event data is used to indicate a blink event.

[0191] The fourth generation unit is used to generate corresponding structured eye movement event data if the state interval of a preset number of consecutive state transitions occurring within a preset time window is within the effective time interval. The first indication information in the eye movement event data is used to indicate continuous blinking events, and the second indication information includes the number of consecutive blinks.

[0192] Optionally, the above-mentioned generation module 62 may include:

[0193] The fifth generation unit is used to generate corresponding structured eye movement event data if the coordinate movement distance of the gaze focus is less than the spatial distance threshold and the duration of the state is greater than the second time threshold. The first indication information in the eye movement event data is used to indicate the gaze event, and the second indication information includes the coordinates of the gaze focus when the gaze event occurs. The second indication information is determined based on multiple gaze coordinates collected within the generation time window of the gaze event.

[0194] Optionally, the above-mentioned generation module 62 may include:

[0195] The sixth generation unit is used to generate corresponding structured eye movement event data if the coordinates of the gaze focus exceed the device boundary range and the duration of the state is greater than a third time threshold. The first indication information in the eye movement event data is used to indicate the gaze exceeding the device boundary event, and the second indication information includes the direction of the excess or the amount of the excess.

[0196] Based on the above analysis, the transmission module 63 may include at least one of the following units:

[0197] A storage unit is used to store the structured eye-tracking event data in the event pool of the operating system for the upper layer of the operating system to read.

[0198] The push unit is used to directly push the structured eye-tracking event data to the upper layer of the operating system through an internal communication process.

[0199] This application also provides a computer-readable storage medium that carries one or more computer programs. When the one or more computer programs are executed by an electronic device, the electronic device can implement any of the eye-tracking data transmission methods provided in this application.

[0200] The computer-readable storage medium can be any available medium that an electronic device can store, or a data storage device such as a training device or data center that integrates one or more available media. Available media can be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., DVDs), or semiconductor media (e.g., solid-state drives (SSDs)).

[0201] This application also provides a computer program product including computer-readable instructions, which, when executed on an electronic device, cause the electronic device to implement any of the eye-tracking data transmission methods provided in this application.

[0202] When computer-readable instructions are loaded and executed on an electronic device, all or part of the processes or functions described in the embodiments of this application are generated. The electronic device may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer-readable instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, computer-readable instructions may be transmitted from one website, computer, training device, or data center to another website, computer, training device, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means, depending on the actual application scenario.

[0203] This application also provides an intelligent program (such as an agent or intelligent assistant) to receive eye-tracking perception data collected by a visual sensor and implement the eye-tracking data transmission method proposed in this application. The implementation process can be referred to the description of the corresponding part of the method embodiment above. In this implementation process, other components of the application program or operating system can also be controlled through interface calls or other interactive methods to respond to the perception results, and there are no limitations on this.

[0204] Reference Figure 7This is a schematic diagram of the hardware structure of an electronic device applicable to the eye-tracking data transmission method proposed in this application. The electronic device can be a smartphone, tablet, laptop, augmented reality (AR) / virtual reality (VR) headset, smart car cockpit system, or any other smart terminal integrating eye-tracking functionality. Figure 7 As shown, there is at least one visual sensor 71 and a processing device 72 coupled to the at least one visual sensor 71. The coupling refers to the connection relationship between the visual sensor 71 and the processing device 72 that enables signal or data transmission, including but not limited to direct electrical connection, indirect electrical connection through intermediate components, bus connection, wireless connection, optical connection, etc.

[0205] The visual sensor 71 can be configured to acquire eye-tracking perception data. It may include a visible light image sensor for capturing eye images, such as RGB or grayscale images of the human eye region, when ambient light is sufficient. It may also include an infrared image sensor that works in conjunction with a matching infrared light source. Infrared light can form a highly reflective "Pulchin spot" on the cornea and a dark spot at the center of the pupil, greatly enhancing pupil tracking accuracy and robustness in low-light conditions or when wearing glasses. It may also include a depth sensor, such as a time-of-flight sensor or a structured light module, for acquiring depth information of the human eye region to assist in more accurate three-dimensional gaze estimation. This application does not limit the type of visual sensor 71 or its connection method with electronic devices.

[0206] The processing device 72 can be configured to implement the eye-tracking data transmission method proposed in this application, and it can be an integration of one or more processing units. These may include a graphics processor or neural processing unit for efficiently performing image processing, computer vision, and deep learning inference tasks. It may also include a digital signal processor or application-specific integrated circuit (ASIC) for hardware acceleration of specific eye-tracking algorithms (such as image feature extraction, filtering, and coordinate calculation) to achieve extreme energy efficiency and low latency. A CPU may also be included as needed.

[0207] In practical applications, refer to Figure 8 The software architecture shown (using the Android operating system as an example; the software framework for other types of operating systems can be adapted accordingly), the processing device 72 includes at least one processing unit (such as...) Figure 8 The algorithm module (used to execute the method proposed in this application) can be located at the hardware layer, independent of the vision sensor 71 or integrated into the vision sensor 71, to implement the method proposed in this application. It transmits the generated structured eye-tracking event data to the system kernel storage, or pushes it to upper layers, such as the system service layer or application layer, through the system kernel. Figure 8The eye-tracking event sensor driver communicates with the hardware layer to cache or transmit the generated eye-tracking event data to the upper layer. This driver can also be used to receive instructions from the upper layer and pass them to the hardware.

[0208] in, Figure 8 The eye-tracking event provider in the system can obtain eye-tracking event data from the kernel-level driver and perform preliminary processing and encapsulation to provide a unified eye-tracking event data interface for the upper layers. The eye-tracking event service in the system framework layer runs as a service in the operating system, responsible for eye-tracking event-related operations, such as receiving data from the eye-tracking event provider and distributing basic eye-tracking events to various applications (Apps). Applications can receive and process eye-tracking events distributed by the eye-tracking event service through the interface provided by the framework layer, implementing interactive functions based on eye-tracking event control, such as scrolling pages and selecting menus based on eye movements.

[0209] The optional processing device 72 can run an intelligent program, which can execute computer programs through the processing device 72 to implement the eye-tracking data transmission method proposed in this application. The intelligent program can be an artificial intelligence assistant embedded in the operating system, which can be activated by voice or specific operations, or it can be an intelligent agent or other program that enables human-computer dialogue interaction, invoking one or more models to perform corresponding tasks as needed through interface calls or other interactive methods. Of course, the intelligent program can also have one or more models; in this case, the models are part of the intelligent program, and can be activated to perform tasks as needed during the operation of the intelligent program. It should be noted that the eye-tracking data transmission method proposed in this application includes, but is not limited to, implementation through an intelligent program.

[0210] Furthermore, the electronic device may also include programs and data necessary for the operation of the electronic device, such as an operating system whose kernel or framework layer contains or integrates eye-tracking event services; device drivers (such as those that control visual sensors and read the eye-tracking perception data they collect); applications (various apps that ultimately consume eye-tracking event data); rule bases (which store predefined rules and thresholds required to generate events); and data acquired or generated during the execution of the method of this application.

[0211] Optionally, the electronic device may also include a system bus that provides a high-speed data channel between sensors, processors, memory and I / O devices, a display screen for displaying an interactive interface, etc. Figure 7 The structure of the electronic device shown does not constitute a limitation on the electronic device in the embodiments of this application. In practical applications, the electronic device may include more than Figure 7The application does not provide detailed examples of the more or fewer components shown, or combinations of certain components, such as gyroscopes, accelerometers, and gravity sensors, other sensing units used to obtain sensing parameters, power management modules, antennas, or other communication elements.

[0212] Finally, it should be noted that the device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. In addition, in the accompanying drawings of the device embodiments provided in this application, the connection relationship between modules indicates that they have a communication connection, which can be implemented as one or more communication buses or signal lines.

[0213] In the above embodiments, the invention can be implemented entirely or partially by software, hardware, firmware, or any combination thereof. Through the description of the above embodiments, those skilled in the art can clearly understand that this application can be implemented using software plus necessary general-purpose hardware, or it can be implemented using dedicated hardware including dedicated integrated circuits, dedicated CPUs, dedicated memory, dedicated components, etc. When implemented using software, it can be implemented entirely or partially in the form of a computer program product. The various embodiments in this specification are described in a progressive or parallel manner, with each embodiment focusing on the differences from other embodiments. Similar or identical parts between embodiments can be referred to mutually. For the apparatuses and electronic devices disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the descriptions are relatively simple, and relevant parts can be referred to the method section.

Claims

1. A method for transmitting eye-tracking data, comprising: Acquire eye-tracking perception data; If the eye movement perception data meets predetermined conditions, structured eye movement event data is generated, and the structured eye movement event data includes first indication information representing the type of eye movement event; The structured eye-tracking event data is transmitted to the upper layers of the operating system.

2. The method according to claim 1, wherein if the eye-tracking perception data meets predetermined conditions, structured eye-tracking event data is generated, comprising: The eye movement perception data is analyzed to determine changes in eye movement state; If the eye movement state change meets the predetermined conditions for any eye movement event type, the corresponding structured eye movement event data is generated; The predetermined conditions are determined based on at least one of a time dimension threshold and a spatial dimension threshold.

3. The method according to claim 1, wherein the structured eye movement event data includes data corresponding to at least one of the following eye movement event types: Eye-closing events, eye-opening events, gaze events, blinking events, continuous blinking events, and gaze beyond device boundaries events.

4. The method according to claim 2, wherein the change in eye movement state includes at least one of the following: The transition between the first state and the second state; The state transition pair formed by the first state and the second state; Changes in the coordinates of the focal point of the line of sight; in, The first state and the second state are two different states, one with eyes open and the other with eyes closed. The state transition pair includes a first state transition and a second state transition that is adjacent to it and has the opposite direction.

5. The method according to claim 1, wherein the structured eye-tracking event data further includes second indication information, the second indication information being used to represent an attribute of the eye-tracking event type, the attribute including at least one of the following: Timestamps of eye-tracking events; Coordinates of the focal point of the gaze during an eye-tracking event; User identifier that generated the eye-tracking event; The eye source type that generates the eye movement event, wherein the eye source type includes the left eye, the right eye, or both eyes.

6. The method according to claim 4, wherein if the eye-tracking perception data meets predetermined conditions, structured eye-tracking event data is generated, comprising: Detect the eye movement state from the first state to the second state; Determine the duration for which the second state is maintained; If the duration exceeds the first time threshold, corresponding structured eye movement event data is generated. The first indication information in the eye movement event data is used to indicate the closed eye event or the open eye event corresponding to the second state.

7. The method according to claim 4, wherein if the eye-tracking perception data meets predetermined conditions, structured eye-tracking event data is generated, comprising: Detect at least one pair of state transitions occurring during eye movement; Obtain the state interval duration between the first state transition and the second state transition in each state transition pair; If the state interval of a state transition pair occurs within the valid time interval, corresponding structured eye movement event data is generated, and the first indication information in the eye movement event data is used to indicate the blink event; If the state interval of a preset number of consecutive state transitions occurring within a preset time window is within the effective time interval, corresponding structured eye movement event data is generated. The first indication information in the eye movement event data is used to indicate consecutive blinking events, and the second indication information includes the number of consecutive blinks.

8. The method according to claim 4, wherein if the eye-tracking perception data meets predetermined conditions, structured eye-tracking event data is generated, including any one of the following: If the coordinate movement distance of the gaze focus is less than the spatial distance threshold, and the duration of this state is greater than the second time threshold, corresponding structured eye movement event data is generated. The first indication information in the eye movement event data is used to indicate the gaze event, and the second indication information includes the coordinates of the gaze focus when the gaze event occurs. The second indication information is determined based on multiple gaze coordinates collected within the generation time window of the gaze event. If the coordinates of the gaze focus exceed the device boundary range and the duration of this state is greater than a third time threshold, corresponding structured eye movement event data is generated. The first indication information in the eye movement event data is used to indicate the gaze exceeding the device boundary event, and the second indication information includes the direction of excess or the amount of excess offset.

9. The method according to claim 1, wherein the transmission of the structured eye-tracking event data to the upper layer of the operating system comprises any one of the following methods: The structured eye-tracking event data is stored in the operating system's event pool for the upper layers of the operating system to read. The structured eye-tracking event data is pushed directly to the upper layer of the operating system through an internal communication process.

10. An electronic device, comprising: At least one visual sensor is used to collect eye-tracking perception data; A processing device, coupled to the at least one vision sensor, is configured to: Acquire the eye-tracking perception data; If the eye movement perception data meets predetermined conditions, structured eye movement event data is generated, and the structured eye movement event data includes first indication information representing the type of eye movement event; The structured eye-tracking event data is transmitted to the upper layers of the operating system.

Images