Input event simulation method and electronic device

Abstract

The application discloses an input event simulation method and an electronic device, relates to the technical field of human-computer interaction, and receives sensing input event data. If the sensing input event data meets predetermined simulation conditions, the sensing input event data is converted into simulation original input event data based on preconfigured event conversion rules. The simulation original input event data is consistent with original input event data generated by a physical input device in terms of data structure. The simulation original input event data is injected into an operating system, and the operating system distributes the simulation original input event data to a native input interface of a target application in an application layer.

Description

Technical Field

[0001] This application relates to the field of human-computer interaction technology, and in particular to an input event simulation method and electronic device. Background Technology

[0002] With the rapid development of sensor technology and artificial intelligence algorithms, electronic devices are able to perceive more and more contactless user input, such as eye tracking, gesture recognition, voice commands, and head posture. These perceptual inputs provide users with more natural and convenient ways to interact.

[0003] However, existing operating systems and applications mainly rely on native input events generated by traditional physical input devices (such as mice, keyboards, and touchscreens) for interaction. This makes it impossible for applications to directly recognize and process the data generated by the aforementioned sensory inputs. Each application needs to develop its own processing logic and input interface for various types of sensory input data. This not only increases the application development cost, but also results in inconsistent user experiences due to the different implementation logic of different applications. Summary of the Invention

[0004] In view of the above problems, this application provides the following solution:

[0005] A first aspect of this application provides an input event simulation method, the method comprising:

[0006] Receive sensor input event data;

[0007] If the perceived input event data meets the predetermined simulation conditions, the perceived input event data is converted into simulated native input event data based on the pre-configured event conversion rules; wherein the simulated native input event data is consistent with the native input event data generated by the physical input device in terms of data structure;

[0008] The simulated native input event data is injected into the operating system, which then distributes it to the native input interface of the target application in the application layer.

[0009] In one possible implementation, the perceived input event data includes at least one of the following: eye-tracking event data, gesture event data, voice command data, and head posture data; the native input event data includes at least one of the following: mouse event data, keyboard event data, and touch event data.

[0010] In one possible implementation, the perceived input event data is converted into simulated native input event data using at least one simulator, each simulator corresponding to a physical input device type. The conversion process includes:

[0011] The sensory input event data is parsed to determine the type of the sensory input event and at least one associated attribute parameter, wherein the attribute parameter includes at least one of spatial attribute parameters, temporal attribute parameters, state attribute parameters and source attribute parameters;

[0012] Based on pre-configured event conversion rules, the perceived input event type is converted into a simulated native input event type, and the at least one attribute parameter is converted into a simulated attribute parameter associated with the simulated native input event type; the simulated native input event type corresponds to the native input event type of the physical input device;

[0013] Based on the native input event data structure, the simulated native input event type and the simulated attribute parameters are encapsulated into structured simulated native input event data.

[0014] In one possible implementation, the event transition rule supports at least one of the following transition modes:

[0015] Multiple perceptual input events occurring within preset spatiotemporal constraints will be combined and converted into a single simulated native input event;

[0016] The continuously changing attribute parameters in the perceived input event are converted into simulated input parameters with continuously changing characteristics in the simulated native input event.

[0017] In one possible implementation, if the perceived input event data is eye-tracking event data, the perceived input event type includes eye-tracking event types defined by one or more combinations of the following: eye-opening event, eye-closing event, gaze event, blinking event, continuous blinking event, and gaze beyond device boundary event.

[0018] The attribute parameters associated with the eye-tracking event type include at least one of the following: occurrence timestamp, gaze focus coordinates, number of event combinations, duration, offset or direction exceeding the boundary, user identifier, and eye source type; the eye source type includes left eye, right eye, or both eyes;

[0019] The simulated attribute parameters include at least one of the following associated with mouse events, keyboard events, or touch events: coordinate parameters, button parameters, status parameters, and device identifier parameters.

[0020] In one possible implementation, the event transformation rule includes at least one of the following transformation relationships:

[0021] Convert the gaze focus coordinates into cursor position parameters or touch point coordinate parameters;

[0022] Convert the number of event combinations into a click action count parameter or a key repetition count parameter;

[0023] Convert the duration of the closed-eye event or the open-eye event into a press duration parameter that simulates the native input event;

[0024] Convert the offset of the line-of-sight-beyond-device boundary event into a scroll increment parameter or a sliding distance parameter that simulates the native input event;

[0025] Convert the combination of gaze-out-of-device-boundary events and gaze events into a continuous scrolling event sequence or a continuous touch-movement event sequence;

[0026] Convert the eye source type into a distinguishing identifier for different event types or different key values ​​that simulate native input events.

[0027] In one possible implementation, the predetermined simulation conditions include at least one of the following:

[0028] The system has enabled the input sensing control mode, or at least one simulator is enabled.

[0029] The confidence level or recognition quality of the perceived input event data is higher than a preset threshold.

[0030] In one possible implementation, the simulated native input event data is injected into the operating system, including any of the following implementation methods:

[0031] The simulated native input event data is submitted to the input event queue of the native input system by calling the native input event injection interface provided by the operating system's native input system.

[0032] The simulated native input event data is submitted to the operating system's native input system using the same communication protocol as the data reported by the physical input device via the simulated input device.

[0033] The native input system is an input system used to receive native input event data generated by the physical input device and distribute it to the application layer.

[0034] A second aspect of this application provides an electronic device, comprising:

[0035] At least one memory stores pre-configured event transition rules;

[0036] At least one processor is configured to run the operating system's awareness event service and perform the following operations:

[0037] Receive sensor input event data;

[0038] If the perceived input event data meets the predetermined simulation conditions, the perceived input event data is converted into simulated native input event data based on the pre-configured event conversion rules; wherein the simulated native input event data is consistent with the native input event data generated by the physical input device in terms of data structure;

[0039] The simulated native input event data is injected into the operating system, which then distributes it to the native input interface of the target application.

[0040] In one possible implementation, the electronic device further includes:

[0041] At least one sensing device is configured to collect corresponding sensing data;

[0042] The at least one processor is further configured to: generate the sensing input event data based on the sensing data, and send the sensing input event data to the sensing event service;

[0043] The sensing device includes an eye-tracking sensor, and the sensing event service includes an eye-tracking event service. Attached Figure Description

[0044] The above and other features, advantages, and aspects of the embodiments of this disclosure will become more apparent from the accompanying drawings and the following detailed description. Throughout the drawings, the same or similar reference numerals denote the same or similar elements. It should be understood that the drawings are schematic, and the originals and elements are not necessarily drawn to scale.

[0045] Figure 1 This is a flowchart illustrating the input event simulation method proposed in Embodiment 1 of this application;

[0046] Figure 2 This is a flowchart illustrating the input event simulation method proposed in Embodiment 2 of this application;

[0047] Figure 3 This is a flowchart illustrating the input event simulation method proposed in Embodiment 3 of this application;

[0048] Figure 4 This is a schematic diagram of the structure of an input event simulation device proposed in an embodiment of this application;

[0049] Figure 5 This is a schematic diagram of the hardware structure of an electronic device proposed in this application;

[0050] Figure 6 This is a schematic diagram of the software architecture of the electronic device proposed in this application. Detailed Implementation

[0051] The embodiments of this application are described below with reference to the accompanying drawings. The terminology used in the implementation section of this application is only for explaining specific embodiments and is not intended to limit the application. Those skilled in the art will recognize that, with technological advancements and the emergence of new scenarios, the technical solutions provided in the embodiments of this application are also applicable to similar technical problems.

[0052] The terms "first," "second," etc., used in the context of this application and the aforementioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such terms are interchangeable where appropriate; this is merely a way of distinguishing objects with the same attributes in the embodiments of this application. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion, so that a process, method, system, product, or apparatus that comprises a series of units is not necessarily limited to those units, but may include other units not explicitly listed or inherent to those processes, methods, products, or apparatuses.

[0053] It is understood that before using the technical solutions disclosed in the embodiments of this application, users should be informed of the types, scope of use, and usage scenarios of the personal information involved in this application in an appropriate manner in accordance with relevant laws and regulations, and user authorization should be obtained. For example, in response to receiving a user's active request, a pop-up window may be used, and a textual prompt message may be presented in the pop-up window to clearly inform the user that the requested operation will require the acquisition and use of the user's personal information. The user can choose whether to provide personal information to the electronic device, application, server, or storage medium or other software or hardware that performs the operation of this application based on the prompt message. This application does not limit the prompt message and the method of user authorization implementation. The data involved in this technical solution (including but not limited to the data itself, the acquisition or use of the data) shall comply with the requirements of relevant laws and regulations and related provisions (such as corporate regulations).

[0054] As described in the background section, the data structure of sensory input events generated by new sensing devices (such as eye trackers, gesture recognition sensors, microphone arrays, or head posture trackers) differs fundamentally from the native input event data structure of traditional physical input devices (such as mice, keyboards, and touchscreens). This difference prevents existing applications (APPs) from directly recognizing and responding to the new sensory inputs, severely limiting the applicability of these devices. Furthermore, each APP develops and integrates its own sensory input processing logic (each integrating its own sensory interaction SDK (Software Development Kit), etc.), resulting in inconsistent user experiences (a sense of fragmentation). Additionally, each APP needs to repeatedly implement complex mapping algorithms from sensory input to business logic, wasting system resources and development costs.

[0055] To address this, a further proposal is to add a sensory input service at the operating system framework layer, converting sensory data into general events and distributing them to applications. However, in vertical applications, due to the incompatibility of general events with the data structures of the native input interfaces (such as Windows' Raw Input, Android's InputManagerService, and Linux's evdev) relied upon by existing applications, existing applications still need to modify their source code and configure the corresponding type of input interface to receive sensory input event data generated by these new sensory devices, failing to achieve plug-and-play functionality. Furthermore, this current solution fails to effectively utilize the existing native input distribution channels of the operating system, resulting in redundant event processing paths and increased latency. Therefore, it is crucial to seamlessly and transparently integrate sensory input events into the existing, mature operating system input processing ecosystem, enabling existing applications to respond to sensory input through their original native input interfaces without any modification, thereby achieving a natural "plug-and-play" interactive experience and reducing development costs. Therefore, this application proposes an input event simulation method, which will be described in detail below with reference to the accompanying drawings.

[0056] Reference Figure 1 This is a flowchart illustrating the input event simulation method proposed in Embodiment 1 of this application. The method proposed in this application can be executed by a perceptual event service (e.g., eye-tracking event service, gesture recognition service, etc.) in the operating system of an electronic device (such as a mobile phone, tablet computer, VR / AR device, smart car, or other user terminal), used to convert various perceptual input events into simulated native input events, enabling existing applications to respond without adaptation. Figure 1 As shown, the input event simulation method proposed in this embodiment may include:

[0057] Step S11: Receive sensor input event data.

[0058] The perceived input event data is structured data with preliminary semantics (such as "open eyes", "close eyes", "gaze", "wave to the left", "say 'click'" or "nod") generated from perceived data from novel sensing devices. It can include at least one of the following: eye-tracking event data (generated by an eye tracker), gesture event data (generated by a depth camera or radar sensor), voice command data (generated by a microphone array and speech recognition engine), and head posture data (generated by an IMU (Inertial Measurement Unit) or a visual tracking system). It represents the user's interactive intent expressed through non-physical contact methods such as eye movements, gestures, and voice. In this application, the data structure of the perceived input event data can be defined by the sensing device's SDK or system sensing service, and differs from the data structure of the operating system's native input events.

[0059] In some embodiments, perceived input event data can be reported by drivers in the operating system kernel layer (KERNEL) or hardware abstraction layer (HAL), or sent by perceived event services (such as eye-tracking event services and gesture recognition services) in the operating system framework layer through inter-process communication mechanisms (such as Binder, Socket, and message queues). For example, the eye-tracking event service sends parsed eye-tracking event data (such as gaze events and blink events) in a structured format to the input event simulation module (such as a simulator); the gesture recognition service sends the recognized gesture event data (such as pinch, swipe, and point) to the same module. This application does not limit the source and content of perceived input event data, and can be determined as appropriate.

[0060] Optionally, the sensing input event data can be generated by the operating system's lower layers (such as the hardware layer, driver layer, or kernel layer) and reported to the upper-layer sensing event service for subsequent steps. The sensing device integrated into the hardware layer, or the processing chip communicating with it, executes the corresponding sensing input event generation logic / program, analyzes the sensing data currently collected by the sensing device, and generates corresponding structured sensing input event data according to the sensing input event transmission protocol, providing it to the upper-layer sensing event service. Alternatively, the generated structured event data indicating the basic sensing event (which corresponds to a single sensing event behavior / action) can be stored in the system kernel or sensing event service, so that when the application layer accesses it, it can perform subsequent operations on the cached corresponding sensing input event data.

[0061] In some embodiments, the perceptual event service can also perform combination and transformation processing on the received basic perceptual events to generate advanced perceptual event data with composite semantics (which correspond to multiple combinations of the same perceptual event behavior, or combinations of different types of perceptual event behavior, etc.), so that the intention expression of the perceptual interaction based thereon is richer and more accurate. Therefore, this application can convert the basic eye-tracking event data reported from the underlying layer into advanced eye-tracking event data with a higher semantic level according to the pre-stored perceptual event conversion rules. Step S11 directly receives the converted advanced eye-tracking time data. Among them, the perceptual event conversion rules define how to convert one or more basic eye-tracking event data into advanced eye-tracking event data with richer semantics, so that it is different from the random or continuous perceptual event stream at the bottom layer of the system. It can be stored in the system firmware or configuration file, or it can be dynamically registered by the user or application, etc. This application does not limit its configuration method and content.

[0062] Therefore, this embodiment achieves unified aggregation of multi-source heterogeneous sensing inputs. Through a standardized receiving interface, event data from different sensing modalities are incorporated into a unified processing pipeline, laying the foundation for subsequent system-level unified simulation and distribution. By decoupling the acquisition of sensing devices (new sensing devices) from the consumption of sensing events, resource competition and logical fragmentation caused by direct interaction between applications and hardware are avoided.

[0063] Step S12: If the perceived input event data meets the predetermined simulation conditions, the perceived input event data is converted into simulated native input event data based on the pre-configured event conversion rules; wherein, the simulated native input event data and the native input event data generated by the physical input device are consistent in data structure.

[0064] In this embodiment, not all received sensor input event data is directly subjected to simulation conversion and injection operations. For each received sensor input event data, it is first determined whether it meets predetermined simulation conditions to avoid false injections caused by factors such as sensor input control not being enabled, low recognition quality, or unauthorized user access, thus ensuring the controllability and security of the system. As can be seen, predetermined simulation conditions are used to filter unnecessary conversions and avoid resource waste. In other words, predetermined simulation conditions are the switch that triggers the simulation process, and can include system-level switch states (such as whether the sensor input control mode is enabled, whether the corresponding simulator is enabled) and data quality thresholds (such as whether the confidence level or recognition quality of the sensor input event data is higher than a preset threshold). Only when the conditions are met is the subsequent conversion process triggered, preventing invalid or low-quality data from entering the subsequent processing stream and saving system resources.

[0065] To achieve the simulated transformation of perceived input event data, this application maintains an event transformation rule base. These rules define the transformation relationships between perceived input event types and native input event types, as well as the transformation logic between attribute parameters. Thus, by selecting a perceived input event type that meets predetermined simulation conditions, the corresponding event transformation rule can be found in the rule base, and the transformation logic executed. Native input event data is a standard data structure defined by the operating system's native input system (such as Windows' Raw Input, Linux's evdev subsystem, and Android's InputManagerService), and can be dynamically adjusted as the standard data structure is updated.

[0066] Based on the physical input device types supported by the application described above, native input event data can include mouse event data (such as mouse movement (MOUSEEVENTF_MOVE), mouse button press / release (MOUSEEVENTF_LEFTDOWN / UP, MOUSEEVENTF_RIGHTDOWN / UP, MOUSEEVENTF_MIDDLEDOWN / UP), mouse wheel scroll (MOUSEEVENTF_WHEEL), mouse horizontal scroll (MOUSEEVENTF_HWHEEL), and mouse double-click (achieved through a rapid combination of two consecutive presses / releases), keyboard event data (such as key press (KEYEVENTF_KEYUP) and key release (KEYEVENTF_KEYDOWN)), and touch event data (such as touch press (ACTION_DOWN), touch movement (ACTION_MOVE), touch release (ACTION_UP), and multi-touch (POINTER_DOWN / UP), etc.). Of course, as the operating system is updated, the allowed native input event types can increase, including but not limited to these three; this application only uses this as an example.

[0067] This application aims to achieve a transformation process that renders the operating system and application layer unable to distinguish between received input events originating from real physical input devices and simulated input events, thereby ensuring that existing applications can process these input events indiscriminately. Therefore, through transformation processing based on event transformation rules, it ensures that the output simulated native input event data is structurally identical (i.e., fully compatible) with the native input event data generated by physical input devices, possessing the same data field definitions, event type encoding methods, and transmission protocols. For example, an eye-tracking "gaze event data" might be transformed into a "simulated mouse movement event data packet" with the same byte sequence as a real mouse movement event.

[0068] In this embodiment, the "simulated" in "simulated native input event data" refers to its functional imitation of the interactive input of physical input devices; the "native" refers to its data structure being completely consistent with the input event data (i.e., native input event data) generated by physical input devices such as mice, keyboards, and touchscreens and natively processed and recognized by the operating system. For example, taking an eye-tracking interaction scenario, if the received gaze event data meets predetermined simulation conditions, based on pre-configured event conversion rules, the gaze focus coordinates (640, 360) are converted into coordinate parameters for a mouse movement event, and the two consecutive blink events are converted into a left-click double-click event. The generated simulated native input event data is a standard mouse event structure, such as [Input Event Type = MOUSEEVENTF_MOVE][Coordinate X = 640][Coordinate Y = 360], and [Input Event Type = MOUSEEVENTF_LEFTDOWN | MOUSEEVENTF_LEFTUP][Repetition Count = 2], whose data structure is completely consistent with the data structure reported by the physical mouse driver.

[0069] Therefore, this embodiment achieves cross-modal conversion from the "perceptual semantic space" to the "native input space." Through consistent configuration of data structures, the operating system and application layer treat perceptual input as input from standard physical input devices, thereby completely solving the compatibility issues of existing applications. By determining predetermined simulation conditions, the controllability of conversion and injection is ensured, avoiding accidental triggering and unauthorized input simulation, thus improving the security and reliability of the system.

[0070] Step S13: Inject simulated native input event data into the operating system, and distribute it from the operating system to the native input interface of the target application in the application layer.

[0071] After generating simulated native input event data through the above steps, the system injects it into the operating system's standard input processing pipeline. This can be done by calling specific interfaces provided by the system or simulating hardware behavior. Once successfully injected, the simulated native input event data appears identical to native input event data from physical input devices to the system. The operating system can then utilize existing input event distribution mechanisms to distribute the data to the native input interfaces of target applications (such as applications that have registered / subscribed / requested this type of input event). It is understood that during this distribution process, the distributed input event data may be simulated native input event data from the perceptual event service or native input event data from physical input devices. The operating system processes and distributes the simulated native input event data according to the same processing path as the native input event data; the implementation process is not detailed in this application.

[0072] In this context, the application's native input interface refers to the application's pre-existing interface for handling physical input device events. This means it's a standard, native callback interface (standard callback function or message handling function) that is written during application development to receive physical input events. Examples include Windows' WndProc message handling function or Android's onTouchEvent callback function. These are part of the standard framework provided to applications by the operating system, not developed separately for sensing input events. The target application can respond to the received input event data (simulated native input event data or native input event data) through the native input interface and execute corresponding business logic. During this response process, even if simulated native input event data is distributed to the target application, the target application will respond as if it had received real native input event data.

[0073] As can be seen, the steps in this embodiment complete the closed loop from sensing input to application response. By reusing the mature native input distribution channel of the operating system, it avoids building parallel input event distribution paths, reducing system complexity and event latency. Since the simulated native input event data is consistent with the data structure of the physical input device (native input event data structure), the target application can respond to new sensing inputs (such as the non-contact sensing input mentioned above) without any modification or adaptation, achieving "zero-modification" compatibility, greatly expanding the applicability of new sensing devices, and reducing ecosystem migration costs.

[0074] For example, this application injects the generated simulated mouse movement event data and double-click event data into the Windows operating system via the SendInput API. The operating system distributes the mouse movement event data to the currently focused drawing application (the target application). After receiving the WM_MOUSEMOVE message, the target application updates the cursor position to (640, 360); subsequently, upon receiving the WM_LBUTTONDBLCLK message, it triggers the double-click confirmation logic (such as selecting an object or opening a file) in the target application. Throughout this process, the target application is completely unaware that the input originates from the eye tracker, treating it as a normal mouse operation.

[0075] In summary, this application's embodiments, through the technical path of "condition triggering → data structure consistency conversion → system-level injection," achieve the "transparent" simulation of perceptual input events as native input events. For applications, what they receive and process is a standard touch, keyboard, or mouse event (a native input event already adapted to the application), without needing to know that the input event originates from perceptual input or to perform any code-level adaptation. That is, existing applications can obtain perceptual input interaction capabilities without any modification, fundamentally solving the compatibility problem between perceptual interaction (a new type of interaction) and the existing application ecosystem. This provides users with a seamless "out-of-the-box" interactive experience, while also relieving application developers of the tedious development burden of adapting to support multiple perceptual interactions, thus greatly lowering the technology adoption threshold. Simultaneously, it ensures the consistency of cross-application interaction logic, saving development costs and system resources.

[0076] Reference Figure 2 This is a flowchart illustrating the input event simulation method proposed in Embodiment 2 of this application. This embodiment describes in detail the conversion process implemented through at least one simulator. The simulator can be a pre-created simulator instance based on the physical input device type. It can exist in the form of a software module, system service, or independent process, and is responsible for converting perceived input events of a specific type or source into native input events of the corresponding physical input device type. It is evident that each simulator corresponds to one physical input device type. The system can pre-create at least one of the following: mouse simulator, keyboard simulator, touchscreen simulator, etc. This application does not limit the method of simulator creation.

[0077] Based on this, such as Figure 2 As shown, the above conversion process implemented by at least one simulator as proposed in this embodiment may include:

[0078] Step S21: Analyze the perceived input event data to determine the type of perceived input event and at least one associated attribute parameter;

[0079] When the simulator is enabled, it can receive sensory input event data from the operating system's underlying layer. This sensory input event data typically includes indications representing the type of sensory input event and its associated attribute parameters, such as binary instruction codes or attribute parameter descriptions. The simulator performs structured parsing on the received sensory input event data, extracting the core semantic elements (key information): the type of sensory input event and at least one associated attribute parameter, etc. This parsing method is not detailed in this application.

[0080] The sensory input event type identifies the category of interactive action performed by the user through the sensing device, such as "gaze event," "pinch gesture," and "voice confirmation command." The attribute parameters provide a detailed context for the event and may include at least one of spatial attribute parameters, temporal attribute parameters, state attribute parameters, and source attribute parameters. This can be determined based on the corresponding sensory input event type and its corresponding business logic requirements, and this application does not impose any restrictions on this.

[0081] In practical applications, spatial attribute parameters can describe the location or geometric features of the perceived input event in space, such as gaze focus coordinates, gesture spatial coordinates (gesture knuckle coordinates), gesture trajectory, gesture posture angle, or head posture angle (yaw, pitch, roll), etc. Temporal attribute parameters can describe the temporal characteristics of the perceived input event, such as event duration, event interval duration, number of event combinations, or event timestamp, etc. State attribute parameters can describe the state or intensity of the perceived input event, such as eyelid opening / closing degree, pupil diameter change, gesture pinching degree, gesture confidence, voice confidence, or touch pressure value, etc. Source attribute parameters can describe the source information of the perceived input event, such as sensor identifier, perception processing framework version, acquisition device identifier, user identifier, gesture hand shape, or eye source type (left eye / right eye / both eyes), etc.

[0082] Taking eye-tracking event data as an example, the perceptual input event types include one or more of the following eye-tracking event types defined by combination: eye-opening event (eye movement state changes from closed to open and lasts for more than a preset duration), eye-closing event (eye movement state changes from open to closed and lasts for more than a preset duration), fixation event (the distance the gaze focus moves is less than a spatial threshold and the duration is greater than the fixation time threshold), blinking event (completing one eye-opening → eye-closing → eye-opening state transition pair and the interval duration is within the effective range), continuous blinking event (multiple effective blinks within a preset time window), gaze beyond device boundary (the gaze exceeds the preset boundary in the X-axis or Y-axis direction of the display device / screen, the gaze focus exceeds the effective area of ​​the screen / device, and lasts for a certain duration), etc., at least one of the following basic eye-tracking events. The corresponding eye movement event type is associated with at least one of the following attribute parameters: occurrence timestamp, gaze focus coordinates, number of event combinations (the number of consecutive occurrences of the same type of basic eye movement event within a preset time window, such as the number of consecutive blinks), duration, offset or direction exceeding the boundary, user identifier, and eye source type.

[0083] Among them, when the eye movement event type is defined by a combination of one or more low-level, physiological basic eye movement events listed in the above paragraph, the eye movement event type can be: multiple combinations of the same basic eye movement event, combinations of multiple different basic eye movement events, or combinations of visual field beyond device boundary events with other basic eye movement events, such as "three consecutive blinks", "fixation and blinking", "visual field beyond lower boundary and fixation", etc., which are high-level eye movement events with complex semantics.

[0084] For example, the simulator parses the received eye-tracking event data as follows: [Perceptual Input Event Type = Continuous Blinking Event][Number of Event Combinations = 3][Occurrence Timestamp = T1][Eye Source Type = Right Eye][Gaze Focus Coordinates = (500, 300)]. The perceptual input event type is extracted as "Continuous Blinking Event", and the attribute parameters include: Number of Event Combinations = 3 (temporal attribute parameter), Eye Source Type = Right Eye (source attribute parameter), and Gaze Focus Coordinates = (500, 300) (spatial attribute parameter).

[0085] Understandably, if the perceived input event data is gesture event data, the gesture event type can be divided into static gestures (such as fist, palm, scissor gesture) and dynamic gestures (such as waving, pinching, grasping, swiping). The associated attribute parameters can include at least one of the following: hand joint coordinates, gesture trajectory, palm orientation, and degree of pinching. If the perceived input event data is voice command data, the corresponding perceived input event type can be divided into those containing wake words, control commands (such as "confirm," "cancel," "page turn," etc.), and continuous voice input. The associated attribute parameters can include at least one of the following: recognized text, confidence level, voiceprint features, and command intent. If the perceived input event data is head posture data, the corresponding head posture type can include: head rotation (yaw, pitch, roll), head position movement, nodding, and shaking. The associated attribute parameters can include at least one of the following: posture angle, displacement vector, motion velocity, and acceleration.

[0086] Therefore, the steps in this embodiment achieve semantic decoupling of the perceived input event data, decomposing the compact structured data into independent sets of type identifiers and attribute parameters, providing clear operational objects for subsequent element-by-element mapping. Standardized attribute classifications (such as spatial, temporal, state, and source) make the configuration of transformation rules more systematic and scalable.

[0087] Step S22: Based on the pre-configured event conversion rules, the perceived input event type is converted into a simulated native input event type, and at least one parsed attribute parameter is converted into a simulated attribute parameter associated with the simulated native input event type.

[0088] After parsing the type and attribute parameters of the perceived input event, pre-configured event conversion rules can be queried to perform bidirectional conversion: event type conversion and attribute parameter conversion. It can be seen that the event conversion rules define the correspondence (conversion relationship) from (perceived input event type, attribute parameter) to (simulated native event type, simulated attribute parameter) to realize the conversion of various perceived inputs to native inputs. It should be noted that the simulated native input event type in this application corresponds to the native input event type of the physical input device, and the simulated native input event can be equated with the corresponding type of native input event.

[0089] The conversion of sensory input event types can include: gaze event → mouse movement event, single blink event → left mouse click, continuous blink event → multiple mouse clicks or repeated keyboard key presses, pinch gesture → touch zoom event, voice "confirm" → keyboard Enter key press, head nod → keyboard down arrow key, etc. The conversion of attribute parameters can include: gaze focus coordinates → mouse cursor coordinates (X, Y) or touch point coordinates; event combination count → number of mouse clicks or key repetitions; duration → key press duration or long press state; offset exceeding the limit → scroll wheel increment or sliding distance; eye source type → differentiation of different key values ​​(left eye → left button, right eye → right button), etc. Therefore, simulated attribute parameters include at least one of the following associated with mouse events, keyboard events, or touch events: coordinate parameters, key parameters, status parameters, and device identification parameters. The parameter content can be determined according to the actual situation and is not limited.

[0090] In practical applications, since each simulator corresponds to a specific physical input device type, it focuses on handling the simulation conversion to that device type. The system can select to activate one or more simulators based on the characteristics of the perceived input event and the current application scenario. For example, based on event conversion rules, the system converts a "continuous blinking event" (event combination count = 3) into a triple left mouse button event (MOUSEEVENTF_LEFTDOWN + MOUSEEVENTF_LEFTUP, repetition count = 3); converts the gaze focus coordinates (500, 300) into mouse cursor coordinates (500, 300); and converts the eye source type = right eye into the device identifier parameter = 0x02 (indicating an auxiliary mouse device). For the keyboard simulator, the system converts the voice "next page" into a keyboard Page Down key event (VK_NEXT), etc.

[0091] As can be seen, the steps in this embodiment achieve accurate translation from perceived semantics to native input semantics, and decompose complex perceived input into standard physical input device-level operations through a simulator architecture. By using pre-configured rules instead of hard-coded logic, the system can flexibly adapt to different sensing devices and application scenarios, and hot updates of rules do not require modification of system code, significantly improving maintainability and scalability.

[0092] Step S23: According to the native input event data structure, the simulated native input event type and simulated attribute parameters are encapsulated into structured simulated native input event data.

[0093] After determining the simulated native input event type and simulated attribute parameters, it can be encapsulated according to the desired native input event data structure. Different operating systems have different native input event data structures: Windows uses the INPUT (Input Device Structure) structure, which describes the attributes and configuration information of a physical input device; Android uses MotionEvent or KeyEvent objects, filling in fields such as action type, coordinates, key code, and timestamp through the constructor or obtain method; Linux uses the input_event structure, writing different field contents through the uinput device. There are no restrictions on this.

[0094] During the implementation of step S23, the simulated native input event type and simulated attribute parameters can be filled into the corresponding positions strictly according to the field definitions and encoding specifications of the structure used by the operating system. This ensures that the generated simulated native input event data packet (such as a binary data stream) has the same layout at the byte level as the data reported by the physical device driver. In other words, the data packet is completely identical to the native input event data reported by the physical input device in terms of memory layout and field semantics (type, encoding, value, timestamp, etc.).

[0095] For example, encapsulating Windows mouse events: In the INPUT structure, type=INPUT_MOUSE, mi.dx=500, mi.dy=300, mi.dwFlags=MOUSEEVENTF_MOVE|MOUSEEVENTF_ABSOLUTE, mi.mouseData=0, mi.time=0, mi.dwExtraInfo=0. Encapsulating keyboard events: In the INPUT structure, type=INPUT_KEYBOARD, ki.wVk=VK_RETURN, ki.wScan=0, ki.dwFlags=0, ki.time=0, ki.dwExtraInfo=0. It can be seen that the steps in this embodiment, by strictly following the operating system's native input data structure, ensure that the operating system's input stack can parse and process these events without discrimination. This application does not describe in detail the standardized encapsulation process for simulating native input event data.

[0096] In summary, this embodiment, through a modular "simulator" design and a "parsing-conversion-encapsulation" processing flow, achieves flexible and configurable conversion of different types of perceptual input events to native input events of different types of physical input devices. This design flexibly adapts to multiple perceptual input sources and multiple physical device types, and allows the system to easily expand to new perceptual interaction methods (adding corresponding parsing logic and mapping rules) or support new simulation device types (adding a new simulator), exhibiting good scalability and maintainability.

[0097] In some embodiments, the event transition rules proposed in this application may also include higher-level transition modes such as event combination transitions and event sequential transitions. These transition rules enable the transformation of complex and natural continuous user behaviors (such as gazing and blinking, or scanning a boundary) into precise, discrete, or continuous interaction commands, which are richer and more natural than simple "button transitions," significantly improving interaction efficiency and user experience. Based on this, refer to... Figure 3 This is a flowchart illustrating the input event simulation method proposed in Embodiment 3 of this application. This embodiment describes the process of converting perceived input event data into simulated native input event data based on these conversion modes, such as... Figure 3 As shown, the conversion process may include:

[0098] Step S31: Analyze the perceived input event data to determine the type of perceived input event and at least one associated attribute parameter.

[0099] Step S32: Based on the type and attribute parameters of the perceived input event, the combination of multiple perceived input events occurring within the preset spatiotemporal constraints is converted into a single simulated native input event.

[0100] In this embodiment, a user's natural interaction behavior often consists of multiple consecutive perceptual input events. These events are correlated in time and space, collectively expressing a complete interaction intent. The system aggregates these scattered perceptual input events into a single simulated native input event with complex semantics through event combination rules, matching the existing application's response logic to the combined input. This allows multiple perceptual input events occurring sequentially within a preset time window to be merged into a single simulated native input event.

[0101] The preset spatiotemporal constraints can include time constraints (such as the time difference between the occurrence of multiple events being less than a preset synchronization threshold, or the events occurring within a preset time window) and spatial constraints (such as the spatial location difference between multiple events being less than a preset distance threshold, or the events occurring within the same interaction area). The combined conversion is triggered only when multiple perceptual input events simultaneously satisfy the spatiotemporal constraints. This application does not limit the threshold values ​​included in the time and spatial constraints; these can be determined as needed.

[0102] For example, a user first performs a "gaze moves to the right edge of the screen" event (gaze beyond device boundary event, direction = right) within 1 second, and then maintains a gaze on the area near the right edge (gaze event, duration = 800ms). According to the combination conversion pattern, these two events are temporally continuous (time interval less than 200ms) and spatially related (gaze point located in the boundary area). These two events are combined into a single simulated native input event of "continuous scrolling to the right" (mouse horizontal scroll wheel event, MOUSEEVENTF_HWHEEL, scroll increment = -3). As another example, if a user performs two consecutive "pinch gestures" within 500ms (event combination count = 2), this is combined into a simulated touch event of "zooming out" (multi-touch zoom, zoom factor = 0.8).

[0103] It is understandable that in the example above, the events of the line of sight going beyond the right boundary and the event of looking at the right edge of the screen can also be combined and converted into page-turning events on other physical input devices, such as touch swipe sequences, and are not limited to this mouse scrolling event. The type of physical input device corresponding to the activated emulator can be determined.

[0104] Therefore, the combined conversion mode proposed in this embodiment achieves semantic aggregation of the perceptual input event sequence. Through spatiotemporal constraints, it combines and converts discrete perceptual actions (or first combines them into advanced perceptual input events, such as the aforementioned advanced eye-tracking events with complex semantics, and then directly converts them) into composite input events that meet application expectations. This avoids the fragmentation of interaction caused by converting each perceptual action (such as basic eye-tracking time and other basic perceptual input events) separately. This combined conversion mode enables perceptual input to trigger advanced interactive functions in the application (such as scrolling, zooming, and dragging), rather than being limited to simple clicks and movements, significantly improving the richness and naturalness of the interaction.

[0105] Step S33: Convert the continuously changing attribute parameters in the perceived input event into simulated input parameters that have continuously changing characteristics in the simulated native input event.

[0106] Sensing devices (especially eye trackers and head trackers) can output continuously changing attribute parameters at high frequency (such as continuous coordinate flow / continuous movement trajectory of gaze focus, continuous angular changes of head posture, and continuous positional changes of gestures). Discretizing these continuous changes into individual events can lead to interaction stutters (such as cursor jumping) or information loss. This application preserves the continuous changing characteristics of the perceived input in the simulated native input events through continuous attribute transformation.

[0107] Optionally, the continuously changing gaze focus coordinate stream can be directly converted into a continuous mouse movement event stream (MOUSEEVENTF_MOVE). Through smoothing filtering and coordinate transformation, the cursor can smoothly follow the gaze, achieving continuous "look-to-point" control. Continuous changes in head tilt angle can be converted into a continuous scroll wheel event stream (MOUSEEVENTF_WHEEL), with a faster scrolling speed for larger head tilt angles. Continuous coordinate changes in gesture trajectories can be converted into a continuous touch movement event stream (ACTION_MOVE) to achieve air swipe operation.

[0108] In practical applications, to match the processing frequency of the operating system's native input system, the system can perform sampling rate conversion and interpolation on the continuous sensory data to ensure that the output event frequency meets the expectations of the native input system (e.g., mouse events are typically processed at 60Hz-120Hz). For example, an eye tracker outputs the gaze focus coordinates at a frequency of 120Hz, continuously moving from (100, 200) to (800, 600). The system uses a mouse simulator to map each frame's coordinate changes to mouse movement events. After denoising and smoothing, it injects continuous MOUSEEVENTF_MOVE events into the operating system at a frequency of 60Hz, causing the cursor to move smoothly on the screen. The application receives continuous WM_MOUSEMOVE messages, achieving a smooth experience consistent with physical mouse movement.

[0109] Therefore, the method described in step S33 achieves a faithful conversion of the continuous characteristics of perceptual input. By converting continuously changing attribute parameters into parameters with continuously changing characteristics in the native input event (such as coordinates and scroll increments), it avoids the problem of inconsistent interaction caused by discretization. This continuous conversion mode makes fine-grained interactions such as eye-tracking cursor control and head-controlled scrolling possible, significantly improving the controllability and usability of perceptual input.

[0110] In summary, the combined conversion mode enhances the intelligence and semantic richness of interaction, enabling simple perceptual actions to express complex commands. The continuous parameter conversion mode, on the other hand, breaks through the limitations of "discrete events," allowing perceptual interaction to achieve sophisticated control requiring continuous input, such as mouse dragging, pen drawing, and smooth view rotation, greatly enriching the possibilities of interaction.

[0111] In some embodiments, in conjunction with the above description of the event conversion rules, the following will take eye-tracking events as an example to specifically illustrate the conversion relationships contained in the rules. The conversion relationships applicable to other types of perceptual input events are similar, and this application will not provide detailed examples for each. For received eye-tracking perception data, if it is determined that it meets predetermined simulation conditions, such as the system has enabled eye-tracking control mode, or the simulator used to convert eye-tracking events into a certain type of native input event is in the startup state, or even if it is determined that the confidence or recognition quality of the eye-tracking perception data is higher than a preset threshold (which is a relatively high value, such as 0.8, to ensure the reliability of eye-tracking interaction), the eye-tracking event data can be parsed through the corresponding simulator to determine the eye-tracking event type and its associated attribute parameters. Then, according to the event conversion rules, the eye-tracking event type and its associated attribute parameters are converted into simulated native input event types and simulated attribute parameters that strictly conform to the native input event data structure of the corresponding type.

[0112] Optionally, if the eye-tracking event type includes gaze events, based on transformation relationship 1: the coordinates of the gaze focus are converted into cursor position parameters (which can be achieved through a mouse simulator) or touch point coordinate parameters (which can be achieved through a touch screen simulator), thereby directly converting the coordinates of the gaze focus on the screen or in space into the mouse cursor position or touch point position, realizing the interactive experience of "gazing is pointing". This allows users to control the screen pointer in the most natural way of gazing without having to learn a new interaction paradigm. This transformation relationship is particularly suitable for assistive operation of disabled users and hands-free control in scenarios where both hands are occupied (such as surgery, industrial maintenance).

[0113] It should be noted that this conversion process includes coordinate system alignment (converting the eye-tracking coordinate system to the screen coordinate system), sensitivity adjustment (adjusting the conversion ratio according to user preferences or application scenarios), and edge processing (snagging or buffering strategies when the gaze approaches the screen edge). For multi-monitor scenarios, the target monitor can also be determined based on the direction of the gaze, enabling cross-screen cursor movement, etc.

[0114] For example, the gaze event data contains the gaze focus coordinates (0.5, 0.5) (normalized coordinates with the screen center as the origin), which is then converted to absolute coordinates (960, 540) on a 1920×1080 resolution screen. This generates simulated mouse movement event data, which can be: mi.dx=(96065535) / 1920, mi.dy=(54065535) / 1080, mi.dwFlags=MOUSEEVENTF_MOVE|MOUSEEVENTF_ABSOLUTE. The operating system then moves the cursor to the center of the screen.

[0115] In practical applications, blink events, a type of eye-tracking event, can be used to implement confirmation operations in eye-tracking interaction. Thus, following the conversion relationship 2—converting the number of event combinations into the number of clicks or key repetitions—the number of consecutive blinks can be converted into the number of mouse clicks or keyboard key repetitions, achieving the intuitive interaction of "N blinks equal N clicks." This conversion relationship can be linear: 1 blink → single click, 2 consecutive blinks → double click, 3 consecutive blinks → triple click. Alternatively, it can be a non-linear mapping: combining the blink frequency within a time window to convert it into a long press or repeated input. The number of event combinations is used as a simulated attribute parameter and filled into the mouseData field of the mouse event (in Windows, a double click can be achieved by quickly pressing DOWN / UP twice) or the repetition count field of the keyboard event, etc.

[0116] For example, if a user blinks three times consecutively within an 800ms time window (event combination count = 3), according to the event conversion rules, this is converted into a triple left mouse click event: three sets of MOUSEEVENTF_LEFTDOWN and MOUSEEVENTF_LEFTUP events are generated sequentially, each set 50ms apart. The operating system distributes these three sets of events to a file manager application, whose response is either a triple click to select a file name (entering rename mode) or opening a file properties dialog box (depending on the application's response logic). This achieves precise encoding of eye-tracking confirmation semantics. Through this conversion relationship, users can perform complex click interactions through instinctive blinking movements without the need for additional physical buttons. This conversion relationship provides important accessibility capabilities for users with motor disabilities.

[0117] In one possible implementation, since prolonged eye-closing or eye-opening can express the semantics of "holding" or "continuing," the pre-configured event conversion rules include conversion relationship 3, which converts the duration of the eye-closing or eye-opening event into a press duration parameter simulating the native input event, enabling interactions such as "gaze + eye-closing" dragging and "eye-opening hold" continuous shooting. This conversion process can include: detecting the start (transition from open-eye state to eye-closing state) and end (transition from eye-closing state to open-eye state) of the eye-closing event, calculating the duration, and mapping the duration to the time interval between pressing and releasing a key in the native input event. If the duration exceeds a long-press threshold, the corresponding long-press interaction semantics (such as popping up a context menu) can also be triggered.

[0118] For example, a user looks at a file icon and then closes their eyes for 1500ms. The system detects the start of the eye-closing event and immediately injects a `MOUSEEVENTF_LEFTDOWN` event (simulating a left mouse button press); it detects the end of the eye-closing event (transitioning to an open-eye state) and injects a `MOUSEEVENTF_LEFTUP` event (simulating a left mouse button release). Since the press duration is 1500ms, which is greater than the long-press threshold of 1000ms, the operating system recognizes it as a long press and displays the file's context menu. This achieves a temporal transition of the semantics of eye-tracking states, enabling eye-tracking interaction to support complex operations that require sustained attention, such as dragging and long presses, significantly expanding the operational dimensions of eye-tracking interaction.

[0119] In one possible implementation, the pre-configured event transformation rules may further include: a transformation relationship 4 that converts the combination of gaze-out-of-device-boundary events and gaze events into a continuous scrolling event sequence or a continuous touch movement event sequence, thereby expressing the intention to continuously scroll. Specifically, the gaze-out-of-device-boundary event can be determined by analyzing whether the gaze focus coordinates (which can be represented by (x, y)) exceed the device screen or a preset effective boundary range (collectively referred to as the device boundary range), and whether the duration of the out-of-bounds state is greater than the corresponding time threshold (which can be detected through a preset time window). This indicates that the user is intentionally performing boundary interactions (such as edge scrolling, page turning, etc.), and the out-of-bounds direction or out-of-bounds offset can be reported simultaneously, allowing the upper layer to implement rich edge control functions.

[0120] In the transformation process based on transformation relationship 4, if a gaze event is detected occurring at the edge of the screen, and subsequently, if a gaze beyond the boundary event is detected within a preset time window, a combination rule is triggered. Based on the direction and offset of the gaze, scroll wheel events or touch movement events are continuously generated at a fixed frequency (e.g., 10 times per second) until the gaze returns to the screen or the gaze event ends. This achieves the transformation from eye-tracking combined semantics to continuous interaction. By converting the combination of gaze and boundary crossing (gaze beyond the boundary) events into a continuous event sequence, false triggering of a single boundary crossing event is avoided. Simultaneously, true continuous control (rather than single-step scrolling) is achieved, allowing users to continuously browse by keeping their gaze outside the screen, resulting in a more natural and fluid interactive experience.

[0121] For example, a user gazes at the right edge of the screen (gaze coordinates = (1910, 540), lasting 300ms), and then their gaze moves off the right edge of the screen (direction = right, offset = 20px). This triggers a combination rule (conversion relationship: the combination of the gaze-out-of-device-boundary event and the gaze event is converted into a continuous scroll event sequence or a continuous touch movement event sequence), and begins continuously generating a horizontal scroll wheel event sequence: a MOUSEEVENTF_HWHEEL event is generated every 100ms, with a scroll increment of -1 (scrolling to the right), continuing until the user's gaze returns to the screen. The browser application receives continuous WM_MOUSEHWHEEL messages, and the page content continues to scroll to the right.

[0122] Based on the above analysis, the event conversion rule of this application may further include: a conversion relationship 5 that converts the offset of the eye movement beyond the boundary event into a scroll increment parameter or sliding distance parameter simulating a native input event. Accordingly, the offset of the eye movement beyond the boundary (the pixel distance or normalized ratio of the eye movement beyond the screen boundary) is converted into a mouse wheel scroll increment or touch sliding distance, achieving eye-tracking controlled scrolling. In this conversion relationship: the larger the offset, the faster the scrolling speed (non-linear mapping, such as an exponential relationship); or the longer the duration of the offset, the longer the cumulative scrolling distance. The offset direction (up / down / left / right) determines the scrolling direction.

[0123] For example, when a user reads a long document, their gaze remains continuously directed beyond the bottom edge of the screen, exceeding the offset by 50 pixels. This application can convert this into mouse wheel event data, which can be represented as: mi.mouseData = -3WHEEL_DELTA (scroll down 3 lines), mi.dwFlags = MOUSEEVENTF_WHEEL. As the offset increases to 150 pixels, the scroll increment increases accordingly to -9WHEEL_DELTA. The document view scrolls smoothly downwards, without the user needing to manually scroll the mouse wheel.

[0124] As can be seen, this application achieves continuous control of eye-tracking interaction semantics through transformation relationship 5. By converting the offset of the gaze beyond the boundary into scroll increments or sliding distances, users can control content browsing through natural eye movement. It is particularly suitable for scenarios such as reading, web browsing, and map navigation, providing an immersive reading experience without manual intervention.

[0125] In the above conversion process, it can also be combined with eye source type, that is, to encode richer interactive commands through binocular coordinated or monocular independent movements. For this, conversion relationship 6 can be pre-configured: converting eye source type into different event types or different key values ​​that simulate native input events, realizing differentiated control for the left and right eyes. Optionally, this conversion relationship 6 can be represented as: left eye blink → left mouse button click, right eye blink → right mouse button click, both eyes blink simultaneously → middle mouse button click; or left eye gaze → main cursor movement, right eye gaze → auxiliary cursor / crosshair movement, etc., thereby enabling different operations to be triggered by different eyes, and the same type of eye movement event can be converted into different native input events.

[0126] For example, when a user blinks their right eye once, the eye source type of this blink event is determined to be the right eye. Based on transformation relationship 6 included in the event transformation rule, it is converted into a right-click event: MOUSEEVENTF_RIGHTDOWN + MOUSEEVENTF_RIGHTUP. The operating system dispatches this event to the drawing application, and the application responds by popping up a context menu (perfectly consistent with the effect of a physical right-click). In this way, the instruction space for eye-tracking interaction is doubled without adding additional hardware buttons, supporting more complex interaction scenarios (such as switching between primary and secondary weapons in games, or selecting primary and secondary tools in design software).

[0127] In summary, by utilizing the six transformation relationships (but not limited to these, and dynamically adjustable as needed) included in the aforementioned fine-grained event transformation rules, the subtle and continuous physiological signals of eye movement are precisely converted into standard input events such as mouse, keyboard, and touch (simulating native input events, each corresponding to a specific type of native input event). This accurately and richly simulates the full interactive capabilities of a mouse, keyboard, and touchscreen. This not only achieves "substitution," allowing users to complete almost all traditional interactive operations with their eyes, but also creates more natural and efficient interaction methods that surpass traditional input by leveraging the unique attributes of eye movement (such as continuous coordinates, blink counts, and boundary offsets). Applications do not need to be aware of the existence of eye tracking. For example, in an unmodified shooting game, players can aim with their gaze and shoot with blinks; in office software, they can scroll documents with their eyes, greatly expanding the applicable scenarios of eye-tracking interaction.

[0128] In practical applications of this application, the aforementioned event conversion rules can be pre-configured in the simulator, manifested as lookup tables, configuration files, or state machine rules, and support dynamic updates. Specifically, the simulator parses the received eye-tracking event data, extracts the event type and attribute parameters, generates simulated native input event data with corresponding data structures based on the event conversion rules, and injects it into the operating system. Furthermore, since the interaction logic of different applications differs, this conversion relationship is not fixed but can be dynamically loaded and switched according to the context of the currently foreground application. That is, the conversion relationship can be dynamically updated based on system configuration or application context to identify and distinguish different application scenarios, enabling system services to provide differentiated and intelligent behaviors for different applications. The implementation process is not detailed in this application.

[0129] In some embodiments, before performing the above conversion steps, it is first determined whether predetermined simulation conditions are met, such as whether the system has enabled the perceptual input control mode, or whether at least one simulator (which may be a simulator corresponding to the physical input device type that needs to be simulated) is enabled. In this application, the system-level switch can be the first line of defense against accidental injection. The perceptual input control mode is a function that the user explicitly enables in the system settings or interaction scenario (such as "enable eye tracking control", "enable gesture navigation", etc.). Only when this mode is enabled will the system allow the perceptual input event data to be converted into simulated native input event data and injected into the operating system.

[0130] Furthermore, the system can individually control the activation status of each simulator. For example, a user might only enable the eye-tracking simulator (for cursor control) while disabling the gesture simulator (to prevent accidental gesture triggers). Sensory input event data for conversion to the corresponding simulator is only processed when the corresponding simulator is enabled. Optionally, this application can enable at least one target simulator from multiple simulators based on system configuration or application context. This allows the system to determine "which app the user is currently operating in" and "what state this app is currently in," thereby automatically selecting and enabling the most suitable simulator as the target simulator to perform the aforementioned conversion operation through the target simulator.

[0131] For example, the user has enabled the "Eye-tracking Mouse" function (sensory input control mode enabled) in the system settings and has activated the mouse emulator and keyboard emulator. When eye-tracking event data is received, it is checked that the eye-tracking control mode is enabled and the mouse emulator is enabled, determining that the predetermined simulation conditions are met, and the subsequent conversion process continues. If the user has disabled the "Eye-tracking Mouse" function, even if eye-tracking event data is received, it is directly discarded or only logged, without performing conversion and injection steps, to avoid the cursor being accidentally controlled by eye movements during normal use.

[0132] Therefore, this application realizes user authorization and scenario adaptation for input event simulation. Through system-level switch or simulator-level switch control, or dual control of system-level switch and simulator-level switch, it ensures that the perceived input only takes effect in the scenario expected by the user, effectively preventing misoperation and privacy infringement, and improving the security and user controllability of the system.

[0133] In some embodiments, considering the uncertainty of the recognition results of the sensing input device (such as the decrease in accuracy of eye tracking under strong light and the decrease in confidence of gesture recognition when occluded), the pre-configured predetermined simulation conditions of this application may also include: the confidence or recognition quality of the sensing input event data is higher than a preset threshold, thereby filtering out low-quality sensing input events, avoiding the conversion of noise or misrecognition into native input events that could lead to erroneous operations, and significantly reducing the erroneous operation rate caused by sensing recognition errors. This quality perception conversion mechanism ensures that only high-confidence sensing inputs can affect the system state, improving the accuracy and reliability of the interaction, which is especially important in complex environments (such as outdoor AR and in-vehicle scenarios).

[0134] The confidence level can be directly provided by the sensing device or sensing event service (such as the confidence field in eye-tracking event data), or it can be dynamically evaluated by the system based on historical data (such as calculating recent recognition accuracy). The recognition quality can be quantified by indicators such as signal strength, feature clarity, and multi-sensor consistency, and can be determined by an AI model through analysis of the original sensing data used to generate the sensing input event data. This application does not impose any restrictions on this.

[0135] For example, the eye-tracking event service reports gaze event data containing a confidence level of 0.65 (below the threshold of 0.8). This indicates insufficient recognition quality of the event data, possibly due to gaze estimation bias caused by a momentary blink or rapid head movement. Therefore, this event data is discarded, and the conversion and injection steps are not triggered. Subsequently reported another gaze event with a confidence level of 0.95, above the threshold, meets the predetermined simulation conditions. This event is then converted and injected into the operating system as mouse movement event data.

[0136] Therefore, this application achieves intelligent process control by introducing "pre-defined simulation conditions," avoiding meaningless conversion and injection operations when the user unintentionally uses the system or the simulation conditions are not met, thus saving computing and communication resources. More importantly, by using data quality thresholds, it ensures the high reliability of the events ultimately injected into the operating system, thereby improving the robustness of the entire simulation interaction system from the source.

[0137] In the input event simulation methods described in the above embodiments, for the simulated native input event data converted by the simulator, an appropriate injection method can be selected to inject it into the operating system according to the operating system type and permission requirements. Injection method one: calling the native input event injection interface provided by the operating system's native input system to perform the injection operation. Injection method two: performing the injection operation through a simulated input device, but not limited to these two methods.

[0138] In injection method one, simulated native input event data can be submitted to the native input system's input event queue through the native input event injection interface provided by the operating system's native input system (such as Windows' SendInput API, Linux's uinput interface, and Android's InputManager.injectInputEvent). This injection interface directly places the simulated native input event data into the same input event queue as the native input event data generated by the physical input device, which is then uniformly distributed by the native input system. This input event queue can include both native input event data and simulated native input event data of the same type. It should be noted that the storage method for input event data is not limited to this event queue method; other storage methods do not affect the implementation of the method in this application.

[0139] This injection method is simple to implement, has low latency, good compatibility with operating system versions, and simulates native input event data with the same priority and processing path as native input event data. For example, on the Windows platform, the system calls `SendInput(2, inputs, sizeof(INPUT))`, where the `inputs` array contains mouse movement event data and left mouse button press event data. The system's original input thread receives these two events, converts them into `WM_MOUSEMOVE` and `WM_LBUTTONDOWN` window messages, and dispatches them to the currently focused application window according to the focus window strategy or event subscription relationship to execute the corresponding business logic.

[0140] As can be seen, the injection operation proposed in this application achieves seamless integration between simulated native input events and the operating system input stack. By calling a standardized injection interface, simulated native input events directly enter the system-level input event queue, ensuring the fairness and consistency of input event distribution. This approach eliminates the need to develop custom drivers or virtual hardware, reducing implementation complexity and system risk.

[0141] Optionally, if injection method two is chosen, the injection operation can be performed using a simulated input device. Simulated native input event data can be submitted to the operating system's native input system using the same communication protocol as data reported by physical input devices. This allows for scenarios requiring the simulation of specific hardware characteristics, support for hot-plug detection, or bypassing certain security restrictions. Virtual input devices (simulated input devices, such as virtual mice, virtual keyboards, virtual touchpads, etc., which can correspond to emulators) can be registered and data reported using standard protocols of the operating system's input driver framework (such as HID and I2C protocols). The simulated input device appears as an input node in the operating system indistinguishable from a physical input device. Applications can query the device and receive its events using standard input APIs (such as GetRawInputDeviceList and evdev enumeration).

[0142] For example, the system creates a virtual mouse device / dev / input / event10 via uinput on a Linux platform. Following the method described in this application, simulated mouse movement and click event data are encapsulated into an input_event structure (type=EV_REL, code=REL_X, value=10, etc.) and written to the uinput device file. The native input system then merges these events with physical mouse events and distributes them to applications subscribed to the device.

[0143] As can be seen, this application achieves customization of simulated native input events at the hardware abstraction layer through injection method two. By creating simulated input devices and reporting data using standard hardware protocols (communication protocols), the operating system and application layer fully recognize the perceived input as standard hardware input. This approach is particularly suitable for scenarios requiring device-level authentication, supporting multi-device concurrency, or requiring precise hardware emulation (such as automated testing and game anti-cheat compatibility), providing the highest level of compatibility assurance, lower latency, and higher throughput.

[0144] In summary, the two injection methods proposed in this application complement each other, enabling the application to adapt to different operating system versions and security policies, ensuring reliable injection of simulated events into the system under various environments. Furthermore, since the processing path after injection is the same as the native input event generated by the physical input device, the application's response behavior is completely identical to when receiving real physical input, requiring no additional adaptation and truly achieving a "zero-cost" natural interaction upgrade.

[0145] It should be noted that the operating system's native input system is used to receive native input event data generated by physical input devices and distribute it to the application layer. In order to ensure that the native input system can receive simulated native input event data, its interfaces (native input event injection interface or interface for receiving data reported by physical input devices) can be adaptively modified. For example, logic code for receiving simulated native input event data can be added, or the user-space interface can be modified to create virtual device nodes through the uinput driver to form a virtual channel, so as to establish a virtual event channel in the operating system input stack, so that the virtual event stream and the native input event stream can be seamlessly integrated at the system level. This application does not restrict the modification method.

[0146] Reference Figure 4 This is a schematic diagram of the structure of an input event simulation device proposed in an embodiment of this application, as shown below. Figure 4 As shown, the input event simulation device may include:

[0147] Receiver module 41 is used to receive sensor input event data;

[0148] The conversion module 42 is used to convert the perceived input event data into simulated native input event data based on pre-configured event conversion rules if the perceived input event data meets the predetermined simulation conditions; wherein the simulated native input event data is consistent with the native input event data generated by the physical input device in terms of data structure.

[0149] Injection module 43 is used to inject simulated native input event data into the operating system, which then distributes it to the native input interface of the target application in the application layer.

[0150] In one possible implementation, the perceived input event data may include at least one of the following: eye-tracking event data, gesture event data, voice command data, and head posture data; the native input event data may include at least one of the following: mouse event data, keyboard event data, and touch event data.

[0151] In one possible implementation, the conversion process of conversion module 42 is implemented by at least one simulator, each simulator corresponding to a physical input device type, and conversion module 42 may include:

[0152] The parsing unit is used to parse the sensing input event data, determine the type of sensing input event and at least one associated attribute parameter, which may include at least one of spatial attribute parameters, temporal attribute parameters, state attribute parameters and source attribute parameters.

[0153] The conversion unit is used to convert the perceived input event type into a simulated native input event type based on pre-configured event conversion rules, and to convert at least one attribute parameter into a simulated attribute parameter associated with the simulated native input event type; the simulated native input event type corresponds to the native input event type of the physical input device;

[0154] The encapsulation unit is used to encapsulate the simulated native input event type and simulated attribute parameters into structured simulated native input event data according to the native input event data structure.

[0155] In one possible implementation, the above event transformation rules support at least one of the following transformation modes:

[0156] The combination conversion unit is used to combine multiple perceptual input events that occur within a preset spatiotemporal constraint into a single analog native input event;

[0157] The continuous conversion unit is used to convert continuously changing attribute parameters in a perceived input event into analog input parameters with continuously changing characteristics in a simulated native input event.

[0158] In one possible implementation, if the perceived input event data is eye-tracking event data, the perceived input event type can include one or more combinations of the following eye-tracking event types: eye-opening event, eye-closing event, gaze event, blinking event, continuous blinking event, and gaze beyond device boundary event. The attribute parameters associated with the eye-tracking event type include at least one of the following: occurrence timestamp, gaze focus coordinates, number of event combinations, duration, offset or direction beyond the boundary, user identifier, and eye source type; the eye source type includes left eye, right eye, or both eyes. Correspondingly, the converted simulated attribute parameters can include at least one of the following associated with mouse events, keyboard events, or touch events: coordinate parameters, button parameters, status parameters, and device identifier parameters.

[0159] In one possible implementation, the transformation unit includes at least one of the following sub-units, each sub-unit corresponding to a transformation relationship in the event transformation rule:

[0160] The coordinate transformation subunit is used to convert the coordinates of the gaze focus into cursor position parameters or touch point coordinate parameters;

[0161] The count conversion subunit is used to convert the event combination count into a click action count parameter or a key repetition count parameter;

[0162] The duration conversion subunit is used to convert the duration of a closed-eye event or an open-eye event into a press duration parameter that simulates the native input event.

[0163] The offset conversion subunit is used to convert the offset of a line-of-sight event beyond the device boundary into a scroll increment parameter or a sliding distance parameter that simulates a native input event.

[0164] The sequence conversion subunit is used to convert a combination of gaze-over-device-boundary events and gaze events into a continuous scrolling event sequence or a continuous touch movement event sequence.

[0165] The eye source conversion subunit is used to convert the eye source type into a distinguishing identifier for different event types or different key values ​​that simulate native input events.

[0166] In one possible implementation, the predetermined simulation conditions described above may include at least one of the following:

[0167] The system has enabled the input sensing control mode, or at least one simulator is enabled.

[0168] The confidence level or recognition quality of the perceived input event data is higher than a preset threshold.

[0169] In one possible implementation, injection module 43 may include at least one of the following units:

[0170] The interface injection unit is used to call the native input event injection interface provided by the operating system's native input system to submit simulated native input event data to the input event queue of the native input system;

[0171] The device injection unit is used to submit simulated native input event data to the operating system's native input system via a simulated input device using the same communication protocol as the data reported by the physical input device.

[0172] The native input system is an input system used to receive native input event data generated by physical input devices and distribute it to the application layer.

[0173] This application also provides a computer-readable storage medium that carries one or more computer programs. When the one or more computer programs are executed by an electronic device, the electronic device can implement any of the input event simulation methods provided in this application.

[0174] The computer-readable storage medium can be any available medium that an electronic device can store, or a data storage device such as a training device or data center that integrates one or more available media. Available media can be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., DVDs), or semiconductor media (e.g., solid-state drives (SSDs)).

[0175] This application also provides a computer program product including computer-readable instructions, which, when executed on an electronic device, cause the electronic device to implement any of the input event simulation methods provided in this application.

[0176] When computer-readable instructions are loaded and executed on an electronic device, all or part of the processes or functions according to the embodiments of this application are generated. The electronic device may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer-readable instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, computer-readable instructions may be transmitted from one website, computer, training device, or data center to another website, computer, training device, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means, which can be determined according to the actual application scenario.

[0177] Reference Figure 5 This is a schematic diagram of the hardware structure of an electronic device proposed in this application, as shown below. Figure 5 As shown, the electronic device may include, but is not limited to, at least one memory 51 and at least one processor 52, wherein: the memory 51 is used to store pre-configured event conversion rules, a computer program implementing the method of this application, and may also store necessary data and programs for the normal operation of the electronic device, such as operating system kernels, sensing event services, simulators, device drivers, and applications. The event conversion rules are pre-configured with conversion relationships from sensing input event types to native input event types, attribute parameter conversion relationships, spatiotemporal constraint thresholds, etc.

[0178] The processor 52 is configured to execute a computer program in the memory 51 to implement the input event simulation method proposed in any embodiment of this application. Therefore, the processor 52 runs the operating system's perceptual event service, receives perceptual input event data, determines that it meets predetermined simulation conditions, converts the perceptual input event data into simulated native input event data based on pre-configured event conversion rules, and injects the simulated native input event data into the operating system, which then distributes it to the native input interface of the target application.

[0179] Furthermore, the electronic device may also include at least one sensing device (i.e., the aforementioned sensing device) 53, which is used to collect corresponding sensing data. The sensing device 53 includes, but is not limited to: eye-tracking sensors (such as infrared cameras or near-eye cameras, used to collect eye images and gaze direction), gesture sensors (such as depth cameras, ToF (Time-of-Flight) sensors, or millimeter-wave radar, used to collect hand postures and movements), voice acquisition devices (such as microphone arrays, used to collect voice signals), and head posture sensors (such as IMU inertial measurement units or head-tracking cameras, used to collect head angles and positions), etc.

[0180] Based on this, the processor 52 is also configured to: generate perceptual input event data based on the perceptual data, and send the perceptual input event data to a perceptual event service. If the perceptual data is eye-tracking perceptual data collected by an eye-tracking sensor, the perceptual event service can be an eye-tracking event service; if it is other types of perceptual data, the perceptual event service can be other types, which will not be described in detail here.

[0181] Optionally, the electronic device may also include audio output devices (such as speakers, headphones), display devices (such as displays, optical waveguides), physical input devices (such as physical mice, keyboards, touchscreens, used to form complementary or backup input channels with sensing inputs), and communication modules (such as Wi-Fi, Bluetooth, 5G, used to communicate with external sensing devices or cloud services). These devices can transmit data, address, and control signals through the high-speed data channel provided by the system bus.

[0182] It should be understood that, Figure 5 The structure of the electronic device shown does not constitute a limitation on the electronic device in the embodiments of this application. In practical applications, the electronic device may include more than Figure 5 The application does not provide detailed examples of the more or fewer components shown, or combinations of certain components, such as gyroscopes, accelerometers, and gravity sensors, other sensing units used to obtain sensing parameters, power management modules, antennas, or other communication elements.

[0183] Based on the above analysis, such as Figure 6The diagram shows the software architecture of the electronic device proposed in this application. This software architecture, from bottom to top, can include: Hardware layer: including the aforementioned sensing device 53, physical input devices, and a dedicated processor for generating sensing input event data, etc. Driver layer / kernel layer: including sensing device drivers (such as eye-tracking drivers, gesture recognition drivers) and physical input device drivers (such as HID mouse drivers, keyboard drivers, touch drivers). The sensing device drivers report raw sensing data / sensing input event data to the Hardware Abstraction Layer (HAL). Hardware Abstraction Layer (HAL): including sensing event providers, which upload sensing input event data to the system framework layer. System framework layer: including sensing event services such as eye-tracking event services, gesture recognition services, and speech recognition services. The input event simulation module (the core component implementing the method of this application) contains simulators corresponding to different types of physical input devices, such as... Figure 6 Simulator0, Simulator1, and Simulator2 in the code can correspond to mouse simulator, keyboard simulator, and touch simulator, respectively. After generating simulated native input event data, the input event simulation module sends it to the native input system (Input flinger) via an injection interface or simulated device. The native input system is responsible for distributing both the native input event data and the simulated native input event data to the application layer. The application layer includes various target applications, such as drawing software, browsers, games, and office software. These applications receive events from the native input system through native input interfaces (such as onTouchEvent, WndProc, and evdev), without needing to distinguish whether the event source is a physical device or a sensor simulation.

[0184] Therefore, this application achieves deep integration of perceptual input and native input at the operating system level through a hierarchical software architecture of "hardware → driver → perceptual service → input event simulation → native input system → application". By embedding the input event simulation module into the operating system's input stack, perceptual input can reuse mature native input distribution mechanisms, ensuring full compatibility with existing applications while maintaining architectural clarity and scalability.

[0185] Finally, it should be noted that the device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. In addition, in the accompanying drawings of the device embodiments provided in this application, the connection relationship between modules indicates that they have a communication connection, which can be implemented as one or more communication buses or signal lines.

[0186] In the above embodiments, the invention can be implemented entirely or partially by software, hardware, firmware, or any combination thereof. Through the description of the above embodiments, those skilled in the art can clearly understand that this application can be implemented using software plus necessary general-purpose hardware, or it can be implemented using dedicated hardware including dedicated integrated circuits, dedicated CPUs, dedicated memory, dedicated components, etc. When implemented using software, it can be implemented entirely or partially in the form of a computer program product. The various embodiments in this specification are described in a progressive or parallel manner, with each embodiment focusing on the differences from other embodiments. Similar or identical parts between embodiments can be referred to mutually. For the apparatuses and electronic devices disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the descriptions are relatively simple, and relevant parts can be referred to the method section.

Claims

1. An input event simulation method, comprising: Receive sensor input event data; If the perceived input event data meets the predetermined simulation conditions, the perceived input event data is converted into simulated native input event data based on the pre-configured event conversion rules; wherein the simulated native input event data is consistent with the native input event data generated by the physical input device in terms of data structure; The simulated native input event data is injected into the operating system, which then distributes it to the native input interface of the target application in the application layer.

2. The method according to claim 1, wherein the perceived input event data includes at least one of the following: Eye-tracking event data, gesture event data, voice command data, head posture data; The native input event data includes at least one of the following: Mouse event data, keyboard event data, touch event data.

3. The method according to claim 2, wherein, The perceived input event data is converted into simulated native input event data using at least one simulator, each simulator corresponding to a type of physical input device. The conversion process includes: The sensory input event data is parsed to determine the type of the sensory input event and at least one associated attribute parameter, wherein the attribute parameter includes at least one of spatial attribute parameters, temporal attribute parameters, state attribute parameters and source attribute parameters; Based on pre-configured event conversion rules, the perceived input event type is converted into a simulated native input event type, and the at least one attribute parameter is converted into a simulated attribute parameter associated with the simulated native input event type; the simulated native input event type corresponds to the native input event type of the physical input device; Based on the native input event data structure, the simulated native input event type and the simulated attribute parameters are encapsulated into structured simulated native input event data.

4. The method according to claim 3, wherein the event transformation rule supports at least one of the following transformation modes: Multiple perceptual input events occurring within preset spatiotemporal constraints will be combined and converted into a single simulated native input event; The continuously changing attribute parameters in the perceived input event are converted into simulated input parameters with continuously changing characteristics in the simulated native input event.

5. The method according to claim 3, wherein if the perceptual input event data is eye-tracking event data, the perceptual input event type includes eye-tracking event types defined by one or more combinations of the following: Eye-opening event, eye-closing event, gaze event, blinking event, continuous blinking event, line-of-sight exceeding device boundary event; The attribute parameters associated with the eye-tracking event type include at least one of the following: The event timestamp, gaze focus coordinates, number of event combinations, duration, offset or direction exceeding the boundary, user identifier, and eye source type; the eye source type includes left eye, right eye, or both eyes; The simulated attribute parameters include at least one of the following associated with mouse events, keyboard events, or touch events: Coordinate parameters, button parameters, status parameters, and device identification parameters.

6. The method according to claim 5, wherein the event transformation rule includes at least one of the following transformation relationships: Convert the gaze focus coordinates into cursor position parameters or touch point coordinate parameters; Convert the number of event combinations into a click action count parameter or a key repetition count parameter; Convert the duration of the closed-eye event or the open-eye event into a press duration parameter that simulates the native input event; Convert the offset of the line-of-sight-beyond-device boundary event into a scroll increment parameter or a sliding distance parameter that simulates the native input event; Convert the combination of gaze-out-of-device-boundary events and gaze events into a continuous scrolling event sequence or a continuous touch-movement event sequence; Convert the eye source type into a distinguishing identifier for different event types or different key values ​​that simulate native input events.

7. The method according to any one of claims 1-6, wherein the predetermined simulation conditions include at least one of the following: The system has enabled the input sensing control mode, or at least one simulator is enabled. The confidence level or recognition quality of the perceived input event data is higher than a preset threshold.

8. The method according to any one of claims 1-6, wherein the simulated native input event data is injected into the operating system, comprising any of the following implementations: The simulated native input event data is submitted to the input event queue of the native input system by calling the native input event injection interface provided by the operating system's native input system. The simulated native input event data is submitted to the operating system's native input system using the same communication protocol as the data reported by the physical input device via the simulated input device. in, The native input system is an input system used to receive native input event data generated by the physical input device and distribute it to the application layer.

9. An electronic device, comprising: At least one memory stores pre-configured event transition rules; At least one processor is configured to run the operating system's awareness event service and perform the following operations: Receive sensor input event data; If the perceived input event data meets the predetermined simulation conditions, the perceived input event data is converted into simulated native input event data based on the pre-configured event conversion rules; wherein the simulated native input event data is consistent with the native input event data generated by the physical input device in terms of data structure; The simulated native input event data is injected into the operating system, which then distributes it to the native input interface of the target application.

10. The electronic device according to claim 9, further comprising: At least one sensing device is configured to collect corresponding sensing data; The at least one processor is further configured to: generate the sensing input event data based on the sensing data, and send the sensing input event data to the sensing event service; The sensing device includes an eye-tracking sensor, and the sensing event service includes an eye-tracking event service.

Images