Method for realizing mouse and virtual key control based on inertial measurement unit

Abstract

The invention discloses a method and equipment for realizing mouse and virtual key control based on an inertial measurement unit, and a computer readable storage medium. The method comprises the following steps: acquiring original three-dimensional motion data of the inertial measurement unit to generate a standard motion feature flow; forming screen key layout information according to the screenshot of the current active application program; determining an inferred running state of the current active application program to obtain an effective virtual key set containing function annotations; determining a current effective interaction mode according to the updated human-computer interaction situation information, the updated current application program inference operation state and a user input instruction; and selecting a mapping algorithm to process the standardized motion feature flow according to the currently effective interaction mode to generate at least one of a mouse cursor movement instruction, a mouse key instruction and a virtual key operation instruction. The method has the advantage that the operation efficiency and the user experience of IMU-based mouse control in multiple scenes are improved.

Description

Technical Field

[0001] The present invention relates to the field of human-computer interaction technology, and in particular to a method, device, and computer-readable storage medium for realizing mouse and virtual key control based on an inertial measurement unit. Background Art

[0002] As a sensor technology capable of capturing the three-dimensional motion and posture of an object, the inertial measurement unit (IMU) demonstrates significant potential in the field of human-computer interaction, particularly in replacing traditional mice for cursor control and enabling new forms of interaction. Existing technologies typically utilize IMUs that integrate accelerometers, gyroscopes, and sometimes magnetometers to acquire raw motion data. These data are then mapped to cursor movements or specific commands on a two-dimensional screen using preprocessing algorithms (such as filtering and sensor fusion).

[0003] However, current IMU-based human-computer interaction technology still has many limitations.

[0004] First, when it comes to cursor control, existing technologies generally use a single sensitivity mapping mode or fixed control logic to handle all operational scenarios. This "one-size-fits-all" design makes it difficult to meet diverse interaction needs. Existing solutions often lack the flexibility to dynamically adjust mapping strategies based on task requirements or user preferences. This results in slow cursor response and inefficient operation in some scenarios, or difficulty in precise positioning in others, resulting in limited control accuracy and a poor user experience.

[0005] Secondly, most existing solutions separate cursor movement control from triggering other core mouse functions (such as button clicks and wheel scrolling), or lack robustness and accuracy in gesture recognition, making it impossible to achieve natural and efficient coordinated control of multimodal operations. After using the IMU to position the cursor, users often find it difficult to smoothly and accurately perform accompanying actions such as clicking, double-clicking, dragging, or scrolling.

[0006] In addition, the limitations of traditional IMU mouse simulation solutions are more prominent when interacting with dynamically changing virtual buttons on the screen (such as the direction keys and skill keys commonly seen in games). Existing solutions usually map recognized IMU gestures to predefined functions or fixed key commands. This mapping relationship is usually static and lacks the ability to perceive real-time changes in the target application interface. When the layout, availability or functionality of the virtual buttons in the application interface changes dynamically with the context, fixed gesture mapping often causes the operation to fail or produce unexpected results. Users need an interaction method that can dynamically perceive the available virtual buttons on the screen, understand their functions in the current state, and intelligently and accurately map user gestures or IMU-controlled mouse operations to the target virtual buttons.

[0007] Therefore, this field urgently needs an IMU human-computer interaction control solution that can provide multi-mode adaptive cursor and scroll wheel control, seamlessly integrate accurate and reliable gesture recognition to achieve multi-modal collaborative operation, and can dynamically perceive and intelligently interact with screen virtual buttons. Summary of the Invention

[0008] The embodiments of the present application provide a method for realizing mouse and virtual button control based on an inertial measurement unit, aiming to provide an IMU human-computer interaction control solution that can provide multi-mode adaptive cursor and scroll wheel control, can seamlessly integrate accurate and reliable gesture recognition to realize multi-modal collaborative operation, and can dynamically perceive and intelligently interact with screen virtual buttons.

[0009] To achieve the above objectives, an embodiment of the present application provides a method for controlling a mouse and virtual buttons based on an inertial measurement unit, comprising:

[0010] Obtain the raw 3D motion data of the inertial measurement unit and perform multi-level preprocessing to generate a standard motion feature stream containing 3D attitude parameters and 3D angular velocity parameters;

[0011] Obtaining a screenshot of the currently active application, and inputting the screenshot into a pre-trained object detection deep learning model to detect virtual buttons and form screen button layout information;

[0012] Determine the inferred running state of the currently active application based on the human-computer interaction context information of the previous cycle, the screen button layout information, and the predefined application state model, and obtain a valid virtual key set containing functional annotations based on the inferred running state and preset virtual key function annotation rules, and simultaneously update the human-computer interaction context information and the inferred running state of the currently active application;

[0013] Determining a currently effective interaction mode based on the updated human-computer interaction context information, the updated inferred running state of the current application, and the user input instruction, wherein the interaction mode includes a mouse-first interaction mode and a virtual key-first interaction mode;

[0014] According to the currently effective interaction mode, selecting a mapping algorithm corresponding to the currently effective interaction mode to process the standardized motion feature stream to generate at least one of a mouse cursor movement instruction, a mouse button instruction, and a virtual button operation instruction;

[0015] Performing command fusion and collaborative event generation on the confirmed mouse cursor movement instructions, the mouse button instructions, the virtual button operation instructions, the currently effective interaction mode, the updated human-computer interaction context information, the updated inferred running state of the currently active application, the screen button layout information, and the valid virtual button set to obtain a unified collaborative interaction event sequence;

[0016] Event conversion is performed on the unified collaborative interaction event sequence to generate a standard input event stream of the target operating system and inject the standard input event stream into the event processing queue of the target operating system.

[0017] To achieve the above-mentioned objectives, an embodiment of the present application also proposes a device for implementing mouse and virtual key control based on an inertial measurement unit, comprising a memory, a processor, and a program for implementing mouse and virtual key control based on an inertial measurement unit stored in the memory and runnable on the processor. When the processor executes the program for implementing mouse and virtual key control based on an inertial measurement unit, the method for implementing mouse and virtual key control based on an inertial measurement unit as described in any one of the above items is implemented.

[0018] To achieve the above-mentioned purpose, an embodiment of the present application also proposes a computer-readable storage medium, on which is stored a program for implementing mouse and virtual key control based on an inertial measurement unit. When the program for implementing mouse and virtual key control based on an inertial measurement unit is executed by a processor, the method for implementing mouse and virtual key control based on an inertial measurement unit as described in any of the above items is implemented.

[0019] The technical solution of this application introduces a method to determine the currently effective interaction mode (at least including the mouse priority interaction mode and the virtual key priority interaction mode) based on the current human-computer interaction context information, the current application inferred running state, and the optional user IMU switching gesture command, which fundamentally solves the problem of single IMU input interpretation and fragmented user experience caused by the lack of dynamic perception of the user's specific operation intentions and application scenarios in the existing technology. This means that the system is no longer fixedly mapping IMU data only to mouse behavior or only to limited fixed gestures, but can actively adjust the core parsing direction of subsequent IMU data according to the most appropriate interaction paradigm at the moment (whether it focuses on the fine pointing of the cursor or the quick operation of the virtual key by gesture). This design gives users different operation requirements (for example, preferring mouse mode when performing file operations on the desktop, and preferring virtual key gesture mode when quickly releasing skills in the game) to obtain a system response that better matches their intentions, greatly improving the intuitiveness and efficiency of the interaction.

[0020] Secondly, the technical solution of this application selects a specific mapping algorithm corresponding to the currently active interaction mode to process the same standardized motion feature stream, thereby generating mouse cursor movement instructions and mouse button instructions (in mouse-priority interaction mode) or virtual key operation instructions (in virtual key-priority interaction mode). This technical feature ensures that even if the user's IMU input may be similar in physical form, the system can correctly interpret it as the most important operation instruction in that mode based on the currently active interaction mode state. For example, in mouse-priority mode, specific IMU movements will be interpreted as smooth cursor movement and simulated mouse click actions; while in virtual key-priority mode, similar IMU movements may be specifically interpreted by the gesture recognition module as direct operation instructions associated with a virtual key function on the screen. This differentiated data parsing and instruction generation based on the current interaction mode effectively avoids confusion and interference between different types of operation intentions (pointing, clicking, function triggering), and achieves clear separation and on-demand activation of the two main interaction modes: mouse operation and virtual key operation.

[0021] Furthermore, the technical solution of the present application forms a valid virtual key set containing functional annotations through dynamic perception and in-depth understanding of the screen content of the currently active application, and performs comprehensive command fusion and collaborative event generation on this information with mouse cursor movement instructions, mouse button instructions, virtual key operation instructions, and the currently effective interaction mode and context information. This series of designs realizes the intelligent association between IMU input and dynamic elements on the screen. The system not only knows what virtual buttons are on the screen and their locations, but also understands their specific functions in the current application state. Therefore, when performing command fusion, whether it is operating the virtual buttons through the cursor and click gestures in mouse priority mode, or directly triggering the virtual button function through specific gestures in virtual button priority mode, the accuracy and effectiveness of the operation can be ensured. For example, a "confirm" gesture will act on the confirmation button pointed to by the cursor in mouse mode, and on the virtual button In this mode, the "Confirm" function button on the current interface may be directly matched. This fusion mechanism overcomes the problems of disconnection between gestures and interfaces and rigid mapping relationships mentioned in the background art, significantly improving the reliability and intelligence of operations in complex dynamic interfaces (such as games and professional software).

[0022] Furthermore, by performing multi-level preprocessing (including filtering, sensor fusion, normalization, and correction) on the raw 3D motion data from the inertial measurement unit to generate a high-quality standard motion feature stream, this provides a stable and reliable data foundation for all subsequent interaction mode judgments, gesture recognition, and mapping algorithms, ensuring the overall control accuracy and response stability from the source. Furthermore, the unified collaborative interaction event sequence is converted into a standard input event stream for the target operating system and injected into the event processing queue, ensuring the broad compatibility of this method and its immediate availability across various operating systems and application environments. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the structures shown in these drawings without paying any creative work.

[0024] Figure 1 This is a module structure diagram of an embodiment of a device for implementing mouse and virtual key control based on an inertial measurement unit according to the present invention;

[0025] Figure 2 This is a flow chart of an embodiment of a method for implementing mouse and virtual button control based on an inertial measurement unit according to the present invention.

[0026] The purpose, features and advantages of the present invention will be further described with reference to the accompanying drawings and in conjunction with the embodiments. DETAILED DESCRIPTION

[0027] It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0028] To better understand the above technical solutions, exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although exemplary embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be limited by the embodiments described herein. Instead, these embodiments are provided to enable a more thorough understanding of the present disclosure and to fully convey the scope of the present disclosure to those skilled in the art.

[0029] It should be noted that in the claims, any reference signs placed between brackets shall not be construed as limiting the claims. The presence of "comprising" in the text does not exclude the presence of components or steps not listed in the claims. The quantifier "a" or "an" preceding a component does not exclude the presence of a plurality of such components. The present invention can be implemented by means of hardware comprising several different components and by means of a suitably programmed computer. In a unit claim that lists several means, several of these means may be embodied by the same hardware item. The use of "first", "second", and "third" etc. does not indicate any order and these words may be interpreted as names.

[0030] like Figure 1 As shown, Figure 1 It is a structural diagram of a server 1 (also called a device for implementing mouse and virtual button control based on an inertial measurement unit) in a hardware operating environment involved in an embodiment of the present invention.

[0031] like Figure 1 As shown, the server 1 includes: a memory 11 , a processor 12 and a network interface 13 .

[0032] The memory 11 includes at least one type of readable storage medium, including a flash memory, a hard disk, a multimedia card, a card-type memory (e.g., an SD or DX memory), a magnetic memory, a magnetic disk, an optical disk, etc. In some embodiments, the memory 11 may be an internal storage unit of the server 1, such as a hard disk of the server 1. In other embodiments, the memory 11 may also be an external storage device of the server 1, such as a plug-in hard disk equipped on the server 1, a smart media card (SMC), a secure digital (SD) card, a flash card, etc.

[0033] Furthermore, the memory 11 may include both an internal storage unit of the server 1 and an external storage device. The memory 11 may be used not only to store application software installed on the server 1 and various data, such as the code of the program 10 for implementing mouse and virtual key control based on the inertial measurement unit, but also to temporarily store data that has been output or is about to be output.

[0034] In some embodiments, the processor 12 can be a central processing unit (CPU), a controller, a microcontroller, a microprocessor, or other data processing chip, used to run the program code or process data stored in the memory 11, such as executing the program 10 based on the inertial measurement unit to implement mouse and virtual button control.

[0035] The network interface 13 may optionally include a standard wired interface or a wireless interface (such as a WI-FI interface), and is generally used to establish a communication connection between the server 1 and other electronic devices.

[0036] The network may be the Internet, a cloud network, a wireless fidelity (Wi-Fi) network, a personal area network (PAN), a local area network (LAN), and / or a metropolitan area network (MAN). Various devices in the network environment may be configured to connect to the communication network according to various wired and wireless communication protocols. Examples of such wired and wireless communication protocols may include, but are not limited to, at least one of the following: Transmission Control Protocol and Internet Protocol (TCP / IP), User Datagram Protocol (UDP), Hypertext Transfer Protocol (HTTP), File Transfer Protocol (FTP), ZigBee, EDGE, IEEE 802.11, Light Fidelity (Li-Fi), 802.16, IEEE 802.11s, IEEE 802.11g, multi-hop communication, wireless access point (AP), device-to-device communication, cellular communication protocol, and / or Bluetooth communication protocol, or a combination thereof.

[0037] Optionally, the server may further include a user interface, which may include a display and an input unit such as a keyboard. The optional user interface may also include a standard wired interface or a wireless interface. Optionally, in some embodiments, the display may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, or an OLED (Organic Light-Emitting Diode) touchscreen. The display, which may also be referred to as a display screen or display unit, is used to display information processed in the server 1 and to display a visual user interface.

[0038] Figure 1Only the server 1 having components 11-13 and the program 10 for implementing mouse and virtual key control based on an inertial measurement unit is shown. It can be understood by those skilled in the art that Figure 1 The illustrated structure does not limit the server 1 and may include fewer or more components than shown, or may combine certain components or arrange the components differently. In this embodiment, the processor 12 may be used to call a program stored in the memory 11 that implements mouse and virtual button control based on an inertial measurement unit.

[0039] Reference Figure 2 , Figure 2 This is an embodiment of a method for implementing mouse and virtual key control based on an inertial measurement unit of the present invention. The method for implementing mouse and virtual key control based on an inertial measurement unit includes the following steps:

[0040] S10. Obtain the original three-dimensional motion data of the inertial measurement unit and perform multi-level preprocessing to generate a standard motion feature stream containing three-dimensional posture parameters and three-dimensional angular velocity parameters. The purpose of this step is to collect the original motion signal from the inertial measurement unit (IMU) and eliminate the interference factors such as noise and drift that may exist therein through a series of systematic processing methods, and finally extract a standardized data stream that can accurately reflect the posture information of the device in three-dimensional space (such as the direction and rotation angle of the device) and the angular velocity information of the motion (that is, the speed of the device rotation). This standardized motion feature stream provides a reliable and consistent data basis for the subsequent accurate recognition of user gestures, judgment of user intentions, and generation of corresponding control instructions. The source of the original three-dimensional motion data can be various devices integrated with IMU, such as smart watches, smart bracelets, VR / AR controllers, etc. This method does not limit the specific hardware implementation of the IMU or the type of device it is in.

[0041] In some embodiments, step S10 may be implemented by the following steps:

[0042] S11, performing preliminary filtering on the original three-dimensional motion data by applying an adaptive low-pass filter, wherein a cutoff frequency of the adaptive low-pass filter is dynamically adjusted according to signal energy of the original three-dimensional motion data;

[0043] S12, performing sensor fusion processing on the raw 3D motion data after preliminary filtering using a Kalman filter to generate the 3D posture parameters and the 3D angular velocity parameters;

[0044] S13. Normalize and offset-correct the three-dimensional posture parameters and the three-dimensional angular velocity parameters to obtain the standardized motion feature flow.

[0045] In step S11, an adaptive low-pass filter is used to remove high-frequency noise from the raw 3D motion data, retaining the low-frequency, valid signals associated with the user's actual movements. The key to the adaptive low-pass filter is its ability to dynamically adjust its cutoff frequency based on the signal's energy, striking a balance between noise removal and signal detail preservation. Signal energy is typically determined by calculating the root mean square value of acceleration or angular velocity data over a period of time. This dynamic adjustment mechanism enables the filter to adapt to changes in the intensity of user movements, such as rapid waving versus slow adjustments.

[0046] In step S12, the Kalman filter comprehensively estimates the device's attitude and angular velocity by fusing data from the accelerometer, gyroscope, and magnetometer. The accelerometer provides information about the device's tilt relative to gravity, the gyroscope calculates angular displacement through integration, and the magnetometer is used to correct heading drift. The Kalman filter uses state prediction and measurement update mechanisms to optimize the integration of multi-sensor data and generate accurate three-dimensional attitude parameters (such as quaternions or Euler angles) and three-dimensional angular velocity parameters. This fusion process significantly reduces the effects of noise and drift in single sensor data, providing a reliable data foundation for subsequent processing.

[0047] In step S13, normalization and offset correction further optimize the generated attitude and angular velocity parameters. Normalization scales parameters of different dimensions to a uniform range (such as [0, 1] or [-1, 1]) to eliminate dimensional differences and facilitate subsequent algorithm processing. Offset correction is used to eliminate zero-point drift of the sensor when it is stationary. For example, a gyroscope may output a non-zero angular velocity when there is no motion. Through these processes, the generated standardized motion feature stream has a consistent format and high precision, which can support accurate mapping of multiple control modes.

[0048] In step S11, adaptive filtering ensures initial data cleanliness. In step S12, Kalman filtering fuses multi-sensor information to generate precise parameters. In step S13, normalization and correction ensure data consistency and reliability. This multi-stage processing mechanism works together to transform raw data into a standard feature stream suitable for complex interactive scenarios, laying a solid foundation for subsequent cursor movement, scroll wheel control, or gesture recognition.

[0049] For example, suppose a user uses a smartwatch equipped with an IMU to interact with a Windows laptop. The IMU outputs raw 3D motion data at a sampling rate of 100Hz, including angular velocity [30, 15, 8] degrees / second and acceleration [0.1, 0.2, -9.7]m / s. 2In step S11, the system calculates the energy of the angular velocity signal and obtains a root mean square value of approximately 19.4 degrees / second. The cutoff frequency of the adaptive low-pass filter is determined to be 6.94Hz according to the formula f_c=0.1*E+5, and the noise above this frequency is filtered out, and the preliminary filtered data is output. In step S12, the Kalman filter fuses the filtered acceleration, angular velocity and magnetometer data to generate three-dimensional attitude parameters (for example, the pitch angle represented by quaternion is 12°, the roll angle is 6°, and the yaw angle is 2°) and angular velocity parameters [30,15,8] degrees / second. In step S13, the posture parameters are normalized using the formula θ_norm = (θ - θ_min) / (θ_max - θ_min). For example, a pitch angle of 12° is normalized to (12 + 180) / (180 - (-180)) = 0.533. The angular velocity is corrected for the offset (e.g., an offset of [0.3, 0.2, 0.1] degrees / second becomes [29.7, 14.8, 7.9] degrees / second after correction). This ultimately generates a standard motion feature stream containing the normalized postures [0.533, 0.517, 0.506] and the corrected angular velocities [29.7, 14.8, 7.9] degrees / second. This feature stream can be used for subsequent cursor control or gesture recognition, for example, to support precise positioning in graphic design scenarios or enable fast navigation in gaming scenarios.

[0050] As can be understood, through the dynamic frequency adjustment of the adaptive low-pass filter, the system balances noise removal and signal preservation at varying motion intensities, significantly improving data quality. Furthermore, by fusing multi-sensor data through a Kalman filter, the system overcomes the limitations of a single sensor and generates precise attitude and angular velocity parameters. Furthermore, through normalization and offset correction, the system ensures data consistency and reliability, providing a high-quality input foundation for subsequent mouse and keyboard control, thereby improving the accuracy and smoothness of interactions.

[0051] S20. Get a screenshot of the currently active application, and input the screenshot into a pre-trained object detection deep learning model to detect virtual buttons and form screen button layout information. This process aims to perceive the virtual buttons in the current application interface (such as skill keys, direction keys or interaction buttons in games) in real time, identify the position, size and function of the buttons through a deep learning model, and provide dynamic interface information support for subsequent interaction mode determination and instruction generation. The screen button layout information contains the bounding box position data and function category labels of the virtual buttons, which can reflect the layout and interaction status of the interface elements. Through this step, the system can intelligently map the user's actions based on the inertial measurement unit (IMU) to the target virtual buttons according to the dynamic changes of the application interface, support the coordinated operation of mouse cursor control and virtual button interaction, and is suitable for dynamic interaction scenarios such as game control and virtual reality interfaces.

[0052] In some embodiments, step S20 may be implemented by the following steps:

[0053] S21. Accurately determine the position and size of each virtual key detected by the pre-trained object detection deep learning model to obtain the bounding box position data of each virtual key;

[0054] S22: performing fine-grained category recognition on each of the detected virtual keys to obtain a category label indicating a function type and a current interaction state of the virtual key;

[0055] S23: Constructing the screen button layout information based on the bounding box position data and the category label.

[0056] In step S21, the system obtains a screenshot of the currently active application (for example, through the Windows PrintWindow API or the Android MediaProjection interface) and inputs a pre-trained object detection deep learning model (for example, a model based on architectures such as YOLO, SSD, or Faster R-CNN). The model outputs bounding box position data for each virtual key, including screen coordinates (such as the upper left corner [x, y] and width and height [w, h]) and a confidence score (such as 0.9), accurately locating the position and size of the key in the interface. The bounding box data ensures that the system can accurately associate the cursor position with the target key.

[0057] In step S22, the system performs fine-grained category recognition on the detected virtual keys and generates category labels indicating the key's function type (e.g., "confirm key," "direction key," "skill key") and current interaction state (e.g., "clickable," "disabled"). Category recognition can be implemented through the model's classification head or post-processing logic, for example based on the function labels and interface context (e.g., key color, text) annotated during training. This ensures that the system understands the interactive meaning of the keys and adapts to dynamic interface changes.

[0058] In step S23, the system integrates the bounding box position data (e.g., [x, y, w, h]) and category label (e.g., "Skill key, clickable") into screen button layout information, structuring it into a list of key-value pairs (e.g., {"Button 1":{"Position":[100,200,50,30], "Category":"Confirm key, clickable"}). This information provides a basis for subsequent interaction mode selection and command mapping, supporting cursor positioning or virtual button triggering.

[0059] By performing precise geometric positioning of the virtual keys in step S21, performing in-depth functional and state semantic analysis on them in step S22, and integrating this information into structured screen key layout information in step S23, step S20 can ensure that the system not only roughly perceives that there are keys on the screen, but can also accurately and in detail understand the position, size, specific functions of these keys and whether they can respond to user interactions at the current moment.

[0060] For example, suppose a user is using an IMU-equipped smartwatch to operate a Windows game application, and the current interface is the character combat interface. In step S21, the system obtains a screenshot via PrintWindow and inputs it into the YOLOv5 model. Three virtual buttons are detected and the output bounding boxes are: button 1 ([150, 300, 60, 40], confidence 0.95), button 2 ([250, 300, 60, 40], confidence 0.93), and button 3 ([350, 300, 60, 40], confidence 0.90). In step S22, the model classification head identifies the button categories: button 1 is "skill key, clickable", button 2 is "direction key, clickable", and button 3 is "confirm key, disabled". In step S23, the system integrates the following on-screen button layout information: {"Skill Key":{"Position":[150,300,60,40],"Status":"Clickable"}, "Direction Key":{"Position":[250,300,60,40],"Status":"Clickable"}, "Confirm Key":{"Position":[350,300,60,40],"Status":"Disabled"}}}. This layout information allows the cursor to subsequently locate the "Skill Key" or trigger its operation.

[0061] As you can understand, through bounding box detection using a deep learning model, the system accurately perceives the position and size of virtual buttons. Furthermore, through fine-grained category recognition, the system dynamically captures button functions and interaction states, adapting to interface changes. At the same time, the integrated on-screen button layout information provides a reliable basis for subsequent interaction mode selection and command generation, significantly improving the accuracy and adaptability of IMU-based interactions in dynamic interface scenarios (such as gaming).

[0062] S30. Determine the inferred running state of the currently active application based on the human-computer interaction context information, screen button layout information, and a predefined application state model from the previous cycle. Based on the inferred running state and pre-defined virtual button function annotation rules, obtain a valid virtual button set containing functional annotations, and simultaneously update the human-computer interaction context information and the inferred running state of the currently active application. This process aims to infer the current running scenario of the application (such as combat mode or menu interface in a game) by analyzing the user interaction context and screen interface state, thereby dynamically filtering available virtual buttons and labeling them with specific functions (such as "attack" and "confirm"), providing accurate contextual basis for subsequent interaction mode selection and command generation. The screen button layout information includes the location and category of virtual buttons, the human-computer interaction context information reflects the user operation mode and application type, and the application state model defines state descriptions in different scenarios. Through this step, the system can dynamically adapt to changes in the application interface, ensure that the interaction based on the inertial measurement unit (IMU) is highly compatible with the current task scenario, and support the coordination of mouse cursor control and virtual button operation.

[0063] In some embodiments, determining the inferred running state of the currently active application based on the human-computer interaction context information of the previous cycle, the screen button layout information, and the predefined application state model includes:

[0064] S31, using the human-computer interaction context information of the previous cycle to identify the user's current interaction mode and the type of the currently active application;

[0065] S32: querying the application state model for a state description that matches the current context based on the interaction mode and the type of the currently active application, in combination with the virtual key category and position in the screen key layout information;

[0066] S33: Inferring the running status of the currently active application based on the matching result.

[0067] In step S31, the system analyzes the human-computer interaction context information from the previous cycle (such as the application name or user interaction mode obtained through the operating system API) and identifies the current interaction mode (such as mouse priority or virtual key priority) and application type (such as game, text editor). For example, if the application obtained through Windows' GetForegroundWindow is "Game.exe", it is determined to be a game type and the interaction mode is virtual key priority.

[0068] In step S32, the system combines the interaction mode, application type, and on-screen button layout information (including the bounding box location and category of virtual buttons, such as "skill button, clickable") to query a predefined application state model. This model, in the form of a state machine or rule library, stores the possible operating states of different applications (such as "battle mode" or "menu mode" in a game) and their characteristics (such as the availability of specific buttons). By matching the current context with the button layout, the system obtains the most relevant state description.

[0069] In step S33, based on the matching results, the system infers the application's current running state, such as "combat mode." Subsequently, based on the inferred state and the preset virtual key function annotation rules (e.g., "In combat mode, the skill key is mapped to attack"), a valid virtual key set (e.g., {"skill key": "attack," "direction key": "move"}) is generated, and the human-computer interaction context information (adding the current state) and the application state (recorded as "combat mode") are updated.

[0070] After determining the inferred running state of the currently active application through steps S31 to S33, the subsequent portion of step S30—namely, "Based on the inferred running state and the preset virtual key function annotation rules, obtaining a valid virtual key set containing functional annotations, and simultaneously updating the human-computer interaction context information and the inferred running state of the currently active application"—begins. The system references this newly inferred application running state and applies a set of preset virtual key function annotation rules to assign specific semantic functions appropriate to the current state to each (or some of the relevant) virtual keys in the screen key layout information. For example, an annotation rule might stipulate, "If the application state is 'Video Playing,' then the function of the virtual key with the pause icon detected on the screen is 'Execute Pause Operation'." In this way, the system is able to form a valid virtual key set that not only includes virtual key location and category information, but more importantly, includes the specific executable functions of the virtual keys in the current application state. Finally, this newly inferred application running state and the valid virtual key set containing functional annotations will jointly update the human-computer interaction context information maintained by the system for use in the next round of decision-making.

[0071] For example, suppose a user is using an IMU-equipped smartwatch to operate a Windows gaming application. The context information from the previous cycle indicates that the application is "Game.exe," the interaction mode is virtual key priority, and the screen key layout information includes three keys: {"Skill Key": {"Position": [150, 300, 60, 40], "Status": "Clickable"}, "Direction Key": {"Position": [250, 300, 60, 40], "Status": "Clickable"}, "Confirm Key": {"Position": [350, 300, 60, 40], "Status": "Disabled"}}. In step S31, the system confirms that the application type is game and the interaction mode is virtual key priority. In step S32, based on the layout information, the application state model is queried (defining that "Game.exe" is in combat mode when "skill keys are clickable and the confirm key is disabled"), and the "combat mode" state description is matched. In step S33, the system infers that the operating state is "combat mode" and generates a valid virtual key set based on the annotation rules: {"skill key": "attack", "direction key": "move"}. The context information is updated (adding "combat mode") and the application state. This set supports subsequent command generation, such as mapping a gesture to an "attack" action.

[0072] It can be understood that by leveraging the recorded human-computer interaction contextual information from the previous interaction cycle to accurately identify the user's current interaction mode and the type of active application, a clear and consistent contextual starting point can be provided for a more precise understanding of the application's internal state and interface element functionality. Furthermore, based on the identified interaction mode and application type, and further combined with the on-screen button layout information generated in real time by the vision module, a pre-built application state model containing typical interface behavior patterns of a specific application at different operational stages is proactively queried. Based on the state description obtained from the query that best matches the current context, the system accurately infers the specific internal operational state of the currently active application through an evidence-based reasoning approach. Based on this newly inferred precise application operational state and referring to a set of pre-set virtual button function annotation rules, each virtual button detected on the screen that has actual interactive significance in its current state is assigned a specific functional meaning that aligns with the current application logic. This transforms the original virtual button detection results, which only contain visual and geometric attributes, into a valid virtual button set containing rich semantic information that can directly guide subsequent interaction behavior decisions. At the same time, by refreshing the overall human-computer interaction context information maintained by the system with the latest inferred application running status and updated functional annotation information at the end of each round of processing, it is ensured that the system can dynamically and continuously track real-time changes in application processes, ensuring that its understanding of virtual button functions and the formulation of subsequent interaction decisions can always be highly synchronized and consistent with the latest internal status and interface presentation of the application.

[0073] S40. Determine the currently effective interaction mode based on the updated human-computer interaction context information, the updated inferred running state of the current application, and the user input instructions. The interaction mode includes a mouse-priority interaction mode and a virtual key-priority interaction mode. This process aims to dynamically select the most suitable interaction mode based on the user's current interaction context, application state, and clear gesture instructions to optimize the synergy between the mouse cursor control and virtual key operations based on the inertial measurement unit (IMU). By querying the predefined "context-interaction mode rule library", the system decides to adopt the mouse-priority interaction mode (suitable for cursor positioning and clicking) or the virtual key-priority interaction mode (suitable for triggering interface buttons, such as game skill keys) to ensure that the interaction method is highly matched with the task scenario.

[0074] In some embodiments, determining the currently effective interaction mode based on the updated human-computer interaction context information, the updated inferred running state of the current application, and the user input instruction includes:

[0075] The running state is inferred based on the updated human-computer interaction context information and the updated current application, combined with the user input instruction, and the currently effective interaction mode is determined according to the predefined "context-interaction mode rule library", wherein the user input instruction is a predefined mode switching gesture instruction obtained by performing specific gesture recognition processing on the standardized motion feature stream.

[0076] Specifically, the system first analyzes updated human-computer interaction context information (such as the application name and interface element status obtained through the operating system API) and the inferred running state of the current application (such as the "battle mode" of a game) to determine the context of the user's operation. User input instructions are generated by performing gesture recognition processing on a standardized motion feature stream (containing three-dimensional posture and angular velocity parameters). For example, the dynamic time warping (DTW) algorithm is used to match mode switching gesture templates in a pre-built hierarchical motion primitive library (such as making a fist to indicate virtual key priority). The system then queries the "context-interaction mode rule library", which defines the mapping relationship between context, state, gesture and interaction mode. For example, the rule may stipulate: in the game "battle mode", if a fist gesture is detected, the virtual key priority mode is selected; if there is no gesture or in a text editing scenario, the mouse priority mode is selected by default. This joint analysis ensures that the interaction mode is consistent with the user's intention and application scenario. By using this decision-making approach based on a predefined rule base and taking into account the real-time updated application state, interface layout understanding, and the user's explicit gesture instructions, step S40 can ensure that the selection of the interaction mode is both intelligent and context-adaptive, while fully respecting and responding to the user's direct control intentions, so that subsequent IMU inputs can be directed to the most appropriate processing flow.

[0077] For example, suppose a user uses an IMU-equipped smartwatch to operate a Windows game application. The updated human-computer interaction context information indicates that the application is "Game.exe", and the operating state is inferred to be "combat mode". The standardized motion feature stream contains an angular velocity of [50, 25, 10] degrees / second (lasting 0.1 seconds). In step S41, the system matches the feature stream with the "fist" template (single rapid angular velocity peak) in the primitive library through the DTW algorithm, and confirms that the mode switching gesture instruction is "virtual key priority". Combining the context information (game type) and the operating state (combat mode), the rule library is queried (defining "combat mode + fist → virtual key priority") to determine that the current effective interaction mode is virtual key priority mode, which is suitable for triggering game skill keys. If there is no fist gesture or the context is a text editor, the rule library selects mouse priority mode to support cursor positioning.

[0078] It can be understood that by relying on a predefined context-interaction mode rule base that includes multiple condition judgments and mode mapping logic, and comprehensively considering three core dynamic input information - namely, the detailed human-computer interaction context information updated in real time by the previous processing link, which already includes annotations of the valid virtual buttons on the screen and their functions, the accurate inference of the specific running stage of the currently active application, which is also updated in real time, and the direct control intention expressed by the user by executing specific predefined mode switching gesture instructions obtained through standardized motion feature stream recognition processing - the system can dynamically and accurately determine the interaction mode that should be most effective at the current moment in a highly intelligent, context-aware and timely responsive to user active intervention manner, whether it should focus on the mouse-priority interaction mode of traditional mouse pointer operations, or the virtual button-priority interaction mode of interacting with the virtual buttons on the screen interface. This sophisticated decision-making mechanism, based on rules and multi-source real-time information input, ensures that the choice of interaction mode is no longer preset, fixed, or dependent solely on single user input or isolated scenario judgment. Instead, it can be flexibly, reasonably, and efficiently adaptively adjusted based on the immediate state of the application interface (for example, whether a large number of virtual buttons are available for operation), the application's internal logical flow (for example, whether the user is currently in a specific functional interface suitable for virtual button input), and the user's explicit switching instructions. This significantly improves the overall control flexibility, operational efficiency, and naturalness and smoothness of the user experience when using IMU devices to interact with modern applications with complex interface elements and diverse interaction requirements (especially those that require both fine pointer control and rely heavily on virtual button input, such as various professional design software, content creation tools, or immersive games). It allows users to seamlessly and intuitively switch between different interaction focuses and task requirements, thereby better realizing the potential of the IMU as a multi-functional input device.

[0079] S50. According to the currently effective interaction mode, select a mapping algorithm corresponding to the currently effective interaction mode to process the standardized motion feature stream to generate at least one of a mouse cursor movement instruction, a mouse button instruction, and a virtual button operation instruction. This process is intended to extract appropriate parameters from the standardized motion feature stream and generate corresponding interaction instructions through a specific algorithm based on the determined interaction mode (mouse priority or virtual key priority), ensuring that the actions based on the inertial measurement unit (IMU) can be accurately mapped to mouse cursor movement, mouse key operation, or virtual key triggering. Through mode-specific mapping and gesture recognition, the system supports the interaction requirements of different scenarios (such as skill key triggering in games or cursor positioning in text editing), and improves the flexibility and accuracy of multimodal interaction.

[0080] In some embodiments, step S50 may be implemented by the following steps:

[0081] S51: If the currently effective interaction mode is the mouse-priority interaction mode, selecting parameters from the standardized motion data stream as basic cursor control inputs based on the current human-computer interaction context information, determining and applying a mapping algorithm corresponding to the currently effective sensitivity mapping mode to process the basic cursor control inputs to generate mouse cursor movement instructions; and performing a first gesture recognition process on the standardized motion feature stream based on a pre-built hierarchical motion primitive library to identify mouse button gestures and parse them into mouse button instructions.

[0082] S52: If the currently effective interaction mode is the virtual key priority interaction mode, a second gesture recognition process is performed on the standardized motion feature stream according to the pre-built hierarchical motion primitive library to recognize the virtual key gesture and parse it into a virtual key operation instruction.

[0083] Alternatively, generating the mouse cursor movement instruction in step S51 may be implemented by the following steps:

[0084] S511. Determine the type of the currently active application or the type of the user interface element under the cursor as current human-computer interaction context information through an application program interface or a user interface analysis tool provided by the operating system;

[0085] S512. Selecting, according to a preset context mapping table, a specific axial combination of three-dimensional gesture parameters or a specific axial combination of three-dimensional angular velocity parameters corresponding to the type of the currently active application or the type of the user interface element under the cursor as a basic cursor control input;

[0086] S513, detecting a motion amplitude or a motion speed related to a basic cursor control input in the standard motion data stream;

[0087] S514: When the movement amplitude or the movement speed exceeds a preset first threshold, determining the preset relative mapping mode as the currently effective sensitivity mapping mode;

[0088] S515: When the movement amplitude or the movement speed is lower than a preset second threshold, determining the preset absolute mapping mode as the currently effective sensitivity mapping mode;

[0089] S516. Read the physical mouse sensitivity setting of the operating system, and calculate the basic mouse sensitivity adjustment parameter according to the physical mouse sensitivity setting;

[0090] S517. Use the basic mouse sensitivity adjustment parameter as the gain adjustment factor of the mapping algorithm corresponding to the currently effective sensitivity mapping mode to process the basic cursor control input, and when the currently effective sensitivity mapping mode changes from the sensitivity mapping mode of the previous control cycle, perform a smooth transition processing based on weighted average based on the current output of the preset mapping algorithm and the final output state data of the previous control cycle to generate a mouse cursor movement instruction.

[0091] In step S511, the operating system's application program interface (e.g., Windows' GetForegroundWindow) or user interface analysis tools (e.g., UI Automation) are used to identify the type of the currently active application (e.g., a text editor, graphic design software) or the type of interface element under the cursor (e.g., a button, text box). This information serves as the basis for the human-computer interaction context, helping the system understand the user's current operation goals and thus optimize the selection of control inputs.

[0092] In step S512, a pre-defined contextual mapping table defines the correspondence between different applications or interface elements and axial combinations of motion parameters. For example, in graphic design software, a combination of pitch and roll angles might be selected for precise cursor positioning; in gaming applications, the X- and Y-axis components of angular velocity parameters might be selected to support rapid movement. This context-adaptive selection mechanism ensures that control inputs match task requirements.

[0093] In step S513, the system detects the movement amplitude (such as the root mean square value of the angular velocity) or movement speed of the basic cursor control input to determine the user's action intensity. This step provides a quantitative basis for subsequent mapping mode selection.

[0094] In steps S5151 and S515, the system determines whether to use relative mapping mode or absolute mapping mode based on the comparison of the motion amplitude or speed with the preset threshold. The relative mapping mode maps motion data (such as angular velocity) to the relative displacement of the cursor, which is suitable for fast, large-scale movements and for scenarios where the amplitude exceeds a first threshold (such as 20 degrees / second). The absolute mapping mode directly maps gesture parameters to absolute screen coordinates, which is suitable for fine manipulation and for scenarios where the amplitude is below a second threshold (such as 10 degrees / second).

[0095] In step S516, the system reads the physical mouse sensitivity setting of the operating system (for example, in the Windows system, the system obtains the mouse sensitivity value set by the user by calling the SystemParametersInfo function (using the SPI_GETMOUSESPEED parameter), which is usually in the range of 1 to 20, and the default value is 10, indicating medium sensitivity), and calculates the basic mouse sensitivity adjustment parameter through a predefined conversion rule (such as a linear formula). This parameter reflects the user's sensitivity preference for a traditional mouse and ensures that the IMU control is consistent with the user's habits. For example, the conversion formula can be defined as S_IMU=k*S_OS+b, where S_OS is the sensitivity value obtained by the operating system, k and b are constants preset according to the IMU sensor characteristics and cursor control requirements, and S_IMU is the calculated basic sensitivity parameter, which is used to subsequently scale the IMU motion data to the cursor movement distance.

[0096] In step S517, the basic mouse sensitivity adjustment parameter is used as a gain factor to adjust the output ratio of the mapping algorithm, for example, scaling the angular velocity to pixel displacement ratio in relative mapping mode. When the mapping mode is switched (e.g., from relative to absolute), the system uses a weighted average method (e.g., C = w(t) * C_prev + (1-w(t)) * C_current) to smoothly transition between the current output and the previous cycle output to avoid cursor jumps.

[0097] Steps S511 and S512 select the appropriate control input based on contextual information and a mapping table. Steps S513 to S515 dynamically determine the mapping mode. Steps S516 and S517 optimize command generation through sensitivity adjustment and smooth transitions. This combination ensures that cursor movement command generation is both user-friendly and adaptable to the current scenario and action characteristics.

[0098] For example, suppose a user uses an IMU-equipped smartwatch to operate on a Windows laptop, the current control mode is cursor control mode, and the normalized motion feature stream contains normalized posture [0.533, 0.517, 0.506] and angular velocity [29.7, 151.8, 7.9] degrees / second. In step S511, the system detects through GetForegroundWindow that the current active application is "Photoshop.exe", indicating that the user is in a graphic design scene. In step S512, according to the context mapping table, the pitch angle and roll angle ([0.533, 0.517]) of the posture parameters are selected as the basic cursor control input, which is suitable for precise positioning. In step S513, the system calculates the angular velocity amplitude as sqrt(29.7 2 +151.8 2 +7.9 2)≈33.1 degrees / second. In steps S5151 and S515, the amplitude exceeds the first threshold of 20 degrees / second but is lower than the second threshold of 10 degrees / second and is not triggered, and the relative mapping mode is selected. In step S516, the system reads the mouse sensitivity setting of 151 through SystemParametersInfo and calculates the basic sensitivity parameter S_IMU=0.051*151+0.2=0.76. In step S517, the system maps the angular velocity [29.7,151.8] degrees / second to the cursor displacement [0.76*29.7≈22.6,0.76*151.8≈11.2] pixels / second. If switched to absolute mode, it smoothly transitions to the absolute coordinates [5183.2,5187.6] through weighted averaging (weight w(t)=0.7) to generate cursor movement instructions, which are suitable for accurately selecting layers in Photoshop.

[0099] As you can see, by selecting control inputs based on the human-computer interaction context, the system adapts to different task requirements and improves interaction efficiency. Furthermore, by dynamically selecting relative or absolute mapping modes, the system balances fast navigation with precise operation. Combined with operating system sensitivity settings and a smooth transition mechanism, this ensures the continuity and consistency of cursor movement, significantly improving the accuracy and naturalness of IMU-based interactions in complex scenarios.

[0100] Alternatively, generating the mouse button movement instruction in step S51 may be implemented by the following steps:

[0101] S5111. Applying a preset sequence matching rule to a portion of the preliminary primitive sequence that is not recognized as the mode switching gesture, the portion is matched with mouse button gesture templates defined in the pre-built hierarchical motion primitive library, where the mouse button gesture templates include a single-click gesture template, a double-click gesture template, and a long-press gesture template.

[0102] S5112, performing duration verification, motion amplitude verification, and confidence threshold filtering based on current human-computer interaction context information on the successfully matched candidate mouse button gestures;

[0103] S5113. Parse the verified candidate mouse button gesture into a corresponding left-click instruction, right-click instruction, double-click instruction, or a button press and release instruction sequence to generate the mouse button instruction.

[0104] In step S5111, the system uses sequence matching rules (e.g., dynamic time warping, or DTW) to compare the non-mode switching portion of the preliminary primitive sequence with a mouse button gesture template. A single-click gesture template might be defined as a single rapid angular velocity peak (e.g., lasting 0.1 seconds); a double-click template as two peaks (less than 0.5 seconds apart); and a long-press template as a sustained stabilization of gesture parameters (e.g., a change of less than 0.01 within 0.5 seconds). The DTW algorithm allows for nonlinear alignment along the timeline by calculating the similarity between the sequence and the template, adapting to differences in user motion rhythms.

[0105] In step S5112, candidate key gestures undergo multiple verifications to ensure reliability. Duration verification checks whether the gesture duration meets template requirements (e.g., a single click is less than 0.2 seconds). Amplitude verification confirms that the angular velocity or posture change exceeds a threshold (e.g., 50 degrees / second) to distinguish intentional motion from noise. Confidence threshold filtering adjusts the verification criteria based on the human-computer interaction context (e.g., the application is a file manager). For example, a higher threshold (e.g., 0.9) may be set in high-precision scenarios (e.g., graphic editing) to reduce false positives.

[0106] In step S5113, the candidate gestures that pass the verification are parsed into standard mouse button commands. For example, a single-click gesture may generate MOUSEEVENTF_LEFTDOWN / UP (left-click) or MOUSEEVENTF_RIGHTDOWN / UP (right-click), a double-click generates two consecutive left-clicks, and a long-press generates a press-and-release sequence (such as a drag operation). This process ensures that the commands are compatible with the operating system's input protocol.

[0107] Step S5111 identifies potential button gestures through template matching, step S5112 ensures accuracy through multiple verifications, and step S5113 converts the gestures into standard commands. This combined mechanism ensures accurate and reliable generation of mouse button commands.

[0108] For example, suppose a user is using an IMU-equipped smartwatch to navigate the Windows file manager. The initial primitive sequence includes the "rapid angular velocity peak" and "stable posture" flags. In step S5111, the system detects two angular velocity peaks ([60, 30, 15] degrees / second and [55, 25, 12] degrees / second, separated by 0.3 seconds). DTW matches the double-click template (two peaks separated by <0.5 seconds) with a similarity score of 0.95. In step S5112, duration verification confirms that the 0.3-second interval meets the requirement, and amplitude verification confirms that the peak exceeds 50 degrees / second. Context analysis (using the application "explorer.exe") sets a confidence threshold of 0.9, and the matching score satisfies the requirements. In step S5113, the double-click gesture is interpreted as a two-mouse event F_LEFTDOWN / UP command sequence, triggering the file open operation.

[0109] As you can see, through sequence matching rules and the DTW algorithm, the system accurately identifies mouse button gestures and adapts to variations in movement. Furthermore, a multi-verification mechanism significantly reduces false positives by filtering by duration, amplitude, and contextual confidence. Furthermore, interpreting gestures as standard commands ensures compatibility with the operating system and improves the accuracy and fluidity of IMU-based mouse interactions in complex scenarios.

[0110] In step S52, when the interaction mode is virtual key priority interaction mode, the system uses the DTW algorithm to match the motion feature stream with the virtual key gesture template in the primitive library (for example, a quick jitter corresponds to a "skill key" trigger). The template is defined based on the screen key layout information (such as the position of the "skill key"), and the recognition result is parsed into a virtual key operation instruction, such as mapping to a specific function key (such as the "Q" key) or interface action (such as "attack") to adapt to dynamic interface scenarios.

[0111] Step S51 supports cursor and key operations in mouse priority mode, and step S52 supports interface button triggering in virtual key priority mode, ensuring that command generation matches the interaction mode.

[0112] For example, assume a user is using an IMU-equipped smartwatch to operate a Windows gaming application. The normalized motion feature stream includes normalized poses [0.533, 0.517, 0.506] and angular velocities [29.9, 14.9, 7.95] degrees / second, and the interaction mode is virtual key-press priority. In step S52, the system detects an angular velocity peak of [50, 25, 10] degrees / second (lasting 0.1 seconds). Using DTW, the system matches the "fast jitter" template (corresponding to the "skill key") in the primitive library to generate a virtual key operation instruction (mapping the "Q" key to trigger "attack"). If the system switches to mouse-priority mode (step S51), the system selects pose parameters [0.533, 0.517]. Since the angular velocity amplitude of 27.6 degrees / second is greater than 20 degrees / second, the system selects relative mapping mode, generates a cursor movement instruction of [16.9, 8.1] pixels, and simultaneously matches the single-click template (single peak) to generate a MOUSEEVENTF_LEFTDOWN / UP instruction.

[0113] As you can understand, through mode-specific mapping algorithms and gesture recognition, the system flexibly generates mouse or virtual button commands to adapt to the needs of different scenarios. In addition, by combining contextual information with the precise matching of primitive libraries, the system improves the pertinence and robustness of command generation. At the same time, dynamic mode support and standard command formats ensure the smoothness and compatibility of interaction, significantly improving the accuracy and efficiency of IMU-based interaction in dynamic scenarios (such as games).

[0114] S60. Perform instruction fusion and collaborative event generation on the confirmed mouse cursor movement instructions, the mouse button instructions, and the virtual button operation instructions to obtain a unified collaborative interaction event sequence. This process aims to integrate multimodal instructions (cursor movement, mouse button, virtual button operation), combine interaction modes and contextual information, and generate a logically consistent and time-coordinated event sequence to ensure that actions based on the inertial measurement unit (IMU) can be accurately mapped to complex interactive behaviors, such as triggering virtual buttons or executing shortcut key operations after cursor positioning. Through priority rules and conflict resolution, the system supports composite operations (such as cursor positioning + skill key triggering) to improve the smoothness and reliability of interaction.

[0115] In some embodiments, step S60 may be implemented by the following steps:

[0116] S61, determining a target virtual key under the current screen cursor position based on the current screen cursor position indicated by the mouse cursor movement instruction and the screen key layout information;

[0117] S62. Based on the currently effective interaction mode, the updated human-computer interaction context information and the updated inferred running status of the currently active application, the preset priority rules are applied to handle the conflicts and ambiguities between the mouse button instructions and the virtual button operation instructions to obtain the unified collaborative interaction event sequence.

[0118] In step S61, the system analyzes the current screen cursor position (e.g., [x, y] coordinates) of the mouse cursor move command and combines it with the on-screen button layout information (e.g., the virtual button bounding box [150, 300, 60, 40]) to determine whether the cursor is within a virtual button area. If the cursor coordinates fall within the button bounding box, the system identifies the button as the target virtual button (e.g., "skill key") and retrieves its function annotation (e.g., "attack") from the valid virtual button set, providing a basis for subsequent command fusion.

[0119] In step S62, the system applies priority rules to handle command conflicts based on the interaction mode (e.g., virtual key priority), human-computer interaction context (e.g., game type), and application state (e.g., "battle mode"). These priority rules are specifically designed to handle complex situations: when the system simultaneously receives or recognizes potentially related mouse button commands (e.g., a general "click" gesture) and virtual key operation commands (e.g., a gesture specifically for "activating a virtual key"), or when the location of a mouse button command coincides with the target virtual key determined in step S61, how to determine the user's true intention and how to decide which operation should ultimately be performed or how to combine these operations. For example, the priority rules may specify that virtual key operation commands take precedence over mouse button commands to support skill triggering in games; when a mouse button command (e.g., single click, confidence level 0.95) and a virtual key command (e.g., "skill key" trigger, confidence level 0.90) occur simultaneously, the system suppresses the lower-priority command through confidence comparison or time window (±50 milliseconds). The conflict resolution logic also considers the relevance of the cursor position to the target button. For example, if the cursor is not within the button area, the virtual button command priority is lowered. After fusion, the system generates a unified collaborative interaction event sequence, including commands with aligned timestamps (such as cursor movement → virtual button triggering).

[0120] Step S61 first clarifies the spatial correspondence between the cursor and the virtual buttons on the screen, and then step S62 uses the preset priority rules based on the current complete interaction context to intelligently process and resolve possible conflicts and semantic ambiguities between mouse button instructions and virtual button operation instructions. Step S60 can efficiently and accurately integrate the various original operation instructions from the user's IMU input and the large amount of contextual information maintained by the system itself into a unified and highly coordinated interaction event sequence that can directly guide the generation of subsequent operating system-level events.

[0121] For example, suppose the user is currently playing a game, and there is an important skill virtual button A on the interface.

[0122] First, through steps S20 and S30, the system has obtained the on-screen button layout information (including the location and function annotation of skill button A, "Release Fireball") and inferred that the current application (game) running state is "In Combat." The human-computer interaction context information is updated accordingly. Through step S40, the currently effective interaction mode is set to virtual button priority interaction mode.

[0123] The user uses the IMU device to move the mouse cursor into the area of ​​skill button A. This mouse cursor movement command is recorded.

[0124] In the implementation of step S61, the system determines that the cursor is located within the boundary box of skill button A based on the current screen cursor position indicated by the mouse cursor movement instruction and referring to the screen button layout information, and therefore determines that skill button A is the target virtual button under the current screen cursor position.

[0125] Next, the user performs a quick downward tap gesture. Through step S50 and its refinements, this gesture may be initially recognized by the first gesture recognition process (for the mouse button) as a mouse button instruction of "left click" and also by the second gesture recognition process (for the virtual button) as a virtual button operation instruction of "activate selected target".

[0126] At this point, in the implementation of step S62, the system applies a preset priority rule. The rule might state: "If the currently active interaction mode is virtual key priority, and the target virtual key (such as skill key A in this case) under the current screen cursor position is confirmed to be valid (for example, its function annotation indicates that it is available in the current 'in combat' state), the virtual key operation command should be prioritized and associated with the target virtual key. At the same time, general mouse button commands that are similar in time and may be generated by the same gesture may need to be suppressed or ignored to avoid duplicate operations or behavioral conflicts." Therefore, the system will ultimately interpret the user's tap gesture as executing the "Release Fireball" action on skill key A. This decision will be recorded in the unified collaborative interaction event sequence, for example, forming an event such as "Activate Virtual Key (ID: Skill Key A, Function: Release Fireball)." If the interaction mode at the time was mouse priority, the priority rule might make a different judgment, for example, still treating it as a normal left-click of the mouse, or generating both a click event and a virtual key activation event (if the virtual key is designed to respond to standard clicks).

[0127] As you can see, by analyzing the correlation between cursor position and button layout, the system accurately identifies the target virtual button. Furthermore, priority rules and conflict resolution logic effectively coordinate multimodal commands, reducing false trigger rates and generating a logically consistent sequence of events. Furthermore, by incorporating contextual and state information, the system adapts to dynamic interaction scenarios, significantly improving the accuracy and fluidity of IMU-based interactions in complex scenarios like gaming.

[0128] S70. Perform event conversion on the unified collaborative interaction event sequence, generate a standard input event stream of the target operating system and inject it into the event processing queue of the target operating system. This process aims to convert the unified collaborative interaction event sequence generated by the above steps, including mouse cursor movement instructions (such as two-dimensional displacement or absolute coordinates), mouse button instructions (such as single-click, double-click) and virtual button operation instructions (such as triggering a "skill key"), into standard mouse and keyboard input events that can be recognized by the operating system, ensuring that the actions based on the inertial measurement unit (IMU) can seamlessly drive complex interactive behaviors, such as cursor positioning, clicking or virtual button activation. After the event sequence is converted, it is injected into the event queue through the input interface of the operating system, and distributed by the system to the currently active application to trigger the corresponding interactive operation, support the collaborative control of the mouse and virtual buttons, and is suitable for dynamic scenes such as game control and virtual reality interaction.

[0129] In some embodiments, step S70 may be implemented by the following steps:

[0130] S71. When the mouse cursor movement instruction indicates absolute coordinate mapping, convert the target position in the mouse cursor movement instruction into normalized absolute screen coordinates and generate a corresponding standard mouse movement event;

[0131] S72. When the mouse cursor movement instruction indicates relative displacement mapping, convert the displacement amount in the mouse cursor movement instruction into a relative movement pixel value relative to the last reported position and generate a corresponding standard mouse movement event;

[0132] S73: Convert the mouse button command into corresponding standard mouse button press and release events, and convert the virtual key activation determined by the virtual key operation command into corresponding standard keyboard key events;

[0133] S74, converting the standard mouse movement event, the standard mouse button press and release event, and the standard keyboard key event into a standard input event format recognizable by the target operating system;

[0134] S75. Inject the standard input event stream into the event processing queue through the event injection interface provided by the target operating system.

[0135] In step S71, for the mouse cursor movement instruction mapped by absolute coordinates (such as the screen coordinates [600, 400] mapped by posture parameters), the system normalizes the target position to the screen resolution range (such as [0, 1], converted to [0.3125, 0.3704] based on 1920x1080 resolution) and generates a standard mouse movement event (such as Windows' MOUSEEVENTF_ABSOLUTE).

[0136] In step S72, for the relative displacement mapping instruction (such as the displacement of [16.9, 8.1] pixels for angular velocity mapping), the system calculates the new position based on the last reported position (such as [590, 395]) and generates a standard mouse movement event (such as MOUSEEVENTF_MOVE in Windows).

[0137] In step S73, mouse button instructions (such as single-click) are converted into standard events (such as Windows' MOUSEEVENTF_LEFTDOWN / UP); virtual key operation instructions (such as "skill key" mapped to "Q" key) are converted into standard keyboard key events (such as KEYEVENTF_KEYDOWN / UP, virtual key code VK_Q).

[0138] In step S74, the system integrates all events into the standard input event format of the target operating system (such as the INPUT structure of Windows), ensuring the alignment of timestamps and the consistency of parameters.

[0139] In step S75, the event stream is injected into the queue through the operating system interface (such as SendInput in Windows or CGEventPost in macOS). The system can provide feedback, such as device vibration (0.2 seconds) or interface prompts, to enhance user perception.

[0140] Through steps S71 and S72, mouse cursor movement instructions in different modes are accurately standardized and converted. Step S73 converts mouse button intentions into standard key events and maps innovative virtual key activations to equivalent keyboard events. Combined with step S74, all these events are finally formatted for the target operating system, and step S75 completes the injection through the standardized system interface. Step S70 can ensure that the rich, complex and highly context-aware interaction intentions generated by the entire IMU control method can ultimately be accurately reflected and executed on the user's computer system.

[0141] For example, suppose a user uses an IMU-equipped smartwatch to operate a Windows gaming application. The unified collaborative interaction event sequence includes cursor movement ([160, 320], t=1000ms), a single click (t=1005ms), and the virtual "skill key" ("Q" key, t=1010ms). In step S71, the system normalizes [160, 320] to [0.0833, 0.2963] (based on 1920x1080) and generates a MOUSEEVENTF_ABSOLUTE event. In step S72, relative mapping is not used and is skipped. In step S73, the single click is converted to MOUSEEVENTF_LEFTDOWN / UP, and the virtual key is converted to KEYEVENTF_KEYDOWN / UP (VK_Q). In step S74, it is integrated into an INPUT structure. In step S75, it is injected into the queue via SendInput, triggering cursor positioning and "skill key" activation, and the watch generates 0.2 seconds of vibration feedback.

[0142] As you can see, through flexible event conversion, the system ensures command compatibility with the operating system. Furthermore, the event injection and feedback mechanisms improve interaction reliability and user experience, significantly enhancing the accuracy and smoothness of IMU-based interactions in dynamic scenarios.

[0143] In addition, an embodiment of the present invention further provides a computer-readable storage medium. The computer-readable storage medium can be any one of, or any combination of, a hard disk, a multimedia card, an SD card, a flash memory card, an SMC, a read-only memory (ROM), an erasable programmable read-only memory (EPROM), a portable compact disk read-only memory (CD-ROM), a USB memory, and the like. The computer-readable storage medium includes a program 10 for implementing mouse and virtual key control based on an inertial measurement unit. The specific implementation of the computer-readable storage medium of the present invention is substantially the same as the specific implementation of the method for implementing mouse and virtual key control based on an inertial measurement unit and the server 1 described above, and will not be further described herein.

[0144] Although preferred embodiments of the present invention have been described, those skilled in the art may make additional changes and modifications to these embodiments once they are aware of the basic inventive concepts. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the invention. Obviously, those skilled in the art may make various changes and modifications to the present invention without departing from the spirit and scope of the invention. Thus, the present invention is intended to include such changes and modifications as fall within the scope of the claims and their equivalents.

Claims

1. A method for controlling a mouse and virtual buttons based on an inertial measurement unit, characterized in that: include: Obtain the raw 3D motion data of the inertial measurement unit and perform multi-level preprocessing to generate a standard motion feature stream containing 3D attitude parameters and 3D angular velocity parameters; Obtaining a screenshot of the currently active application, and inputting the screenshot into a pre-trained object detection deep learning model to detect virtual buttons and form screen button layout information; Determine the inferred running state of the currently active application based on the human-computer interaction context information of the previous cycle, the screen button layout information, and the predefined application state model, and obtain a valid virtual key set containing functional annotations based on the inferred running state and preset virtual key function annotation rules, and simultaneously update the human-computer interaction context information and the inferred running state of the currently active application; Determining a currently effective interaction mode based on the updated human-computer interaction context information, the updated inferred running state of the current application, and the user input instruction, wherein the interaction mode includes a mouse-first interaction mode and a virtual key-first interaction mode; According to the currently effective interaction mode, selecting a mapping algorithm corresponding to the currently effective interaction mode to process the standardized motion feature stream to generate at least one of a mouse cursor movement instruction, a mouse button instruction, and a virtual button operation instruction; Performing command fusion and collaborative event generation on the confirmed mouse cursor movement command, the mouse button command, and the virtual button operation command to obtain a unified collaborative interaction event sequence; Event conversion is performed on the unified collaborative interaction event sequence to generate a standard input event stream of the target operating system and inject the standard input event stream into the event processing queue of the target operating system.

2. The method for realizing mouse and virtual key control based on an inertial measurement unit according to claim 1, wherein: The raw 3D motion data of the inertial measurement unit is acquired and multi-level preprocessing is performed to generate a standard motion feature stream containing 3D attitude parameters and 3D angular velocity parameters, including: Applying an adaptive low-pass filter to perform preliminary filtering on the original three-dimensional motion data, wherein a cutoff frequency of the adaptive low-pass filter is dynamically adjusted according to signal energy of the original three-dimensional motion data; Performing sensor fusion processing on the raw three-dimensional motion data after preliminary filtering using a Kalman filter to generate the three-dimensional posture parameters and the three-dimensional angular velocity parameters; The three-dimensional posture parameters and the three-dimensional angular velocity parameters are normalized and offset corrected to obtain the standardized motion feature flow.

3. The method for realizing mouse and virtual key control based on an inertial measurement unit according to claim 1, wherein: Capture a screenshot of the currently active application and feed the screenshot into a pre-trained object detection deep learning model to detect virtual buttons and generate screen button layout information, including: Accurately determining the position and size of each virtual key detected by the pre-trained object detection deep learning model to obtain the bounding box position data of each virtual key; performing fine-grained category recognition on each of the detected virtual keys to obtain a category label indicating a function type and a current interaction state of the virtual key; The screen button layout information is constructed based on the bounding box position data and the category label.

4. The method for realizing mouse and virtual key control based on an inertial measurement unit according to claim 1, wherein: The inferred running state of the currently active application is determined based on the human-computer interaction context information of the previous cycle, screen button layout information, and a predefined application state model, including: Identifying the user's current interaction mode and the type of the currently active application program using the human-computer interaction context information of the previous cycle; querying the application state model for a state description that matches the current context based on the interaction mode and the type of the currently active application, in combination with the virtual key category and position in the screen key layout information; The running state of the currently active application is inferred based on the matching result.

5. The method for realizing mouse and virtual key control based on an inertial measurement unit according to claim 1, wherein: Based on the updated human-computer interaction context information, the updated inferred running state of the current application, and the user input command, the currently effective interaction mode is determined, including: The running state is inferred based on the updated human-computer interaction context information and the updated current application, combined with the user input instruction, and the currently effective interaction mode is determined according to a predefined "context-interaction mode rule library", wherein the user input instruction is a predefined mode switching gesture instruction obtained by performing specific gesture recognition processing on the standardized motion feature stream.

6. The method for realizing mouse and virtual key control based on an inertial measurement unit according to claim 1, wherein: Selecting a mapping algorithm corresponding to the currently effective interaction mode according to the currently effective interaction mode to process the standardized motion feature stream includes: If the currently effective interaction mode is the mouse-priority interaction mode, parameters are selected from the standardized motion data stream as basic cursor control inputs based on the current human-computer interaction context information, a mapping algorithm corresponding to the currently effective sensitivity mapping mode is determined and applied to process the basic cursor control inputs to generate mouse cursor movement instructions; and a first gesture recognition process is performed on the standardized motion feature stream based on a pre-built hierarchical motion primitive library to identify mouse button gestures and parse them into mouse button instructions; If the currently effective interaction mode is the virtual key priority interaction mode, a second gesture recognition process is performed on the standardized motion feature stream according to the pre-built hierarchical motion primitive library to recognize virtual key gestures and parse them into virtual key operation instructions.

7. The method for realizing mouse and virtual key control based on an inertial measurement unit according to claim 1, wherein: Performing command fusion and collaborative event generation on the confirmed mouse cursor movement command, the mouse button command, and the virtual button operation command to obtain a unified collaborative interaction event sequence, including: Determining a target virtual key under the current screen cursor position based on the current screen cursor position indicated by the mouse cursor movement instruction and the screen key layout information; and According to the currently effective interaction mode, the updated human-computer interaction context information and the updated inferred running status of the currently active application, the preset priority rules are applied to handle the conflicts and ambiguities between the mouse button instructions and the virtual button operation instructions to obtain the unified collaborative interaction event sequence.

8. The method for realizing mouse and virtual key control based on an inertial measurement unit according to claim 1, wherein: Performing event conversion on the unified collaborative interaction event sequence to generate a standard input event stream of the target operating system and injecting the stream into the event processing queue of the target operating system includes: When the mouse cursor movement instruction indicates absolute coordinate mapping, converting the target position in the mouse cursor movement instruction into normalized absolute screen coordinates and generating a corresponding standard mouse movement event; When the mouse cursor movement instruction indicates a relative displacement mapping, converting the displacement amount in the mouse cursor movement instruction into a relative movement pixel value relative to a last reported position and generating a corresponding standard mouse movement event; Converting the mouse button press instructions into corresponding standard mouse button press and release events, and converting the virtual button activation determined by the virtual button operation instructions into corresponding standard keyboard button events; Converting the standard mouse movement event, the standard mouse button press and release event, and the standard keyboard key event into a standard input event format recognizable by the target operating system; The standard input event stream is injected into the event processing queue through an event injection interface provided by the target operating system.

9. A device for realizing mouse and virtual key control based on an inertial measurement unit, characterized in that: The invention comprises a memory, a processor and a program for controlling a mouse and virtual keys based on an inertial measurement unit, which is stored in the memory and can be run on the processor. When the processor executes the program for controlling a mouse and virtual keys based on an inertial measurement unit, the method for controlling a mouse and virtual keys based on an inertial measurement unit as described in any one of claims 1 to 8 is implemented.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a program for implementing mouse and virtual key control based on an inertial measurement unit. When the program for implementing mouse and virtual key control based on an inertial measurement unit is executed by a processor, the method for implementing mouse and virtual key control based on an inertial measurement unit as described in any one of claims 1-8 is implemented.

Images