Self-adaptive mouse control method based on inertial measurement unit

Abstract

The invention discloses a self-adaptive mouse control method and device based on an inertial measurement unit and a computer readable storage medium. The method comprises the steps that basic mouse sensitivity adjusting parameters are calculated according to physical mouse sensitivity setting; obtaining original three-dimensional motion data of the inertial measurement unit to generate a standard motion feature flow; selecting parameters from the standard motion feature flow as basic cursor control input based on current human-computer interaction situation information; using the basic sensitivity parameter as an adjusting parameter, and processing the basic cursor control input by applying a preset mapping algorithm corresponding to the current effective sensitivity mapping mode to generate a mouse cursor moving instruction; and matching the standard motion feature flow with a pre-constructed hierarchical motion primitive library to obtain a mouse key instruction. The method has the advantage that the operation efficiency and the user experience of IMU-based mouse control in multiple scenes are improved.

Description

Technical Field

[0001] The present invention relates to the field of human-computer interaction technology, and in particular to an adaptive mouse control method and device based on an inertial measurement unit, and a computer-readable storage medium. Background Art

[0002] Inertial measurement units (IMUs), as sensor technology capable of capturing the three-dimensional motion and posture of objects, have shown significant potential in the field of human-computer interaction, particularly in replacing traditional mice for cursor control. Existing technologies typically use IMUs integrating accelerometers, gyroscopes, and magnetometers to acquire raw motion data. Through preprocessing and algorithms, they map the user's intended movements in three-dimensional space into cursor movements on a two-dimensional screen.

[0003] However, the current IMU-based cursor control technology faces two core challenges, which seriously restrict its performance in practical applications and user acceptance. First, the existing technology generally adopts a single sensitivity mapping mode, that is, a fixed mapping method is used to handle all operation scenarios. This "one-size-fits-all" design has significant limitations in practical applications and is difficult to meet diverse interaction needs. For example, in scenarios that require rapid positioning (such as games or large-scale interface navigation), if the absolute mapping mode is used, it may cause the cursor to respond slowly and reduce operational efficiency; conversely, in scenarios that require precise pointing (such as text editing or graphic design), if the relative mapping mode is used, the positioning accuracy may be affected due to the lack of a fixed reference. The problems of insufficient adaptability, limited control accuracy and poor user experience caused by the use of a single sensitivity mapping mode are key pain points that need to be urgently addressed in this field.

[0004] Secondly, most existing technical solutions separate the cursor movement control from the triggering operation of the mouse buttons (such as the left and right buttons), or lack the robustness and accuracy of gesture recognition, resulting in the inability to achieve natural and efficient coordinated control of the cursor movement and buttons at the same time. After using the IMU to position the cursor, users often find it difficult to perform accompanying operations such as clicking smoothly and accurately, or easily interfere with the stability of the cursor when trying to execute gesture commands, and vice versa. This incoordination between control modes makes the IMU-based mouse simulation solution far from the level of traditional mice in terms of functional completeness and operational consistency, greatly limiting its potential as a mainstream input method.

[0005] Therefore, in response to the above problems, especially the limitations of a single mapping mode and the lack of coordinated control of the cursor and buttons, the field urgently needs an improved solution that can provide multi-mode adaptive cursor control and seamlessly integrate precise gesture recognition to achieve multi-modal collaborative operation. Summary of the Invention

[0006] The embodiments of the present application provide an adaptive mouse control method based on an inertial measurement unit, aiming to provide users with an IMU mouse control solution with multi-mode adaptive cursor control and seamless integration of precise gesture recognition to achieve multi-modal collaborative operation.

[0007] To achieve the above objectives, an embodiment of the present application provides an adaptive mouse control method based on an inertial measurement unit, comprising:

[0008] Reading the physical mouse sensitivity setting of the operating system, and calculating the basic mouse sensitivity adjustment parameter according to the physical mouse sensitivity setting;

[0009] Obtain the raw 3D motion data of the inertial measurement unit and perform multi-level preprocessing to generate a standard motion feature stream containing 3D attitude parameters and 3D angular velocity parameters;

[0010] Acquiring current human-computer interaction context information, and selecting parameters from the standard motion feature stream as basic cursor control input based on the current human-computer interaction context information;

[0011] Using the basic mouse sensitivity adjustment parameter as an adjustment parameter, determining and applying a mapping algorithm corresponding to a currently effective sensitivity mapping mode to process the basic control cursor input, and generating a mouse cursor movement instruction;

[0012] Matching the standard motion feature stream with a pre-built hierarchical motion primitive library to obtain mouse button commands;

[0013] Performing command fusion and disambiguation on the mouse cursor movement command and the mouse button command to generate a mouse event sequence;

[0014] The mouse event sequence is converted into a standard mouse input event of a target operating system, and the standard mouse input event is injected into an event processing queue of the target operating system.

[0015] In one embodiment, reading a physical mouse sensitivity setting of an operating system and calculating a basic mouse sensitivity adjustment parameter based on the physical mouse sensitivity setting includes:

[0016] Obtain the physical mouse sensitivity value set by the user in the operating system through the application programming interface provided by the operating system;

[0017] According to a predefined conversion rule, the physical mouse sensitivity value is converted into a basic sensitivity parameter suitable for inertial measurement unit control.

[0018] In one embodiment, raw 3D motion data from an inertial measurement unit is obtained and multi-stage preprocessing is performed to generate a standard motion feature stream containing 3D attitude parameters and 3D angular velocity parameters, including:

[0019] Applying an adaptive low-pass filter to perform preliminary filtering on the original three-dimensional motion data, wherein a cutoff frequency of the adaptive low-pass filter is dynamically adjusted according to signal energy of the original three-dimensional motion data;

[0020] Performing sensor fusion processing on the raw three-dimensional motion data after preliminary filtering using a Kalman filter to generate the three-dimensional posture parameters and the three-dimensional angular velocity parameters;

[0021] The three-dimensional posture parameters and the three-dimensional angular velocity parameters are normalized and offset corrected to obtain the standardized motion feature flow.

[0022] In one embodiment, obtaining current human-computer interaction context information and selecting parameters from the standard motion feature stream as basic cursor control input based on the current human-computer interaction context information includes:

[0023] Determining, through an application program interface or a user interface analysis tool provided by the operating system, the type of the currently active application or the type of the user interface element under the cursor as the current human-computer interaction context information;

[0024] According to a preset context mapping table, a specific axial combination of the three-dimensional posture parameters or a specific axial combination of the three-dimensional angular velocity parameters corresponding to the type of the currently active application or the type of the user interface element under the cursor is selected as the basic cursor control input.

[0025] In one embodiment, the at least two preset sensitivity mapping modes include an absolute mapping mode for mapping the device posture to an absolute position of the screen based on the three-dimensional posture parameters, and a relative mapping mode for mapping the device angular velocity to a relative displacement or velocity of the screen based on the three-dimensional angular velocity parameters;

[0026] The method uses the basic mouse sensitivity adjustment parameter as an adjustment parameter, determines and applies a mapping algorithm corresponding to a currently effective sensitivity mapping mode to process the basic control cursor input, and generates a mouse cursor movement instruction, including:

[0027] detecting a motion amplitude or a motion speed associated with the basic cursor control input in the standard motion data stream,

[0028] When the movement amplitude or the movement speed exceeds a preset first threshold, determining that the relative mapping mode is the currently effective sensitivity mapping mode,

[0029] When the movement amplitude or the movement speed is lower than a preset second threshold, determining the absolute mapping mode as the currently effective sensitivity mapping mode;

[0030] The basic mouse sensitivity adjustment parameter is used as the gain adjustment factor of the mapping algorithm corresponding to the currently effective sensitivity mapping mode to process the basic cursor control input, and when the currently effective sensitivity mapping mode changes from the sensitivity mapping mode of the previous control cycle, a smooth transition processing based on weighted average is performed based on the current output of the preset mapping algorithm and the final output state data of the previous control cycle to generate the mouse cursor movement instruction.

[0031] In one embodiment, the standard motion feature stream is matched with a pre-built hierarchical motion primitive library to obtain a mouse button command, including:

[0032] Applying a multi-scale dynamic time window to segment the normalized motion feature stream to extract candidate gesture segments;

[0033] Using a dynamic time warping algorithm, the candidate motion segments are matched with micro-motion primitives and combined motion primitives defined in the pre-built hierarchical motion primitive library and based on the mouse button gesture template;

[0034] Confidence evaluation and validity verification based on the current human-computer interaction context information are performed on the matching result to generate the mouse button instruction, wherein the mouse button instruction includes a single-click instruction, a double-click instruction, or a long-press instruction.

[0035] In one embodiment, the mouse cursor movement instruction and the mouse button instruction are fused and disambiguated to generate a mouse event sequence, including:

[0036] Aligning and synchronizing the timestamp of the mouse cursor movement instruction and the triggering timestamp of the mouse button instruction;

[0037] When the confidence level of the mouse button instruction is higher than a preset button confirmation threshold, suppressing the mouse cursor displacement instruction caused by the gesture action within the triggering time window of the mouse button instruction;

[0038] When the mouse cursor movement instruction represents high-speed continuous movement, increasing a key confirmation threshold for confirming the mouse button instruction;

[0039] generating, based on the current human-computer interaction context information and the type of the mouse button command, a drag operation sequence including move, press, move while dragging, and release events;

[0040] The combined instruction sequence is subjected to a check including a context rationality check and a repeated command suppression check to generate the mouse event sequence.

[0041] In one embodiment, converting the mouse event sequence into a standard mouse input event of the target operating system includes:

[0042] Converting the mouse cursor movement instructions in the mouse event sequence into standard mouse movement events containing absolute screen coordinates or relative displacements according to the currently effective sensitivity mapping mode;

[0043] Converting the mouse button commands in the mouse event sequence into corresponding standard mouse button events of mouse button press and button release;

[0044] At least one multimodal feedback of tactile, auditory or visual sense is provided for the successful generation of the mouse button instruction or the switching of the sensitivity mapping mode.

[0045] To achieve the above-mentioned objectives, an embodiment of the present application also proposes an adaptive mouse control device based on an inertial measurement unit, comprising a memory, a processor, and an adaptive mouse control program based on an inertial measurement unit stored in the memory and executable on the processor. When the processor executes the adaptive mouse control program based on the inertial measurement unit, the adaptive mouse control method based on the inertial measurement unit as described in any one of the above items is implemented.

[0046] To achieve the above-mentioned objectives, an embodiment of the present application also proposes a computer-readable storage medium, on which an adaptive mouse control program based on an inertial measurement unit is stored. When the adaptive mouse control program based on an inertial measurement unit is executed by a processor, the adaptive mouse control method based on an inertial measurement unit as described in any of the above items is implemented.

[0047] The adaptive mouse control method based on the inertial measurement unit of the technical solution of the present application fundamentally solves the limitations of the single mode of the existing technology by introducing at least two preset sensitivity mapping modes and determining the technical features of the currently effective sensitivity mapping mode based on user input or system detection conditions. This means that the system is no longer limited to a fixed mapping method, but can flexibly switch between a mode of absolute position mapping based on the three-dimensional posture information of the device (suitable for precise pointing) and a mode of relative displacement mapping based on three-dimensional angular velocity information (suitable for fast, large-scale movement). This multi-mode design allows users to call the most appropriate control logic when facing operating tasks of different natures - whether fast positioning or fine manipulation is required - so that better operating effects can be obtained in various application scenarios, which directly makes up for the defects of the single absolute mapping mode pointed out in the background technology, that is, the slow response in fast scenarios and the inaccurate positioning of the single relative mapping mode in precise scenarios.

[0048] Secondly, combined with the technical features of obtaining human-computer interaction context information and selecting a specific axis combination of specified three-dimensional posture parameters or three-dimensional angular velocity parameters from the standard motion data stream as the basic cursor control input based on the context, this application makes the selection and application of sensitivity mapping modes more intelligent and targeted. The system can automatically or assist the user in selecting the basic control parameters (posture or angular velocity) that best match the context and activate the corresponding mapping mode based on the type of application currently running or the characteristics of the interface elements the user is interacting with. This contextual adaptive capability ensures that the control method can dynamically adapt to the current task requirements. For example, when making detailed depictions in drawing software, the system may tend to adopt an absolute positioning mode based on posture; when browsing wide web pages or playing games, it can switch to a relatively fast movement mode based on angular velocity, thereby significantly improving the intuitiveness and efficiency of control.

[0049] What is particularly critical is that when the sensitivity mapping mode changes, the present application applies the preset mapping algorithm corresponding to the current effective mode, and uses the basic sensitivity parameters as adjustment parameters to process the basic cursor control input, and then performs a smooth transition process based on the output of the preset mapping algorithm and the final output state data of the previous control cycle. This ensures the smoothness of mode switching and the continuity of user experience. This smooth transition mechanism can effectively eliminate or significantly reduce the sudden jumps, loss of control or response interruptions of the pointer that may be caused by changes in mapping logic or sensitivity characteristics, so that users can hardly feel the abrupt changes during the mode conversion process. This not only improves the user's willingness and comfort to switch between different modes, but also ensures the consistency and accuracy of operations in dynamic task flows, overcoming the problem that is not explicitly mentioned in the background technology but actually exists, that is, users are unwilling to take advantage of the advantages of multiple modes due to abrupt switching.

[0050] Furthermore, this application performs preliminary intent separation on the motion feature stream, specifically identifying mouse button commands based on a hierarchical library of motion primitives, and fusing and disambiguating mouse cursor movement commands and mouse button commands, thus achieving integrated and coordinated control of multiple mouse operation modes. This means that the system can not only control cursor movement, but also accurately identify and execute operations such as clicking and scrolling, and ensure the coordination of these operations through fusion and disambiguation mechanisms to avoid mutual interference.

[0051] In addition, this application calculates the basic sensitivity parameters by reading the physical mouse sensitivity settings of the operating system, providing a sensitivity starting point that conforms to the user's existing habits for all mapping modes, further improving the user-friendliness. At the same time, the original three-dimensional motion data is preprocessed to generate a high-quality standard motion data stream, providing a stable and reliable data foundation for various mapping modes and subsequent precise calculations, ensuring the accuracy of the overall control. Finally, the final instruction is converted into a standard operating system mouse event and injected into the queue, ensuring the wide compatibility and immediate availability of this method. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the structures shown in these drawings without paying any creative work.

[0053] Figure 1 This is a module structure diagram of an embodiment of an adaptive mouse control device based on an inertial measurement unit according to the present invention;

[0054] Figure 2 2 is a flow chart of an embodiment of an adaptive mouse control method based on an inertial measurement unit according to the present invention.

[0055] The purpose, features and advantages of the present invention will be further described with reference to the accompanying drawings and in conjunction with the embodiments. DETAILED DESCRIPTION

[0056] It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0057] To better understand the above technical solutions, exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although exemplary embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be limited by the embodiments described herein. Instead, these embodiments are provided to enable a more thorough understanding of the present disclosure and to fully convey the scope of the present disclosure to those skilled in the art.

[0058] It should be noted that in the claims, any reference signs placed between brackets shall not be construed as limiting the claims. The presence of "comprising" in the text does not exclude the presence of components or steps not listed in the claims. The quantifier "a" or "an" preceding a component does not exclude the presence of a plurality of such components. The present invention can be implemented by means of hardware comprising several different components and by means of a suitably programmed computer. In a unit claim that lists several means, several of these means may be embodied by the same hardware item. The use of "first", "second", and "third" etc. does not indicate any order and these words may be interpreted as names.

[0059] like Figure 1 As shown, Figure 1 It is a structural diagram of a server 1 (also called an adaptive mouse control device based on an inertial measurement unit) in a hardware operating environment involved in an embodiment of the present invention.

[0060] The server of the embodiment of the present invention is a device with display function such as "Internet of Things devices", smart air conditioners, smart lights, smart power supplies with networking functions, AR / VR devices with networking functions, smart speakers, self-driving cars, PCs, smart phones, tablet computers, e-book readers, portable computers, etc.

[0061] like Figure 1 As shown, the server 1 includes: a memory 11 , a processor 12 and a network interface 13 .

[0062] The memory 11 includes at least one type of readable storage medium, including a flash memory, a hard disk, a multimedia card, a card-type memory (e.g., an SD or DX memory), a magnetic memory, a magnetic disk, an optical disk, etc. In some embodiments, the memory 11 may be an internal storage unit of the server 1, such as a hard disk of the server 1. In other embodiments, the memory 11 may also be an external storage device of the server 1, such as a plug-in hard disk equipped on the server 1, a smart media card (SMC), a secure digital (SD) card, a flash card, etc.

[0063] Furthermore, the memory 11 may include both an internal storage unit of the server 1 and an external storage device. The memory 11 may be used not only to store application software installed on the server 1 and various data, such as the code of the adaptive mouse control program 10 based on the inertial measurement unit, but also to temporarily store data that has been output or is about to be output.

[0064] In some embodiments, the processor 12 can be a central processing unit (CPU), a controller, a microcontroller, a microprocessor, or other data processing chip, used to run the program code stored in the memory 11 or process data, such as executing the adaptive mouse control program 10 based on the inertial measurement unit.

[0065] The network interface 13 may optionally include a standard wired interface or a wireless interface (such as a WI-FI interface), and is generally used to establish a communication connection between the server 1 and other electronic devices.

[0066] The network may be the Internet, a cloud network, a wireless fidelity (Wi-Fi) network, a personal area network (PAN), a local area network (LAN), and / or a metropolitan area network (MAN). Various devices in the network environment may be configured to connect to the communication network according to various wired and wireless communication protocols. Examples of such wired and wireless communication protocols may include, but are not limited to, at least one of the following: Transmission Control Protocol and Internet Protocol (TCP / IP), User Datagram Protocol (UDP), Hypertext Transfer Protocol (HTTP), File Transfer Protocol (FTP), ZigBee, EDGE, IEEE 802.11, Light Fidelity (Li-Fi), 802.16, IEEE 802.11s, IEEE 802.11g, multi-hop communication, wireless access point (AP), device-to-device communication, cellular communication protocol, and / or Bluetooth communication protocol, or a combination thereof.

[0067] Optionally, the server may further include a user interface, which may include a display and an input unit such as a keyboard. The optional user interface may also include a standard wired interface or a wireless interface. Optionally, in some embodiments, the display may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, or an OLED (Organic Light-Emitting Diode) touchscreen. The display, which may also be referred to as a display screen or display unit, is used to display information processed in the server 1 and to display a visual user interface.

[0068] Figure 1 Only the server 1 having components 11-13 and the adaptive mouse control program 10 based on the inertial measurement unit is shown. It can be understood by those skilled in the art that Figure 1 The structure shown does not constitute a limitation on the server 1 , and the server 1 may include fewer or more components than shown in the figure, or combine certain components, or arrange the components differently.

[0069] In this embodiment, the processor 12 may be configured to call an adaptive mouse control program based on an inertial measurement unit stored in the memory 11 .

[0070] Based on the hardware architecture of the aforementioned IMU-based adaptive mouse control device, an embodiment of the present invention's IMU-based adaptive mouse control method is proposed. This IMU-based adaptive mouse control method aims to provide users with an IMU mouse control solution that enables multi-modal adaptive cursor control and seamlessly integrates precise gesture recognition to achieve multi-modal collaborative operation.

[0071] Reference Figure 2 , Figure 2 This is an embodiment of the adaptive mouse control method based on an inertial measurement unit of the present invention, and the adaptive mouse control method based on an inertial measurement unit includes the following steps:

[0072] S10. Read the physical mouse sensitivity setting of the operating system, and calculate the basic mouse sensitivity adjustment parameters based on the physical mouse sensitivity setting. This step is intended to provide a sensitivity benchmark for the cursor control based on the inertial measurement unit (IMU) that is consistent with the user's traditional mouse operation habits, thereby ensuring that the user can experience the familiar cursor movement speed and response characteristics when switching from a traditional mouse to IMU control, reducing the cost of learning and adaptation. The physical mouse sensitivity setting is usually a control parameter configured by the user in the operating system to adjust the proportional relationship between the movement of a traditional mouse and the displacement of the screen cursor. By reading this setting and converting it into parameters suitable for IMU motion data, the system can map the user's hand or device movement (such as angular velocity or posture change) to the screen cursor movement at a speed that meets the user's expectations.

[0073] In some embodiments, the above reading and calculating process of step S10 can be implemented by following steps S11 to S12:

[0074] In step S11, the physical mouse sensitivity value set by the user in the operating system is obtained through the application programming interface provided by the operating system. The operating system usually provides a specific interface function for accessing the mouse sensitivity parameters configured by the user in the system settings. For example, in the Windows system, the system obtains the mouse sensitivity value set by the user by calling the SystemParametersInfo function (using the SPI_GETMOUSESPEED parameter). The value is usually in the range of 1 to 20, and the default value is 10, indicating medium sensitivity. In the macOS system, the mouse tracking speed setting can be obtained through the IOHIDGetParameter function. The sensitivity value obtained reflects the user's preference for the cursor movement speed. For example, a higher sensitivity value means that a smaller mouse movement can produce a larger cursor displacement.

[0075] In step S12, the obtained physical mouse sensitivity value is converted into a basic sensitivity parameter suitable for inertial measurement unit control according to a predefined conversion rule. IMU control relies on the motion data of the device (such as angular velocity, in degrees per second, or attitude angle, in degrees), and the sensitivity value of the operating system is usually designed for the two-dimensional displacement of a traditional mouse, so the two need to be associated through specific conversion rules. The conversion rule usually uses a linear mapping or other mathematical formula to ensure that the IMU motion data can drive the cursor in a way that is familiar to the user. For example, the conversion formula can be defined as S_IMU=k*S_OS+b, where S_OS is the sensitivity value obtained by the operating system, k and b are constants preset according to the IMU sensor characteristics and cursor control requirements, and S_IMU is the calculated basic sensitivity parameter, which is used to subsequently scale the IMU motion data to the cursor movement distance.

[0076] Steps S11 and S12 above work together through specific logic to implement the functionality of step S10. During execution, the system first captures the user's current mouse sensitivity setting through the operating system API, ensuring that the acquired value directly reflects the user's preference in traditional mouse operation. Next, based on predefined conversion rules, the system maps this sensitivity value to parameters suitable for IMU control, taking into account the relationship between the dimension of the IMU data (such as angular velocity or attitude angle) and the screen pixel displacement. For example, assume that in a Windows system, the sensitivity value obtained through SystemParametersInfo(SPI_GETMOUSESPEED,0,&sensitivity,0) is 14, indicating that the user prefers a faster mouse response speed. In step S12, the system applies the conversion rule S_IMU = 0.04*S_OS + 0.2 to calculate S_IMU = 0.04*14 + 0.2 = 0.76. This basic sensitivity parameter S_IMU is then used to adjust the IMU motion data, for example, converting an angular velocity of 50 degrees / second into a cursor movement speed of 50*0.76=38 pixels / second, thereby maintaining an experience consistent with the user's traditional mouse operation.

[0077] For example, suppose a user uses an IMU-equipped smartwatch to control the cursor on a Windows laptop. In step S11, the system calls SystemParametersInfo(SPI_GETMOUSESPEED, 0, &sensitivity, 0) to obtain the mouse sensitivity value set by the user in the operating system's control panel. Assume it is 16, indicating a preference for faster mouse response. In step S12, the system applies the conversion rule S_IMU = 0.04 * S_OS + 0.2, resulting in S_IMU = 0.04 * 16 + 0.2 = 0.84. This base sensitivity parameter, S_IMU = 0.84, is used in subsequent IMU data processing. For example, if the IMU detects that the device is rotating at an angular velocity of 60 degrees per second about the X-axis, the cursor's speed on the screen's X-axis is 60 * 0.84 = 50.4 pixels per second. The system then applies this speed to the generated cursor movement commands, ensuring that cursor movement is consistent with the user's experience with traditional mouse operations, providing fast and intuitive responses in text editing or graphic design scenarios.

[0078] It can be understood that since the physical mouse sensitivity value set by the user is directly obtained through the application programming interface provided by the operating system, the system can accurately capture the user's personalized preference for cursor movement speed, thereby providing a sensitivity starting point that conforms to the user's habits for IMU control. Moreover, by mapping the operating system sensitivity value to parameters suitable for IMU control through predefined conversion rules, the system can effectively adapt to the characteristics of IMU motion data, such as the relationship between the dimension of angular velocity or posture change and screen pixel displacement, thereby achieving intuitive and accurate cursor movement. At the same time, this method significantly reduces the user's adaptation cost when switching to IMU control by reusing the user's existing settings in the operating system, so that the IMU-based cursor control can provide a familiar operating experience during initial use, laying the foundation for subsequent dynamic mapping and contextual adaptation.

[0079] S20. Obtain the original three-dimensional motion data of the inertial measurement unit and perform multi-level preprocessing to generate a standard motion feature stream containing three-dimensional attitude parameters and three-dimensional angular velocity parameters. This step aims to systematically process the original motion data collected by the inertial measurement unit (IMU), eliminate interference such as noise and drift, generate high-quality, standardized motion feature data, and provide reliable input for subsequent cursor control. The source of the original three-dimensional motion data can be a smart wearable device (such as a smart watch), AR / VR device or other device with integrated IMU. This application does not limit the specific type or hardware implementation of the IMU.

[0080] In some embodiments, the acquisition and preprocessing process of step S20 may be implemented by following the steps S21 to S23:

[0081] In step S21, an adaptive low-pass filter is applied to the raw three-dimensional motion data for preliminary filtering, where the cutoff frequency of the adaptive low-pass filter is dynamically adjusted based on the signal energy of the raw three-dimensional motion data. The raw data of the IMU includes three-dimensional acceleration provided by the accelerometer, three-dimensional angular velocity provided by the gyroscope, and magnetic field strength provided by the magnetometer. These data are usually collected at a high frequency (e.g., 100 Hz) but are easily interfered with by high-frequency noise. The adaptive low-pass filter improves data quality by limiting high-frequency components and retaining effective low-frequency motion signals. Its cutoff frequency is dynamically adjusted based on the signal energy to balance the denoising effect and the retention of signal details. For example, the signal energy can be determined by calculating the root mean square (RMS) value of the data over a period of time using the formula E = sqrt((1 / N) * Σ(x_i^2)), where x_i is the angular velocity or acceleration sample and N is the number of samples. The cutoff frequency f_c can be calculated using the formula f_c = k * E + f_0, where k and f_0 are preset constants that control the sensitivity and base frequency of the frequency adjustment, respectively.

[0082] In step S22, a Kalman filter is used to perform sensor fusion processing on the raw three-dimensional motion data after preliminary filtering to generate three-dimensional attitude parameters and three-dimensional angular velocity parameters. The Kalman filter comprehensively estimates the attitude and angular velocity of the device by fusing data from the accelerometer, gyroscope, and magnetometer. The accelerometer data provides information about the tilt of the device relative to the direction of gravity, the gyroscope data calculates the angular displacement by integration, and the magnetometer data corrects the heading drift. The Kalman filter uses state prediction and measurement update mechanisms to optimize the fusion of multi-sensor data and generate accurate three-dimensional attitude parameters (such as pitch angle, roll angle, and yaw angle expressed in quaternions or Euler angles) and three-dimensional angular velocity parameters (such as angular velocity around the X, Y, and Z axes). This process can effectively reduce the impact of sensor noise and drift and improve data reliability.

[0083] In step S23, the generated three-dimensional attitude parameters and three-dimensional angular velocity parameters are normalized and offset corrected to obtain a standardized motion feature stream. Normalization scales parameters of different dimensions to a uniform range (such as [0,1] or [-1,1]) to eliminate dimensional differences. For example, attitude parameters (such as Euler angles, in degrees) can be normalized by the formula θ_norm = (θ-θ_min) / (θ_max-θ_min), where θ_min and θ_max are angular ranges (such as -180° to 180°). Offset correction is used to eliminate zero-point drift of the sensor when it is stationary. For example, the gyroscope may output a non-zero angular velocity ω_offset when it is stationary, and the corrected angular velocity is ω_corrected = ω-ω_offset. The standardized motion feature stream contains the normalized and corrected attitude and angular velocity parameters, providing a consistent input format for subsequent cursor control.

[0084] For example, suppose a user uses a smartwatch equipped with an IMU to control the cursor on a Windows laptop. The IMU outputs raw 3D motion data at a sampling rate of 100 Hz, including angular velocity ω = [30, 15, 8] degrees / second and acceleration a = [0.1, 0.2, -9.7] m / s. 2 In step S21, the system calculates the angular velocity signal energy E = sqrt((30 2 +15 2 +8 2) / 3)≈19.4 degrees / second. Applying the formula f_c=0.1*E+5, we obtain a cutoff frequency f_c=0.1*19.4+5=6.94Hz. An adaptive low-pass filter removes noise above 6.94Hz and outputs the filtered data. In step S22, the Kalman filter fuses the filtered angular velocity, acceleration, and magnetometer data to generate three-dimensional attitude parameters (e.g., a quaternion representing a pitch angle of 12°, a roll angle of 6°, and a yaw angle of 2°) and an angular velocity parameter ω=[30,15,8] degrees / second. In step S23, the posture parameters are normalized. For example, a pitch angle of 12° is normalized to (12+180) / (180-(-180))=0.533. The angular velocity is corrected by an offset (e.g., ω_offset=[0.3, 0.2, 0.1] degrees / second, resulting in a correction of ω=[29.7, 14.8, 7.9] degrees / second). This generates a standard motion feature stream containing the normalized postures [0.533, 0.517, 0.506] and the corrected angular velocities [29.7, 14.8, 7.9] degrees / second. This feature stream is then used for cursor control, for example, selecting posture parameters for precise positioning in graphic design scenarios or selecting angular velocity parameters for rapid movement in gaming scenarios.

[0085] It can be understood that since the adaptive low-pass filter dynamically adjusts the cutoff frequency according to the signal energy, the system can balance the effects of denoising and signal retention under different motion intensities, thereby improving the quality and stability of the data and providing reliable input for subsequent processing. In addition, by fusing multi-sensor data through the Kalman filter, the system can integrate the information of the accelerometer, gyroscope and magnetometer to generate accurate three-dimensional posture and angular velocity parameters, overcome the limitations of single sensor noise and drift, and significantly improve the reliability of motion data. At the same time, through normalization processing and offset correction, the system eliminates the influence of parameter dimension differences and sensor zero-point drift, generates a standardized motion feature stream, ensures the consistency and portability of data in different devices or scenarios, and provides a high-quality data foundation for subsequent context-adaptive cursor control, thereby improving the accuracy and smoothness of cursor movement.

[0086] S30, obtain the current human-computer interaction context information, and select parameters from the standard motion feature stream as basic cursor control input based on the current human-computer interaction context information. This step aims to dynamically select the most suitable inertial measurement unit (IMU) motion parameter combination by sensing the user's current operating environment to optimize the performance of cursor control in different application scenarios. The standard motion feature stream contains three-dimensional posture parameters and three-dimensional angular velocity parameters, which are suitable for different interaction needs. For example, in scenarios that require precise pointing (such as text editing or graphic design), posture parameters can provide stable absolute positioning control; while in scenarios that require fast navigation (such as games or web browsing), angular velocity parameters are more suitable for supporting dynamic cursor movement. By analyzing the current human-computer interaction context, the system can select a suitable parameter combination from the standard motion feature stream as the input for cursor control, thereby improving the accuracy and adaptability of the control. The human-computer interaction context information can come from the application status or user interface element information provided by the operating system. This application does not limit the specific acquisition method or source of the context information.

[0087] In some embodiments, the above-mentioned acquisition and selection process of step S30 may be implemented by the following steps S31 to S32:

[0088] In step S31, the type of the currently active application or the type of the user interface element under the cursor is determined as the current human-computer interaction context information through the application program interface or user interface analysis tool provided by the operating system. The operating system usually provides an application program interface (API) for obtaining application information of the currently active window, such as the program name or type; the user interface analysis tool can further identify the user interface element where the cursor is currently located, such as a button, text box, or canvas area. For example, in the Windows system, the system can obtain the program name of the currently active window by calling the GetForegroundWindow and GetClassName functions (such as "notepad.exe" for a text editor and "photoshop.exe" for graphic design software). In addition, the UI Automation framework of Windows can detect the type of interface element that the cursor is hovering over, such as "button" or "canvas". This information together constitutes the human-computer interaction context information, reflecting the user's current operation context.

[0089] In step S32, according to the preset context mapping table, a specific axial combination of three-dimensional posture parameters or a specific axial combination of three-dimensional angular velocity parameters corresponding to the type of the currently active application or the type of user interface element under the cursor is selected as the basic cursor control input. The context mapping table is a predefined set of rules that corresponds different application types or interface element types to suitable IMU parameter combinations. For example, the mapping table may stipulate that for the "text editing" scenario, the pitch angle (Y axis) and yaw angle (Z axis) combination of the three-dimensional posture parameters is selected to achieve precise cursor positioning; for the "game" scenario, the X axis and Y axis combination of the three-dimensional angular velocity parameters is selected to support fast cursor movement. The selected parameter combination is extracted from the standard motion feature stream as a direct input for cursor movement, driving the cursor behavior on the screen.

[0090] For example, assuming a user uses an IMU-equipped smartwatch to control the cursor on a Windows laptop, the standard motion feature stream includes normalized three-dimensional pose parameters (pitch angle 0.533, roll angle 0.517, yaw angle 0.506) and corrected three-dimensional angular velocity parameters (X axis 29.7 degrees / second, Y axis 14.8 degrees / second, Z axis 7.9 degrees / second). In step S31, the system detects that the current active application is Adobe Photoshop (type is "graphic design") through GetForegroundWindow, and identifies that the cursor is hovering over the canvas area (element type is "canvas") through the UI Automation framework. In step S32, the system queries the preset scenario mapping table, which stipulates that the "graphic design" or "canvas" scene uses a combination of pitch and roll angles of three-dimensional pose parameters. The system then extracts normalized pose parameters [0.533, 0.517] (corresponding to pitch and roll angles) from the standard motion feature stream as basic cursor control inputs, which are then mapped to two-dimensional cursor coordinates on the screen. For example, pitch controls Y-axis movement, and roll controls X-axis movement, enabling precise canvas manipulation. If the user subsequently switches to a gaming application (such as "game.exe"), the system detects the application type as "game" and selects angular velocity parameters [29.7, 14.8] degrees / second (X and Y axes) based on the mapping table to support fast cursor movement.

[0091] It can be understood that since the type of currently active application or the type of element under the cursor is obtained through the application program interface or user interface analysis tool provided by the operating system, the system can accurately capture the context of the user's operation, thereby realizing context-aware dynamic parameter selection. In addition, by selecting a specific axial combination of three-dimensional posture or angular velocity parameters that matches the context according to the preset context mapping table, the system can optimize the cursor control input to adapt to different task requirements, such as providing stable positioning control in precise editing scenarios and supporting dynamic movement response in fast navigation scenarios. At the same time, this context-adaptive mechanism significantly improves the accuracy and efficiency of cursor control in various application scenarios by dynamically adjusting the control parameters, reduces the user's adaptation cost when switching between different scenarios, and enhances the intuitiveness and smoothness of the interaction.

[0092] S40: Using the basic mouse sensitivity adjustment parameters as adjustment parameters, determine and apply the mapping algorithm corresponding to the currently effective sensitivity mapping mode to process the basic control cursor input and generate a mouse cursor movement instruction. This step is intended to dynamically select an appropriate sensitivity mapping mode based on the dynamic characteristics of the user's operation, and, in combination with the basic mouse sensitivity adjustment parameters, convert the basic cursor control input (such as a specific axial combination of a three-dimensional posture or angular velocity parameter) into an on-screen cursor movement instruction, thereby achieving precise and smooth cursor control.

[0093] In some embodiments, at least two preset sensitivity mapping modes include an absolute mapping mode that maps the device posture to the absolute screen position based on three-dimensional posture parameters, and a relative mapping mode that maps the device angular velocity to the relative displacement or velocity of the screen based on three-dimensional angular velocity parameters. The absolute mapping mode is suitable for scenarios that require precise positioning (such as text editing or graphic design), and directly maps posture parameters (such as pitch angle and roll angle) to screen coordinates; the relative mapping mode is suitable for fast navigation scenarios (such as gaming or web browsing), and generates the relative displacement of the cursor through angular velocity parameters.

[0094] Alternatively, the determination and application process of step S40 may be implemented by following steps S41 to S44:

[0095] In step S41, the motion amplitude or motion velocity associated with the basic cursor control input in the standard motion data stream is detected. The standard motion feature stream contains normalized three-dimensional posture parameters and corrected three-dimensional angular velocity parameters, and the basic cursor control input is a specific parameter combination selected from them (such as pitch and roll angles for posture, or X-axis and Y-axis for angular velocity). The motion amplitude is determined by calculating the absolute value or variation range of the parameter. For example, for angular velocity input, the amplitude is the vector modulus A=sqrt(ω_x 2 +ω_y 2 +ω_z 2), in degrees per second. For attitude input, the amplitude is the root mean square of the angle difference. The motion speed is directly expressed as the angular velocity value or attitude change rate.

[0096] In step S42, when the motion amplitude or speed exceeds a preset first threshold, the relative mapping mode is determined to be the currently active sensitivity mapping mode. Relative mapping mode maps angular velocity parameters to relative cursor displacement and is suitable for fast, large-scale movements. A preset first threshold (e.g., 20 degrees / second) is used to determine whether the user is performing rapid movements. If this threshold is exceeded, the system assumes that the user intends to perform dynamic navigation and selects relative mapping mode to ensure rapid cursor response.

[0097] In step S43, when the motion amplitude or speed falls below a preset second threshold, absolute mapping mode is determined to be the currently active sensitivity mapping mode. Absolute mapping mode maps gesture parameters to absolute screen coordinates and is suitable for precise pointing scenarios. A preset second threshold (e.g., 10 degrees / second) is used to determine whether the user is performing slow or fine movements. If the threshold is below this, the system assumes that the user intends to perform precise manipulation and selects absolute mapping mode to provide stable positioning control.

[0098] In step S44, the basic mouse sensitivity adjustment parameter is used as a gain adjustment factor for the mapping algorithm corresponding to the currently effective sensitivity mapping mode. Basic cursor control input is processed. When the current sensitivity mapping mode changes from the sensitivity mapping mode of the previous control cycle, a weighted average smoothing process is performed based on the current output of the preset mapping algorithm and the final output state data of the previous control cycle to generate mouse cursor movement instructions. In absolute mapping mode, the posture parameters are converted to screen coordinates using a mapping algorithm (e.g., [x, y] = [S_IMU*θ_pitch*W, S_IMU*θ_roll*H], where W and H are the screen width and height). In relative mapping mode, the angular velocity parameters are converted to cursor displacement using a mapping algorithm (e.g., [Δx, Δy] = [S_IMU*ω_x*Δt, S_IMU*ω_y*Δt], where Δt is the time step). The basic mouse sensitivity adjustment parameter S_IMU is used as a gain factor to adjust the output amplitude to suit user habits. When the mode is switched, the smooth transition is performed by calculating the cursor movement instruction through weighted average: C = w(t)*C_prev+(1-w(t))*C_current, where w(t) = 1-t / T is the weight function, T is the transition time (such as 0.5 seconds), and C_prev and C_current are the outputs of the previous cycle and the current cycle respectively.

[0099] For example, assuming that a user is using an IMU-equipped smartwatch on a Windows laptop, the standard motion feature stream contains normalized 3D pose parameters [0.533, 0.517, 0.506] and corrected angular velocity parameters [29.7, 14.8, 7.9] degrees / second, and the basic cursor control input is an angular velocity of [29.7, 14.8] degrees / second (based on the game scene). The basic mouse sensitivity adjustment parameter is S_IMU = 0.84 (calculated in step S10). In step S41, the system calculates the motion amplitude A = sqrt(29.7 2 +14.8 2 )≈33.1 degrees / second. In step S42, 33.1 degrees / second exceeds the first threshold of 20 degrees / second, and the system selects relative mapping mode. In step S44, the system applies the mapping algorithm [Δx, Δy]=[0.84*29.7*0.01,0.84*14.8*0.01]≈[0.249,0.124] pixels to generate cursor displacement instructions. If the user then slowly moves the device and the movement amplitude drops to 8 degrees / second, which is lower than the second threshold of 10 degrees / second in step S43, the system switches to absolute mapping mode, processes the gesture input [0.533,0.517] to generate the coordinates [448,434] pixels. When switching, assuming that the output of the previous cycle is [500,500] pixels, the transition time is 0.5 seconds, the current time is 0.2 seconds, the weight w(0.2) = 0.6, and the cursor instruction is C = 0.6*[500,500]+0.4*[448,434]≈[483.2,487.6] pixels to ensure a smooth transition.

[0100] It can be understood that by detecting the motion amplitude or speed in the standard motion data stream, the system can capture the dynamic characteristics of the user's operation in real time, thereby providing a basis for the selection of the sensitivity mapping mode. In addition, by setting the first threshold and the second threshold to select the relative mapping mode and the absolute mapping mode respectively, the system can adapt to different operational intentions of fast navigation and precise positioning, and significantly improve the flexibility and adaptability of cursor control. At the same time, by using the basic mouse sensitivity adjustment parameter as a gain factor and performing a smooth transition process, the system ensures that the cursor movement is consistent with the user's habits and avoids jumping when switching modes, thereby improving the continuity and smoothness of the control and enhancing the user's interactive experience in different scenarios.

[0101] S50. Match the standard motion feature stream with the pre-built hierarchical motion primitive library to obtain mouse button instructions. This step aims to recognize the user's gesture actions and convert them into corresponding mouse button instructions (such as single click, double click or long press) by analyzing the standardized motion feature stream generated by the inertial measurement unit (IMU), thereby realizing somatosensory-based button control. The hierarchical motion primitive library is a predefined set of gesture templates that contains micro-motion primitives (such as short device jitter) and combined motion primitives (such as continuous jitter or specific trajectory) that describe mouse button operations. By matching the standard motion feature stream with the primitive library, the system can recognize the user's intention and generate corresponding button instructions, providing discrete control functions for IMU-based interaction. The generation of mouse button instructions depends on the accurate recognition and contextual verification of gestures to ensure the reliability of the instructions and their adaptability to the current operating scenario.

[0102] In some embodiments, the matching and generating process of step S50 may be implemented by following the steps S51 to S53:

[0103] In step S51, a multi-scale dynamic time window is applied to segment the standardized motion feature stream to extract candidate gesture segments. The standardized motion feature stream is a continuous time series that contains dynamic changes in three-dimensional posture and angular velocity parameters. The multi-scale dynamic time window analyzes the feature stream within different time spans (such as 0.1 seconds, 0.3 seconds, and 0.5 seconds) to segment out segments that may correspond to gestures. The size of the time window is adaptively adjusted according to the dynamic characteristics of the signal. For example, when a sharp fluctuation in angular velocity or posture changes is detected, a shorter window is used to capture fast movements; when the signal is relatively stable, a longer window is used to identify continuous movements. The segmented candidate gesture segments are a subset of the time series and contain potential gesture features. For example, a short angular velocity peak may correspond to a single click, and a continuous peak may correspond to a double click.

[0104] In step S52, the dynamic time warping (DTW) algorithm is used to match the candidate gesture segments with the micro-motion primitives and combined motion primitives based on the mouse button gesture template defined in the pre-built hierarchical motion primitive library. The hierarchical motion primitive library contains two types of templates: micro-motion primitives (such as a single fast shake, indicating a single click) and combined motion primitives (such as two fast shakes, indicating a double click, or a continuous posture maintenance, indicating a long press). The DTW algorithm allows nonlinear alignment on the time axis by calculating the time series similarity between the candidate gesture segment and the primitive template to adapt to individual differences in user movement speed and rhythm. For example, DTW can determine the most matching primitive template by minimizing the Euclidean distance between the segment and the template, thereby identifying the corresponding gesture type.

[0105] In step S53, the matching results are subjected to confidence assessment and validity verification based on the current human-computer interaction context information to generate mouse button instructions, where the mouse button instructions include single-click instructions, double-click instructions, or long-press instructions. The confidence assessment determines the degree of matching between the candidate gesture segment and the primitive template by calculating the similarity score of the DTW matching. For example, a score higher than a preset threshold (such as 0.9) indicates a high-confidence match. The validity verification combines the current human-computer interaction context information (such as the type of active application or the interface element under the cursor) to ensure that the recognized gesture is consistent with the operation scenario. For example, in a text editing scenario, a single-click gesture may be to select text, while in a game scenario, it may be to trigger an action. After verification, the system generates the corresponding mouse button instructions, such as a single-click instruction corresponds to a single press of the left mouse button, a double-click instruction corresponds to two quick presses, and a long-press instruction corresponds to a continuous press.

[0106] The above steps S51 to S53 work together in a specific logical order to achieve the function of step S50. First, the system segments the standard motion feature stream through a multi-scale dynamic time window, extracts segments that may contain gestures, and ensures that motion features at different time scales are captured. Next, the DTW algorithm matches these segments with templates in the hierarchical motion primitive library to identify gesture patterns corresponding to operations such as single-click, double-click, or long press. The system then filters out matching results with low confidence or that do not match the current scene through confidence evaluation and context verification, and finally generates reliable mouse button instructions. For example, assuming that the user performs a fast jitter gesture in graphic design software, the system first segments the angular velocity peak segment, then matches it to the single-click template through DTW, and finally generates a single-click instruction combined with canvas context verification to select graphic elements.

[0107] For example, assume a user is using an IMU-equipped smartwatch on a Windows laptop. The standard motion feature stream contains normalized 3D pose parameters of [0.533, 0.517, 0.506] and calibrated angular velocity parameters of [29.7, 14.8, 7.9] degrees / second. In step S51, the system detects two rapid angular velocity peaks within 0.3 seconds (e.g., [60, 30, 15] degrees / second and [55, 25, 12] degrees / second) and segments a candidate gesture segment using a 0.3-second time window. In step S52, the DTW algorithm matches this segment with a double-click template (defined as two rapid angular velocity peaks separated by less than 0.5 seconds) from the hierarchical motion primitive library, calculating a similarity score of 0.95. In step S53, the current human-computer interaction context is "File Explorer" ("explorer.exe" detected via GetForegroundWindow), and the cursor is hovering over a file icon (identified as "icon" by the UIAutomation framework). If the confidence level of 0.95 is higher than the threshold of 0.9 and the double-click gesture is compatible with the file open operation, the system generates a double-click command to trigger the file open operation. If the user maintains the gesture for 0.8 seconds in the same context (e.g., the gesture parameters are stable within [0.535, 0.518, 0.507]), the system segments the long press segment, matches it to the long press template (score 0.93), and after verification, generates a long press command to trigger the right-click menu.

[0108] It can be understood that due to the application of multi-scale dynamic time windows to segment the standard motion feature stream, the system can flexibly capture gesture features at different time scales, thereby improving the adaptability and accuracy of gesture recognition. In addition, by matching the dynamic time warping algorithm with the hierarchical motion primitive library, the system can tolerate differences in user movement speed and rhythm and accurately identify gesture patterns such as single-click, double-click, or long press. At the same time, through confidence evaluation and effectiveness verification based on the current human-computer interaction scenario, the system ensures that the generated gesture commands are both reliable and match the operation scenario, thereby significantly reducing the misrecognition rate and improving the robustness of IMU-based button control in complex interaction scenarios and the naturalness of the user experience.

[0109] S60. Perform command fusion and disambiguation on the mouse cursor movement command and the mouse button command to generate a mouse event sequence. This step aims to integrate and optimize the cursor movement command and button command generated based on the inertial measurement unit to generate an event sequence that meets the operating system standards to achieve smooth and accurate mouse interaction. Since the user's actions in IMU control may trigger movement and button commands at the same time, for example, gestures may cause unexpected displacement of the cursor, command fusion and disambiguation ensure that the generated mouse event sequence reflects the user's intention and avoids conflicts or redundancies through time alignment, misoperation suppression and situational verification. The mouse event sequence includes events such as movement, pressing, moving while dragging, and releasing, which can adapt to a variety of interaction scenarios, such as text selection, file drag and drop, or graphic editing.

[0110] In some embodiments, the above fusion and disambiguation process of step S60 may be implemented by following the steps S61 to S65:

[0111] In step S61, the timestamp of the mouse cursor movement instruction and the trigger timestamp of the mouse button instruction are aligned and synchronized. The cursor movement instruction and the button instruction are generated by the above steps respectively, and each has a timestamp reflecting the moment of its generation or triggering. Time alignment ensures their temporal correspondence by mapping the timestamps of the two types of instructions to a unified time axis. For example, the system records the instruction timestamps in milliseconds. If the movement instruction generates the coordinates [500,500] at t=1000ms and the button instruction generates a click event at t=1010ms, the system aligns the two through a time window (such as ±20ms) to confirm that they belong to the same interactive action. The synchronization process ensures that the instructions are processed in chronological order to avoid event misordering due to processing delays.

[0112] In step S62, when the confidence level of the mouse button command is higher than the preset button confirmation threshold, the mouse cursor displacement command caused by the gesture action within the button command trigger time window is suppressed. The confidence level of the button command comes from the matching score of the gesture recognition (such as the similarity score of the dynamic time warping algorithm), and the preset button confirmation threshold (such as 0.9) is used to judge the reliability of the gesture. If the confidence level is higher than the threshold, it indicates that the button command (such as a single click) is the user's clear intention. The system suppresses accidental cursor displacement that may be caused by gesture actions (such as jitter) within the trigger time window (such as ±50ms). For example, executing a single-click gesture may cause a brief fluctuation in angular velocity, generating small displacement commands. The system prevents the cursor from jittering when the button is triggered by suppressing these displacement commands.

[0113] In step S63, when the mouse cursor movement instruction represents high-speed continuous movement, the key confirmation threshold for confirming the mouse button instruction is increased. High-speed continuous movement is characterized by the displacement speed or frequency of the cursor movement instruction. For example, in the relative mapping mode, the displacement speed exceeds a certain threshold (such as 50 pixels / second) or the frequency of the continuous movement instruction is higher than a certain value (such as 10 times / second). In this case, the system increases the key confirmation threshold (such as from 0.9 to 0.95) to reduce the number of key instructions that are misidentified during rapid movement. For example, when moving the cursor quickly in a game scene, the user may unintentionally trigger an action similar to a click. By increasing the threshold, the system ensures that only clear gestures are recognized as key instructions.

[0114] In step S64, based on the current human-computer interaction context information and the type of mouse button instruction, a drag operation sequence including move, press, move while dragging, and release events is generated. The current human-computer interaction context information (such as application type or interface element) is obtained by the above steps and is used to guide the instruction combination logic. For example, in a file manager, a long press instruction combined with a continuous move instruction can generate a drag operation sequence: press (long press trigger) → move while dragging (continuous cursor displacement) → release (long press ends). The system generates an event sequence that conforms to the operation logic based on the context (such as "file icon") and button type (such as long press) to ensure that it is consistent with the user's intention. Figure 1 To.

[0115] In step S65, the combined instruction sequence is subjected to a check including a context rationality check and a repeated command suppression to generate a mouse event sequence. The context rationality check checks whether the instruction sequence conforms to the current context. For example, in a text editor, the drag sequence should be compatible with the text selection operation. Repeated command suppression merges or eliminates redundant instructions by detecting repeated instructions (such as multiple clicks) within a time window (such as 100ms). For example, if the system detects that two click instructions are generated within 0.1 seconds and the confidence levels are close, they are merged into a single click event. The mouse event sequence finally generated includes standard events such as moving, pressing, dragging, and releasing, which conform to the input format of the operating system.

[0116] The above steps S61 to S65 work together in a specific logical order to achieve the function of step S60. First, the system ensures that the cursor movement instructions and key instructions are correctly associated on the timeline through timestamp alignment and synchronization to form a unified instruction stream. Then, the confidence threshold and time window are used to suppress accidental displacement caused by gestures and improve the reliability of key instructions. Then, the key confirmation threshold is increased in high-speed movement scenarios to reduce false triggers. Subsequently, the system combines the human-computer interaction scenario and key type to generate an instruction sequence containing drag logic. Finally, through context verification and repeated command suppression, the system optimizes the instruction sequence to ensure that its logic is reasonable and without redundancy. For example, in the file drag and drop scenario, the system aligns long press and move instructions, suppresses accidental displacement, generates a drag sequence, verifies its rationality in the file manager context, and finally outputs a standard event sequence.

[0117] For example, assume a user is using an IMU-equipped smartwatch on a Windows laptop. The standard motion feature stream generates cursor movement instructions (coordinates [500, 500], timestamp t = 1000ms; displacement [0.249, 0.124] pixels, t = 1010ms) and key commands (long press, confidence 0.93, t = 1005ms). The current context is "file manager" ("explorer.exe", cursor on the "file icon"). In step S61, the system aligns the timestamps and confirms that the movement and long press instructions are related within ± 20ms. In step S62, the long press confidence of 0.93 is higher than the threshold of 0.9, and the system suppresses displacement instructions within t = 1005ms ± 50ms (such as [0.249, 0.124]). In step S63, the movement speed is less than 50 pixels / second, and the threshold is maintained at 0.9. In step S64, the system generates a drag sequence based on the "File Manager" context and the long-press command: press (t = 1005ms) → move during drag (coordinates [500, 500] to [510, 510]) → release (t = 1500ms). In step S65, the context check confirms that the drag sequence is suitable for file drag and drop, and repeated command suppression removes redundant movement instructions within 0.1 seconds, generating the final mouse event sequence.

[0118] It can be understood that due to the alignment and synchronization of timestamps for cursor movement instructions and key instructions, the system can ensure the temporal coordination of the two instructions, thus providing a reliable basis for fusion. Moreover, by suppressing accidental displacements caused by gestures during high-confidence key instructions, the system effectively reduces cursor jitter caused by gestures and enhances the stability of key operations. At the same time, by increasing the key confirmation threshold during high-speed movement, the system reduces the risk of false triggering and improves the robustness of fast interaction scenarios. In addition, by combining the human-computer interaction scenario to generate a drag operation sequence and perform context verification and repeated command suppression, the system ensures a high degree of match between the mouse event sequence and the user's intention and operation scenario, thereby significantly improving the accuracy and smoothness of IMU-based interactions in complex scenarios.

[0119] S70: Convert the mouse event sequence into standard mouse input events of the target operating system and inject the standard mouse input events into the event processing queue of the target operating system. This step aims to convert the mouse event sequence generated in the previous step (including cursor movement instructions and key commands) into standard input events that the target operating system can recognize and execute, ensuring that the inertial measurement unit (IMU)-based cursor control and key operations are seamlessly integrated with the native input mechanism of the operating system.

[0120] In some embodiments, the conversion and injection process of step S70 may be implemented by following the steps S71 to S73:

[0121] In step S71, the mouse cursor movement instructions in the mouse event sequence are converted into standard mouse movement events containing absolute screen coordinates or relative displacements according to the currently effective sensitivity mapping mode. The mouse cursor movement instructions are derived from the aforementioned steps, and absolute coordinates (such as [x, y] pixels) or relative displacements (such as [Δx, Δy] pixels) are generated according to the current sensitivity mapping mode (absolute mapping mode or relative mapping mode). In absolute mapping mode, the movement instructions are directly converted into absolute positioning events of the target operating system. For example, in Windows systems, the MOUSEEVENTF_ABSOLUTE event is injected through the SendInput function, and the coordinates need to be normalized to the range of 0-65535 to adapt to the system resolution (such as a screen of 1000x1000 pixels, and the coordinates [500, 500] are mapped to [500 / 100065535, 500 / 100065535]≈[32768, 32768]). In relative mapping mode, movement instructions are converted into relative displacement events, such as directly passing the displacement amount (such as [0.249, 0.124] pixels) through the MOUSEEVENTF_MOVE event. The converted standard mouse movement events are compatible with the input format of the operating system, ensuring that the cursor movement is executed accurately.

[0122] In step S72, the mouse button instructions in the mouse event sequence are converted into standard mouse button events of corresponding mouse button presses and button releases. Key instructions (such as single-click, double-click, and long press) are generated by gesture recognition and need to be converted into standard key events of the operating system. For example, in the Windows system, a single-click instruction is converted into a pair of MOUSEEVENTF_LEFTDOWN (press) and MOUSEEVENTF_LEFTUP (release) events, a double-click instruction generates two consecutive press and release events, and a long press instruction generates a continuous press event until the release triggers. The system generates a corresponding standard event sequence based on the timestamp and type of the key instruction (such as a single trigger of a single click or the duration of a long press) to ensure that it is consistent with the user's gesture intention. Figure 1 To.

[0123] In step S73, at least one multimodal feedback of tactile, auditory or visual is provided for the successful generation of the mouse button instruction or the switching of the sensitivity mapping mode. The successful generation of the button instruction or the switching of the sensitivity mapping mode (absolute or relative mode) may change the interactive behavior, and the user needs to perceive these changes to maintain the continuity of the operation. Tactile feedback is achieved through the vibration motor of the device (such as a smart watch), for example, triggering a short vibration of 0.2 seconds; auditory feedback is achieved through system sound effects, such as playing a "beep" sound; visual feedback is achieved through screen prompts, such as the cursor briefly flashing for 0.5 seconds. These feedbacks are implemented through the operating system API (such as the Windows PlaySound function) or the device driver interface to enhance the user's awareness of the interactive status.

[0124] The above steps S71 to S73 work together in a specific logical order to achieve the function of step S70. First, the system converts the cursor movement instruction into the standard movement event of the operating system according to the current sensitivity mapping mode, ensuring that the cursor behavior is consistent with the user's intention. Figure 1 Next, the system converts the key commands into standard key events, accurately reflecting operations such as single-click, double-click, or long press. The system then prompts the user of the generation of key commands or mode switching through multimodal feedback to improve the intuitiveness of the interaction. Finally, the converted standard mouse input events are injected into the event processing queue of the operating system, processed and executed by the system. For example, in the file drag and drop scenario, the system converts the movement and key commands of the drag sequence into standard events, which are injected into the queue to trigger the file movement, and at the same time prompts the user through vibration feedback that the long press operation has been triggered.

[0125] For example, suppose a user is using an IMU-equipped smartwatch on a Windows laptop. The mouse event sequence includes a move command (relative displacement [0.249, 0.124] pixels, timestamp t = 1010ms) and a key command (long press, timestamp t = 1005ms, lasting until t = 1500ms). The current context is "File Manager" (the cursor is on the "file icon"). In step S71, the system recognizes that the current mapping mode is relative and converts the displacement [0.249, 0.124] into a MOUSEEVENTF_MOVE event with parameters [0.249, 0.124]. In step S72, the long press command is converted into a MOUSEEVENTF_LEFTDOWN event (t = 1005ms) and a MOUSEEVENTF_LEFTUP event (t = 1500ms), forming the press and release events of the drag operation. In step S73, after the long-press command is generated, the system triggers the smartwatch to vibrate for 0.2 seconds (via the device driver API), play a beeping sound (via PlaySound), and flash the cursor for 0.5 seconds (via the graphics API). These standard events are injected into the Windows event queue. The operating system updates the cursor position and executes the file drag operation, providing feedback to the user indicating that the operation was successfully triggered.

[0126] It can be understood that by converting the mouse event sequence into standard mouse input events of the target operating system, the system ensures seamless compatibility between IMU control and the operating system, thereby achieving accurate execution of cursor movement and key operations. Moreover, by converting cursor movement instructions and key commands into standard movement events and key events respectively, the system can accurately reflect the user's somatosensory operation intentions and adapt to a variety of interaction scenarios. At the same time, by providing tactile, auditory or visual multimodal feedback for key command generation or sensitivity mode switching, the system enhances the user's perception of changes in interaction status, thereby reducing cognitive load and improving the intuitiveness and smoothness of IMU-based mouse control in complex scenarios.

[0127] In addition, embodiments of the present invention further provide a computer-readable storage medium. The computer-readable storage medium can be any one of, or any combination of, a hard disk, a multimedia card, an SD card, a flash memory card, an SMC, a read-only memory (ROM), an erasable programmable read-only memory (EPROM), a portable compact disk read-only memory (CD-ROM), or a USB memory device. The computer-readable storage medium includes an inertial measurement unit-based adaptive mouse control program 10. The specific implementation of the computer-readable storage medium of the present invention is substantially the same as the specific implementation of the inertial measurement unit-based adaptive mouse control method and server 1 described above, and will not be further described herein.

[0128] It will be understood by those skilled in the art that embodiments of the present invention may be provided as methods, systems, or computer program products. Thus, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0129] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowcharts and / or block diagrams, as well as combinations of processes and / or blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowcharts and / or block diagrams. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0130] Obviously, those skilled in the art may make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if such changes and modifications fall within the scope of the claims and their equivalents, the present invention is intended to include such changes and modifications.

Claims

1. An adaptive mouse control method based on an inertial measurement unit, characterized in that: include: Reading the physical mouse sensitivity setting of the operating system, and calculating the basic mouse sensitivity adjustment parameter according to the physical mouse sensitivity setting; Obtain the raw 3D motion data of the inertial measurement unit and perform multi-level preprocessing to generate a standard motion feature stream containing 3D attitude parameters and 3D angular velocity parameters; Acquiring current human-computer interaction context information, and selecting parameters from the standard motion feature stream as basic cursor control input based on the current human-computer interaction context information; Using the basic mouse sensitivity adjustment parameter as an adjustment parameter, determining and applying a mapping algorithm corresponding to a currently effective sensitivity mapping mode to process the basic control cursor input, and generating a mouse cursor movement instruction; Matching the standard motion feature stream with a pre-built hierarchical motion primitive library to obtain mouse button commands; Performing command fusion and disambiguation on the mouse cursor movement command and the mouse button command to generate a mouse event sequence; The mouse event sequence is converted into a standard mouse input event of a target operating system, and the standard mouse input event is injected into an event processing queue of the target operating system.

2. The adaptive mouse control method based on an inertial measurement unit according to claim 1, wherein: Reading the physical mouse sensitivity setting of the operating system and calculating the basic mouse sensitivity adjustment parameter according to the physical mouse sensitivity setting, including: Obtain the physical mouse sensitivity value set by the user in the operating system through the application programming interface provided by the operating system; According to a predefined conversion rule, the physical mouse sensitivity value is converted into a basic sensitivity parameter suitable for inertial measurement unit control.

3. The adaptive mouse control method based on inertial measurement unit according to claim 1, wherein: The raw 3D motion data of the inertial measurement unit is acquired and multi-level preprocessing is performed to generate a standard motion feature stream containing 3D attitude parameters and 3D angular velocity parameters, including: Applying an adaptive low-pass filter to perform preliminary filtering on the original three-dimensional motion data, wherein a cutoff frequency of the adaptive low-pass filter is dynamically adjusted according to signal energy of the original three-dimensional motion data; Performing sensor fusion processing on the raw three-dimensional motion data after preliminary filtering using a Kalman filter to generate the three-dimensional posture parameters and the three-dimensional angular velocity parameters; The three-dimensional posture parameters and the three-dimensional angular velocity parameters are normalized and offset corrected to obtain the standardized motion feature flow.

4. The adaptive mouse control method based on an inertial measurement unit according to claim 1, wherein: Acquiring current human-computer interaction context information, and selecting parameters from the standard motion feature stream as basic cursor control input based on the current human-computer interaction context information, including: Determining, through an application program interface or a user interface analysis tool provided by the operating system, the type of the currently active application or the type of the user interface element under the cursor as the current human-computer interaction context information; According to a preset context mapping table, a specific axial combination of the three-dimensional posture parameters or a specific axial combination of the three-dimensional angular velocity parameters corresponding to the type of the currently active application or the type of the user interface element under the cursor is selected as the basic cursor control input.

5. The adaptive mouse control method based on inertial measurement unit according to claim 1, wherein: The at least two preset sensitivity mapping modes include an absolute mapping mode for mapping the device posture to an absolute position of the screen based on the three-dimensional posture parameters, and a relative mapping mode for mapping the device angular velocity to a relative displacement or velocity of the screen based on the three-dimensional angular velocity parameters; The method uses the basic mouse sensitivity adjustment parameter as an adjustment parameter, determines and applies a mapping algorithm corresponding to a currently effective sensitivity mapping mode to process the basic control cursor input, and generates a mouse cursor movement instruction, including: detecting a motion amplitude or a motion speed associated with the basic cursor control input in the standard motion data stream, When the movement amplitude or the movement speed exceeds a preset first threshold, determining that the relative mapping mode is the currently effective sensitivity mapping mode, When the movement amplitude or the movement speed is lower than a preset second threshold, determining the absolute mapping mode as the currently effective sensitivity mapping mode; The basic mouse sensitivity adjustment parameter is used as the gain adjustment factor of the mapping algorithm corresponding to the currently effective sensitivity mapping mode to process the basic cursor control input, and when the currently effective sensitivity mapping mode changes from the sensitivity mapping mode of the previous control cycle, a smooth transition processing based on weighted average is performed based on the current output of the preset mapping algorithm and the final output state data of the previous control cycle to generate the mouse cursor movement instruction.

6. The adaptive mouse control method based on an inertial measurement unit according to claim 1, wherein: The standard motion feature stream is matched with a pre-built hierarchical motion primitive library to obtain mouse button commands, including: Applying a multi-scale dynamic time window to segment the normalized motion feature stream to extract candidate gesture segments; Using a dynamic time warping algorithm, the candidate motion segments are matched with micro-motion primitives and combined motion primitives defined in the pre-built hierarchical motion primitive library and based on the mouse button gesture template; Confidence evaluation and validity verification based on the current human-computer interaction context information are performed on the matching result to generate the mouse button instruction, wherein the mouse button instruction includes a single-click instruction, a double-click instruction, or a long-press instruction.

7. The adaptive mouse control method based on an inertial measurement unit according to claim 1, wherein: Performing command fusion and disambiguation on the mouse cursor movement command and the mouse button command to generate a mouse event sequence, including: Aligning and synchronizing the timestamp of the mouse cursor movement instruction and the triggering timestamp of the mouse button instruction; When the confidence level of the mouse button instruction is higher than a preset button confirmation threshold, suppressing the mouse cursor displacement instruction caused by the gesture action within the triggering time window of the mouse button instruction; When the mouse cursor movement instruction represents high-speed continuous movement, increasing a key confirmation threshold for confirming the mouse button instruction; generating, based on the current human-computer interaction context information and the type of the mouse button command, a drag operation sequence including move, press, move while dragging, and release events; The combined instruction sequence is subjected to a check including a context rationality check and a repeated command suppression check to generate the mouse event sequence.

8. The adaptive mouse control method based on an inertial measurement unit according to claim 1, wherein: Convert the mouse event sequence into standard mouse input events of the target operating system, including: Converting the mouse cursor movement instructions in the mouse event sequence into standard mouse movement events containing absolute screen coordinates or relative displacements according to the currently effective sensitivity mapping mode; Converting the mouse button commands in the mouse event sequence into corresponding standard mouse button events of mouse button press and button release; At least one multimodal feedback of tactile, auditory or visual sense is provided for the successful generation of the mouse button instruction or the switching of the sensitivity mapping mode.

9. An adaptive mouse control device based on an inertial measurement unit, characterized in that: The invention comprises a memory, a processor and an adaptive mouse control program based on an inertial measurement unit stored in the memory and executable on the processor, wherein when the processor executes the adaptive mouse control program based on the inertial measurement unit, the adaptive mouse control method based on the inertial measurement unit as described in any one of claims 1 to 8 is implemented.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium stores an adaptive mouse control program based on an inertial measurement unit. When the adaptive mouse control program based on an inertial measurement unit is executed by a processor, the adaptive mouse control method based on an inertial measurement unit as described in any one of claims 1 to 8 is implemented.

Images