Multi-mode mouse control method based on inertial measurement unit

Abstract

The invention discloses a multi-mode mouse control method and equipment based on an inertial measurement unit, and a computer readable storage medium. The method comprises the following steps: acquiring original three-dimensional motion data of the inertial measurement unit to generate a standard motion feature flow; performing multi-scale dynamic time window segmentation on the standardized motion feature flow, and matching the segmented standardized motion feature flow with a template of a pre-constructed hierarchical motion primitive library to obtain a preliminary primitive sequence; analyzing the preliminary primitive sequence to recognize the occurrence or ending state of a mode switching gesture, and determining a current control mode according to the recognition result of the mode switching gesture; according to the current control mode, selecting a corresponding mapping algorithm to process the standardized motion feature flow, and generating a mouse cursor movement instruction or a mouse wheel instruction; and analyzing a part which is not identified as a mode switching gesture in the initial primitive sequence, and analyzing the part as a mouse key instruction. The method has the advantage that the operation efficiency and the user experience of IMU-based mouse control in multiple scenes are improved.

Description

Technical Field

[0001] The present invention relates to the field of human-computer interaction technology, and in particular to a multimodal mouse control method and device based on an inertial measurement unit, and a computer-readable storage medium. Background Art

[0002] As a sensor technology capable of capturing the three-dimensional motion and posture of an object, the inertial measurement unit (IMU) shows significant potential in the field of human-computer interaction, particularly in replacing traditional mice for cursor control. Existing technologies typically use IMUs that integrate accelerometers, gyroscopes, and sometimes magnetometers to acquire raw motion data. Through preprocessing and algorithms, these IMUs map the user's intended movements in three-dimensional space into cursor movements on a two-dimensional screen.

[0003] However, current IMU-based cursor control technology faces two core challenges, which seriously restrict its performance in practical applications and user acceptance.

[0004] First, existing technologies generally use a single sensitivity mapping mode or fixed control logic to handle all operating scenarios. This "one-size-fits-all" design has significant limitations in practical applications and is difficult to meet diverse interaction needs. For example, users may expect to use a fast-responding relative mapping mode when navigating a large-scale interface, and switch to a more stable and accurate absolute mapping mode or a low-sensitivity relative mode when performing detailed graphic editing or text selection. Existing solutions often lack the flexibility to dynamically adjust mapping strategies based on task requirements or user preferences, resulting in slow cursor response and low operational efficiency in some scenarios, or difficulty in precise positioning in other scenarios, affecting operational accuracy. The problems of limited control accuracy and poor user experience caused by the single control mode and insufficient adaptability are key pain points that need to be urgently addressed in this field.

[0005] Secondly, most existing solutions separate cursor movement control from triggering other core mouse functions (such as button clicks and wheel scrolling), or lack robustness and accuracy in gesture recognition, resulting in an inability to achieve natural and efficient coordinated control of multimodal operations. After using an IMU to locate the cursor, users often struggle to smoothly and accurately perform accompanying actions such as clicking, double-clicking, dragging, or scrolling. When attempting to execute these discrete commands through specific gestures, the gestures themselves can easily disrupt cursor stability, causing pointer drift or misoperation. Conversely, when focusing on cursor movement, subtle, unconscious movements can be mistakenly recognized as gesture commands. Furthermore, existing solutions lack a mature and user-friendly solution for implementing continuous and directional operations such as mouse scrolling through a limited set of easy-to-learn IMU gestures. This lack of coordination and conflict between different control modes—cursor movement, button clicks, and wheel scrolling—as well as the limited expressiveness of gesture command sets, mean that IMU-based mouse emulation solutions fall far short of the functional completeness and operational consistency of traditional mice, significantly limiting their potential as a mainstream input method.

[0006] Therefore, in response to the above-mentioned problems, especially the limitations of a single mapping mode, the lack of coordinated control of multimodal operations (cursor movement, button clicks, and scrolling), and the limitations of gesture interaction, the field urgently needs an improved mouse control solution that can provide multimodal adaptive cursor and scroll wheel control and can seamlessly integrate accurate and reliable gesture recognition to achieve multimodal collaborative operations. Summary of the Invention

[0007] The embodiments of the present application provide a multimodal mouse control method based on an inertial measurement unit, aiming to provide an improved mouse control solution that provides multimodal adaptive cursor and scroll wheel control and can seamlessly integrate accurate and reliable gesture recognition to achieve multimodal collaborative operation.

[0008] To achieve the above objectives, an embodiment of the present application provides a multimodal mouse control method based on an inertial measurement unit, comprising:

[0009] Obtain the raw 3D motion data of the inertial measurement unit and perform multi-level preprocessing to generate a standard motion feature stream containing 3D attitude parameters and 3D angular velocity parameters;

[0010] Applying multi-scale dynamic time window segmentation to the standardized motion feature stream to extract candidate gesture segments, and matching the candidate gesture segments with templates of a pre-constructed hierarchical motion primitive library to obtain a preliminary primitive sequence;

[0011] Analyzing the preliminary primitive sequence according to the definition of the mode switching gesture in the pre-built hierarchical motion primitive library to identify the occurrence or end state of the mode switching gesture, and determining whether the current control mode is the cursor control mode or the scroll wheel control mode according to the recognition result of the mode switching gesture;

[0012] According to the current control mode, a corresponding mapping algorithm is selected to process the standardized motion feature stream to generate a mouse cursor movement instruction or a mouse wheel instruction;

[0013] Analyzing the portion of the preliminary primitive sequence that is not recognized as the mode switching gesture according to the definition of the mouse button gesture in the pre-built hierarchical motion primitive library to recognize the mouse button gesture and parse it into a mouse button command;

[0014] combining the identified mouse cursor movement instructions, the mouse button instructions, and the mouse wheel instructions to generate a mouse event sequence;

[0015] The mouse event sequence is converted into a standard mouse input event of a target operating system, and the standard mouse input event is injected into an event processing queue of the target operating system.

[0016] To achieve the above-mentioned objectives, an embodiment of the present application also proposes a multimodal mouse control device based on an inertial measurement unit, comprising a memory, a processor, and a multimodal mouse control program based on an inertial measurement unit stored in the memory and executable on the processor. When the processor executes the multimodal mouse control program based on the inertial measurement unit, the multimodal mouse control method based on the inertial measurement unit as described in any one of the above items is implemented.

[0017] To achieve the above-mentioned objectives, an embodiment of the present application further proposes a computer-readable storage medium, on which a multimodal mouse control program based on an inertial measurement unit is stored. When the multimodal mouse control program based on an inertial measurement unit is executed by a processor, the multimodal mouse control method based on an inertial measurement unit as described in any one of the above items is implemented.

[0018] The inertial measurement unit-based mouse control method of the technical solution of the present application explicitly determines whether the current control mode is the cursor control mode or the scroll wheel control mode by introducing a specific mode switching gesture (such as a "fist" gesture) performed by the user, which fundamentally solves the pain points in the prior art that the cursor movement and the scroll wheel scrolling operation are easily confused and the control intention is unclear due to the similarity of gestures. This means that the system no longer relies solely on subtle differences in motion characteristics to passively infer user intentions, but empowers the user to define how subsequent IMU data should be parsed through a clear and active action. When the user does not perform a mode switching gesture (for example, the hand is in a natural and relaxed state), the system defaults to the cursor control mode; when the user performs a "fist" gesture and holds it, the system switches to the scroll wheel control mode. This design allows users to intuitively and reliably separate the two operation intentions, greatly improving the control accuracy and user confidence when cursor pointing and content scrolling are required at the same time.

[0019] Secondly, the technical solution of the present application selects and applies a specific mapping algorithm corresponding to the determined cursor control mode or scroll wheel control mode to process the same standardized multi-dimensional motion feature stream to generate mouse cursor movement instructions or mouse scroll wheel instructions respectively. This technical feature ensures that even if the user performs hand movements with similar physical forms in different modes, the system can correctly interpret them as two-dimensional plane movement of the cursor (in cursor control mode) or unidirectional scrolling of the scroll wheel (in scroll wheel control mode) based on the currently activated mode state. For example, in cursor mode, slight pitch and deflection of the wrist will be converted by the cursor mapping algorithm into smooth movement of the pointer on the screen; while in the scroll wheel mode activated by "clenching a fist", similar up and down swings of the wrist (if defined as scroll wheel gestures) will be specifically interpreted by the scroll wheel mapping algorithm as up and down scrolling of the page. This mode-based differentiated data parsing and instruction generation directly solves the problem of interference between cursor and scroll wheel operations due to the lack of an effective distinction mechanism, and realizes clear separation and independent control of these two core mouse functions.

[0020] Furthermore, the technical solution of the present application identifies and parses mouse button commands from the preliminary primitive sequence (after excluding mode switching gestures) through an independent path, and then combines and verifies the mouse button commands with the mouse cursor movement commands or mouse wheel commands in step six, and finally generates a confirmed mouse event sequence. This design realizes the effective coordination and integration of the three core mouse function modes of cursor movement, wheel scrolling and mouse button operation. The user can trigger events such as mouse clicks by executing predefined button gestures (such as tapping and clicking) in any control mode (cursor mode or wheel mode), and the recognition logic of these button gestures is relatively independent of the continuous control logic of the cursor / wheel. The instruction combination and verification mechanism further ensures that the commands from these different sources can be reasonably synchronized (for example, the click event accurately acts on the current cursor position) and filters out potential conflicts or redundant commands generated by fast operation sequences. This overcomes the problem of separation of cursor control and button triggering and difficulty in natural and efficient coordination pointed out in the background art, significantly improving the consistency of operation and the smoothness of overall interaction.

[0021] In addition, the technical solution of the present application performs multi-level preprocessing (including adaptive filtering, sensor fusion, dynamic normalization and online offset correction) on the original three-dimensional motion data of the inertial measurement unit to generate a high-quality standard motion feature stream, which provides a stable and reliable data basis for all subsequent mode judgments, gesture recognition and mapping algorithms, and ensures the accuracy of the overall control and the stability of the response from the source. The use of multi-scale dynamic time window segmentation and hierarchical motion primitive library template matching to obtain a preliminary primitive sequence improves the accuracy of gesture recognition and the adaptability to gestures of different speeds and amplitudes. Finally, the confirmed mouse event sequence is converted into a standard mouse input event of the target operating system and injected into the event processing queue, ensuring the wide compatibility of this method and its immediate availability in various operating systems and application environments. These auxiliary technical features work together to further enhance the overall performance of this solution as a practical and efficient IMU mouse control solution. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the structures shown in these drawings without paying any creative work.

[0023] Figure 1 This is a module structure diagram of an embodiment of a multi-modal mouse control device based on an inertial measurement unit according to the present invention;

[0024] Figure 2 FIG. 4 is a flow chart of an embodiment of a multimodal mouse control method based on an inertial measurement unit according to the present invention.

[0025] The purpose, features and advantages of the present invention will be further described with reference to the accompanying drawings and in conjunction with the embodiments. DETAILED DESCRIPTION

[0026] It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0027] To better understand the above technical solutions, exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although exemplary embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be limited by the embodiments described herein. Instead, these embodiments are provided to enable a more thorough understanding of the present disclosure and to fully convey the scope of the present disclosure to those skilled in the art.

[0028] It should be noted that in the claims, any reference signs placed between brackets shall not be construed as limiting the claims. The presence of "comprising" in the text does not exclude the presence of components or steps not listed in the claims. The quantifier "a" or "an" preceding a component does not exclude the presence of a plurality of such components. The present invention can be implemented by means of hardware comprising several different components and by means of a suitably programmed computer. In a unit claim that lists several means, several of these means may be embodied by the same hardware item. The use of "first", "second", and "third" etc. does not indicate any order and these words may be interpreted as names.

[0029] like Figure 1 As shown, Figure 1 It is a structural diagram of a server 1 (also called a multimodal mouse control device based on an inertial measurement unit) in a hardware operating environment involved in an embodiment of the present invention.

[0030] The server of the embodiment of the present invention is a device with display function such as "Internet of Things devices", smart air conditioners, smart lights, smart power supplies with networking functions, AR / VR devices with networking functions, smart speakers, self-driving cars, PCs, smart phones, tablets, e-book readers, portable computers, etc.

[0031] like Figure 1 As shown, the server 1 includes: a memory 11 , a processor 12 and a network interface 13 .

[0032] The memory 11 includes at least one type of readable storage medium, including a flash memory, a hard disk, a multimedia card, a card-type memory (e.g., an SD or DX memory), a magnetic memory, a magnetic disk, an optical disk, etc. In some embodiments, the memory 11 may be an internal storage unit of the server 1, such as a hard disk of the server 1. In other embodiments, the memory 11 may also be an external storage device of the server 1, such as a plug-in hard disk equipped on the server 1, a smart media card (SMC), a secure digital (SD) card, a flash card, etc.

[0033] Furthermore, the memory 11 may include both an internal storage unit of the server 1 and an external storage device. The memory 11 can be used not only to store application software installed on the server 1 and various data, such as the code of the multimodal mouse control program 10 based on the inertial measurement unit, but also to temporarily store data that has been output or is about to be output.

[0034] In some embodiments, the processor 12 can be a central processing unit (CPU), a controller, a microcontroller, a microprocessor, or other data processing chip, used to run the program code stored in the memory 11 or process data, such as executing the multimodal mouse control program 10 based on the inertial measurement unit.

[0035] The network interface 13 may optionally include a standard wired interface or a wireless interface (such as a WI-FI interface), and is generally used to establish a communication connection between the server 1 and other electronic devices.

[0036] The network may be the Internet, a cloud network, a wireless fidelity (Wi-Fi) network, a personal area network (PAN), a local area network (LAN), and / or a metropolitan area network (MAN). Various devices in the network environment may be configured to connect to the communication network according to various wired and wireless communication protocols. Examples of such wired and wireless communication protocols may include, but are not limited to, at least one of the following: Transmission Control Protocol and Internet Protocol (TCP / IP), User Datagram Protocol (UDP), Hypertext Transfer Protocol (HTTP), File Transfer Protocol (FTP), ZigBee, EDGE, IEEE 802.11, Light Fidelity (Li-Fi), 802.16, IEEE 802.11s, IEEE 802.11g, multi-hop communication, wireless access point (AP), device-to-device communication, cellular communication protocol, and / or Bluetooth communication protocol, or a combination thereof.

[0037] Optionally, the server may further include a user interface, which may include a display and an input unit such as a keyboard. The optional user interface may also include a standard wired interface or a wireless interface. Optionally, in some embodiments, the display may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, or an OLED (Organic Light-Emitting Diode) touchscreen. The display, which may also be referred to as a display screen or display unit, is used to display information processed in the server 1 and to display a visual user interface.

[0038] Figure 1 Only the server 1 having components 11-13 and the multimodal mouse control program 10 based on the inertial measurement unit is shown. It can be understood by those skilled in the art that Figure 1 The structure shown does not constitute a limitation on the server 1 , and the server 1 may include fewer or more components than shown in the figure, or combine certain components, or arrange the components differently.

[0039] In this embodiment, the processor 12 may be configured to call an inertial measurement unit-based multimodal mouse control program stored in the memory 11 .

[0040] Based on the hardware architecture of the above-mentioned multimodal mouse control device based on an inertial measurement unit, an embodiment of the multimodal mouse control method based on an inertial measurement unit of the present invention is proposed. The multimodal mouse control method based on an inertial measurement unit of the present invention aims to provide an improved mouse control solution that can seamlessly integrate accurate and reliable gesture recognition to achieve multimodal collaborative operation, with multimodal adaptive cursor and scroll wheel control.

[0041] Reference Figure 2 , Figure 2 This is an embodiment of a multimodal mouse control method based on an inertial measurement unit of the present invention, and the multimodal mouse control method based on an inertial measurement unit includes the following steps:

[0042] S10. Obtain the original three-dimensional motion data of the inertial measurement unit and perform multi-level preprocessing to generate a standard motion feature stream containing three-dimensional posture parameters and three-dimensional angular velocity parameters. The purpose of this step is to collect the original motion signal from the inertial measurement unit (IMU) and eliminate the interference factors such as noise and drift that may exist therein through a series of systematic processing methods, and finally extract a standardized data stream that can accurately reflect the posture information of the device in three-dimensional space (such as the direction and rotation angle of the device) and the angular velocity information of the motion (that is, the speed of the device rotation). This standardized motion feature stream provides a reliable and consistent data basis for the subsequent accurate recognition of user gestures, judgment of user intentions, and generation of corresponding control instructions. The source of the original three-dimensional motion data can be various devices integrated with IMU, such as smart watches, smart bracelets, VR / AR controllers, etc. This method does not limit the specific hardware implementation of the IMU or the type of device it is in.

[0043] In some embodiments, the acquisition and preprocessing process of step S10 may be implemented by the following steps:

[0044] S11, performing preliminary filtering on the original three-dimensional motion data by applying an adaptive low-pass filter, wherein a cutoff frequency of the adaptive low-pass filter is dynamically adjusted according to signal energy of the original three-dimensional motion data;

[0045] S12, performing sensor fusion processing on the raw 3D motion data after preliminary filtering using a Kalman filter to generate the 3D posture parameters and the 3D angular velocity parameters;

[0046] S13. Normalize and offset-correct the three-dimensional posture parameters and the three-dimensional angular velocity parameters to obtain the standardized motion feature flow.

[0047] In step S11, an adaptive low-pass filter is used to remove high-frequency noise from the raw 3D motion data, retaining the low-frequency, valid signals associated with the user's actual movements. The key to the adaptive low-pass filter is its ability to dynamically adjust its cutoff frequency based on the signal's energy, striking a balance between noise removal and signal detail preservation. Signal energy is typically determined by calculating the root mean square value of acceleration or angular velocity data over a period of time. This dynamic adjustment mechanism enables the filter to adapt to changes in the intensity of user movements, such as rapid waving versus slow adjustments.

[0048] In step S12, the Kalman filter comprehensively estimates the device's attitude and angular velocity by fusing data from the accelerometer, gyroscope, and magnetometer. The accelerometer provides information about the device's tilt relative to gravity, the gyroscope calculates angular displacement through integration, and the magnetometer is used to correct heading drift. The Kalman filter uses state prediction and measurement update mechanisms to optimize the integration of multi-sensor data and generate accurate three-dimensional attitude parameters (such as quaternions or Euler angles) and three-dimensional angular velocity parameters. This fusion process significantly reduces the effects of noise and drift in single sensor data, providing a reliable data foundation for subsequent processing.

[0049] In step S13, normalization and offset correction further optimize the generated attitude and angular velocity parameters. Normalization scales parameters of different dimensions to a uniform range (such as [0, 1] or [-1, 1]) to eliminate dimensional differences and facilitate subsequent algorithm processing. Offset correction is used to eliminate zero-point drift of the sensor when it is stationary. For example, a gyroscope may output a non-zero angular velocity when there is no motion. Through these processes, the generated standardized motion feature stream has a consistent format and high precision, which can support accurate mapping of multiple control modes.

[0050] In step S11, adaptive filtering ensures initial data cleanliness. In step S12, Kalman filtering fuses multi-sensor information to generate precise parameters. In step S13, normalization and correction ensure data consistency and reliability. This multi-stage processing mechanism works together to transform raw data into a standard feature stream suitable for complex interactive scenarios, laying a solid foundation for subsequent cursor movement, scroll wheel control, or gesture recognition.

[0051] For example, suppose a user uses a smartwatch equipped with an IMU to interact with a Windows laptop. The IMU outputs raw 3D motion data at a sampling rate of 100Hz, including angular velocity [30, 15, 8] degrees / second and acceleration [0.1, 0.2, -9.7]m / s. 2In step S11, the system calculates the energy of the angular velocity signal and obtains a root mean square value of approximately 19.4 degrees / second. The cutoff frequency of the adaptive low-pass filter is determined to be 6.94Hz according to the formula f_c=0.1*E+5, and the noise above this frequency is filtered out, and the preliminary filtered data is output. In step S12, the Kalman filter fuses the filtered acceleration, angular velocity and magnetometer data to generate three-dimensional attitude parameters (for example, the pitch angle represented by quaternion is 12°, the roll angle is 6°, and the yaw angle is 2°) and angular velocity parameters [30,15,8] degrees / second. In step S13, the posture parameters are normalized using the formula θ_norm = (θ - θ_min) / (θ_max - θ_min). For example, a pitch angle of 12° is normalized to (12 + 180) / (180 - (-180)) = 0.533. The angular velocity is corrected for the offset (e.g., an offset of [0.3, 0.2, 0.1] degrees / second becomes [29.7, 14.8, 7.9] degrees / second after correction). This ultimately generates a standard motion feature stream containing the normalized postures [0.533, 0.517, 0.506] and the corrected angular velocities [29.7, 14.8, 7.9] degrees / second. This feature stream can be used for subsequent cursor control or gesture recognition, for example, to support precise positioning in graphic design scenarios or enable fast navigation in gaming scenarios.

[0052] It can be understood that by using an adaptive low-pass filter to dynamically adjust the cutoff frequency according to the signal energy, the system can balance the effects of denoising and signal retention under different motion intensities, thereby improving the quality and stability of the data and providing reliable input for subsequent processing. In addition, by fusing multi-sensor data through the Kalman filter, the system can integrate information from the accelerometer, gyroscope, and magnetometer to generate accurate three-dimensional posture and angular velocity parameters, overcoming the limitations of single sensor noise and drift, and significantly improving the reliability of motion data. At the same time, through normalization processing and offset correction, the system eliminates the influence of parameter dimension differences and sensor zero-point drift, generates a standardized motion feature stream, ensures the consistency and portability of data in different devices or scenarios, and thus provides a high-quality data foundation for subsequent contextual adaptive control and gesture recognition, improving the accuracy and fluency of interaction.

[0053] S20. Apply multi-scale dynamic time window segmentation to the standardized motion feature stream to extract candidate gesture segments, and match the candidate gesture segments with templates of a pre-constructed hierarchical motion primitive library to obtain a preliminary primitive sequence. The core purpose of this step is to intelligently segment the continuous standardized motion feature stream generated in the previous step into a series of shorter segments that may contain user-specific intentions, namely candidate gesture segments. Subsequently, by comparing and matching these segments with templates in a pre-constructed hierarchical motion primitive library that contains definitions of various basic motion patterns, the specific motion primitives corresponding to each segment are identified, and finally a preliminary primitive sequence consisting of these identified motion primitives arranged in chronological order is formed. This preliminary primitive sequence provides basic, classified motion units for subsequent complex gesture analysis (such as recognition of mode switching gestures or mouse button gestures).

[0054] In some embodiments, step S30 may be implemented by the following steps:

[0055] S31, applying a three-level parallel sliding window process comprising a micro-window, a medium-window, and a macro-window to the standardized motion feature stream, and dynamically adjusting the window size and overlap ratio of each level of windows according to the energy change rate and complexity index of the standardized motion feature stream to extract the candidate gesture segments;

[0056] S32. Extract time domain statistical features, frequency domain distribution characteristics and posture trajectory features from the motion data in the candidate gesture segment, and perform pattern matching between the extracted features and the primitive templates of the corresponding level in the pre-built hierarchical motion primitive library using a dynamic time warping algorithm or a support vector machine model to obtain the preliminary primitive sequence.

[0057] In step S31, three-level parallel sliding window processing captures potential gesture fragments in the standardized motion feature stream by setting windows of different time spans (micro-window, medium-window, macro-window). Micro-windows (such as 0.1 seconds) are suitable for fast and short actions, such as the jitter of a single-click gesture; medium-windows (such as 0.3 seconds) are suitable for actions of medium duration, such as double-clicks; and macro-windows (such as 0.5 seconds) are used to identify longer-duration actions, such as long presses or posture holds. The window size and overlap rate are dynamically adjusted based on the energy change rate (such as the root mean square value of the angular velocity change) and complexity indicators (such as signal entropy or spectral complexity) of the feature stream. This adaptive mechanism ensures that the system can flexibly adapt to the diversity of user actions, such as the different rhythms of fast waving and slow adjustments, thereby improving the accuracy of gesture fragment extraction.

[0058] In step S32, features are extracted from the extracted candidate gesture segments, including time domain statistical features (such as mean, variance, and peak), frequency domain distribution characteristics (such as the main frequency component after Fourier transform), and posture trajectory features (such as the continuously changing trajectory of the posture angle). These features comprehensively describe the motion pattern of the segment and provide multi-dimensional information for subsequent matching. The pre-built hierarchical motion primitive library contains templates at different levels, such as micro-motion primitives (such as single jitter) for fast movements and combination primitives (such as continuous jitter) for complex gestures. The dynamic time warping (DTW) algorithm allows nonlinear time alignment by calculating the time series similarity between the segment features and the template, and adapts to individual differences in user movement speed; the support vector machine (SVM) model uses a classifier to identify the degree of match between the features and the template. The two are combined to generate a preliminary primitive sequence, which contains structured identifiers that may correspond to gestures, such as "fast jitter" or "continuous gesture."

[0059] The multi-scale dynamic window segmentation strategy of step S31 ensures that potential gesture segments of different durations and dynamic characteristics are effectively captured from continuous motion data. Subsequently, step S32 extracts the deep features of each segment and uses advanced pattern matching techniques (such as DTW or SVM) to accurately map these segments to the corresponding entries in the predefined motion primitive library. Such a process from intelligent segmentation to fine feature extraction to robust pattern matching enables step S20 to convert the original standardized motion feature stream into a structured preliminary sequence composed of identified motion primitives, laying a solid foundation for subsequent higher-level semantic understanding (such as parsing of specific gesture commands).

[0060] For example, suppose a user interacts with a Windows laptop using an IMU-equipped smartwatch. The normalized motion feature stream contains normalized 3D pose parameters [0.533, 0.517, 0.506] and calibrated angular velocity parameters [29.7, 14.8, 7.9] degrees / second. In step S31, the system detects two rapid angular velocity peaks within 0.3 seconds (e.g., [60, 30, 15] degrees / second and [55, 25, 12] degrees / second). The system calculates the energy change rate (RMS value approximately 45 degrees / second) and complexity index (high signal entropy). Based on this, the micro-window is adjusted to 0.1 seconds (overlap ratio 0.5) and the medium window to 0.3 seconds (overlap ratio 0.3), extracting the candidate gesture segment containing these two peaks. In step S32, the system extracts temporal features (e.g., peak value 60 degrees / second, variance 10), frequency features (e.g., main frequency 5Hz), and gesture trajectory features (e.g., pitch angle change trajectory) from this segment. Using the DTW algorithm, the segment features were matched with a double-click template in the primitive library (defined as two rapid angular velocity peaks separated by less than 0.5 seconds), achieving a similarity score of 0.95. The SVM model further confirmed the match and generated a preliminary primitive sequence, labeled the "double-click primitive." This sequence was then used to identify the double-click gesture and trigger the file open operation.

[0061] It can be understood that by applying multi-scale dynamic time window segmentation, the system can flexibly capture gesture features at different time scales and adapt to the diversity of fast or continuous movements, thereby improving the accuracy and coverage of gesture extraction. In addition, by extracting time domain, frequency domain and trajectory features and combining DTW or SVM for template matching, the system can tolerate individual differences in user movement speed and rhythm, accurately identify complex gesture patterns, and generate reliable primitive sequences. At the same time, this combination of multi-dimensional feature extraction and hierarchical template matching significantly improves the robustness of gesture recognition, reduces the misrecognition rate, and provides a high-quality input foundation for subsequent mode switching and command generation, thereby optimizing the accuracy and naturalness of IMU-based multimodal interaction in complex scenarios.

[0062] S30. According to the definition of the mode switching gesture in the pre-built hierarchical motion primitive library, the preliminary primitive sequence is analyzed to identify the occurrence or end state of the mode switching gesture, and the current control mode is determined to be the cursor control mode or the scroll wheel control mode according to the recognition result of the mode switching gesture. The key to this step is to give the system the ability to understand the user's intention to switch between different control focuses. The system no longer simply interprets all actions as cursor movement or clicks, but can determine whether the user wants to control the movement of the mouse pointer or the scrolling of a page or list (i.e., the scroll wheel function) by identifying specific user actions - i.e., mode switching gestures. This is achieved by analyzing the preliminary primitive sequence generated in the previous step and comparing it with the definition of the mode switching gesture in the pre-built hierarchical motion primitive library. Once the occurrence or end state of the mode switching gesture is identified, the system will update the current control mode accordingly and decide whether the subsequent motion data should be interpreted as cursor control instructions or scroll wheel control instructions.

[0063] In some embodiments, step S30 may be implemented by:

[0064] S31, matching the preliminary primitive sequence with a specific continuous primitive combination or a specific posture holding pattern defined in the pre-built hierarchical motion primitive library, wherein the specific continuous primitive combination or the specific posture holding pattern corresponds to one of a fist-clenching start state, a fist-clenching continue state, or a fist-clenching end state;

[0065] S32: when the fist-clenching start state or the fist-clenching continued state is identified, determining that the current control mode is the roller control mode;

[0066] S33: When the fist-clenching state is identified to have ended or the fist-clenching state is not detected to have continued within a preset time, determining that the current control mode is the cursor control mode.

[0067] In step S31, the specific continuous primitive combination and the specific posture holding mode are predefined templates in the hierarchical motion primitive library, which are used to describe the different stages of the fist-clenching action. The continuous primitive combination includes a series of rapid motion changes, such as a sequence from static to rapid angular velocity peak, indicating the start of fist-clenching. Alternatively, a rapid rotation primitive followed by a short static primitive can also be used to indicate the start of fist-clenching; the posture holding mode refers to the stable state of posture parameters within a certain period of time, for example, the three-dimensional posture parameters change less than a certain threshold within 0.5 seconds, indicating that the fist-clenching continues. These templates identify the occurrence, continuation or end state of the fist-clenching action by matching with the preliminary primitive sequence. The matching process is usually based on a dynamic time warping algorithm, which allows nonlinear alignment on the time axis to adapt to individual differences in the user's movement rhythm.

[0068] In step S32, when the system detects either the fist-clenching start state or the fist-clenching hold state, it determines that the current control mode is the scroll wheel control mode. The fist-clenching start state typically corresponds to the user performing a specific action, such as rapidly clenching their hand, which generates a brief peak in angular velocity or acceleration. The fist-clenching hold state indicates that the user maintains the fist-clenching posture, such as continuously clenching their fist to trigger a scrolling action in a virtual reality scene. Activating the scroll wheel control mode maps subsequent motion data to interface scrolling instructions, making it suitable for scenarios requiring up-and-down or left-and-right scrolling, such as browsing long documents or web pages.

[0069] In step S33, when the system recognizes that the fist-clenching state has ended or the fist-clenching state has not been continuously detected within a preset time (such as 1 second), it is determined that the current control mode is the cursor control mode. The fist-clenching state may be manifested as the user releasing the hand, resulting in a rapid change in angular velocity or posture parameters; the failure to continuously detect the fist-clenching state indicates that the user has not maintained the fist-clenching action, and the system defaults to the cursor control mode. The cursor control mode is suitable for scenarios that require precise pointer positioning, such as graphic editing or file selection, to ensure that the switching of the control mode is consistent with the user's intention. Figure 1 To.

[0070] Step S31 uses template matching to identify different states of the fist-clenching gesture, and steps S32 and S33 dynamically determine the control mode based on the recognition results. This combination enables the system to flexibly switch between cursor control and scroll wheel control based on the user's gesture intent, adapting to different interaction needs.

[0071] For example, suppose a user uses a smartwatch equipped with an inertial measurement unit to interact with a Windows laptop. The preliminary primitive sequence contains primitives identified as "rapid angular velocity peak" and "stable posture". In step S31, the system detects the rapid angular velocity peak in the primitive sequence (such as [50, 25, 10] degrees / second, lasting 0.1 seconds), and matches it with the "fist start" template in the hierarchical motion primitive library through the dynamic time warping algorithm. The similarity score is 0.92, confirming the fist start state. Subsequently, the posture parameters for 0.6 seconds in the sequence (such as [0.540, 0.520, 0.510], with a change of less than 0.01) match the "fist continued" template with a score of 0.95. In step S32, based on the fist continued state, the system determines that the current mode is the scroll wheel control mode, which is suitable for users to browse long web pages. If the user subsequently releases their hand, generating a new angular velocity peak (e.g., [-45, -20, -8] degrees / second), the "end of fist clenching" template is matched in step S31, resulting in a score of 0.90. Based on this, step S33 switches back to cursor control mode, allowing the user to select files. Alternatively, if no continued fist clenching is detected within 1 second, the system also switches back to cursor control mode.

[0072] It can be understood that by matching the preliminary primitive sequence with the fist-clenching template in the hierarchical motion primitive library, the system can accurately identify the start, duration, and end states of the fist-clenching action, thereby providing a reliable basis for mode switching. In addition, by activating the scroll wheel control mode according to the start or duration of the fist-clenching, and switching back to the cursor control mode according to the end of the fist-clenching or non-continuation of the fist-clenching, the system realizes the dynamic adaptation of the control mode, significantly improving the matching degree between the interaction mode and the user's intention and scenario requirements. At the same time, this mode switching mechanism based on gesture state, combined with the matching method of dynamic time regularization, tolerates individual differences in the rhythm of user movements, enhances the robustness of gesture recognition, and thus optimizes the smoothness and naturalness of multimodal interaction in complex scenarios.

[0073] S40. According to the current control mode, select the corresponding mapping algorithm to process the standardized motion feature stream and generate a mouse cursor movement instruction or a mouse wheel instruction. This step is intended to dynamically select a suitable algorithm based on the previously identified control mode (cursor control mode or wheel control mode) and the user's current operating environment to map the three-dimensional posture parameters or angular velocity parameters in the standardized motion feature stream to specific mouse instructions. The standardized motion feature stream contains pre-processed three-dimensional posture and angular velocity parameters, which can reflect the user's hand or device movement. In cursor control mode, the system maps the motion data to two-dimensional movement of the screen cursor, which is suitable for precise pointing or navigation; in wheel control mode, the system maps the motion data to interface scrolling instructions, which is suitable for browsing long pages or lists. By dynamically selecting the mapping algorithm, the system can adapt to the needs of different interactive scenarios and improve the accuracy and naturalness of the control.

[0074] In some embodiments, when the control mode is the cursor control mode, selecting a corresponding mapping algorithm to process the standardized motion feature stream and generating a mouse cursor movement instruction may be achieved by the following steps:

[0075] S41. Determine, through an application program interface or a user interface analysis tool provided by the operating system, the type of the currently active application or the type of the user interface element under the cursor as the current human-computer interaction context information;

[0076] S42. Selecting, according to a preset context mapping table, a specific axial combination of the three-dimensional gesture parameters or a specific axial combination of the three-dimensional angular velocity parameters corresponding to the type of the currently active application or the type of the user interface element under the cursor as a basic cursor control input;

[0077] S43, detecting the movement amplitude or movement speed related to the basic cursor control input in the standard movement data stream,

[0078] S44: When the movement amplitude or the movement speed exceeds a preset first threshold, determining the preset relative mapping mode as the currently effective sensitivity mapping mode,

[0079] S45. When the movement amplitude or the movement speed is lower than a preset second threshold, determining the preset absolute mapping mode as the currently effective sensitivity mapping mode;

[0080] S46. Read the physical mouse sensitivity setting of the operating system, and calculate the basic mouse sensitivity adjustment parameter according to the physical mouse sensitivity setting;

[0081] S47. Use the basic mouse sensitivity adjustment parameter as the gain adjustment factor of the mapping algorithm corresponding to the currently effective sensitivity mapping mode to process the basic cursor control input, and when the currently effective sensitivity mapping mode changes from the sensitivity mapping mode of the previous control cycle, perform a smooth transition processing based on weighted average based on the current output of the preset mapping algorithm and the final output state data of the previous control cycle to generate the mouse cursor movement instruction.

[0082] In step S41, the operating system's application program interface (e.g., Windows' GetForegroundWindow) or user interface analysis tools (e.g., UI Automation) are used to identify the type of the currently active application (e.g., a text editor, graphic design software) or the type of interface element under the cursor (e.g., a button, text box). This information serves as the basis for the human-computer interaction context, helping the system understand the user's current operation goals and thus optimize the selection of control inputs.

[0083] In step S42, a pre-set contextual mapping table defines the correspondence between different applications or interface elements and axial combinations of motion parameters. For example, in graphic design software, a combination of pitch and roll angles might be selected for precise cursor positioning; in gaming applications, the X- and Y-axis components of angular velocity might be selected to support rapid movement. This contextually adaptive selection mechanism ensures that control inputs match task requirements.

[0084] In step S43, the system detects the movement amplitude (such as the root mean square value of the angular velocity) or movement speed of the basic cursor control input to determine the user's action intensity. This step provides a quantitative basis for subsequent mapping mode selection.

[0085] In steps S44 and S45, the system determines whether to use relative mapping mode or absolute mapping mode based on the comparison of the motion amplitude or speed with the preset threshold. The relative mapping mode maps motion data (such as angular velocity) to the relative displacement of the cursor, which is suitable for fast, large-scale movements and for scenarios where the amplitude exceeds a first threshold (such as 20 degrees / second). The absolute mapping mode directly maps gesture parameters to absolute screen coordinates, which is suitable for fine manipulation and for scenarios where the amplitude is below a second threshold (such as 10 degrees / second).

[0086] In step S46, the system reads the physical mouse sensitivity setting of the operating system (for example, in the Windows system, the system obtains the mouse sensitivity value set by the user by calling the SystemParametersInfo function (using the SPI_GETMOUSESPEED parameter), which is usually in the range of 1 to 20, and the default value is 10, indicating medium sensitivity), and calculates the basic mouse sensitivity adjustment parameter through a predefined conversion rule (such as a linear formula). This parameter reflects the user's sensitivity preference for a traditional mouse and ensures that the IMU control is consistent with the user's habits. For example, the conversion formula can be defined as S_IMU=k*S_OS+b, where S_OS is the sensitivity value obtained by the operating system, k and b are constants preset according to the IMU sensor characteristics and cursor control requirements, and S_IMU is the calculated basic sensitivity parameter, which is used to subsequently scale the IMU motion data to the cursor movement distance.

[0087] In step S47, the basic mouse sensitivity adjustment parameter is used as a gain factor to adjust the output ratio of the mapping algorithm, for example, scaling the angular velocity to pixel displacement ratio in relative mapping mode. When the mapping mode is switched (e.g., from relative to absolute), the system uses a weighted average method (e.g., C = w(t) * C_prev + (1-w(t)) * C_current) to smoothly transition between the current output and the previous cycle output to avoid cursor jumps.

[0088] Steps S41 and S42 select the appropriate control input based on contextual information and a mapping table. Steps S43 to S45 dynamically determine the mapping mode. Steps S46 and S47 optimize command generation through sensitivity adjustment and smooth transitions. This combination ensures that cursor movement command generation is both user-friendly and adaptable to the current scenario and action characteristics.

[0089] For example, suppose a user uses an IMU-equipped smartwatch to operate on a Windows laptop, the current control mode is cursor control mode, and the standardized motion feature stream contains normalized posture [0.533, 0.517, 0.506] and angular velocity [29.7, 14.8, 7.9] degrees / second. In step S41, the system detects through GetForegroundWindow that the current active application is "Photoshop.exe", indicating that the user is in a graphic design scene. In step S42, according to the context mapping table, the pitch angle and roll angle ([0.533, 0.517]) of the posture parameters are selected as the basic cursor control input, which is suitable for precise positioning. In step S43, the system calculates the angular velocity amplitude as sqrt(29.7 2 +14.8 2 +7.9 2 )≈33.1 degrees / second. In steps S44 and S45, the amplitude exceeds the first threshold of 20 degrees / second but is lower than the second threshold of 10 degrees / second and is not triggered, and the relative mapping mode is selected. In step S46, the system reads the mouse sensitivity setting of 14 through SystemParametersInfo and calculates the basic sensitivity parameter S_IMU=0.04*14+0.2=0.76. In step S47, the system maps the angular velocity [29.7,14.8] degrees / second to the cursor displacement [0.76*29.7≈22.6,0.76*14.8≈11.2] pixels / second. If switched to absolute mode, it smoothly transitions to the absolute coordinates [483.2,487.6] through weighted averaging (weight w(t)=0.7) to generate cursor movement instructions, which are suitable for precise layer selection in Photoshop.

[0090] It can be understood that by determining the type of the currently active application or the type of user interface element under the cursor as the current human-computer interaction context information, the system can make subsequent selection of cursor control parameters more targeted, thereby optimizing the control experience and efficiency in different application scenarios. In addition, according to a preset context mapping table, a specific axial combination of three-dimensional posture parameters or three-dimensional angular velocity parameters corresponding to the context information is selected as the basic cursor control input. This can match the most intuitive and efficient original motion signal source for different types of interactive tasks, thereby improving the naturalness of control. Furthermore, by detecting the movement amplitude or speed of the basic cursor control input and dynamically switching the currently effective sensitivity mapping mode between a preset relative mapping mode (suitable for fast, large-scale movement) and an absolute mapping mode (suitable for fine, small-scale positioning), the system can intelligently adapt to the user's control intentions in different operating scenarios, significantly enhancing the flexibility of cursor control and its ability to meet different task requirements. At the same time, by reading the operating system's physical mouse sensitivity settings and calculating the basic mouse sensitivity adjustment parameters, which are applied as gain adjustment factors to the currently active mapping algorithm, the IMU-based mouse control ensures that its movement speed is consistent with the user's existing traditional mouse usage habits, effectively reducing the user's learning and adaptation costs. Furthermore, when the sensitivity mapping mode changes, a weighted average-based smooth transition process is performed based on the output of the currently active mapping algorithm and the final output state data of the previous control cycle. This effectively eliminates or significantly reduces sudden pointer jumps, loss of control, or response interruptions that may result from changes in mapping logic or sensitivity characteristics, ensuring the continuity of cursor control and the smoothness of the user experience in dynamic task flows.

[0091] In some embodiments, when the control mode is wheel control mode, a corresponding mapping algorithm is selected to process the standardized motion feature stream and generate mouse wheel commands, which can be achieved by following the steps S48-S49. In wheel control mode, the system focuses on the motion changes of a single axis and uses a specific mapping algorithm to convert it into the amplitude and direction of the wheel scrolling, ensuring that the generated commands are highly consistent with the user's action intention.

[0092] S48, extracting the specific single axial change of the three-dimensional posture parameter or the specific single axial component of the three-dimensional angular velocity parameter from the standardized motion feature stream as a basic roller control input;

[0093] S49: Apply a preset wheel mapping algorithm that maps the amplitude or duration of the basic wheel control input to the wheel scrolling amplitude and direction to generate the mouse wheel instruction.

[0094] Here, the core of step S48 is to specify a clear and user-friendly input source for the scroll wheel control function. When the system is in scroll wheel control mode (for example, the user has performed a fist gesture), not all IMU motion data will be used for scroll wheel control. Instead, information of specific dimensions will be selectively extracted from the complete standardized motion feature stream. This can be a change in a specific single axis in the three-dimensional posture parameter, for example, the change in the tilt angle of the device around one of its axes (such as the horizontal or vertical axis); it can also be a component value in a specific single axis in the three-dimensional angular velocity parameter, for example, the angular velocity of the device rotating around one of its axes. Which parameter and which axis to choose are usually pre-defined during system design to correspond to the user's intuitive scrolling operation habits, such as swinging the wrist up and down or rotating the wrist left and right. The specific motion information extracted in this way constitutes the basic scroll wheel control input.

[0095] Step S49 is responsible for converting this relatively primitive basic scroll wheel control input into a mouse scroll wheel command that can be recognized by the operating system. This step is completed through a preset scroll wheel mapping algorithm. The core function of the algorithm is to map the characteristics of the basic scroll wheel control input - mainly its amplitude (for example, how large the tilt angle is, or how fast the angular velocity is) or its duration (for example, how long a certain tilt posture is maintained or rotated at a certain angular velocity) - into a specific scroll wheel scroll amplitude and scroll direction. For example, the algorithm may stipulate that tilting the device upward at a specific angle corresponds to scrolling up a certain number of units, and tilting it downward corresponds to scrolling down; or, the longer the device rotates at a certain angular velocity, the greater the amount of scrolling generated. Through this mapping, the degree and manner of the user's physical action can be accurately converted into the scrolling behavior of the scroll wheel.

[0096] Step S48 accurately extracts the motion information of a specific single axis used for scroll wheel control, and then combines the mapping algorithm defined in step S49 to convert the amplitude or duration of the motion information into a specific scrolling amplitude and direction. Step S40 can effectively convert the user's specific body movements into intuitive and controllable mouse scroll wheel operations in the scroll wheel control mode.

[0097] For example, suppose a user is browsing the web on a Windows laptop using a smartwatch equipped with an inertial measurement unit. The user is currently in scroll wheel control mode. The normalized motion feature stream contains normalized attitude parameters [0.533, 0.517, 0.506] and angular velocity parameters [29.7, 14.8, 7.9] degrees per second. In step S48, the system extracts the pitch angle change of the three-dimensional attitude parameters (e.g., from 0.533 to 0.550, a change of 0.017, or approximately 3.06 degrees) as the basic scroll wheel control input, which is suitable for vertical scrolling. In step S49, the system applies a scroll wheel mapping algorithm, such as R = k * Δθ, where the gain factor k = 50 pixels / degree, to map the pitch angle change of 3.06 degrees to a scroll amplitude of 3.06 * 50 = 153 pixels, with a positive direction (scrolling upward). If the pitch angle continues to change for 0.5 seconds, the system further adjusts the scrolling speed based on the duration, generating continuous scroll wheel commands to trigger the web page to scroll up 153 pixels, which is suitable for browsing long articles or lists.

[0098] It can be understood that by selectively extracting a specific single-axis change in a 3D posture parameter or a specific single-axis component in a 3D angular velocity parameter from the standardized motion feature stream as the basic scroll wheel control input, a clear, unambiguous, and typically consistent with the user's intuitive operating habits (such as swinging / rotating the device up and down or left and right) device action can be assigned to the scroll wheel control function as an input source, effectively simplifying the operational complexity of the user expressing scrolling intentions in scroll wheel mode. Furthermore, a preset scroll wheel mapping algorithm is applied to map the amplitude (such as the size of the movement angle, the speed of the angular velocity) or the duration (such as the length of time a specific posture or movement is maintained) of this basic scroll wheel control input to the scroll wheel's scroll amplitude and scroll direction. This allows the degree and manner of the specific action performed by the user to be accurately and quantitatively converted into corresponding scrolling amount and scrolling direction instructions, achieving an intuitive, controllable, and simulated reproduction of the traditional mouse wheel function, thereby providing a convenient scrolling interaction experience without the need for a physical scroll wheel.

[0099] S50. According to the definition of mouse button gestures in the pre-built hierarchical motion primitive library, analyze the part of the preliminary primitive sequence that is not recognized as the mode switching gesture to identify the mouse button gesture and parse it into a mouse button instruction. The purpose of this step is that when the user does not perform the mode switching gesture, the system can identify the specific gesture of intending to execute the mouse button (such as single click, double click, long press, etc.) from the user's other body movements. It is achieved by carefully analyzing the parts of the preliminary primitive sequence generated in the previous step (such as step S20) that are not determined as mode switching signals by step S30. The system will refer to the gesture patterns specifically defined for various mouse button operations in the pre-built hierarchical motion primitive library to match and identify these remaining primitive sequence fragments. Once the recognition is successful, it will be parsed into a specific mouse button instruction, thereby completing the contactless key input.

[0100] In some embodiments, step S50 may be implemented by the following steps:

[0101] S51. Applying a preset sequence matching rule to a portion of the preliminary primitive sequence that is not recognized as the mode switching gesture, the portion is matched with mouse button gesture templates defined in the pre-built hierarchical motion primitive library, where the mouse button gesture templates include a single-click gesture template, a double-click gesture template, and a long-press gesture template.

[0102] S52, performing duration verification, motion amplitude verification, and confidence threshold filtering based on current human-computer interaction context information on the successfully matched candidate mouse button gestures;

[0103] S53: Parse the verified candidate mouse button gesture into a corresponding left-click instruction, right-click instruction, double-click instruction, or a button press and release instruction sequence.

[0104] Here, step S51 describes in detail how to preliminarily screen out fragments that may represent mouse button operations from the preliminary primitive sequence. The system will select those parts of the primitive sequence that have not been identified as being used for mode switching by the previous steps, and then apply a series of preset sequence matching rules. These rules define how to compare these primitive sequence fragments with various mouse button gesture templates stored in a pre-built hierarchical motion primitive library. These templates are pre-defined typical motion patterns representing specific button actions. For example, the library will contain a single-click gesture template for identifying a single-click action, a double-click gesture template for identifying two clicks in quick succession, and a long-press gesture template for identifying the action of holding down a button for a period of time. Each template itself may be composed of one or more specific motion primitives in a specific timing relationship or feature combination.

[0105] Step S52 then performs a series of rigorous validations on the candidate mouse button gestures initially matched in step S51 to improve recognition accuracy and reduce misuse. This includes duration validation, which checks whether the entire candidate gesture process is within a reasonable, pre-set time range for the specific button type (such as a single or double click). Actions that are too short or too long may be excluded. It also includes motion amplitude validation, which assesses whether the physical amplitude of the action (e.g., the distance the IMU device moves in space, the angle of rotation, or the maximum angular velocity achieved) meets a preset minimum threshold to be considered a valid intention, while not exceeding a certain upper limit to avoid being identified as other unrelated, large-scale actions. Furthermore, confidence threshold filtering based on the current human-computer interaction context is performed. This means that for a matched candidate gesture, its match score (confidence) must be above a specific threshold to be accepted; and this threshold itself can be dynamically adjusted based on the nature of the application the user is currently using or the interface element they are interacting with. For example, in scenarios requiring high-precision operations, the threshold may be set higher to ensure that only very clear and unambiguous gestures are recognized as buttons.

[0106] Step S53 is the link that finally converts the candidate mouse button gesture that has passed all verifications into computer executable instructions. Once a candidate gesture successfully passes the various tests of step S52, the system will parse it into a specific mouse button instruction. This may be a left-click instruction, or a right-click instruction (the distinction between left and right buttons may depend on the subtle differences in the gesture itself, or in combination with other contextual information), or a double-click instruction. For operations such as long presses, the system may generate a key press instruction and generate a key release instruction when the gesture ends, thus forming a complete sequence of key press and release instructions. These instructions can then be sent to the event queue of the operating system.

[0107] Through the preliminary recognition based on template matching in step S51, combined with the rigorous multi-dimensional and context-adaptive verification in step S52, and the precise analysis of specific key events in step S53, step S50 can accurately and reliably extract various mouse button intentions from the user's continuous actions and convert them into actual control commands.

[0108] For example, suppose a user is using a smartwatch with an IMU on a Windows laptop. Its preliminary primitive sequence contains segments labeled "rapid angular velocity peaks" and "sustained gesture." In step S51, the system detects two rapid angular velocity peaks ([60, 30, 15] degrees / second and [55, 25, 12] degrees / second, separated by 0.3 seconds) and uses DTW to match them to a double-click template (defined as two peaks within 0.5 seconds), achieving a similarity score of 0.95. In step S52, duration verification confirms that the interval between the peaks (0.3 seconds) is within an acceptable range, amplitude verification ensures that the peak amplitude exceeds a threshold (e.g., 50 degrees / second), and context analysis (e.g., the active application is "explorer.exe") sets a high confidence threshold (0.9), which satisfies the score. In step S53, the verified double-click gesture is interpreted as a double-click command, triggering the opening of the selected file in File Explorer.

[0109] As can be seen, by matching preliminary primitive sequences with predefined mouse button gesture templates, the system can accurately identify the user's intended actions, ensuring reliable command generation. Furthermore, through duration, amplitude, and context-based confidence verification, the system minimizes false detections, thereby improving gesture recognition accuracy. Furthermore, parsing verified gestures into standard commands ensures compatibility with a variety of applications, thereby improving the accuracy and responsiveness of IMU-based mouse interactions in complex scenarios.

[0110] S60. Combine the identified mouse cursor movement instructions, the mouse button instructions and the mouse wheel instructions to generate a mouse event sequence. This process aims to integrate the various instructions generated in the previous steps to form a coordinated and consistent event sequence, ensuring that cursor movement, button operations and wheel scrolling are seamlessly connected in time and logic, reflecting the user's complete intention in the interactive scenario. The mouse cursor movement instruction controls the displacement of the screen pointer, the mouse button instruction corresponds to single-click, double-click or long press operations, and the mouse wheel instruction implements interface scrolling. By performing time alignment, conflict resolution and context verification on these instructions, the system generates a standardized mouse event sequence to support complex interactive tasks such as file dragging, menu selection or page browsing, and improve the fluency and reliability of the control method based on the inertial measurement unit (IMU).

[0111] In some embodiments, step S60 may be implemented by the following steps:

[0112] S61, aligning and synchronizing the timestamps of the identified mouse cursor movement instructions, the mouse button instructions, and the mouse wheel instructions;

[0113] S62: When the confidence level of the mouse button command is higher than a preset button confirmation threshold, suppressing a mouse cursor displacement command or a mouse wheel command caused by a gesture action within a triggering time window of the mouse button command;

[0114] S63, when the mouse cursor movement instruction or the mouse wheel instruction represents high-speed continuous movement, increasing the key confirmation threshold for confirming the mouse button instruction;

[0115] S64: generating a drag operation sequence including move, press, move while dragging, and release events based on the current human-computer interaction context information and the type of the mouse button command;

[0116] S65. Perform checks including context rationality check and repeated command suppression on the combined instruction sequence to generate the mouse event sequence.

[0117] In step S61, timestamp alignment and synchronization ensure that commands are correctly sequenced. Each command (such as a cursor movement, key press, or scroll wheel) is accompanied by a timestamp reflecting the moment it was generated. The system aligns commands within a uniform time window (e.g., ±20 milliseconds), eliminating timing deviations caused by processing delays or sensor sampling variations, ensuring that the event sequence reflects the actual order of user actions.

[0118] In step S62, when the confidence level of a mouse button command exceeds a preset threshold (e.g., 0.9), the system suppresses cursor movement or scroll wheel commands within the button triggering time window (e.g., ±50 milliseconds). This is because button gestures (e.g., single clicks) may be accompanied by slight movements, which can easily trigger movement or scrolling commands by mistake. This suppression mechanism prioritizes ensuring the execution of button commands and reduces erroneous operations.

[0119] In step S63, when the cursor movement or scroll wheel command represents high-speed continuous motion (e.g., angular velocity exceeding 30 degrees per second), the system increases the key command confirmation threshold (e.g., from 0.9 to 0.95). This prevents false triggering of key commands due to noise or unintentional movement during high-speed motion, and enhances the reliability of command differentiation.

[0120] In step S64, the system generates a drag operation sequence including move, press, move while dragging, and release based on the current human-computer interaction context (e.g., the application type is a file manager) and the key command type (e.g., long press). For example, in a file manager, a long press followed by cursor movement generates a drag sequence to support file drag and drop operations.

[0121] In step S65, the system performs contextual rationality check on the combined instruction sequence (such as verifying whether the drag sequence complies with the file manager operation logic) and repeated command suppression (such as ignoring continuously repeated click instructions) to generate the final mouse event sequence to ensure logical consistency and efficient operation.

[0122] Step S61 ensures accurate instruction timing, steps S62 and S63 eliminate conflicts through confidence and threshold adjustment, and steps S64 and S65 generate an event sequence that meets the user's intent through context combination and verification. This combination mechanism achieves the coordination and unification of multimodal instructions.

[0123] For example, suppose a user is using an IMU-equipped smartwatch in a Windows file manager. The generated cursor movement command is a displacement of [22.6, 11.2] pixels / second, the key command is a long press (confidence 0.93), and the scroll wheel command is a scroll up 153 pixels. In step S61, the system aligns the timestamps of these commands (e.g., t = 1000ms, 1005ms, 1010ms) to ensure timing consistency. In step S62, the long press command confidence of 0.93 exceeds the threshold of 0.9, and the system suppresses cursor movement commands within the time window (1000ms ± 50ms) to avoid false triggering. In step S63, the cursor movement speed is less than 30 degrees / second, so the high threshold adjustment is not triggered, and the key threshold is maintained at 0.9. In step S64, the system recognizes the file manager context and the long press command and generates a drag sequence: press (t = 1005ms) → move [22.6, 11.2] pixels → release (t = 1015ms). In step S65 , the system verifies that the drag sequence is consistent with the file manager operation, suppresses repeated press instructions, generates a final event sequence, and triggers the file drag and drop operation.

[0124] It can be understood that through timestamp alignment and synchronization, the system ensures the timing consistency of multimodal commands, providing a reliable foundation for complex interactions. In addition, through confidence suppression and dynamic threshold adjustment, the system effectively reduces the conflict between key and movement or scroll wheel commands, and improves the accuracy of command recognition. At the same time, by combining the context to generate drag sequences and perform rationality verification and repetition suppression, the system generates a logically clear and efficient operation sequence of events, which significantly improves the fluency and accuracy of IMU-based interactions in multi-tasking scenarios.

[0125] S70, converting the mouse event sequence into a standard mouse input event of the target operating system, and injecting the standard mouse input event into the event processing queue of the target operating system. This process aims to convert the mouse event sequence generated by the above steps, including cursor movement, key operations and wheel scrolling instructions, into standardized input events that the operating system can directly recognize and execute, thereby achieving a user interaction experience consistent with traditional mouse operations. The mouse event sequence integrates cursor movement instructions (such as two-dimensional displacement), key instructions (such as single click, double click, long press) and wheel instructions (such as scrolling amplitude and direction). By converting them into a standard event format and injecting them into the operating system queue, the system can seamlessly drive interface operations such as file selection, dragging or page scrolling. This process ensures that the control method based on the inertial measurement unit (IMU) is compatible with the input mechanism of the target operating system and supports smooth interaction in a variety of application scenarios.

[0126] In the implementation of step S70, the system first maps the various instructions in the mouse event sequence to standard mouse input events defined by the target operating system. For example, in the Windows system, the cursor movement instruction can be converted into MOUSEEVENTF_MOVE (relative displacement) or MOUSEEVENTF_ABSOLUTE (absolute coordinate) events, the key instruction can be converted into MOUSEEVENTF_LEFTDOWN / UP (left button pressed / released) or MOUSEEVENTF_RIGHTDOWN / UP (right button pressed / released), and the scroll wheel instruction is converted into MOUSEEVENTF_WHEEL (vertical scrolling) or MOUSEEVENTF_HWHEEL (horizontal scrolling). In the macOS system, similar events can be generated by the CGEventCreateMouseEvent function. These standard events carry specific parameters, such as displacement, scrolling distance or key status, and comply with the input protocol of the operating system. Subsequently, the system injects these events into the event processing queue through the input interface of the operating system (such as the SendInput function of Windows or the CGEventPost function of macOS), and the operating system distributes them to the currently active application to trigger the corresponding interactive behavior.

[0127] To enhance the user experience, the system can provide feedback after event injection, such as by generating vibrations (e.g., a slight vibration of 0.2 seconds) through the device's haptic module, emitting a prompt sound (e.g., a short "beep") through the speaker, or displaying a flashing cursor on the interface. These feedback mechanisms help users confirm that the command was successfully executed, particularly in contactless interaction scenarios, making operations more intuitive.

[0128] For example, suppose a user uses an IMU-equipped smartwatch on a Windows laptop. The generated mouse event sequence includes a cursor movement command (displacement [22.6, 11.2] pixels), a key command (left button long press for 0.5 seconds), and a drag sequence (press → move → release). In step S70, the system converts the cursor movement command into a MOUSEEVENTF_MOVE event with parameters [22.6, 11.2] pixels; the long press command is converted into a MOUSEEVENTF_LEFTDOWN (t = 1005ms) and MOUSEEVENTF_LEFTUP (t = 1015ms) event; and the drag sequence is integrated into a continuous event stream. The system injects these events into the Windows event queue via the SendInput function, triggering a file drag-and-drop operation in the file manager. After injection, the smartwatch generates a 0.2-second vibration feedback to notify the user that the operation is complete. This process ensures that the event sequence seamlessly drives the file drag and drop while providing an intuitive user experience.

[0129] As can be understood, by converting mouse event sequences into standard input events for the target operating system, the system achieves compatibility with traditional mouse input, ensuring correct command execution. Furthermore, through event injection and multimodal feedback mechanisms, the system not only ensures the reliability and smoothness of interaction, but also enhances the user's perception of successful operation through tactile, auditory, or visual feedback, significantly improving the practicality and user experience of IMU-based contactless interaction in complex scenarios.

[0130] In addition, embodiments of the present invention further provide a computer-readable storage medium. The computer-readable storage medium can be any one of, or any combination of, a hard disk, a multimedia card, an SD card, a flash memory card, an SMC, a read-only memory (ROM), an erasable programmable read-only memory (EPROM), a portable compact disk read-only memory (CD-ROM), or a USB memory device. The computer-readable storage medium includes an inertial measurement unit-based multimodal mouse control program 10. The specific implementation of the computer-readable storage medium of the present invention is substantially the same as the specific implementation of the inertial measurement unit-based multimodal mouse control method and server 1 described above, and will not be further described herein.

[0131] It will be understood by those skilled in the art that embodiments of the present invention may be provided as methods, systems, or computer program products. Thus, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0132] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowcharts and / or block diagrams, as well as combinations of processes and / or blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowcharts and / or block diagrams. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0133] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0134] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0135] Although the preferred embodiments of the present invention have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the present invention.

[0136] Obviously, those skilled in the art may make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if such changes and modifications fall within the scope of the claims and their equivalents, the present invention is intended to include such changes and modifications.

Claims

1. A multimodal mouse control method based on an inertial measurement unit, characterized in that: include: Obtain the raw 3D motion data of the inertial measurement unit and perform multi-level preprocessing to generate a standard motion feature stream containing 3D attitude parameters and 3D angular velocity parameters; Applying multi-scale dynamic time window segmentation to the standardized motion feature stream to extract candidate gesture segments, and matching the candidate gesture segments with templates of a pre-constructed hierarchical motion primitive library to obtain a preliminary primitive sequence; Analyzing the preliminary primitive sequence according to the definition of the mode switching gesture in the pre-built hierarchical motion primitive library to identify the occurrence or end state of the mode switching gesture, and determining whether the current control mode is the cursor control mode or the scroll wheel control mode according to the recognition result of the mode switching gesture; According to the current control mode, a corresponding mapping algorithm is selected to process the standardized motion feature stream to generate a mouse cursor movement instruction or a mouse wheel instruction; Analyzing the portion of the preliminary primitive sequence that is not recognized as the mode switching gesture according to the definition of the mouse button gesture in the pre-built hierarchical motion primitive library to recognize the mouse button gesture and parse it into a mouse button command; combining the identified mouse cursor movement instructions, the mouse button instructions, and the mouse wheel instructions to generate a mouse event sequence; The mouse event sequence is converted into a standard mouse input event of a target operating system, and the standard mouse input event is injected into an event processing queue of the target operating system.

2. The multimodal mouse control method based on an inertial measurement unit according to claim 1, wherein: The raw 3D motion data of the inertial measurement unit is acquired and multi-level preprocessing is performed to generate a standard motion feature stream containing 3D attitude parameters and 3D angular velocity parameters, including: Applying an adaptive low-pass filter to perform preliminary filtering on the original three-dimensional motion data, wherein a cutoff frequency of the adaptive low-pass filter is dynamically adjusted according to signal energy of the original three-dimensional motion data; Performing sensor fusion processing on the raw three-dimensional motion data after preliminary filtering using a Kalman filter to generate the three-dimensional posture parameters and the three-dimensional angular velocity parameters; The three-dimensional posture parameters and the three-dimensional angular velocity parameters are normalized and offset corrected to obtain the standardized motion feature flow.

3. The multimodal mouse control method based on an inertial measurement unit according to claim 1, wherein: Applying multi-scale dynamic time window segmentation to the standardized motion feature stream to extract candidate gesture segments, and matching the candidate gesture segments with templates of a pre-constructed hierarchical motion primitive library to obtain a preliminary primitive sequence, including: Applying a three-level parallel sliding window process comprising a micro-window, a medium-window, and a macro-window to the standardized motion feature stream, and dynamically adjusting the window size and overlap ratio of each level of windows according to an energy change rate and a complexity index of the standardized motion feature stream to extract the candidate gesture segments; The time domain statistical features, frequency domain distribution characteristics and posture trajectory features are extracted from the motion data in the candidate gesture segment, and the extracted features are matched with the primitive templates of the corresponding level in the pre-built hierarchical motion primitive library using a dynamic time warping algorithm or a support vector machine model to obtain the preliminary primitive sequence.

4. The multimodal mouse control method based on an inertial measurement unit according to claim 1, wherein: Identifying the occurrence or end state of a predefined mode switching gesture, and determining whether the current control mode is a cursor control mode or a scroll wheel control mode according to the recognition result of the mode switching gesture, including: Matching the preliminary primitive sequence with a specific continuous primitive combination or a specific posture holding pattern defined in the pre-built hierarchical motion primitive library, wherein the specific continuous primitive combination or the specific posture holding pattern corresponds to one of a fist-clenching start state, a fist-clenching continue state, or a fist-clenching end state; When a fist-clenching start state or a fist-clenching continued state is identified, determining that the current control mode is the roller control mode; When the fist-clenching state is identified to have ended or the fist-clenching state is not continuously detected within a preset time, the current control mode is determined to be the cursor control mode.

5. The multimodal mouse control method based on an inertial measurement unit according to claim 1, wherein: Selecting a corresponding mapping algorithm to process the standardized motion feature stream and generate a mouse cursor movement instruction includes: Determining, through an application program interface or a user interface analysis tool provided by the operating system, the type of the currently active application or the type of the user interface element under the cursor as the current human-computer interaction context information; selecting, according to a preset context mapping table, a specific axial combination of the three-dimensional gesture parameters or a specific axial combination of the three-dimensional angular velocity parameters corresponding to the type of the currently active application or the type of the user interface element under the cursor as a basic cursor control input; detecting a motion amplitude or a motion speed associated with the basic cursor control input in the standard motion data stream, When the movement amplitude or the movement speed exceeds a preset first threshold, determining the preset relative mapping mode as the currently effective sensitivity mapping mode, When the movement amplitude or the movement speed is lower than a preset second threshold, determining the preset absolute mapping mode as the currently effective sensitivity mapping mode; Reading the physical mouse sensitivity setting of the operating system, and calculating the basic mouse sensitivity adjustment parameter according to the physical mouse sensitivity setting; The basic mouse sensitivity adjustment parameter is used as the gain adjustment factor of the mapping algorithm corresponding to the currently effective sensitivity mapping mode to process the basic cursor control input, and when the currently effective sensitivity mapping mode changes from the sensitivity mapping mode of the previous control cycle, a smooth transition processing based on weighted average is performed based on the current output of the preset mapping algorithm and the final output state data of the previous control cycle to generate the mouse cursor movement instruction.

6. The multimodal mouse control method based on an inertial measurement unit according to claim 1, wherein: Selecting a corresponding mapping algorithm to process the standardized motion feature stream and generate a mouse wheel instruction includes: Extracting a specific single axial change of the three-dimensional posture parameter or a specific single axial component of the three-dimensional angular velocity parameter from the standardized motion feature stream as a basic roller control input; A preset wheel mapping algorithm is applied to map the amplitude or duration of the basic wheel control input to the wheel scrolling amplitude and direction to generate the mouse wheel instruction.

7. The multimodal mouse control method based on an inertial measurement unit according to claim 1, wherein: According to the definition of mouse button gestures in the pre-built hierarchical motion primitive library, analyzing the portion of the preliminary primitive sequence that is not recognized as the mode switching gesture to recognize the mouse button gestures and parse them into mouse button commands, including: Applying a preset sequence matching rule to a portion of the preliminary primitive sequence that is not recognized as the mode switching gesture, the portion is matched with mouse button gesture templates defined in the pre-built hierarchical motion primitive library, where the mouse button gesture templates include a single-click gesture template, a double-click gesture template, and a long-press gesture template; Perform duration verification, motion amplitude verification, and confidence threshold filtering based on the current human-computer interaction context information on the successfully matched candidate mouse button gestures; The verified candidate mouse button gesture is parsed into a corresponding left-click instruction, right-click instruction, double-click instruction, or a button press and release instruction sequence.

8. The multimodal mouse control method based on an inertial measurement unit according to claim 1, wherein: Combining the identified mouse cursor movement instruction, the mouse button instruction, and the mouse wheel instruction to generate a mouse event sequence, including: aligning and synchronizing timestamps of the identified mouse cursor movement instructions, the mouse button instructions, and the mouse wheel instructions; When the confidence level of the mouse button command is higher than a preset button confirmation threshold, suppressing a mouse cursor displacement command or a mouse wheel command caused by a gesture action within a triggering time window of the mouse button command; When the mouse cursor movement instruction or the mouse wheel instruction represents high-speed continuous movement, increasing a key confirmation threshold for confirming the mouse button instruction; generating, based on the current human-computer interaction context information and the type of the mouse button command, a drag operation sequence including move, press, move while dragging, and release events; The combined instruction sequence is subjected to a check including a context rationality check and a repeated command suppression check to generate the mouse event sequence.

9. A multimodal mouse control device based on an inertial measurement unit, characterized in that: The invention comprises a memory, a processor and a multimodal mouse control program based on an inertial measurement unit stored in the memory and executable on the processor, wherein when the processor executes the multimodal mouse control program based on an inertial measurement unit, the multimodal mouse control method based on an inertial measurement unit as described in any one of claims 1 to 8 is implemented.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a multimodal mouse control program based on an inertial measurement unit. When the multimodal mouse control program based on an inertial measurement unit is executed by a processor, the multimodal mouse control method based on an inertial measurement unit as described in any one of claims 1-8 is implemented.

Images