Intelligent handle mouse control method and system based on dynamic variable pyramid

Abstract

The application provides a smart handle mouse control method and system based on a dynamic variable pyramid, which integrates a multi-modal dimension sense stereoscopic perception system module, collects pressure distribution, touch trajectory and rocker motion data in real time, and dynamically generates a multi-modal feature matrix; a dynamic variable scale pyramid module is introduced, local and global features are extracted through real-time adjustment of weights, down-sampling steps and dynamic scale switching, multi-scale information fusion and precise adaptation are realized; a multi-level neural network model is used to establish an interaction relationship between modes, fuse multi-scale features, and accurately identify complex operation modes; and the application generates personalized feedback and optimization strategies based on real-time operation and historical data, significantly improves interaction efficiency and adaptability, effectively solves the obstacles of the finger-disabled population in traditional device operation, and realizes intelligent and efficient human-computer interaction control.

Description

Technical Field

[0001] The present invention relates to the technical field of intelligent auxiliary equipment and human-computer interaction, and in particular to an intelligent handle mouse control method and system based on a dynamically variable pyramid. Background Art

[0002] Existing mouse and controller devices require high hand dexterity, making it difficult for people with limited finger control to achieve precise control and complex interactions. Existing assistive devices (such as head mice and eye-tracking devices) are expensive, have limited functionality, and are poorly adaptable, making it difficult to meet personalized needs. These devices often lack comprehensive adaptability to various operating modes, requiring users to make numerous repetitive adjustments when faced with complex interaction needs, further increasing the difficulty of use and the learning curve.

[0003] Current technologies are deficient in multimodal data fusion and dynamic adaptability, making it impossible to achieve detailed modeling and real-time adjustment of multimodal data such as force, touch, and motion, making it difficult for devices to take into account both global and local information in complex interactive scenarios. In addition, traditional modeling methods have limited perception of dynamic changes, making it difficult to capture subtle features during rapid operations, resulting in delayed device responses or reduced recognition accuracy. For example, when processing rapid operations, traditional mouse and handle devices lack the ability to perceive dynamic changes, making it difficult to capture subtle features during rapid operations, resulting in delayed device responses or reduced recognition accuracy. In addition, traditional modeling methods have limited perception of dynamic changes and are unable to adjust feature extraction and weight distribution in real time, further affecting device performance and user experience.

[0004] As users place higher demands on intelligent interactive devices, how to enhance the interactive experience through efficient dynamic feature extraction and adaptive optimization strategies has become a key challenge for the development of the industry. Summary of the Invention

[0005] In response to the technical problems in the prior art that users with limited finger control suffer from poor operational flexibility, low precision, and lack of adaptability when using mouse and handle devices, the present invention proposes an intelligent handle mouse control method and system based on a dynamic variable pyramid. By using multimodal sensing technology to collect pressure distribution, touch trajectory, and motion data during user operations, the method extracts multi-scale features by combining a dynamic variable scale pyramid module, a multi-level graph neural network, and a global-local collaborative dynamic weight adjustment framework. This provides users with an efficient and personalized human-computer interaction experience, realizes all the functions of a normal handle and mouse, and provides a more accurate and rich control experience.

[0006] In order to achieve the above object, the technical solution of the present invention is achieved as follows:

[0007] A method for controlling a mouse using an intelligent handle based on a dynamic variable pyramid comprises the following steps:

[0008] Multimodal data acquisition: Based on the multimodal dimensional sensing system, multimodal data of user operations are collected in real time, including pressure perception data, touch path data, and motion trajectory data;

[0009] Data processing and dynamic feature generation: Perform time synchronization correction and normalization on the collected multimodal data to generate a multimodal dynamic feature matrix;

[0010] Dynamic scalable feature extraction: The dynamic scalable pyramid module intelligently decomposes the multimodal dynamic feature matrix to extract global and local multi-level features;

[0011] Dynamic weight optimization: Global and local multi-level features are integrated through a global-local collaborative dynamic weight adjustment mechanism to generate a comprehensive feature matrix;

[0012] Feature modeling and fusion: We build an intelligent modeling system using the spatiotemporal graph neural network (ST-GNN), hypergraph neural network (ST-HyperGNN), and temporal recurrent network (ST-RNN). We use this intelligent modeling system to process the comprehensive feature matrix and form comprehensive operational features for multimodal fusion.

[0013] Output user operation instructions: Generate user operation instructions through comprehensive operation features: including complex interactive instructions such as dragging, clicking, and rotating, to support users with limited finger operation to complete various interactive tasks.

[0014] Preferably, the multimodal dimensional stereoscopic sensing system includes an X-axis potential sensor, a Y-axis potential sensor, a MXene piezoresistive sensor, an acceleration sensor and a touch distribution sensor; pressure sensing data is captured by the MXene piezoresistive sensor, touch path data is recorded by the touch distribution sensor, and motion trajectory data is obtained by the X-axis potential sensor, the Y-axis potential sensor and the acceleration sensor.

[0015] Preferably, the motion trajectory data includes joystick tilt angle and acceleration information.

[0016] Preferably, the method for intelligently decomposing the multimodal dynamic feature matrix through the dynamically variable scale pyramid module is as follows: dynamically adjusting the downsampling step size according to the operation mode: when the feature changes gently, a larger downsampling step size is used to focus on capturing macro trends, such as the overall operation trajectory, global pressure distribution, or continuity of large-scale movements; and when the feature changes dramatically, the system dynamically reduces the downsampling step size to deeply extract the refined features of the operation, such as rapid pressure changes in local areas, small motion paths, or subtle touch trajectories, thereby achieving multi-scale information extraction from global trends to local details;

[0017] Dynamically adjust the downsampling ratio according to the changes in multimodal features:

[0018]

[0019] in, represents the change of the feature matrix of the lth layer, ε is the feature change threshold, α and β are the step size scaling coefficients; represents the eigenvalue of node i at time t-1; represents the eigenvalue of node i at time t;

[0020] Multi-scale feature fusion: Features at each layer are fused through an intelligent weighting mechanism to generate a dynamic feature pyramid that integrates global and local information: H fusion =concat(H local ,H global ), where H fusion Represents the dynamic feature pyramid matrix after fusion; H local represents the local feature matrix; H global Represents the global feature matrix.

[0021] Preferably, the formula of the global-local collaborative dynamic weight adjustment mechanism is as follows:

[0022]

[0023] in, represents the eigenvalue of node i at time t+1; represents the eigenvalue of the adjacent node j at time t; w ij represents the connection weight between nodes i and j; σ(·) represents the activation function; N(i) is the set of neighbor nodes of node i.

[0024] Preferably, the method of processing the comprehensive feature matrix using the intelligent modeling system to form comprehensive operational features of multimodal fusion is:

[0025] First, the local dynamic features of a single sensor node are captured through a spatiotemporal graph neural network. For different modal data, an adjacency matrix and a dynamic feature update mechanism are constructed:

[0026]

[0027] Among them, b i represents the bias term of node i;

[0028] Secondly, a hypergraph neural network is used to model the cross-modal interaction characteristics of multimodal data. Different modal data are used as node inputs to construct a high-order hypergraph structure, where each hyperedge connects multiple modal nodes, representing high-order associations between modalities. The node feature update mechanism is as follows:

[0029]

[0030] Among them, w e is the weight of the hyperedge, j ′ ∈e represents the set of nodes in the hyperedge, E i is the set of hyperedges connected to node i; finally, the temporal recursive network is combined to model the dynamic changes in the time dimension of multimodal features; for the temporal evolution of the node feature matrix, a temporal recursive mechanism is adopted:

[0031] H(t)=f(H(t-1),X(t))

[0032] Among them, H (t) represents the node status at time t, X (t) is the input data at time t, and f is the recursive function.

[0033] Preferably, the global-local collaborative dynamic weight adjustment mechanism includes a feedback adjustment module, a topology analysis module, a hotspot weighting module and a dynamic collaborative multimodal fusion module;

[0034] Feedback adjustment module: Dynamically optimizes the weight distribution of a single modality based on real-time sensor data;

[0035] Topology analysis module: Optimizes the weights of sensor nodes and edges through graph neural networks to strengthen the correlation between local and global data;

[0036] Hotspot weighting module: Through real-time detection of hotspot areas, it increases the weight of key areas and highlights regional features;

[0037] Dynamic collaborative multimodal fusion module: Based on the correlation between global and local modal features, it dynamically optimizes the fusion weights between modalities.

[0038] An intelligent handle mouse control system based on a dynamic variable pyramid comprises: a multimodal dimensional sensing stereoscopic sensing system module, a dynamic weight adjustment module, a processing unit module, a power supply system module, a communication system module, a user interaction and feedback system module, and a handle control system module; the multimodal dimensional sensing stereoscopic sensing system module is respectively connected to the dynamic weight adjustment module, the processing unit module, and the handle control system module; the processing unit module is respectively connected to the power supply system module, the communication system module, and the handle control system module; the power supply system module is connected to the communication system module; and the communication system module is connected to the user interaction and feedback system module.

[0039] Preferably, the multimodal dimensional sensing system module includes an X-axis potential sensor, a Y-axis potential sensor, a MXene piezoresistive sensor, an acceleration sensor and a touch distribution sensor, for collecting multimodal operation data;

[0040] The processing unit module includes a microcontroller for extracting global features, local features and dynamic scale pyramid features;

[0041] The dynamic weight adjustment module is used to dynamically optimize sensor weights, network node connection weights and hotspot area weights to achieve dynamic optimization from local to global;

[0042] The user interaction and feedback system module includes a Web interface, a sound feedback module and an LED indicator light; it provides real-time vibration feedback and status display according to user operation instructions.

[0043] Preferably, the handle control system module includes a 3D printed handle shell, a joystick body, a rebound spring, a vibration feedback module, a universal joint rod structure, and a pressure button assembly; the joystick body, the rebound spring, the universal joint rod structure, and the pressure button assembly enable multiple interaction modes, including complex operations such as dragging, clicking, and rotating; and the vibration feedback module provides real-time feedback;

[0044] The communication system module includes a Bluetooth module and a USB interface; through the Bluetooth module and the USB interface, wireless data transmission and wired connection are supported.

[0045] Beneficial effects of the present invention:

[0046] 1) This paper proposes a dynamically rescalable pyramid module that dynamically adjusts the downsampling step size according to real-time operation changes to achieve accurate extraction of global and local multi-scale features.

[0047] 3) This paper proposes a multi-level graph neural network structure to model local features, global features and inter-modal interactions layer by layer, significantly improving the fusion accuracy of multimodal data and the ability to recognize operational intentions.

[0048] 4) The present invention proposes a global-local collaborative dynamic weight adjustment framework to dynamically optimize sensor modal weights, topological node connection weights, and hotspot area weights to achieve efficient collaboration and refined adjustment between modalities.

[0049] 5) The present invention provides real-time feedback and personalized operation settings through the vibration feedback module and Bluetooth wireless transmission function, thereby improving user experience and operational adaptability.

[0050] 6) The present invention supports multifunctional operation in mouse mode and handle mode to meet different operation scenarios and needs, and provides an efficient, accurate and intelligent human-computer interaction experience for people with finger-impaired operation. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0052] Figure 1 The process of the method of the present invention is provided.

[0053] Figure 2 This is a block diagram of the overall structure of the handle mouse system proposed in the present invention.

[0054] Figure 3 This is a structural diagram of the multimodal data processing and dynamic optimization system of the present invention.

[0055] Figure 4 This is a structural diagram of the global-local collaborative dynamic weight adjustment framework of the present invention.

[0056] Figure 5 This is a schematic diagram of the operating mode of the handle mouse proposed in the present invention. DETAILED DESCRIPTION

[0057] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without creative work are within the scope of protection of the present invention.

[0058] Example 1, as Figure 1As shown in the figure, a method for controlling an intelligent handle mouse based on a dynamic variable pyramid includes the following steps: multimodal data acquisition, data preprocessing and dynamic variable scale feature decomposition, local and global feature modeling, cross-modal feature fusion, operation intention prediction, and real-time feedback generation. This method analyzes user operation behavior in real time by fusing multimodal data collected by force sensors, position sensors, acceleration sensors, and touch distribution sensors, accurately identifies complex operation intentions, and provides efficient feedback and dynamic adjustments. The method includes the following steps:

[0059] Multimodal data acquisition: Based on the multimodal dimensional sensing system, multimodal data of user operations are collected in real time, including pressure perception data, touch path data and motion trajectory data; the multimodal dimensional sensing system includes an X-axis potential sensor 801, a Y-axis potential sensor 802, a MXene piezoresistive sensor 803, an acceleration sensor 804 and a touch distribution sensor 805; pressure perception data is captured by the MXene piezoresistive sensor 803, touch path data is recorded by the touch distribution sensor 805, and motion trajectory data is obtained by the X-axis potential sensor 801, the Y-axis potential sensor 802 and the acceleration sensor 804; the motion trajectory data includes the joystick tilt angle and acceleration information.

[0060] The multimodal signal acquisition method of the present invention achieves efficient acquisition, time synchronization correction, and fusion calculation of multi-sensor data, ultimately providing support for dynamic feedback and user operation recognition. The overall process includes data acquisition, spatiotemporal synchronization, feature fusion, and feedback adjustment. It aims to solve the spatiotemporal inconsistency problem caused by asynchronous sensor sampling and improve the system's recognition accuracy in complex operation scenarios through feature fusion. The system signal acquisition includes three types of data: pressure distribution data, motion posture data, and touch path data. The specific implementation is as follows:

[0061] 1. Pressure distribution data

[0062] The pressure distribution data is collected by the pressure sensor matrix at the bottom and side of the handle, and a three-axis pressure sensor (such as Honeywell FSS-SMT series) is used to obtain three-dimensional force data F. x ,F y ,F z The output matrix P(x,y) of the sensor matrix is ​​defined as follows: Each matrix element Fij represents the three-axis force data of the sensor, and the sampling frequency fp = 100 Hz.

[0063] 2. Motion posture data

[0064] Motion posture data is collected by the inertial measurement unit (IMU), which provides angular velocity ω x ,ω y ,ωz and acceleration a x ,z y ,a z , output motion trajectory matrix T(t): The IMU sampling frequency fm = 200 Hz is used to capture the user's rapid rotation and acceleration events.

[0065] 3. Touch path data

[0066] The touch path data is collected by the touch sensor array on the surface of the handle (such as Synaptics T1000), which outputs the touch path matrix S(u,v,t). The touch path sampling frequency ft = 120Hz is used for multi-touch and path tracking.

[0067] Data processing and dynamic feature generation: The collected multimodal data is time-synchronized and normalized to generate a multimodal dynamic feature matrix, providing basic input for subsequent modeling.

[0068] In order to ensure the time and space consistency of various signal data, the system uses the time and space synchronization module to perform time correction. sync Defined as: Among them, the IMU provides a global time reference to ensure that the data of each modality is matched and fused on a unified time axis. Based on the spatiotemporal synchronization, the system realizes the fusion calculation of data through the multimodal fusion module. The fusion weight is dynamically allocated by the attention mechanism. The formula is as follows: Among them, α i′ represents the fusion weight of the i′th modality, Q i′ represents the query vector of the i′th modality, K i′ represents the key vector of the i′th mode; the final fusion result is: Among them, H sync,i′ It represents the feature matrix after the synchronization of the i′th modality, which integrates the spatiotemporal characteristics of pressure, motion, and touch data and can accurately reflect the user's operation status.

[0069] The sensitivity of each mode is dynamically adjusted based on the feedback module. For pressure-motion feedback, the dynamic adjustment formula is: For touch-pressure feedback, the sensitivity adjustment formula is: Among them, γ and β are dynamic adjustment coefficients, The pressure gradient reflects the directional force applied by the user. Through the above steps, the present invention realizes the collection, synchronization, fusion and dynamic adjustment of multimodal data. The system can recognize the user's operation intention in real time and optimize the feedback control effect.

[0070] Dynamic scalable feature extraction: The dynamic scalable pyramid module intelligently decomposes the multimodal dynamic feature matrix to extract global and local multi-level features;

[0071] The downsampling step size is dynamically adjusted according to the operation mode: when the feature changes smoothly, the system adopts a larger downsampling step size to focus on capturing macro trends, such as the overall operation trajectory, global pressure distribution, or the continuity of large-scale movements; when the feature changes drastically, the system dynamically reduces the downsampling step size to deeply extract the refined features of the operation, such as rapid pressure changes in local areas, small motion paths, or subtle touch trajectories, thereby realizing multi-scale information extraction from global trends to local details.

[0072] Multi-scale feature fusion: Features at each layer are fused through an intelligent weighting mechanism to generate a dynamic feature pyramid that integrates global and local information: H fusion =concat(H local ,H global ). Among them, H fusion Represents the dynamic feature pyramid matrix after fusion; H local represents the local feature matrix; H global Represents the global feature matrix.

[0073] like Figure 3 As shown in the figure, the system enters the dynamic variable scale pyramid feature decomposition module, dynamically adjusts the downsampling step of the feature according to the operation mode, and intelligently decomposes the multimodal features. The system calculates the feature change ΔH l To judge the intensity of features at different levels, and dynamically adjust the downsampling step size s l The specific definitions are as follows:

[0074]

[0075] in, Indicates the change in the feature matrix of the first layer, ∈ is the feature change threshold, and α and β are the step size scaling coefficients. In specific applications, for example:

[0076] (1) When ΔHl=0.05<∈=0.1, the step size is increased to 4 to capture global features;

[0077] (2) When ΔHl=0.2>∈, the step size is reduced to 1 and fine-grained local features are extracted.

[0078] Dynamic weight optimization: Through the global-local collaborative dynamic weight adjustment mechanism, global and local multi-level features are integrated to generate a comprehensive feature matrix, realizing feature modeling and adaptation from global to local.

[0079] The formula of the global-local collaborative dynamic weight adjustment mechanism is as follows:

[0080]

[0081] in, represents the eigenvalue of node i at time t+1; represents the eigenvalue of the adjacent node j at time t; w ij represents the connection weight between nodes i and j; σ(·) represents the activation function; N(i) is the set of neighbor nodes of node i.

[0082] The multi-scale features obtained by decomposition are fused through a dynamic weighting mechanism to generate a comprehensive feature matrix H fusion : In actual applications, for example, when the user performs a "drag + rotate" operation, the system automatically reduces the downsampling step size to extract fine-grained touch path and force distribution features, dynamically allocates weights to focus on hot spots, and ensures accurate recognition of complex operation modes.

[0083] Feature modeling and fusion: Figure 4 As shown in the figure, in the cross-modal modeling stage, an intelligent modeling system is constructed through the spatiotemporal graph neural network (ST-GNN), hypergraph neural network (ST-HyperGNN) and temporal recurrent network (ST-RNN). The intelligent modeling system is used to process the comprehensive feature matrix to form comprehensive operational features of multimodal fusion; dynamic modeling of multimodal data, feature fusion and accurate identification of operation modes are realized.

[0084] First, the local dynamic features of a single sensor node are captured through a spatiotemporal graph neural network (ST-GNN). For different modal data (pressure, motion, touch), an adjacency matrix and a dynamic feature update mechanism are constructed:

[0085]

[0086] in, represents the eigenvalue of neighbor node j at time t, b i represents the bias term of node i, w ij represents the edge weight between node i and its adjacent node j.

[0087] Through the above mechanism, the spatiotemporal graph neural network can capture the spatiotemporal dynamic characteristics of different sensor nodes and generate a spatiotemporal-related node feature matrix to describe the dynamic response behavior of each modal data.

[0088] Secondly, a hypergraph neural network (ST-HyperGNN) is used to model the cross-modal interaction characteristics of multimodal data. Different modal data (pressure, motion, touch) are used as node inputs to construct a high-order hypergraph structure, where each hyperedge connects multiple modal nodes, representing high-order associations between modalities. The node feature update mechanism is as follows:

[0089]

[0090] Among them, w e is the weight of the hyperedge, j′∈e represents the node set in the hyperedge, E i is the set of hyperedges connected to node i; through hypergraph modeling, high-order interaction relationship modeling between modalities is realized, and the cross-modal weight distribution is dynamically optimized to enhance modal synergy.

[0091] Finally, the temporal recurrent network (ST-RNN) is combined to model the dynamic changes in the time dimension of multimodal features. A temporal recursive mechanism is used for the temporal evolution of the node feature matrix:

[0092] H(t)=f(H(t-1),X(t))

[0093] Among them, H (t) represents the node status at time t, X (t) is the input data at time t, and f is a recursive function. Through time series modeling, we can capture the temporal dependencies of different modal operation characteristics and accurately identify the dynamic changes in operation modes.

[0094] In a multi-layer graph neural network, the first layer of the network is used to extract local operation features: extracting local information during the operation process, including the pressure distribution gradient, the short-term change rate of the motion trajectory, and the curvature characteristics of the touch path, capturing fine-grained operation patterns, and generating a local feature matrix. The second layer of the network is used for global feature modeling: the local feature matrix is ​​input into the second layer of the hybrid graph neural network, integrating data from different sensor modalities, and extracting overall trend information, including the global change pattern of pressure distribution and the continuity characteristics of the motion trajectory, for analyzing the macro pattern of the operation. The third layer of the network is used for modal interaction modeling: the global features generated by the second layer are input into the third layer of the hybrid graph neural network to model the interaction relationship between multiple modalities, combining global features with local features to generate comprehensive features of multimodal fusion, which are used to accurately describe the user's operation intentions.

[0095] To optimize the weight distribution of sensor modal fusion, the present invention proposes a global-local collaborative dynamic weight adjustment framework, which includes a feedback adjustment module, a topology analysis module, a hotspot weighting module, and a dynamic collaboration module. The feedback adjustment module dynamically optimizes the weight distribution of single modalities (force, motion, touch) according to changes in sensor data, improving the accuracy of single modalities; the topology analysis module optimizes the weights of sensor nodes and edges through graph neural networks, strengthening the correlation between local and global data; the hotspot weighting module increases the weight of key areas by real-time detection of operation hotspot areas, highlighting regional characteristics; the dynamic collaboration module optimizes multimodal fusion weights based on task requirements, achieving efficient collaboration of the three modalities of force, motion, and touch.

[0096] Output user operation instructions: Generate user operation instructions through comprehensive operation features, support multiple interaction methods, including complex interaction instructions such as dragging, clicking, and rotating, and support users who are not convenient to operate with fingers to complete various interaction tasks.

[0097] like Figure 5 As shown in the figure, the system generates real-time feedback through the operation intention prediction module, outputting commands such as click, drag, and rotation to the user interaction system. At the same time, the dynamic feedback generation module is combined to optimize the operation sensitivity and weight distribution. For example, when the system detects a significant increase in pressure in a specific area, the sensitivity is enhanced by increasing the hotspot weight to ensure operation accuracy. In addition, the feedback results can be transmitted to the user end via the Bluetooth module, providing real-time feedback through the web interface, LED lights, or vibration modules to enhance the user experience.

[0098] Example 2, as Figure 2 As shown, an intelligent handle mouse control system based on a dynamic variable pyramid includes: a multimodal dimensional sensing stereoscopic sensing system module, a dynamic weight adjustment module, a processing unit module, a power supply system module, a communication system module, a user interaction and feedback system module, and a handle control system module; the multimodal dimensional sensing stereoscopic sensing system module is respectively connected to the dynamic weight adjustment module, the processing unit module, and the handle control system module; the processing unit module is respectively connected to the power supply system module, the communication system module, and the handle control system module; the power supply system module is connected to the communication system module; and the communication system module is connected to the user interaction and feedback system module.

[0099] The multimodal dimensional sensing system module includes an X-axis potential sensor 801, a Y-axis potential sensor 802, a MXene piezoresistive sensor 803, an acceleration sensor 804, and a touch distribution sensor 805 for collecting multimodal operation data. The processing unit module includes a microcontroller 401 for extracting global features, local features, and dynamic scale pyramid features.

[0100] The dynamic scale-adjustable pyramid feature is implemented through the following mechanisms:

[0101] (1) Dynamically adjust the downsampling step size according to the rate of change of input data;

[0102] (2) Dynamically assign weights to local, intermediate, and global scales to prioritize key operational areas;

[0103] (3) Realize dynamic scale selection and switching, actively perceive data features, and perform local to global feature transfer and fusion.

[0104] The dynamic weight adjustment module is used to dynamically optimize sensor weights, network node connection weights, and hotspot area weights, achieving local optimization and refined feature fusion under global guidance. The user interaction and feedback system module includes a web interface 601, an audio feedback module 602, and an LED indicator 603. Users can use the web interface 601 or a mobile app to configure operations and monitor status, allowing them to flexibly adjust system parameters. Real-time vibration feedback and status display are provided based on user operation instructions.

[0105] The handle control system module includes a 3D-printed handle housing 701, a joystick body 702, a rebound spring 703, a vibration feedback module 704, a universal joint rod structure 705, and a pressure button assembly 706. The joystick body 702, rebound spring 703, universal joint rod structure 705, and pressure button assembly 706 enable multiple interaction methods, including complex operations such as dragging, clicking, and rotating. The vibration feedback module 704 provides real-time feedback, enhancing the operational experience and interaction accuracy. Users can freely switch between mouse mode and handle mode according to scenario requirements, and the system automatically adapts to parameter configurations in different modes. The communication system module includes a Bluetooth module 501 and a USB interface 502. Through the Bluetooth module 501 and USB interface 502, both wireless data transmission and wired connection are supported.

[0106] The system also integrates a collaborative reinforcement learning module to optimize feature modeling and weight adjustment strategies based on real-time and historical user operation data. Through continuous adaptive optimization, the system generates personalized operating parameter configurations and provides users with operational efficiency analysis reports, achieving long-term optimization and personalized interactive support.

[0107] Specific example: Real-time recognition and response to "rapid rotation and frequent pressing" operations. The following is a complete example of system processing.

[0108] 1. Data collection and time synchronization

[0109] The user holds the control handle to operate, and the system collects pressure, IMU motion data, and touch path data in real time:

[0110] (1) Pressure data P(x,y): The pressure distribution of the user's palm is recorded through the bottom pressure sensor matrix, with a resolution of 0.01N and a sampling frequency fp = 100Hz.

[0111] (2) Motion data T(t): The IMU sensor captures the rotational angular velocity and linear acceleration of the handle in real time, with a sampling frequency of fm = 200 Hz.

[0112] (3) Touch data S(u, v, t): The touch sensor records the finger sliding path and pressing position with a resolution of 0.1 mm and a sampling frequency of ft = 120 Hz.

[0113] Since the sampling time of each sensor is different, the system uses the time synchronization correction module to align the data in time and space to generate a unified time and space synchronization matrix H sync .

[0114] 2. Dynamic Variable Scale Pyramid Feature Extraction

[0115] Based on synchronous data H sync , the system uses a dynamic and rescalable pyramid module to extract multi-level features:

[0116] (1) Detect feature changes: Calculate the feature change ΔH for each layer l , used to determine the intensity of user operations: For example, in this example, when the user performs a fast rotation operation, the angular velocity data changes dramatically, resulting in ΔH l =0.3>∈=0.1.

[0117] (2) Dynamically adjust the step size: Dynamically adjust the downsampling step size according to the feature change

[0118] As ΔH l When β = 0.3, set β = 0.5, then sl = 1. The system captures fine-grained local features and focuses on subtle operational changes during the rotation process.

[0119] 3. Graph Neural Network (GNN) Modeling and Operation Recognition

[0120] The system inputs multi-layer features into the graph neural network to perform global and local feature modeling:

[0121] (1) Local feature modeling: ST-GNN captures the short-term spatial correlation between nodes and identifies that the pressure points are concentrated near (x5, y5).

[0122] (2) Global feature analysis: Through topological analysis, the system identifies drastic changes in angular velocity accompanied by frequent touch path oscillations.

[0123] (3) Cross-modal fusion: ST-HyperGNN fuses pressure, motion, and touch features and outputs a comprehensive feature matrix Hfusion of the operation mode:

[0124] Through the above modeling, the system accurately identifies the user's operation mode as "rapid rotation and frequent pressing".

[0125] 4. Feedback adjustment and real-time response

[0126] After identifying "rapid rotation and frequent pressing", the system dynamically adjusts the sensitivity through the feedback module:

[0127] (1) Motion sensitivity feedback: sensitivity is adjusted by combining pressure gradient and rotational angular velocity: Current

[0128] Then w adjusted =ω+0.32

[0129] (2) Touch sensitivity feedback: Dynamically adjust the touch response according to pressure changes: When P(x5,y5)=2.0, the touch sensitivity increases to u adjusted =u+1.0. The system also uses LED flashing and vibration feedback to remind users of changes in operation accuracy and force.

[0130] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A method for controlling a mouse using an intelligent handle based on a dynamic variable pyramid, characterized in that: The following steps are involved: Multimodal data acquisition: Based on the multimodal dimensional sensing system, multimodal data of user operations are collected in real time, including pressure perception data, touch path data, and motion trajectory data; Data processing and dynamic feature generation: Perform time synchronization correction and normalization on the collected multimodal data to generate a multimodal dynamic feature matrix; Dynamic scalable feature extraction: The dynamic scalable pyramid module intelligently decomposes the multimodal dynamic feature matrix to extract global and local multi-level features; Dynamic weight optimization: Global and local multi-level features are integrated through a global-local collaborative dynamic weight adjustment mechanism to generate a comprehensive feature matrix; Feature modeling and fusion: Build an intelligent modeling system through spatiotemporal graph neural networks, hypergraph neural networks, and temporal recursive networks. Use the intelligent modeling system to process the comprehensive feature matrix to form comprehensive operational features for multimodal fusion. Output user operation instructions: Generate user operation instructions through comprehensive operation features, including complex interactive instructions such as dragging, clicking, and rotating, to support users with limited finger operation to complete various interactive tasks; The method for intelligently decomposing a multimodal dynamic feature matrix using a dynamically scalable pyramid module comprises: dynamically adjusting the downsampling step size based on the operation mode: when feature changes are gentle, the downsampling step size is used to focus on capturing macro trends, including the overall operation trajectory, global pressure distribution, or the continuity of large-scale movements; whereas, when feature changes are drastic, the system dynamically reduces the downsampling step size to deeply extract refined features of the operation, including rapid pressure changes in local areas, small motion paths, or subtle touch trajectories, thereby achieving multi-scale information extraction from global trends to local details; Dynamically adjust the downsampling ratio according to the changes in multimodal features: in, represents the change of the feature matrix of the lth layer, ε is the feature change threshold, α and β are the step size scaling coefficients; represents the eigenvalue of node i at time t-1; represents the eigenvalue of node i at time t; Multi-scale feature fusion: Features at each layer are fused through an intelligent weighting mechanism to generate a dynamic feature pyramid that integrates global and local information: H fusion =concat(H local ,H global ), where H fusion Represents the dynamic feature pyramid matrix after fusion; H local represents the local feature matrix; H global represents the global feature matrix; The formula of the global-local collaborative dynamic weight adjustment mechanism is as follows: in, represents the eigenvalue of node i at time t+1; represents the eigenvalue of the adjacent node j at time t; w ij represents the connection weight between nodes i and j; σ(·) represents the activation function; N(i) is the set of neighbor nodes of node i; The method of using the intelligent modeling system to process the comprehensive feature matrix to form comprehensive operational features of multimodal fusion is: First, the local dynamic features of a single sensor node are captured through a spatiotemporal graph neural network. For different modal data, an adjacency matrix and a dynamic feature update mechanism are constructed: Among them, b i represents the bias term of node i; Secondly, a hypergraph neural network is used to model the cross-modal interaction characteristics of multimodal data. Different modal data are used as node inputs to construct a high-order hypergraph structure, where each hyperedge connects multiple modal nodes, representing high-order associations between modalities. The node feature update mechanism is as follows: Among them, w e is the weight of the hyperedge, j′∈e represents the node set in the hyperedge, E i is the set of hyperedges connected to node i; finally, the temporal recursive network is combined to model the dynamic changes in the time dimension of multimodal features; for the temporal evolution of the node feature matrix, a temporal recursive mechanism is adopted: H(t)=f(H(t-1),X(t)) Among them, H (t) represents the node status at time t, X (t) is the input data at time t, and f is the recursive function.

2. The intelligent handle mouse control method based on a dynamic variable pyramid according to claim 1, characterized in that: The multimodal dimensional stereoscopic sensing system includes an X-axis potential sensor (801), a Y-axis potential sensor (802), a MXene piezoresistive sensor (803), an acceleration sensor (804) and a touch distribution sensor (805); pressure sensing data is captured by the MXene piezoresistive sensor (803), touch path data is recorded by the touch distribution sensor (805), and motion trajectory data is obtained by the X-axis potential sensor (801), the Y-axis potential sensor (802) and the acceleration sensor (804).

3. The intelligent handle mouse control method based on a dynamic variable pyramid according to claim 1 or 2, characterized in that: The motion trajectory data includes the joystick tilt angle and acceleration information.

4. The intelligent handle mouse control method based on dynamic variable pyramid according to claim 1, characterized in that: The global-local collaborative dynamic weight adjustment mechanism includes a feedback adjustment module, a topology analysis module, a hotspot weighting module and a dynamic collaborative multimodal fusion module; Feedback adjustment module: Dynamically optimizes the weight distribution of a single modality based on real-time sensor data; Topology analysis module: Optimizes the weights of sensor nodes and edges through graph neural networks to strengthen the correlation between local and global data; Hotspot weighting module: Through real-time detection of hotspot areas, it increases the weight of key areas and highlights regional features; Dynamic collaborative multimodal fusion module: Based on the correlation between global and local modal features, it dynamically optimizes the fusion weights between modalities.

5. An intelligent handle mouse control system based on a dynamic variable pyramid, used to implement the method according to any one of claims 1 to 4, characterized in that: include: Multimodal dimensional sensing stereoscopic sensing system module, dynamic weight adjustment module, processing unit module, power supply system module, communication system module, user interaction and feedback system module and handle control system module; the multimodal dimensional sensing stereoscopic sensing system module is respectively connected to the dynamic weight adjustment module, the processing unit module and the handle control system module, the processing unit module is respectively connected to the power supply system module, the communication system module and the handle control system module, the power supply system module is connected to the communication system module, and the communication system module is connected to the user interaction and feedback system module.

6. The intelligent handle mouse control system based on a dynamic variable pyramid according to claim 5, characterized in that: The multimodal dimensional sensing system module includes an X-axis potential sensor (801), a Y-axis potential sensor (802), a MXene piezoresistive sensor (803), an acceleration sensor (804) and a touch distribution sensor (805), and is used to collect multimodal operation data; The processing unit module includes a microcontroller (401) for extracting global features, local features and dynamic scale pyramid features; The dynamic weight adjustment module is used to dynamically optimize sensor weights, network node connection weights and hotspot area weights to achieve dynamic optimization from local to global; The user interaction and feedback system module comprises a Web interface (601), a sound feedback module (602) and an LED indicator light (603); and provides real-time vibration feedback and status display according to user operation instructions.

7. The intelligent handle mouse control system based on a dynamic variable pyramid according to claim 6, characterized in that: The handle control system module includes a 3D printed handle housing (701), a rocker body (702), a rebound spring (703), a vibration feedback module (704), a universal joint rod structure (705), and a pressure button assembly (706); a variety of interaction modes are realized through the rocker body (702), the rebound spring (703), the universal joint rod structure (705), and the pressure button assembly (706), including complex operations such as dragging, clicking, and rotating; Providing real-time feedback via a vibration feedback module (704); The communication system module comprises a Bluetooth module (501) and a USB interface (502); wireless data transmission and wired connection are supported via the Bluetooth module (501) and the USB interface (502).

Images