Intelligent handle mouse control method and system based on dynamic variable pyramid

Abstract

The invention provides an intelligent handle mouse control method and system based on a dynamic variable pyramid, and the system integrates a multi-modal three-dimensional sensing system module, collects pressure distribution, a touch track and rocker motion data in real time, and dynamically generates a multi-modal feature matrix; a dynamic variable-scale pyramid module is introduced, local and global features are extracted through real-time weight adjustment, down-sampling step length, dynamic scale switching and the like, and multi-scale information fusion and precise adaptation are achieved; and a multi-level neural network model is adopted to establish an interaction relationship between modals, multi-scale features are fused, and a complex operation mode is accurately identified. Based on real-time operation and historical data, a personalized feedback and optimization strategy is generated, and the interaction efficiency and adaptability are remarkably improved; obstacles of people with inconvenient fingers in operation of traditional equipment are effectively solved, and intelligent and efficient man-machine interaction control is achieved.

Description

Technical Field

[0001] The present invention relates to the technical field of intelligent auxiliary equipment and human-computer interaction, and in particular to an intelligent handle mouse control method and system based on a dynamically variable pyramid. Background Art

[0002] Existing mouse and handle devices require high hand flexibility, and people with limited finger control have difficulty achieving precise control and complex interactions when using them. Existing auxiliary devices (such as head mice and eye tracking devices) are expensive, have single functions, poor adaptability, and are difficult to meet personalized needs. Such devices usually lack the ability to fully adapt to operating modes, and users need to make a lot of repetitive adjustments when faced with complex interaction needs, which further increases the difficulty of use and learning costs.

[0003] Current technologies are deficient in multimodal data fusion and dynamic adaptability, and are unable to achieve detailed modeling and real-time adjustment of multimodal data such as force, touch, and motion, making it difficult for devices to take into account both global and local information in complex interactive scenarios. In addition, traditional modeling methods have limited perception of dynamic changes, making it difficult to capture subtle features during rapid operations, resulting in delayed device responses or reduced recognition accuracy. For example, when traditional mouse and handle devices are processing rapid operations, they lack the ability to perceive dynamic changes, making it difficult to capture subtle features during rapid operations, resulting in delayed device responses or reduced recognition accuracy. In addition, traditional modeling methods have limited perception of dynamic changes, making it impossible to adjust feature extraction and weight distribution in real time, further affecting device performance and user experience.

[0004] As users place higher demands on intelligent interactive devices, how to improve the interactive experience through efficient dynamic feature extraction and adaptive optimization strategies has become a key challenge for the development of the industry. Summary of the invention

[0005] In view of the technical problems in the prior art that users with disabled fingers have poor operating flexibility, low precision and lack of adaptability when using mouse and handle devices, the present invention proposes an intelligent handle mouse control method and system based on a dynamic variable pyramid. The pressure distribution, touch trajectory and motion data of the user's operation are collected through multimodal sensing technology, and multi-scale features are extracted by combining a dynamic variable scale pyramid module, a multi-level graph neural network and a global-local collaborative dynamic weight adjustment framework, so as to provide users with an efficient and personalized human-computer interaction experience, realize all the functions of a normal handle and mouse, and provide a more accurate and rich control experience.

[0006] In order to achieve the above object, the technical solution of the present invention is achieved as follows:

[0007] A method for controlling a mouse using an intelligent handle based on a dynamic variable pyramid comprises the following steps:

[0008] Multimodal data collection: Based on the multimodal dimensional sensing system, multimodal data of user operation is collected in real time, including pressure perception data, touch path data and motion trajectory data;

[0009] Data processing and dynamic feature generation: Perform time synchronization correction and normalization on the collected multimodal data to generate a multimodal dynamic feature matrix;

[0010] Dynamic scalable feature extraction: The multimodal dynamic feature matrix is ​​intelligently decomposed through the dynamic scalable pyramid module to extract global and local multi-level features;

[0011] Dynamic weight optimization: The global and local multi-level features are integrated through the global-local collaborative dynamic weight adjustment mechanism to generate a comprehensive feature matrix;

[0012] Feature modeling and fusion: An intelligent modeling system is constructed through spatiotemporal graph neural network (ST-GNN), hypergraph neural network (ST-HyperGNN) and temporal recurrent network (ST-RNN). The intelligent modeling system is used to process the comprehensive feature matrix to form comprehensive operational features of multi-modal fusion;

[0013] Output user operation instructions: Generate user operation instructions through comprehensive operation features: including complex interactive instructions such as dragging, clicking, and rotating, to support users with limited finger operation to complete various interactive tasks.

[0014] Preferably, the multimodal dimensional stereoscopic sensing system includes an X-axis potential sensor, a Y-axis potential sensor, a MXene piezoresistive sensor, an acceleration sensor and a touch distribution sensor; pressure sensing data is captured by the MXene piezoresistive sensor, touch path data is recorded by the touch distribution sensor, and motion trajectory data is acquired by the X-axis potential sensor, the Y-axis potential sensor and the acceleration sensor.

[0015] Preferably, the motion trajectory data includes joystick tilt angle and acceleration information.

[0016] Preferably, the method of intelligently decomposing the multimodal dynamic feature matrix through the dynamic variable scale pyramid module is: dynamically adjusting the downsampling step size according to the operation mode: when the feature changes slowly, a larger downsampling step size is used to focus on capturing macro trends, such as the overall operation trajectory, global pressure distribution, or the continuity of a large range of actions; and when the feature changes dramatically, the system dynamically reduces the downsampling step size to deeply extract the refined features of the operation, such as the rapid pressure changes in the local area, the small motion path, or the subtle touch trajectory, thereby realizing multi-scale information extraction from global trends to local details;

[0017] Dynamically adjust the downsampling ratio according to the changes in multimodal features:

[0018]

[0019] in, represents the change of the feature matrix of the lth layer, ε is the feature change threshold, α and β are the step size scaling coefficients; represents the eigenvalue of node i at time t-1; represents the characteristic value of node i at time t;

[0020] Multi-scale feature fusion: Features at each layer are fused through an intelligent weighting mechanism to generate a dynamic feature pyramid that integrates global and local information: fusion =concat(H local ,H global ), where H fusion represents the dynamic feature pyramid matrix after fusion; H local represents the local feature matrix; H global Represents the global feature matrix.

[0021] Preferably, the formula of the global-local collaborative dynamic weight adjustment mechanism is as follows:

[0022]

[0023] in, represents the eigenvalue of node i at time t+1; represents the eigenvalue of the adjacent node j at time t; w ij represents the connection weight between nodes i and j; σ(·) represents the activation function; N(i) is the set of neighbor nodes of node i.

[0024] Preferably, the method of processing the comprehensive feature matrix using the intelligent modeling system to form comprehensive operation features of multi-modal fusion is:

[0025] First, the local dynamic features of a single sensor node are captured through a spatiotemporal graph neural network. For different modal data, an adjacency matrix and a dynamic feature update mechanism are constructed:

[0026]

[0027] Among them, b i represents the bias term of node i;

[0028] Secondly, a hypergraph neural network is used to model the cross-modal interaction characteristics of multimodal data. Different modal data are used as node inputs to construct a high-order hypergraph structure, in which each hyperedge connects multiple modal nodes, indicating high-order associations between modalities. The node feature update mechanism is as follows:

[0029]

[0030] Among them, w e is the weight of the hyperedge, j ′ ∈e represents the set of nodes in the hyperedge, E i is the set of hyperedges connected to node i; finally, the temporal recursive network is combined to model the dynamic changes in the time dimension of multimodal features; for the temporal evolution of the node feature matrix, the temporal recursive mechanism is adopted:

[0031] H(t)=f(H(t-1),X(t))

[0032] Among them, H (t) represents the node state at time t, X (t) is the input data at time t, and f is the recursive function.

[0033] Preferably, the global-local collaborative dynamic weight adjustment mechanism includes a feedback adjustment module, a topology analysis module, a hotspot weighting module and a dynamic collaborative multimodal fusion module;

[0034] Feedback adjustment module: dynamically optimizes the weight distribution of a single mode based on real-time sensor data;

[0035] Topology analysis module: optimizes the weights of sensor nodes and edges through graph neural networks to strengthen the correlation between local and global data;

[0036] Hotspot weighting module: through real-time detection of hotspot areas, increase the weight of key areas and highlight regional features;

[0037] Dynamic collaborative multimodal fusion module: Dynamically optimizes the fusion weights between modalities based on the correlation between global and local modal features.

[0038] An intelligent handle mouse control system based on a dynamic variable pyramid comprises: a multi-modal dimensional sensing stereoscopic sensing system module, a dynamic weight adjustment module, a processing unit module, a power system module, a communication system module, a user interaction and feedback system module and a handle control system module; the multi-modal dimensional sensing stereoscopic sensing system module is respectively connected to the dynamic weight adjustment module, the processing unit module and the handle control system module, the processing unit module is respectively connected to the power system module, the communication system module and the handle control system module, the power system module is connected to the communication system module, and the communication system module is connected to the user interaction and feedback system module.

[0039] Preferably, the multi-modal dimensional sensing system module includes an X-axis potential sensor, a Y-axis potential sensor, a MXene piezoresistive sensor, an acceleration sensor and a touch distribution sensor, which are used to collect multi-modal operation data;

[0040] The processing unit module includes a microcontroller for extracting global features, local features and dynamic scale pyramid features;

[0041] The dynamic weight adjustment module is used to dynamically optimize sensor weights, network node connection weights and hotspot area weights to achieve dynamic optimization from local to global;

[0042] The user interaction and feedback system module includes a Web interface, a sound feedback module and an LED indicator light; it provides real-time vibration feedback and status display according to user operation instructions.

[0043] Preferably, the handle control system module includes a 3D printed handle shell, a rocker body, a rebound spring, a vibration feedback module, a universal joint rod structure, and a pressure button assembly; a variety of interaction modes are realized through the rocker body, the rebound spring, the universal joint rod structure, and the pressure button assembly, including complex operations such as dragging, clicking, and rotating; and real-time feedback is provided through the vibration feedback module;

[0044] The communication system module includes a Bluetooth module and a USB interface; through the Bluetooth module and the USB interface, wireless data transmission and wired connection are supported.

[0045] Beneficial effects of the present invention:

[0046] 1) The present invention proposes a dynamically variable scale pyramid module, which dynamically adjusts the downsampling step size according to real-time operation changes to achieve accurate extraction of global and local multi-scale features.

[0047] 3) The present invention proposes a multi-level graph neural network structure, which models local features, global features and inter-modal interactions layer by layer, significantly improving the fusion accuracy of multimodal data and the ability to recognize operational intentions.

[0048] 4) The present invention proposes a global-local collaborative dynamic weight adjustment framework to dynamically optimize sensor modal weights, topological node connection weights and hot spot area weights to achieve efficient collaboration and refined adjustment between modalities.

[0049] 5) The present invention provides real-time feedback and personalized operation settings through a vibration feedback module and Bluetooth wireless transmission function, thereby improving user experience and operational adaptability.

[0050] 6) The present invention supports multi-functional operation of mouse mode and handle mode to meet different operation scenarios and needs, and provides efficient, accurate and intelligent human-computer interaction experience for people with limited finger operation. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0052] Figure 1 The process of the method of the present invention is provided.

[0053] Figure 2 This is the overall structural block diagram of the handle mouse system proposed by the present invention.

[0054] Figure 3 This is a structural diagram of the multimodal data processing and dynamic optimization system of the present invention.

[0055] Figure 4 This is a structural diagram of the global-local collaborative dynamic weight adjustment framework of the present invention.

[0056] Figure 5 This is a schematic diagram of the operation mode of the handle mouse proposed in the present invention. DETAILED DESCRIPTION

[0057] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0058] Embodiment 1, as Figure 1As shown, a method for controlling an intelligent handle mouse based on a dynamic variable pyramid includes the steps of multimodal data collection, data preprocessing and dynamic variable scale feature decomposition, local and global feature modeling, cross-modal feature fusion, operation intention prediction, and real-time feedback generation. This method analyzes user operation behavior in real time, accurately identifies complex operation intentions, and provides efficient feedback and dynamic adjustment by fusing multimodal data collected by force sensors, position sensors, acceleration sensors, and touch distribution sensors. It includes the following steps:

[0059] Multimodal data acquisition: Based on the multimodal dimensional sensing system, multimodal data of user operation is collected in real time, including pressure perception data, touch path data and motion trajectory data; the multimodal dimensional sensing system includes an X-axis potential sensor 801, a Y-axis potential sensor 802, a MXene piezoresistive sensor 803, an acceleration sensor 804 and a touch distribution sensor 805; pressure perception data is captured by the MXene piezoresistive sensor 803, touch path data is recorded by the touch distribution sensor 805, and motion trajectory data is obtained by the X-axis potential sensor 801, the Y-axis potential sensor 802 and the acceleration sensor 804; the motion trajectory data includes the tilt angle of the joystick and acceleration information.

[0060] The multimodal signal acquisition method of the present invention realizes the efficient acquisition, time synchronization correction and fusion calculation of multi-sensor data, and finally provides support for dynamic feedback and user operation recognition. The overall process includes data acquisition, time and space synchronization, feature fusion and feedback adjustment, aiming to solve the problem of data time and space inconsistency caused by asynchronous sampling of sensors, and improve the recognition accuracy of the system in complex operation scenarios through feature fusion. The acquisition of system signals includes three types of data: pressure distribution data, motion posture data and touch path data. The specific implementation is as follows:

[0061] 1. Pressure distribution data

[0062] The pressure distribution data is collected by the pressure sensor matrix at the bottom and side of the handle, and a three-axis pressure sensor (such as Honeywell FSS-SMT series) is used to obtain three-dimensional force data F x ,F y ,F z The output matrix P(x,y) of the sensor matrix is ​​defined as follows: Each matrix cell Fij represents the three-axis force data of the sensor, and the sampling frequency fp = 100 Hz.

[0063] 2. Motion posture data

[0064] Motion posture data is collected through the inertial measurement unit (IMU), which provides angular velocity ω x ,ω y ,ωz and acceleration a x ,z y ,a z , output motion trajectory matrix T(t): The IMU sampling frequency fm = 200 Hz is used to capture the user's rapid rotation and acceleration events.

[0065] 3. Touch path data

[0066] The touch path data is collected by the touch sensor array on the surface of the handle (such as Synaptics T1000), and the touch path matrix S (u, v, t) is output. The touch path sampling frequency ft = 120 Hz is used for multi-touch and path tracking.

[0067] Data processing and dynamic feature generation: Perform time synchronization correction and normalization on the collected multimodal data to generate a multimodal dynamic feature matrix to provide basic input for subsequent modeling.

[0068] In order to ensure the time and space consistency of various signal data, the system uses the time and space synchronization module to perform time correction. sync Defined as: Among them, IMU provides a global time reference to ensure that the data of each modality is matched and fused on a unified time axis. On the basis of spatiotemporal synchronization, the system realizes data fusion calculation through a multimodal fusion module, and the fusion weight is dynamically allocated by the attention mechanism. The formula is as follows: Among them, α i′ represents the fusion weight of the i′th modality, Q i′ represents the query vector of the i′th modality, K i′ represents the key vector of the i′th mode; the final fusion result is: Among them, H sync,i′ It represents the feature matrix after the synchronization of the i′th modality, which integrates the spatiotemporal characteristics of pressure, motion, and touch data, and can accurately reflect the user's operation status.

[0069] The sensitivity of each mode is dynamically adjusted based on the feedback module. For pressure-motion feedback, the dynamic adjustment formula is: For touch-pressure feedback, the sensitivity adjustment formula is: Among them, γ and β are dynamic adjustment coefficients, It represents the pressure gradient and reflects the directional force applied by the user. Through the above steps, the present invention realizes the collection, synchronization, fusion and dynamic adjustment of multimodal data, and the system can recognize the user's operation intention in real time and optimize the feedback control effect.

[0070] Dynamic scalable feature extraction: The multimodal dynamic feature matrix is ​​intelligently decomposed through the dynamic scalable pyramid module to extract global and local multi-level features;

[0071] The downsampling step size is dynamically adjusted according to the operation mode: when the feature changes smoothly, the system uses a larger downsampling step size to focus on capturing macro trends, such as the overall operation trajectory, global pressure distribution, or the continuity of a large range of actions; when the feature changes dramatically, the system dynamically reduces the downsampling step size to deeply extract the refined features of the operation, such as rapid pressure changes in local areas, small motion paths, or subtle touch trajectories, thereby achieving multi-scale information extraction from global trends to local details.

[0072] Multi-scale feature fusion: Features at each layer are fused through an intelligent weighting mechanism to generate a dynamic feature pyramid that integrates global and local information: fusion =concat(H local ,H global ). Among them, H fusion represents the dynamic feature pyramid matrix after fusion; H local represents the local feature matrix; H global Represents the global feature matrix.

[0073] like Figure 3 As shown in the figure, the system enters the dynamic variable scale pyramid feature decomposition module, dynamically adjusts the feature downsampling step size according to the operation mode, and intelligently decomposes the multimodal features. The system calculates the feature change ΔH l To judge the severity of features at different levels, and dynamically adjust the downsampling step size s l The specific definitions are as follows:

[0074]

[0075] in, represents the change of the feature matrix of the lth layer, ∈ is the feature change threshold, and α and β are the step scaling coefficients. In specific applications, for example:

[0076] (1) When ΔHl=0.05<∈=0.1, the step size is increased to 4 to capture global features;

[0077] (2) When ΔHl=0.2>∈, the step size is reduced to 1 and fine-grained local features are extracted.

[0078] Dynamic weight optimization: The global and local multi-level features are integrated through the global-local collaborative dynamic weight adjustment mechanism to generate a comprehensive feature matrix, realizing feature modeling and adaptation from global to local.

[0079] The formula of the global-local collaborative dynamic weight adjustment mechanism is as follows:

[0080]

[0081] in, represents the eigenvalue of node i at time t+1; represents the eigenvalue of the adjacent node j at time t; w ij represents the connection weight between nodes i and j; σ(·) represents the activation function; N(i) is the set of neighbor nodes of node i.

[0082] The multi-scale features obtained by decomposition are fused through a dynamic weighting mechanism to generate a comprehensive feature matrix H fusion : In actual applications, for example, when the user performs a "drag + rotate" operation, the system automatically reduces the sampling step size to extract fine-grained touch path and force distribution features, dynamically allocates weights to focus on hot spots, and ensures accurate recognition of complex operation modes.

[0083] Feature modeling and fusion: Figure 4 As shown in the figure, in the cross-modal modeling stage, an intelligent modeling system is constructed through the spatiotemporal graph neural network (ST-GNN), hypergraph neural network (ST-HyperGNN) and temporal recurrent network (ST-RNN). The intelligent modeling system is used to process the comprehensive feature matrix to form comprehensive operation features of multimodal fusion; dynamic modeling of multimodal data, feature fusion and accurate identification of operation modes are realized.

[0084] First, the local dynamic features of a single sensor node are captured through the spatiotemporal graph neural network (ST-GNN); for different modal data (pressure, motion, touch), the adjacency matrix and dynamic feature update mechanism are constructed:

[0085]

[0086] in, represents the eigenvalue of neighbor node j at time t, b i represents the bias term of node i, w ij represents the edge weight between node i and its adjacent node j.

[0087] Through the above mechanism, the spatiotemporal graph neural network can capture the spatiotemporal dynamic characteristics of different sensor nodes and generate a spatiotemporal related node feature matrix to describe the dynamic response behavior of each modal data.

[0088] Secondly, a hypergraph neural network (ST-HyperGNN) is used to model the cross-modal interaction characteristics of multimodal data. Different modal data (pressure, motion, touch) are used as node inputs to construct a high-order hypergraph structure, in which each hyperedge connects multiple modal nodes, indicating high-order associations between modalities. The node feature update mechanism is as follows:

[0089]

[0090] Among them, w e is the weight of the hyperedge, j′∈e represents the node set in the hyperedge, E i is the set of hyperedges connected to node i; through hypergraph modeling, high-order interaction relationship modeling between modalities is realized, and the cross-modal weight distribution is dynamically optimized to enhance modal synergy.

[0091] Finally, the temporal recursive network (ST-RNN) is combined to model the dynamic changes of the time dimension of multimodal features; the temporal recursive mechanism is adopted for the temporal evolution of the node feature matrix:

[0092] H(t)=f(H(t-1),X(t))

[0093] Among them, H (t) represents the node state at time t, X (t) is the input data at time t, and f is a recursive function. Through time series modeling, the time dependency of different modal operation features is captured, and the dynamic changes of operation modes are accurately identified.

[0094] In the multi-level graph neural network, the first layer of the network is used to extract local operation features: extract local information during the operation, including the pressure distribution gradient, the short-term change rate of the motion trajectory, and the curvature characteristics of the touch path, capture fine-grained operation patterns, and generate a local feature matrix. The second layer of the network is used for global feature modeling: the local feature matrix is ​​input into the second layer of the hybrid graph neural network, and the data of different sensor modalities are integrated to extract overall trend information, including the global change pattern of pressure distribution and the continuity characteristics of the motion trajectory, which are used to analyze the macro mode of the operation. The third layer of the network is used for modal interaction modeling: the global features generated by the second layer are input into the third layer of the hybrid graph neural network to model the interaction relationship between multiple modalities, combine global features with local features, and generate comprehensive features of multimodal fusion to accurately describe the user's operation intentions.

[0095] In order to optimize the weight distribution of sensor modal fusion, the present invention proposes a global-local collaborative dynamic weight adjustment framework, including a feedback adjustment module, a topology analysis module, a hotspot weighting module and a dynamic collaboration module. The feedback adjustment module dynamically optimizes the weight distribution of a single modality (force, motion, touch) according to changes in sensor data to improve the accuracy of the single modality; the topology analysis module optimizes the weights of sensor nodes and edges through a graph neural network to strengthen the correlation between local and global data; the hotspot weighting module increases the weight of key areas and highlights regional characteristics by real-time detection of operation hotspot areas; the dynamic collaboration module optimizes the multimodal fusion weights based on task requirements to achieve efficient collaboration of the three modalities of force, motion and touch.

[0096] Output user operation instructions: Generate user operation instructions through comprehensive operation features, support multiple interaction methods, including complex interaction instructions such as dragging, clicking, rotating, etc., and support users who are not convenient to operate with fingers to complete various interaction tasks.

[0097] like Figure 5 As shown in the figure, the system generates real-time feedback through the operation intention prediction module, and outputs commands including click, drag, and rotate to the user interaction system. At the same time, the dynamic feedback generation module is combined to optimize the operation sensitivity and weight distribution. For example, when the system detects that the pressure in a specific area has increased significantly, the sensitivity is enhanced by increasing the hotspot weight to ensure the operation accuracy. In addition, the feedback results can be transmitted to the user end through the Bluetooth module, and real-time feedback is provided through the Web interface, LED light or vibration module to improve the user experience.

[0098] Embodiment 2, as Figure 2 As shown, an intelligent handle mouse control system based on a dynamic variable pyramid includes: a multimodal dimensional sensing system module, a dynamic weight adjustment module, a processing unit module, a power system module, a communication system module, a user interaction and feedback system module and a handle control system module; the multimodal dimensional sensing system module is respectively connected to the dynamic weight adjustment module, the processing unit module and the handle control system module, the processing unit module is respectively connected to the power system module, the communication system module and the handle control system module, the power system module is connected to the communication system module, and the communication system module is connected to the user interaction and feedback system module.

[0099] The multi-modal dimensional sensing system module includes an X-axis potential sensor 801, a Y-axis potential sensor 802, a MXene piezoresistive sensor 803, an acceleration sensor 804 and a touch distribution sensor 805, which are used to collect multi-modal operation data. The processing unit module includes a microcontroller 401, which is used to extract global features, local features and dynamic scale pyramid features.

[0100] The dynamic variable scale pyramid feature is realized by the following mechanism:

[0101] (1) Dynamically adjust the downsampling step size according to the rate of change of input data;

[0102] (2) Dynamically assign weights to local, intermediate, and global scales to prioritize key operational areas;

[0103] (3) Realize dynamic scale selection and switching, actively perceive data features, and perform local to global feature transfer and fusion.

[0104] The dynamic weight adjustment module is used to dynamically optimize sensor weights, network node connection weights and hotspot area weights to achieve local optimization and refined feature fusion under global guidance. The user interaction and feedback system module includes a Web interface 601, a sound feedback module 602 and an LED indicator 603; users can use the Web interface 601 or a mobile phone application to implement operation settings and status monitoring, and flexibly adjust system parameters. Provide real-time vibration feedback and status display according to user operation instructions.

[0105] The handle control system module includes a 3D printed handle shell 701, a rocker body 702, a rebound spring 703, a vibration feedback module 704, a universal joint rod structure 705, and a pressure button assembly 706; a variety of interaction methods are realized through the rocker body 702, the rebound spring 703, the universal joint rod structure 705, and the pressure button assembly 706, including complex operations such as dragging, clicking, and rotating; real-time feedback is provided through the vibration feedback module 704 to enhance the operation experience and interaction accuracy; users can freely switch between mouse mode and handle mode according to scene requirements, and the system automatically adapts to parameter configurations in different modes. The communication system module includes a Bluetooth module 501 and a USB interface 502; through the Bluetooth module 501 and the USB interface 502, wireless data transmission and wired connection are supported.

[0106] In addition, the system also integrates a collaborative reinforcement learning module to optimize feature modeling and weight adjustment strategies based on real-time and historical user operation data. Through continuous adaptive optimization, the system generates personalized operation parameter configurations and provides users with operation efficiency analysis reports to achieve long-term optimization and personalized interactive support.

[0107] Specific example: Real-time recognition and response to "rapid rotation and frequent pressing" operations. The following is a complete example of system processing.

[0108] 1. Data collection and time synchronization

[0109] The user holds the control handle to operate, and the system collects pressure, IMU motion data and touch path data in real time:

[0110] (1) Pressure data P(x, y): The pressure distribution of the user's palm is recorded through the bottom pressure sensor matrix, with a resolution of 0.01N and a sampling frequency of fp = 100 Hz.

[0111] (2) Motion data T(t): The IMU sensor captures the rotational angular velocity and linear acceleration of the handle in real time, with a sampling frequency of fm = 200 Hz.

[0112] (3) Touch data S (u, v, t): The touch sensor records the finger sliding path and pressing position with a resolution of 0.1 mm and a sampling frequency of ft = 120 Hz.

[0113] Since the sampling time of each sensor is different, the system uses the time synchronization correction module to align the data in time and space to generate a unified time and space synchronization matrix H sync .

[0114] 2. Dynamic Variable Scale Pyramid Feature Extraction

[0115] Based on synchronous data H sync , the system uses a dynamic and rescalable pyramid module to extract multi-level features:

[0116] (1) Detect feature changes: Calculate the feature change ΔH for each layer l , used to determine the severity of the user's operation: For example, in this example, when the user performs a fast rotation operation, the angular velocity data changes dramatically, resulting in ΔH l =0.3>∈=0.1.

[0117] (2) Dynamically adjust the step size: dynamically adjust the downsampling step size according to the feature change

[0118] As ΔH l = 0.3, β = 0.5, then sl = 1. The system captures fine-grained local features and focuses on subtle operational changes during the rotation process.

[0119] 3. Graph Neural Network (GNN) Modeling and Operation Recognition

[0120] The system inputs multi-layer features into the graph neural network to perform global and local feature modeling:

[0121] (1) Local feature modeling: ST-GNN captures the short-term spatial correlation between nodes and identifies the pressure points concentrated on (x 5 ,y 5 )nearby.

[0122] (2) Global feature analysis: Through topological analysis, the system identifies dramatic changes in angular velocity accompanied by frequent touch path oscillations.

[0123] (3) Cross-modal fusion: ST-HyperGNN fuses pressure, motion, and touch features and outputs a comprehensive feature matrix Hfusion of the operation mode:

[0124] Through the above modeling, the system accurately identifies the user's operation mode as "rapid rotation and frequent pressing".

[0125] 4. Feedback adjustment and real-time response

[0126] After identifying "rapid rotation and frequent pressing", the system dynamically adjusts the sensitivity through the feedback module:

[0127] (1) Motion sensitivity feedback: sensitivity is adjusted by combining pressure gradient and rotation angular velocity: current

[0128] Then adjusted =ω+0.32

[0129] (2) Touch sensitivity feedback: Dynamically adjust the touch response according to pressure changes: When P(x 5 ,y 5 )=2.0, the touch sensitivity increases to u adjusted =u+1.0. The system also uses LED flashing and vibration feedback to remind users of changes in operation accuracy and force.

[0130] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A smart handle mouse control method based on a dynamic variable pyramid, characterized in that: The following steps are involved: Multimodal data collection: Based on the multimodal dimensional sensing system, multimodal data of user operation is collected in real time, including pressure perception data, touch path data and motion trajectory data; Data processing and dynamic feature generation: Perform time synchronization correction and normalization on the collected multimodal data to generate a multimodal dynamic feature matrix; Dynamic scalable feature extraction: The multimodal dynamic feature matrix is ​​intelligently decomposed through the dynamic scalable pyramid module to extract global and local multi-level features; Dynamic weight optimization: The global and local multi-level features are integrated through the global-local collaborative dynamic weight adjustment mechanism to generate a comprehensive feature matrix; Feature modeling and fusion: An intelligent modeling system is constructed through spatiotemporal graph neural network (ST-GNN), hypergraph neural network (ST-HyperGNN) and temporal recurrent network (ST-RNN). The intelligent modeling system is used to process the comprehensive feature matrix to form comprehensive operational features of multimodal fusion. Output user operation instructions: Generate user operation instructions through comprehensive operation features: including complex interactive instructions such as dragging, clicking, and rotating, to support users with limited finger operation to complete various interactive tasks.

2. The intelligent handle mouse control method based on dynamic variable pyramid according to claim 1 is characterized in that: The multi-modal dimensional sensing system comprises an X-axis potential sensor (801), a Y-axis potential sensor (802), a MXene piezoresistive sensor (803), an acceleration sensor (804) and a touch distribution sensor (805); pressure sensing data is captured by the MXene piezoresistive sensor (803), touch path data is recorded by the touch distribution sensor (805), and motion trajectory data is acquired by the X-axis potential sensor (801), the Y-axis potential sensor (802) and the acceleration sensor (804).

3. The intelligent handle mouse control method based on the dynamic variable pyramid according to claim 1 or 2, characterized in that: The motion trajectory data includes the joystick tilt angle and acceleration information.

4. The intelligent handle mouse control method based on dynamic variable pyramid according to claim 1 is characterized in that: The method for intelligently decomposing a multimodal dynamic feature matrix through a dynamically variable scale pyramid module is as follows: dynamically adjusting the downsampling step size according to the operation mode: when the feature changes slowly, a larger downsampling step size is used to focus on capturing macro trends, such as the overall operation trajectory, global pressure distribution, or continuity of a large range of actions; and when the feature changes dramatically, the system dynamically reduces the downsampling step size to deeply extract refined features of the operation, such as rapid pressure changes in a local area, a small motion path, or a subtle touch trajectory, thereby achieving multi-scale information extraction from global trends to local details; Dynamically adjust the downsampling ratio according to the changes in multimodal features: in, represents the change of the feature matrix of the lth layer, ε is the feature change threshold, α and β are the step size scaling coefficients; represents the eigenvalue of node i at time t-1; represents the characteristic value of node i at time t; Multi-scale feature fusion: Features at each layer are fused through an intelligent weighting mechanism to generate a dynamic feature pyramid that integrates global and local information: fusion =concat(H local ,H global ), where H fusion represents the dynamic feature pyramid matrix after fusion; H local represents the local feature matrix; H global Represents the global feature matrix.

5. The intelligent handle mouse control method based on dynamic variable pyramid according to claim 4 is characterized in that: The formula of the global-local collaborative dynamic weight adjustment mechanism is as follows: in, represents the eigenvalue of node i at time t+1; represents the eigenvalue of the adjacent node j at time t; w ij represents the connection weight between nodes i and j; σ(·) represents the activation function; N(i) is the set of neighbor nodes of node i.

6. The intelligent handle mouse control method based on dynamic variable pyramid according to claim 5 is characterized in that: The method of using the intelligent modeling system to process the comprehensive feature matrix to form a comprehensive operation feature of multi-modal fusion is: First, the local dynamic features of a single sensor node are captured through a spatiotemporal graph neural network. For different modal data, an adjacency matrix and a dynamic feature update mechanism are constructed: Among them, b i represents the bias term of node i; Secondly, a hypergraph neural network is used to model the cross-modal interaction characteristics of multimodal data. Different modal data are used as node inputs to construct a high-order hypergraph structure, in which each hyperedge connects multiple modal nodes, indicating high-order associations between modalities. The node feature update mechanism is as follows: Among them, w e is the weight of the hyperedge, j′∈e represents the node set in the hyperedge, E i is the set of hyperedges connected to node i; finally, the temporal recursive network is combined to model the dynamic changes in the time dimension of multimodal features; for the temporal evolution of the node feature matrix, the temporal recursive mechanism is adopted: H(t)=f(H(t-1),X(t)) Among them, H (t) represents the node state at time t, X (t) is the input data at time t, and f is the recursive function.

7. The intelligent handle mouse control method based on dynamic variable pyramid according to claim 1 is characterized in that: The global-local collaborative dynamic weight adjustment mechanism includes a feedback adjustment module, a topology analysis module, a hotspot weighting module and a dynamic collaborative multimodal fusion module; Feedback adjustment module: dynamically optimizes the weight distribution of a single mode based on real-time sensor data; Topology analysis module: optimizes the weights of sensor nodes and edges through graph neural networks to strengthen the correlation between local and global data; Hotspot weighting module: through real-time detection of hotspot areas, increase the weight of key areas and highlight regional features; Dynamic collaborative multimodal fusion module: Dynamically optimizes the fusion weights between modalities based on the correlation between global and local modal features.

8. An intelligent handle mouse control system based on a dynamically variable pyramid, used to implement any of the methods described in claims 1 to 7, characterized in that: include: Multimodal dimensional sensing stereoscopic sensing system module, dynamic weight adjustment module, processing unit module, power system module, communication system module, user interaction and feedback system module and handle control system module; the multimodal dimensional sensing stereoscopic sensing system module is respectively connected to the dynamic weight adjustment module, the processing unit module and the handle control system module, the processing unit module is respectively connected to the power system module, the communication system module and the handle control system module, the power system module is connected to the communication system module, and the communication system module is connected to the user interaction and feedback system module.

9. The intelligent handle mouse control system based on dynamic variable pyramid according to claim 8, characterized in that: The multi-modal dimensional sensing system module includes an X-axis potential sensor (801), a Y-axis potential sensor (802), a MXene piezoresistive sensor (803), an acceleration sensor (804) and a touch distribution sensor (805), which are used to collect multi-modal operation data; The processing unit module includes a microcontroller (401) for extracting global features, local features and dynamic scale pyramid features; The dynamic weight adjustment module is used to dynamically optimize sensor weights, network node connection weights and hotspot area weights to achieve dynamic optimization from local to global; The user interaction and feedback system module comprises a Web interface (601), a sound feedback module (602) and an LED indicator light (603); and provides real-time vibration feedback and status display according to user operation instructions.

10. The intelligent handle mouse control method based on dynamic variable pyramid according to claim 8, characterized in that: The handle control system module comprises a 3D printed handle housing (701), a rocker body (702), a rebound spring (703), a vibration feedback module (704), a universal joint rod structure (705), and a pressure button assembly (706); a variety of interaction modes are realized through the rocker body (702), the rebound spring (703), the universal joint rod structure (705), and the pressure button assembly (706), including complex operations such as dragging, clicking, and rotating; Providing real-time feedback via a vibration feedback module (704); The communication system module comprises a Bluetooth module (501) and a USB interface (502); wireless data transmission and wired connection are supported through the Bluetooth module (501) and the USB interface (502).

Images