Hand wearable man-machine interaction system based on DeepSeek-R1 architecture

Abstract

The invention discloses a hand wearable man-machine interaction system based on a DeepSeek-R1 framework, and belongs to the field of intelligent medical equipment. The system is composed of a medical hand sensing terminal and a remote diagnosis and treatment terminal, the terminal integrates a medical flexible finger joint sensor, a nine-axis IMU and a touch force feedback module, and hand anatomical-level motion data are accurately captured. And carrying out transfer learning on clinical operation data based on a DeepSeek-R1 neural network, constructing a three-dimensional gesture model library conforming to medical specifications, and realizing microscopic gesture reconstruction with millimeter-level precision. The action synchronization of the surgical robot is realized through a medical 5G network, and the biomechanical characteristics such as tissue elasticity and vascular pulsation are converted into programmable tactile feedback by applying an FOC algorithm. And the system optimizes the AI model and establishes an expert gesture database. According to the scheme, the problems of force feedback distortion, large delay and the like of a traditional VR medical system are solved, high-immersion interaction is realized in the fields of remote operation guidance, minimally invasive training, neural rehabilitation treatment and the like, and a safe and reliable high-simulation solution is provided for intelligent medical treatment.

Description

(1) Technical field

[0001] This invention belongs to the field of intelligent medical devices and artificial intelligence technology, specifically a hand gesture capture and remote force feedback system based on multi-sensor fusion and deep learning. This system addresses the technical bottlenecks of force feedback distortion and motion delay in medical VR interactions, and is suitable for remote surgical guidance, minimally invasive surgical training, and neurorehabilitation. (2) Background technology

[0002] Existing hand gesture capture devices often rely on a single sensor, such as an inertial sensor or optical marker, and suffer from issues such as insufficient accuracy, high latency, and a lack of force feedback. For example, vision-based hand tracking is susceptible to occlusion, while traditional data gloves typically only provide one-way motion capture and lack tactile feedback. Furthermore, the AI models of existing systems are mostly fixed architectures, making it difficult to adapt to the personalized gesture characteristics of different users. Therefore, a high-precision, low-latency, and dynamically optimized hand interaction system is urgently needed to meet the needs of real-time, two-way human-computer interaction in fields such as remote control, medical rehabilitation, and virtual reality. (3) Summary of the invention

[0003] The present invention proposes a hand interaction system with multimodal perception and adaptive learning, the technical solution of which includes:

[0004] Sensor Network Architecture: Flexible Flexure Sensor Array: Utilizes five 0.2mm thick nanopiezoresistive thin film sensors with a linearity error of <1%, embedded in the proximal (PIP) and distal (DIP) joints of the finger cuff. Each sensor outputs a 0-5V analog signal corresponding to joint angles from 0° extension to 90° full flexion. Six-axis Inertial Sensor: Integrated with an MPU-6050 chip, the accelerometer has a ±16g range and a gyroscope has a ±2000° / s range. Fixed to the back of the hand, the global hand pose is calculated using quaternions (heading angle accuracy of ±0.5°). Haptic Feedback Actuator: Five micro brushless motors (8mm diameter, 0.15N·m torque) are embedded in the fingertips, using a field-oriented control (FOC) algorithm to achieve 0-10N continuous force feedback with a resolution of 0.05N.

[0005] The system uses multiple sensors to construct a multidimensional dataset, including spatial coordinates (x, y, and z axis positions and Euler angles output by six-axis sensors); joint angles (knuckle curvature mapped by flexible sensor flexure); and dynamic features (hand movement velocity, acceleration, and trajectory continuity). Knowledge distillation training is performed using the DeepSeek-R1 neural network architecture: the teacher model, a multimodal fusion model based on DeepSeek-R1 (10B parameters), extracts spatiotemporal features from raw data through supervised learning; the student model, a lightweight convolutional network (500M parameters), inherits the teacher model's knowledge through a distillation loss function, enabling real-time inference (latency < 20ms).

[0006] Posture reconstruction: The sensor signal is input into the student model, and the three-dimensional coordinates of 17 joints are output (in line with the MANO hand model standard), with a reconstruction error of <2mm; remote control: Through the low-latency communication module based on 5G transmission, the transmission delay is <10ms to drive the robotic arm to synchronously reproduce hand movements; force feedback: The tactile sensor at the end of the robotic arm collects contact force data, which is converted into a fingertip motor drive signal through the FOC (magnetic field oriented control) algorithm to achieve force-electric coupling feedback, with a force control accuracy of ±0.1N; online optimization: User data is continuously collected during system operation, and the student model parameters are updated through incremental learning, so that the gesture recognition accuracy increases with usage time (optimization iteration once a day). (IV) Description of the accompanying drawings

[0007] Figure 1 This is a schematic diagram of the sensor layout of the hand-worn device;

[0008] Figure 2 It is a flowchart of multimodal data fusion and model training;

[0009] Figure 3 It is the logic block diagram of bidirectional force feedback control. (V) Specific implementation methods

[0010] The following will be combined with the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the embodiments described are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0011] During the data collection phase, users wear the device and execute a standardized gesture library (including 300 types of actions). After the sensor data is de-noised using a Kalman filter, it is divided into a training set and a test set in a 7:3 ratio. The following optimization strategy is used when training the DeepSeek-R1 model: Loss function:

[0012] Ltotal =λ1L position (P pred ,P gt )+λ2L orientation (Q pred, Q gt ) Among them, λ1=0.7 and λ2=0.3 constrain the joint position error and hand orientation quaternion error respectively.

[0013] The six-dimensional force sensor at the end of the hand sensing terminal collects the contact force F ext , mapped to the fingertip motor target torque through the following formula:

[0014] where K p =0.15N·mm / mN,K d =0.02N·mm·s / mN, ensuring the dynamic responsiveness of force feedback.

[0015] When the system is running, a sliding window mechanism is used to store the latest 200 sets of gesture data, and the model is updated daily using the following incremental learning formula:

[0016] Among them, η = 0.001 is the learning rate, and β = 0.1 is the elastic weight to prevent catastrophic forgetting.

[0017] Sensor calibration process: The user performs a standardized calibration motion (clenching a fist, spreading five fingers, and rotating the wrist) to collect 10 seconds of baseline data. The mapping relationship between the sensor output and the optical motion capture system (Vicon) data is fitted by the least squares method to establish a calibration matrix satisfy:

[0018] θ calibrated =M·[V flex ,a x ,a y ,a z ,ω x ,ω y ] T Where V flex is the voltage value of the flexible sensor.

[0019] Real-time inference optimization: The TensorRT engine is deployed to perform INT8 quantization on the student model, reducing inference time from 15ms to 8ms. A double-buffering mechanism is used to preload the next frame of sensor data while processing the current frame, reducing end-to-end latency.

[0020] Dynamic force mapping strategy: Define a material hardness database (rubber: stiffness coefficient 50N / mm, metal: 200N / mm). When the robot arm contacts an object, the feedback force is calculated based on the preset material type:

[0021] F feedback =min(K·δ X ,F max ) where δ x is the contact point displacement, F max =8N is the safety threshold. By modulating the motor drive signal with PWM, force control with a resolution of 0.1N is achieved.

[0022] User personalized adaptation: Provide "Force Sensitivity" slider (0-100%) in the settings interface, mapped to K p coefficient:

[0023] Among them S user The sensitivity value set by the user.

[0024] Incremental learning implementation steps: Data preprocessing: Time alignment and normalization of the newly added gesture data, and removal of outliers (Z-score>3). Feature extraction: Freeze the convolutional layer of the student model and only update the weights of the fully connected layer.

[0025] Training configuration: Optimizer:

[0026] AdamW(Ir=3e-5,weight_decay=0.01) Batch size: 32, iterations: 100 steps / day. Performance verification: After daily updates, test on a retained validation set (2000 data points). If the accuracy drops by >2%, rollback the model.

[0027] The above disclosure is only a preferred embodiment of the present invention, and certainly cannot be used to limit the scope of the rights of the present invention. Ordinary technicians in this field can understand that all or part of the processes of the above embodiment and equivalent changes made in accordance with the claims of the present invention are still within the scope of the invention.

Claims

1. The hand-worn human-computer interaction system based on the DeepSeek-R1 architecture is characterized by: include: A hand-worn terminal integrating a flexible bending sensor, a six-axis inertial sensor, and five brushless motors; a remote actuator, including a robotic arm and a six-dimensional force sensor; A data processing module for fusing multi-source sensor data and constructing a spatiotemporal feature dataset; A deep learning model module that uses the AI distillation model of the DeepSeek-R1 architecture to output hand posture parameters in real time; A two-way interaction module that enables synchronous control of hand movements to the robotic arm and tactile force feedback; An online learning module that dynamically optimizes model parameters through incremental learning.

2. The wearable terminal according to claim 1, characterized in that The hand-worn terminal further includes: the flexible bending sensor is a nano-piezoresistive film sensor, distributed on the proximal interphalangeal joint (PIP) and distal interphalangeal joint (DIP) of the finger, outputting a 0-5V analog signal with a linearity error of less than 1%; the six-axis inertial sensor integrates an MPU-6050 chip, fixed to the base on the back of the hand, with a sampling frequency of ≥200Hz; the brushless motor is embedded in the fingertip, has a torque of 0.15N·m, and supports a field-oriented control (FOC) algorithm.

3. The actuator according to claim 1, characterized in that: The remote actuator comprises: Signal synchronization unit uses IEEE 1588 protocol to align multi-source data, with synchronization error less than 1ms; Feature encoding unit, which extracts motion trajectory features through LSTM network and joint coupling features through 1D convolutional network; The fusion unit uses a gated attention mechanism to dynamically weight multimodal features, and the weight coefficient α is generated by a two-layer MLP.

4. The deep learning model according to claim 1, characterized in that The deep learning model module and the online learning module perform the following operations: the teacher model is a 12-layer Transformer with an input dimension of 256 and 8 multi-head attention heads; the student model is a 5-layer deep separable convolutional network with 5% of the parameters of the teacher model; the knowledge distillation loss function is: Where T(x i )、S(x i ) are the output logits of the teacher and student models respectively. Forward control channel: transmits hand posture data to the robotic arm based on the UWB communication protocol. The control command generation formula is: in is the pseudo-inverse of the Jacobian matrix of the robot arm, x target is the target hand pose; Reverse feedback channel: The six-dimensional force sensor at the end of the robotic arm collects the contact force F ext , which is converted into motor drive current through the impedance control model: Data cache: A circular buffer is used to store the gesture data of the last 72 hours, with a maximum capacity of 100,000 groups; Incremental training: Freeze the first three convolutional layers of the student model, and update the weights of the fully connected layer daily. The loss function is: L online =L mse +10 3 ·∑ i F i (θ i -θ old,i ) 2 Among them F i is the diagonal element of the Fisher information matrix; Abnormal rollback mechanism: If the accuracy of the validation set drops by more than 2% after the update, it will automatically revert to the previous version of the model. Material hardness mapping: Calculate the feedback force based on the preset stiffness coefficient K (rubber 50N / mm, metal 200N / mm): F feedback =min(K·δ x ,8N) where δ x is the contact displacement; User adaptive adjustment: Dynamically adjust the proportional coefficient K through the sensitivity slider p : Among them S user User-settable value (0-100%).

5. A hand-worn human-computer interaction system based on the DeepSeek-R1 architecture according to any one of claims 1 to 4, characterized in that: The following steps are involved: The medical-grade hand sensing terminal consists of five flexible knuckle bending sensors encapsulated in medical silicone, a high-precision nine-axis IMU, and a biocompatible tactile feedback unit, enabling real-time capture of anatomical-level hand motion parameters. The DeepSeek-R1 architecture is used to train an AI distillation model, ensuring it can adapt to different hand sizes and can be self-trained and optimized during use, outputting hand posture parameters in real time. The data processing module uses the DeepSeek-R1 neural network to transfer learning from clinical surgical data, constructing a 3D gesture model library that complies with medical standards and achieving millimeter-level precision in microsurgical gesture reconstruction. A dedicated medical 5G communication module transmits multimodal surgical dynamics data and ensures sub-second latency for operational commands. It also uses a brushless motor and mechanical mechanism to synchronize hand movements with the robotic arm, providing tactile force feedback, with customizable force feedback levels. A tactile feedback module uses a Field-of-Center (FOC) algorithm to convert tissue elasticity and vascular pulsation characteristics into programmable tactile feedback force.

Images