Hand rehabilitation training system based on virtual reality and gesture interaction

Abstract

The invention relates to the technical field of rehabilitation medical treatment, in particular to a hand rehabilitation training system based on virtual reality and gesture interaction, and the system comprises a gesture sensing module which collects a hand movement track, a muscle electric signal, grip pressure and a joint movement angle; the VR rehabilitation scene construction module constructs a rehabilitation training virtual scene, simulates a hand interaction behavior and a force feedback effect through an adaptive physical engine and a neural action mapping algorithm, and outputs simulation data; the intelligent evaluation center evaluates the action standard degree and the muscle activation efficiency based on an improved CNN-GRU model, identifies an action compensation mode and abnormal force generation, and outputs an evaluation report; the interactive training unit renders a motion state and a training process in real time, and generates personalized training guidance in combination with voice, visual and tactile prompts; and the personalized regulation and control unit predicts a rehabilitation trend and a stage target through a reinforcement learning algorithm, and dynamically adjusts training parameters and scene difficulty. Therefore, the problems of low interaction precision, incapability of sharing and the like in the prior art are solved.

Description

Technical Field

[0001] This invention relates to the field of rehabilitation medicine technology, specifically a hand rehabilitation training system based on virtual reality and gesture interaction. Background Technology

[0002] Hand dysfunction is a common complication following stroke, spinal cord injury, trauma, and other diseases or injuries, severely impacting patients' daily living abilities and quality of life. Hand rehabilitation training is a key means of improving hand dysfunction; continuous and scientific training can promote the recovery of neuromuscular function in the hand.

[0003] Current hand rehabilitation training methods mainly include traditional manual rehabilitation training and rehabilitation training based on simple mechanical devices. Traditional manual rehabilitation training relies on one-on-one guidance from professional rehabilitation therapists, which results in low training efficiency, high labor costs, and monotonous training content, leading to poor patient compliance and difficulty in ensuring training effectiveness. Rehabilitation training based on simple mechanical devices mostly adopts a fixed trajectory mechanical drive method, lacking personalized adaptation, unable to dynamically adjust the training difficulty according to the patient's rehabilitation progress, and making it difficult to achieve accurate training effect assessment.

[0004] With the development of virtual reality (VR) technology, some rehabilitation training equipment has begun to incorporate VR technology to construct virtual training scenarios to enhance the fun of training. However, existing VR rehabilitation training equipment still has many shortcomings: on the one hand, the precision of gesture interaction is low, making it difficult to accurately recognize the subtle hand movements of patients, resulting in unsmooth virtual scene interaction and affecting the training experience; on the other hand, there is a lack of collection and analysis of patients' physiological signals (such as electromyographic signals), making it impossible to comprehensively and accurately assess the training effect, and also making it difficult to dynamically optimize the training plan based on the patient's physiological state; in addition, training data is mostly limited to single-institution storage, making it impossible to achieve cross-institution sharing, which is not conducive to medical staff fully grasping the patient's rehabilitation progress, and also restricts the precise optimization of rehabilitation training plans.

[0005] Therefore, developing a hand rehabilitation training system with high-precision gesture interaction, accurate training assessment, personalized training program generation, and cross-institutional data sharing capabilities has become the key to solving existing technical problems. Summary of the Invention

[0006] This application provides a hand rehabilitation training system based on virtual reality and gesture interaction to solve the problems of low interaction accuracy and inability to share existing technologies.

[0007] The first aspect of this application provides a hand rehabilitation training system based on virtual reality and gesture interaction, including: a gesture perception module, a VR rehabilitation scene construction module, an intelligent assessment center, an interactive training unit, and a personalized control unit; wherein, the gesture perception module is used to collect hand movement trajectory, electromyographic signals, grip force and pressure data, and joint range of motion; the VR rehabilitation scene construction module is used to construct a personalized rehabilitation training virtual scene based on the patient's rehabilitation stage and the hand movement trajectory, electromyographic signals, grip force and pressure data, and joint range of motion, and to simulate hand interaction behavior and force in a real environment through an adaptive physics engine and a neuro-motor mapping algorithm. The feedback effect outputs patient hand simulation data; the intelligent assessment center is used to evaluate the standardization of rehabilitation training movements and muscle activation efficiency, identify movement compensation patterns and abnormal force characteristics based on the patient hand simulation data and combined with an improved CNN-GRU model, and output a rehabilitation assessment report; the interactive training unit is used to render the hand movement state and training progress in real time according to the rehabilitation assessment report, and generate personalized training guidance by combining voice, visual and tactile prompts; the personalized control unit is used to predict rehabilitation trends and stage goals through reinforcement learning algorithms based on the rehabilitation assessment report and real-time training data, and dynamically adjust training parameters and scene difficulty.

[0008] Preferably, the gesture sensing module includes a motion capture unit, an electromyography (EMG) sensor group, a grip force and pressure sensor, and a joint angle acquisition unit. The motion capture unit is used to acquire the three-dimensional movement trajectory of the hand in real time. The EMG sensor group is used to conform to the forearm muscle group and acquire surface EMG signals during muscle contraction. The grip force and pressure sensor is integrated into the handle of the training device to acquire data on the magnitude and distribution of hand grip force. The joint angle acquisition unit is used to monitor the movement angle and rotation rate of the fingers and wrist joints in real time.

[0009] Preferably, the VR rehabilitation scene construction module includes a scene generation unit, a physics engine adaptation unit, and a neuro-motion mapping unit. The scene generation unit constructs a personalized rehabilitation training virtual scene based on the patient's rehabilitation stage, hand movement trajectory, muscle electrical signals, grip force pressure data, and joint range of motion. The physics engine adaptation unit dynamically adjusts the physical properties of objects in the scene using an adaptive physics engine to simulate real contact force feedback. The neuro-motion mapping unit establishes a precise mapping relationship between hand movement data and virtual scene interaction actions using a neuro-motion mapping algorithm, outputting hand simulation data including movement trajectory and force feedback.

[0010] Preferably, the intelligent assessment center includes a movement standardization assessment unit, a muscle activation analysis unit, and an abnormal pattern recognition unit. The movement standardization assessment unit uses an improved CNN-GRU model to extract movement features from hand simulation data, compares them with standard rehabilitation movement templates, calculates movement similarity scores, and assesses movement standardization. The muscle activation efficiency analysis unit calculates muscle activation rate and synergistic work efficiency based on electromyographic signal simulation data. The abnormal pattern recognition unit identifies compensatory movements and abnormal features of force imbalance, generating a rehabilitation assessment report containing assessment indicators and abnormal analysis.

[0011] Preferably, the interactive training unit includes a VR rendering engine, a multimodal prompting unit, and a training guidance generation unit. The VR rendering engine is used to render the movement state of the virtual hand model, the progress of the training task, and details of the scene environment in real time. The multimodal prompting module is used to provide voice command prompts, visual highlighting guidance, and tactile vibration feedback. The training guidance generation unit is used to generate targeted training guidance based on the weaknesses identified in the rehabilitation assessment report.

[0012] Preferably, the personalized control unit includes a rehabilitation trend prediction unit, a training parameter adjustment unit, and a scenario difficulty adaptation unit. The rehabilitation trend prediction unit is used to predict the recovery trend and phased goals of hand function by using a reinforcement learning algorithm, combined with historical training data and evaluation reports. The training parameter adjustment unit is used to dynamically adjust the training duration, the number of repetitions of the movement, and the target threshold parameters of grip strength. The scenario difficulty adaptation unit is used to gradually increase the complexity of the scenario according to the rehabilitation progress.

[0013] The second aspect of this application provides a hand rehabilitation training method based on virtual reality and gesture interaction, comprising: acquiring hand movement trajectory, electromyographic signals, grip force and pressure data, and joint motion angles; constructing a personalized rehabilitation training virtual scene based on the patient's rehabilitation stage and the hand movement trajectory, electromyographic signals, grip force and pressure data, and joint motion angles; simulating hand interaction behavior and force feedback effects in a real environment through an adaptive physics engine and a neuro-motor mapping algorithm, and outputting patient hand simulation data; evaluating the standardization of rehabilitation training movements and muscle activation efficiency based on the patient's hand simulation data, combined with an improved CNN-GRU model, identifying movement compensation patterns and abnormal force characteristics, and outputting a rehabilitation assessment report; simultaneously rendering the hand movement state and training progress in real time, and generating personalized training guidance by combining voice, visual, and tactile cues; and predicting rehabilitation trends and stage goals through a reinforcement learning algorithm based on the rehabilitation assessment report and real-time training data, and dynamically adjusting training parameters and scene difficulty.

[0014] A third aspect of this application provides an electronic device, including: a memory, a processor, and a computer program stored in the memory and executable on the processor. The processor executes the program to implement a hand rehabilitation training method based on virtual reality and gesture interaction as described in the above embodiments.

[0015] A fourth aspect of this application provides a computer-readable storage medium having a computer program stored thereon, which is executed by a processor to implement a hand rehabilitation training method based on virtual reality and gesture interaction as described in the above embodiments.

[0016] A fifth aspect of this application provides a computer program product, including a computer program or instructions, for implementing a hand rehabilitation training method based on virtual reality and gesture interaction as described in the above embodiments.

[0017] Therefore, this application has the following beneficial effects:

[0018] This application's embodiments comprehensively collect hand movement trajectories, muscle electrical signals, grip pressure data, and joint range of motion through a gesture perception module, providing multi-dimensional and accurate basic data for rehabilitation training and assessment. The VR rehabilitation scene construction module, combining an adaptive physics engine and a neuro-motion mapping algorithm, accurately simulates hand interaction behavior and force feedback effects in a real-world environment, achieving dynamic mapping between hand movement data and virtual scenes, overcoming the limitations of traditional rehabilitation training scenarios being singular and having delayed feedback. The intelligent assessment center, utilizing an improved CNN-GRU model to integrate hand simulation data, enhances the accuracy of rehabilitation training movement standard assessment and muscle activation efficiency analysis, and strengthens the ability to identify movement compensation patterns and force abnormalities. The immersive VR training scene constructed by the interactive training unit intuitively presents hand movement status and training progress, facilitating the efficient generation of personalized training guidance. The personalized control unit predicts rehabilitation trends and stage goals based on rehabilitation assessment reports and real-time training data, dynamically adjusting training parameters and scene difficulty, effectively improving the targeting, personalization, and hand function recovery efficiency of rehabilitation training. Thus, it solves the problems of low interactive accuracy and inability to share data in existing technologies.

[0019] Additional aspects and advantages of this application will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of this application. Attached Figure Description

[0020] The above and / or additional aspects and advantages of this application will become apparent and readily understood from the following description of the embodiments taken in conjunction with the accompanying drawings, wherein:

[0021] Figure 1 This is a schematic diagram of a hand rehabilitation training system based on virtual reality and gesture interaction provided in an embodiment of this application;

[0022] Figure 2 This is a schematic diagram of a gesture sensing module provided according to an embodiment of this application;

[0023] Figure 3 This is a schematic diagram of a VR rehabilitation scene construction module provided according to an embodiment of this application;

[0024] Figure 4 This is a schematic diagram of an intelligent evaluation center provided according to an embodiment of this application;

[0025] Figure 5 This is a schematic diagram of an interactive training unit provided according to an embodiment of this application;

[0026] Figure 6 This is a schematic diagram of a personalized control unit provided according to an embodiment of this application;

[0027] Figure 7 This is a flowchart of a hand rehabilitation training system based on virtual reality and gesture interaction provided according to an embodiment of this application;

[0028] Figure 8 This is a flowchart illustrating a hand rehabilitation training method based on virtual reality and gesture interaction according to an embodiment of this application;

[0029] Figure 9 This is a schematic diagram of a hand rehabilitation training method based on virtual reality and gesture interaction according to an embodiment of this application;

[0030] Figure 10 This is a schematic diagram of the structure of an electronic device provided according to an embodiment of this application. Detailed Implementation

[0031] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0032] The following description, with reference to the accompanying drawings, describes a hand rehabilitation training system based on virtual reality and gesture interaction, according to an embodiment of this application. To address the issue of data sharing limitations mentioned in the background, this application provides a hand rehabilitation training system based on virtual reality and gesture interaction. In this system, a gesture perception module comprehensively collects hand movement trajectories, muscle electrical signals, grip pressure data, and joint range of motion, providing multi-dimensional and accurate basic data for rehabilitation training and assessment. A VR rehabilitation scene construction module, combining an adaptive physics engine and a neuro-motion mapping algorithm, accurately simulates hand interaction behavior and force feedback effects in a real-world environment, achieving dynamic mapping between hand movement data and virtual scenes, overcoming the limitations of traditional rehabilitation training scenarios that are singular and have delayed feedback. An intelligent assessment center, utilizing an improved CNN-GRU model to integrate hand simulation data, enhances the accuracy of rehabilitation training movement standard assessment and muscle activation efficiency analysis, and strengthens the ability to identify movement compensation patterns and force abnormalities. An immersive VR training scene constructed by an interactive training unit intuitively presents hand movement status and training progress, facilitating the efficient generation of personalized training guidance. A personalized control unit predicts rehabilitation trends and stage goals based on rehabilitation assessment reports and real-time training data, dynamically adjusting training parameters and scene difficulty, effectively improving the targeting, personalization, and efficiency of hand function recovery in rehabilitation training. This solves the problems of low interaction accuracy and inability to share existing technologies.

[0033] Figure 1 This is a schematic diagram of the structure of a hand rehabilitation training system based on virtual reality and gesture interaction, provided in an embodiment of this application.

[0034] This application provides a hand rehabilitation training system based on virtual reality and gesture interaction. The system 10 includes:

[0035] Gesture perception module 100, VR rehabilitation scene construction module 200, intelligent assessment center 300, interactive training unit 400, and personalized control unit 500.

[0036] The system comprises the following components: a gesture perception module 100, which collects hand movement trajectories, electromyographic signals, grip pressure data, and joint motion angles; a VR rehabilitation scene construction module 200, which constructs personalized rehabilitation training virtual scenes based on the patient's rehabilitation stage and hand movement trajectories, electromyographic signals, grip pressure data, and joint motion angles, simulating hand interaction behavior and force feedback effects in a real environment through an adaptive physics engine and neuro-motor mapping algorithm, and outputting simulated hand data; an intelligent assessment center 300, which, based on the simulated hand data and an improved CNN-GRU model, assesses the standardization of rehabilitation training movements, muscle activation efficiency, identifies movement compensation patterns and abnormal force characteristics, and outputs a rehabilitation assessment report; an interactive training unit 400, which, based on the rehabilitation assessment report, renders the hand movement state and training progress in real time, and generates personalized training guidance by combining voice, visual, and tactile cues; and a personalized control unit 500, which, based on the rehabilitation assessment report and real-time training data, predicts rehabilitation trends and stage goals through reinforcement learning algorithms, and dynamically adjusts training parameters and scene difficulty.

[0037] It is understood that in this embodiment, the gesture perception module comprehensively collects hand movement trajectories, muscle electrical signals, grip pressure data, and joint range of motion, providing multi-dimensional and accurate basic data for rehabilitation training and assessment. The VR rehabilitation scene construction module combines an adaptive physics engine and a neuro-motion mapping algorithm to accurately simulate hand interaction behavior and force feedback effects in a real environment, achieving dynamic mapping between hand movement data and virtual scenes, breaking through the limitations of traditional rehabilitation training scenarios being singular and having delayed feedback. The intelligent assessment center uses an improved CNN-GRU model to integrate hand simulation data, improving the accuracy of rehabilitation training movement standard assessment and muscle activation efficiency analysis, and enhancing the ability to identify movement compensation patterns and force abnormalities. The immersive VR training scene constructed by the interactive training unit intuitively presents the hand movement state and training process, helping to efficiently generate personalized training guidance. The personalized control unit predicts rehabilitation trends and stage goals based on rehabilitation assessment reports and real-time training data, dynamically adjusting training parameters and scene difficulty, effectively improving the pertinence, personalization, and hand function recovery efficiency of rehabilitation training. Thus, it solves the problems of low interactive accuracy and inability to share data in existing technologies.

[0038] In this embodiment of the application, the gesture sensing module 100 includes: Figure 2 As shown, the device includes a motion capture unit, an electromyography sensor group, a grip force pressure sensor, and a joint angle acquisition unit.

[0039] Among them, the motion capture unit is used to collect the three-dimensional motion trajectory of the hand in real time; the electromyography sensor group is used to fit the forearm muscle group and collect the surface electromyography signals when the muscles contract; the grip force and pressure sensor is integrated into the handle of the training device to collect the grip force and pressure distribution data of the hand; and the joint angle acquisition unit is used to monitor the movement angle and rotation rate of the finger and wrist joints in real time.

[0040] It is understood that the embodiments of this application acquire the three-dimensional motion trajectory of the hand in real time through the motion capture unit, accurately track the spatial position changes of the hand movements, and provide basic trajectory data for gesture recognition and action analysis; the electromyography sensor group collects surface electromyography signals by conforming to the forearm muscle group, captures the intensity and frequency characteristics of muscle contraction, and reflects the muscle driving state of the gesture movement; the grip force pressure sensor is integrated into the handle of the training device to collect the magnitude and pressure distribution data of the hand grip force, and quantifies the hand force exertion and grip posture; the joint angle acquisition unit monitors the activity angle and rotation rate of the finger and wrist joints in real time, accurately captures the details of joint movement, and provides multi-dimensional and high-precision raw data for gesture intention interpretation, action standard evaluation and training effect optimization, thereby improving the comprehensiveness and accuracy of gesture perception.

[0041] For example, taking the rehabilitation training scenario of patients with hand dysfunction after stroke as an example, the high-precision infrared optical capture technology equipped in the motion capture unit can collect the complete three-dimensional motion trajectory of the patient's hand from extension, grasping to release in real time. It can accurately track the opening and closing range of each finger, the rotation angle of the wrist joint, and the dynamic parameters such as the overall movement rate and acceleration, clearly presenting the spatial posture and motion continuity of the patient's hand movements, providing an objective and quantitative core basis for the standardized assessment of rehabilitation movements. Furthermore, the collected real-time trajectory data is compared with the standard rehabilitation movement model built into the system at the millisecond level, automatically calculating the deviation value between the actual movement and the standard movement, accurately identifying problems such as movement compensation and trajectory deviation when the patient exerts force during grasping, helping rehabilitation therapists to quickly locate the weak links in the patient's hand function recovery, adjust the rehabilitation training plan in a targeted manner, avoid the solidification of incorrect movements, and improve the scientificity and safety of rehabilitation training. At the same time, the long-term collected trajectory data can build a personalized rehabilitation progress database for patients, realize the dynamic tracking and trend analysis of rehabilitation effects, provide data support for the iterative optimization of subsequent rehabilitation programs, and help patients recover fine motor function and daily living activities of the hand faster and better.

[0042] In this embodiment of the application, the VR rehabilitation scene construction module 200 includes: as follows Figure 3 As shown, there are scene generation unit, physics engine adaptation unit, and neural motion mapping unit.

[0043] The scene generation unit constructs a personalized virtual rehabilitation training scene based on the patient's rehabilitation stage, hand movement trajectory, muscle electrical signals, grip pressure data, and joint movement angles; the physics engine adaptation unit is used to dynamically adjust the physical properties of objects in the scene through an adaptive physics engine to simulate real contact force feedback; the neural motion mapping unit is used to establish a precise mapping relationship between hand movement data and virtual scene interaction actions through a neural motion mapping algorithm, and output hand simulation data including movement trajectory and force feedback.

[0044] It is understood that, through the scene generation unit, this application constructs a personalized virtual training scene that meets the patient's rehabilitation needs based on the patient's rehabilitation stage and multi-dimensional information such as hand movement trajectory, muscle electromyography, grip force pressure data, and joint range of motion. This achieves precise adaptation and dynamic adjustment of rehabilitation training content. The physics engine adaptation unit adjusts the physical properties of objects in the scene in real time through an adaptive physics engine, accurately simulating the contact force feedback in the real environment. This allows patients to obtain a tactile experience close to reality in virtual interaction, enhancing the immersion and realism of training. The neuro-motion mapping unit establishes a precise correspondence between hand movement data and virtual scene interaction actions based on the neuro-motion mapping algorithm. It outputs hand simulation data including movement trajectory and force feedback, ensuring the smoothness and accuracy of virtual interaction. This provides patients with a natural and efficient immersive rehabilitation training experience, improving the targeting, fun, and training effect of rehabilitation training. At the same time, it provides comprehensive data support for rehabilitation progress assessment and program optimization.

[0045] It should be noted that, based on the patient's rehabilitation stage and hand movement trajectory, electromyographic signals, grip strength and pressure data, and joint range of motion, a personalized virtual rehabilitation training scenario is constructed. First, the patient's rehabilitation stage is used as the core basis, combined with clinical rehabilitation assessment results (such as muscle strength level, joint range of motion, and motor control ability) to determine the target difficulty, movement type, and training intensity of the rehabilitation training, forming the basic framework of the virtual scenario. Simultaneously, a motion capture unit collects the patient's three-dimensional hand movement trajectory, extracting features such as spatial position, movement rate, and trajectory continuity to analyze the flexibility and coordination of hand movements. Furthermore, an electromyographic sensor array collects surface electromyographic signals of the forearm muscle groups, analyzing the intensity, frequency, and duration of muscle contractions to assess muscle activation. The system analyzes hand posture and force application patterns. It acquires hand grip strength and pressure distribution data via grip pressure sensors to quantify the balance of hand force application and grip stability. A joint angle acquisition unit monitors the range of motion and rotation rate of the fingers and wrist joints, accurately capturing the range and flexibility of joint movement. Subsequently, it correlates multi-dimensional hand data with rehabilitation stage goals, combining the patient's strengths and weaknesses to specifically adjust task settings, interaction methods, and difficulty levels in the virtual environment. By integrating motion feedback mechanisms, visual guidance elements, and real-time assessment modules, it ultimately constructs a virtual training scenario tailored to the patient's individual rehabilitation needs. This provides a precisely adapted and dynamically adjustable interactive environment for rehabilitation training, enhancing the training's relevance, engagement, and rehabilitation effectiveness.

[0046] By dynamically adjusting the physical properties of objects in the scene through an adaptive physics engine to simulate realistic contact force feedback, and guided by the interactive needs of hand rehabilitation training, this system leverages the real-time parallel computing capabilities of the adaptive physics engine to construct a basic physical property parameter library covering interactive objects of different materials and shapes. This library includes core indicators such as material hardness, elastic coefficient, coefficient of friction, mass distribution, and damping characteristics, providing a dynamically adjustable parameter basis for training needs at different rehabilitation stages and muscle strength levels. Simultaneously, it receives real-time muscle contraction intensity signals from electromyography (EMG) sensors, force magnitude and pressure distribution data from grip force sensors, and movement rate and posture parameters from joint angle acquisition units. Through the engine's built-in adaptive matching algorithm, it adjusts the physical properties of interactive objects in the scene in milliseconds based on the patient's actual hand force state, grip posture, and movement continuity. For example, for patients in the early stages of rehabilitation with weaker muscles, it automatically reduces the hardness and frictional resistance of objects to alleviate the burden on the hand and prevent muscle strain. For patients with excessive muscle fatigue and those in the middle to late stages of muscle recovery who have recovered relatively well, the mass, elasticity coefficient, and collision damping of objects are gradually increased to enhance the difficulty and challenge of training. Furthermore, a force feedback signal conversion algorithm is used to accurately convert the dynamic adjustment results of the object's physical properties into corresponding contact force feedback signals. This simulates the tactile sensation of holding an object, the resistance when pressing, the reaction force when grasping, and the inertial dragging sensation when rotating—all realistic physical experiences. For example, when a patient grasps a virtual water cup, the grip strength is fed back in real time; when the wrist rotates, the weight of the cup due to gravity is simulated; and when the object is placed, the collision feedback when it contacts a table is simulated. Ultimately, this achieves precise dynamic matching between the physical properties of objects in the virtual scene and the patient's hand state. This ensures both the safety and personalized adaptability of rehabilitation training, while enhancing the immersion and realism of virtual training through highly realistic contact force feedback. This helps patients establish correct force perception and motor control patterns, improving the scientific nature and effectiveness of rehabilitation training.

[0047] Adaptive matching algorithm formula:

[0048]

[0049] in, Parameters for dynamically adjusting the physical properties of virtual objects; These are the weighting coefficients corresponding to the electromyographic signals; This is the normalized value of the electromyographic signal intensity; The weighting coefficient corresponding to grip strength; This is the real-time grip strength value; These are the weighting coefficients corresponding to the joint angles; This represents the standardized deviation value of the joint angle; The weighting coefficients corresponding to the motion speed; This is the normalized value of the hand movement rate; The weighting coefficients corresponding to the rehabilitation stage; This represents the coefficient for the recovery stage.

[0050] Force feedback signal conversion algorithm formula:

[0051]

[0052] in, This is the output contact force feedback signal; For force feedback mapping coefficients; This represents the actual force exerted by the patient's hand. This corresponds to the standard exertion threshold for the rehabilitation stage; This is the basic offset for force feedback.

[0053] Neural Action Mapping Algorithm Formula:

[0054]

[0055]

[0056]

[0057] in, This is a feature vector of interactive actions in a virtual scene. It is a neural action mapping function; This is the preprocessed hand motion data matrix; This is the weight parameter matrix of the mapping model; This is the bias vector of the mapping model; for Real-time hand simulation motion trajectory coordinates; This is the motion trajectory transformation matrix; The initial coordinates of the hand; For trajectory correction terms; This refers to the force feedback vector in the virtual scene. This is the virtual contact stiffness coefficient; for The coordinates of the surface contact point of the virtual object at any given time; The distance between the simulated hand trajectory and the virtual object; This is the normal vector of the contact point of the virtual object.

[0058] For example, in a VR industrial-grade motor repair training scenario: after wearing data gloves integrating an inertial measurement unit and a bending sensor, and a VR headset equipped with a scene rendering module, the trainee first locates the M12 bolt area of ​​the end cap of a certain type of three-phase asynchronous motor in the virtual environment; when the trainee makes the hand action of "holding the bolt and turning it clockwise", the neural motion mapping unit will collect multi-dimensional data such as the bending angle of the index finger / thumb joint, the angular velocity of wrist rotation, and the acceleration of hand movement transmitted back by the gloves in real time. The data is first preprocessed by denoising and time-series normalization, and then matched with a pre-trained "industrial bolt turning action library" (containing virtual interaction parameters corresponding to different torques and angles) through the built-in CNN-LSTM hybrid neural motion mapping algorithm to establish a precise mapping relationship between the hand operation data and the "virtual M12 bolt rotation" action, and output the hand simulation simultaneously. In the real data, the motion trajectory is refined into a "continuous spatial coordinate sequence from the initial grip coordinates (X:120, Y:85, Z:210) of the virtual bolt to the center of the bolt, rotating 32° clockwise." The force feedback combines the rigidity parameters of the virtual bolt's 45# steel to calculate parameters such as "initial tightening resistance of 1.2 N·m, and resistance linearly increasing to 3.5 N·m when the torque approaches the preset threshold." This is then transmitted to the micro force feedback vibration module of the data glove, allowing trainees to experience a "smooth initial movement followed by a stuttering" tactile sensation consistent with that of a real bolt. If the trainee's tightening angle deviation exceeds 5°, the unit will also adjust the virtual motion matching degree through trajectory correction items and simultaneously pop up motion specification prompts in the VR field of view. Ultimately, this achieves a training effect of "no physical equipment damage, highly realistic operation tactile sensation, and real-time motion error correction," significantly reducing the cost and risk of industrial maintenance practical training.

[0059] In this embodiment of the application, the intelligent evaluation center 300 includes: Figure 4 As shown, there are three units: standardization assessment unit, muscle activation analysis unit, and abnormal pattern recognition unit.

[0060] The unit includes a movement standardization assessment unit, which uses an improved CNN-GRU model to extract movement features from hand simulation data, compares them with standard rehabilitation movement templates, calculates movement similarity scores, and assesses movement standardization; a muscle activation efficiency analysis unit, which calculates muscle activation rate and synergistic work efficiency based on electromyographic signal simulation data; and an abnormal pattern recognition unit, which identifies compensatory movements and abnormal features of force imbalance, and generates a rehabilitation assessment report that includes assessment indicators and abnormal analysis.

[0061] It is understood that the action standardization assessment unit in this application relies on an improved CNN-GRU model to efficiently extract action features from hand simulation data, accurately compares them with standard rehabilitation action templates, and generates a similarity score, providing a quantitative basis for action standardization; the muscle activation efficiency analysis unit, based on electromyographic signal simulation data, accurately calculates muscle activation rate and synergistic efficiency, deeply reflecting the state of muscle function recovery; the abnormal pattern recognition unit can intelligently capture abnormal features such as compensatory movements and force imbalances, and integrate multi-dimensional assessment indicators to generate a structured rehabilitation assessment report. This not only realizes the full-chain assessment from action form to muscle function, improving the scientificity and comprehensiveness of rehabilitation assessment, but also provides data support for the optimization of subsequent personalized rehabilitation plans, effectively reducing the risk of compensatory injury, accelerating the rehabilitation process, and significantly improving the pertinence and effectiveness of rehabilitation training.

[0062] It should be noted that the improved CNN-GRU model formula is as follows:

[0063]

[0064]

[0065]

[0066]

[0067]

[0068]

[0069] in, This is the stitched multimodal hand simulation data matrix; Parameters for interactive actions in the virtual scene; For force feedback simulation data; For dimension is × The real number space; For feature dimensions; The length of the time series sequence; The CNN spatial features of the action trajectory at time t; This is a two-dimensional convolution operation; The motion trajectory data at time t; The weights of the convolution kernels for the action trajectory branches; The convolution bias for the motion trajectory branch; It is activated by the ReLU function; This is a two-dimensional max pooling operation; The CNN spatial features represent the force feedback at time t; The force feedback data at time t; The convolution kernel weights are for the force feedback branch; The convolution bias is used for the force feedback branch; Let be the fused spatial feature vector at time t; For feature splicing operations; The feature fusion weight matrix; For feature fusion bias; Let t be the update gate output of the GRU; Use the Sigmoid activation function; To update the weight matrix of the gate; Let t be the hidden state of the GRU at time t-1; To update the bias vector of the gate; The output of the GRU reset gate at time t; To reset the weight matrix of the gate; To reset the gate's bias vector; Let be the candidate hidden state of GRU at time t; It is the hyperbolic tangent activation function; Let be the weight matrix of the candidate hidden states; is the bias vector of the candidate hidden state; Let t be the temporal feature vector output by the GRU at time t; Let t be the temporal attention weight; It is an exponential function; This is the transpose of the attention weight vector; This is the attention transformation matrix; For attentional bias; This is the index of the time series sequence at time k; The time-series feature vector output by the GRU at time k This is the weighted and fused global action feature vector; This is the index of the time series sequence at time t; Score the similarity of actions; This is the transpose of the feature vector of the standard rehabilitation movement template. It is the L2 norm of the vector; The feature vector is a standard rehabilitation movement template.

[0070] The muscle activation efficiency analysis unit is used to calculate muscle activation rate and synergistic efficiency based on electromyography (EMG) signal simulation data (i.e., the temporal signals of contraction intensity of various forearm muscle groups synchronized with hand interaction movements output by the neuro-motor mapping unit). First, the EMG signal simulation data is processed by muscle group (e.g., distinguishing different functional muscle groups such as forearm flexors and extensors, and filtering signal noise). Then, the muscle activation rate is calculated using a muscle activation rate calculation algorithm. Based on the standard EMG activation threshold for the corresponding rehabilitation movement, the proportion of time the EMG signal intensity of the target muscle group exceeds the threshold within the entire movement cycle is statistically analyzed. Simultaneously, the coverage ratio of the actually activated muscle area and the area required for activation in the standard movement is combined to obtain the activation rate of a single muscle group, thus quantifying the single muscle group. The study assesses the sufficiency of muscle activation by using a muscle group coordination efficiency calculation algorithm. This involves analyzing the temporal synchronicity, phase difference, and activation intensity matching of electromyographic (EMG) signals from multiple related muscle groups (such as flexors and extensors that assist in stabilization during grasping movements). For example, it calculates the cross-correlation coefficients of EMG signals from different muscle groups to assess whether their activation sequence aligns with the standard coordination pattern. Simultaneously, it statistically analyzes the percentage of time spent in synchronous activation across multiple muscle groups to obtain the level of coordination between muscle groups. Finally, it integrates these two quantitative indicators to output evaluation results, helping rehabilitation therapists accurately locate areas of insufficient muscle activation or problems with muscle group coordination disorder. This allows for targeted adjustments to the design of rehabilitation exercises and force guidance, improving the accuracy and efficiency of muscle function recovery.

[0071] Muscle activation rate calculation algorithm:

[0072]

[0073] in, The activation rate of a single muscle; This is the first weighting coefficient; The duration for which the electromyographic signal intensity exceeds the standard activation threshold; The total duration of the entire rehabilitation exercise cycle; This is the second weighting coefficient; This represents the area of ​​the muscle region that was actually activated. This refers to the area of ​​muscle regions that need to be activated for standard rehabilitation exercises.

[0074] Algorithm for calculating muscle group coordination efficiency:

[0075]

[0076]

[0077]

[0078] in, Let be the cross-correlation coefficient between muscle group i and muscle group j; For covariance; The electromyographic signal timing data for muscle group i; The electromyographic signal timing data for muscle group j; The standard deviation of the electromyographic signal of muscle group i; represents the standard deviation of the electromyographic signal of muscle group j; The activation phase difference between muscle group i and muscle group j; The phase angle of the signal spectrum; For Fast Fourier Transform; To improve the efficiency of muscle group coordination; The number of muscle groups involved in the synergy; To traverse all different muscle group pairs; These are the weighting coefficients corresponding to the cross-correlation coefficients; The absolute value of the cross-correlation coefficient between muscle group i and muscle group j; These are the weighting coefficients corresponding to the phase difference; The absolute value of the activation phase difference between muscle group i and muscle group j; Pi; The weighting coefficients corresponding to the activation strength matching degree; It is the smaller value between the maximum activation intensities of muscle groups i and j; It is the larger of the maximum activation intensities of muscle groups i and j.

[0079] To identify compensatory movements and abnormal features of force imbalance, a rehabilitation assessment report containing evaluation indicators and anomaly analysis is generated. First, an anomaly identification dataset is constructed by integrating hand movement trajectories output from the neuro-motor mapping unit, force feedback simulation data, single muscle group activation rates and muscle group synergistic efficiency calculated by the muscle activation efficiency analysis unit, and movement similarity scores output by an improved CNN-GRU model. Then, for compensatory movement identification, the difference in activation rates between target and non-target muscle groups is compared, combined with spatial deviation characteristics of the movement trajectory (e.g., when a patient relies excessively on shoulder compensation to drive hand movement due to weakness of the target muscle group, the hand trajectory will show non-standard deviations), and simultaneously matched with preset compensatory movement feature templates (e.g., insufficient flexor activation but abnormally high biceps activation rate in grasping movements), to identify the specific manifestations and timing of compensatory movements. For abnormal force imbalance identification, through… By analyzing indicators such as whether the efficiency of muscle group coordination is lower than a preset threshold, the temporal synchronization deviation of multi-muscle group electromyographic signals, and the symmetry of grip force pressure distribution, the type of force imbalance (such as activation intensity imbalance, temporal coordination imbalance, and pressure distribution imbalance) is determined. Based on this, the unit will quantitatively compare the identified abnormal features with the standard thresholds of the rehabilitation stage, extract key information such as the degree of abnormality and frequency of occurrence, and finally generate a rehabilitation assessment report containing core assessment indicators (movement similarity score, muscle activation rate, muscle group coordination efficiency, frequency of abnormal occurrence), abnormal-specific analysis (abnormality type, specific manifestation, scope of impact, potential causes), and rehabilitation optimization suggestions. This provides rehabilitation therapists with data support and decision-making basis for accurately grasping the problems in the patient's rehabilitation process, adjusting personalized training programs, effectively avoiding training injuries caused by compensatory movements, and improving the scientific nature and pertinence of rehabilitation training.

[0080] In this embodiment of the application, the interactive training unit 400 includes, as follows: Figure 5 As shown, the VR rendering engine, multimodal prompting unit, and training guidance generation unit are included.

[0081] The VR rendering engine is used to render the movement state of the virtual hand model, the progress of the training task, and the details of the scene environment in real time; the multimodal prompting module is used to provide voice command prompts, visual highlighting guidance, and tactile vibration feedback; and the training guidance generation unit is used to generate targeted training guidance based on the weak points in the rehabilitation assessment report.

[0082] It is understood that the embodiments of this application use a VR rendering engine to render the movement state of the virtual hand model, the progress of the training task, and scene details in real time. This allows patients to intuitively and synchronously see the virtual mapping of their own movements, enhancing the immersion and engagement of the training, and also enabling them to clearly grasp the progress of the task, thus improving the goal-orientation and continuity of the training. The multimodal prompting unit uses a combination of multisensory guidance, including voice commands, visual highlighting, and tactile vibration, to adapt to the perceptual habits of different patients, making the movement guidance more precise and intuitive, helping to quickly correct deviations, standardize movements, and reduce the occurrence of compensatory movements. The training guidance generation unit generates targeted guidance based on the weaknesses identified in the rehabilitation assessment report, accurately focusing on the patient's movement shortcomings and insufficient muscle activation, achieving personalized adaptation of training content, avoiding the inefficiency of blind training, improving rehabilitation effects, and providing a good interactive experience for rehabilitation training.

[0083] For example, taking a patient in the early stages of hand rehabilitation as an example, their rehabilitation assessment report clearly identifies the weaknesses as "flexor activation rate of only 40% (below the standard threshold of 70% for the same stage), finger opening and closing deviation in grasping movements of 15%, and shoulder compensatory movements occurring 3 times per training cycle." The training guidance generation unit first analyzes the type and degree of these weaknesses, and then matches corresponding specialized content from the rehabilitation training resource library: for insufficient flexor activation, "progressive virtual grip strengthener training" is selected; for the weakness in finger opening and closing, "virtual small bead fine pinching task" is matched; for shoulder compensation, "shoulder fixed hand directional grasping training" is called, and specific training guidelines are generated: specifying 2 training sessions per day. The training consisted of 15 repetitions per set, with a difficulty gradient rule of "increasing the resistance of the hand gripper after the flexor activation rate increases to 50%". Key points to monitor during training, including "flexor activation state, finger opening and closing trajectory, and shoulder range of motion," were also highlighted. This guidance was then synchronized to the VR rendering engine (rendering the corresponding training scene) and the multimodal prompting unit (providing voice prompts and tactile vibration feedback for flexor activation points). After three cycles of training according to this guidance, the patient's flexor activation rate increased to 62%, the deviation in finger opening and closing range of motion decreased to 8%, and the frequency of shoulder compensation dropped to 0 times per cycle. This precisely strengthened weak abilities while avoiding the inefficiency of blind training, effectively accelerating the rehabilitation process.

[0084] In this embodiment of the application, the personalized control unit 500 includes, as follows: Figure 6 As shown, there are three units: rehabilitation trend prediction unit, training parameter adjustment unit, and scenario difficulty adaptation unit.

[0085] Among them, the rehabilitation trend prediction unit is used to predict the recovery trend and phased goals of hand function by combining reinforcement learning algorithms with historical training data and evaluation reports; the training parameter adjustment unit is used to dynamically adjust the training duration, number of repetitions of the movement, and grip strength target threshold parameters; and the scenario difficulty adaptation unit is used to gradually increase the scenario complexity according to the rehabilitation progress.

[0086] Understandably, the rehabilitation trend prediction unit in this application uses reinforcement learning algorithms to deeply mine the patient's historical training data and previous rehabilitation assessment reports to accurately predict the recovery trend of hand function and formulate phased rehabilitation goals. This provides a scientific basis for subsequent training adjustments and allows patients to clearly see the rehabilitation path, enhancing their sense of purpose and motivation in training. The training parameter adjustment unit dynamically optimizes core parameters such as training duration, number of repetitions of movements, and grip strength target threshold based on the predicted recovery trend and real-time training status. This avoids inefficiency caused by insufficient training intensity or muscle fatigue and injury caused by excessive intensity, ensuring that training always matches the patient's current muscle strength level and recovery rhythm. The scenario difficulty adaptation unit gradually upgrades the scenario complexity according to the rehabilitation progress. By progressively increasing the difficulty, it continuously applies appropriate challenges to the patient's hand function, avoiding the bottleneck of ability improvement caused by staying at the same difficulty level for a long time. This allows rehabilitation training to shift from a fixed mode to dynamic adaptation, accurately matching the patient's personalized recovery rhythm and continuously promoting the improvement of hand function to a higher level, effectively shortening the rehabilitation cycle and improving the scientific and efficient nature of rehabilitation training.

[0087] It should be noted that the reinforcement learning algorithm formula is as follows:

[0088]

[0089]

[0090]

[0091]

[0092]

[0093]

[0094] in, Let be the state vector at time t; For the action-related indicators at time t; The similarity score at time t; For evaluation metrics at time t; These are the virtual training parameters at time t; The training progress parameter at time t; This is the transpose operation for a vector; It is a 5x1 vector space of real numbers; Let t be the action vector at time t; At time t+1 The predicted value of the indicator; At time t+1 The predicted value of the indicator; At time t+1 The predicted value of the indicator; Let be the reward value at time t; for The weighting coefficients corresponding to the indicators; It is an exponential function; for The mean of the indicator; for The weighting coefficients corresponding to the indicators; for The mean of the indicator; for The weighting coefficients corresponding to the indicators; for The mean of the indicator; State at time t ,action The action value function; These are the model parameters for the Q function; Discount factor; This is the operation that takes the maximum value of all actions at time t+1; Let t+1 be the state vector. This is the action vector at time t+1; Let be the action value function of the target Q-network at time t+1; These are the model parameters for the target Q-network; The loss function is the Q-function; For the state and actions Expectation calculation; Let be the target Q value at time t.

[0095] The training parameter adjustment unit dynamically adjusts training duration, repetition count, and grip strength target threshold parameters. Regarding training duration, the unit adjusts based on the patient's real-time muscle activation stability, movement accuracy fluctuations, and fatigue feedback data. If the patient's muscle activation rate remains at the target level and there are no obvious signs of fatigue, the duration of each training session is appropriately extended to enhance the training effect. If fatigue signals such as a sudden drop in movement accuracy or a decline in muscle group coordination efficiency occur, the duration is shortened promptly to avoid overtraining and muscle injury. Regarding the repetition count, the unit dynamically adjusts based on the movement similarity score and the achievement of the target muscle activation rate: for training tasks with high movement accuracy and sufficient muscle activation, the number of repetitions is appropriately reduced to avoid ineffective repetitions; for weak movements (such as gripping movements corresponding to insufficient flexor activation), the number of repetitions is increased, and progressive repetition rules are set to strengthen muscle memory and movement accuracy through high-frequency targeted training. To address the target grip strength threshold, the unit combines muscle activation efficiency analysis results with rehabilitation stage goals to develop a step-by-step improvement strategy. Initially, it starts with a low threshold to ensure patients can complete basic movements and build confidence. As muscle activation rate gradually increases, the target grip strength threshold is steadily raised according to a preset gradient, continuously providing appropriate challenges to the muscles and promoting gradual muscle strength recovery. Simultaneously, the unit synchronizes the adjusted parameters in real time to the VR rendering engine and multimodal prompting unit, ensuring precise matching between the training scenario, prompt content, and new parameters. This avoids the rigidity of fixed parameter training and allows for flexible adaptation based on the patient's real-time status and recovery pace. While ensuring training safety, it maximizes the targeting and efficiency of rehabilitation training, helping patients steadily achieve their stage-by-stage rehabilitation goals.

[0096] The scenario difficulty adaptation unit integrates the achievement status of rehabilitation assessment reports, the recovery stage judgment of the rehabilitation trend prediction unit, and the data on movement standardization and muscle activation efficiency in real-time training as the core basis for adjusting scenario complexity. It dynamically adapts to the patient's rehabilitation progress in a step-by-step manner. Initially, it starts with extremely simple single-movement scenarios, such as grasping static virtual objects without interference. Only the core training objectives are retained in the scenario, reducing environmental interference and helping patients focus on movement standardization and muscle activation. When the patient's movement similarity, muscle activation rate and other indicators are stable and meet the standards, the scenario complexity is gradually increased. For example, dynamic obstacles, multi-tasking requirements, or adjustments to the physical properties of virtual objects are added to the training scenario. As the rehabilitation progresses, the scenario is further upgraded to simulate real-life scenarios, such as taking and putting away tableware and organizing clothes in a virtual kitchen, allowing patients to complete training in an environment close to real life and improve their practical application ability of hand function. At the same time, the unit will work in conjunction with the training parameter adjustment unit to ensure that the increase in scenario difficulty is matched with the adjustment of training duration, number of repetitions of movements, and grip strength target threshold, so as to avoid the difficulty being out of touch with the patient's current ability, and to always maintain an appropriate level of challenge in training to improve hand flexibility.

[0097] This application proposes a hand rehabilitation training system based on virtual reality and gesture interaction. The system comprehensively collects hand movement trajectories, muscle electrical signals, grip pressure data, and joint range of motion through a gesture perception module, providing multi-dimensional and accurate basic data for rehabilitation training and assessment. A VR rehabilitation scene construction module, combining an adaptive physics engine and a neuro-motor mapping algorithm, accurately simulates hand interaction behavior and force feedback effects in a real-world environment, achieving dynamic mapping between hand movement data and the virtual scene, overcoming the limitations of traditional rehabilitation training methods such as single scene design and delayed feedback. An intelligent assessment center, utilizing an improved CNN-GRU model to integrate hand simulation data, enhances the accuracy of rehabilitation training movement standard assessment and muscle activation efficiency analysis, and strengthens the identification of movement compensation patterns and force abnormalities. An immersive VR training scene constructed by the interactive training unit intuitively presents hand movement status and training progress, facilitating the efficient generation of personalized training guidance. A personalized control unit predicts rehabilitation trends and stage goals based on rehabilitation assessment reports and real-time training data, dynamically adjusting training parameters and scene difficulty, effectively improving the targeting, personalization, and efficiency of hand function recovery in rehabilitation training. This solves the problems of low interactive accuracy and lack of data sharing in existing technologies.

[0098] The following will illustrate a hand rehabilitation training system based on virtual reality and gesture interaction through a specific embodiment, such as... Figure 7 As shown, it includes:

[0099] In the neurorehabilitation training room of a rehabilitation medicine center, personalized rehabilitation training services are being provided for patients with hand dysfunction after stroke. Traditional rehabilitation training relies on therapist guidance, which suffers from problems such as homogenized training content, highly subjective movement assessment, and low patient participation, making it difficult to accurately match the recovery pace of different patients. A hand rehabilitation training system based on virtual reality and gesture interaction, through the deep integration of multimodal perception, virtual scene interaction, and intelligent assessment and control, has achieved precision, personalization, and efficiency in hand rehabilitation training, significantly improving patient rehabilitation outcomes and training compliance.

[0100] System hardware deployment and network architecture

[0101] The gesture perception module, serving as the core of the system's data acquisition, comprises an optical motion capture unit, a flexible electromyography (EMG) sensor array, an array-type grip force and pressure sensor, and a joint angle acquisition unit, forming a comprehensive hand data acquisition network. The optical motion capture unit uses eight infrared cameras (sampling frequency 120Hz, positioning accuracy ≤0.1mm) to capture real-time three-dimensional hand movement trajectories and posture changes via reflective markers attached to key nodes of the patient's hand (fingertips, metacarpophalangeal joints, wrist joints), accurately reproducing subtle movements such as finger opening and closing, and wrist flexion and extension. The flexible EMG sensor array consists of six patch-type sensors attached to functional muscle groups such as the forearm flexors and extensors, collecting surface EMG signals during muscle contraction (sampling frequency 1000Hz, signal-to-noise ratio ≥80dB), providing raw data for muscle activation efficiency analysis. Data: An array of grip force sensors is integrated into the palm and fingertips of a customized training glove, employing piezoresistive sensing technology to collect data on hand grip force (measurement range 0-500N, accuracy ±5%) and pressure distribution (resolution 256×256 pixels), reflecting the symmetry and uniformity of hand force application in real time. A joint angle acquisition unit, based on a miniature inertial measurement unit (IMU), integrates an accelerometer, gyroscope, and magnetometer to monitor the movement angles of the fingers and wrist joints in real time (measurement range 0-180°, accuracy ±1°) and rotation speed (measurement range 0-300° / s). This multimodal data establishes a low-latency transmission link (end-to-end latency ≤20ms) through a 5G industrial gateway and edge computing nodes. After preprocessing (Butterworth filtering, signal noise reduction, time synchronization), the data is uploaded to the rehabilitation center's cloud database, ensuring real-time performance and integrity, providing high-quality data support for subsequent scenario construction, evaluation analysis, and control decisions.

[0102] Implementation Details of VR Rehabilitation Scene Construction Module

[0103] The VR rehabilitation scene construction module, through the collaborative work of the scene generation unit, the physics engine adaptation unit, and the neuro-motor mapping unit, achieves the precise construction of personalized rehabilitation scenes and the realistic simulation of hand interaction behaviors. The scene generation unit, using Unity3D as the development engine, constructs hierarchical personalized rehabilitation training scenes based on the patient's rehabilitation stage (e.g., initial, intermediate, and recovery phase) and hand movement data collected by the gesture perception module: for patients in the initial phase, it constructs interference-free static grasping scenes (e.g., cylinders and cubes on a virtual desktop), simplifying scene elements and highlighting training objectives; for patients in the intermediate phase, it constructs fine manipulation scenes including dynamic obstacles (e.g., picking up and placing items on a virtual shelf); and for patients in the recovery phase, it constructs complex scenes simulating daily life (e.g., organizing tableware and folding clothes in a virtual kitchen). The physics engine adaptation unit uses the NVIDIA PhysX physics engine, which dynamically adjusts the physical properties of objects in the scene through an adaptive parameter adjustment algorithm: the weight (range 0.1-5kg), friction coefficient (range 0.1-0.8), and collision response parameters of virtual objects are adjusted in real time based on the patient's grip strength data to simulate the force feedback of objects in a real environment; when the patient's hand comes into contact with the virtual object, tactile stimulation with corresponding vibration intensity (vibration frequency 50-200Hz, amplitude 0.1-0.5mm) is output through the haptic feedback glove to achieve cross-sensory linkage of "vision-touch". The neural motion mapping unit, based on an improved LSTM-attention mechanism model, establishes a precise mapping relationship between hand multimodal data and virtual scene interactive actions: it fuses hand movement trajectory, electromyographic signals, and grip pressure data into a temporal feature vector, extracts action temporal features through LSTM, introduces an attention mechanism to strengthen the weight of key action frames, and finally outputs hand simulation data containing action trajectory (3D coordinate error ≤ 0.2 mm) and force feedback (pressure error ≤ 3%), achieving real-time synchronization between the patient's hand movements and the virtual model (synchronization delay ≤ 30 ms), ensuring the realism and smoothness of virtual interaction.

[0104] Intelligent evaluation central algorithm implementation

[0105] The intelligent assessment center consists of a movement standardization assessment unit, a muscle activation efficiency analysis unit, and an abnormal pattern recognition unit, enabling multi-dimensional and accurate assessment of rehabilitation training effects. The movement standardization assessment unit employs an improved CNN-GRU hybrid model to extract movement features from hand simulation data and compare them with standard rehabilitation movement templates: the CNN part extracts local features (such as finger opening and closing amplitude, palm pressure center offset) from the spatial dimension (movement trajectory, pressure distribution) of the hand simulation data through three convolutional layers (kernel sizes 3×3, 5×5, 7×7); the GRU part captures the temporal evolution of movements (such as movement continuity, speed change trends) through two hidden layers (256 neurons), and introduces a temporal attention mechanism (attention coefficient dynamically adjusted through a softmax function) to strengthen the feature weights of key movement stages. After training with 50,000 sets of labeled data (including 10 types of rehabilitation movements such as grasping, extending, and rotating), the model achieves a 95% accuracy rate in calculating movement similarity scores, with a single movement assessment time ≤0.3 seconds. The muscle activation efficiency analysis unit calculates the muscle activation rate based on electromyographic signal simulation data through threshold determination and a dual-dimensional proportion fusion algorithm. Simultaneously, it uses cross-correlation coefficients and phase difference analysis to evaluate the efficiency of muscle group coordination: using the standard electromyographic activation threshold at the same rehabilitation stage as a benchmark, it statistically analyzes the percentage of time the target muscle group's electromyographic signal exceeds the threshold and the percentage of activation area coverage, weighting them to obtain the muscle activation rate (weighting coefficients are 0.6 and 0.4, respectively). By calculating the cross-correlation coefficients (range -1 to 1) and activation phase differences (range 0 to π) of multi-muscle group electromyographic signals, it quantifies the synchronicity and coordination of muscle group coordination, with a coordination efficiency calculation error ≤5%. The abnormal pattern recognition unit, based on rule matching and machine learning classification algorithms, identifies compensatory movements and force imbalances: by comparing the activation rate differences between target and non-target muscle groups (e.g., insufficient flexor activation but abnormally high biceps activation), and combining this with spatial deviations in hand movement trajectories (e.g., trajectory deviation caused by shoulder-driven hand movement), it identifies the type of compensatory movement (e.g., shoulder compensation, wrist compensation) and the time of occurrence (time accuracy ≤ 0.1 seconds); by analyzing whether muscle group synergy efficiency is lower than a preset threshold (e.g., synergy efficiency < 0.6) and the symmetry deviation of grip force distribution (e.g., left-right grip force difference > 20%), it determines the type of force imbalance (e.g., activation intensity imbalance, timing coordination imbalance). Finally, the unit integrates multi-dimensional assessment indicators to generate a standardized rehabilitation assessment report including movement similarity score, muscle activation rate, muscle group synergy efficiency, frequency of abnormal occurrence, and specific analysis, with an output time ≤ 1 second.

[0106] Interactive training unit application

[0107] The interactive training unit, through the collaboration of a VR rendering engine, a multimodal prompting unit, and a training guidance generation unit, constructs an immersive and highly interactive rehabilitation training experience. The VR rendering engine employs physically based rendering (PBR) to render the hand virtual model's movement state, training task progress, and scene environment details in real time at a frame rate of 90fps. The hand virtual model uses skeletal binding technology, with a synchronization error of ≤0.1 seconds with the patient's hand movements, accurately reproducing the subtle flexion and extension of the fingers and wrist rotation. The training task progress is fed back in real time through virtual progress bars, voice broadcasts, and other means to help patients understand their training goals. The scene environment details use high-definition texture mapping and global illumination rendering to create a realistic visual immersion. The multimodal prompting unit provides multi-sensory guidance combining voice commands, visual highlighting, and tactile vibration: voice commands use TTS speech synthesis technology, supporting Chinese semantic broadcasting (recognition accuracy of 98%), and providing real-time prompts for proper action (such as "please increase the range of finger opening and closing"); visual highlighting uses color gradients of virtual models (such as the target muscle group area changing from blue to green) to highlight training focus; tactile vibration feedback uses wearable tactile gloves to output vibration prompts at corresponding locations when there is a deviation in action (vibration intensity is positively correlated with the degree of deviation), with a response time of ≤0.1 seconds. The training guidance generation unit, based on weaknesses identified in the rehabilitation assessment report, matches targeted training content from the rehabilitation training resource library: For insufficient flexor activation, it generates a "progressive grip strength increasing training" guide, specifying the training frequency (2 sets per day, 15 repetitions per set) and difficulty gradient rules (increasing the weight of the virtual object if muscle activation rate increases by 10% after every 3 training sessions); for weak finger opening and closing range, it generates a "virtual fine motor skills training" guide, setting training tasks for pinching virtual objects of different sizes (5mm to 20mm in diameter); for compensatory movements, it generates a "target muscle group isolation activation training" guide, strengthening the force control of the target muscle group through restraint training with a fixed shoulder. Simultaneously, the unit synchronizes the training guidance in real-time to the VR rendering engine and multimodal prompting unit, achieving precise matching of training content with scenarios and prompts, forming a closed-loop training process of "guidance-execution-feedback," significantly improving patient training compliance and movement standardization.

[0108] Personalized control unit operation mechanism

[0109] The personalized control unit achieves dynamic optimization and precise control of rehabilitation training through the linkage of the rehabilitation trend prediction unit, training parameter adjustment unit, and scenario difficulty adaptation unit. The rehabilitation trend prediction unit is based on a deep reinforcement learning algorithm (DQN model), combined with the patient's historical training data and rehabilitation assessment report, to predict the trend of hand function recovery and the stage goals: using muscle activation rate, movement similarity score, muscle group coordination efficiency, cumulative training time, and rehabilitation stage level as state vectors, and the stage goal value of each indicator as the action vector. The model parameters are optimized through an immediate reward function (integrating the achievement degree of each indicator). After training with 10,000 sets of historical patient data, the model achieved a rehabilitation trend prediction accuracy of 92%, and the rationality score of the stage goal setting was ≥4.5 / 5 points (therapist assessment). The training parameter adjustment unit dynamically optimizes three core parameters—training duration, number of repetitions, and grip strength target threshold—based on rehabilitation trend prediction results and real-time training data. When the patient's muscle activation rate consistently meets the target and there are no signs of fatigue, the training duration of a single session is appropriately extended (e.g., from 20 minutes to 30 minutes). When the movement similarity score is ≥90 points, the number of repetitions of that movement is reduced (e.g., from 15 repetitions per set to 10 repetitions). When grip strength increases to 80% of the stage target, the grip strength target threshold is steadily increased in a 5% increment to ensure that the training intensity is precisely matched with the patient's recovery status. The scenario difficulty adaptation unit gradually increases the complexity of the scenarios according to the rehabilitation progress, strictly adhering to the principle of gradual progression: before each difficulty upgrade, the patient's mastery of the current difficulty is verified (e.g., a similarity score of ≥85 points and an abnormal occurrence frequency of ≤1 time / training cycle); it gradually upgrades from the initial grasping of a single static object to the intermediate fine picking and placing in a multi-obstacle environment, and then to the later simulation of daily life movements; at the same time, the increase in scenario difficulty and the adjustment of training parameters are synchronized to ensure that the difficulty matches the patient's current ability, avoiding the deformation of movements or frustration caused by excessive difficulty, and preventing the inability to effectively stimulate hand function improvement due to insufficient difficulty. Through a dynamic control mechanism, the system realizes the transformation of rehabilitation training from a "fixed mode" to "personalized dynamic adaptation", effectively shortening the patient's rehabilitation cycle and improving the efficiency and quality of hand function recovery.

[0110] In summary, this application's embodiments utilize a gesture perception module to collect multi-dimensional and precise hand data, providing a reliable input foundation for system operation; a VR rehabilitation scene construction module enables personalized scene adaptation and realistic interactive simulation, enhancing training immersion and engagement; an intelligent assessment center, through improved algorithms, accurately assesses movement standardization, analyzes muscle activation efficiency, and identifies abnormalities, avoiding the subjective biases of traditional manual assessments; an interactive training unit, through multimodal prompts and targeted training guidance, forms a closed-loop training process, significantly improving patient training compliance and movement standardization; and a personalized control unit dynamically optimizes training parameters and scene difficulty, achieving precise matching between training and patient recovery rhythm, effectively solving the problems of homogenization and poor adaptability in traditional rehabilitation training, shortening the rehabilitation cycle, improving the quality of hand function recovery, and helping patients quickly return to daily life.

[0111] Next, referring to the accompanying drawings, a hand rehabilitation training method based on virtual reality and gesture interaction is described according to an embodiment of this application.

[0112] like Figure 8 As shown, this hand rehabilitation training method based on virtual reality and gesture interaction includes the following steps:

[0113] In step S101, the hand movement trajectory, muscle electrical signals, grip pressure data, and joint range of motion are acquired.

[0114] It is understood that, by acquiring hand movement trajectories, muscle electrical signals, grip force and pressure data, and joint range of motion, the system can accurately perceive the patient's hand movement state, muscle activation, and force characteristics. It can capture key rehabilitation training information in real time, such as hand movement deviations, insufficient muscle activation, and force imbalances. This provides high-quality data for subsequent VR personalized rehabilitation scene construction, intelligent assessment of movement standards and anomaly identification, targeted guidance generation for interactive training units, and parameter and difficulty optimization for personalized control units. Real-time perception and precise control of the patient's hand rehabilitation status avoids the problem of poor training adaptability caused by inaccurate and inconsistent data collection in traditional rehabilitation training. This lays a reliable data foundation for the precise and personalized implementation of rehabilitation training, improves the targeting and effectiveness of training, and helps shorten the rehabilitation cycle and improve the quality of hand function recovery.

[0115] In step S102, based on the patient's rehabilitation stage and hand movement trajectory, muscle electrical signals, grip force pressure data and joint movement angles, a personalized rehabilitation training virtual scene is constructed. Through an adaptive physics engine and a neuro-motion mapping algorithm, the hand interaction behavior and force feedback effect in a real environment are simulated, and the patient's hand simulation data is output.

[0116] Among them, the adaptive physics engine is a core component of physical simulation that can dynamically adjust the physical properties of objects in a virtual scene, simulate real contact force feedback, and adapt to the patient's hand state to ensure the realism and adaptability of virtual interaction.

[0117] It is understood that the embodiments of this application utilize an adaptive physics engine, which can dynamically adjust the physical properties of virtual objects based on the patient's rehabilitation stage and multi-dimensional hand data. This accurately simulates hand interaction behavior and force feedback in a real environment, ensuring the realism and adaptability of virtual interaction. At the same time, it provides support for the output of accurate hand simulation data by the neuro-motor mapping algorithm, laying the foundation for subsequent intelligent assessment, personalized training guidance generation, and parameter control. This effectively enhances the patient's immersion and sense of involvement in training, avoids the problems of unrealistic and poor adaptability in traditional virtual training interactions, and strengthens the targeting and effectiveness of training.

[0118] It should be noted that the neural motion mapping algorithm is an algorithm that integrates deep learning temporal modeling. It is used to integrate multimodal information such as hand movement trajectory, muscle electrical signals, grip force pressure data and joint range of motion to establish a precise mapping relationship between real hand movements and virtual scene interaction behavior. It outputs hand simulation data containing movement trajectory and force feedback, realizes real-time synchronization between the patient's hand movements and the virtual model, and ensures the realism and smoothness of virtual interaction.

[0119] For example, taking a stroke patient with early-stage hand dysfunction as an example, the adaptive physics engine played a crucial role in hand rehabilitation training based on virtual reality and gesture interaction: When the patient wore gesture-sensing gloves for virtual grasping training, the engine received grip force data collected in real time from an array of grip force pressure sensors (initially only reaching 20N), automatically reducing the weight of the virtual cylinder from the default 0.5kg to 0.2kg, while adjusting the coefficient of friction to 0.3 to reduce the difficulty of grasping and adapt to the patient's current muscle strength level; when the patient's hand came into contact with the virtual cylinder, the engine triggered the haptic feedback glove to output a vibration signal with a frequency of 80Hz and an amplitude of 0.2mm, simulating the tactile feedback of a real object; as the patient's rehabilitation training progressed and the grip force gradually increased to 50N, the engine dynamically increased the weight of the virtual object to 0.8kg and adjusted the coefficient of friction to 0.6, increasing the training challenge; in addition, when the patient's hand posture deviated due to uneven force exertion, the engine would also adjust the collision response parameters in real time to prevent the virtual object from "slipping" due to unstable movements, ensuring the continuity of the training process. This dynamic adaptation process allows patients to build confidence in training within their capabilities, while gradually increasing the difficulty of training as muscle strength recovers. At the same time, it enhances immersion through realistic force feedback, effectively avoiding frustration or ineffective training caused by a mismatch between training difficulty and the patient's ability, and significantly improving the pertinence and effectiveness of rehabilitation training.

[0120] In step S103, based on the patient's hand simulation data and combined with the improved CNN-GRU model, the standardization of rehabilitation training movements and muscle activation efficiency are evaluated, movement compensation patterns and abnormal force characteristics are identified, and a rehabilitation assessment report is output. At the same time, the hand movement status and training progress are rendered in real time, and personalized training guidance is generated by combining voice, visual and tactile prompts.

[0121] Among them, the improved CNN-GRU model is a hybrid deep learning model that integrates the spatial feature extraction capability of convolutional neural networks with the temporal modeling advantages of gated recurrent units and introduces a temporal attention mechanism. It is used to extract the spatiotemporal features of hand simulation data in hand rehabilitation training and compare them with standard rehabilitation action templates to accurately evaluate the standardization of the actions.

[0122] It is understood that this application's embodiments are based on patient hand simulation data, combined with standard rehabilitation movement templates and electromyographic signal simulation data. Through an improved CNN-GRU model that integrates the spatial feature extraction capabilities of convolutional neural networks, the temporal modeling advantages of gated recurrent units, and the feature focusing capabilities of temporal attention mechanisms, it achieves precise extraction of spatiotemporal features of hand rehabilitation training movements and quantitative calculation of multi-dimensional evaluation indicators. This provides high-precision intelligent analysis support for assessing the standardization of rehabilitation training movements, analyzing muscle activation efficiency, and identifying compensatory movement patterns and abnormal force characteristics. By comparing the extracted spatiotemporal features with standard rehabilitation movement templates, the objectivity and accuracy of the assessment are ensured. Deep analysis of spatiotemporal features enhances the comprehensiveness and accuracy of rehabilitation assessment. Combined with real-time rendered hand movement states and training progress, and with voice, visual, and tactile cues, personalized training guidance is generated. This accurately identifies rehabilitation training problems such as non-standard movements, insufficient muscle activation, and compensatory movements, scientifically outputs rehabilitation assessment reports, optimizes personalized rehabilitation training programs, reduces the risk of compensatory injury, accelerates the patient's rehabilitation process, and improves the intelligence level and training effect of rehabilitation training.

[0123] For example, in the scenario of hand function rehabilitation training for stroke patients: when patients wear data gloves with integrated motion sensors and electromyography (EMG) acquisition modules to perform designated rehabilitation exercises such as wrist flexion and extension and finger grasping, the system collects hand simulation data in real time, including hand movement trajectory, force feedback, and EMG signals, and inputs it into an improved CNN-GRU model. The model first uses a convolutional neural network (CNN) to accurately extract the spatial features of hand movements, such as joint angle distribution and movement trajectory curvature, as well as other static and dynamic spatial information. Then, it uses a gated recurrent unit (GRU) to model the temporally continuous hand movement data, capturing the temporal correlation features of the movements. At the same time, through the introduction of a temporal attention mechanism, it automatically focuses on key movements such as "peak force" and "limit joint angle". The system first assigns feature weights to frames; then compares the extracted spatiotemporal fusion features with preset standard rehabilitation movement templates to calculate movement similarity scores. Simultaneously, it analyzes muscle activation efficiency using electromyography (EMG) signal features to accurately identify abnormal patterns such as "shoulder compensatory force" and "finger flexion and extension incoordination" that occur during patient training. Finally, it outputs a structured rehabilitation assessment report containing movement standardization scores, muscle activation efficiency data, and abnormal movement analysis. Simultaneously, it generates targeted voice correction prompts and visual guidance animations in real time to help patients standardize their movements. This not only significantly improves the accuracy and comprehensiveness of rehabilitation assessments but also enhances the model's computational efficiency by focusing on key movement features, providing reliable data support for optimizing personalized rehabilitation training programs and effectively shortening the patient's rehabilitation cycle.

[0124] In step S104, based on the rehabilitation assessment report and real-time training data, the rehabilitation trend and stage goals are predicted through reinforcement learning algorithms, and the training parameters and scenario difficulty are dynamically adjusted.

[0125] Among them, reinforcement learning algorithms are machine learning algorithms that enable agents to continuously interact with the environment, execute actions based on real-time state feedback, and accumulate reward signals in order to autonomously learn and optimize strategies and achieve specific goals.

[0126] It is understood that the embodiments of this application use reinforcement learning algorithms to predict rehabilitation trends and set phased goals, integrate rehabilitation assessment reports and real-time training data, and construct a personalized rehabilitation training dynamic control system. This system transforms abstract data on the patient's hand function recovery status and training effects into quantifiable rehabilitation trend predictions and phased rehabilitation goals. Simultaneously, it links real-time training feedback to form appropriate training parameters and scenario difficulty adjustment strategies. Through the dynamic reward mechanism and autonomous iterative optimization capability of reinforcement learning, it achieves precise matching between training control and the patient's recovery status. Efficient rehabilitation trend prediction allows therapists and patients to clearly grasp the rehabilitation path, reducing the subjectivity and blindness of training planning. The adjustment strategies generated by combining real-time training data can accurately adapt to the patient's personalized recovery pace, reducing the negative impact of ineffective and overtraining. The system dynamically optimizes training parameters and scenario difficulty according to the rehabilitation progress, adapting to the patient's real needs for hand function recovery and improving the pertinence and efficiency of rehabilitation training.

[0127] This application proposes a hand rehabilitation training method based on virtual reality and gesture interaction. A gesture perception module comprehensively collects hand movement trajectories, muscle electrical signals, grip pressure data, and joint range of motion, providing multi-dimensional and accurate basic data for rehabilitation training and assessment. A VR rehabilitation scene construction module, combining an adaptive physics engine and a neuro-motion mapping algorithm, accurately simulates hand interaction behavior and force feedback effects in a real environment, achieving dynamic mapping between hand movement data and the virtual scene, overcoming the limitations of traditional rehabilitation training methods with single scenes and delayed feedback. An intelligent assessment center, using an improved CNN-GRU model to integrate hand simulation data, improves the accuracy of rehabilitation training movement standard assessment and muscle activation efficiency analysis, enhancing the ability to identify movement compensation patterns and force abnormalities. An immersive VR training scene constructed by an interactive training unit intuitively presents hand movement status and training progress, facilitating the efficient generation of personalized training guidance. A personalized control unit predicts rehabilitation trends and stage goals based on rehabilitation assessment reports and real-time training data, dynamically adjusting training parameters and scene difficulty, effectively improving the targeting, personalization, and hand function recovery efficiency of rehabilitation training. This solves the problems of low interaction accuracy and inability to share data in existing technologies.

[0128] The following will illustrate a hand rehabilitation training method based on virtual reality and gesture interaction through a specific embodiment, such as... Figure 9 As shown, it includes:

[0129] A stroke patient with mid-stage hand dysfunction (right-sided hemiplegia, wrist range of motion 60°, fingers capable of simple grasping, grip strength approximately 35N) was selected as the training subject. A hand rehabilitation training system based on virtual reality and gesture interaction was deployed in the neurorehabilitation training room of a rehabilitation medicine center for an 8-week personalized rehabilitation training program. The core equipment of the system included: an optical motion capture system, a flexible electromyography (EMG) sensing kit, customized haptic feedback gloves, a VR headset, and an edge computing server. All equipment was connected to a 5G industrial gateway to build a low-latency data transmission network, ensuring real-time performance and smooth operation during the training process.

[0130] The data acquisition phase focuses on the comprehensive and accurate acquisition of multimodal hand data. Multiple types of sensing devices are deployed on the patient's right hand: Motion capture utilizes OptiTrackPrime13 optical cameras (8 units, sampling frequency 120Hz, positioning accuracy ≤0.1mm), with reflective markers attached to 7 key nodes: the fingertips (thumb to little finger), metacarpophalangeal joints, and wrist joint. This captures the hand's three-dimensional motion trajectory (coordinate range X: -50~50cm, Y: -30~30cm, Z: 0~80cm) and posture changes in real time, accurately reconstructing finger opening and closing angles (measurement range 0~90°), wrist flexion, extension, and rotation. The flexible electromyography (EMG) sensor group uses MyoWare EMG sensors (6 patch sensors, sampling frequency 1000Hz, signal-to-noise ratio ≥80dB), which are respectively attached to the patient's forearm flexors (radial and ulnar wrist flexors) and extensors (radial and ulnar wrist flexors). The extensor carpi ulnaris, extensor carpi ulnaris, flexor digitorum, and extensor digitorum muscles are used to collect surface electromyographic signals during muscle contraction. The signal bandwidth is set to 20~500Hz to filter environmental interference. A customized tactile feedback glove integrates an array of piezoresistive sensors (measurement range 0~500N, accuracy ±5%, resolution 256×256 pixels) at the palm and fingertips to collect grip force and pressure distribution data in real time during grip training. The glove has a built-in micro vibration motor (vibration frequency 50~200Hz, amplitude 0.1~0.5mm) for force feedback output. Joint angle acquisition uses an MPU9250 micro inertial measurement unit (IMU, integrating accelerometer, gyroscope, and magnetometer), which is fixed to each finger joint and wrist joint to monitor joint motion angle (measurement range 0~180°, accuracy ±1°) and rotation rate (0~300° / s) in real time. The sampling frequency is set to 200Hz to capture rapid movement changes. All sensor data is transmitted to an edge computing server (Intel Core i7-12700H processor, 32GB memory) via a 5G module, with transmission latency controlled within 30ms. InfluxDB time-series database is used to store high-frequency motion and electromyography data (retained for 30 days), and MySQL is used to store basic patient information and training plan data, thus building a complete data traceability and management system.

[0131] The raw data needs to undergo systematic processing to support the operation of subsequent modules. The processing flow covers three core stages: cleaning, feature extraction, and standardization. Data cleaning adopts differentiated strategies for different types of signals: EMG signals have power frequency interference (50Hz) and EMG noise, which are reduced by 3-layer decomposition using db4 wavelet basis (high-frequency coefficients are hard-thresholded, with the threshold set to 0.02), improving the signal-to-noise ratio to over 35dB; outliers in motion trajectory and joint angle data (such as coordinate jumps caused by occlusion) are identified using the 3σ criterion, and data exceeding the range of [μ-3σ,μ+3σ] are corrected by weighted averaging of the nearest 5 points (weights are allocated according to the reciprocal of the distance); occasional missing (<5%) of grip force pressure data is filled by linear interpolation, and data with more than 10 consecutive missing sampling points are marked as invalid and trigger sensor calibration prompts. Feature extraction takes into account both the spatiotemporal characteristics of the signal and the needs of rehabilitation assessment: Temporal features extract eight statistical quantities from electromyography (EMG) signals, including peak value, root mean square value, and integral EMG value, reflecting muscle activation intensity; and six parameters, including amplitude, velocity, and acceleration, are extracted from motion trajectory data to characterize the smoothness of movement. Frequency domain features convert EMG signals to the frequency domain through FFT transformation, extracting the energy proportion in the 100-300Hz frequency band to correlate with muscle fatigue state. Spatial features extract four indicators, including pressure center coordinates and pressure uniformity, from pressure distribution data to assess force symmetry. Standardization uses the Z-score formula to map all features to the [-1,1] interval to eliminate dimensional differences. Finally, a structured dataset containing timestamps, 26-dimensional feature vectors (8 temporal domain + 6 frequency domain + 12 spatial domain), and training phase labels is constructed, divided into a training set (40,000 records) and a test set (10,000 records) in an 8:2 ratio, stored in Parquet format (30% compression), and a real-time data pipeline is built using Apache Kafka to provide standardized input for model inference and scene interaction.

[0132] The VR rehabilitation scenario construction and interactive simulation focuses on the fine motor skills training needs of patients in the middle stage of rehabilitation. Using Unity3D 2022.3 as the development engine, it achieves personalized scenario construction and realistic interactive simulation through multi-unit collaboration. The scenario generation unit, based on the patient's rehabilitation stage (middle stage) and collected hand data, constructs a "virtual supermarket shelf retrieval and placement" scenario. The scenario includes three shelves displaying virtual goods (beverage bottles, snack boxes, canned goods) of different sizes (3-8cm in diameter) and weights (0.2-2kg). Movable obstacles (moving speed 0.1-0.3m / s) are placed between the shelves. The training objective is for the patient to complete the task of retrieving and placing designated goods while avoiding obstacles. The geometric modeling adopts the "CAD basics + motion capture correction" approach: import a standard human hand CAD model, combine it with point cloud data captured from the patient's hand motion (point density 80 points / mm²), and perform registration using GeomagicWrap software (ICP algorithm, registration error <0.03mm). Correct personalized parameters such as finger length and joint angles, and build a 1:1 scale virtual hand model in Unity3D, supporting LOD level of detail display (simplified polygon count to 50,000 at a distance, and 1.5 million polygon count for a high-precision model at close range). The physics engine adaptation unit uses the NVIDIA PhysX 5.1 physics engine, which dynamically optimizes the physical properties of virtual objects through an adaptive parameter adjustment algorithm: it receives real-time patient grip force data, and when the grip force is <30N, it automatically reduces the weight of the virtual product by 10% and increases the coefficient of friction to 0.7 to reduce the difficulty of grasping; when the grip force is ≥40N, it increases the weight of the product by 5% increments and adjusts the coefficient of friction to 0.4 to increase the training challenge; when the patient's hand comes into contact with the virtual product, the system controls the vibration intensity of the haptic feedback glove according to the contact pressure (vibration frequency of 150Hz when pressure is >30N, and vibration frequency of 80Hz when pressure is <15N), realizing cross-sensory linkage of "vision-touch". The neural action mapping unit employs an improved LSTM-attention mechanism model to construct a precise mapping between hand multimodal data and virtual actions: the input layer receives a 26-dimensional feature vector, which is processed by a 2-layer LSTM (128 hidden units, dropout=0.3) to extract action temporal features. An attention mechanism is introduced (weights are dynamically allocated through the softmax function) to enhance the feature weights of key action frames such as grasping and releasing. The output layer outputs the three-dimensional coordinates and pose parameters of the hand virtual model, achieving real-time synchronization between the patient's hand movements and the virtual model (synchronization delay ≤30ms, coordinate error ≤0.2mm), ensuring the smoothness and realism of the interaction.

[0133] The intelligent evaluation and interactive training units work together to achieve accurate evaluation of training results and generation of personalized guidance. The action standard evaluation adopts an improved CNN-GRU hybrid model: the input layer receives hand simulation data (action trajectory, pressure distribution), and extracts spatial features (such as finger opening and closing amplitude deviation and pressure center shift) through 3 layers of CNN (convolution kernel size 3×3, 5×5, 7×7, number of kernels 32, 64, 128 respectively, activation function ReLU). The 128-dimensional features are output through a global average pooling layer; 2 layers of GRU (64 hidden units, dropout=0.3) are connected to capture temporal correlations (such as the trend of action speed change and coherence), and finally the action similarity score (0~100 points) is output through a fully connected layer (activation function Sigmoid). Model training was performed on an NVIDIA RTX 3090 graphics card using the Adam optimizer (learning rate 0.001, β1=0.9, β2=0.999), with cross-entropy loss as the optimization objective. After 50 epochs of training, the training set loss decreased to 0.003, the test set accuracy reached 95%, and the single-action evaluation time was ≤0.3 seconds. Muscle activation efficiency analysis was based on electromyography (EMG) simulation data. Muscle activation rate was calculated using a threshold determination and a dual-dimensional proportion fusion algorithm: using the standard EMG activation threshold (root mean square 0.05mV) of mid-term patients as a benchmark, the proportion of time the target muscle group's EMG signal exceeded the threshold (weight 0.6) and the proportion of activation area coverage (weight 0.4) were statistically analyzed, and the weighted average was used to obtain the muscle activation rate (range 0~1); the cross-correlation coefficient (range -1~1) and phase difference (range 0~π) were used to analyze the multi-muscle group synergy efficiency, and a synergy efficiency <0.6 was judged as force imbalance. Abnormal pattern recognition is achieved through rule matching and feature comparison: when the flexor activation rate is <0.3 and the biceps activation rate is >0.6, and the hand movement trajectory deviates from the standard trajectory by >5mm, it is determined to be shoulder compensation; when the left-right deviation of the grip force pressure distribution is >20%, and the wrist joint angle fluctuation is >15°, it is determined to be force imbalance, with a recognition time ≤0.1 seconds. After the assessment is completed, a standardized rehabilitation assessment report is generated, including indicators such as movement similarity score, muscle activation rate, coordination efficiency, abnormality type, and frequency of occurrence.

[0134] The VR interactive training scene is built using the URP rendering pipeline in Unity3D to ensure an immersive experience and real-time performance: real-time global illumination is configured (baking precision 512 texels / m), with lightmap baking time of approximately 1.5 hours, maintaining shadow quality at 90fps; a PBR material system is used, with differentiated material parameters set for the virtual product surfaces (beverage bottle metallicity 0.3, roughness 0.2; snack box metallicity 0, roughness 0.6) to simulate the texture of real objects; a multi-level LOD model is implemented, switching to a simplified scene model (200,000 polygons) when the view distance is >3m to ensure smooth operation. Dynamic rendering focuses on key training information: the virtual hand model uses a semi-transparent effect, joints are highlighted in red, and the movement trajectory is updated in real time with blue dashed lines (retaining the most recent 100 trajectory points, approximately 0.8 seconds of history); when the movement deviates, the deviated part is indicated by a yellow flashing indicator (frequency 2Hz), and a comparison of the standard movement ghost image is simultaneously displayed in the scene; training progress is displayed in real time through a virtual progress bar, and a "task completed" voice prompt pops up after completing a single task. The multimodal prompting unit integrates voice, visual, and tactile feedback: voice prompts use TTS speech synthesis technology (98% accuracy in Chinese semantic recognition) to broadcast action specifications in real time (such as "please reduce the wrist rotation angle" and "distribute grip force evenly"); visual prompts guide the training path with arrows in the scene, and the target product is highlighted with a green halo; tactile feedback prompts action deviations through changes in the intensity and frequency of glove vibration (strong vibration when deviation > 10mm, and moderate vibration when deviation 3~10mm), with a response time ≤ 0.1 seconds. The training guidance generation unit matches targeted content from the training resource library based on the weaknesses identified in the rehabilitation assessment report: For patients with insufficient flexor activation (activation rate 0.4), a sub-task of "progressive grip strength increasing training" is generated, with 2 sets per day, 15 repetitions per set. If the activation rate increases by ≥10% after every 3 training sessions, the weight of the virtual item is increased by 0.1kg; For patients with mild shoulder compensation, a "restrained grip training" is generated, adding a virtual shoulder support to the scene to restrain shoulder movements and strengthen the force control of the target muscle group. After the guidance is generated, it is synchronized to the VR scene and the patient's training plan in real time.

[0135] Personalized control and a dual-end collaborative system enable dynamic optimization and full-scenario monitoring of the training process. The rehabilitation trend prediction employs a deep reinforcement learning (DQN) model. The input layer receives rehabilitation assessment report indicators and real-time training data (muscle activation rate, movement similarity score, coordination efficiency, and cumulative training time), constructing a 5-dimensional state vector. The output layer outputs stage-specific rehabilitation goals (e.g., movement similarity ≥ 85 points and muscle activation rate ≥ 0.6 after 2 weeks), with a 3-dimensional movement vector. The model is trained using historical training data from 10,000 patients (including the rehabilitation trajectories of 500 mid-term patients), employing an ε-greedy exploration strategy (ε linearly decays from 0.9 to 0.1), with mean squared error as the loss function. After training, the rehabilitation trend prediction accuracy reaches 92%, and the stage-specific goal rationality score is ≥ 4.5 / 5 points (therapist assessment). The training parameter adjustment unit dynamically optimizes based on prediction results and real-time data: when the patient's similarity score for three consecutive training sessions is ≥85 and there are no signs of fatigue (energy percentage of 200-300Hz in the electromyography signal is <15%), the training duration for a single session is extended from 25 minutes to 30 minutes; when grip strength steadily increases to 40N, the target grip strength threshold is increased in a 5% gradient; when the similarity score for a movement is <75, the number of repetitions of that movement is increased (from 15 to 20 per set). Scene difficulty adaptation follows a gradual approach: after each training round, the patient's mastery is verified. When the similarity score for a movement is ≥85 and the frequency of abnormal occurrences is ≤1 per training cycle, the scene difficulty is increased, such as increasing the number of obstacles (from 2 to 4), increasing the obstacle movement speed (from 0.1m / s to 0.2m / s), and adding a task of stacking and retrieving goods; after increasing the difficulty, the physics engine parameters are adjusted synchronously to ensure that the training difficulty matches the patient's ability.

[0136] The dual-terminal collaborative system constructs a "local training + remote monitoring" model: Local training is deployed in the rehabilitation training room, where patients wear ValveIndex VR headsets (1600×1440 resolution / eye, 144Hz refresh rate) and achieve immersive interaction through haptic feedback gloves. The training room is equipped with a 27-inch 4K monitor, which displays the patient's training scene, real-time curves of assessment indicators, and abnormal warning windows in a split-screen format. Therapists can use the mouse to click and mark the movements that need correction, and the marked information is synchronized to the patient's VR field of view in real time (indicated by red arrows). Remote monitoring adopts a lightweight WebGL solution, converting the Unity3D scene to glTF format (compressing the file size to 1 / 5 of the original model), and rendering it in a browser through the Three.js engine, supporting access on PC (Chrome / Firefox) and mobile devices (WeChat mini-program). Administrators can view the patient's real-time training data, rehabilitation trend curves, and training plan completion status, and provide feedback to therapists through the voice message function. Therapists can remotely adjust training parameters (such as temporarily reducing the difficulty of the scene), and adjustment commands are transmitted through the 5G network with an execution latency of <50ms. Adaptive regulation and emergency response ensure training safety and effectiveness: The rehabilitation progress tracking module generates a rehabilitation curve based on daily training data, compares the differences between stage goals, and automatically optimizes the training plan when the actual progress lags behind the target by more than 10%, such as increasing the frequency of targeted sub-task training and adjusting the parameter improvement range. During training, the patient's fatigue status is monitored in real time. When electromyographic signal noise increases or movement speed decreases by more than 20%, a fatigue warning is triggered, a rest prompt pops up, and training is paused for 5 minutes. An emergency response mechanism addresses sudden situations (such as dizziness or severe movement distortion): Level 1 response (0~200ms): A red warning interface is displayed in the VR scene, and a voice announcement says "Please stop training"; Level 2 response (200~500ms): The haptic feedback glove emits continuous vibration prompts and automatically saves the current training data; Level 3 response (500ms~1s): The therapist receives an emergency notification, can remotely pause training, check the patient's status, and terminate the current training session and adjust the plan if necessary.

[0137] After 8 weeks of systematic training, the patient's right hand function improved significantly: wrist joint range of motion increased from 60° to 105°, grip strength increased from 35N to 62N, movement similarity score increased from 68 points to 92 points, muscle activation rate increased from 0.4 to 0.75, abnormal movements such as shoulder compensation were completely eliminated, and the patient can independently complete daily activities such as picking up and putting away tableware and folding clothes. The rehabilitation effect is better than the traditional manual training mode.

[0138] In summary, this application's embodiments, through multimodal hand data acquisition and standardized processing, combined with personalized VR rehabilitation scenario construction and realistic interactive simulation, improved CNN-GRU model for accurate evaluation, and deep reinforcement learning for trend prediction, achieve objective quantitative assessment of rehabilitation training movement standardization and muscle activation efficiency, as well as rapid identification of compensatory abnormalities, providing scientific support for rehabilitation decision-making. Personalized control and dual-end collaborative monitoring mechanisms dynamically optimize training parameters and scenario difficulty, accurately adapting to the patient's recovery pace, reducing ineffective and overtraining, and ensuring training safety and continuity. Multimodal prompts and targeted training guidance effectively improve patient training compliance and movement standardization. Adaptive emergency response and progress tracking ensure the training process is controllable and adjustable, significantly improving the patient's core hand function, completely eliminating abnormal movements, and enabling independent completion of daily living activities. The rehabilitation effect is superior to traditional manual training modes, comprehensively improving the accuracy, personalization level, and rehabilitation quality of hand rehabilitation training, shortening the rehabilitation cycle, and helping patients quickly return to daily life.

[0139] Figure 10 A schematic diagram of the structure of an electronic device provided in an embodiment of this application. The electronic device may include:

[0140] The memory 1001, the processor 1002, and the computer program stored on the memory 1001 and capable of running on the processor 1002.

[0141] When the processor 1002 executes the program, it implements a hand rehabilitation training method based on virtual reality and gesture interaction provided in the above embodiments.

[0142] Furthermore, electronic devices also include:

[0143] Communication interface 1003 is used for communication between memory 1001 and processor 1002.

[0144] The memory 1001 is used to store computer programs that can run on the processor 1002.

[0145] The memory 1001 may include high-speed RAM (Random Access Memory) memory, and may also include non-volatile memory, such as at least one disk storage.

[0146] If the memory 1001, processor 1002, and communication interface 1003 are implemented independently, then the communication interface 1003, memory 1001, and processor 1002 can be interconnected via a bus to complete communication between them. The bus can be an ISA (Industry Standard Architecture) bus, a PCI (Peripheral Component Interconnect) bus, or an EISA (Extended Industry Standard Architecture) bus, etc. The bus can be divided into address bus, data bus, control bus, etc. For ease of representation, Figure 10 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.

[0147] Optionally, in a specific implementation, if the memory 1001, processor 1002, and communication interface 1003 are integrated on a single chip, then the memory 1001, processor 1002, and communication interface 1003 can communicate with each other through an internal interface.

[0148] The processor 1002 may be a CPU (Central Processing Unit), an ASIC (Application Specific Integrated Circuit), or one or more integrated circuits configured to implement the embodiments of this application.

[0149] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described hand rehabilitation training method based on virtual reality and gesture interaction.

[0150] Furthermore, this application also provides a computer program product, including a computer program or instructions, which, when executed, implement the aforementioned hand rehabilitation training method based on virtual reality and gesture interaction.

[0151] In the description of this specification, the references to "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of this application. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0152] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this application, "multiple" means at least two, such as two, three, etc., unless otherwise explicitly specified.

[0153] Any process or method description in the flowchart or otherwise herein can be understood as representing a module, segment, or portion of code comprising one or more executable instructions for implementing custom logic functions or processes, and the scope of the preferred embodiments of this application includes additional implementations in which functions may be performed not in the order shown or discussed, including substantially simultaneously or in reverse order depending on the functions involved, as should be understood by those skilled in the art to which embodiments of this application pertain.

[0154] It should be understood that various parts of this application can be implemented using hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented using software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware as in another embodiment, it can be implemented using any of the following techniques known in the art, or a combination thereof: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0155] Those skilled in the art will understand that all or part of the steps of the methods described in the above embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, the program includes one or a combination of the steps of the method embodiments.

[0156] Although embodiments of this application have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting this application. Those skilled in the art can make changes, modifications, substitutions and variations to the above embodiments within the scope of this application.

Claims

1. A hand rehabilitation training system based on virtual reality and gesture interaction, characterized in that, include: The system comprises a gesture perception module, a VR rehabilitation scene construction module, an intelligent assessment center, an interactive training unit, and a personalized control unit; among which, The gesture sensing module is used to collect hand movement trajectory, muscle electrical signals, grip pressure data, and joint range of motion. The VR rehabilitation scene construction module is used to construct a personalized rehabilitation training virtual scene based on the patient's rehabilitation stage and the hand movement trajectory, muscle electrical signals, grip force pressure data and joint movement angle. Through an adaptive physics engine and a neuro-motor mapping algorithm, it simulates hand interaction behavior and force feedback effect in a real environment and outputs patient hand simulation data. The intelligent assessment center is used to assess the standardization of rehabilitation training movements and muscle activation efficiency based on the patient's hand simulation data and combined with the improved CNN-GRU model, identify movement compensation patterns and abnormal force characteristics, and output a rehabilitation assessment report. The interactive training unit is used to render the hand movement status and training process in real time based on the rehabilitation assessment report, and generate personalized training guidance by combining voice, visual and tactile prompts. The personalized control unit is used to predict rehabilitation trends and stage goals based on the rehabilitation assessment report and real-time training data, and dynamically adjust training parameters and scenario difficulty through reinforcement learning algorithms.

2. The hand rehabilitation training system based on virtual reality and gesture interaction according to claim 1, characterized in that, The gesture sensing module includes a motion capture unit, an electromyography (EMG) sensor group, a grip force and pressure sensor, and a joint angle acquisition unit. The motion capture unit is used to collect the three-dimensional movement trajectory of the hand in real time. The EMG sensor group is used to conform to the forearm muscle group and collect surface EMG signals during muscle contraction. The grip force and pressure sensor is integrated into the handle of the training device to collect data on the magnitude and distribution of hand grip force. The joint angle acquisition unit is used to monitor the range of motion and rotation rate of the fingers and wrist joints in real time.

3. The hand rehabilitation training system based on virtual reality and gesture interaction according to claim 1, characterized in that, The VR rehabilitation scene construction module includes a scene generation unit, a physics engine adaptation unit, and a neuro-motion mapping unit. The scene generation unit constructs a personalized rehabilitation training virtual scene based on the patient's rehabilitation stage, hand movement trajectory, muscle electrical signals, grip force pressure data, and joint movement angles. The physics engine adaptation unit dynamically adjusts the physical properties of objects in the scene through an adaptive physics engine to simulate real contact force feedback. The neuro-motion mapping unit establishes a precise mapping relationship between hand movement data and virtual scene interaction actions through a neuro-motion mapping algorithm, outputting hand simulation data including movement trajectory and force feedback.

4. The hand rehabilitation training system based on virtual reality and gesture interaction according to claim 1, characterized in that, The intelligent assessment center includes a movement standardization assessment unit, a muscle activation analysis unit, and an abnormal pattern recognition unit. The movement standardization assessment unit uses an improved CNN-GRU model to extract movement features from hand simulation data, compares them with standard rehabilitation movement templates, calculates movement similarity scores, and assesses movement standardization. The muscle activation efficiency analysis unit calculates muscle activation rate and synergistic work efficiency based on electromyographic signal simulation data. The abnormal pattern recognition unit identifies compensatory movements and force imbalance abnormalities, generating a rehabilitation assessment report that includes assessment indicators and abnormal analysis.

5. A hand rehabilitation training system based on virtual reality and gesture interaction according to claim 1, characterized in that, The interactive training unit includes a VR rendering engine, a multimodal prompting unit, and a training guidance generation unit. The VR rendering engine is used to render the movement state of the virtual hand model, the progress of the training task, and the details of the scene environment in real time. The multimodal prompting module is used to provide voice command prompts, visual highlighting guidance, and tactile vibration feedback. The training guidance generation unit is used to generate targeted training guidance based on the weaknesses in the rehabilitation assessment report.

6. A hand rehabilitation training system based on virtual reality and gesture interaction according to claim 1, characterized in that, The personalized control unit includes a rehabilitation trend prediction unit, a training parameter adjustment unit, and a scenario difficulty adaptation unit. The rehabilitation trend prediction unit is used to predict the recovery trend and phased goals of hand function by using a reinforcement learning algorithm combined with historical training data and evaluation reports. The training parameter adjustment unit is used to dynamically adjust the training duration, number of repetitions of the movement, and grip strength target threshold parameters. The scenario difficulty adaptation unit is used to gradually increase the complexity of the scenario according to the rehabilitation progress.

7. A method for applying a hand rehabilitation training system based on virtual reality and gesture interaction as described in any one of claims 1-6, characterized in that, The method includes: Acquire hand movement trajectory, muscle electrical signals, grip force and pressure data, and joint range of motion; Based on the patient's rehabilitation stage and the hand movement trajectory, electromyographic signals, grip force and pressure data, and joint movement angle, a personalized rehabilitation training virtual scenario is constructed. Through an adaptive physics engine and a neuro-motor mapping algorithm, the hand interaction behavior and force feedback effect in a real environment are simulated, and the patient's hand simulation data is output. Based on the patient's hand simulation data, combined with the improved CNN-GRU model, the standardization of rehabilitation training movements and muscle activation efficiency are evaluated, movement compensation patterns and abnormal force characteristics are identified, and a rehabilitation assessment report is output. At the same time, the hand movement status and training progress are rendered in real time, and personalized training guidance is generated by combining voice, visual and tactile prompts. Based on the rehabilitation assessment report and real-time training data, the rehabilitation trend and stage goals are predicted through reinforcement learning algorithms, and the training parameters and scenario difficulty are dynamically adjusted.

8. An electronic device, characterized in that, It includes a memory, a processor, and a computer program stored in the memory and executable on the processor. The processor executes the program to implement the hand rehabilitation training method based on virtual reality and gesture interaction as claimed in claim 7.

9. A computer-readable storage medium having a computer program or instructions stored thereon, characterized in that, When a computer program or instruction is executed, it implements the hand rehabilitation training method based on virtual reality and gesture interaction as claimed in claim 7.

10. A computer program product, comprising a computer program or instructions, characterized in that, When a computer program or instruction is executed, it implements the hand rehabilitation training method based on virtual reality and gesture interaction as claimed in claim 7.

Images