A hand rehabilitation robot training method and system based on multi-modal fusion

Abstract

The application discloses a kind of based on multimodal fusion hand rehabilitation robot training method and system, it is related to the technical field of rehabilitation robot.The method comprises: the finger bending angle is collected by angle sensor built-in in glove, angular velocity signal is obtained by differential calculation and power spectral density feature is extracted, and output kinematics fatigue degree;Surface electromyogram signal is collected by forearm electromyogram patch, and physiological muscle fatigue degree is output by time-frequency domain processing and time series prediction model;Face image sequence is collected by front camera, and emotional state score is output by expression recognition model;Three indexes are normalized and rehabilitation exercise participation degree is calculated according to weighted fusion formula F , F Angle tracking error is input into iterative learning model predictive controller (ILMPC) to realize multi-objective optimization control, and picture style and sound effect type of virtual reality training scene are switched according to emotional state.The application can find muscle fatigue trend in advance by kinematics, physiology and psychology three modal decision level fusion, and real-time understand patient training state, which is helpful to maintain training enthusiasm while preventing overtraining.

Description

Technical Field

[0001] This invention relates to the field of rehabilitation robot technology, and in particular to a training method and system for a hand rehabilitation robot based on multimodal fusion. Background Technology

[0002] Hand rehabilitation robots assist patients in performing repetitive training movements such as grasping and extending, promoting the recovery of hand motor function, and are gradually being applied in clinical rehabilitation after stroke and hand injuries. With the increasing demands for precision and personalization in clinical rehabilitation training, the need for real-time monitoring and comprehensive assessment of patients' multidimensional states during training is becoming increasingly urgent.

[0003] Currently, status monitoring solutions for hand rehabilitation robots are mainly divided into two categories: monomodal and multimodal. Monomodal solutions collect only single-dimensional data. For example, relying solely on motion sensors to monitor finger movement trajectories cannot reveal the physiological fatigue level of the forearm muscles; relying solely on electromyography (EMG) sensors to collect surface EMG signals makes it difficult to capture the kinematic fatigue characteristics at the finger movement level, resulting in a one-sided assessment perspective. Existing multimodal solutions mainly focus on pairwise combinations of kinematics and EMG, or kinematics and force feedback: while the former can take into account both muscle physiological state and movement information, and the latter can introduce force feedback, neither incorporates the emotional dimension and cannot perceive changes in the patient's psychological state. When negative emotions such as irritability and frustration occur, the system cannot detect them, making it difficult to ensure training compliance. Although some solutions integrate multiple sensors, they are essentially still a superposition of kinematic or mechanical sensors, failing to include the psychological dimension in the monitoring scope. Therefore, in the context of hand rehabilitation, simultaneous coverage of the three core dimensions of kinematics, physiology, and psychology remains a gap.

[0004] Furthermore, in terms of assistive force regulation, existing solutions mostly employ fixed thresholds or proportional controls, lacking a closed-loop incentive mechanism that dynamically adjusts task difficulty based on the patient's real-time participation. Studies have shown that optimizing control indicators with rehabilitation exercise participation as a multi-objective goal and achieving ILMPC closed-loop control in conjunction with trajectory accuracy can significantly improve muscle activation levels and subjective participation, but this mechanism has not yet been realized in integrated multimodal systems. Due to the lack of simultaneous perception across the aforementioned kinematic, physiological, and psychological dimensions, the system not only fails to identify early muscle fatigue and negative emotions but also struggles to detect compensatory exertion through multimodal cross-validation, thus making it difficult to simultaneously prevent overtraining and maintain training motivation. Summary of the Invention

[0005] This invention addresses the shortcomings of existing hand rehabilitation robot state monitoring solutions by providing a training method and system based on multimodal fusion, mainly comprising:

[0006] A training method for a hand rehabilitation robot based on multimodal fusion includes: S1. The angle sensor built into the glove collects the finger bending angle in real time, obtains the angular velocity signal through differential calculation, extracts the angular velocity power spectral density characteristics, and outputs the kinematic fatigue index. S2. Real-time acquisition of surface electromyography signals through forearm electromyography patches, time-frequency domain processing and feature extraction, prediction of fatigue trends in future periods using time-series prediction models, and output of physiological muscle fatigue index. S3. Real-time acquisition of patient facial image sequences via front-facing camera, extraction of facial dynamic features and identification of emotion categories using expression recognition model, and output of emotion state score. S4. After normalizing the scores of kinematic fatigue, physiological muscle fatigue, and emotional state, calculate the rehabilitation exercise participation using a weighted fusion formula. F ;Will F Together with the finger bending angle tracking error, it serves as a multi-objective optimization control index, constructing a system that includes... F The MPC cost function and ILC iterative learning law for tracking error terms enable the controller to optimize engagement levels while optimizing finger movement trajectories; at the same time, when the emotional state score falls into the negative emotional range, the visual style and sound effect type of the virtual reality training scene are switched.

[0007] Further, in step S4, the weighted fusion formula is: In the formula F K , F P , E M These are the normalized scores for kinematic fatigue, physiological muscle fatigue, and emotional state. F K , F P ∈[0,1] , E M ∈[−1,1] ; α K , α P , α E For the weighting coefficients, satisfying α K +α P +α E =1 and α P >α K >α E。 The switching of virtual reality training scenes includes at least three preset scenario modes: when the scene is identified as pleasant or focused, bright colors and upbeat sound effects are presented; when the scene is identified as calm or neutral, the current scene is maintained; when the scene is identified as anxious or tense, soft colors and soothing music are switched; when the scene is identified as frustrated or tired, warm colors are switched to simplify the scene and meditation guidance audio are switched.

[0008] The present invention also provides a hand rehabilitation robot training system based on multimodal fusion, comprising: The kinematic modality module is used to collect the finger bending angle through the angle sensor built into the glove, obtain the angular velocity signal through differentiation, extract the power spectral density characteristics, and output the kinematic fatigue level. The electromyography modal module is used to collect surface electromyography signals through forearm electromyography patches, and output physiological muscle fatigue after time-frequency domain processing and time-series prediction model. The emotion modality module is used to capture facial image sequences through the front-facing camera and output an emotion state score through an expression recognition model. The fusion decision module is used to normalize the output indicators of the three modalities and then calculate the rehabilitation exercise participation rate according to the weighted fusion formula. F As one of the multi-objective optimization control indicators, it is input together with the angle tracking error into the iterative learning model predictive controller (ILMPC) to optimize the participation of rehabilitation movement while optimizing the finger movement trajectory, and to apply emotional adaptive correction (scene switching) when there is emotional abnormality. The virtual reality adaptive module is used to switch the visual style and sound effects of the virtual scene according to the emotion category.

[0009] A computer-readable storage medium storing a plurality of instructions adapted for loading and execution of the method by a terminal device processor.

[0010] A terminal device includes a processor and a computer-readable storage medium, the processor being configured to implement instructions, and the computer-readable storage medium being configured to store a plurality of instructions adapted to be loaded by the processor and executed by the method.

[0011] This invention addresses three key shortcomings of existing solutions in condition monitoring through trimodal decision-level fusion, including: (1) The temporal prediction ability of electromyography can capture the trend of physiological fatigue in advance before the finger movements show obvious changes, thus avoiding muscle damage caused by improper training load; (2) The emotional modality senses the patient's psychological state in real time. When negative emotions are detected, the color tone, sound effects and guidance content of the virtual scene are automatically switched to help maintain training enthusiasm and improve compliance. (3) Multimodal cross-validation can identify compensatory force exertion: When the kinematic parameters are normal but the electromyographic activation mode is abnormal, the system can prompt the patient to adjust the force exertion mode and reduce ineffective training. Attached Figure Description

[0012] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used in conjunction with embodiments of the invention to explain the invention and do not constitute a limitation thereof. In the drawings: Figure 1 is a flowchart of the overall system architecture proposed in this invention; Figure 2 is a flowchart of the multimodal fusion ILMPC control closed loop proposed in this invention; Figure 3 is a schematic diagram of the virtual reality scene adaptation proposed in this invention. Detailed Implementation

[0013] The present invention will now be described in further detail with reference to the accompanying drawings. These drawings are simplified schematic diagrams, illustrating only the basic structure of the invention, and therefore only show the components relevant to the invention.

[0014] Referring to Figures 1 to 3, a training method for a hand rehabilitation robot based on multimodal fusion includes the following steps: S1. Simultaneously collect finger bending angle, forearm surface electromyography signals, and patient facial image sequences through the built-in angle sensor, forearm electromyography patch, and front-facing camera in the glove. S2. The finger angular velocity signal is obtained by differential calculation of the finger bending angle, and the kinematic fatigue degree is output by power spectral density analysis. S3. Perform time-frequency domain processing on the electromyography signals on the forearm surface and extract features. Use a time-series prediction model to predict the fatigue trend in future periods and output the physiological muscle fatigue level. S4. Preprocess the patient's facial image sequence, extract facial dynamic features using an expression recognition model, identify emotion categories, and output an emotion state score. S5. After normalizing the scores of kinematic fatigue, physiological muscle fatigue, and emotional state, calculate the rehabilitation exercise participation using a weighted fusion formula. F ,Will F The angle tracking error is input into ILMPC for multi-objective optimization control, outputting air pressure control commands and switching virtual reality training scenes according to emotional state; S6. Execute ILMPC air pressure control commands and virtual reality scene switching commands, and continuously collect three signals to form a closed loop.

[0015] In this embodiment, step S1 specifically includes: S11. The patient wears flexible rehabilitation gloves, with Leap Motion angle sensors embedded in the inner walls of each finger to collect real-time sequences of bending angles at each finger joint at a sampling rate of not less than 40 Hz; electromyography patches are attached to the muscle bellies of the flexor and extensor muscles of the forearm to collect surface electromyography signals at a sampling rate of not less than 1000 Hz; a front-facing camera is set in front of the training table to capture facial image sequences at a rate of not less than 15 frames / second. S12. Add a synchronization timestamp to the three signals and temporarily cache them in the local storage module.

[0016] In this embodiment, step S2 specifically includes: S21. For the finger bending angle sequence obtained in step S1, calculate the finger angular velocity signal by dividing the angle difference between consecutive frames by the time interval. S22. Perform power spectral density analysis on the angular velocity signal, and use continuous wavelet transform to extract the power spectral amplitude in the motion frequency band corresponding to the rhythm of hand grasping / extending movements. S23. Compare the real-time power spectrum amplitude with the patient's individual baseline value. When the power spectrum amplitude increases significantly within the motion frequency band, it is determined that kinematic fatigue has occurred. The kinematic fatigue level is then output after normalization. F K ∈[0,1] The higher the value, the greater the fatigue level at the finger movement level.

[0017] In this embodiment, step S3 specifically includes: S31. Bandpass filtering is performed on the electromyography signal on the forearm surface to remove motion artifacts and low-frequency drift. A 50 Hz notch filter is used to remove power frequency interference, and then normalization is performed. S32. Segment the signal using a 500-point sliding window, extract the root mean square value and integral electromyography value in the time domain, and extract the median frequency and average power frequency in the frequency domain. S33. Input the time-frequency domain features into the time-series prediction model, which is composed of a convolutional neural network, an attention mechanism and a long short-term memory network connected in series: the convolutional layer automatically extracts features, the attention mechanism weights important time segments, and the long short-term memory network captures time-series dependencies. S34. The model outputs the fatigue trend prediction results for the next 5 seconds, which are then normalized to obtain the physiological muscle fatigue level. F P ∈[0,1] The higher the value, the higher the degree of muscle physiological fatigue.

[0018] In this embodiment, step S4 specifically includes: S41. The facial image sequence acquired in step S1 is scaled to a uniform size using bilinear interpolation, and the CLAHE method is used to enhance brightness and local contrast. After normalizing the pixel values, they are stacked in chronological order to form a four-dimensional image tensor. S42. Input the image tensor into the improved BiFormer network, which includes a face region guidance module, a two-level routing attention module, a lightweight temporal modeling module, and a two-branch output module; S43. The facial region guidance module extracts key feature points such as the inner and outer corners of the eyebrows, the outer corners of the eyebrows, the corners of the eyes, and the corners of the mouth, and generates a mask matrix with higher weights for areas with significant facial expressions such as the eyebrows, eyes, nose, and lips. S44. The dual-level routing attention module divides the image into tokens and clusters them according to facial regions before performing self-attention calculation to capture local expression features. S45. The lightweight temporal modeling module models the emotional changes between consecutive frames through one-dimensional convolution. S46. The dual-branch output module outputs the main emotion category through the Softmax function, outputs the emotion dimension score through linear regression, and outputs the confidence parameter through the Sigmoid function, thus obtaining a comprehensive emotion state score. E M ,in E M >0 Indicates positive emotions. E M <0 Indicates negative emotions. |E M | The larger the value, the higher the intensity of the emotion.

[0019] In this embodiment, step S5 specifically includes: S51. Define reference values ​​for rehabilitation exercise participation. F ref This value represents the engagement level of healthy individuals when performing the same training task, obtained through pre-testing or calculation; a reference trajectory at a given angle is also provided. i ref (t)。 S52. Calculate rehabilitation exercise participation in real time using a weighted fusion formula in the multimodal fusion module. F ,Will F With angle i Simultaneously serving as the optimization objective of the controller. Based on the ILMPC framework of claim 1 (whose predictive model employs linearization modeling using the Koopman operator, and whose control law integrates MPC and ILC), the cost function of model predictive control (MPC) is: S53. Introduce an participation error term into the input update law of Iterative Learning Control (ILC): In the formula L θ and L F These are the iterative learning gains for angle error and participation error, respectively, with subscripts... j This indicates the iteration (training repetition cycle) number; the update law is consistent with claim 1 and corresponds to the input iterative learning law in the Koopman-ILMPC controller. S54. In each iteration (training repetition cycle) or at each sampling time, the system updates the... F And the angle was re-optimized to control the input, so that i and F At the same time, they approach their respective reference values. S55. Referring to Figure 2, form a closed loop: Multimodal sensor acquires data → Calculation F → F Feedback to MPC / ILC → Output pneumatic control command → Change patient's finger movement and participation status → Collect data again → Recalculate F The system continues to run until the set number of training iterations is completed. S56. When the emotional state score shows a specific emotion, the virtual scene switching in step S6 is triggered to maintain the training challenge and fun to a certain extent.

[0020] In this embodiment, step S6 specifically includes: S61. Referring to Figure 3, the virtual training scenario has four preset scenario modes, which automatically switch according to the emotion category identified in step S4: (1) When the mood is "pleasure" or "focus", the scene is bright and warm in color, such as a sunny grass scene, and light music is played; (2) When the mood is “calm” or “neutral”, maintain the standard training scene, neutral gray and white tone, and soft ambient sound; (3) When the mood is "anxious" or "tense", switch to a low saturation soft color tone, such as a starry sky or twilight scene, and play soothing music; (4) When the mood is "frustrated" or "fatigued", switch to warm soft light to simplify the scene, play meditation guidance audio, and display prompts such as "Please follow the rhythm to breathe" on the interface; S62. The above scene switching is achieved through real-time rendering using the Unity engine, and the color saturation and the number of scene elements are adjusted proportionally according to the emotional state score. S63. The air pressure control command and the virtual scene switching command are simultaneously sent to the hand rehabilitation robot glove for execution. The system continuously collects three signals and repeats steps S2 to S6 to form a real-time closed loop.

Claims

1. A training method for a hand rehabilitation robot based on multimodal fusion, characterized in that, Includes the following steps: S1. The angle sensor built into the glove collects the finger bending angle in real time, obtains the angular velocity signal through differential calculation, extracts the angular velocity power spectral density characteristics, and outputs the kinematic fatigue index. S2. Real-time acquisition of surface electromyography signals through forearm electromyography patches, time-frequency domain processing and feature extraction, prediction of fatigue trends in future periods using time-series prediction models, and output of physiological muscle fatigue index. S3. Real-time acquisition of patient facial image sequences via front-facing camera, extraction of facial dynamic features and identification of emotion categories using expression recognition model, and output of emotion state score. S4. After normalizing the kinematic fatigue, physiological muscle fatigue, and emotional state scores output by S1, S2, and S3, calculate the rehabilitation exercise participation rate using a weighted fusion formula. F Then F Together with the finger bending angle tracking error, these are used as multi-objective optimization control indices to construct the model predictive control (MPC) cost function: And the Iterative Learning Control (ILC) input update law: in F ref This serves as a reference value for participation when healthy individuals complete the same task. θ ref As an angle reference trajectory, e θ j =θ ref − θ j For angle tracking error, e F j =F ref −F j For participation tracking error, Q θ , w F , R These are respectively angle tracking, participation tracking, and control quantity weights. L θ , L F This represents the corresponding iterative learning gain. N To predict the time domain, the controller optimizes the rehabilitation exercise participation while optimizing the finger bending angle trajectory, forming a closed loop of "multimodal perception → participation calculation → multi-objective optimization control → pneumatic execution → state update".

2. The method according to claim 1, characterized in that: In step S4, the degree of participation in rehabilitation exercises is calculated. F The weighted fusion formula used is: In the formula F K , F P , E M These are the normalized scores for kinematic fatigue, physiological muscle fatigue, and emotional state, respectively. F K , F P ∈[0,1] , E M ∈[−1,1] ( E M >0 Indicates positive emotions, E M <0 (Indicates negative emotions). α K , α P , α E For the weighting coefficients, satisfying Preferably, α P ∈[0.40,0.50] , α K ∈[0.30,0.40] , α E ∈[0.15,0.25]。 3. The method according to claim 1, characterized in that: Step S1 specifically includes: S11. For the acquired finger bending angle sequence, calculate the finger angular velocity signal by dividing the angle difference between consecutive frames by the time interval; S12. Perform power spectral density analysis on the angular velocity signal to extract the power spectral amplitude in the motion frequency band corresponding to the hand grasping / extending action rhythm; S13. Compare the real-time power spectrum amplitude with the patient's personal baseline value. When the power spectrum amplitude increases significantly in the motion frequency band, it is determined that fatigue has occurred. After normalization, the kinematic fatigue degree is output.

4. The method according to claim 1, characterized in that: Step S2 specifically includes: S21. Perform 20~450 Hz bandpass filtering and 50 Hz notch filtering on the electromyography signal on the forearm surface, and then perform Z-score normalization. S22. Segment the signal using a sliding window, extract the root mean square value and integral electromyography value in the time domain, and extract the median frequency and average power frequency in the frequency domain. S23. Input the time-frequency domain features into the time-series prediction model, which is composed of a convolutional neural network, an attention mechanism, and a long short-term memory network connected in series. The model outputs the fatigue trend prediction result within the next 5 seconds, and the result is normalized to obtain the physiological muscle fatigue level. F P ∈[0,1]。 5. The method according to claim 1, characterized in that: Step S3 specifically includes: S31. Perform size normalization, brightness equalization and pixel value normalization on the acquired facial image sequence, and stack them in time order to form a four-dimensional image tensor. S32. Input the image tensor into the facial expression recognition model. The facial expression recognition model includes a facial region guidance module, a two-level routing attention module, a lightweight temporal modeling module, and a two-branch output module. It outputs the main emotion category, emotion dimension score, and confidence parameter, and obtains the comprehensive emotion state score. E M ,in E M >0 Indicates positive emotions. E M <0 It indicates negative emotions.

6. The method according to claim 1, characterized in that: The switching of the visual style and sound effects of the virtual reality training scene specifically includes: the virtual reality training scene has at least three preset scene modes. When it is recognized as pleasant or focused, it presents bright colors and light sound effects; when it is recognized as calm or neutral, it maintains the current scene; when it is recognized as anxious or tense, it switches to soft colors and soothing music; when it is recognized as frustrated or tired, it switches to warm colors to simplify the scene and meditation guidance audio.

7. A hand rehabilitation robot training system based on multimodal fusion, characterized in that, include: The kinematic modality module is used to collect the finger bending angle through the angle sensor built into the glove, obtain the angular velocity signal through differentiation, extract the power spectral density characteristics, and output the kinematic fatigue level. The electromyography modal module is used to collect surface electromyography signals through forearm electromyography patches, and output physiological muscle fatigue after time-frequency domain processing and time-series prediction model. The emotion modality module is used to capture facial image sequences through the front-facing camera and output an emotion state score through an expression recognition model. The fusion decision module is used to normalize the output indicators of the three modalities and then calculate the rehabilitation exercise participation rate according to the weighted fusion formula. F As one of the multi-objective optimization control indicators, it is input together with the angle tracking error into the iterative learning model predictive controller (ILMPC). The controller introduces an engagement tracking term into the MPC cost function. w F (F ref −F) 2 Introduce a participation error term into the ILC update law. L F e F j This allows for the optimization of finger movement trajectories while simultaneously increasing participation in rehabilitation exercises; additionally, it applies adaptive emotion correction (scene switching) when emotional abnormalities occur. The virtual reality adaptive module is used to switch the visual style and sound effect type of the virtual reality training scene according to the emotion category.

8. A computer-readable storage medium storing a plurality of instructions, characterized in that, The instructions are adapted to be loaded by a terminal device processor and executed as described in any one of claims 1 to 6.

9. A terminal device, comprising a processor and a computer-readable storage medium, characterized in that, The processor is used to implement the instructions, and the computer-readable storage medium is used to store a plurality of instructions adapted to be loaded by the processor and executed as described in any one of claims 1 to 6.

Images