A hand rehabilitation evaluation system and method based on multi-modal data fusion

Abstract

The application discloses a kind of hand rehabilitation evaluation system and method based on multi-modal data fusion, it is related to hand rehabilitation evaluation technical field, the system includes: sensor acquisition module and host computer, host computer includes data processing module, user interaction module, quick assessment module and individualized assessment module, quick assessment module is used to input three-axis attitude angle and target surface myoelectric signal into trained deep learning network, obtains first hand rehabilitation evaluation result;Individualized assessment module is used to calculate the motion similarity corresponding to the completion degree according to the reference data corresponding to the time sequence of the user's to-be-evaluated action and completion degree, to determine the second hand rehabilitation evaluation result of user.This application carries out hand rehabilitation evaluation through hand posture information and surface myoelectric signal and other multi-modal data, improves the comprehensiveness and accuracy of evaluation, and through individualized assessment module, the healthy side of patient can be compared to realize evaluation, and the rehabilitation progress of patient is accurately reflected.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of hand rehabilitation evaluation, and particularly relates to a hand rehabilitation evaluation system and method based on multi-modal data fusion. BACKGROUND

[0002] Traditional rehabilitation evaluation of subjects with hand movement dysfunction is mainly performed by manual evaluation by doctors or assisted evaluation by simple rehabilitation equipment. The traditional rehabilitation evaluation has the problems of great labor intensity, long evaluation time, strong subjectivity, high cost, boring process, poor initiative of subjects, and prolonged rehabilitation period. In recent years, rehabilitation treatment technologies for post-stroke hand movement dysfunction have been continuously emerging, such as forced movement therapy, transcranial magnetic stimulation therapy, and movement imagination therapy. Among them, the hand rehabilitation training method based on electromyography can significantly improve the hand movement function of stroke subjects, and the hand movement function of the subjects can be evaluated through inertial information. However, the evaluation mode is single and the interactivity is poor, which cannot adapt to the rehabilitation training needs of different subjects and different rehabilitation stages, that is, the individual adaptability and the initiative of the subjects are poor, and the evaluation of the hand movement function is not perfect. The existing automatic hand rehabilitation evaluation system based on sensors generally acquires hand information by wearing wearable data acquisition devices by patients, and evaluates the completion degree of actions. However, the existing automatic hand rehabilitation evaluation system based on sensors has the following defects: (1) complex wearing: because the motion information needs to be collected by using sensors, the patients need to wear wearable data acquisition devices, which may increase the motion burden and discomfort of the patients and affect the action effect; (2) high cost: the use of high-tech devices such as bio-information sensors may cause high system cost and high technical cost of product development; (3) poor interactivity: the interactive interface design is relatively flat, and the human-computer friendliness is not considered, so the enthusiasm and participation of the patients are reduced, and the rehabilitation effect is affected; (4) weak generalization: because the medical history, physical condition and rehabilitation stage of the patients are different, only the sensor data of the affected side is compared with the database data, it is difficult to realize individualized evaluation and rehabilitation scheme; (5) insufficient result effectiveness: the qualitative evaluation results in the evaluation results of the existing system lack accuracy, and the scientificity of the quantitative results is difficult to verify, which cannot fully meet the needs of clinical practice.

[0003] In order to solve the above problems, there are currently two technical solutions. In the existing solution one, the subject wears gloves on both healthy and affected sides, and collects inertial sensor and surface electromyography signal data for evaluation during rehabilitation. The collected electromyography signals create a classifier to identify the main movements in the virtual scene, including "picking fruit" and "building blocks". The main actions are grasping and stretching. The electromyography and inertial signals on the first and second arms of the subject are collected synchronously to evaluate the rehabilitation status of the subject, evaluate the current rehabilitation of the patient, and facilitate subsequent rehabilitation training. For the two types of data, after pre-processing the data and extracting information through feature engineering, the correlation between the healthy and affected sides is calculated, and the mean of the results obtained by the acceleration and electromyography signals is taken as the final result, and a qualitative evaluation grade is given. At the same time, different virtual scenes and leap motion are combined to realize the "four-hand linkage" human-computer interaction of the virtual hand and the healthy and affected sides.

[0004] In the existing solution two, the production of the glove hardware is described in detail. The sensors used are thin film pressure sensors installed on the fingertips and bending angle sensors on the palms of the fingers. The upper computer provides action guidance, and the specific actions are "palm stretching", "fist clenching", "thumbs up", "thumbs down", and "ok". This solution also evaluates rehabilitation by independently collecting electromyography and inertial data. Surface electromyography signals are used for healthy side hand action intention recognition to drive the affected side rehabilitation. In the evaluation, only the kinematic signals collected by the pressure sensors and bending sensors are used to collect inertial data. The logarithmic likelihood probability of the movement function of the affected hand relative to the healthy hand is calculated by a hidden Markov model to give a score.

[0005] Although the above two solutions can provide effective solutions to these problems, there are still many defects in terms of completeness. For the existing solution one, although there is a comparison between the healthy and affected sides, the action needs to be recorded in advance to establish the model, which is not very convenient. For the existing solution two, the electromyography signal is only used for action classification and recognition in the existing motion prompt software, which is not significant, which also leads to the participation of only inertial single modal data, and the perfection needs to be improved. On the other hand, in terms of the action strategy used in the evaluation, only a few simple actions such as grasping and stretching are used, so only the basic action function of the hand can be evaluated. Because there are no detailed actions, the evaluation report is relatively simple, so it is difficult to surpass the traditional physician-assisted rehabilitation scale evaluation in terms of effect, and further, the authority is also difficult to verify.

[0006] Therefore, there is an urgent need for a hand rehabilitation evaluation system with higher comprehensive and accurate evaluation ability for the rehabilitation status of the hands of stroke patients. SUMMARY

[0007] The application aims to provide a hand rehabilitation evaluation system and method based on multi-modal data fusion, which can evaluate hand rehabilitation through multi-modal data such as hand posture information and surface electromyography signals, improve the comprehensiveness, accuracy and robustness of stroke patient hand rehabilitation evaluation, and through the individualized scoring mechanism of the individualized evaluation module, compare the healthy side and the diseased side of the patient to realize evaluation and accurately reflect the rehabilitation progress of the patient.

[0008] To achieve the above-mentioned purpose, the application provides the following scheme:

[0009] In a first aspect, the application provides a hand rehabilitation evaluation system based on multi-modal data fusion, comprising:

[0010] A sensor acquisition module is configured to acquire original hand posture information and original surface electromyography signals of a user, wherein the original hand posture information and the original surface electromyography signals are acquired when the user performs a set action.

[0011] The upper computer includes a data processing module, a user interaction module, a rapid evaluation module and an individualized evaluation module.

[0012] The data processing module is configured to perform first preprocessing on the original hand posture information to obtain three-axis posture angles, and perform second preprocessing on the original surface electromyography signals to obtain target surface electromyography signals.

[0013] The user interaction module is configured to select the rapid evaluation module or the individualized evaluation module according to an input instruction of the user.

[0014] The rapid evaluation module is configured to input the three-axis posture angles and the target surface electromyography signals into a trained deep learning network to obtain a predicted score corresponding to the set action, and determine a first hand rehabilitation evaluation result of the user according to the scores corresponding to all set actions, wherein the hand rehabilitation evaluation result includes complete completion, partial completion and basically unable to complete.

[0015] The individualized evaluation module is configured to calculate an action similarity corresponding to a completion degree for each reference data corresponding to the completion degree according to a time sequence of an action to be evaluated of the user and the reference data corresponding to the completion degree, and determine a second hand rehabilitation evaluation result of the user according to all action similarities corresponding to the completion degree, wherein the reference data corresponding to the completion degree is hand data acquired when a healthy side hand of the user performs a preset action and the hand rehabilitation evaluation result is the completion degree, the time sequence of the action to be evaluated of the user is hand data acquired when a diseased side hand of the user performs the preset action, and the hand data includes three-axis posture angles and target surface electromyography signals.

[0016] In a second aspect, the present application provides a hand rehabilitation evaluation method based on the hand rehabilitation evaluation system based on multi-modal data fusion in the first aspect, comprising:

[0017] Collecting original hand posture information and original surface electromyography signals of a user; the original hand posture information and the original surface electromyography signals are collected when the user makes a set action;

[0018] Firstly, the original hand posture information is pre-processed to obtain three-axis posture angles; secondly, the original surface electromyography signals are pre-processed to obtain target surface electromyography signals;

[0019] The three-axis posture angles and the target surface electromyography signals are input into a trained deep learning network to obtain a predicted score corresponding to the set action; a first hand rehabilitation evaluation result of the user is determined according to the scores corresponding to all set actions; the hand rehabilitation evaluation result includes complete completion, partial completion and basically unable to complete;

[0020] For each reference data corresponding to a completion degree, an action similarity corresponding to the completion degree is calculated according to a time sequence of a to-be-evaluated action of the user and the reference data corresponding to the completion degree; a second hand rehabilitation evaluation result of the user is determined according to all action similarities corresponding to the completion degrees; the reference data corresponding to the completion degree is hand data collected when the healthy side hand of the user has the hand rehabilitation evaluation result of the completion degree when performing a preset action; the time sequence of the to-be-evaluated action of the user is hand data collected when the diseased side hand of the user performs the preset action; the hand data includes position, speed, acceleration, angle and electromyography signal.

[0021] According to the specific embodiments provided by the application, the following technical effects are disclosed: the application provides a hand rehabilitation evaluation system and method based on multi-modal data fusion, comprising a sensor acquisition module and an upper computer, the upper computer comprising a data processing module, a user interaction module, a rapid evaluation module and a personalized evaluation module, the rapid evaluation module being used for inputting three-axis attitude angle and target surface electromyographic signals into a trained deep learning network to obtain a predicted score corresponding to a set action; determining a first hand rehabilitation evaluation result of a user according to the scores corresponding to all set actions; the personalized evaluation module is used for calculating an action similarity corresponding to a completion degree according to a time sequence of a to-be-evaluated action of the user and the reference data corresponding to the completion degree for each completion degree corresponding reference data; determining a second hand rehabilitation evaluation result of the user according to the action similarities corresponding to all completion degrees. The application performs hand rehabilitation evaluation through multi-modal data such as hand posture information and surface electromyographic signals, improves the comprehensiveness, accuracy and robustness of hand rehabilitation evaluation, and through the personalized scoring mechanism of the personalized evaluation module, the healthy side of a stroke patient can be compared to realize evaluation, and the rehabilitation progress of the patient can be accurately reflected. BRIEF DESCRIPTION OF DRAWINGS

[0022] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings needed in the embodiments will be briefly introduced as follows. Obviously, the drawings in the following description only constitute some embodiments of the present application, and for those skilled in the art, other drawings can also be obtained without creative labor on the basis of these drawings.

[0023] Figure 1 A hand rehabilitation evaluation system structure schematic diagram based on multi-modal data fusion is provided for embodiment 1 of the present application.

[0024] Figure 2 A data glove schematic diagram is provided for embodiment 1 of the present application.

[0025] Figure 3 An electromyographic arm ring expansion structure schematic diagram is provided for embodiment 1 of the present application.

[0026] Figure 4 A reinforcement training module interface schematic diagram is provided for embodiment 1 of the present application.

[0027] Figure 5 A hand rehabilitation evaluation flowchart is provided for embodiment 1 of the present application.

[0028] Figure 6 A hand rehabilitation evaluation work flowchart is provided for embodiment 1 of the present application. DETAILED DESCRIPTION

[0029] With reference to the accompanying drawings, the technical solutions in the embodiments of the present application will be described clearly and completely. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative efforts belong to the scope of the present application.

[0030] The present application aims to provide a hand rehabilitation evaluation system and method based on multi-modal data fusion, which aims to evaluate hand rehabilitation through multi-modal data such as hand posture information and surface electromyography signals, improving the comprehensiveness, accuracy and robustness of hand rehabilitation evaluation, and through the individualized scoring mechanism of the individualized evaluation module, the patient's healthy and affected sides can be compared for evaluation, accurately reflecting the patient's rehabilitation progress. Provide accurate rehabilitation evaluation and individualized training for patients with post-stroke hand dysfunction. The system combines data gloves and electromyography arm rings to collect hand kinematics and electromyography signals in real time, processes and analyzes the scores using machine learning and deep learning algorithms, including a fast evaluation model based on the LSTM-CNN model and an individualized evaluation model based on the DTW algorithm. The system improves the accuracy and robustness of the results through multi-modal fusion technology, can quickly generate an individualized rehabilitation evaluation report, obtain the patient's muscle and nerve rehabilitation status, record historical evaluation information to generate a rehabilitation curve and provide rehabilitation recommendations.

[0031] In order to make the above-mentioned purposes, features and advantages of the present application more obvious and easy to understand, the present application will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0032] Embodiment 1

[0033] As shown in Figure 1 The present embodiment is used to provide a hand rehabilitation evaluation system based on multi-modal data fusion, which includes:

[0034] The sensor acquisition module is used to acquire the original hand posture information and the original surface electromyography signal of the user; the original hand posture information and the original surface electromyography signal are acquired when the user makes a set action.

[0035] The host computer includes a data processing module, a user interaction module, a fast evaluation module and an individualized evaluation module.

[0036] The data processing module is used to: first preprocess the original hand posture information to obtain three-axis posture angles; secondly preprocess the original surface electromyography signal to obtain a target surface electromyography signal.

[0037] The user interaction module is used to select the fast evaluation module or the individualized evaluation module according to the input instruction of the user.

[0038] The rapid assessment module is configured to: input the triaxial attitude angle and the target surface electromyogram into a trained deep learning network to obtain a predicted score corresponding to a set action; and determine a first hand rehabilitation assessment result of the user according to scores corresponding to all set actions, wherein the hand rehabilitation assessment result includes complete completion, partial completion, and substantially unable to complete.

[0039] The personalized assessment module is configured to: for each reference data corresponding to a completion degree, calculate an action similarity corresponding to the completion degree according to a time sequence of an action to be assessed of the user and the reference data corresponding to the completion degree; and determine a second hand rehabilitation assessment result of the user according to all action similarities corresponding to the completion degrees, wherein the reference data corresponding to the completion degree is hand data collected when a healthy hand of the user has a hand rehabilitation assessment result corresponding to the completion degree when performing a preset action, the time sequence of the action to be assessed of the user is hand data collected when a diseased hand of the user performs the preset action, and the hand data includes a triaxial attitude angle and a target surface electromyogram.

[0040] The hand rehabilitation assessment process is as shown in Figure 5 The workflow is as shown in Figure 6

[0041] The sensor acquisition module includes a data glove and an electromyographic armband, and the sampling frequencies of the two are 100 Hz and 1000 Hz, respectively.

[0042] 1. As shown in Figure 2 The data glove is composed of six inertial sensors and five bend sensors, and is used to collect original hand posture information of the user; the inertial sensors are arranged at the fingertip parts of the glove, and the bend sensors are installed at each finger joint; the original hand posture information includes acceleration, angular velocity, hand direction relative to the earth's magnetic field, and finger bending degree of the hand.

[0043] The present embodiment efficiently and accurately monitors and analyzes the hand actions and postures of the user through the data glove. The following is a detailed description of the hardware of the data glove:

[0044] ​The data glove in this embodiment is equipped with multiple high-precision sensors, including six inertial sensors (IMU) and five bend sensors, to comprehensively capture the motion and posture changes of the hand. These sensors work together to provide rich hand motion data, reflecting the dynamic activities of the user's hand. (1) Inertial sensors: including a three-axis accelerometer and a three-axis gyroscope, these sensors can capture the linear acceleration and angular velocity of the hand in real time, accurately tracking hand movements. The accelerometer has a range of ±16g, and the gyroscope has a range of ±2000dps, ensuring data accuracy even in fast and complex hand movements. (2) Bend sensors: installed at each finger joint, used to detect the degree of finger bending, providing more detailed hand motion data. These sensors enhance the precision of hand motion capture by detecting changes in finger bending.

[0045] The data glove uses Kalman filtering algorithm to fuse sensor data in real time, effectively reducing noise and interference, improving data stability and accuracy. Calibration ensures data reliability, so that the data glove can provide accurate hand motion data in various environments. The data glove is equipped with a rechargeable battery, supporting up to 7 hours of continuous use, suitable for long-term motion capture needs. Charging through USB interface simplifies the charging process and improves the convenience of use.

[0046] Connection and configuration: The data glove is connected to the PC through the data receiver, supporting 2.4GHz and 5.8GHz wireless dual-frequency communication, ensuring the stability and reliability of data transmission. Provides multiple hardware connection methods, including POE router, POE switch and ordinary home router, to adapt to different use environments and needs.

[0047] The data receiver provides isolated synchronous input and non-isolated synchronous output BNC interfaces, allowing time synchronization with other motion capture devices or systems, ensuring data consistency when multiple devices work together.

[0048] The design of the data glove takes into account the comfort and convenience of the user, the skin is adhered to ensure that the glove can fit the hand to accurately capture the motion; the components are relatively independent, and the dirty, old, and damaged skin can be replaced directly.

[0049] In the first preprocessing of the original hand posture information, the three-axis attitude angle is obtained, and the data processing module is used for:

[0050] Using Kalman filtering algorithm, the original hand posture information is fused to obtain the three-axis attitude angle; the three-axis attitude angle includes roll attitude angle, pitch attitude angle and yaw attitude angle.

[0051] The data glove collects human motion data through built-in sensors. Each data glove contains 6 inertial sensors and 5 bend sensors, which work together to collect high-dynamic human motion information to avoid motion distortion. The raw hand pose information collected includes:

[0052] Accelerometer data: range of ±16g, used to measure the acceleration of the hand.

[0053] Gyroscope data: range of ±2000dps, used to measure the angular velocity of the hand.

[0054] Magnetometer data: used to detect the direction of the hand relative to the Earth's magnetic field.

[0055] Bend sensor data: installed on the fingers, used to measure the degree of finger bending.

[0056] By fusing the collected raw data through the Kalman Filter algorithm, this algorithm can effectively integrate acceleration, angular velocity, magnetic value and bend sensor data to obtain high-precision pose angle. In addition, the data glove is calibrated to ensure the accuracy of the data. The Kalman Filter algorithm specifically includes the following steps:

[0057] Step 1. State prediction:

[0058]

[0059] where, is the state prediction at time step k, Φ k|k-1 is the state transition matrix, is the state estimate at the previous time step, u k is the process noise.

[0060] Step 2. Covariance prediction:

[0061]

[0062] where, P k|k-1 is the covariance estimate of the predicted state, Q k is the covariance matrix of the process noise.

[0063] Step 3. Kalman gain calculation:

[0064]

[0065] where, K k is the Kalman gain, H k is the observation model matrix, R k is the covariance matrix of the observation noise.

[0066] Step 4. State update:

[0067]

[0068] where, is the updated state estimate, z k is the actual observation.

[0069] Step 5. Covariance update:

[0070] P k = (I - K k H k )P k|k-1 (5);

[0071] where, P k is the updated state estimate's covariance. In the above formula, denotes the state estimate, P denotes the state estimate's covariance, K denotes the Kalman gain, u denotes the process noise, z denotes the observation, H denotes the observation model matrix, Q and R denote the covariance matrices of the process noise and the observation noise, respectively.

[0072] High-precision three-axis attitude angles, including Roll (roll attitude angle), Pitch (pitch attitude angle), and Yaw (yaw attitude angle), are obtained through data fusion. Dynamic and static accuracy: the measurement accuracy of Roll, Pitch, and Yaw under dynamic and static conditions is given, respectively.

[0073] 2. As shown in Figure 3 , the electromyographic arm ring (i.e., the surface electromyographic sensor) includes three dry electrode sheets and a magnetically attracted watchband for collecting the user's original surface electromyographic signals.

[0074] The surface electromyographic sensor in this embodiment adopts dry electrode technology and detects human surface electromyographic signals through active sensing, thereby reflecting the activity of human muscles and nerves. The surface electromyographic sensor is built-in with a filtering and amplifying circuit, which can amplify the weak electromyographic signals within the range of ±1.5 mV by 1000 times. Through differential input and analog filtering circuit, noise, especially power frequency interference, is effectively suppressed, and the output analog signal range is 0-3.0 V. Taking 1.5 V as the reference voltage, the output signal size directly reflects the activity intensity of the measured muscle, providing convenience for the analysis and research of surface electromyographic signals.

[0075] The supply voltage of the myoelectric arm ring ranges from 3.3 to 5.5 V, and the supply current is not less than 20 mA, so as to ensure stable operation and minimize ripple and noise. In order to ensure the performance of the sensor, a stabilized DC power supply is used. The effective frequency spectrum range of the myoelectric signal is set to 20 Hz to 500 Hz, therefore, the resolution of the analog-to-digital converter (ADC) used is not less than 8 bits, and the effective sampling frequency is not less than 1 KHz, so as to maximize the preservation of the original signal information and improve the accuracy and reliability of signal processing.

[0076] In this embodiment, a specially designed metal dry electrode plate is used, and the consistency of the electrode plate with the muscle direction is considered in the design, so as to optimize the signal capture efficiency and quality. The use of dry electrodes simplifies the preparation work, without the need for conductive gel, which is convenient for users to use, and improves the service life and stability of the electrode, and is especially suitable for daily use by non-medical professionals.

[0077] The dry electrode needs to be closely attached to the skin surface to accurately measure the surface electromechanical signal, and considering the convenience of wearing the myoelectric arm ring, the myoelectric arm ring is connected to the sensor through an elastic rope and is worn through a magnetic sticker, so as to balance the convenience and reliability.

[0078] In the second preprocessing of the original surface myoelectric signal to obtain the target surface myoelectric signal, the data processing module is used to:

[0079] The original surface myoelectric signal is sequentially subjected to deletion of noise, non-numerical characters and null values, rejection of interference, removal of direct current, band pass filtering, full wave rectification and normalization processing to obtain an intermediate surface myoelectric signal.

[0080] The intermediate surface myoelectric signal is divided based on a sliding window method to obtain a plurality of time-sliced surface myoelectric signals. For each time slice, the time-sliced surface myoelectric signal is subjected to feature extraction to obtain the myoelectric signal feature of the time slice. All the myoelectric signal features of the time slices constitute a target surface myoelectric signal. The myoelectric signal feature includes root mean square, average absolute value, waveform length, Willision amplitude, slope sign change frequency, median frequency and average power frequency.

[0081] Specifically, in this embodiment, a dry electrode muscle electrical signal sensor is used to collect the surface myoelectric signal of the contacted muscle, and an Arduino Uno is used for data reading and transmission. The received data is preprocessed, including deletion of noise, non-numerical characters and null values, and rejection of a maximum amplitude disturbance occurring when starting the test data, to obtain a preliminarily processed signal, i.e. an intermediate surface myoelectric signal.

[0082] The energy distribution of the myoelectric signal is usually concentrated in the range of 0 to 500 Hz, and the main component is in the range of 20 to 150 Hz. Due to the influence of various noises such as other bioelectric signals, motion artifacts, baseline drift, power frequency interference and electrode-skin noise, the myoelectric signal often needs to be preprocessed to obtain accurate analysis results. The following preprocessing process is performed on the intermediate surface myoelectric signal using the third-party library pyemgpipeline. The specific process is as follows: first, remove the direct current and band-pass filter to remove the influence of direct current signals, electrocardiogram signals, power frequency interference and other clutter, and retain the main frequency component of the myoelectric signal of interest. After filtering, the myoelectric signal often still contains negative parts, while users are usually interested in the amplitude of the signal and only analyze the positive part. Therefore, the signal is full-wave rectified, that is, the absolute value is taken, so as to facilitate subsequent analysis and processing. Finally, in order to eliminate the difference in signal amplitude under different collection conditions, the signal is standardized or normalized. This means that the signal is adjusted to the same amplitude range to ensure consistency and comparability of subsequent analysis.

[0083] After the above preprocessing, feature extraction of the surface myoelectric signal is also needed. For time-series myoelectric signals, the embodiment extracts the following 7 features from each time slice in time series in a sliding window manner. Preferably, the window size and step length are set to 300 and 100 respectively, so that the window is short enough to reduce the decision-making time of simultaneous control, while ensuring that each data window overlaps with the previous data window to prevent feature loss during segmentation.

[0084] 1) Root Mean Square (RMS)

[0085]

[0086] where x i represents the amplitude of the signal at the i-th time point, and N represents the total number of samples. When the muscle is under greater load or fatigue, the RMS decreases, and vice versa.

[0087] 2) Mean Absolute Value (MAV)

[0088]

[0089] where x i represents the amplitude of the signal at the i-th time point, and N represents the total number of samples. When the muscle is under greater load or fatigue, the MAV decreases as the RMS, but MAV does not consider the square of the signal when calculating, but directly averages the absolute value of the amplitude, so MAV is less affected by high-frequency noise and more stable than RMS.

[0090] 3) Waveform Length (ML)

[0091]

[0092] where x i represents the amplitude of the signal at the i-th time point, and N represents the total number of samples. When the muscle contracts, the waveform of the sEMG signal becomes more complex, the curve becomes more "steep", and the waveform length increases.

[0093] 4) Willison amplitude WA

[0094]

[0095] where

[0096] where x i represents the amplitude of the signal at the i-th time point, and N represents the total number of samples. The number of times the amplitude of the surface electromyogram signal changes by more than the threshold value, which is generally 50-100 uv, is counted, and the waveform length WA is changed.

[0097] 5) Slope sign change frequency SSC

[0098]

[0099] where:

[0100]

[0101] where x i represents the amplitude of the signal at the i-th time point, and N represents the total number of samples. Its change is the same as the waveform length WA.

[0102] Secondly, the frequency domain characteristics of the signal are generally used to judge the degree of muscle fatigue, and the left shift of the characteristic frequency may represent the occurrence of fatigue. When extracting the frequency domain signal, the amplitude-frequency spectrum is first obtained by fast Fourier transform (FFT), and the energy density spectrum is obtained by squaring the amplitude-frequency. The energy density ratio of time gets the power spectrum PSD(f). Because of the involvement of FFT, only short-time stationary signals can be analyzed and the calculation is relatively complex. Among them, the fast Fourier transform formula is as follows:

[0103]

[0104] 6) Median frequency MF

[0105]

[0106] where PSD(f) represents the power spectral density of frequency f. Short-time stationary has good characteristics in terms of sensitivity to physiological parameters and ability to resist noise interference, and is the first choice for researchers when analyzing the frequency domain of the signal.

[0107] 7) Average power frequency MPF

[0108]

[0109] where PSD(f) represents the power spectral density at frequency f, and f represents the sampling frequency.

[0110] The action scheme is specifically introduced as follows:

[0111] The embodiment designs a matching action scheme for post-stroke hand rehabilitation condition evaluation, which involves 14 actions of 21 degrees of freedom of wrist joint, metacarpophalangeal joint and interphalangeal joint, and guides the user to complete them in sequence under the guidance of a software interface demonstration video. The design of the action scheme is based on reference to the authoritative scale, in-depth anatomical mechanism, and improvement combined with the actual scene.

[0112] Scale evaluation is the most recognized clinical movement disorder evaluation method at present, and the Fugl-Meyer Assessment for Upper Extremity (FMA-UE) scale is the authoritative scale for stroke rehabilitation. In order to improve the rationality and persuasiveness, the design scheme includes all the actions of the Fugl-Meyer hand, thereby connecting to the traditional scale evaluation system.

[0113] On this basis, the principle of human anatomy and physiology is further studied, and the muscles involved in each action and their innervation are summarized from the anatomical structure of the forearm, wrist and hand muscles, tendon sheath and nerves, and the traditional actions are further optimized to achieve comprehensive coverage of each muscle. There are the following three parts of detail modification:

[0114] (1) Wrist rotation: the scale lacks separate examination of the function of wrist internal rotation and external rotation, and the muscles responsible for wrist rotation in kinematics, i.e. supinator, pronator quadratus, pronator teres and brachioradialis, are relatively independent in function, so the item of "wrist rotation" is added to examine the function of these corresponding muscles.

[0115] (2) Thumb and little finger opposition: opposition movement refers to the movement of the palmar surface of the thumb tip and the palmar surface of each finger, which is unique to the human hand and plays an important role in fine manipulation and gripping action. Therefore, the item of "thumb and little finger opposition" is added to examine the function of the long flexor muscle of the thumb, the adductor muscle of the thumb, the short flexor muscle of the thumb, the palmaris muscle of the thumb and the palmaris muscle of the little finger in the medial and lateral muscle groups of the hand.

[0116] (3) Extending the index finger and the little finger: the scale's examination of hand mobility is limited to the coordinated movement of the five fingers, while each finger moves independently, so the items of "extending the index finger" and "extending the little finger" are added to check the condition of the extensor muscle of the index finger, the extensor muscle of the little finger, the lumbrical muscle, the dorsal interosseous muscle and the palmar interosseous muscle. Since the middle finger and the ring finger do not have independent tendons, they often move together with other fingers in a coordinated manner in daily life, so they are not separately examined.

[0117] (4) Wrist adduction and wrist abduction: In order to make the definition of "inadequate completion" clearer, the "wrist circular movement" of the wrist in the scale is broken down into two parts: wrist adduction (ulnar deviation) and wrist abduction (radial deviation).

[0118] Subsequently, improvements were made to the invention based on the specific scenario of independent home rehabilitation for patients. Since patients living independently cannot complete the assessment method in the scale that involves observing resistance to certain movements, the method was improved by prompting patients to exert sustained force after completing the movement, causing isometric contraction of the skeletal muscles, and then acquiring muscle strength information through electromyography (EMG). The improved movement plan and textual description are shown in Table 1.

[0119] Table 1: Description of Action Plan

[0120]

[0121]

[0122] In this embodiment, the host computer comprises two parts: the hardware and software environment and the software.

[0123] (1) The hardware component is described as follows: The host computer's hardware and software environment is based on a computer host with a Windows operating system installed. This host is responsible for wireless communication with the slave computer, realizing data acquisition and processing, and providing guidance, progress display, and result presentation during user operation. The hardware environment includes the computer host and related peripherals, while the software environment covers the Windows operating system and the developed dedicated software.

[0124] (2) The host computer software is described as follows: The host computer software mainly includes two modules: data parsing and algorithm analysis.

[0125] The data parsing module utilizes Windows' wireless and Bluetooth modules to read and parse data packets from the data glove and electromyography (EMG) armband. This module is responsible for extracting motion data and EMG signals recorded by the sensors, storing the read data as a file, and then inputting it into the model after processing to obtain evaluation results.

[0126] The algorithm analysis module preprocesses and extracts features from the patient's motion data, and then inputs it into the evaluation model to generate evaluation results.

[0127] The host computer includes a personal database module: before database entry, the approximate time required and the purpose of establishing the database are explained to the user, and the collected data only includes kinematic and electromyographic data for specific movements on the healthy side, without involving critical personal privacy information. During entry, the healthy side wears a data glove, and for each movement in the assessment movement plan, three completion statuses are simulated: complete completion, partial completion, and basically unable to complete (representing the hand rehabilitation assessment results).

[0128] When entering the action database, for each action, action guidance is given in the interface, including 3D modeling animation, hand real scene animation and text description, prompting the user to perform each action once, without limiting the action duration. A recording progress bar is displayed. The start and end of action data reading are realized by the user clicking the "start action" and "next action" user actions respectively.

[0129] User interface: Design an interactive interface that integrates rehabilitation assessment and motor training, including two modes of quick assessment and personalized assessment, as well as a query function to provide historical rehabilitation information to meet the needs of different users.

[0130] The evaluation part of this embodiment provides two modes of "quick assessment" and "personalized assessment". Among them, the "quick assessment" mode is suitable for patient product experience scenarios, without the need for pre-entry of patient data; while the "personalized assessment" mode requires the user to log in and establish a personal hand action database to achieve personalized assessment. During the entry process, the system provides detailed action guidance and entry progress prompts to ensure the accuracy and completeness of the data. At the same time, the system can guide the patient to complete the relevant actions through video and voice guidance, while presenting the modeling of the collected data in real time on the interface to enable the patient to understand their own movement situation. Each action performed by the patient is accompanied by a popular science explanation, enhancing the patient's participation in decision-making and improving the patient's acceptance. In addition, after completing a set of assessment, the patient not only obtains the corresponding FMA-UE scale score, but also obtains the rehabilitation information of the relevant muscles and nerves, records the historical assessment information, generates a rehabilitation curve, and provides rehabilitation suggestions for targeted strengthening training to improve rehabilitation effect.

[0131] The user first starts the data collection sequence by clicking the "start" button on the graphical user interface (GUI). This action activates a preset function that calls a specific executable file to read data from the inertial measurement unit (IMU) in real time and records these data in a text file in sequence. At the same time, a specially developed electromyographic signal acquisition program is also activated, which receives electromyographic data from the Arduino device through a serial port and records it. In order to meet the continuity requirements of data acquisition and signal processing, while avoiding the mutual blocking of programs due to single-threaded execution, a multi-threaded execution strategy is adopted to ensure the smoothness and real-time response capability of software operation.

[0132] To further process and manage these collected EMG and inertial data, the embodiment imports them into a main processing program. In this program, the data is converted and managed, and an automatic scoring process is triggered. When performing a quick test, the embodiment calls a pre-trained deep learning network (a long short-term memory network LSTM-convolutional neural network CNN fusion model) to evaluate the IMU and EMG data. Using the weights determined based on experimental data regression analysis, a final score is calculated, and the score result is saved in an action score list and directly displayed to the user through a pop-up window.

[0133] The trained deep learning network in the quick evaluation module includes an LSTM module, a convolution module, a global average pooling layer (GAP), a BN layer, and a classification module connected in sequence; the LSTM module includes a first LSTM layer and a second LSTM layer connected in sequence, each LSTM layer has 32 neurons, and the activation function is Relu; the convolution module includes a first convolution layer, a max pooling layer, and a second convolution layer, the convolution layer is used to extract spatial features, the first convolution layer (first CNN layer) has 64 neurons, the second convolution layer (second CNN layer) has 128 neurons, and a max pooling layer is used between the two CNN layers for downsampling operation; the global average pooling layer converts multi-dimensional feature mapping into a one-dimensional feature vector to reduce global model parameters; the BN layer is used to improve the convergence of the model, thereby ensuring the stability and accuracy of the model; and the classification module includes a fully connected layer and a Softmax classifier.

[0134] The training process of the deep learning network is as follows:

[0135] First, the model architecture of the LSTM layer and then the CNN layer can use the LSTM layer to process and remember the time dependence in the time series, and then use the CNN layer to extract the spatial features of the sequence. This combination method enables the model to more comprehensively understand and classify hand movements, as it not only considers the dependence of movements over time, but also considers the spatial form and important local features in the movements.

[0136] (1) Construction of the deep learning network:

[0137] (a). LSTM layer

[0138] The LSTM model inherits the advantages of recurrent neural networks (RNN) in feature extraction of sequence data, while solving the problem of gradient disappearance. The length of hand movement data may vary due to the complexity of the movement and the execution speed. LSTM can handle this variable-length input sequence, allowing the model to adapt to different lengths of movement data.

[0139] The storage structure thereof includes an input gate, a forget gate and an output gate, and the activation function is as follows:

[0140] h t = σ(w i，h · x t + w h，h · h t-1 + b) (14);

[0141] wherein σ represents a sigmoid activation function, w i，h , w h，h is a weight matrix, b is a bias, h t-1 is a hidden state at a previous moment, and t is an input at a current moment.

[0142] In the model used in the present embodiment, two layers of LSTM are constructed after inputting data to extract time features in sequence data. The number of storage units of each layer of LSTM is 32. To adapt to the input shape of the CNN convolutional layer, the output of the second layer of LSTM is expanded from three dimensions to four dimensions.

[0143] (b). CNN convolutional layer and pooling layer

[0144] Through convolution and pooling operations, CNN can reduce the dimensionality of data while retaining important feature information, not only reducing the required computing resources for subsequent processing, but also helping to reduce the risk of overfitting, and more effectively extracting spatial features. The convolutional layer can identify local patterns in the data, which may appear multiple times in the entire time series.

[0145] The CNN convolutional layer uses a convolution kernel to convolve the input and is activated by a nonlinear activation function:

[0146]

[0147] wherein W m,n is a convolution kernel, b is a bias term, and X i+m,j+n is an input feature. The result Y i,j of the convolution operation is nonlinearly transformed by an activation function f to obtain an activated output feature map Z i,j . The activation function uses a rectified linear unit (ReLU) to calculate the feature map of the convolutional layer:

[0148] o(x) = max(0, x) (16);

[0149] wherein x represents a value in the input feature map. In the convolutional layer, 64 convolution kernels are used for feature extraction, each convolution kernel has a size of 1x5, and the sliding step of the convolution window is 2. There is a max pooling layer between the two convolutional layers, which is used to perform downsampling operations.

[0150] (c). Global average pooling layer and batch normalization layer

[0151] This embodiment improves the classic CNN, replaces the fully connected layer after the convolutional layer with a global average pooling layer to reduce the model parameters, and at the same time improves the robustness to the spatial transformation of the input. A batch normalization layer is added after it to normalize and reconstruct the input data of each batch of training samples to speed up the convergence of the model.

[0152] (d). Output layer (classification module)

[0153] The output layer is composed of a fully connected layer and a Softmax classifier, where the fully connected layer extracts features from the upper layer, and the Softmax classifier classifies the rehabilitation completion degree:

[0154]

[0155] where z i is the original output of the model for the i-th class, is the exponential function of z i . The denominator is the sum of all , which ensures that the sum of the probabilities of all classes is 1.

[0156] (2) Training set construction and model training:

[0157] For each action, 1000s (i.e. one hundred thousand frames) of each of the three completion conditions of the action are recorded, and after preprocessing, the three completion conditions of the action are trained using a deep learning network model. The data can be cut into multiple segments according to fixed time intervals by defining a time step window and a step, and each segment contains a predetermined number of time point data. During this process, the label column and the specified non-feature column are automatically excluded, and the key feature information is retained.

[0158] Using the constructed data set, the classification model is trained and learned through back propagation and gradient descent algorithm, and the model parameters with high prediction accuracy and strong generalization performance are selected for saving.

[0159] (3) Application of the trained deep learning network:

[0160] The data to be tested is preprocessed, including standardization and feature extraction, and the data quality is improved by using wavelet transform denoising and standardization processing to effectively extract key features. And the data is reshaped to a format suitable for the model, and each sensor column data is processed by dividing the sliding time window to ensure that the model can capture the subtle differences of the action. The trained three-classification model is used to predict which class the strange data belongs to among the three classes, so as to obtain the completion degree score of the action.

[0161] The quality of the trained deep learning network depends on a large amount of collection of patient data, and the number, diversity and collection time of the patients have considerable influence on the evaluation effect of the model.

[0162] The personalized evaluation module is configured to: for each reference data corresponding to a completion degree, calculate the action similarity corresponding to the completion degree according to the time sequence of the action to be evaluated of the user and the reference data corresponding to the completion degree by using a dynamic time warping algorithm.

[0163] The dynamic time warping algorithm (DTW algorithm) is to construct a warping network, which finds the best match between two time series by minimizing the global distance between them. Let there be two time series X=(x1, x2,..., x N ) and Y=(y1, y2,..., y M ), where N and M are the lengths of sequences X and Y respectively. The goal of the DTW algorithm is to find a path from sequence X to sequence Y with the minimum total distance.

[0164] 1. Distance metric: First, define a distance function d(x i ,g j ) to measure the distance between two points x i and y j . Common distance metrics include Euclidean distance and Manhattan distance, and the embodiment selects to use the Euclidean distance calculation method.

[0165]

[0166] where p=(p1, p2,..., p n ) and q=(q1, q2,..., q n ) are the coordinates of two points in n-dimensional space, and p i and q i are the coordinate values of point p and q in the i-th dimension.

[0167] 2. Cumulative distance: For each pair of points (x i , y j ), calculate its cumulative distance D(i, j), which represents the minimum path distance from the beginning of sequence X to point x i , and from the beginning of sequence Y to y j . The cumulative distance can be calculated using the following recursive formula:

[0168] D(i, j) = min{D(i-1, j), D(i, j-1), D(i-1, j-1)} + d(x i , y j ) (19);

[0169] where D(0,0) = 0.

[0170] 3. The path is regularized: by accumulating the distance matrix D, we can trace back a path from (x N , y M ) to (x1, y1) with the total distance D(N, M).

[0171] In this task, the input of the DTW algorithm is two time series: one is the time series of the reference action (i.e. the benchmark data), and the other is the time series of the action to be evaluated. These time series are composed of hand data collected from inertial sensors, bending sensors, surface electromyography sensors after processing, including position, velocity, acceleration, angle, electromyography signal features, etc.

[0172] Input two time series X and Y, where X is the time series of the reference action, and Y is the time series of the action to be evaluated. The output is a single numerical value representing the similarity between the action to be evaluated and the reference action. This value is the DTW distance between the two time series, which reflects the degree of completion of the action to be evaluated. The smaller the DTW distance, the more similar the two actions, i.e. the higher the degree of completion of the action to be evaluated, indicating that the function of the affected hand has recovered better. By comparing the test data with the benchmark data of the same action in the individual database with three degrees of completion, the smaller distance corresponds to the score of this action.

[0173] The DTW algorithm can effectively evaluate the degree of completion of hand actions, even if there are differences in the speed or duration of the action, which can be aligned through the algorithm. It is worth noting that this algorithm has high requirements for data preprocessing, feature engineering, and the quality of benchmark actions, see the individual database entry block in software design. Only a small amount of information from the healthy side of the patient is needed to achieve personalized evaluation, improving the efficiency and accuracy of the evaluation.

[0174] The dynamic time warping algorithm can quantify the similarity between two time series with incomplete matching of positions on the speed or time axis. When using the dynamic time warping algorithm for personalized evaluation, the following advantages are obtained:

[0175] 1. Less data required: when directly applying this model, only a set of data for each type of scoring action is needed to score based on this benchmark, without the need for patients to repeatedly enter data, and without the need for a large training sample.

[0176] 2. Good generalization, can compare healthy and affected sides: By collecting the motion information of the healthy side of the individual, the healthy and affected sides can be compared, so that the score can be based on the individual's rehabilitation condition; especially for the electromyography module, the data difference between individuals is large, and referring to other people's data for scoring is often not accurate enough - by comparing and scoring with the individual's own data, such evaluation is more personalized.

[0177] For users who need personalized testing, the system first checks whether the corresponding user's benchmark database can be accessed. If the database does not exist, the system will prompt the user to establish a personalized database or choose to use the quick test function. Once the existence of the benchmark database is confirmed, the system will process the IMU and electromyography data using the dynamic time warping (DTW) algorithm, respectively, and through a series of accurate weighted calculations, the final score will be obtained and presented to the user in the form of a pop-up window. In particular, the DTW algorithm mainly performs column deletion operations when processing IMU data, and performs regression analysis and weight distribution based on feature classification when processing electromyography data.

[0178] The upper computer further includes a rehabilitation evaluation report obtaining module configured to: after obtaining the hand rehabilitation evaluation result, determine a rehabilitation evaluation report according to the hand rehabilitation evaluation result; the rehabilitation evaluation report includes a predicted score of each set action in the action scheme, a Fugl-Meyer scale score, a B scale score, a radar chart of completion conditions of several basic actions, a radar chart of contraction abilities of several muscle groups, joint activities, and subsequent training suggestions.

[0179] The upper computer further includes a reinforcement training module configured to: perform action training by using the sensor acquisition module.

[0180] The reinforcement training module formulates a rehabilitation plan for active rehabilitation for the patient, and obtains reinforcement training suggestions through evaluation, so that the user can repeatedly train specific actions in this module, thereby enhancing the related muscle and nerve abilities.

[0181] When the user has performed the evaluation, the user can click the reinforcement training function. In this interface, the system pushes the top 5 actions most effective for rehabilitation, and the patient can switch actions by clicking the lower right corner of the interface, imitate the action in the video example, and try to complete the action. At this time, the wearing of the data glove and the electromyography armband actually provides a certain movement resistance, which helps the patient to resist active movement, and at the same time reduces muscle synergy to a certain extent.

[0182] As Figure 4As shown, the interface design of the reinforcement training module places particular emphasis on user-friendliness in human-computer interaction. The software not only displays a progress bar and time elapsed, but also provides rich prompts, including text descriptions and video demonstrations, to guide users in performing the movements correctly. The video tutorials guide users through the exercises in a loop, while the text descriptions include detailed breakdowns of the movements to help users standardize their operations, as well as popular science information about muscles and nerves to enhance users' understanding of muscle movement principles. Furthermore, by utilizing MotionVenus modeling and simulation technology, the software can reflect the user's hand movements in real time, achieving two-way information feedback between the human and the machine. Users can switch between two interfaces of different sizes. Initially, the video tutorials are played primarily on the larger interface. As users become familiar with the movement flow, they can switch to the modeling and simulation interface for a more intuitive observation of their movements, increasing the enjoyment of the rehabilitation process.

[0183] This embodiment has the following beneficial effects:

[0184] 1. This embodiment integrates an inertial measurement unit, a flexion sensor, and a surface electromyography (EMG) sensor to acquire information on the patient's limb movement status from multiple dimensions. This multimodal comprehensive assessment can more comprehensively reflect the patient's motor ability and rehabilitation status. Compared with methods relying on a single data source, it can provide more accurate and comprehensive data support, thereby improving the accuracy of the assessment.

[0185] 2. This embodiment employs a fusion model of LSTM and CNN, combined with a personalized scoring mechanism based on DTW technology. This allows for a comparative assessment of the patient's healthy and affected sides, accurately reflecting the patient's rehabilitation progress. The fusion model of Long Short-Term Memory Network and Convolutional Neural Network, combined with dynamic time planning technology, enables the scoring mechanism to be personalized for different usage scenarios. This method considers both the model's prediction accuracy in specific scenarios and enhances its generalization ability, thereby enabling more precise assessment and monitoring of subtle changes in patients during neuromuscular rehabilitation.

[0186] 3. This embodiment describes a motion adaptation method designed for specific hardware and application scenarios. Through data analysis during execution, a performance score is derived and matched with the Fugl-Meyer Assessment for Upper Extremity scale. Simultaneously, the motion is linked to muscle and nerve function, thus mapping the results to specific muscle and nerve recovery status. This not only increases the accuracy of the assessment but also makes the assessment process more aligned with actual rehabilitation needs, helping medical personnel and patients to better understand the rehabilitation progress.

[0187] 4、The embodiment designs a user-friendly and easy-to-operate interactive interface, which enables patients to perform rehabilitation training independently without the on-site guidance of physicians. This not only improves the convenience and accessibility of rehabilitation training, but also may increase the rehabilitation enthusiasm of patients, thereby accelerating the rehabilitation process through daily self-training.

[0188] The embodiment comprehensively realizes significant advantages in accuracy, personalization, practicality, and user experience through a multi-modal comprehensive evaluation system, a personalized scoring mechanism based on advanced deep learning models, a motion adaptation method in specific hardware and application scenarios, and a user-friendly interactive interface design. This unique comprehensive methodology enables the embodiment to comprehensively capture the motion state information of the patient's limbs in multiple dimensions, achieving highly accurate rehabilitation progress monitoring and evaluation. Through a personalized scoring mechanism, it adapts to diverse use scenarios, providing a unique neuromuscular rehabilitation solution. In addition, considering the autonomy and convenience of patients during rehabilitation training, the human-computer interaction design of the embodiment significantly improves the rehabilitation experience and training enthusiasm of patients. These unique technologies and methods ensure that no other related inventions can achieve the same comprehensive effect as the embodiment, and have obvious application value in the field of neuromuscular rehabilitation.

[0189] Embodiment 2

[0190] The embodiment provides a multi-modal data fusion-based hand rehabilitation evaluation method based on the multi-modal data fusion-based hand rehabilitation evaluation system of embodiment 1, comprising:

[0191] Collecting original hand posture information and original surface electromyography signals of a user; the original hand posture information and the original surface electromyography signals are collected when the user performs a set action;

[0192] Firstly, the original hand posture information is pre-processed to obtain three-axis posture angles; secondly, the original surface electromyography signals are pre-processed to obtain target surface electromyography signals;

[0193] The three-axis posture angles and the target surface electromyography signals are input into a trained deep learning network to obtain a predicted score corresponding to the set action; a first hand rehabilitation evaluation result of the user is determined according to the scores corresponding to all set actions; the hand rehabilitation evaluation result includes complete completion, partial completion, and basically unable to complete;

[0194] For each reference data corresponding to a completion degree, a motion similarity corresponding to the completion degree is calculated according to a time sequence of a to-be-evaluated motion of the user and the reference data corresponding to the completion degree; a second hand rehabilitation evaluation result of the user is determined according to all the motion similarities corresponding to the completion degrees; the reference data corresponding to the completion degree is hand data collected when a healthy hand of the user performs a preset motion and the hand rehabilitation evaluation result is the completion degree; the time sequence of the to-be-evaluated motion of the user is hand data collected when a diseased hand of the user performs the preset motion; and the hand data includes position, speed, acceleration, angle and electromyographic signal.

[0195] The technical features of the above embodiments can be combined in any manner. To make the description concise, not all possible combinations of the technical features in the above embodiments are described, but as long as the combinations of the technical features do not exist contradictions, they should be considered as the scope of the present disclosure.

[0196] The principles and implementation manners of the present application are described by using specific examples in the present disclosure, and the above description of the embodiments is only used to help understand the method of the present application and its core idea; meanwhile, for those skilled in the art, the specific implementation manners and application scopes can be changed according to the idea of the present application. In conclusion, the content of the present disclosure should not be understood as a limitation of the present application.

Claims

1. A hand rehabilitation assessment system based on multimodal data fusion, characterized in that, include: The sensor acquisition module is used to: acquire the user's raw hand posture information and raw surface electromyography signals; The original hand posture information and the original surface electromyography signal are collected when the user performs the set action; The host computer includes a data processing module, a user interaction module, a rapid evaluation module, and a personalized evaluation module; The data processing module is configured to: perform a first preprocessing on the original hand posture information to obtain three-axis posture angles; perform a second preprocessing on the original surface electromyography (EMG) signal to obtain a target surface EMG signal; regarding the first preprocessing of the original hand posture information to obtain three-axis posture angles, the data processing module is configured to: use a Kalman filter algorithm to perform data fusion on the original hand posture information to obtain three-axis posture angles; the three-axis posture angles include roll posture angle, pitch posture angle, and yaw posture angle; regarding the second preprocessing of the original surface EMG signal to obtain a target surface EMG signal, the data processing module is configured to: sequentially perform noise removal, non-numerical characters and null values ​​removal, interference removal, DC removal, bandpass filtering, full-wave rectification, and normalization on the original surface EMG signal to obtain an intermediate surface EMG signal; The intermediate surface electromyography (EMG) signal is divided into several time slices based on the sliding window method to obtain surface EMG signals. For each time slice, feature extraction is performed on the surface electromyography (EMG) signal of the time slice to obtain the EMG signal features of the time slice; The electromyographic signal features of all the time slices constitute the electromyographic signal of the target surface; The electromyographic signal characteristics include root mean square, mean absolute value, waveform length, Willision amplitude, number of slope sign changes, median frequency, and average power frequency. The user interaction module is used to select between the quick assessment module and the personalized assessment module based on the user's input instructions. A rapid assessment module is used to: input triaxial pose angles and target surface electromyography signals into a trained deep learning network to obtain predicted scores corresponding to set actions; determine the user's first hand rehabilitation assessment result based on the scores corresponding to all set actions; the hand rehabilitation assessment result includes complete completion, partial completion, and basically no completion; the trained deep learning network includes a sequentially connected LSTM module, a convolutional module, a global average pooling layer, a batch normalization (BN) layer, and a classification module; the LSTM module includes a sequentially connected first LSTM layer and a second LSTM layer; the convolutional module includes a first convolutional layer, a max pooling layer, and a second convolutional layer; the classification module includes a fully connected layer and a Softmax classifier; The personalized assessment module is used to: calculate the similarity of the action corresponding to each completion level based on the user's action time sequence and the baseline data corresponding to the completion level; determine the user's second hand rehabilitation assessment result based on the action similarity corresponding to all completion levels; the baseline data corresponding to the completion level is the hand data collected when the user's healthy hand performs the preset action and the hand rehabilitation assessment result is the completion level; the user's action time sequence is the hand data collected when the user's diseased hand performs the preset action; the hand data includes three-axis posture angles and target surface electromyography signals.

2. The hand rehabilitation assessment system based on multimodal data fusion according to claim 1, characterized in that, The sensor acquisition module includes a data glove and an electromyography arm loop; The data glove consists of six inertial sensors and five bending sensors, used to collect the user's original hand posture information; the inertial sensors are located at the fingertips of the glove, and the bending sensors are installed at each finger joint; the original hand posture information includes the hand's acceleration, angular velocity, the direction of the hand relative to the Earth's magnetic field, and the degree of finger bending. The electromyography armband includes three dry electrode pads and a magnetic strap, used to collect the user's raw surface electromyography signals.

3. The hand rehabilitation assessment system based on multimodal data fusion according to claim 1, characterized in that, The host computer also includes a rehabilitation assessment report acquisition module, used for: After obtaining the hand rehabilitation assessment results, a rehabilitation assessment report is determined based on the hand rehabilitation assessment results; the rehabilitation assessment report includes the predicted score of each set movement in the movement program, the Fugl-Meyer scale score, the B scale score, the radar chart of movement completion, the radar chart of muscle group contraction ability, the range of motion of each joint, and subsequent training suggestions.

4. The hand rehabilitation assessment system based on multimodal data fusion according to claim 1, characterized in that, The personalized evaluation module is used to: for each level of completion, use a dynamic time warping algorithm to calculate the action similarity corresponding to the level of completion based on the user's action time sequence to be evaluated and the benchmark data corresponding to the level of completion.

5. A hand rehabilitation assessment system based on multimodal data fusion according to claim 1, characterized in that, The host computer also includes an enhanced training module, used for: performing motion training using the sensor acquisition module.

6. A hand rehabilitation assessment method based on multimodal data fusion, based on the hand rehabilitation assessment system based on multimodal data fusion as described in claim 1, characterized in that, include: Collect the user's raw hand posture information and raw surface electromyography signals; The original hand posture information and the original surface electromyography signal are collected when the user performs the set action; The original hand posture information is subjected to a first preprocessing to obtain the three-axis posture angle; the original surface electromyography signal is subjected to a second preprocessing to obtain the target surface electromyography signal. The three-axis posture angles and electromyographic signals of the target surface are input into a trained deep learning network to obtain a predicted score corresponding to the set action; the user's first hand rehabilitation assessment result is determined based on the scores corresponding to all set actions; the hand rehabilitation assessment result includes complete completion, partial completion, and basically no completion; For each level of completion, the baseline data is used to calculate the similarity of the action corresponding to the level of completion based on the user's action time sequence and the baseline data. The user's second hand rehabilitation assessment result is determined based on the similarity of the actions corresponding to all levels of completion. The baseline data corresponding to the level of completion is the hand data collected when the user's healthy hand performs the preset action and the hand rehabilitation assessment result is the level of completion. The user's action time sequence is the hand data collected when the user's diseased hand performs the preset action. The hand data includes position, velocity, acceleration, angle, and electromyographic signals.

Images