Post-stroke hand movement function rehabilitation state evaluation system and method and training system

Abstract

The invention discloses a post-stroke hand movement function rehabilitation state evaluation system and method and a training system, and relates to the field of medical rehabilitation, the evaluation system comprises a signal acquisition device, a communication device and an upper computer; the training system comprises an evaluation system and a virtual reality device. The signal acquisition device acquires multi-modal motion signals of a stroke patient completing hand rehabilitation training, a multi-modal hand motion state evaluation model is deployed in the upper computer and used for preprocessing the multi-modal motion signals, inputting the preprocessed multi-modal motion signals into the multi-modal hand motion state evaluation model and outputting a hand motion state score prediction value, and the score prediction value is used for evaluating the hand motion state of the stroke patient. And performing hand motion function rehabilitation state evaluation according to the hand motion state score prediction value. According to the invention, the accuracy of evaluating the rehabilitation state of the hand movement function can be improved.

Description

Technical Field

[0001] The present application relates to the field of medical rehabilitation technology, and in particular to a post-stroke hand motor function rehabilitation status assessment system, method and training system. Background Art

[0002] Hand function rehabilitation is an important part of stroke patients' recovery of daily living ability. Fine movements of the hands have a vital impact on the individual's self-care ability and quality of life. With the development of intelligent technology and robotics technology, hand function rehabilitation training systems based on data gloves provide patients with new treatment options. These systems use sensors to capture hand movements and help patients rebuild hand movement control through real-time feedback and personalized training programs. Data gloves integrate advanced sensor technology to accurately monitor and evaluate the range of motion, strength and coordination of hand joints, providing a quantitative and traceable evaluation method for rehabilitation treatment.

[0003] There are many data glove-based hand function rehabilitation programs on the market, and they have different focuses in design and function. Some data gloves focus on providing passive or auxiliary hand movement training, that is, exoskeletons, which simulate the natural movement of the hands through built-in sensors and actuators to help patients restore muscle strength and joint flexibility; other programs combine gamification elements to stimulate patients' training interest and participation through interactive games, while also adding fun to the rehabilitation process. These gloves are usually used in conjunction with professional rehabilitation software, which can customize training plans according to the patient's specific situation and monitor the training effect in real time.

[0004] However, although these programs have shown certain effects in clinical trials, there are still some challenges. For example, existing rehabilitation treatment evaluation methods are generally based on single-modality evaluation and lack more objective multimodal evaluation methods. The accuracy of evaluation results needs to be improved. Summary of the invention

[0005] The purpose of this application is to provide a post-stroke hand motor function rehabilitation status assessment system, method and training system to solve the problem of low accuracy of existing hand motor function rehabilitation status assessment.

[0006] To achieve the above objectives, this application provides the following solutions:

[0007] In a first aspect, the present application provides a post-stroke hand motor function rehabilitation status assessment system, comprising: a signal acquisition device, a communication device and a host computer;

[0008] The signal acquisition device is worn on a stroke patient, and is used to acquire multimodal motion signals of the stroke patient who has completed hand rehabilitation training; the multimodal motion signals include electroencephalogram signals, electromyography signals, and hand motion signals;

[0009] The host computer is connected to the signal acquisition device through the communication device, and a multimodal hand motion state evaluation model is deployed in the host computer; the host computer is used to preprocess the received multimodal motion signal, and input the preprocessed multimodal motion signal into the multimodal hand motion state evaluation model, output a hand motion state score prediction value, and perform hand motion function rehabilitation state evaluation based on the hand motion state score prediction value; the multimodal hand motion state evaluation model is obtained by training a neural network model using training samples; the training samples include multimodal motion signal sample data and corresponding hand motion state score sample values.

[0010] Optionally, the signal acquisition device includes: an EEG cap, a wireless myoelectric sensor and a wearable data glove;

[0011] The EEG cap is connected to the host computer via the communication device, and is used to collect the patient's EEG signals and send the EEG signals to the host computer;

[0012] The wireless myoelectric sensor is connected to the host computer through the communication device, and is used to collect the myoelectric signals of the forearm muscle group of the patient's hand, and send the myoelectric signals to the host computer;

[0013] The wearable data glove is connected to the host computer through the communication device, and is used to capture the patient's hand motion signals in three-dimensional space and send the hand motion signals to the host computer; the hand motion signals include hand motion trajectory information, hand posture information and hand pressure signals.

[0014] Optionally, the host computer includes:

[0015] An EEG signal preprocessing unit is used to perform bandpass filtering, ICA artifact removal and downsampling processing on the EEG signal in sequence, and intercept the downsampled EEG signal of a preset window length to obtain a preprocessed EEG signal;

[0016] The electromyographic signal preprocessing unit is used to perform high-pass filtering, low-pass filtering and downsampling processing on the electromyographic signal in sequence, and intercept the downsampled electromyographic signal of a preset window length to obtain a preprocessed electromyographic signal;

[0017] The hand motion signal preprocessing unit is used to adopt a Kalman filter algorithm to calculate the IMU linear acceleration based on the hand motion signal, and to form a hand motion signal time series based on the IMU linear acceleration and three-axis angular velocity, and to intercept the hand motion signal time series of a preset window length to obtain a preprocessed hand motion signal.

[0018] Optionally, the multimodal hand motion state assessment model comprises a first embedding layer, a second embedding layer, a third embedding layer, a cross attention mechanism layer and a regression network; the first embedding layer, the second embedding layer and the third embedding layer are respectively connected to the input end of the cross attention mechanism layer, and the output end of the cross attention mechanism layer is connected to the regression network;

[0019] The first embedding layer is used to extract features from the preprocessed EEG signal to obtain low-dimensional features of the EEG signal;

[0020] The second embedding layer is used to extract features from the preprocessed electromyographic signal to obtain low-dimensional features of the electromyographic signal;

[0021] The third embedding layer is used to extract features of the preprocessed hand motion signal to obtain low-dimensional features of the hand motion signal;

[0022] The cross attention mechanism layer is used to fuse the low-dimensional features of the EEG signal, the low-dimensional features of the EMG signal and the low-dimensional features of the hand movement signal to obtain a fused feature;

[0023] The regression network is used to output a hand movement state score prediction value according to the fusion feature.

[0024] Optionally, the regression network includes an input layer, a hidden layer and an output layer connected in sequence; the hidden layer includes a plurality of fully connected layers;

[0025] The input layer is used to receive the fusion feature;

[0026] The hidden layer is used to obtain nonlinear features based on the fused features;

[0027] The output layer is used to predict the hand movement state score based on the nonlinear features and output the hand movement state score prediction value.

[0028] In a second aspect, the present application provides a method for assessing the state of hand motor function rehabilitation after stroke, the method for assessing the state of hand motor function rehabilitation after stroke is applied to the system for assessing the state of hand motor function rehabilitation after stroke, and the method for assessing the state of hand motor function rehabilitation after stroke comprises:

[0029] Collecting multimodal motion signals of stroke patients who have completed hand rehabilitation training; the multimodal motion signals include electroencephalogram signals, electromyography signals and hand movement signals;

[0030] The multimodal motion signal is preprocessed, and the preprocessed multimodal motion signal is input into a multimodal hand motion state assessment model, a hand motion state score prediction value is output, and a hand motor function rehabilitation state assessment is performed based on the hand motion state score prediction value; the multimodal hand motion state assessment model is obtained by training a neural network model using training samples; the training samples include multimodal motion signal sample data and corresponding hand motion state score sample values.

[0031] In a third aspect, the present application provides a post-stroke hand motor function rehabilitation training system, the post-stroke hand motor function rehabilitation training system comprising the post-stroke hand motor function rehabilitation status assessment system and a virtual reality device;

[0032] The virtual reality device is worn on the head of a stroke patient and is connected to the host computer and the signal acquisition device respectively through the communication device, and the host computer is deployed with a virtual rehabilitation screen and a preset rehabilitation task; the virtual reality device is used to display the virtual rehabilitation screen and the preset rehabilitation task to the stroke patient when the received hand movement state score is lower than a preset threshold, and the stroke patient performs the preset rehabilitation task to complete the corresponding hand rehabilitation training.

[0033] Optionally, the host computer is further used to score the rehabilitation movements of the stroke patient performing the preset rehabilitation task according to the multimodal motion feedback signal collected when the stroke patient performs the preset rehabilitation task, to obtain a scoring result;

[0034] The virtual reality device is also used to display the scoring results to the stroke patient.

[0035] Optionally, the virtual reality device is a virtual reality head-mounted display.

[0036] According to the specific embodiments provided in this application, this application discloses the following technical effects:

[0037] The present application provides a system, method and training system for assessing the rehabilitation status of hand motor function after stroke. Multimodal sensors are used to collect multimodal motion signals of the hands of stroke patients who have completed hand rehabilitation training. The acquired signals are richer, thereby achieving a more refined assessment of the rehabilitation status of hand motor function and enhancing the objectivity and accuracy of the assessment. The hand motor status of stroke patients is scored based on a multimodal hand motor status assessment model to achieve effective multimodal information fusion, obtain reliable hand motor function rehabilitation status assessment results, and improve the accuracy of hand motor function rehabilitation status assessment. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0039] Figure 1 A schematic diagram of functional modules of a post-stroke hand motor function rehabilitation status assessment system and a rehabilitation training system provided in one embodiment of the present application;

[0040] Figure 2 A schematic diagram of functional modules of a multimodal hand motion state assessment model provided in one embodiment of the present application;

[0041] Figure 3 A schematic flow chart of a method for assessing the state of hand motor function rehabilitation after stroke provided in another embodiment of the present application;

[0042] Figure 4 A schematic diagram of the working principle of a rehabilitation training system provided in one embodiment of the present application.

[0043] Reference numerals:

[0044] Signal acquisition equipment-1, EEG cap-11, wireless electromyography sensor-12, wearable data gloves-13, communication equipment-2, host computer-3, virtual reality equipment-4. DETAILED DESCRIPTION

[0045] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0046] In order to make the above-mentioned objects, features and advantages of the present application more obvious and easy to understand, the present application is further described in detail below in conjunction with the accompanying drawings and specific implementation methods.

[0047] In an exemplary embodiment, Figure 1 As shown, a post-stroke hand motor function rehabilitation status assessment system is provided, comprising: a signal acquisition device 1, a communication device 2 and a host computer 3.

[0048] The signal acquisition device 1 is worn on a stroke patient, and is used to acquire multimodal motion signals of the stroke patient who has completed hand rehabilitation training; the multimodal motion signals include electroencephalogram signals, electromyography signals and hand motion signals.

[0049] As an optional implementation, the signal acquisition device 1 includes: an EEG cap 11 , a wireless electromyography sensor 12 and a wearable data glove 13 .

[0050] The EEG cap 11 is connected to the host computer 3 via the communication device 2 , and is used to collect the patient's EEG signals and send the EEG signals to the host computer 3 .

[0051] The wireless electromyographic sensor 12 is connected to the host computer 3 via the communication device 2 , and is used to collect electromyographic signals of the forearm muscle groups of the patient's hand, and send the electromyographic signals to the host computer 3 .

[0052] The wearable data glove 13 is connected to the host computer 3 via the communication device 2, and is used to capture the patient's hand motion signal in three-dimensional space and send the hand motion signal to the host computer 3; the hand motion signal includes hand motion trajectory information and hand posture information.

[0053] Specifically, the signal acquisition device 1, the communication device 2 and the host computer 3 constitute a rehabilitation status assessment system, which aims to comprehensively assess the patient's hand motor function rehabilitation status. The rehabilitation status assessment system utilizes a variety of sensor technologies, including a wearable dry electrode EEG cap 11, a wireless electromyography sensor 12 and a wearable data glove 13, to collect the patient's biological signals and motion data.

[0054] The wearable dry electrode EEG cap 11 is used to obtain EEG (Electroencephalogram, EEG) signals. EEG signals are a method of measuring and recording brain activity, especially the electrophysiological activity of neurons in the cerebral cortex; the EEG cap 11 can non-invasively record the patient's EEG signals and provide information about brain activity, especially EEG signals in the motor cortex area of ​​the brain. The parameters of a single device are 64 channels and 2000HZ sampling rate.

[0055] Use wireless EMG sensor 12 (EMG armband) to obtain the EMG (electromyography, EMG) signal of the forearm muscle group. EMG signal is a bioelectric signal that detects the muscle in the active state, which is used to evaluate the state of muscle and nerve function. The movement of the hand is controlled by the forearm flexor and extensor muscles. The wireless EMG sensor 12 is used to measure the patient's EMG signal. The system can monitor the EMG activity of the patient's hand forearm muscle group in real time. The parameters of a single device are 4 leads, a sampling rate of more than 2000HZ, an AD conversion bit of 24bit, and wireless communication function.

[0056] The wearable data glove 13 is used to obtain the motion signal of the hand inertial measurement unit (IMU). IMU is an integrated sensor used to measure and report the orientation, position and speed of an object, usually including sensors such as accelerometers and gyroscopes. Using IMU technology, the motion trajectory and posture of the hand in three-dimensional space can be captured to provide more detailed hand movement data. The IMU is attached to the glove and is located at the middle finger joint when worn; based on piezoelectric tactile feedback, since the fingertips are the most sensitive part of the human hand, the tactile sensor is located at the distal finger joint (fingertips) of the hand, and tactile feedback is achieved by applying pressure to the finger pads. The parameters of a single device are: 6 IMU sensors, accelerometer range ±16g, angular velocity range ±2000dps, magnetometer ±2Gauss, data sampling rate 200HZ; 5 tactile sensors, the tactile sensors use piezoelectric technology and have wireless communication functions. The stroke patient performs a clinical assessment task, obtains a multimodal motion signal according to the signal acquisition device 1, applies a multimodal fusion algorithm, and generates a fine-grained score of the hand movement state through regression fitting.

[0057] When the patient performs clinical assessment tasks on the hand, the system collects the patient's multimodal motion signals, including EEG signals, EMG signals and hand motion signals, through multimodal sensors (i.e., signal acquisition devices), filters and reduces noise on the collected time series signals, and divides the data in the time window corresponding to the execution of the task according to the changes in hand movement posture.

[0058] As an optional implementation, the host computer 3 includes: an electroencephalogram signal preprocessing unit, an electromyography signal preprocessing unit and a hand movement signal preprocessing unit.

[0059] The EEG signal preprocessing unit is used to perform bandpass filtering, ICA artifact removal and downsampling on the EEG signal in sequence, and intercept the downsampled EEG signal of a preset window length to obtain the preprocessed EEG signal. The EEG signal is bandpass filtered at [0.5, 100] Hz, ICA artifacts are removed, the signal is downsampled to 100 Hz, and a 3-second window data is selected.

[0060] The electromyographic signal preprocessing unit is used to perform high-pass filtering, low-pass filtering and downsampling processing on the electromyographic signal in sequence, and intercept the downsampled electromyographic signal of a preset window length to obtain a preprocessed electromyographic signal. The electromyographic signal is processed using a 3rd-order Butterworth high-pass filter (20Hz) and a low-pass filter (500Hz), and then the signal is downsampled to 500HZ, and a 3-second window data is selected.

[0061] The hand motion signal preprocessing unit is used to use the Kalman filter algorithm to calculate the IMU linear acceleration based on the hand motion signal, and to form a hand motion signal time series based on the IMU linear acceleration and the three-axis angular velocity, and to intercept the hand motion signal time series of a preset window length to obtain the preprocessed hand motion signal. Specifically, the Kalman filter algorithm is first used to calculate the IMU attitude and orientation information from the acceleration, angular velocity, and magnetometer data. The specific steps are as follows:

[0062] 1) Initialization:

[0063] Define the state vector X = [q, b], where q is the attitude quaternion and b is the gyroscope bias. Initialize the state vector x 0 and the covariance matrix P 0 , define the process noise covariance matrix Q and the measurement noise covariance matrix R.

[0064] 2) Prediction:

[0065] Read the angular velocity data ω and update the attitude quaternion q and covariance matrix P:

[0066]

[0067] P k∣k-1 =FP k-1 F T +Q.

[0068] Among them, q k+1 represents the attitude quaternion at time k+1; Δt represents the time interval; ω k represents the noise matrix at time k; b k represents the systematic error in the gyroscope measurement at time k; P k∣k-1 represents the covariance matrix at time k-1; F represents the state transfer matrix; F T Represents the transpose of the state transition matrix.

[0069] 3) Update:

[0070] Read the acceleration and magnetometer data, record it as Z, and calculate the Kalman gain:

[0071] K k=P k∣k-1 H T (HP k∣k-1 H T +R) -1 .

[0072] Among them, K k represents the Kalman gain at time k, H represents the measurement matrix; H T represents the transpose of the measurement matrix; R represents the measurement noise covariance matrix.

[0073] Update the state vector x k and the covariance matrix P k :

[0074] x k =x k∣k-1 +K k (z k -h(x k∣k-1 )).

[0075] P k =(IK k H)P k∣k-1 .

[0076] Among them, z k represents the acceleration and magnetometer data at time k; h(x) represents the measurement function in the Kalman filter, which maps the state variables to the measurement space and represents the measurement value that should be observed in the current state; I represents the unit matrix.

[0077] Use the orientation attitude quaternion q calculated by Kalman filtering to calculate the linear acceleration of the IMU after removing the gravity acceleration. The steps are as follows:

[0078] 1) Express IMU acceleration as quaternion Use attitude quaternion to realize acceleration coordinate system rotation:

[0079]

[0080] in, Represents the acceleration quaternion in the world coordinate system, q * Represents the conjugate of the attitude quaternion.

[0081] Then subtract the acceleration due to gravity:

[0082]

[0083] in, The acceleration in the world coordinate system without gravity.

[0084] Finally, the linear acceleration is

[0085]

[0086] The linear acceleration modulus is calculated using the three-axis acceleration data. The linear acceleration modulus and the three-axis angular velocity form a 4-channel time series, and a 3-second window of data is selected.

[0087] The host computer 3 is connected to the signal acquisition device 1 through the communication device 2, and a multimodal hand motion state evaluation model is deployed in the host computer 3; the host computer 3 is used to preprocess the received multimodal motion signal, and input the preprocessed multimodal motion signal into the multimodal hand motion state evaluation model, output the hand motion state score prediction value, and perform hand motion function rehabilitation state evaluation according to the hand motion state score prediction value; the multimodal hand motion state evaluation model is obtained by training a neural network model with training samples; the training samples include multimodal motion signal sample data and corresponding hand motion state score sample values.

[0088] As an optional implementation, Figure 2 As shown, the multimodal hand motion state evaluation model includes a first embedding layer, a second embedding layer, a third embedding layer, a cross-attention mechanism layer and a regression network; the first embedding layer, the second embedding layer, and the third embedding layer are respectively connected to the input end of the cross-attention mechanism layer, and the output end of the cross-attention mechanism layer is connected to the regression network.

[0089] The first embedding layer is used to perform feature extraction on the preprocessed EEG signal to obtain low-dimensional features of the EEG signal; the second embedding layer is used to perform feature extraction on the preprocessed EMG signal to obtain low-dimensional features of the EMG signal; the third embedding layer is used to perform feature extraction on the preprocessed hand movement signal to obtain low-dimensional features of the hand movement signal; the cross-attention mechanism layer is used to fuse the low-dimensional features of the EEG signal, the low-dimensional features of the EMG signal and the low-dimensional features of the hand movement signal to obtain fused features; the regression network is used to output a predicted value of the hand movement state score based on the fused features.

[0090] Specifically, the embedding layer is used to convert the high-dimensional original sequence into a low-dimensional feature with consistent shape. Since the data is of three modes, a separate encoder is used for each mode, so the first embedding layer, the second embedding layer and the third embedding layer are set; each encoder is set to extract the features of the modal data. For multivariate time series data, 1d-CNN is generally used as the encoder, and its embedding layer is used to extract data features. The final output embedding has a uniform shape - m×n.

[0091] The cross-attention layer is used to process the semantic relationship between two sequences by calculating attention on two different sequences, so as to better integrate multimodal information. In multimodal crossattention, the representation (or feature) of each modality is used to generate queries, keys, and values. In the standard Transformer model, the calculation of cross attention can be expressed as:

[0092] Given three modalities A, B, C, each modality has a corresponding query Q A , Q B , Q C , key K A , K B , K C Sum value V A 、V B 、V C . The calculation of the cross attention value of modality A can be expressed as:

[0093]

[0094] Among them, d k is the dimension of the key vector, used as a scaling factor to avoid gradient vanishing or exploding problems caused by large dimensions.

[0095] For the three modalities, we need to calculate the cross-attention values ​​of modalities A, B, and C. The representation of modal A is:

[0096] O A =Attention(Q A ,K B ,V B )+Attention(Q A ,K C ,V C )

[0097] Similarly, the outputs of modal B and C are calculated, and finally the fusion feature H = (O A ,O B ,O C ), which is used in the subsequent regression network to determine the predicted value of the hand movement state score.

[0098] As an optional implementation, the regression network includes an input layer, a hidden layer and an output layer connected in sequence; the hidden layer includes a plurality of fully connected layers;

[0099] The input layer is used to receive the fused features; the hidden layer is used to obtain nonlinear features based on the fused features; the output layer is used to predict the hand movement state score based on the nonlinear features and output the hand movement state score prediction value.

[0100] Specifically, the fusion feature H is regressed through a regression network to calculate the hand movement state score, which is generally composed of a multi-layer fully connected neural network. Among them, the input layer is used to receive the input of feature data; the hidden layer is composed of multiple fully connected layers, each layer contains a number of neurons, and the fully connected layers are connected through a weight matrix, and the output of each neuron is transformed by a nonlinear activation function (such as ReLU, Sigmoid or Tanh) to introduce nonlinear characteristics; the last layer of the output layer is a linear layer, which contains only one neuron, and the output of this neuron is used as a regression score to predict the value of the hand movement state score.

[0101] Based on the same inventive concept, the embodiment of the present application also provides a method for implementing the above-mentioned method for evaluating the rehabilitation status of hand motor function after stroke. The implementation scheme for solving the problem provided by the method is similar to the implementation scheme recorded in the above-mentioned system, so the specific limitations in one or more embodiments of the method for evaluating the rehabilitation status of hand motor function after stroke provided below can be referred to the limitations of the system for evaluating the rehabilitation status of hand motor function after stroke above, and will not be repeated here.

[0102] In an exemplary embodiment, Figure 3 As shown, a method for assessing the rehabilitation status of hand motor function after stroke is provided. The method is executed by a computer device, and can be executed by a computer device such as a terminal or a server alone, or by a terminal and a server together. In the embodiment of the present application, the method is applied to a server as an example for explanation, and includes the following steps S1 to S2. Among them:

[0103] Step S1, collecting multimodal motion signals of a stroke patient who has completed hand rehabilitation training; the multimodal motion signals include electroencephalogram signals, electromyography signals and hand movement signals.

[0104] Step S2, preprocessing the multimodal motion signal, and inputting the preprocessed multimodal motion signal into a multimodal hand motion state evaluation model, outputting a hand motion state score prediction value, and performing hand motion function evaluation based on the hand motion state score prediction value; the multimodal hand motion state evaluation model is obtained by training a neural network model using training samples; the training samples include multimodal motion signal sample data and corresponding hand motion state score sample values.

[0105] In an exemplary embodiment, Figure 1As shown, a post-stroke hand motor function rehabilitation training system is provided, including the post-stroke hand motor function rehabilitation status evaluation system and a virtual reality device 4.

[0106] The virtual reality (VR) device 4 is worn on the head of a stroke patient, and is connected to the host computer 3 and the signal acquisition device 1 respectively through the communication device 2, and a virtual rehabilitation screen and a preset rehabilitation task are deployed in the host computer 3; the virtual reality device 4 is used to display the virtual rehabilitation screen and the preset rehabilitation task to the stroke patient when the received hand movement state score is lower than a preset threshold, and the stroke patient performs the preset rehabilitation task to complete the corresponding hand rehabilitation training.

[0107] As an optional implementation, the host computer 3 is also used to score the rehabilitation movements of the stroke patient according to the multimodal motion feedback signal generated when the stroke patient performs the preset rehabilitation task to obtain a scoring result; the virtual reality device 4 is also used to display the scoring result to the stroke patient.

[0108] Specifically, the virtual reality device 4 is a virtual reality head mounted display. The rehabilitation training system includes a virtual reality head mounted display, a virtual rehabilitation screen, and uses a motor imagery detection algorithm, combined with virtual hand movement and tactile feedback mechanism to complete hand rehabilitation training for stroke patients. The system provides a variety of rehabilitation methods. Figure 4 As shown in the figure, the rehabilitation training system mainly involves the following aspects:

[0109] 1) Virtual rehabilitation screen (i.e., VR environment presented in a virtual reality head-mounted display): provides a virtual hand environment that simulates the real world. The patient enters the environment through a virtual reality head-mounted display and can observe the virtual hand. The palm IMU of the data glove realizes the posture and gesture control of the virtual hand. All calculations are completed in the host computer 3 and output on the virtual reality head-mounted display.

[0110] 2) Multimodal motor imagery detection: A wearable dry electrode EEG cap 11 is used to detect the patient's motor imagery signals and provide feedback for rehabilitation training.

[0111] 3) EMG threshold detection: Based on the static and dynamic detection results of the forearm muscle group EMG, various algorithms (such as mean, root mean square, percentile, etc.) are used to determine the signal threshold, and the signal exceeding the threshold is regarded as a valid movement signal, and the EMG intensity is determined. The details are as follows:

[0112] The root mean square (RMS) method is used to extract the characteristic value of the electromyographic signal as an indicator of muscle strength under different states. The calculation formula is:

[0113]

[0114] Where M represents the RMS value of muscle strength, x i Represents the electromyographic signal value at the sampling moment, n represents the number of sampling points within a period of time calculated according to the sampling rate, i represents the i-th sampling point, and the value ranges from 0 to n.

[0115] Calculate the relaxed muscle strength and maximum muscle strength of the patient's healthy side and affected side to obtain the relaxed comprehensive muscle strength M relax and maximum comprehensive muscle strength M max .

[0116] In rehabilitation training, the RMS value of the electromyographic signal is used as feedback and as a performance score of the training action in the virtual environment. The score formula is:

[0117]

[0118] Among them, x is the muscle strength value of the affected side in the current time window, the a in the upper right corner of M represents the affected side, and h represents the healthy side.

[0119] 4) Tactile feedback: Based on specific signals, the tactile sensor realizes tactile feedback of the hand through pressure.

[0120] 5) Preset rehabilitation tasks: Patients can choose a variety of rehabilitation tasks for hand rehabilitation training. The rehabilitation tasks are mainly presented in the form of game interaction. Among them, two rehabilitation tasks are listed as follows:

[0121] Preset rehabilitation task 1:

[0122] 1. System Settings

[0123] 1.1 Wearing the device

[0124] The stroke patient wore a VR head mounted display, a wearable data glove13 on his hand, and a wireless electromyography sensor12 on his forearm.

[0125] 1.2 Mode Setting

[0126] In the current mode, the virtual hand is not displayed, but training is carried out in augmented reality (AR).

[0127] 2. Training process

[0128] 2.1 Object Display and Movement

[0129] A moving object appears on the VR head-mounted display screen, and when the object moves to the virtual hand, the patient needs to perform a grasping action.

[0130] 2.2 Data Collection

[0131] The patient's hand motion signals are collected through the wearable data glove 13, and the myoelectric signals of the forearm are collected through the wireless myoelectric sensor 12.

[0132] 2.3 Posture Calculation and Grasp Detection

[0133] The host computer 3 calculates the hand posture information according to the hand motion signal collected by the wearable data glove 13, and detects the grasping action.

[0134] 2.4 Grip score

[0135] Grasp success detection: If the grasping action is successful, the system calculates the score.

[0136] EMG data analysis: The system determines the score based on the EMG level. A higher EMG amplitude indicates that the patient is in a more focused training state.

[0137] 2.5 Feedback Mechanism

[0138] The system score provides on-screen feedback, which includes the score, and the pressure level of the tactile sensor is determined according to the score to encourage the patient to continue the game training. The pressure level is controlled as follows:

[0139]

[0140] Among them, V max is the maximum driving voltage that the sensor can withstand; V min is the minimum voltage; P is the percentage of the control amplitude, V(P) is the drive voltage that should be applied in percentage.

[0141] Preset rehabilitation task 2:

[0142] 1. System Settings

[0143] 1.1 Wearing the device

[0144] The patient wears a VR head mounted display, an EEG cap 11 , a wearable data glove 13 , and a forearm wireless EMG sensor 12 .

[0145] 1.2 Mode Setting

[0146] This mode is based on motor imagery (MI) for rehabilitation training. A virtual hand holding an object is displayed on the screen of the VR head-mounted display.

[0147] 2. Training process

[0148] 2.1 Motor imagery instructions

[0149] Stroke patients perform grasping movements using motor imagery.

[0150] 2.2 Signal Detection and Acquisition

[0151] EEG signal detection: The EEG system collects EEG signals in real time and detects motor imagery signals based on the EEG signals.

[0152] 2.3 Virtual Hand Control and Haptic Feedback

[0153] Motor imagery signal recognition: When the EEG system detects a valid motor imagery signal, the virtual hand starts to perform a grasping action.

[0154] Haptic feedback: The tactile sensor begins to provide corresponding tactile feedback, simulating the feeling of real grasping.

[0155] 2.4 Grip score

[0156] Successful detection: Each successful grasping movement imagination is counted as an effective grasp. A successful grasping movement imagination can score points. The score is determined according to the electromyographic intensity and is accumulated.

[0157] 2.5 Feedback Mechanism

[0158] Real-time visual feedback (i.e., screen feedback) is provided based on the score, and the deformation of the virtual object is displayed at the same time. The success rate of five consecutive motor imagery tests controls the pressure level of the tactile sensor to motivate the patient to continue training.

[0159] Beneficial effects of this application:

[0160] 1. Use multimodal sensors (i.e., signal acquisition devices) to obtain more abundant data on the patient's hand motion status, thereby achieving a more refined evaluation. The collected multimodal motion signals also enhance the objectivity of the evaluation.

[0161] 2. Use a multimodal hand movement state assessment model based on cross attention. This model can fuse biological signals of multiple modalities, process cross-modal data, achieve effective multimodal signal fusion, obtain reliable hand movement function rehabilitation state assessment results, and improve the accuracy of hand movement function rehabilitation state assessment.

[0162] 3. Use virtual reality-based methods to design rehabilitation training tasks and implement multiple rehabilitation theories through a variety of wearable sensors. This not only promotes neuroplasticity in multiple dimensions, but also interesting rehabilitation games greatly stimulate patients' interest in rehabilitation training.

[0163] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant regulations.

[0164] Those of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM may be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM).

[0165] The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. The non-relational database may include a distributed database based on blockchain, etc., but is not limited thereto. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., but is not limited thereto.

[0166] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0167] This article uses specific examples to illustrate the principles and implementation methods of this application. The description of the above embodiments is only used to help understand the method and core ideas of this application. At the same time, for those skilled in the art, according to the ideas of this application, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as limiting this application.

Claims

1. A post-stroke hand motor function rehabilitation status assessment system, characterized in that: The post-stroke hand motor function rehabilitation status assessment system comprises: a signal acquisition device, a communication device and a host computer; The signal acquisition device is worn on a stroke patient, and is used to acquire multimodal motion signals of the stroke patient who has completed hand rehabilitation training; the multimodal motion signals include electroencephalogram signals, electromyography signals, and hand motion signals; The host computer is connected to the signal acquisition device through the communication device, and a multimodal hand motion state evaluation model is deployed in the host computer; the host computer is used to preprocess the received multimodal motion signal, and input the preprocessed multimodal motion signal into the multimodal hand motion state evaluation model, output a hand motion state score prediction value, and perform hand motion function rehabilitation state evaluation based on the hand motion state score prediction value; the multimodal hand motion state evaluation model is obtained by training a neural network model using training samples; the training samples include multimodal motion signal sample data and corresponding hand motion state score sample values.

2. The post-stroke hand motor function rehabilitation status assessment system according to claim 1, characterized in that: The signal acquisition equipment includes: an EEG cap, a wireless myoelectric sensor and a wearable data glove; The EEG cap is connected to the host computer via the communication device, and is used to collect the patient's EEG signals and send the EEG signals to the host computer; The wireless myoelectric sensor is connected to the host computer through the communication device, and is used to collect the myoelectric signals of the forearm muscle group of the patient's hand, and send the myoelectric signals to the host computer; The wearable data glove is connected to the host computer through the communication device, and is used to capture the patient's hand motion signals in three-dimensional space and send the hand motion signals to the host computer; the hand motion signals include hand motion trajectory information, hand posture information and hand pressure signals.

3. The post-stroke hand motor function rehabilitation status assessment system according to claim 2, characterized in that: The host computer comprises: An EEG signal preprocessing unit is used to perform bandpass filtering, ICA artifact removal and downsampling processing on the EEG signal in sequence, and intercept the downsampled EEG signal of a preset window length to obtain a preprocessed EEG signal; The electromyographic signal preprocessing unit is used to perform high-pass filtering, low-pass filtering and downsampling processing on the electromyographic signal in sequence, and intercept the downsampled electromyographic signal of a preset window length to obtain a preprocessed electromyographic signal; The hand motion signal preprocessing unit is used to adopt a Kalman filter algorithm to calculate the IMU linear acceleration based on the hand motion signal, and to form a hand motion signal time series based on the IMU linear acceleration and three-axis angular velocity, and to intercept the hand motion signal time series of a preset window length to obtain a preprocessed hand motion signal.

4. The post-stroke hand motor function rehabilitation status assessment system according to claim 3, characterized in that: The multimodal hand motion state assessment model comprises a first embedding layer, a second embedding layer, a third embedding layer, a cross attention mechanism layer and a regression network; the first embedding layer, the second embedding layer and the third embedding layer are respectively connected to the input end of the cross attention mechanism layer, and the output end of the cross attention mechanism layer is connected to the regression network; The first embedding layer is used to extract features from the preprocessed EEG signal to obtain low-dimensional features of the EEG signal; The second embedding layer is used to extract features from the preprocessed electromyographic signal to obtain low-dimensional features of the electromyographic signal; The third embedding layer is used to extract features of the preprocessed hand motion signal to obtain low-dimensional features of the hand motion signal; The cross attention mechanism layer is used to fuse the low-dimensional features of the EEG signal, the low-dimensional features of the EMG signal and the low-dimensional features of the hand movement signal to obtain a fused feature; The regression network is used to output a hand movement state score prediction value according to the fusion feature.

5. The post-stroke hand motor function rehabilitation status assessment system according to claim 4, characterized in that: The regression network includes an input layer, a hidden layer and an output layer connected in sequence; the hidden layer includes a plurality of fully connected layers; The input layer is used to receive the fusion feature; The hidden layer is used to obtain nonlinear features based on the fused features; The output layer is used to predict the hand movement state score based on the nonlinear features and output the hand movement state score prediction value.

6. A method for assessing the rehabilitation status of hand motor function after stroke, characterized in that: The method for assessing the state of hand motor function rehabilitation after stroke is applied to the system for assessing the state of hand motor function rehabilitation after stroke according to claims 1 to 5, and the method for assessing the state of hand motor function rehabilitation after stroke comprises: Collecting multimodal motion signals of stroke patients who have completed hand rehabilitation training; the multimodal motion signals include electroencephalogram signals, electromyography signals and hand movement signals; The multimodal motion signal is preprocessed, and the preprocessed multimodal motion signal is input into a multimodal hand motion state assessment model, a hand motion state score prediction value is output, and a hand motor function rehabilitation state assessment is performed based on the hand motion state score prediction value; the multimodal hand motion state assessment model is obtained by training a neural network model using training samples; the training samples include multimodal motion signal sample data and corresponding hand motion state score sample values.

7. A post-stroke hand motor function rehabilitation training system, comprising the post-stroke hand motor function rehabilitation state assessment system according to any one of claims 1 to 5 and a virtual reality device; The virtual reality device is worn on the head of a stroke patient and is connected to the host computer and the signal acquisition device respectively through the communication device, and the host computer is deployed with a virtual rehabilitation screen and a preset rehabilitation task; the virtual reality device is used to display the virtual rehabilitation screen and the preset rehabilitation task to the stroke patient when the received hand movement state score is lower than a preset threshold, and the stroke patient performs the preset rehabilitation task to complete the corresponding hand rehabilitation training.

8. The post-stroke hand motor function rehabilitation training system according to claim 7, characterized in that: The host computer is also used to score the rehabilitation movements of the stroke patient performing the preset rehabilitation task according to the multimodal motion feedback signal collected when the stroke patient performs the preset rehabilitation task, and obtain a scoring result; The virtual reality device is also used to display the scoring results to the stroke patient.

9. The post-stroke hand motor function rehabilitation training system according to claim 7, characterized in that: The virtual reality device is a virtual reality head-mounted display.

Images