Multi-modal information-based motion function rehabilitation state detection method

Abstract

The invention relates to a motion function rehabilitation state detection method based on multi-modal information, and the method comprises the steps: collecting a wrist electromyographic signal when a target person executes an action, carrying out the preprocessing and dimension reduction of the electromyographic signal, and extracting the features of the electromyographic signal; collecting a video when the target person executes the action, and performing preprocessing and recognition based on the collected video to obtain an action classification result as an action recognition feature; detection data of a wearable dynamic sensor worn on the body of the target person when the target person executes the action are collected, and kinematic characteristics are extracted and obtained, and the kinematic characteristics comprise acceleration, angular velocity and force information; splicing the electromyographic signal features, the action recognition features and the kinematics features under the same time window to obtain a detection feature vector, and calculating the Euclidean distance between the detection feature vector and the standard feature vector to obtain a rehabilitation state detection result. Compared with the prior art, the method has the advantages of high detection accuracy and the like.

Description

Technical Field

[0001] The present invention relates to the field of motor function rehabilitation status detection based on artificial intelligence, and in particular to a motor function rehabilitation status detection method based on multimodal information. Background Art

[0002] Rehabilitation assessment is the core link in rehabilitation treatment. By systematically collecting and analyzing various data generated by patients during the rehabilitation process, it helps doctors, therapists and patients understand the rehabilitation effect, judge whether the current rehabilitation plan is effective, and provide guidance for subsequent treatment plans. However, traditional rehabilitation assessment programs usually rely on scales and subjective scores. Therapists or patients fill out the assessment form themselves, and assess the progress and effect of rehabilitation through observation and self-reporting. Although scale assessment simplifies the assessment process to a certain extent, this method also has some disadvantages that cannot be ignored: it is highly subjective, easily affected by bias, and highly dependent on rehabilitation physicians; the amount of information is limited, and there is a lack of quantitative analysis and long-term tracking.

[0003] With the continuous development of rehabilitation medicine, intelligent rehabilitation equipment has gradually become an important auxiliary tool for clinical rehabilitation training. The evaluation module is the core component of intelligent rehabilitation equipment to improve rehabilitation efficiency. Compared with traditional scale evaluation, intelligent rehabilitation equipment can collect various physiological and motion data detected during rehabilitation training, and visualize the data to help doctors efficiently analyze the patient's motor function improvement, further optimize and adjust the treatment plan, and achieve long-term data comparison and tracking.

[0004] At present, the main evaluation technology uses dynamic data analysis to describe the rehabilitation training process by collecting the patient's kinematic parameters (such as joint angle, torque and displacement speed). However, its disadvantages are that it cannot reflect the functional state of the neuromuscular system, it is difficult to capture the true state of the patient's internal neuromuscular system, and it cannot identify the movement intention, so it is difficult to evaluate whether the patient has autonomous control ability or whether there is abnormal movement pattern; and it is difficult to capture small movement changes by only collecting dynamic data, and it is not sensitive enough to identify fine movements of the hand and wrist and coordination problems.

[0005] In addition, some intelligent rehabilitation equipment also collects electrophysiological signals during muscle activity through electromyographic signals to reflect the patient's muscle activation level and neural control ability. The disadvantage of this method is that electromyographic signals are very sensitive to noise, the signals are easily interfered with, and it is difficult to accurately evaluate the patient's overall motor function based solely on electromyographic signals.

[0006] Other intelligent devices use visual data to conduct rehabilitation assessments. Through cameras or depth sensors, they can capture the patient's movement trajectory and limb posture in real time and perform three-dimensional motion analysis. This solution is highly dependent on the environment and is affected by factors such as ambient light and camera viewing angle. It also cannot provide information about the patient's internal muscle and nerve activity, and the assessment dimension is limited.

[0007] The above three solutions all analyze the rehabilitation situation through the collection of a single data modality, which leads to a significant disadvantage of low credibility of the evaluation results. Therefore, the evaluation modules of many intelligent rehabilitation equipment fail to achieve the expected design effects, and it is difficult to effectively improve medical efficiency when they are put into use in hospitals. Summary of the invention

[0008] The purpose of this invention is to provide a method for detecting the rehabilitation status of exercise function based on multimodal information, combining surface electromyographic signals (sEMG), visual signals and dynamic information to form a multimodal evaluation method. This innovation breaks through the limitations of traditional single-modality evaluation and can fully reflect the relationship between the patient's external movement performance and internal neuromuscular activity, thereby achieving more accurate rehabilitation status detection.

[0009] The purpose of the present invention can be achieved by the following technical solutions:

[0010] A method for detecting the rehabilitation state of exercise function based on multimodal information, comprising:

[0011] Collect wrist electromyographic signals when the target person performs an action, and extract electromyographic signal features after preprocessing and dimensionality reduction processing on the electromyographic signals, wherein the electromyographic signal features include waveform length, number of zero crossing points, electromyographic integral value, variance and root mean square;

[0012] Collect videos of the target person performing actions, and obtain action classification results based on the pre-processed videos as action recognition features;

[0013] Collecting detection data of a wearable dynamic sensor worn on the target person when the target person performs an action, and extracting kinematic features, wherein the kinematic features include acceleration, angular velocity and force information;

[0014] The detection feature vector is obtained by splicing the electromyographic signal features, motion recognition features and kinematic features in the same time window, and the corresponding standard feature vector is found based on the physiological characteristics and motion classification results of the target person. The Euclidean distance between the detection feature vector and the standard feature vector is calculated, and the rehabilitation status detection result is obtained based on the Euclidean distance between the detection feature vector and the standard feature vector.

[0015] The preprocessing process of EMG signals includes normalization, label processing and bandpass filtering, and the wavelet analysis method is used to reduce the dimension of EMG signals.

[0016] The action classification result obtained by pre-processing the collected video as the action recognition feature includes:

[0017] Video input preprocessing and patch embedding: The captured video is segmented into patches to generate non-overlapping 3D patches. Then, the image blocks are embedded and the positions are embedded using a linear layer. Each 3D patch is mapped to a high-dimensional embedding vector through a convolution or linear layer to form a preliminary token representation. The output is in shape N×d, where N is the number of preliminary tokens and d is the embedding dimension.

[0018] Local attention mechanism: self-attention is calculated for the preliminary tokens in each window;

[0019] Action classification: Through global pooling or dimensionality reduction, self-attention is calculated based on the preliminary tokens in each window to obtain the global features of the video, and then input into the classifier to obtain the action classification results as action recognition features.

[0020] The number of preliminary tokens is:

[0021] N=(T×H×W) / (Tp×Hp×Wp)

[0022] Where: T is the duration of the captured video, H is the height of the captured video frame, W is the width of the captured video frame, Tp is the time dimension of the 3D patch, indicating the number of consecutive frames contained in each patch, Hp is the height of the 3D patch, and Wp is the width of the 3D patch.

[0023] The classifier is a linear classification head.

[0024] The wearable dynamic sensor includes at least an IMU sensing device and a force sensor.

[0025] The process of collecting detection data of the wearable dynamic sensor includes:

[0026] Use the unified network time protocol time to add the timestamp Ti when the data is generated;

[0027] Use the Network Time Protocol to ensure time synchronization between multiple devices and ensure the relative consistency of timestamps;

[0028] Incremental transmission: The sender only sends the data added since the last synchronization, and uses a queue or buffer to temporarily store the data that has not been sent to ensure timing integrity. The receiver receives and stores the data in the order of timestamps.

[0029] The receiving end uses timestamps to fuse data onto a unified timeline, and data from different sensor devices are aligned through timestamps for synchronization or joint analysis.

[0030] The physiological characteristics include height and weight, and the height and weight corresponding to the standard feature vector are similar to the height and body shape of the target person.

[0031] A device for detecting the state of rehabilitation of exercise function based on multimodal information comprises a memory, a processor, and a program stored in the memory, wherein the processor implements the above method when executing the program.

[0032] A storage medium stores a program, which implements the above method when executed.

[0033] Compared with the prior art, the present invention has the following beneficial effects:

[0034] 1. Combining surface electromyography (sEMG), visual signals and dynamic information to form a multimodal assessment method. This innovation breaks through the limitations of traditional single-modality assessment and can fully reflect the relationship between the patient's external movement performance and internal neuromuscular activity, thereby achieving more accurate rehabilitation status detection.

[0035] 2. Through electromyographic signal analysis, not only can the activation degree of the patient's muscles and neural control ability be captured, but also high-resolution description of muscle activity can be achieved by combining multi-feature extraction technology. The introduction of visual signals, with the help of Video SwinTransformer technology, can extract the global and local features of the patient's motion trajectory and accurately identify subtle movement differences. At the same time, the collection of dynamic information covers acceleration, angular velocity and force, which can comprehensively quantify the dynamic characteristics of the patient's movement.

[0036] 3. In the process of data processing, efficient synchronization and fusion of multimodal signals are achieved. Through the timestamp-based incremental synchronization mechanism, the consistency of different modal data on the time axis is ensured, thereby achieving precise alignment of signals. Finally, the action score is calculated at the decision level through feature fusion and template matching. This scoring method is more accurate than the Fugl-Meyer evaluation scale, ensuring the quantification and objectivity of the evaluation results.

[0037] 4. It effectively improves the accuracy and reliability of rehabilitation assessment. Compared with the traditional rough assessment method that relies on scales or the assessment method of single modality analysis, it significantly reduces the dependence on professional physicians and provides a scientific basis for the formulation of personalized rehabilitation treatment plans.

[0038] 5. All data collection is done through non-invasive evaluation methods to optimize the patient experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1 The figure is a schematic flow chart of the main steps of the method of the present invention. DETAILED DESCRIPTION

[0040] The present invention is described in detail below in conjunction with the accompanying drawings and specific embodiments. This embodiment is implemented based on the technical solution of the present invention, and provides a detailed implementation method and specific operation process, but the protection scope of the present invention is not limited to the following embodiments.

[0041] A method for detecting the rehabilitation status of exercise function based on multimodal information, such as Figure 1 As shown, including:

[0042] Step S1: collecting wrist electromyographic signals when the target person performs an action, and extracting electromyographic signal features after preprocessing and dimensionality reduction processing on the electromyographic signals, wherein the electromyographic signal features include waveform length, number of zero crossing points, electromyographic integral value, variance and root mean square;

[0043] In the specific wrist electromyographic signal collection process, a surface electromyographic signal, namely, sEMG signal, is collected using an electromyographic signal sensor group installed on the wrist to collect the corresponding surface electromyographic signal and extract the time domain information using a sliding window method;

[0044] The preprocessing process of EMG signals includes normalization, label processing and bandpass filtering, and the wavelet analysis method is used to reduce the dimension of EMG signals.

[0045] Specifically, the electromyographic signal is preprocessed, including: normalization and labeling, using Butterworth bandpass filtering to filter low-frequency and high-frequency noise, and using notch filter to remove noise; wavelet analysis method is used for dimensionality reduction processing to extract five time series characteristic values ​​such as waveform length, number of zero crossing points, electromyographic integral value, variance, and root mean square.

[0046] Step S2: Collecting a video of the target person performing an action, and obtaining an action classification result as an action recognition feature based on the pre-processed video.

[0047] The action classification results are obtained based on the preprocessing of the collected video as action recognition features, including:

[0048] Video input preprocessing and patch embedding: The collected video with a shape of T×H×W×C (representing time, space and channel dimensions) is patch segmented to generate non-overlapping 3D patches. The size of each 3D patch is a cubic block of Tp×Hp×Wp. Then, the image blocks are embedded and the position is embedded using a linear layer. Each 3D patch is mapped to a high-dimensional embedding vector through a convolution or linear layer to form a preliminary token representation. The output is in the shape of N×d, where N is the number of preliminary tokens and d is the embedding dimension.

[0049] The initial number of tokens is:

[0050] N=(T×H×W) / (Tp×Hp×Wp)

[0051] Where: T is the duration of the captured video, H is the height of the captured video frame, W is the width of the captured video frame, Tp is the time dimension of the 3D patch, indicating the number of consecutive frames contained in each patch, Hp is the height of the 3D patch, and Wp is the width of the 3D patch.

[0052] Local attention mechanism: self-attention is calculated for the preliminary tokens in each window;

[0053] Action classification: Through global pooling or dimensionality reduction, self-attention is calculated based on the preliminary tokens in each window to obtain the global features of the video, and the results are input into the classifier to obtain the action classification results as action recognition features. In this embodiment, the classifier is a linear classification head. For example, the classification result of an action is "lateral raise". These actions are completed under the instructions of a rehabilitation assessor.

[0054] Step S3: collecting detection data of a wearable dynamic sensor worn on the target person when the target person performs an action, and extracting kinematic features, wherein the kinematic features include acceleration, angular velocity and force information;

[0055] The wearable dynamic sensor includes at least an IMU sensing device and a force sensor.

[0056] The process of collecting detection data from wearable dynamic sensors includes:

[0057] Use the unified network time protocol time to add the timestamp Ti when the data is generated;

[0058] Use the Network Time Protocol to ensure time synchronization between multiple devices and ensure the relative consistency of timestamps;

[0059] Incremental transmission: The sender only sends the data added since the last synchronization, and uses a queue or buffer to temporarily store the data that has not been sent to ensure timing integrity. The receiver receives and stores the data in the order of timestamps.

[0060] The receiving end uses timestamps to fuse data onto a unified timeline, and data from different sensor devices are aligned through timestamps for synchronization or joint analysis.

[0061] Specifically, it includes: ① When data is generated, a timestamp Ti is added, which is usually the system clock of the device or the unified Network Time Protocol (NTP) time. ② Ensure time synchronization between multiple devices (such as using the NTP protocol) to ensure the relative consistency of timestamps. The receiver maintains the last received timestamp T_last and only receives new data with T>T_last. ③ Incremental transmission: The sender: only sends the newly added data since the last synchronization (timestamp T>Tlast). Use a queue or buffer to temporarily store data that has not been sent to ensure timing integrity. The receiver: Receive and store data in timestamp order. If the timestamp interval is abnormal (such as lost data), the data request compensation mechanism can be triggered. ④ At the receiving end, the timestamp is used to merge the data into a unified time axis, and the data from different devices are aligned through timestamps for synchronization or joint analysis. ⑤ Real-time update and feedback: The receiving end confirms the synchronization result (such as the timestamp range) and feeds it back to the sending end. If data delay or missing is found, a retransmission or compensation strategy can be triggered.

[0062] Step S4: The electromyographic signal features, motion recognition features and kinematic features in the same time window are concatenated to obtain a detection feature vector, and the corresponding standard feature vector is found based on the physiological characteristics and motion classification results of the target person, the Euclidean distance between the detection feature vector and the standard feature vector is calculated, and the rehabilitation status detection result is obtained based on the Euclidean distance between the detection feature vector and the standard feature vector.

[0063] In the splicing process, the weights of each modal data are considered equal. In addition, physiological characteristics include height and weight. The height and weight corresponding to the standard feature vector are similar to the height and body shape of the target person, that is, the data group that is closest to the height and weight of the target person in all standard action libraries in the database.

[0064] In this embodiment, the Euclidean distance between the detection feature vector and the standard feature vector is specifically mapped to the 0, 1, and 2 discrete scores of the Fugl-Meyer assessment scale, thereby completing the entire rehabilitation status detection process.

[0065] If the above functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc., which can store program code.

Claims

1. A method for detecting the rehabilitation state of exercise function based on multimodal information, characterized in that: include: Collect wrist electromyographic signals when the target person performs an action, and extract electromyographic signal features after preprocessing and dimensionality reduction processing on the electromyographic signals, wherein the electromyographic signal features include waveform length, number of zero crossing points, electromyographic integral value, variance and root mean square; Collect videos of the target person performing actions, and obtain action classification results based on the pre-processed videos as action recognition features; Collecting detection data of a wearable dynamic sensor worn on the target person when the target person performs an action, and extracting kinematic features, wherein the kinematic features include acceleration, angular velocity and force information; The detection feature vector is obtained by splicing the electromyographic signal features, motion recognition features and kinematic features in the same time window, and the corresponding standard feature vector is found based on the physiological characteristics and motion classification results of the target person. The Euclidean distance between the detection feature vector and the standard feature vector is calculated, and the rehabilitation status detection result is obtained based on the Euclidean distance between the detection feature vector and the standard feature vector.

2. The method for detecting the state of exercise function rehabilitation based on multimodal information according to claim 1, characterized in that: The preprocessing process of EMG signals includes normalization, label processing and bandpass filtering, and the wavelet analysis method is used to reduce the dimension of EMG signals.

3. The method for detecting the state of exercise function rehabilitation based on multimodal information according to claim 1, characterized in that: The action classification result obtained by pre-processing the collected video as the action recognition feature includes: Video input preprocessing and patch embedding: The captured video is segmented into patches to generate non-overlapping 3D patches. Then, the image blocks are embedded and the positions are embedded using a linear layer. Each 3D patch is mapped to a high-dimensional embedding vector through a convolution or linear layer to form a preliminary token representation. The output is in shape N×d, where N is the number of preliminary tokens and d is the embedding dimension. Local attention mechanism: self-attention is calculated for the preliminary tokens in each window; Action classification: Through global pooling or dimensionality reduction, self-attention is calculated based on the preliminary tokens in each window to obtain the global features of the video, and then input into the classifier to obtain the action classification results as action recognition features.

4. The method for detecting the state of exercise function rehabilitation based on multimodal information according to claim 3, characterized in that: The number of preliminary tokens is: N=(T×H×W) / (Tp×Hp×Wp) Where: T is the duration of the captured video, H is the height of the captured video frame, W is the width of the captured video frame, Tp is the time dimension of the 3D patch, indicating the number of consecutive frames contained in each patch, Hp is the height of the 3D patch, and Wp is the width of the 3D patch.

5. The method for detecting the state of exercise function rehabilitation based on multimodal information according to claim 3, characterized in that: The classifier is a linear classification head.

6. The method for detecting the state of exercise function rehabilitation based on multimodal information according to claim 1, characterized in that: The wearable dynamic sensor includes at least an IMU sensing device and a force sensor.

7. The method for detecting the state of exercise function rehabilitation based on multimodal information according to claim 6, characterized in that: The process of collecting detection data of the wearable dynamic sensor includes: Use the unified network time protocol time to add a timestamp Ti when the data is generated; Use the Network Time Protocol to ensure time synchronization between multiple devices and ensure the relative consistency of timestamps; Incremental transmission: The sender only sends the data added since the last synchronization, and uses a queue or buffer to temporarily store the data that has not been sent to ensure timing integrity. The receiver receives and stores the data in the order of timestamps. The receiving end uses timestamps to fuse data onto a unified timeline, and data from different sensor devices are aligned through timestamps for synchronization or joint analysis.

8. The method for detecting the state of exercise function rehabilitation based on multimodal information according to claim 6, characterized in that: The physiological characteristics include height and weight, and the height and weight corresponding to the standard feature vector are similar to the height and body shape of the target person.

9. A device for detecting the state of rehabilitation of exercise function based on multimodal information, comprising a memory, a processor, and a program stored in the memory, characterized in that: When the processor executes the program, the method according to any one of claims 1 to 8 is implemented.

10. A storage medium having a program stored thereon, characterized in that: When the program is executed, the method according to any one of claims 1 to 8 is implemented.

Images