Rehabilitation evaluation method, system and equipment fusing multi-modal data and medium

Abstract

The invention discloses a multi-modal data fused rehabilitation assessment method, system, equipment and medium, and the method comprises the steps: collecting high-density surface electromyogram signals, muscle blood flow ultrasonic data, individualized central function data and final rehabilitation information of a to-be-assessed person, carrying out the feature extraction of the high-density surface electromyogram signals, obtaining an extraction result, and carrying out the rehabilitation assessment of the to-be-assessed person. Inputting the individualized central function data, the mobile rehabilitation information and the extraction result into a first machine learning model, so that the first machine learning model fuses the individualized central function data, the mobile rehabilitation information and the extraction result according to the characteristics of the data; and inputting the fused data into a second machine learning model for analysis processing to obtain an analysis report, and performing evaluation according to the analysis report to obtain a new rehabilitation scheme. According to the method, data of different modes are subjected to fusion analysis, so that the accuracy and reliability of rehabilitation diagnosis evaluation are improved, and an accurate rehabilitation scheme is obtained.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data processing, and in particular to a rehabilitation evaluation method and system fusing multi-modal data, a device and a medium. BACKGROUND

[0002] Robot-assisted rehabilitation training intervention should be combined with continuous objective quantitative motor function evaluation to monitor the rehabilitation progress, realize process optimization, and achieve individualized optimal rehabilitation effect. However, the current rehabilitation content and treatment design and efficacy evaluation are mostly fixed or subjective treatment, and the training content is based on a single clinical representation, such as a 2-3 week hospitalization period in a general hospital, 1 hour of upper limb training per day, or the popular early high-intensity rehabilitation intervention and research in recent years, which determines the efficacy according to the comparison of clinical scales and performance evaluation before and after the training period. However, such evaluation cannot reveal the saturation and characteristic changes of the rehabilitation process. In addition, existing technologies such as magnetic resonance imaging (fMRI), near-infrared (fNIRs), and electroencephalography (EEG) can provide more objective function measurements, but due to the complexity of professional operation and high cost, and the different information included in each modal data, they cannot comprehensively evaluate the rehabilitation and optimize the rehabilitation plan. SUMMARY

[0003] To solve the above technical problems, the present application provides a rehabilitation evaluation method, system, device and medium fusing multi-modal data, which improves the accuracy of rehabilitation diagnosis and evaluation by fusing and analyzing the data of different modalities obtained by monitoring the individual rehabilitation process.

[0004] The first aspect of the embodiment of the present application provides a rehabilitation diagnosis and evaluation method, which comprises:

[0005] Collecting initial diagnosis parameters of the person to be evaluated, wherein the diagnosis parameters include high-density surface electromyography signals, muscle blood flow ultrasound data, and individualized central function data;

[0006] Real-time collection of rehabilitation information of the person to be evaluated, calculation of the rehabilitation progress saturation degree according to the rehabilitation information, and output of the rehabilitation information as the final rehabilitation information if the rehabilitation progress saturation degree is greater than the preset saturation degree, wherein the rehabilitation information includes EMG level, CIs, and ROM;

[0007] Feature extraction is performed on the high-density surface electromyography signals to obtain an extraction result, and the individualized central function data, the final rehabilitation information, and the extraction result are input into a first machine learning model to fuse the individualized central function data, the rehabilitation information, and the extraction result according to the characteristics of the data, and obtain a plurality of fused data;

[0008] The fused data is input into a second machine learning model for analysis and processing to obtain an analysis report, and the analysis report is evaluated to obtain a new rehabilitation scheme.

[0009] In a possible implementation manner of the first aspect, feature extraction is performed on the high-density surface electromyography signal to obtain an extraction result, including:

[0010] The high-density surface electromyography signal is subjected to component decomposition to obtain neural drive information corresponding to each component;

[0011] The neural drive information is subjected to time sequence feature extraction to obtain micro features and macro features, and the macro features and the micro features are fused by using a trained time sequence neural network to obtain the extraction result.

[0012] In a possible implementation manner of the first aspect, the individualized central function data includes a shaping degree of a muscle cortex coupling center, a lateralization index of a cortex brain function network, a central and peripheral blood oxygen change rate, and upper limb muscle group coordination,

[0013] The shaping degree of the muscle cortex coupling center is obtained by performing whole brain CMC measurement on the to-be-evaluated person, and the lateralization index of the cortex brain function network is obtained by measuring the brain network of the to-be-evaluated object in the upper limb motor perception function evaluation by using an electroencephalogram collector;

[0014] The central and peripheral blood oxygen change rate is obtained by measuring the blood flow and blood oxygen of the central and upper limb muscles in the passive, active and resistance exercise modes by using fNIRS, and the upper limb muscle group coordination is obtained by performing upper limb muscle group autonomous movement coordination evaluation on the to-be-evaluated object.

[0015] In a possible implementation manner of the first aspect, after obtaining the analysis report, the method further includes:

[0016] The second machine learning model is used to calculate the extraction result, the final rehabilitation information and the individualized central and peripheral function indicators respectively to obtain a plurality of evaluation results, the evaluation results are compared to obtain a comparison result, and the analysis report is verified according to the verification result.

[0017] The second aspect of the embodiment of the present application provides a rehabilitation evaluation system fusing multi-modal data, including a multi-modal rehabilitation diagnosis evaluation module, a mobile rehabilitation module and a diagnosis module, wherein,

[0018] The diagnosis module is configured to collect initial diagnosis parameters of a to-be-evaluated person;

[0019] The mobile rehabilitation module is used to collect rehabilitation information of a person to be evaluated in real time, calculate a rehabilitation progress saturation degree according to the rehabilitation information, and output the rehabilitation information as final rehabilitation information if the rehabilitation progress saturation degree is greater than a preset saturation degree, wherein the rehabilitation information includes EMG levels, CIs and ROM

[0020] The multi-modal rehabilitation diagnosis evaluation module is used to extract features from high-density surface electromyography signals to obtain extraction results, input individualized central function data, final rehabilitation information and the extraction results into a first machine learning model to make the first machine learning model fuse the individualized central function data, the rehabilitation information and the extraction results according to characteristics of the data to obtain a plurality of fused data, input each fused data into a second machine learning model for analysis and processing to obtain an analysis report, evaluate the analysis report to obtain a new rehabilitation plan.

[0021] In a possible implementation manner of the second aspect, the feature extraction from the high-density surface electromyography signals to obtain the extraction results comprises:

[0022] The high-density surface electromyography signals are subjected to component decomposition to obtain neural drive information corresponding to each component;

[0023] The neural drive information is subjected to time sequence feature extraction to obtain micro features and macro features, and a trained time sequence neural network is used to fuse the macro features and the micro features to obtain the extraction results.

[0024] In a possible implementation manner of the second aspect, the individualized central function data comprises a shaping degree of a muscle cortex coupling center, a lateralization index of a cortex brain function network, a central and peripheral blood oxygen change rate and an upper limb muscle group coordination,

[0025] The shaping degree of the muscle cortex coupling center is obtained by measuring the whole brain CMC of the person to be evaluated, and the lateralization index of the cortex brain function network is obtained by measuring the brain network of the person to be evaluated in upper limb motor perception function evaluation by using an electroencephalogram collector;

[0026] The central and peripheral blood oxygen change rate is obtained by measuring the blood oxygen of the blood flow and blood oxygen of the central and upper limb muscles in passive, active and resistance exercise modes by fNIRS, and the upper limb muscle group coordination is obtained by evaluating the self-motion coordination of the upper limb muscle group of the person to be evaluated.

[0027] In a possible implementation manner of the second aspect, after the analysis report is obtained, the method further comprises:

[0028] The second machine learning model is used to calculate the extraction result, rehabilitation information and individualized central and peripheral function indicators respectively, a plurality of evaluation results are obtained, the evaluation results are compared, and a comparison result is obtained, so that the personnel verifies the analysis report according to the verification result.

[0029] The third aspect of the embodiment of the present application provides a computing device, comprising:

[0030] a memory for storing a computer program;

[0031] a processor for executing the computer program to realize the rehabilitation evaluation method of fusing multi-modal data according to the first aspect.

[0032] The fourth aspect of the embodiment of the present application provides a non-transitory computer readable storage medium, which stores a computer program, and the computer program is executed by a processor to realize the steps of the rehabilitation evaluation method of fusing multi-modal data according to the first aspect.

[0033] The embodiment of the present application provides the initial diagnosis parameters of the personnel to be evaluated; the rehabilitation information of the personnel to be evaluated is collected in real time, the rehabilitation progress saturation is calculated according to the rehabilitation information, if the rehabilitation progress saturation is greater than the preset saturation, the rehabilitation information is output as the final rehabilitation information, the feature extraction is performed on the high-density surface electromyogram signal, the extraction result is obtained, the individualized central function data, the final rehabilitation information and the extraction result are input into the first machine learning model, so that the first machine learning model fuses the individualized central function data, the rehabilitation information and the extraction result according to the characteristics of the data, and a plurality of fused data are obtained; each fused data is input into the second machine learning model for analysis and processing, an analysis report is obtained, the analysis report is evaluated, and a new rehabilitation scheme is obtained. The method can improve the accuracy and reliability of the rehabilitation diagnosis and evaluation by fusing and analyzing the data of different modalities after monitoring the rehabilitation process of the personnel to be evaluated, so as to obtain an accurate rehabilitation scheme. BRIEF DESCRIPTION OF DRAWINGS

[0034] Figure 1 : the flowchart of the rehabilitation diagnosis and evaluation method provided by the present application is an embodiment;

[0035] Figure 2 : the structure and function diagram of the Io-ENMS upper limb individualized precise rehabilitation diagnosis and treatment platform of the rehabilitation diagnosis and evaluation method provided by the present application is an embodiment;

[0036] Figure 3 : the data collection diagram of the Io-ENMS upper limb individualized precise rehabilitation diagnosis and treatment platform of the rehabilitation diagnosis and evaluation method provided by the present application is an embodiment;

[0037] Figure 4 : A brain network diagram of a stroke patient in the assessment of upper limb motor sensory function, as provided in an embodiment of the rehabilitation diagnostic assessment method of the present invention;

[0038] Figure 5 : A comparison of the EEG brain networks of stroke patients with the affected and unaffected upper limbs when receiving tactile stimulation, and those of normal individuals, in an embodiment of the rehabilitation diagnosis and assessment method provided by the present invention;

[0039] Figure 6 : A system structure block diagram of an embodiment of the rehabilitation diagnosis and assessment method provided by the present invention. Detailed Implementation

[0040] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0041] Please refer to Figure 1 This is a flowchart illustrating one embodiment of the rehabilitation diagnosis and assessment method provided by the present invention, including steps S101 to S103.

[0042] S101. Collect diagnostic parameters of the person to be evaluated, wherein the diagnostic parameters include high-density surface electromyography signals, muscle blood flow ultrasound data, individualized central and peripheral function data, and rehabilitation information.

[0043] In this embodiment, current rehabilitation interventions do not yet achieve objective testing of individual patients' training tolerance, such as controlling fatigue levels during training to prevent insufficient or excessive training intensity. Therefore, the Io-ENMS upper limb individualized precision rehabilitation diagnosis and treatment platform is proposed, such as... Figure 2 As shown, Figure 2 This is a schematic diagram of the structure and function of the Io-ENMS upper limb individualized precision rehabilitation diagnosis and treatment platform. It utilizes individualized central and peripheral functional fNIRS, EEG and EMG data, high-density surface electromyography (HD-sEMG) data, and muscle blood flow ultrasound data obtained through inpatient and outpatient testing during hospitalization and discharge. The mobile Io-ENMS platform collects continuous upper limb functional rehabilitation information from hospitalization to discharge. For example... Figure 3 As shown, the Io-ENMS platform collects data such as rehabilitation progress saturation and fatigue tolerance during edge regulation. Then, a multimodal rehabilitation diagnostic assessment platform is used to comprehensively evaluate the collected information.

[0044] The multi-modal rehabilitation diagnostic evaluation platform integrates a comprehensive evaluation system of multi-modal diagnostic parameters collected during hospitalization and after discharge, and also incorporates continuous evaluation information of the lo-ENMS; the multi-modal rehabilitation diagnostic evaluation platform also has cloud information storage, operation, terminal retrieval and feedback functions. In addition to managing the parameter set of the lo-ENMS (EMG level; CIs; ROM), the multi-modal rehabilitation diagnostic evaluation platform also incorporates multi-modal diagnostic parameters: central and peripheral fNIRS, EEG and EMG, HD-sEMG, muscle blood flow ultrasound for post-stroke stepwise comprehensive evaluation.

[0045] In some embodiments, the "individualized central and peripheral function data" in step S101 includes but is not limited to the following steps:

[0046] The individualized central and peripheral function indicators include the shaping degree of the muscle-cortical coupling center, the lateralization index of the cortical brain function network, the central and peripheral blood oxygen change rate, and the upper limb muscle group coordination;

[0047] The shaping degree of the muscle-cortical coupling center is obtained by measuring the whole brain CMC of the person to be evaluated, and the lateralization index of the cortical brain function network is obtained by measuring the brain network of the person to be evaluated in the upper limb motor perception function evaluation using an electroencephalograph;

[0048] The central and peripheral blood oxygen change rate is obtained by measuring the blood oxygen of the central and upper limb muscles in passive, active and resistance exercise modes using fNIRS, and the upper limb muscle group coordination is obtained by measuring the four-channel electromyogram of the person to be evaluated during training and the upper limb muscle group coordination of the person to be evaluated using the lo-ENMS.

[0049] In this embodiment, the muscle-cortical coupling (CMC) in the shaping degree of the muscle-cortical coupling center is the power cross spectrum of EEG and EMG signals, which can measure the functional coupling strength of the cortical site and muscle spontaneous contraction. The shaping degree of the target muscle motor center can be evaluated by whole brain CMC measurement. Figure 4 The strength and position changes of the CMC center of the small finger extensor muscle found in the previous study during the 20 ENMS assisted processes experienced by the stroke patient. The increase in the strength of the center CMC represents the activation of the center, and the migration of the CMC peak to the affected half represents the benign contralateral motor function recovery.

[0050] Lateralization index of the cortical brain function network: EEG and fNIRS will be further used for structural research on brain network in task state. Figure 5The brain network of stroke patients in the upper limb motor sensory function evaluation measured by EEG is utilized. When the brain network related to motor function increases in focus and shifts to the affected hemisphere, it represents functional recovery and reduced compensation; when the brain network remains in the unaffected hemisphere, it represents weak functional recovery and dependence on compensation. The brain network of stroke patients requires more extensive cortical resources and is distributed ipsilaterally and centrally compared to normal people.

[0051] The central and peripheral blood oxygen change rates are analyzed by fNIRS, blood flow and blood oxygen of the central and peripheral muscles in passive, active and resistance exercise modes, and blood flow velocity and blood oxygen concentration changes during rehabilitation training. Previous studies have shown that the increase in central and peripheral muscle blood flow velocity and blood oxygen consumption is related to exercise mode, providing supporting evidence for neural-vascular oxygen changes in muscle fatigue and central brain function reorganization.

[0052] Upper limb muscle group coordination is measured by Io-ENMS in training to calculate muscle coordination in addition to four-channel EMG. The multi-modal rehabilitation diagnostic evaluation platform will provide detailed upper limb muscle self-motion coordination evaluation for inpatients and outpatients, mainly by 8-channel EMG to measure CI values between AD, PD and BIC, TRI, ECR, FCR, ED and FD in daily upper limb movements to evaluate the normalization ability of peripheral muscle function.

[0053] S102: Real-time acquisition of rehabilitation information of the person to be evaluated, and calculation of the rehabilitation progress saturation degree according to the rehabilitation information, if the rehabilitation progress saturation degree is greater than the preset saturation degree, output the rehabilitation information as the final rehabilitation information, wherein the rehabilitation information includes EMG level, CI and ROM 。

[0054] In this embodiment, a mobile rehabilitation robot, i.e., a mobile rehabilitation platform, is used to integrate multi-modal evaluation systems in the rehabilitation application / method of patients at different recovery stages after stroke, specifically:

[0055] In the early fatigue controllable enhanced rehabilitation intervention stage of stroke, real-time monitoring and training of EMG level, CI and ROM before, 1 week and 2 weeks after the comprehensive diagnosis and treatment platform evaluation. The advantage of this stage is to grasp the intensity of early rehabilitation training. According to the parameter set of EMG level, CI and ROM measured by the patient's daily training, the individual's daily rehabilitation training before and after the change is monitored through cloud computing and intelligent algorithm, and the training saturation and rehabilitation progress are defined according to the individual brain function data, combined with self-feedback, and the upper limb joint chain movement scheme is adjusted combined with high-density surface electromyographic signals, so that individualized rehabilitation is further optimized, in order to achieve more efficient motor function rehabilitation.

[0056] Select the first single hemiplegia within 6 months of stroke patients, for a period of 2 weeks, 2 times a day, 6-8 hours apart, each 2 hours (the specific length of the course of treatment refer to the following explanation) of mobile Io-ENMS assisted upper limb training. Real-time monitoring of EMG+CIs+ROM by mobile Io-ENMS, determine the fatigue tolerance and rehabilitation process. In each training, by mobile Io-ENMS real-time control of muscle fatigue degree MPF is not less than 70% of the baseline; otherwise give appropriate rest recovery; then continue training until 2 hours complete a single training. The length of the course of treatment by measured training saturation to determine the end time. In each training, by mobile Io-ENMS monitoring real-time fatigue degree determines the length of training. According to the individual rehabilitation parameter set <EMG level, CIs, ROM> uploaded by mobile Io-ENMS to calculate the saturation of rehabilitation process, when it is considered to achieve training saturation, stop training (i.e. a course of treatment ends).

[0057] It should be noted that the rehabilitation information refers to the EMG level, CIs, ROM parameter information uploaded by the mobile Io-ENMS. The person to be evaluated refers to inpatients and outpatients, i.e. patients undergoing rehabilitation intervention. The Io-ENMS platform is a mobile automated stroke upper limb rehabilitation platform for optimizing the continuous upper limb function rehabilitation of stroke individuals from hospitalization to discharge. The platform includes innovative continuous individual rehabilitation process automatic monitoring and control technology, which can characterize the rehabilitation process and tolerance characteristics of stroke individuals by using physiological kinematic parameters measured by the robot during training, realize efficient, objective and automated evaluation, and control the robot-assisted upper limb training course and scheme to prevent undertraining and overtraining.

[0058] S103: feature extraction is performed on the high-density surface electromyogram signal to obtain an extraction result, and the individualized central function data, the final rehabilitation information and the extraction result are input into a first machine learning model, so that the first machine learning model fuses the individualized central function data, the rehabilitation information and the extraction result according to the characteristics of the data to obtain a plurality of fused data.

[0059] In this embodiment, the high-density surface electromyogram signal (HD-sEMG) has the advantages of non-invasiveness, convenience and multi-channel, and can provide rich muscle space activation information in time and space. The technology uses a high-density surface electromyogram sensor to collect HD-sEMG, which can evaluate more accurate muscle strength and autonomous control ability of muscle movement units derived from nerves. HD-sEMG is a non-stationary physiological signal, which is easily disturbed by noise such as adjacent muscle signals and body movement electric waves, thereby limiting the improvement of muscle strength estimation accuracy. Therefore, feature extraction is performed on the HD-sEMG to basically realize the fusion of macroscopic features and microscopic neural driving information, thereby improving the muscle strength estimation accuracy.

[0060] After feature extraction of high-density surface electromyography signals, the individualized central and peripheral function indicators, mobile rehabilitation information, and extraction results are fused using a machine learning model. The individualized evaluation of multi-modal data complements and cooperates with each other, thereby providing a more comprehensive rehabilitation evaluation result.

[0061] In this embodiment, the electromyography signals are classified and action pattern recognition is performed based on PCA and LDA, and then the surface electromyography signals are reduced and classified using a BP neural network. A method for decomposing macro features and micro neural drive information of high-density surface electromyography (HD-sEMG) is realized. The temporal and spatial fusion of macro and micro features is realized using a time series deep learning method, the problem of HD-sEMG noise interference is solved, and the requirement of high-precision muscle strength estimation is met. Finally, rehabilitation diagnosis and evaluation are realized based on multi-modal fusion (including upper limb motion trajectory, surface electromyography, and patient motion posture, etc.).

[0062] It should be noted that the first machine learning model mainly uses convolutional neural networks and self-attention deep learning models (Transformer) to fuse multi-modal data. By using different machine learning models to fuse multi-class data recognition tasks, a machine learning model with high accuracy is obtained.

[0063] According to the characteristics of the data, the multi-modal data are fused. The advantage of multi-modal fusion is to comprehensively utilize the information of different modalities, complement and cooperate with each other, and thereby provide a more comprehensive rehabilitation evaluation result. For example, fNIRS and EEG data can be fused to obtain information on brain function activity and neural connectivity, further understanding the brain function state and rehabilitation potential of the patient. In addition, EMG and muscle blood flow ultrasound data can be fused to comprehensively evaluate the muscle function and blood supply of the patient, providing a more accurate basis for the development of rehabilitation programs.

[0064] It should be noted that fNIRs refers to functional near-infrared spectroscopy technology, EEG refers to electroencephalogram, and EMG refers to electromyogram.

[0065] In some embodiments, the step of "extracting features from the high-density surface electromyography signals to obtain extraction results" in step S103 includes but is not limited to the following steps:

[0066] The high-density surface electromyography signals are decomposed into components to obtain neural drive information corresponding to each component;

[0067] The neural drive information is subjected to time series feature extraction to obtain micro features and macro features, and the trained time series neural network is used to fuse the macro features and the micro features to obtain the extraction results.

[0068] Specifically, the action potential sequence capable of reflecting microscopic neural drive is obtained by using the HD-sEMG decomposition technology, and the HD-sEMG signal can be decomposed into multiple components, each component corresponds to different types of neural drive information, so as to obtain more fine-grained neural drive information. Then, deep learning has the advantages of high information utilization rate, high generalization ability and autonomous feature extraction, and is applied to a large number of HD-sEMG macro-micro time sequences to extract related features, and the macro features realize the fusion in time and space; and the time sequence neural network is used to automatically learn the time sequence relationship between the features, and the macro features and the micro features are effectively integrated, so as to realize high-precision muscle force estimation. HD-EMG will become a diagnostic platform for more accurate phased evaluation, and will complement the continuous measurement of Io-ENNMS.

[0069] S104, input each fused data into a second machine learning model for analysis and processing to obtain an analysis report, evaluate the analysis report to obtain a new rehabilitation scheme.

[0070] In this embodiment, each fused data is input into a second machine learning model for analysis and processing to generate a comprehensive visual report; and a quick diagnosis reference is provided for clinical medical staff, such as ANN and SVM projection to clinical scale values. The team adjusts the rehabilitation scheme of the stroke individual according to the report as necessary.

[0071] It should be noted that the second machine learning model is also a machine learning model based on a convolutional neural network and a self-attention deep learning model (Transformer).

[0072] In some embodiments, the step S104 includes but is not limited to the following steps after obtaining the analysis report:

[0073] The second machine learning model is used to calculate the extraction result, the rehabilitation information and the individualized central and peripheral function index respectively to obtain a plurality of evaluation results, compare each evaluation result to obtain a comparison result, so that the personnel verify the analysis report according to the verification result.

[0074] Specifically, the multi-modal fusion rehabilitation diagnosis evaluation can also improve the reliability and accuracy of diagnosis by verifying and cross-verifying the results of different modalities. By comparing the results of fNIRS and EEG, it can be verified whether the patient's brain function activity is consistent with the neural electrical activity, thereby increasing the credibility of the results. In addition, by comprehensively analyzing the results of multiple modalities, the patient's rehabilitation progress and effect can be better understood, providing a scientific basis for the adjustment and optimization of the rehabilitation program. The current multi-modal model has the problem of modality processing limitation. In order to solve this problem, a multi-modal rehabilitation diagnosis evaluation framework based on knowledge graph and causal reasoning will be studied, and the model will continuously learn the growing external knowledge, including multi-modal adaptive contrast learning, symbolic causal graph neural network, and causal transfer learning.

[0075] It should be understood that, although Figure 1 The steps in the flowchart of the application are shown in sequence according to the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order limitation for the execution of these steps, and these steps can be executed in other orders. Moreover, Figure 1 At least part of the steps in the flowchart of the application can include multiple steps or stages, which are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily sequential, but can be executed in rotation or alternation with at least part of other steps or steps or stages in other steps.

[0076] In one embodiment, as Figure 6 shown, it shows a block diagram of a rehabilitation diagnosis evaluation system 600 provided by the embodiments of the application, which includes a collection module 601, a fusion module 602 and an analysis module 603, wherein:

[0077] The collection module 601 is configured to collect diagnosis parameters of a person to be evaluated, wherein the diagnosis parameters include high-density surface electromyography signals, muscle blood flow ultrasound data, individualized central and peripheral function data, and rehabilitation information.

[0078] The fusion module 602 is configured to extract features from the high-density surface electromyography signals to obtain extraction results, and input the individualized central and peripheral function indicators, the mobile rehabilitation information and the extraction results into a first machine learning model, so that the first machine learning model fuses the individualized central and peripheral function indicators, the rehabilitation information and the extraction results according to the characteristics of the data to obtain a plurality of fused data.

[0079] The analysis module 603 is configured to input each of the fused data into a second machine learning model for analysis and processing to obtain an analysis report, evaluate according to the analysis report, and obtain a new rehabilitation program.

[0080] In one embodiment, feature extraction is performed on the high-density surface electromyography signal to obtain an extraction result, including:

[0081] The high-density surface electromyography signal is decomposed into components to obtain neural drive information corresponding to each component;

[0082] The neural drive information is subjected to time series feature extraction to obtain micro features and macro features, and the trained time series neural network is used to fuse the macro features and the micro features to obtain the extraction result.

[0083] In one embodiment, the individualized central and peripheral function indicators include the shaping degree of the muscle cortex coupling center, the lateralization index of the cortical brain function network, the central and peripheral blood oxygen change rate, and the upper limb muscle group coordination,

[0084] The shaping degree of the muscle cortex coupling center is obtained by measuring the whole brain CMC of the to-be-evaluated person, and the lateralization index of the cortical brain function network is obtained by measuring the brain network of the to-be-evaluated object in the upper limb motor perception function evaluation using an electroencephalograph;

[0085] The central and peripheral blood oxygen change rate is obtained by measuring the blood flow and blood oxygen of the central and upper limb muscles in passive, active and resistance exercise modes using fNIRS, and the upper limb muscle group coordination is obtained by measuring the four-channel electromyography of the to-be-evaluated object in training and the upper limb muscle group coordination of the to-be-evaluated object using Io-ENMS.

[0086] In one embodiment, after obtaining the analysis report, further comprising:

[0087] The extraction result, the rehabilitation information and the individualized central and peripheral function indicators are calculated using a second machine learning model to obtain a plurality of evaluation results, the evaluation results are compared to obtain a comparison result, and the analysis report is verified according to the verification result.

[0088] As an example of the present embodiment,

[0089] In one embodiment of the present application, a computing device is provided, which includes a memory and a processor, the memory stores a computer program, and the processor implements the above steps when executing the computer program; the computing device provided by the embodiment of the present application has similar implementation principles and technical effects to the above method embodiments, which will not be described here.

[0090] In one embodiment of the present application, a non-transitory computer-readable storage medium is provided, and the computer program is stored in the computer-readable storage medium. The computer program is executed by a processor to further implement the above steps. The computer-readable storage medium provided in the embodiment has similar implementation principles and technical effects to the above method embodiments, and will not be described here.

[0091] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, the processes of the above-mentioned embodiments can be included. Any reference to memory, storage, database or other medium used in the embodiments provided by the present application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration but not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0092] The technical features of the above embodiments can be combined in any way. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as the combinations of the technical features do not contradict, they should be considered within the scope of the present application.

[0093] The above specific embodiments further illustrate the purpose, technical solutions and beneficial effects of the present application. It should be understood that the above description is only for specific embodiments of the present application and does not limit the protection scope of the present application. It is particularly pointed out that any modification, equivalent replacement, improvement, etc. made by those skilled in the art within the spirit and principles of the present application should be included in the protection scope of the present application.

Claims

1. A method of fusing multi-modal data for rehabilitation assessment, the method comprising: The method comprises the following steps: collecting initial diagnosis parameters of a person to be evaluated, wherein the diagnosis parameters include high-density surface electromyography signals, muscle blood flow ultrasound data, and individualized central function data; real-time collection of rehabilitation information of the person to be evaluated, calculation of rehabilitation progress saturation degree according to the rehabilitation information, and output of the rehabilitation information as final rehabilitation information if the rehabilitation progress saturation degree is greater than a preset saturation degree, wherein the rehabilitation information includes EMG level, CIs, and ROM; feature extraction of the high-density surface electromyography signals to obtain an extraction result, input of the individualized central function data, the final rehabilitation information, and the extraction result into a first machine learning model to make the first machine learning model fuse the individualized central function data, the rehabilitation information, and the extraction result according to the characteristics of the data to obtain a plurality of fused data; analysis and processing of each of the fused data by the second machine learning model to obtain an analysis report, evaluation of the analysis report, and obtaining of a new rehabilitation plan.

2. The rehabilitation assessment method of fusing multi-modal data as claimed in claim 1, wherein, The feature extraction of the high-density surface electromyography signals to obtain an extraction result comprises the following steps: component decomposition of the high-density surface electromyography signals to obtain neural drive information corresponding to each component; time series feature extraction of the neural drive information to obtain micro features and macro features, fusion of the macro features and the micro features by using a trained time series neural network to obtain the extraction result.

3. The rehabilitation assessment method of fusing multi-modal data as claimed in claim 1, wherein, The individualized central function data includes the shaping degree of muscle cortex coupling center, the lateralization index of cortex brain function network, the central and peripheral blood oxygen change rate, and the upper limb muscle group coordination, wherein the shaping degree of muscle cortex coupling center is obtained by measuring the whole brain CMC of the person to be evaluated, and the lateralization index of cortex brain function network is obtained by measuring the brain network of the person to be evaluated in the upper limb motor perception function evaluation by using an electroencephalogram collector; the central and peripheral blood oxygen change rate is obtained by measuring the blood oxygen of the passive, active, and resistance exercise modes of the fNIRS central and upper limb muscle blood flow and blood oxygen, and the upper limb muscle group coordination is obtained by evaluating the self-motion coordination of the upper limb muscle group of the person to be evaluated.

4. The rehabilitation assessment method of fusing multi-modal data as claimed in claim 1, wherein, After obtaining the analysis report, the method further comprises the following steps: calculating the extraction result, the final rehabilitation information, and the individualized central and peripheral function indicators by using the second machine learning model to obtain a plurality of evaluation results, comparing each evaluation result to obtain a comparison result, and verifying the analysis report according to the verification result.

5. A rehabilitation assessment system fusing multi-modal data, characterized by, The method comprises the following steps: The method comprises the following steps: The diagnostic module is used to collect initial diagnosis parameters of a person to be evaluated; The mobile rehabilitation module is used to real-time collection of rehabilitation information of the person to be evaluated, calculation of rehabilitation progress saturation degree according to the rehabilitation information, and output of the rehabilitation information as final rehabilitation information if the rehabilitation progress saturation degree is greater than a preset saturation degree, wherein the rehabilitation information includes EMG level, CIs, and ROM The multi-modal rehabilitation diagnosis evaluation module is used for feature extraction on the high-density surface electromyography signal to obtain an extraction result, inputting the individualized central function data, the final rehabilitation information and the extraction result into a first machine learning model, so that the first machine learning model fuses the individualized central function data, the rehabilitation information and the extraction result according to the characteristics of data to obtain a plurality of fused data; inputting each of the fused data into the second machine learning model for analysis and processing to obtain an analysis report, evaluating the analysis report to obtain a new rehabilitation scheme.

6. The rehabilitation assessment system fusing multi-modal data of claim 5, wherein, The feature extraction on the high-density surface electromyography signal to obtain an extraction result comprises: component decomposition on the high-density surface electromyography signal to obtain neural drive information corresponding to each component; time sequence feature extraction on the neural drive information to obtain micro features and macro features, and fusion of the macro features and the micro features by using a trained time sequence neural network to obtain the extraction result.

7. The rehabilitation assessment system fusing multi-modal data of claim 5, wherein, The individualized central function data comprises a shaping degree of a muscle cortex coupling center, a lateralization index of a cortex brain function network, a central and peripheral blood oxygen change rate and an upper limb muscle group coordination, The shaping degree of the muscle cortex coupling center is obtained by whole brain CMC measurement on the to-be-evaluated personnel, and the lateralization index of the cortex brain function network is obtained by measuring the brain network of the to-be-evaluated object in upper limb motor perception function evaluation by using an electroencephalogram collector; The central and peripheral blood oxygen change rate is obtained by blood oxygen measurement of blood flow and blood oxygen of the central and upper limb muscles in passive, active and resistance exercise modes by fNIRS, and the upper limb muscle group coordination is obtained by upper limb muscle group autonomous movement coordination evaluation on the to-be-evaluated object.

8. The rehabilitation assessment system fusing multi-modal data of claim 5, wherein, After obtaining the analysis report, the method further comprises: calculating the extraction result, the rehabilitation information and the individualized central and peripheral function indicators by using the second machine learning model respectively to obtain a plurality of evaluation results, comparing each evaluation result to obtain a comparison result, so that a person verifies the analysis report according to the verification result.

9. A computing device, comprising: Comprise: a memory for storing a computer program; a processor for executing the computer program to realize the rehabilitation evaluation method of fused multi-modal data according to any one of claims 1 to 4.

10. A non-transitory computer-readable storage medium, comprising: A computer program is stored thereon, and the computer program is executed by a processor to realize the rehabilitation evaluation method of fused multi-modal data according to any one of claims 1 to 4.

Images