Multimodal rehabilitation assessment method, system, computer device and storage medium

Abstract

This invention discloses a multimodal rehabilitation assessment method, system, computer device, and storage medium. It proposes multi-sensor information input and a rehabilitation task-oriented rehabilitation assessment solution that incorporates a model transfer strategy and hybrid experts. This method enables comprehensive and accurate assessments. By collecting diverse data through multiple sensors and integrating multimodal information, it comprehensively reflects the patient's rehabilitation status and improves assessment accuracy. It also enables personalized customization, providing patients with personalized rehabilitation recommendations and solutions through a hybrid subtask assessment system that combines knowledge from different fields. It also enables real-time monitoring and dynamic updating of assessment results and solutions to adapt to changes in the patient's rehabilitation process. It also improves rehabilitation outcomes, with accurate assessments and personalized solutions contributing to improved rehabilitation efficiency and quality.

Description

Technical Field

[0001] The present invention belongs to the technical field of medical equipment, and in particular to a multimodal rehabilitation assessment method, system, computer equipment and storage medium. Background Art

[0002] In recent years, an increasing number of patients with functional disabilities have required rehabilitation training to restore normal bodily function. This need for long-term rehabilitation is essential. Professional and timely rehabilitation assessments are essential during the rehabilitation process. Traditional training methods involve users participating in a training program or rehabilitation course, supervised by a specialist, who then prescribes one or more exercises. This can lead to regimens that are overly rigid and lack flexibility, and patients may become overly reliant on expert guidance and supervision. Furthermore, this approach places significant pressure on the healthcare system. Most importantly, without expert supervision, users are unable to effectively manage their condition, resulting in a delay in understanding their rehabilitation outcomes and significantly reduced compliance with training.

[0003] Most traditional assessments focus on a single body function, such as simply monitoring heart rate and blood pressure to judge cardiopulmonary rehabilitation, while ignoring related factors such as muscle strength and joint range of motion.

[0004] Various types of rehabilitation testing equipment continue to emerge, generating massive amounts of fragmented data, such as motion data from wearable devices and physiological structure data from hospital imaging equipment, forming data islands that make it impossible to form a complete assessment.

[0005] The Chinese patent (CN117894428A) introduces a rehabilitation robot control method based on multi-sensor data. By fusing multi-sensor data and utilizing machine learning and artificial intelligence technologies, it provides patients with more accurate and personalized rehabilitation plans, thereby improving rehabilitation efficiency and safety.

[0006] The Chinese patent (CN116370259A) introduces a human-computer interaction upper limb rehabilitation system based on multi-sensor information fusion. Through the collaborative work of data gloves, binocular cameras, a host computer control center, an exoskeleton upper limb rehabilitation robot and an augmented virtual reality game center, patients can complete rehabilitation tasks in an entertaining game, improve rehabilitation effects and patient compliance, and achieve comprehensive monitoring and intelligent interaction of patients' upper limb rehabilitation training.

[0007] In general, existing rehabilitation assessment methods focus more on the evaluation of single functions and indicators and are unable to comprehensively evaluate the patient's rehabilitation progress; or they use one or several devices and are unable to be compatible with multiple sensing devices to more accurately evaluate the user's functions. Summary of the Invention

[0008] The purpose of the present invention is to address the problems existing in the above-mentioned prior art and to provide a rehabilitation assessment system that proposes multi-sensory information input, is rehabilitation task-oriented, combines model transfer strategy, and hybrid experts.

[0009] The technical solution to achieve the purpose of the present invention is: a multimodal rehabilitation assessment method, which includes:

[0010] Step 1: Confirm the rehabilitation assessment tasks, including main tasks and subtasks;

[0011] Step 2, collecting multi-source data related to rehabilitation of the subject through multiple sensor channels;

[0012] Step 3: preprocessing the multi-source data to correct data quality and form a task data set;

[0013] Step 4: Based on the sensor channel type and rehabilitation assessment task type, train on the corresponding open source dataset to obtain a pre-trained model;

[0014] Step 5: further optimizing and training the pre-trained model using the task dataset to adapt it to the characteristics of the rehabilitation data in the task dataset and the requirements of the rehabilitation assessment task, and extracting feature vectors of the data from each sensor channel using the optimized model;

[0015] Step 6: Establish a multi-source sensor model fusion strategy for a single subtask based on the rehabilitation assessment task, and use this strategy to fuse the features of the data from each sensor channel, and use the fused features as the subtask features;

[0016] Step 7: Pass the subtask features through an independent fully connected module to output the evaluation results of each subtask;

[0017] Step 8: Establish a multi-task evaluation system to fuse all sub-task features corresponding to the main task;

[0018] In step 9, the features fused in step 8 are used as the main task features, and after passing through independent fully connected modules and MLP models, the evaluation results of each main task are output.

[0019] Furthermore, the main task in step 1 includes at least a rehabilitation assessment table, and each attribute in the table serves as a subtask; the rehabilitation assessment table includes a clinical scale or a clinical scale customized based on the clinical scale.

[0020] Furthermore, the multi-source data in step 2 includes at least motion data, mechanical data, electromyographic signals and image data of the subject to be rehabilitated; the multiple sensing channels in step 1 include at least sensing channels formed by acceleration sensors, gyroscopes, pressure sensors, electromyographic sensors and visual sensors respectively.

[0021] Furthermore, in step 4, training is performed on the corresponding open source dataset according to the sensor channel type and rehabilitation assessment task type to obtain a pre-trained model, which specifically includes:

[0022] Step 4-1: Select a number of open source datasets with the same number of sensor channels according to the type of rehabilitation assessment task;

[0023] Step 4-2: Generate feature embedding for the open source dataset, specifically:

[0024] For each sensor channel type, the rehabilitation data in the corresponding open source dataset is used as the input signal segment , input encoder Generate feature embeddings:

[0025]

[0026] Where, represents a high-dimensional embedding vector;

[0027] Step 4-3, performing self-supervised learning on the open source dataset to obtain a pre-trained model to extract common features related to the rehabilitation assessment task; specifically, including:

[0028] Step 4-3-1, select the contrastive learning framework;

[0029] Step 4-3-2: Based on the feature embedding generation result, execute a positive sample generation strategy to generate positive samples; the positive sample generation strategy is: generate positive sample pairs through at least two data enhancement methods;

[0030] Step 4-3-3, construct the objective function L SSL :

[0031]

[0032] Where N is the total batch of input signal segments, M is the total number of candidate samples, Represents The corresponding positive sample, represents a randomly selected negative sample; is the similarity function;

[0033] Step 4-3-4, the objective function L SSL Optimization is performed to maximize the similarity of positive sample pairs and minimize the similarity of negative sample pairs, thereby obtaining a pre-trained model.

[0034] Furthermore, step 5 specifically includes:

[0035] Step 5-1: Generate feature embedding for the task dataset, specifically:

[0036] The rehabilitation data collected by each sensor channel is used as the input signal segment , input encoder Generate feature embeddings:

[0037]

[0038] Where, represents a high-dimensional embedding vector;

[0039] Step 5-2: Based on the result of step 5-1 and the pre-trained model obtained in step 4, transfer learning is performed to generate a pre-trained model corresponding to the task dataset;

[0040] Step 5-3, establish the transfer learning loss function :

[0041]

[0042] in, is the classification cross entropy loss function:

[0043]

[0044] is the regularization term:

[0045]

[0046] Where, is the true label, is the predicted probability, N represents the total batch of input signal segments, is the encoder parameter;

[0047] Step 5-4, optimize the pre-trained model generated in step 5-2: update the parameters of the pre-trained model using stochastic gradient descent or Adam optimizer by minimizing the transfer learning loss function:

[0048]

[0049]

[0050] Where t represents the number of optimization iterations, Represents the classification head, is the learning rate, and are the gradients of the encoder and classification head, respectively.

[0051] Furthermore, in the process of optimizing the pre-trained model generated in step 5-2, step 5-4 further includes: adjusting the output result of the pre-trained model by freezing and unfreezing the encoder parameters.

[0052] Furthermore, in step 6, a multi-source sensor model fusion strategy for a single subtask is established based on the rehabilitation assessment task, and the features of the data from each sensor channel are fused using the strategy, specifically including:

[0053] Step 6-1: Build a multi-attention model. For a single subtask, perform embedding operations on the feature vectors of different sensor channels extracted in step 5 to generate input vectors:

[0054]

[0055] Where, represents the feature vector of the nth sensing channel, where n represents the number of sensing channels;

[0056] Step 6-2, for each attention , calculate the sensing channel coding:

[0057]

[0058] in,

[0059]

[0060]

[0061] Split the Q, K, and V matrices into multiple subspaces according to the number of heads h, and each subspace corresponds to an attention head:

[0062]

[0063] Where, is the key vector dimension scaling factor, Q is the query vector, K is the key vector, V is the value vector, is the activation function, , , , is the output linear transformation matrix, , represents the feature dimension of the model, and It is the dimension set to adapt to the computing needs;

[0064] Step 6-3: concatenate and linearly transform the encoded feature vectors output by each sensor channel to obtain the final fused feature vector :

[0065]

[0066] Where, Represents the concatenation function.

[0067] Furthermore, the establishment of a multi-task evaluation system in step 8 specifically includes:

[0068] Step 8-1, assigning a separate subtask evaluation system to each subtask based on the subtask characteristics in step 6;

[0069] Step 8-2: Input each subtask feature into the corresponding subtask evaluation system to obtain the output of the subtask evaluation system;

[0070] Step 8-3: Create all subtask features corresponding to the main task The multi-task evaluation system for fusion is:

[0071]

[0072] in,

[0073]

[0074]

[0075]

[0076] Where, is the weight factor of the evaluation system for the i-th subtask, is the output of the i-th subtask evaluation system, m is the total number of subtask evaluation systems, is the output of the multi-task evaluation system, is the normal distribution function, is the activation function, is the noise of each subtask feature component, is the pre-trained weight of the component.

[0077] Furthermore, the weight factors of the subtask evaluation system need to satisfy:

[0078]

[0079]

[0080] in, Subtask features The balance weight factor, is the preset scaling factor, The input feature vector is the subtask feature The correlation coefficient of represents the balanced loss function.

[0081] Furthermore, the method further comprises:

[0082] Step 10: Generate a comprehensive evaluation result based on the evaluation results of each main task.

[0083] Furthermore, step 10 specifically includes:

[0084] Step 10-1, feature mapping the evaluation results of each main task;

[0085] Step 10-2: Establish a decision output model and input the feature mapping result into the decision output model to obtain a comprehensive evaluation result;

[0086] The decision output model is:

[0087]

[0088] Where, Represents the prediction result of the maximum likelihood output, is the output result vector of the main task, Indicates the comprehensive evaluation results. is the conditional probability, which means belong The probability of the category, is the feature mapping result corresponding to the jth main task, and q represents the total number of main tasks.

[0089] Compared with the prior art, the present invention has the following significant advantages:

[0090] (1) Comprehensive and accurate assessment: Multiple sensor channels, including accelerometers, gyroscopes, pressure sensors, myoelectric sensors, and visual sensors, are used to collect multi-source data, including motion data, mechanical data, myoelectric signals, and image data of the rehabilitation subject. During data processing, a multi-attention model is used to fuse the feature vectors of different sensor channels. This multimodal information fusion method comprehensively reflects the patient's rehabilitation status and can more accurately assess the patient's rehabilitation progress compared to traditional single body function assessment methods.

[0091] (2) Personalized customization: Establish a multi-task evaluation system and assign a separate subtask evaluation model to each subtask. Combining knowledge from different fields and evaluating each subtask based on its characteristics and importance, it can provide patients with rehabilitation suggestions and plans that better meet their individual needs, achieving personalized customization.

[0092] Based on clinical tasks: The main tasks include clinical scales or custom clinical scales based on clinical scales. This allows the assessment to be closely aligned with actual clinical needs, tailoring individual assessment plans to the specific conditions and rehabilitation stages of different patients, further enhancing the targeted and personalized nature of rehabilitation assessments.

[0093] (3) Dynamic adaptability: When training the model, transfer learning is used, and the output of the pre-trained model is adjusted by freezing and unfreezing the encoder parameters. This approach enables the model to continuously adjust its own parameters according to changes in the patient's recovery process, better adapting to new data and task requirements, thereby achieving dynamic updates of evaluation results and plans to adapt to changes in the patient's recovery process.

[0094] (4) Improve rehabilitation effects: Through comprehensive and accurate assessment and personalized customization, more accurate rehabilitation plans can be developed for patients. Accurate assessment can accurately determine the patient's rehabilitation stage and existing problems, and personalized plans can provide targeted rehabilitation training suggestions for these problems. A multi-task evaluation system is established to integrate all sub-task features corresponding to the main task, and generate comprehensive evaluation results through a decision output model. This multi-task evaluation and comprehensive decision-making method can comprehensively consider multiple aspects of the patient's rehabilitation process, avoid the limitations of single indicator evaluation, provide a more comprehensive and scientific basis for rehabilitation treatment, and help improve rehabilitation effects.

[0095] The present invention is further described in detail below with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0096] Figure 1 Schematic diagram of the principle of the multimodal rehabilitation assessment method of the present invention in one embodiment.

[0097] Figure 2 Schematic diagram of the mapping between main task output and comprehensive evaluation output in one embodiment. DETAILED DESCRIPTION

[0098] In order to make the purpose, technical solutions and advantages of this application more clear, the following further describes this application in detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application.

[0099] It should be noted that if there are descriptions involving "first", "second", etc. in the embodiments of the present invention, the descriptions of "first", "second", etc. are only for descriptive purposes and cannot be understood as indicating or implying their relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of such features. In addition, the technical solutions between the various embodiments can be combined with each other, but they must be based on the ability of ordinary technicians in this field to implement them. When the combination of technical solutions is contradictory or cannot be implemented, it should be deemed that such a combination of technical solutions does not exist and is not within the scope of protection required by the present invention.

[0100] In one embodiment, combined Figure 1 , provides a multimodal rehabilitation assessment method, the method comprising:

[0101] Step 1: Confirm the rehabilitation assessment tasks, including main tasks and subtasks;

[0102] Step 2, collecting multi-source data related to rehabilitation of the subject through multiple sensor channels;

[0103] Step 3: preprocessing the multi-source data to correct data quality and form a task data set;

[0104] Step 4: Based on the sensor channel type and rehabilitation assessment task type, train on the corresponding open source dataset to obtain a pre-trained model;

[0105] Step 5: further optimizing and training the pre-trained model using the task dataset to adapt it to the characteristics of the rehabilitation data in the task dataset and the requirements of the rehabilitation assessment task, and extracting feature vectors of the data from each sensor channel using the optimized model;

[0106] Step 6: Establish a multi-source sensor model fusion strategy for a single subtask based on the rehabilitation assessment task, and use this strategy to fuse the features of the data from each sensor channel, and use the fused features as the subtask features;

[0107] Step 7: Pass the subtask features through an independent fully connected module to output the evaluation results of each subtask;

[0108] Step 8: Establish a multi-task evaluation system to fuse all sub-task features corresponding to the main task;

[0109] In step 9, the features fused in step 8 are used as the main task features, and after passing through independent fully connected modules and MLP models, the evaluation results of each main task are output.

[0110] Furthermore, in one embodiment, the main task in step 1 includes a rehabilitation assessment table, and each attribute in the table serves as a subtask; the rehabilitation assessment table includes a clinical scale or a clinical scale customized based on the clinical scale.

[0111] Furthermore, in one embodiment, the multi-source data in step 2 includes at least motion data, mechanical data, electromyographic signals and image data of the subject to be rehabilitated; the multiple sensing channels in step 1 include at least sensing channels formed by motion sensors (such as accelerometers, gyroscopes), force sensors (such as pressure sensors, force-sensitive resistors), physiological sensors (such as heart rate monitors, blood oxygen saturation sensors), and visual sensors.

[0112] Here, data from various sensors are collected synchronously and in real time to ensure that the data can accurately reflect the patient's exercise performance, force application, and changes in physiological state.

[0113] Preferably, in some embodiments, the preprocessing of the multi-source data in step 3 at least includes cleaning, denoising and normalizing the multi-source data.

[0114] Furthermore, in one embodiment, step 4 is performed on a corresponding open source dataset according to the sensor channel type and the rehabilitation assessment task type to obtain a pre-trained model, specifically including:

[0115] Step 4-1: Select a number of open source datasets with the same number of sensor channels according to the type of rehabilitation assessment task;

[0116] Step 4-2: Generate feature embedding for the open source dataset, specifically:

[0117] For each sensor channel type, the rehabilitation data in the corresponding open source dataset is used as the input signal segment , input encoder Generate feature embeddings:

[0118]

[0119] Where, represents a high-dimensional embedding vector;

[0120] This process is to transform the original time series into a vector representation rich in semantic information and compact structure, so that the model can learn time series dynamics, long-term dependencies and complex patterns more efficiently.

[0121] Step 4-3, performing self-supervised learning on the open source dataset to obtain a pre-trained model to extract common features related to the rehabilitation assessment task; specifically, including:

[0122] Step 4-3-1, select the contrastive learning framework;

[0123] Step 4-3-2: Based on the feature embedding generation result, execute a positive sample generation strategy to generate positive samples; the positive sample generation strategy is: generate positive sample pairs through at least two data enhancement methods;

[0124] Here, there are temporal reversal, random cropping, signal jittering, noise injection, time wrap, etc.

[0125] This process is to define the irrelevant changes (invariance) that the model should ignore and the semantic core (variance) that should be retained. By controlling the difficulty of generating positive samples, the model's discriminative ability is improved.

[0126] Step 4-3-3, construct the objective function L SSL :

[0127]

[0128] Where N is the total batch of input signal segments, M is the total number of candidate samples, Represents The corresponding positive sample, represents a randomly selected negative sample; is a similarity function, such as cosine similarity: ;

[0129] Step 4-3-4, the objective function L SSL Optimization is performed to maximize the similarity of positive sample pairs and minimize the similarity of negative sample pairs, thereby obtaining a pre-trained model.

[0130] Furthermore, in one embodiment, step 5 specifically includes:

[0131] Step 5-1: Generate feature embedding for the task dataset, specifically:

[0132] The rehabilitation data collected by each sensor channel is used as the input signal segment , input encoder Generate feature embeddings:

[0133]

[0134] Where, represents a high-dimensional embedding vector;

[0135] Step 5-2: Based on the result of step 5-1 and the pre-trained model obtained in step 4, transfer learning is performed to generate a pre-trained model corresponding to the task dataset;

[0136] Step 5-3, establish the transfer learning loss function :

[0137]

[0138] in, is the classification cross entropy loss function:

[0139]

[0140] is the regularization term:

[0141]

[0142] Where, is the true label, is the predicted probability, N represents the total batch of input signal segments, is the encoder parameter;

[0143] Step 5-4, optimize the pre-trained model generated in step 5-2: update the parameters of the pre-trained model using stochastic gradient descent or Adam optimizer by minimizing the transfer learning loss function:

[0144]

[0145]

[0146] Where t represents the number of optimization iterations, Represents the classification head, is the learning rate, and are the gradients of the encoder and classification head, respectively.

[0147] Here, in some embodiments, the process of optimizing the pre-trained model generated in step 5-2 in step 5-4 further includes: adjusting the output of the pre-trained model by freezing and unfreezing the encoder parameters. Specifically:

[0148] Freeze encoder parameters: fix the encoder parameters in the early stages and only update the classification head :

[0149]

[0150] Unfreeze encoder parameters: In the subsequent stages, the encoder parameters are gradually unfrozen, allowing and Also optimize:

[0151]

[0152] The fine-tuning process is achieved by minimizing Optimize the model to suit specific rehabilitation movement assessment tasks.

[0153] Furthermore, in one embodiment, step 6 establishes a multi-source sensor model fusion strategy for a single subtask based on the rehabilitation assessment task, and utilizes the strategy to fuse the features of the data from each sensor channel, specifically including:

[0154] Step 6-1: Build a multi-attention model. For a single subtask, perform embedding (splicing) operations on the feature vectors of different sensor channels extracted in step 5 to generate an input vector:

[0155]

[0156] Where, represents the feature vector of the nth sensing channel, where n represents the number of sensing channels;

[0157] Step 6-2, for each attention , calculate the sensing channel coding:

[0158]

[0159] in,

[0160]

[0161]

[0162] Split the Q, K, and V matrices into multiple subspaces according to the number of heads h, and each subspace corresponds to an attention head:

[0163]

[0164] Where, is the key vector dimension scaling factor, Q is the query vector, K is the key vector, V is the value vector, is the activation function, , , , is the output linear transformation matrix, , represents the feature dimension of the model, and It is the dimension set to adapt to the computing needs;

[0165] Step 6-3: concatenate and linearly transform the encoded feature vectors output by each sensor channel to obtain the final fused feature vector :

[0166]

[0167] Where, Represents the concatenation function.

[0168] Furthermore, in one embodiment, the step 8 of establishing a multi-task evaluation system specifically includes:

[0169] Step 8-1, assigning a separate subtask evaluation system to each subtask based on the subtask characteristics in step 6;

[0170] Step 8-2: Input each subtask feature into the corresponding subtask evaluation system to obtain the output of the subtask evaluation system;

[0171] Step 8-3: Create all subtask features corresponding to the main task The multi-task evaluation system for fusion is:

[0172]

[0173] in,

[0174]

[0175]

[0176]

[0177] Where, is the weight factor of the evaluation system for the i-th subtask, is the output of the i-th subtask evaluation system, m is the total number of subtask evaluation systems, is the output of the multi-task evaluation system, is the normal distribution function, is the activation function, is the noise of each subtask feature component, is the pre-trained weight of the component.

[0178] Preferably, in some embodiments, the weight factors of the subtask evaluation system need to satisfy:

[0179]

[0180]

[0181] in, Subtask features The balance weight factor, is the preset scaling factor, The input feature vector is the subtask feature The correlation coefficient of represents the balanced loss function.

[0182] Furthermore, in one embodiment, the method further comprises:

[0183] Step 10: Generate a comprehensive evaluation result based on the evaluation results of each main task.

[0184] Here, in some embodiments, step 10 specifically includes:

[0185] Step 10-1, feature mapping the evaluation results of each main task;

[0186] Each main task will generate a main task evaluation result, such as:

[0187]

[0188] Where, The dimension of the result vector of each main task is not necessarily the same. After feature mapping, the main task result becomes ;

[0189] The form is as follows

[0190]

[0191] in is the matrix dimension of the comprehensive evaluation task.

[0192] Step 10-2: Establish a decision output model and input the feature mapping results into the decision output model to obtain a comprehensive evaluation result; the main task evaluation result and the comprehensive evaluation result mapping diagram are shown in FIG. Figure 2 As shown;

[0193] The decision output model is:

[0194]

[0195] Where, Represents the prediction result of the maximum likelihood output, is the output result vector of the main task, Indicates the comprehensive evaluation results. is the conditional probability, which means belong The probability of the category, is the feature mapping result corresponding to the jth main task, and q represents the total number of main tasks.

[0196] Here, preferably, the Beyes decision method is used to establish the decision output model. Depending on the task, DS theory, addition, summation, voting and other strategies may also be used.

[0197] In one embodiment, a multimodal rehabilitation assessment system is provided, comprising:

[0198] The first module is used to confirm the rehabilitation assessment tasks, including main tasks and subtasks;

[0199] The second module is used to collect multi-source data related to rehabilitation of the subject through multiple sensor channels;

[0200] A third module is used to pre-process the multi-source data to correct data quality and form a task data set;

[0201] The fourth module is used to train on the corresponding open source dataset according to the sensor channel type and rehabilitation assessment task type to obtain a pre-trained model;

[0202] A fifth module is configured to further optimize and train the pre-trained model using the task dataset to adapt it to the characteristics of the rehabilitation data in the task dataset and the requirements of the rehabilitation assessment task, and to extract feature vectors of the data from each sensor channel using the optimized model;

[0203] The sixth module is used to establish a multi-source sensor model fusion strategy for a single subtask based on the rehabilitation assessment task, and use this strategy to fuse the features of the data from each sensor channel and use the fused features as the subtask features;

[0204] The seventh module is used to pass the subtask features through an independent fully connected module to output the evaluation results of each subtask;

[0205] The eighth module is used to establish a multi-task evaluation system to integrate the features of all sub-tasks corresponding to the main task;

[0206] The ninth module is used to use the features fused by the eighth module as the main task features, and output the evaluation results of each main task through an independent fully connected module and MLP model.

[0207] Furthermore, in one embodiment, the system further comprises:

[0208] The tenth module is used to generate comprehensive evaluation results based on the evaluation results of each main task.

[0209] For the specific definition of the multimodal rehabilitation assessment system, please refer to the definition of the multimodal rehabilitation assessment method above and will not be repeated here. The various modules in the above-mentioned multimodal rehabilitation assessment system can be implemented in whole or in part through software, hardware, or a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in hardware form, or can be stored in the memory of the computer device in software form, so that the processor can call and execute the corresponding operations of the above-mentioned modules.

[0210] In one embodiment, a computer device is provided, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the following is achieved:

[0211] Step 1: Confirm the rehabilitation assessment tasks, including main tasks and subtasks;

[0212] Step 2, collecting multi-source data related to rehabilitation of the subject through multiple sensor channels;

[0213] Step 3: preprocessing the multi-source data to correct data quality and form a task data set;

[0214] Step 4: Based on the sensor channel type and rehabilitation assessment task type, train on the corresponding open source dataset to obtain a pre-trained model;

[0215] Step 5: further optimizing and training the pre-trained model using the task dataset to adapt it to the characteristics of the rehabilitation data in the task dataset and the requirements of the rehabilitation assessment task, and extracting feature vectors of the data from each sensor channel using the optimized model;

[0216] Step 6: Establish a multi-source sensor model fusion strategy for a single subtask based on the rehabilitation assessment task, and use this strategy to fuse the features of the data from each sensor channel, and use the fused features as the subtask features;

[0217] Step 7: Pass the subtask features through an independent fully connected module to output the evaluation results of each subtask;

[0218] Step 8: Establish a multi-task evaluation system to fuse all sub-task features corresponding to the main task;

[0219] Step 9: The features fused in step 8 are used as the main task features. After passing through independent fully connected modules and MLP models, the evaluation results of each main task are output.

[0220] Step 10: Generate a comprehensive evaluation result based on the evaluation results of each main task.

[0221] For the specific limitations of each step, please refer to the limitations of the multimodal rehabilitation assessment method above and will not be repeated here.

[0222] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the computer program implements:

[0223] Step 1: Confirm the rehabilitation assessment tasks, including main tasks and subtasks;

[0224] Step 2, collecting multi-source data related to rehabilitation of the subject through multiple sensor channels;

[0225] Step 3: preprocessing the multi-source data to correct data quality and form a task data set;

[0226] Step 4: Based on the sensor channel type and rehabilitation assessment task type, train on the corresponding open source dataset to obtain a pre-trained model;

[0227] Step 5: further optimizing and training the pre-trained model using the task dataset to adapt it to the characteristics of the rehabilitation data in the task dataset and the requirements of the rehabilitation assessment task, and extracting feature vectors of the data from each sensor channel using the optimized model;

[0228] Step 6: Establish a multi-source sensor model fusion strategy for a single subtask based on the rehabilitation assessment task, and use this strategy to fuse the features of the data from each sensor channel, and use the fused features as the subtask features;

[0229] Step 7: Pass the subtask features through an independent fully connected module to output the evaluation results of each subtask;

[0230] Step 8: Establish a multi-task evaluation system to fuse all sub-task features corresponding to the main task;

[0231] Step 9: The features fused in step 8 are used as the main task features. After passing through independent fully connected modules and MLP models, the evaluation results of each main task are output.

[0232] Step 10: Generate a comprehensive evaluation result based on the evaluation results of each main task.

[0233] For the specific limitations of each step, please refer to the limitations of the multimodal rehabilitation assessment method above and will not be repeated here.

[0234] The above shows and describes the basic principles, main features and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The above embodiments and descriptions are only illustrative of the principles of the present invention. Without departing from the spirit and scope of the present invention, any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention shall be included in the scope of protection of the present invention.

Claims

1. A multimodal rehabilitation assessment method, characterized in that: The method comprises: Step 1: Confirm the rehabilitation assessment tasks, including main tasks and subtasks; Step 2: collecting multi-source data related to rehabilitation of the subject through multiple sensor channels; Step 3: preprocessing the multi-source data to correct data quality and form a task data set; Step 4: Based on the sensor channel type and rehabilitation assessment task type, train on the corresponding open source dataset to obtain a pre-trained model; Step 5: further optimizing and training the pre-trained model using the task dataset to adapt it to the characteristics of the rehabilitation data in the task dataset and the requirements of the rehabilitation assessment task, and extracting feature vectors of the data from each sensor channel using the optimized model; Step 6: Establish a multi-source sensor model fusion strategy for a single subtask based on the rehabilitation assessment task, and use this strategy to fuse the features of the data from each sensor channel, and use the fused features as the subtask features; Step 7: Pass the subtask features through an independent fully connected module to output the evaluation results of each subtask; Step 8: Establish a multi-task evaluation system to fuse all sub-task features corresponding to the main task; Step 9: The features fused in step 8 are used as the main task features. After passing through independent fully connected modules and MLP models, the evaluation results of each main task are output. The main task in step 1 includes at least a rehabilitation assessment table, and each attribute in the table serves as a subtask; the rehabilitation assessment table includes a clinical scale or a clinical scale customized based on the clinical scale; Step 8 describes establishing a multi-task evaluation system, specifically including: Step 8-1, assigning a separate subtask evaluation system to each subtask based on the subtask characteristics in step 6; Step 8-2: Input each subtask feature into the corresponding subtask evaluation system to obtain the output of the subtask evaluation system; Step 8-3: Create all subtask features corresponding to the main task The multi-task evaluation system for fusion is: ； in, ； ； ； Where, is the weight factor of the evaluation system for the i-th subtask, is the output of the i-th subtask evaluation system, m is the total number of subtask evaluation systems, is the output of the multi-task evaluation system, is the normal distribution function, is the activation function, is the noise of each subtask feature component, is the pre-trained weight of the component; The weight factors of the subtask evaluation system need to satisfy: ； ； in, Subtask features The balance weight factor, is the preset scaling factor, The input feature vector is the subtask feature The correlation coefficient of represents the balanced loss function.

2. The multimodal rehabilitation assessment method according to claim 1, characterized in that: In step 4, according to the sensor channel type and rehabilitation assessment task type, training is performed on the corresponding open source dataset to obtain a pre-trained model, which specifically includes: Step 4-1: Select a number of open source datasets with the same number of sensor channels according to the type of rehabilitation assessment task; Step 4-2: Generate feature embedding for the open source dataset, specifically: For each sensor channel type, the rehabilitation data in the corresponding open source dataset is used as the input signal segment , input encoder Generate feature embeddings: ； Where, represents a high-dimensional embedding vector; Step 4-3, performing self-supervised learning on the open source dataset to obtain a pre-trained model to extract common features related to the rehabilitation assessment task; specifically, including: Step 4-3-1, select the contrastive learning framework; Step 4-3-2: Based on the feature embedding generation result, execute a positive sample generation strategy to generate positive samples; the positive sample generation strategy is: generate positive sample pairs through at least two data enhancement methods; Step 4-3-3, construct the objective function L SSL : ； Where N is the total batch of input signal segments, M is the total number of candidate samples, Represents The corresponding positive sample, represents a randomly selected negative sample; is the similarity function; Step 4-3-4, for the objective function L SSL Optimization is performed to maximize the similarity of positive sample pairs and minimize the similarity of negative sample pairs, thereby obtaining a pre-trained model.

3. The multimodal rehabilitation assessment method according to claim 1, characterized in that: Step 5 specifically includes: Step 5-1: Generate feature embedding for the task dataset, specifically: The rehabilitation data collected by each sensor channel is used as the input signal segment , input encoder Generate feature embeddings: ； Where, represents a high-dimensional embedding vector; Step 5-2: Based on the result of step 5-1 and the pre-trained model obtained in step 4, transfer learning is performed to generate a pre-trained model corresponding to the task dataset; Step 5-3, establish the transfer learning loss function : ； in, is the classification cross entropy loss function: ； is the regularization term: ； Where, is the true label, is the predicted probability, N represents the total batch of input signal segments, is the encoder parameter; Step 5-4, optimize the pre-trained model generated in step 5-2: update the parameters of the pre-trained model using stochastic gradient descent or Adam optimizer by minimizing the transfer learning loss function: ； ； Where t represents the number of optimization iterations, Represents the classification head, is the learning rate, and are the gradients of the encoder and classification head, respectively.

4. The multimodal rehabilitation assessment method according to claim 3, characterized in that: The process of optimizing the pre-trained model generated in step 5-2 in step 5-4 also includes: adjusting the output result of the pre-trained model by freezing and unfreezing the encoder parameters.

5. The multimodal rehabilitation assessment method according to claim 1, characterized in that: Step 6 establishes a multi-source sensor model fusion strategy for a single subtask based on the rehabilitation assessment task, and uses this strategy to fuse the features of the data from each sensor channel, specifically including: Step 6-1: Build a multi-attention model. For a single subtask, perform embedding operations on the feature vectors of different sensor channels extracted in step 5 to generate input vectors: ； Where, represents the feature vector of the nth sensing channel, where n represents the number of sensing channels; Step 6-2, for each attention , calculate the sensing channel coding: ； in, ； ； Split the Q, K, and V matrices into multiple subspaces according to the number of heads h, and each subspace corresponds to an attention head: ； Where, is the key vector dimension scaling factor, Q is the query vector, K is the key vector, V is the value vector, is the activation function, , , , is the output linear transformation matrix, , represents the feature dimension of the model, and It is the dimension set to adapt to the computing needs; Step 6-3: concatenate and linearly transform the encoded feature vectors output by each sensor channel to obtain the final fused feature vector : ； Where, Represents the concatenation function.

6. The multimodal rehabilitation assessment method according to claim 1, characterized in that: The method further comprises: Step 10: Generate a comprehensive evaluation result based on the evaluation results of each main task.

7. The multimodal rehabilitation assessment method according to claim 6, characterized in that: Step 10 specifically includes: Step 10-1, feature mapping the evaluation results of each main task; Step 10-2: Establish a decision output model and input the feature mapping result into the decision output model to obtain a comprehensive evaluation result; The decision output model is: ； Where, Represents the prediction result of the maximum likelihood output, is the output result vector of the main task, Indicates the comprehensive evaluation results. is the conditional probability, which means belong The probability of the category, is the feature mapping result corresponding to the jth main task, and q represents the total number of main tasks.

8. A multimodal rehabilitation assessment system based on the method according to any one of claims 1 to 7, comprising: The first module is used to confirm the rehabilitation assessment tasks, including main tasks and subtasks; The second module is used to collect multi-source data related to rehabilitation of the subject through multiple sensor channels; A third module is used to pre-process the multi-source data to correct data quality and form a task data set; The fourth module is used to train on the corresponding open source dataset according to the sensor channel type and rehabilitation assessment task type to obtain a pre-trained model; A fifth module is configured to further optimize and train the pre-trained model using the task dataset to adapt it to the characteristics of the rehabilitation data in the task dataset and the requirements of the rehabilitation assessment task, and to extract feature vectors of the data from each sensor channel using the optimized model; The sixth module is used to establish a multi-source sensor model fusion strategy for a single subtask based on the rehabilitation assessment task, and use this strategy to fuse the features of the data from each sensor channel and use the fused features as the subtask features; The seventh module is used to pass the subtask features through an independent fully connected module to output the evaluation results of each subtask; The eighth module is used to establish a multi-task evaluation system to integrate the features of all sub-tasks corresponding to the main task; The ninth module is used to use the features fused by the eighth module as the main task features, and output the evaluation results of each main task through an independent fully connected module and MLP model.

9. The multimodal rehabilitation assessment system according to claim 8, characterized in that: The system further comprises: The tenth module is used to generate comprehensive evaluation results based on the evaluation results of each main task.

Images