Peripheral nerve rehabilitation evaluation system and method based on multi-modal large model

Abstract

The invention provides a peripheral nerve rehabilitation evaluation system and method based on a multi-modal large model, and the system comprises a data preprocessing module and a multi-modal data fusion module which are used for extracting key features from data of different modals; fusing the features of different modes by adopting a multi-feature fusion method; using the fused feature data to train a multi-modal large model; the trained multi-modal large model is evaluated by using an independent test data set; the rehabilitation evaluation module is used for evaluating the movement function of the patient according to the biomechanical data and the video image data; and evaluating the sensory function of the patient in combination with the surface electromyogram signal and the electroencephalogram signal. According to the technical scheme, data fusion and analysis are carried out based on the multi-modal large model, the rehabilitation condition of the patient can be evaluated more accurately, and the accuracy and reliability of evaluation are improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of medical rehabilitation, and in particular to a peripheral nerve rehabilitation evaluation system and method based on a multi-modal large model. BACKGROUND

[0002] The peripheral nervous system is composed of cranial nerves, spinal nerves and autonomic nerves, including nerve fibers, ganglia, nerve plexuses, peripheral nerve trunks and peripheral nerve endings. When the peripheral nervous system is damaged, patients may experience motor disorders, sensory abnormalities, autonomic nervous dysfunction and other symptoms, which seriously affect the quality of life. Traditional rehabilitation evaluation methods mainly rely on the experience and manual operation of professionals, and have the problems of strong subjectivity, low efficiency, lack of comprehensiveness and accuracy. With the development of artificial intelligence technology, multi-modal large models have shown great capabilities in data processing and analysis, providing new ideas for solving the shortcomings of traditional rehabilitation evaluation methods. SUMMARY

[0003] The present application provides a peripheral nerve rehabilitation evaluation system and method based on a multi-modal large model to improve the accuracy and reliability of the evaluation.

[0004] In a first aspect, a peripheral nerve rehabilitation evaluation system based on a multi-modal large model is provided, comprising: A data preprocessing module is configured to clean the collected multi-modal data, remove noise and outliers, normalize different types of data, align the collection time and frequency of different modal data, and enhance the multi-modal data. A multi-modal data fusion module is configured to extract key features from different modal data, fuse features of different modalities using multiple feature fusion methods, train a multi-modal large model using the fused feature data, and evaluate the trained multi-modal large model using an independent test data set. A rehabilitation evaluation module is configured to evaluate the patient's motor function based on biomechanical data and video image data, evaluate the patient's sensory function by combining surface electromyography and electroencephalography, evaluate the patient's autonomic nervous function based on physiological indicators, and comprehensively evaluate the patient's overall rehabilitation by considering the evaluation results of motor function, sensory function and autonomic nervous function.

[0005] In the above technical solution, by setting the data preprocessing module, the collected multi-modal data is cleaned to remove noise and outliers; different types of data are normalized; the collection time and frequency of different modal data are aligned; the multi-modal data is enhanced; the multi-modal data fusion module is used to extract key features from different modal data; a variety of feature fusion methods are used to fuse features of different modalities; the fused feature data is used to train a multi-modal large model; and the trained multi-modal large model is evaluated using independent test data set; the rehabilitation evaluation module is used to evaluate the patient's motor function according to the biomechanical data and video image data; the patient's sensory function is evaluated by combining surface electromyographic signals and electroencephalographic signals; the patient's autonomic nervous function is evaluated according to the patient's physiological indicators; and the overall rehabilitation of the patient is comprehensively evaluated by considering the evaluation results of motor function, sensory function and autonomic nervous function; based on the multi-modal large model, data fusion and analysis can more accurately evaluate the patient's rehabilitation, improving the accuracy and reliability of the evaluation.

[0006] In a specific embodiment, it also includes: a data acquisition module for acquiring joint range of motion parameters, muscle strength grade parameters and gait cycle parameters of patients using biomechanical sensors; using surface electromyographic signal analysis technology to acquire the electrical activity signals of the patient's muscles; using electroencephalographic topographic mapping technology to acquire the neural electrical activity signals of the patient's brain; using a high-definition camera to acquire the patient's rehabilitation training video images, recording the patient's action posture information and motion trajectory information; obtaining the multi-modal data.

[0007] In a specific embodiment, it also includes: A rehabilitation training scheme generation module for developing a personalized rehabilitation training scheme for the patient according to the patient's rehabilitation evaluation results, combining the patient's age, gender, and disease severity factors; setting the training parameters according to the rehabilitation training scheme; recommending suitable training equipment according to the rehabilitation training scheme and training parameters; and developing a rehabilitation training schedule to clearly define the training goals and tasks for each stage.

[0008] In a specific embodiment, it also includes: A rehabilitation training monitoring module for real-time acquisition of patient training data during the patient's rehabilitation training process; monitoring the patient's training status according to the real-time acquired data; periodically evaluating the patient's rehabilitation training effect; and remotely monitoring the patient's rehabilitation training.

[0009] In a specific embodiment, it also includes: A system management module for managing users of the system; managing data in the system; generating various types of reports according to user needs.

[0010] In a second aspect, a multi-modal large model-based peripheral nerve rehabilitation evaluation method is provided, comprising the following steps: The data preprocessing module is used to clean the collected multi-modal data, remove noise and outliers, normalize different types of data, align the collection time and frequency of different modal data, and enhance the multi-modal data; The multi-modal data fusion module extracts key features from different modal data, uses multiple feature fusion methods to fuse features of different modalities, trains a multi-modal large model using the fused feature data, and evaluates the trained multi-modal large model using an independent test data set; The rehabilitation evaluation module evaluates the patient's motor function based on biomechanical data and video image data, evaluates the patient's sensory function in combination with surface electromyographic signals and electroencephalographic signals, evaluates the patient's autonomic nervous function based on physiological indicators, and comprehensively evaluates the patient's overall rehabilitation based on the evaluation results of motor function, sensory function and autonomic nervous function.

[0011] In a specific embodiment, it further comprises: The data acquisition module uses biomechanical sensors to collect joint range of motion parameters, muscle strength grade parameters and gait cycle parameters of the patient, uses surface electromyographic signal analysis technology to collect muscle electrical activity signals of the patient, uses electroencephalographic topographic mapping technology to collect neural electrical activity signals of the patient's brain, uses a high-definition camera to collect rehabilitation training video images of the patient, records the patient's action posture information and motion trajectory information, and obtains the multi-modal data.

[0012] In a specific embodiment, it further comprises: The rehabilitation training scheme generation module formulates a personalized rehabilitation training scheme for the patient based on the patient's rehabilitation evaluation results, in combination with the patient's age, gender and disease severity factors, sets training parameters according to the rehabilitation training scheme, recommends suitable training equipment according to the rehabilitation training scheme and training parameters, and formulates a rehabilitation training progress plan, clearly defining training goals and tasks at each stage.

[0013] In a specific embodiment, it further comprises: The rehabilitation training monitoring module collects training data of the patient in real time during the patient's rehabilitation training process, monitors the patient's training status based on the real-time collected data, periodically evaluates the patient's rehabilitation training effect, and remotely monitors the patient's rehabilitation training.

[0014] In a specific embodiment, it further comprises: The system management module is used for managing users of the system, managing data in the system, and generating various types of reports according to the needs of the users.

[0015] In the above technical solution, by setting the data preprocessing module, the collected multi-modal data is cleaned to remove noise and outliers; different types of data are normalized; the collection time and frequency of different modal data are aligned; the multi-modal data is enhanced; the multi-modal data fusion module is used to extract key features from different modal data; a plurality of feature fusion methods are used to fuse features of different modalities; the fused feature data is used to train a multi-modal large model; and the trained multi-modal large model is evaluated using an independent test data set; the rehabilitation evaluation module is used to evaluate the motor function of the patient according to the biomechanical data and video image data; the sensory function of the patient is evaluated in combination with the surface electromyography signal and the electroencephalogram signal; the autonomic nervous function of the patient is evaluated according to the physiological indicators; and the overall rehabilitation of the patient is comprehensively evaluated by considering the evaluation results of the motor function, the sensory function and the autonomic nervous function; the data fusion and analysis based on the multi-modal large model can more accurately evaluate the rehabilitation of the patient, and the accuracy and reliability of the evaluation are improved. BRIEF DESCRIPTION OF DRAWINGS

[0016] Figure 1 A structural block diagram of a peripheral nerve rehabilitation evaluation system based on a multi-modal large model is provided for the embodiments of the present application. Figure 2 A flowchart of a peripheral nerve rehabilitation evaluation method based on a multi-modal large model is provided for the embodiments of the present application. DETAILED DESCRIPTION

[0017] The present application will be further described in detail by the accompanying drawings and embodiments. Through these descriptions, the features and advantages of the present application will become clearer and more explicit.

[0018] The word "exemplary" is used herein to mean "serving as an example, instance, or illustration." Any implementation described herein as "exemplary" is not necessarily to be construed as preferred or advantageous over other implementations. Unless specifically stated otherwise, the drawings are not drawn to scale and are merely intended to conceptually illustrate aspects of the embodiments.

[0019] In addition, the technical features involved in different embodiments of the present application described below can be combined with each other as long as there is no conflict.

[0020] To facilitate the understanding of the peripheral nerve rehabilitation evaluation system and method based on a multi-modal large model provided by the embodiments of the present application, the application scenario is first described. The peripheral nerve rehabilitation evaluation system and method based on a multi-modal large model provided by the embodiments of the present application are used to improve the accuracy and reliability of the evaluation. The peripheral nervous system is composed of cranial nerves, spinal nerves and autonomic nerves, including nerve fibers, ganglia, nerve plexuses, peripheral nerve trunks and peripheral nerve devices. When the peripheral nervous system is damaged, patients may have symptoms such as motor disorders, sensory abnormalities, autonomic nervous function disorders, and the like, which seriously affect the quality of life. Traditional rehabilitation evaluation methods mainly rely on the experience and manual operation of professionals, and have problems such as strong subjectivity, low efficiency, lack of comprehensiveness and accuracy. With the development of artificial intelligence technology, multi-modal large models have shown great capabilities in data processing and analysis, providing a new way to solve the shortcomings of traditional rehabilitation evaluation methods. Therefore, the embodiments of the present application provide a peripheral nerve rehabilitation evaluation system and method based on a multi-modal large model to improve the accuracy and reliability of the evaluation. The embodiments will be described in detail below with reference to the specific drawings.

[0021] Reference Figure 1 and Figure 2 , Figure 1 The structure block diagram of the peripheral nerve rehabilitation evaluation system based on a multi-modal large model provided by the embodiments of the present application is shown in FIG. 1. Figure 2 The flowchart of the peripheral nerve rehabilitation evaluation method based on a multi-modal large model provided by the embodiments of the present application is shown in FIG. 2.

[0022] In Figure 1 the embodiments of the present application, a peripheral nerve rehabilitation evaluation system based on a multi-modal large model is provided, which comprises: a data preprocessing module for cleaning the collected multi-modal data, removing noise and outliers; normalizing different types of data; performing data alignment processing on the collection time and frequency of different modal data; and performing enhancement processing on the multi-modal data; a multi-modal data fusion module for extracting key features from different modal data respectively; using a plurality of feature fusion methods to fuse features of different modalities; training a multi-modal large model using the fused feature data; and evaluating the trained multi-modal large model using an independent test data set; a rehabilitation evaluation module for evaluating the motor function of a patient according to biomechanical data and video image data; evaluating the sensory function of a patient in combination with surface electromyography signals and electroencephalography signals; evaluating the autonomic nervous function of a patient according to physiological indicators; and comprehensively evaluating the overall rehabilitation of a patient by considering the evaluation results of motor function, sensory function and autonomic nervous function.

[0023] In the above technical solution, by setting the data preprocessing module, the collected multi-modal data is cleaned to remove noise and outliers; different types of data are normalized; the collection time and frequency of different modal data are aligned; the multi-modal data is enhanced; the multi-modal data fusion module is used to extract key features from different modal data; a variety of feature fusion methods are used to fuse features of different modalities; the fused feature data is used to train a multi-modal large model; and the trained multi-modal large model is evaluated using independent test data sets; the rehabilitation evaluation module is used to evaluate the patient's motor function according to biomechanical data and video image data; the patient's sensory function is evaluated by combining surface electromyographic signals and electroencephalographic signals; the patient's autonomic nervous function is evaluated according to the patient's physiological indicators; and the overall rehabilitation of the patient is comprehensively evaluated by considering the evaluation results of motor function, sensory function and autonomic nervous function; based on the multi-modal large model, data fusion and analysis can more accurately evaluate the patient's rehabilitation, improving the accuracy and reliability of the evaluation.

[0024] Specifically, the beneficial effects include: 1. Improve evaluation accuracy Comprehensive data fusion analysis Traditional peripheral nerve rehabilitation evaluation often relies on single modal data, such as only through the doctor's clinical examination or simple physiological indicator measurement, which is difficult to fully reflect the patient's rehabilitation condition. This system integrates multi-source data such as biomechanical data, video image data, surface electromyographic signals, electroencephalographic signals and patient's physiological indicators with the help of multi-modal large model. For example, biomechanical data can accurately reflect joint range of motion and muscle strength changes, video image data can capture patient's action posture and motion trajectory, surface electromyographic signals can reveal muscle electrical activity, and electroencephalographic signals can reflect the activation degree of brain nerve function. The multi-modal large model deeply fuses and analyzes these rich data, avoiding the limitations of single data source, and comprehensively evaluates the patient's rehabilitation from multiple dimensions, greatly improving the accuracy of the evaluation.

[0025] Precise feature extraction and fusion After data preprocessing, the multi-modal data fusion module extracts key features from different modalities of data. These features, carefully selected, accurately represent the core information of each modality of data. For example, the joint angle change features extracted from biomechanical data accurately reflect the degree of joint function recovery; the time and frequency domain features extracted from surface electromyography signals can effectively evaluate muscle contraction ability and neuromuscular transmission function. Subsequently, various advanced feature fusion methods, such as weighted fusion and feature fusion networks in deep learning, are used to organically fuse features from different modalities. This fusion method fully considers the correlation and complementarity between features from different modalities, making the fused features more comprehensive and accurate in describing the patient's rehabilitation status, providing a solid data foundation for subsequent rehabilitation evaluation.

[0026] 2. Enhance evaluation reliability Data preprocessing ensures quality The data preprocessing module comprehensively processes the collected multi-modal data, including cleaning, normalization, alignment, and enhancement. Data cleaning removes noise and outliers to ensure data purity and avoid interference factors affecting the evaluation results. For example, during biomechanical data collection, external environmental interference may produce abnormal fluctuation values, which can be effectively removed through data cleaning. Normalization processing unifies different types of data to the same dimension and range, making different modalities of data comparable, facilitating subsequent fusion and analysis. Data alignment solves the problem of inconsistent data collection time and frequency, ensuring data consistency and synchronization on the time axis, and improving the reliability of evaluation results. Data augmentation increases the diversity and richness of data, improving the generalization ability of the model, so that the system can give accurate evaluation when facing different patients and complex situations.

[0027] Model evaluation and verification After training the multi-modal large model using the fused feature data, an independent test data set is used to evaluate the trained model. Various evaluation indicators such as accuracy, recall rate, and F1 value are used to comprehensively and objectively evaluate the performance and effect of the model. This strict evaluation and verification mechanism ensures that the multi-modal large model has high reliability and stability, and can accurately evaluate the patient's rehabilitation status in actual application, providing reliable reference for doctors and therapists.

[0028] Optimize rehabilitation decisions Comprehensive evaluation guides program The rehabilitation evaluation module not only evaluates the motor function, sensory function and autonomic nervous function of the patient separately, but also comprehensively considers the evaluation results of these three aspects to comprehensively evaluate the overall rehabilitation of the patient. This comprehensive evaluation method can fully understand the rehabilitation status of the patient and find the mutual influence and correlation between different functions. For example, sensory dysfunction may affect the recovery of motor function, and autonomic nervous dysfunction may adversely affect the overall rehabilitation process. Based on the comprehensive evaluation results, doctors and therapists can develop more scientific and reasonable rehabilitation training programs, develop personalized rehabilitation plans for the specific problems of patients, and improve the effectiveness and efficiency of rehabilitation treatment.

[0029] In a specific implementable embodiment, it further comprises a data acquisition module for acquiring joint range of motion parameters, muscle strength level parameters and gait cycle parameters of the patient using biomechanical sensors; acquiring muscle electrical activity signals of the patient using surface electromyography analysis technology; acquiring neural electrical activity signals of the brain of the patient using electroencephalographic topography technology; acquiring rehabilitation training video images of the patient using a high-definition camera, recording the action posture information and motion trajectory information of the patient; obtaining the multi-modal data.

[0030] Specifically, the beneficial effects include: Data comprehensiveness is improved: The data acquisition module uses multiple technical means to obtain patient information from different dimensions. Biomechanical sensors can accurately collect joint range of motion, muscle strength level and gait cycle parameters, directly reflecting the physical state of the patient's motor system; surface electromyography analysis technology can capture muscle electrical activity signals to deeply understand muscle function and neuromuscular transmission; electroencephalographic topography technology can obtain brain neural electrical activity signals to understand the functional status of the nervous system; high-definition cameras record rehabilitation training video images to present the patient's action posture and motion trajectory. Multiple data complement each other, comprehensively cover the key information required for peripheral nerve rehabilitation evaluation, and avoid the one-sidedness of single data source, laying a solid foundation for subsequent accurate evaluation.

[0031] Evaluation objectivity is enhanced: Compared with the traditional evaluation method relying on the subjective judgment of doctors, this module uses advanced sensors and imaging technology to record the indicators of the patient with objective data and images. These data are not disturbed by human factors, have high accuracy and reliability, and can truly reflect the rehabilitation status of the patient, making the evaluation results more objective and fair, and providing a strong basis for developing scientific and reasonable rehabilitation programs.

[0032] In a specific implementable embodiment, it further comprises: The rehabilitation training scheme generation module is configured to formulate a personalized rehabilitation training scheme for the patient according to the rehabilitation evaluation result of the patient, in combination with the age, gender, and severity of the patient's condition; set training parameters according to the rehabilitation training scheme; recommend suitable training equipment according to the rehabilitation training scheme and the training parameters; and formulate a rehabilitation training progress plan, and clearly define the training targets and tasks of each stage.

[0033] Specifically, the beneficial effects include: Personalized adaptation improves the effect: The rehabilitation training scheme generation module fully considers the individual differences of the patient, formulates a personalized scheme according to the rehabilitation evaluation result, in combination with the age, gender, and severity of the patient's condition. The physical functions and recovery abilities of patients of different ages and genders are different, and the severity of the condition directly affects the training intensity and method. This precise adaptation can ensure that the training scheme meets the actual needs of the patient, avoids the "one-size-fits-all" mode, effectively stimulates the physical potential of the patient, speeds up the rehabilitation process, and improves the rehabilitation effect.

[0034] Training guidance is precise and scientific: The module not only formulates a scheme, but also sets training parameters, recommends equipment, and plans progress. Clear training parameters provide standards for training intensity, frequency, etc., ensuring the scientificity and standardization of training; appropriate training equipment recommendations can assist the patient in better completing training movements and reduce the risk of injury; a clear progress plan clearly defines the goals and tasks of each stage, allowing the patient and medical staff to have a clear understanding of the rehabilitation process, facilitating timely adjustment and optimization of training, and making rehabilitation training more systematic and coherent.

[0035] In one specific implementation scheme, it further includes: The rehabilitation training monitoring module is configured to collect training data of the patient in real time during the rehabilitation training of the patient; monitor the training state of the patient according to the real-time collected data; periodically evaluate the rehabilitation training effect of the patient; and remotely monitor the rehabilitation training of the patient.

[0036] Specifically, the beneficial effects include: Real-time feedback ensures training safety and effectiveness: The rehabilitation training monitoring module collects training data in real time and monitors the training state, which can timely discover abnormalities in the patient's training, such as non-standard movements, excessive exercise intensity, or body discomfort, etc. Once a problem is found, it can be immediately adjusted to avoid secondary damage due to improper training and ensure training safety. At the same time, real-time data can enable medical staff to accurately grasp the patient's current training performance, provide timely guidance and encouragement, ensure that training movements are standardized and intensity is reasonable, and improve training effectiveness.

[0037] Periodic evaluation of optimized training programs: Regularly evaluate the effectiveness of rehabilitation training to intuitively understand the progress of patients and determine whether the current training program is effective. Based on the evaluation results, training parameters can be adjusted, training methods or equipment can be replaced, and the training program can always meet the needs of patient rehabilitation to avoid ineffective training and improve rehabilitation efficiency.

[0038] Remote monitoring improves the convenience of medical services: Remote monitoring breaks the time and space constraints, allowing doctors and therapists to view training data and status at any time without being constantly present with the patient, and providing timely remote guidance and advice. This is especially convenient for patients in remote areas, allowing high-quality medical resources to be more widely utilized.

[0039] In one specific implementation, it also includes: A system management module for managing users of the system, managing data in the system, and generating various types of reports based on user needs.

[0040] Specifically, the beneficial effects include: User management ensures system security and order: The system management module manages users in detail and can set permissions for different roles (such as doctors, therapists, and patients). Doctors and therapists can access and process relevant patient data, develop and adjust rehabilitation programs based on their permissions; patients can only view their own rehabilitation information. This hierarchical management effectively prevents data leakage and misuse, ensures the safe and stable operation of the system, and ensures that users can efficiently use the system within their responsibilities, improving overall work order and efficiency.

[0041] Data management enables efficient use: Comprehensive data management in the system includes data storage, backup, and recovery. Scientific storage makes data classification clear and easy to query, making it easy for medical staff to quickly access the information they need; regular backups prevent data loss and ensure data integrity; data can be recovered in a timely manner when problems arise, ensuring the normal operation of the system. This provides reliable data support for rehabilitation assessment and training, promoting the deep mining and utilization of data.

[0042] Report generation aids decision analysis: Generate various reports based on user needs, such as rehabilitation assessment reports and training effect reports. These reports present data in intuitive charts and text, helping medical staff quickly understand patient conditions and providing strong evidence for developing and adjusting rehabilitation strategies, improving the scientificity and accuracy of decision-making.

[0043] In a specific and implementable embodiment, the system is based on a multi-modal large model, integrating multiple data collection methods, including biomechanical sensors, surface electromyography analysis, electroencephalography topography, video images, etc., and can comprehensively obtain rehabilitation information of patients. Through deep fusion and analysis of multi-modal data, the rehabilitation status of patients with peripheral nervous system diseases is accurately evaluated, and personalized rehabilitation training programs are provided for patients. Including: 1. Data collection module Biomechanical data collection unit: biomechanical sensors are used to collect joint range of motion, muscle strength level, gait cycle and other parameters of patients. For example, during the patient's walking process, through the pressure sensors installed on the soles and the angle sensors at the joints, the gait information and joint activity of the patient are obtained in real time.

[0044] Surface electromyography signal collection unit: surface electromyography signal analysis technology is used to collect the electrical activity signals of the patient's muscles. By pasting electrode patches on the surface of the relevant muscles of the patient, the electrical signal changes during muscle contraction are recorded to reflect the functional status of the muscles.

[0045] Electroencephalography signal collection unit: electroencephalography technology is used to collect the neural electrical activity signals of the patient's brain. By wearing an electroencephalography cap, the electroencephalography signals of different brain regions of the patient are collected to analyze the activation degree of the brain neural function.

[0046] Video image collection unit: high-definition cameras are used to collect video images of the patient's rehabilitation training, recording the patient's action posture, motion trajectory and other information. For example, when the patient performs upper limb rehabilitation training, the camera captures the motion of the upper limb in real time, providing a basis for subsequent motion analysis and evaluation.

[0047] 2. Data preprocessing module Data cleaning unit: the collected multi-modal data is cleaned to remove noise and outliers. For example, for abnormal fluctuation values in biomechanical data, reasonable threshold values are set for screening and correction.

[0048] Data normalization unit: different types of data are normalized to have the same dimension and range. For example, joint range of motion data and muscle strength level data are normalized to the [0, 1] interval, which facilitates subsequent data fusion and analysis.

[0049] Data alignment unit: due to the differences in collection time and frequency of different modal data, data alignment processing is needed. Dynamic time warping (DTW) algorithm is used to align different modal data on the time axis to ensure the consistency and synchronization of the data.

[0050] Data augmentation unit: In order to increase the diversity and richness of data, the original data is enhanced. For example, rotate, flip, scale, etc. operations are performed on video image data to generate more training samples and improve the generalization ability of the model.

[0051] 3. Multimodal data fusion module Feature extraction unit: Extract key features from different modalities of data. For example, extract joint angle change features and muscle strength change features from biomechanical data; extract time domain features and frequency domain features from surface electromyography signals; extract electroencephalogram power spectrum features from electroencephalogram signals.

[0052] Feature fusion unit: Use multiple feature fusion methods to fuse features from different modalities. For example, use weighted fusion method to give different weights according to the importance and correlation of different modalities of features, and linearly combine the features; or use deep learning feature fusion network, such as convolutional neural network (CNN) and long short-term memory network (LSTM) combined model, to realize deep fusion of features.

[0053] Model training unit: Use the fused feature data to train the multimodal large model. Use deep learning frameworks such as TensorFlow or PyTorch to build the structure of the multimodal large model, and use a large amount of labeled data to train and optimize the parameters of the model.

[0054] Model evaluation unit: Use independent test data set to evaluate the trained multimodal large model. Use multiple evaluation indicators such as accuracy, recall rate, F1 value, etc. to evaluate the performance and effect of the model, and ensure that the model has high reliability and accuracy.

[0055] 4. Rehabilitation evaluation module Motor function evaluation unit: Evaluate the patient's motor function according to biomechanical data and video image data. For example, by analyzing joint range of motion and gait cycle parameters, evaluate the patient's motor coordination and balance ability; by comparing the similarity of the patient's action with the standard action, evaluate the completion, fluency, accuracy and symmetry of the action.

[0056] Sensory function evaluation unit: Evaluate the patient's sensory function by combining surface electromyography signals and electroencephalogram signals. For example, by analyzing the electrical activity signals of the muscles and the activation degree of the brain neural function, determine whether the patient's sensory conduction pathway is normal and whether there is sensory disturbance.

[0057] Autonomic nervous function evaluation unit: Evaluate the patient's autonomic nervous function according to physiological indicators such as heart rate, blood pressure, respiratory rate, etc. By analyzing the changes of these indicators, determine whether the patient's autonomic nervous regulation function is impaired.

[0058] Comprehensive assessment unit: comprehensively consider the assessment results of motor function, sensory function and autonomic nervous function, and comprehensively assess the overall rehabilitation of the patient. Give rehabilitation grade score, and generate detailed assessment report, provide comprehensive reference for doctors and therapists.

[0059] 5. Rehabilitation training scheme generation module Personalized scheme development unit: according to the rehabilitation assessment results of patients, combined with the age, sex, severity of disease and other factors of patients, develop personalized rehabilitation training scheme for patients. For example, for patients with mild motor dysfunction, develop a scheme mainly based on active exercise training; for patients with severe motor dysfunction, develop a scheme mainly based on passive exercise training and assisted exercise training.

[0060] Training parameter setting unit: set the parameters of training according to the rehabilitation training scheme, such as training intensity, training time, training frequency, etc. For example, for muscle strength training, set the weight, times and sets of training; for balance training, set the difficulty level and duration of training.

[0061] Training equipment recommendation unit: recommend suitable training equipment according to the rehabilitation training scheme and training parameters. For example, for upper limb rehabilitation training, recommend using upper limb rehabilitation robot, elastic band and other equipment; for lower limb rehabilitation training, recommend using balance bar, treadmill and other equipment.

[0062] Training progress planning unit: develop a progress plan for rehabilitation training, and clearly define the training goals and tasks of each stage. According to the rehabilitation progress of patients, dynamically adjust the training progress to ensure the scientificity and effectiveness of training.

[0063] 6. Rehabilitation training monitoring module Real-time data acquisition unit: in the process of rehabilitation training of patients, real-time acquisition of training data of patients, such as biomechanical data, surface electromyogram, electroencephalogram, etc. Through the connection with training equipment, obtain the running parameters of equipment and the training state information of patients.

[0064] Training state monitoring unit: according to the real-time collected data, monitor the training state of patients. For example, monitor whether the movement posture of patients is correct, whether the movement intensity is appropriate, whether there is fatigue and pain, etc. Discover the problems existing in the training process in time, and give prompt and correction.

[0065] Training effect evaluation unit: regularly evaluate the rehabilitation training effect of patients. By comparing the rehabilitation assessment results before and after training, analyze the change situation of training effect. According to the evaluation results, adjust the rehabilitation training scheme and training parameters, and optimize the training effect.

[0066] Remote monitoring unit: Support remote monitoring function, doctors and therapists can view the training data and training status of patients in real time through the network. Even if not in the patient's side, also can understand the rehabilitation of patients in time, give remote guidance and suggestion.

[0067] 7. System management module User management unit: Manage the users of the system, including doctors, therapists, patients and other users with different roles. Implement user registration, login, permission setting and other functions to ensure the security of the system and the confidentiality of data.

[0068] Data management unit: Manage the data in the system, including data storage, backup, recovery and other operations. Establish a data warehouse to classify and manage multi-modal data, making it easy to query and use data.

[0069] System setting unit: Provide system setting function, users can set the parameters of the system according to actual demand, such as data acquisition frequency, evaluation index threshold, training equipment parameters, etc. At the same time, support system upgrade and maintenance, ensure the stability and performance optimization of the system.

[0070] Report generation unit: According to the needs of users, generate various types of reports, such as rehabilitation evaluation report, training effect report, equipment use report, etc. The report shows data in the form of charts and text, intuitive and clear, convenient for users to analyze data and make decisions.

[0071] In this embodiment, the beneficial effects include: Comprehensiveness: This system integrates multiple data acquisition methods, which can comprehensively obtain the rehabilitation information of patients, including motor function, sensory function and autonomic nervous function, avoiding the one-sidedness of traditional evaluation methods.

[0072] Accuracy: Based on multi-modal large model for data fusion and analysis, it can more accurately evaluate the rehabilitation of patients, improve the accuracy and reliability of evaluation.

[0073] Personalization: According to the specific situation of patients, formulate individualized rehabilitation training plan, meet the rehabilitation needs of different patients, improve the effect of rehabilitation training.

[0074] Real-time: Real-time monitoring of patient training status and training effect during rehabilitation training, timely adjustment of training scheme and parameters, ensure the scientificity and effectiveness of training.

[0075] Convenience: The system supports remote monitoring function, doctors and therapists can understand the rehabilitation of patients anytime and anywhere, give remote guidance and suggestion, improve the convenience and efficiency of medical service.

[0076] In one specific implementation, the working process of the system is as follows: Data collection: Use biomechanical sensors, surface electromyography acquisition equipment, electroencephalography, and high-definition cameras to collect data from a patient with peripheral nerve injury. Collect biomechanical data, surface electromyography signals of upper limb muscles, electroencephalography signals of brain, and video images of rehabilitation training when the patient walks.

[0077] Data preprocessing: Clean, normalize, align, and enhance the collected data. Remove abnormal fluctuation values in biomechanical data, normalize different types of data to the [0, 1] interval, use the DTW algorithm to time-align multi-modal data, and perform rotation and flipping operations on video image data.

[0078] Multi-modal data fusion: Extract key features from different modalities of data, such as joint angle change features in biomechanical data, time-domain features in surface electromyography signals, and electroencephalography power spectrum features in brain signals. Use a weighted fusion method to fuse features from different modalities, and use the fused feature data to train a multi-modal large model.

[0079] Rehabilitation assessment: Based on the trained multi-modal large model, assess the patient's motor function, sensory function, and autonomic nervous function. Analyze the patient's joint range of motion, gait cycle, muscle electrical activity signal, and brain neural function activation degree, and give a rehabilitation level score and detailed assessment report.

[0080] Rehabilitation training scheme generation: Based on the rehabilitation assessment results, develop a personalized rehabilitation training scheme for the patient. Set training parameters such as training intensity, training time, and training frequency, recommend suitable training equipment such as balance bars and elastic bands, and develop a training schedule.

[0081] Rehabilitation training monitoring: During the patient's rehabilitation training process, real-time training data is collected to monitor the training status. Regularly assess training effectiveness and adjust rehabilitation training schemes and training parameters based on assessment results.

[0082] System management: Manage users of the system and set user permissions. Store and backup data in the system, and generate rehabilitation assessment reports and training effectiveness reports based on user needs.

[0083] In one specific implementation, the working process of the system is as follows: Data collection: Collect data from a patient with peripheral neuropathy using the same data collection equipment as in the above example, including biomechanical data, surface electromyography signals of lower limb muscles, electroencephalography signals of brain, and video images of rehabilitation training when the patient stands.

[0084] Data preprocessing: The collected data is preprocessed according to the method of the above embodiment to ensure the quality and consistency of the data.

[0085] Multi-modal data fusion: A feature fusion network in deep learning is used, such as a model combining CNN and LSTM, to deeply fuse features of different modalities and improve the representation ability of the features.

[0086] Rehabilitation assessment: The integrated feature data is used to comprehensively assess the patient's rehabilitation, focusing on the patient's sensory function and autonomic nervous function recovery, and giving targeted assessment recommendations.

[0087] Rehabilitation training plan generation: Based on the assessment results, a rehabilitation training plan is developed for the patient, focusing on sensory training and autonomic nervous regulation training. Appropriate training equipment such as sensory training balls and biofeedback therapy devices are selected, and reasonable training parameters are set.

[0088] Rehabilitation training monitoring: Through remote monitoring, doctors and therapists can view the patient's training data and training status in real time. Timely problems are discovered and remote guidance is given to ensure the smooth progress of training.

[0089] System management: Regular maintenance and upgrades are performed to optimize system performance. Based on user feedback, system parameter settings and report generation formats are adjusted to improve user satisfaction.

[0090] In Figure 2 the present application, an embodiment of a peripheral nerve rehabilitation assessment method based on a multi-modal large model is provided, comprising the following steps: The data preprocessing module is used to clean the collected multi-modal data, remove noise and outliers, normalize different types of data, align the collection time and frequency of different modal data, and enhance the multi-modal data; The multi-modal data fusion module extracts key features from different modal data, uses multiple feature fusion methods to fuse features of different modalities, trains a multi-modal large model using the fused feature data, and evaluates the trained multi-modal large model using an independent test data set; The rehabilitation assessment module assesses the patient's motor function based on biomechanical data and video image data, assesses the patient's sensory function based on surface electromyography and electroencephalography, assesses the patient's autonomic nervous function based on physiological indicators, and comprehensively assesses the patient's overall rehabilitation based on the assessment results of motor function, sensory function and autonomic nervous function.

[0091] In one specific implementation, it also includes: The data acquisition module collects joint range of motion parameters, muscle strength level parameters and gait cycle parameters of the patient using biomechanical sensors; surface electromyography signal analysis technology is used to collect the electrical activity signals of the patient's muscles; electroencephalography technology is used to collect the neural electrical activity signals of the patient's brain; a high-definition camera is used to collect the patient's rehabilitation training video images, record the patient's action posture information and motion trajectory information; and the multi-modal data is obtained.

[0092] In one specific embodiment, it further comprises: The rehabilitation training scheme generation module formulates a personalized rehabilitation training scheme for the patient according to the patient's rehabilitation evaluation results, combined with the patient's age, gender and disease severity factors; sets the training parameters according to the rehabilitation training scheme; recommends suitable training equipment according to the rehabilitation training scheme and the training parameters; and formulates a rehabilitation training progress plan, clearly defining the training targets and tasks of each stage.

[0093] In one specific embodiment, it further comprises: The rehabilitation training monitoring module collects the patient's training data in real time during the patient's rehabilitation training process; monitors the patient's training state according to the real-time collected data; periodically evaluates the patient's rehabilitation training effect; and remotely monitors the patient's rehabilitation training.

[0094] In one specific embodiment, it further comprises: The system management module manages the users of the system; manages the data in the system; and generates various types of reports according to the user's needs.

[0095] In the above technical solution, the data preprocessing module is set to clean the collected multi-modal data, remove noise and outliers; normalize different types of data; perform data alignment processing on the collection time and frequency of different modal data; and perform enhancement processing on the multi-modal data; the multi-modal data fusion module is used to extract key features from different modal data; a variety of feature fusion methods are used to fuse features of different modalities; the fused feature data is used to train a multi-modal large model; and an independent test data set is used to evaluate the trained multi-modal large model; the rehabilitation evaluation module is used to evaluate the patient's motor function according to biomechanical data and video image data; evaluate the patient's sensory function in combination with surface electromyography signals and electroencephalography signals; evaluate the patient's autonomic nervous function according to the patient's physiological indicators; and comprehensively consider the evaluation results of motor function, sensory function and autonomic nervous function to comprehensively evaluate the patient's overall rehabilitation; data fusion and analysis based on the multi-modal large model can more accurately evaluate the patient's rehabilitation, improving the accuracy and reliability of the evaluation.

[0096] Those skilled in the art can appreciate that the present application can be embodied as a system, a method, or a computer program product.

[0097] Therefore, the present disclosure can be embodied in the form of a complete hardware, a complete software (including firmware, resident software, microcode, etc.), or a combination of hardware and software, which is generally referred to as "circuitry", "module" or "system" herein. In addition, in some embodiments, the present application can also be embodied in the form of a computer program product in one or more computer readable media, which contains computer readable program codes.

[0098] Any combination of one or more computer readable medium can be employed. The computer readable medium can be a computer readable signal medium or a computer readable storage medium. A computer readable storage medium can be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the above. More specific examples (a non-exhaustive list) of the computer readable storage medium include: an electrical connection having one or more wires, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this document, the computer readable storage medium can be any tangible medium that contains or stores a program that can be used by or in connection with an instruction execution system, apparatus, or device.

[0099] Although the embodiments of the present application have been shown and described above, it should be understood that the above-described embodiments are exemplary, and should not be construed as limiting the present application, and those skilled in the art can make changes, modifications, replacements and variations to the above-described embodiments within the scope of the present application. On this basis, various replacements and improvements can be made to the present application, and these all fall within the protection scope of the present application.

Claims

1. A peripheral nerve rehabilitation assessment system based on a multi-modal large model, characterized in that, The application relates to a multi-modal data fusion system and method for rehabilitation evaluation. The application comprises: a data preprocessing module for cleaning collected multi-modal data, removing noise and outliers; normalizing different types of data; performing data alignment processing on the collection time and frequency of different modal data; performing enhancement processing on the multi-modal data; a multi-modal data fusion module for extracting key features from data of different modalities respectively; using multiple feature fusion methods to fuse features of different modalities; using fused feature data to train a multi-modal large model; and using an independent test data set to evaluate the trained multi-modal large model; a rehabilitation evaluation module for evaluating the motor function of a patient according to biomechanical data and video image data; combining surface electromyography signals and electroencephalography signals to evaluate the sensory function of the patient; evaluating the autonomic nervous function of the patient according to physiological indicators; 2. The peripheral nerve rehabilitation assessment system based on a multi-modal large model according to claim 1, wherein, and comprehensively evaluating the overall rehabilitation condition of the patient by comprehensively considering the evaluation results of the motor function, the sensory function and the autonomic nervous function. The application further comprises: a data acquisition module for acquiring joint range of motion parameters, muscle strength grade parameters and gait cycle parameters of a patient by using biomechanical sensors; using surface electromyography signal analysis technology to acquire the electrical activity signals of the muscles of the patient; using electroencephalography topography technology to acquire the neural electrical activity signals of the brain of the patient; using a high-definition camera to acquire rehabilitation training video images of the patient, and recording the action posture information and motion trajectory information of the patient; 3. The multimodal large model-based peripheral nerve rehabilitation assessment system according to claim 2, wherein, obtaining the multi-modal data. The application further comprises: a rehabilitation training scheme generation module for formulating a personalized rehabilitation training scheme for a patient according to the rehabilitation evaluation result of the patient, in combination with the age, gender and disease severity factors of the patient; setting training parameters according to the rehabilitation training scheme; recommending suitable training equipment according to the rehabilitation training scheme and the training parameters; 4. The multimodal large model-based peripheral nerve rehabilitation assessment system according to claim 3, wherein, and formulating a rehabilitation training progress plan, and clearly defining the training targets and tasks of each stage. The application further comprises: a rehabilitation training monitoring module for acquiring training data of a patient in real time during the rehabilitation training process of the patient; monitoring the training state of the patient according to the real-time acquired data; regularly evaluating the rehabilitation training effect of the patient; 5. The multimodal large model-based peripheral nerve rehabilitation assessment system according to claim 4, wherein, and remotely monitoring the rehabilitation training of the patient. The application further comprises: a system management module for managing users of the system; managing data in the system; 6. A peripheral nerve rehabilitation evaluation method based on a multi-modal large model, characterized in that, generating various types of reports according to the requirements of the users. The application comprises the following steps: using the data preprocessing module to clean the collected multi-modal data, remove noise and outliers; and normalizing different types of data; performing data alignment processing on the collection time and frequency of different modal data; performing enhancement processing on the multi-modal data; using the multi-modal data fusion module to extract key features from data of different modalities respectively; using multiple feature fusion methods to fuse features of different modalities; using fused feature data to train a multi-modal large model; and using an independent test data set to evaluate the trained multi-modal large model; using the rehabilitation evaluation module to evaluate the motor function of a patient according to biomechanical data and video image data; The surface electromyography signal and the electroencephalogram signal are combined to evaluate the sensory function of the patient; the autonomic nervous function of the patient is evaluated according to the physiological indexes of the patient; The evaluation results of the motor function, the sensory function and the autonomic nervous function are comprehensively considered to comprehensively evaluate the overall rehabilitation condition of the patient.

7. The multi-modal large model based peripheral nerve rehabilitation assessment method according to claim 6, characterized in that, Further comprising: The data acquisition module is used to collect the joint range of motion parameters, muscle strength grade parameters and gait cycle parameters of the patient by using biomechanical sensors; The surface electromyography signal analysis technology is used to collect the electrical activity signals of the muscles of the patient; the electroencephalogram topography technology is used to collect the neural electrical activity signals of the brain of the patient; The high-definition camera is used to collect the rehabilitation training video images of the patient, and the action posture information and the motion trajectory information of the patient are recorded; The multi-modal data are obtained.

8. The multi-modal large model-based peripheral nerve rehabilitation assessment method according to claim 7, characterized in that, Further comprising: The rehabilitation training scheme generation module is used to formulate a personalized rehabilitation training scheme for the patient according to the rehabilitation evaluation results of the patient, in combination with the age, gender and disease severity factors of the patient; The training parameters are set according to the rehabilitation training scheme; The training equipment suitable for the patient is recommended according to the rehabilitation training scheme and the training parameters; The progress plan of the rehabilitation training is formulated, and the training targets and tasks of each stage are defined.

9. The multi-modal large model-based peripheral nerve rehabilitation assessment method according to claim 8, characterized in that, Further comprising: The rehabilitation training monitoring module is used to collect the training data of the patient in real time during the rehabilitation training of the patient; The training state of the patient is monitored according to the real-time collected data; the rehabilitation training effect of the patient is evaluated regularly; The rehabilitation training of the patient is remotely monitored.

10. The multi-modal large model based peripheral nerve rehabilitation assessment method according to claim 9, characterized in that, Further comprising: The system management module is used to manage the users of the system; The data in the system are managed; Various types of reports are generated according to the requirements of the users.

Images