Multi-modal rehabilitation data intelligent evaluation method and system based on large model

Abstract

The invention discloses a multi-modal rehabilitation data intelligent evaluation method and system based on a large model, and relates to the field related to data processing, and the method comprises the steps: carrying out the data collection based on a plurality of data sources, obtaining and integrating multi-modal data collection results, obtaining a multi-modal data set, synchronizing the multi-modal data set to an information extraction model, and carrying out the retrieval, a rehabilitation training scheme is executed to track the rehabilitation progress of a target user in real time; rehabilitation effect data is obtained and synchronized to the rehabilitation evaluation large model; a multi-dimensional evaluation result is obtained to perform feedback optimization on the rehabilitation training scheme; and a rehabilitation optimization training scheme is determined. The technical problem that traditional rehabilitation evaluation depends on professionals and lacks enough professional knowledge and experience, so that the accuracy and efficiency of evaluation are affected is solved, and the technical effects of performing intelligent personalized evaluation on rehabilitation training and improving the precision and efficiency of rehabilitation evaluation are achieved.

Description

Technical Field

[0001] The present application relates to data processing related fields, and in particular to a method and system for intelligent evaluation of multimodal rehabilitation data based on a large model. Background Art

[0002] In the current field of rehabilitation medicine, rehabilitation assessment is increasingly important as a key link in formulating personalized rehabilitation plans, monitoring rehabilitation progress, and evaluating rehabilitation effects. With the continuous advancement of medical technology and the diversification of patient needs, the limitations of traditional rehabilitation assessment methods have gradually been exposed. First, traditional rehabilitation assessment relies on professional medical teams, but the scarcity of professional talent resources means that the assessment process may face challenges such as lack of experience or knowledge updates. Secondly, traditional assessment methods mostly rely on manual observation and simple scales, which to a certain extent limits the accuracy and comprehensiveness of the assessment. Therefore, in the existing technology, traditional rehabilitation assessment relies on professional talents and faces the lack of sufficient professional knowledge and experience, which leads to technical problems that affect the accuracy and efficiency of the assessment. Summary of the invention

[0003] This application provides a multimodal rehabilitation data intelligent assessment method and system based on a large model, which solves the technical problem that traditional rehabilitation assessment relies on professional talents and lacks sufficient professional knowledge and experience, thereby affecting the accuracy and efficiency of the assessment. It achieves the technical effect of conducting intelligent personalized assessment of rehabilitation training and improving the accuracy and efficiency of rehabilitation assessment.

[0004] The present application provides a large-scale model-based multimodal rehabilitation data intelligent evaluation method, which is applied to a large-scale model-based multimodal rehabilitation data intelligent evaluation system, including: performing data collection based on multiple data sources to obtain multimodal data collection results, integrating the multimodal data collection results to obtain a multimodal data set; synchronizing the multimodal data set to an information extraction model for retrieval to determine a target information set; performing training verification based on the target information set to construct a large-scale rehabilitation evaluation model; executing a rehabilitation training plan to track the rehabilitation progress of a target user in real time to obtain rehabilitation effect data; synchronizing the rehabilitation effect data of the target user to the large-scale rehabilitation evaluation model to obtain a multidimensional evaluation result, performing feedback optimization on the rehabilitation training plan based on the multidimensional evaluation result, and determining a rehabilitation optimization training plan.

[0005] In a possible implementation, the multimodal data set is synchronized to an information extraction model for retrieval, a target information set is determined, and the following processing is performed: data annotation is performed based on the data type of the multimodal data set to obtain data annotation results; the multimodal data set is divided into multiple text data and multiple image data according to the data annotation results; the information extraction model includes a text feature extraction layer, an image feature extraction layer, and an extracted feature fusion layer; the multiple text data are synchronized to the text feature extraction layer to obtain multiple text features; the multiple image data are synchronized to the image feature extraction layer to obtain multiple image features; the multiple text features and the multiple image features are synchronized to the extracted feature fusion layer to obtain a multimodal feature fusion vector; according to the multimodal feature fusion vector, the multimodal data set is traversed for retrieval to determine the target information set.

[0006] In a possible implementation, training verification is performed based on the target information set, a rehabilitation assessment model is constructed, and the following processing is performed: data preprocessing is performed based on the target information set to generate multiple target information processing results; the multiple target information processing results are divided into a training data set, a verification data set, and a test data set; supervised training is performed based on the training data set combined with the verification data set to construct a text processing layer and an image processing layer, wherein the text processing layer is obtained by training based on the text data in the training data set, and the image processing layer is obtained by training based on the image data in the training data set; historical health assessment files are retrieved, and an information assessment layer is constructed in combination with the multimodal feature fusion vector; the output end of the text processing layer and the output end of the image processing layer are communicatively connected to the input end of the information assessment layer; the test data set is synchronized to the information assessment layer for testing, and when the test passes, the rehabilitation assessment model is determined.

[0007] In a possible implementation, the rehabilitation effect data of the target user is synchronized to the rehabilitation assessment model to obtain a multidimensional assessment result, and the following processing is performed: data formatting is performed based on the rehabilitation effect data of the target user to generate rehabilitation effect text data and rehabilitation effect image data; the rehabilitation effect text data is synchronized to the text processing layer of the rehabilitation assessment model to obtain a text processing result; the rehabilitation effect image data is synchronized to the image processing layer of the rehabilitation assessment model to obtain an image processing result; the text processing result and the image processing result are weightedly fused using an attention mechanism to obtain a weighted fusion result; an evaluation index is set based on the multimodal feature fusion vector, and a plurality of evaluation dimensions are determined according to the evaluation index; a multidimensional assessment is performed according to the plurality of evaluation dimensions based on the weighted fusion result to generate the multidimensional assessment result.

[0008] In a possible implementation, the rehabilitation effect text data is synchronized to the text processing layer of the rehabilitation assessment large model to obtain a text processing result, and the following processing is performed: the rehabilitation effect text data is segmented to generate multiple text words; the multiple text words are vectorized to obtain multiple text word vectors; context information is extracted based on the rehabilitation effect text data, and text understanding is performed based on the multiple text word vectors combined with the context information to obtain text understanding information; the association relationship between the text understanding information and the rehabilitation effect data is extracted to construct an association network; the text understanding information is extracted according to the association network, and the text processing result is determined based on the extracted information.

[0009] In a possible implementation, the rehabilitation effect image data is synchronized to the image processing layer of the rehabilitation assessment large model to obtain an image processing result, and the following processing is performed: the rehabilitation effect image data is enhanced to generate a plurality of enhanced images; a standardized conversion is performed based on the plurality of enhanced images to obtain a plurality of standard images; a convolutional neural network is used to traverse the plurality of standard images for transfer learning to obtain a plurality of image migration features; a region segmentation detection is performed on the plurality of standard images based on the plurality of image migration features to generate an image detection result; an evaluation is performed based on the image detection result, and the image processing result is obtained based on the evaluation result.

[0010] In a possible implementation, the rehabilitation training program is feedback-optimized based on the multidimensional evaluation results, and the rehabilitation optimization training program is determined, and the following processing is performed: a feedback cycle is determined based on the execution timing of the rehabilitation training program; multiple mining directions are determined based on the multidimensional evaluation results, and the rehabilitation training program is traversed according to the multiple mining directions to perform data mining to obtain multiple rehabilitation training data; the multiple rehabilitation training data are regularly evaluated based on the feedback cycle to generate a label set of data to be optimized; the label set of data to be optimized is synchronized to the rehabilitation training program through a feedback mechanism for identification, and multiple data to be optimized are determined; the rehabilitation training program is optimized according to the multiple data to be optimized to obtain the rehabilitation optimization training program.

[0011] The present application also provides a multimodal rehabilitation data intelligent evaluation system based on a large model, including: a data integration module, the data integration module is used to collect data based on multiple data sources to obtain multimodal data collection results, integrate the multimodal data collection results to obtain a multimodal data set; a data retrieval module, the data retrieval module is used to synchronize the multimodal data set to an information extraction model for retrieval, and determine a target information set; a training verification module, the training verification module is used to perform training verification based on the target information set and construct a large rehabilitation evaluation model; a data tracking module, the data tracking module is used to execute a rehabilitation training plan to track the rehabilitation progress of a target user in real time and obtain rehabilitation effect data; a feedback optimization module, the feedback optimization module is used to synchronize the rehabilitation effect data of the target user to the large rehabilitation evaluation model to obtain a multidimensional evaluation result, perform feedback optimization on the rehabilitation training plan based on the multidimensional evaluation result, and determine a rehabilitation optimization training plan.

[0012] One or more technical solutions provided in this application have at least the following technical effects or advantages:

[0013] The large-model-based multimodal rehabilitation data intelligent assessment method and system provided in the present application relate to the field of data processing technology. They solve the technical problem that traditional rehabilitation assessment relies on professional talents and faces a lack of sufficient professional knowledge and experience, thereby affecting the accuracy and efficiency of the assessment. They achieve the technical effect of conducting intelligent individual assessment of rehabilitation training and improving the accuracy and efficiency of rehabilitation assessment. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] In order to more clearly illustrate the technical scheme of the embodiment of the present application, the accompanying drawings of the embodiment of the present application will be briefly introduced below. A flow chart is used in the present application to illustrate the operations performed by the system according to the embodiment of the present application. It should be understood that the preceding or following operations are not necessarily performed accurately in order. On the contrary, various steps can be processed in reverse order or simultaneously as needed. At the same time, other operations can also be added to these processes, or a certain step or several steps of operations can be removed from these processes.

[0015] Figure 1 A schematic diagram of a flow chart of a large-model-based multimodal rehabilitation data intelligent evaluation method provided in an embodiment of the present application;

[0016] Figure 2 A schematic diagram of the structure of a large-model-based multimodal rehabilitation data intelligent evaluation system provided in an embodiment of the present application. DETAILED DESCRIPTION

[0017] The above description is only an overview of the technical solution of the present application. In order to more clearly understand the technical means of the present application, it can be implemented in accordance with the contents of the specification. In order to make the above and other purposes, features and advantages of the present application more obvious and easy to understand, the specific implementation methods of the present application are listed below.

[0018] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application will be further described in detail below in conjunction with the accompanying drawings. The described embodiments should not be regarded as limiting the present application. All other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of this application.

[0019] In the following description, reference is made to "some embodiments", which describe a subset of all possible embodiments, but it is understood that "some embodiments" may be the same subset or different subsets of all possible embodiments, and may be combined with each other without conflict, and the terms "first\second" involved are merely to distinguish similar objects and do not represent a specific ordering of objects. The terms "including" and "having" and any variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product, or server that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or modules that are not clearly listed or inherent to these processes, methods, products, or devices. Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those generally understood by technicians in the technical field of this application. The terms used herein are for the purpose of describing the embodiments of the present application only.

[0020] The embodiment of the present application provides a multimodal rehabilitation data intelligent evaluation method based on a large model, and the method is applied to a multimodal rehabilitation data intelligent evaluation system based on a large model, such as Figure 1 As shown, the method includes:

[0021] Execute step A100, collect data based on multiple data sources, obtain multimodal data collection results, integrate the multimodal data collection results, and obtain a multimodal data set; first, determine multiple data sources for acquiring data, and the multiple data sources may include multimodal data sources such as electronic medical records, patient self-reported symptoms, doctor's diagnosis reports, and imaging files. At the same time, for each data source, a corresponding data collection strategy may be designed, including sampling frequency, data format, storage method, etc., to ensure the integrity and consistency of the data, and then extract and summarize the required data based on multiple data sources, that is, by integrating multi-source data such as electronic medical records, patient self-reports, doctor's diagnoses, and imaging materials, a comprehensive reflection of the patient's health status is constructed. Data set, thereby determining the multimodal data collection results. Furthermore, since data of different modalities may come from different time points and spatial locations, time synchronization and spatial alignment are required. At the same time, in order to ensure the consistency between the data of each modality, it is necessary to integrate the multimodal data collection results, which means: extracting key features from each modal data, which can be simple features such as pixel value and frequency, or complex features such as the theme of the text and the content description of the image, and fusing the extracted multimodal features to form a unified multimodal representation, that is, a multimodal data set. This multi-dimensional data input provides a solid foundation for subsequent precise analysis, thereby improving the accuracy and depth of subsequent evaluation and analysis.

[0022] Execute step A200, synchronize the multimodal data set to the information extraction model for retrieval, and determine the target information set; in a possible implementation, step A200 further includes step A210, perform data annotation based on the data type of the multimodal data set to obtain data annotation results; execute step A220, divide the multimodal data set into multiple text data and multiple image data according to the data annotation results; execute step A230, the information extraction model includes a text feature extraction layer, an image feature extraction layer, and an extraction feature fusion layer; execute step A240, synchronize the multiple text data to the text feature extraction layer to obtain multiple text features; execute step A250, synchronize the multiple image data to the image feature extraction layer to obtain multiple image features; execute step A260, synchronize the multiple text features and the multiple image features to the extraction feature fusion layer to obtain a multimodal feature fusion vector; execute step A270, traverse the multimodal data set according to the multimodal feature fusion vector to perform retrieval, and determine the target information set.

[0023] First, the data type of the multimodal dataset obtained above is extracted, and the data type can be a text data type or an image data type. On this basis, data annotation is performed on the multimodal dataset, which means annotating the multimodal dataset according to the data type corresponding to each data, namely, the text data type and the image data type. According to the result of the data annotation, the multimodal dataset is automatically divided into multiple text data and multiple image data.

[0024] Further, multiple text data and multiple image data are synchronized to the information extraction model respectively, and the information extraction model includes a text feature extraction layer, an image feature extraction layer, and an extracted feature fusion layer. Then, the multiple text data are subjected to feature extraction through the text feature extraction layer to determine multiple text features. The text feature extraction layer is used to process the text data set and convert the text data into numerical features. The multiple image data are subjected to feature extraction through the image feature extraction layer to determine multiple image features. The image feature extraction layer is used to process the image data set and extract key features in the image, which may include image edges, image textures, image objects and other data. Finally, the multiple text features output by the text feature extraction layer are The multiple image features output by the image feature extraction layer are synchronized to the extraction feature fusion layer. In the extraction feature fusion layer, feature splicing is used to fuse the text features and the image features to form a multimodal feature fusion vector. The multimodal feature fusion vector is information that contains both text and image, which can describe the data content more comprehensively. Then, according to the multimodal feature fusion vector, the multimodal data set is traversed for retrieval, which means that based on the multimodal feature fusion vector, a query request is constructed, and the request is converted into a multimodal query vector to traverse the multimodal data set, and the similarity between the query vector and the multimodal feature fusion vector of each sample in the data set is calculated. According to the similarity sorting result, several samples with the highest similarity are selected as the target information set for output.

[0025] Execute step A300, perform training verification based on the target information set, and construct a large rehabilitation assessment model;

[0026] In a possible implementation, step A300 further includes step A310, performing data preprocessing based on the target information set to generate multiple target information processing results; executing step A320, dividing the multiple target information processing results into a training data set, a verification data set, and a test data set; executing step A330, performing supervised training based on the training data set in combination with the verification data set, and constructing a text processing layer and an image processing layer, wherein the text processing layer is obtained by training based on the text data in the training data set, and the image processing layer is obtained by training based on the image data in the training data set; executing step A340, retrieving historical health assessment files, and constructing an information evaluation layer in combination with the multimodal feature fusion vector; executing step A350, communicating and connecting the output end of the text processing layer and the output end of the image processing layer to the input end of the information evaluation layer; executing step A360, synchronizing the test data set to the information evaluation layer for testing, and when the test passes, determining the rehabilitation assessment model.

[0027] In order to improve the accuracy of model building, it is first necessary to perform data preprocessing on the target information set obtained above. The process of data preprocessing can be to further preprocess each sample in the target information set. This may include text data cleaning (removing stop words, punctuation marks, non-alphanumeric characters, etc.), normalization (word frequency normalization, TF-IDF weighting, etc.), word segmentation or word embedding, etc.; image data cropping, scaling, normalization (pixel value normalization), enhancement (rotation, flipping, noise addition, etc.), etc., so as to generate multiple target information processing results, and further divide the multiple target information processing results into training data sets, verification data sets, and test data sets, which means randomly dividing the multiple target information processing results into training data sets, verification data sets, and test data sets. Among them, the training data set accounts for the largest proportion, which can account for 70%, the verification data set is second, which can account for 15%, and the test data set is the smallest, which can account for 15%. When dividing, it should be ensured that the three data sets are as consistent as possible in data distribution.

[0028] The text processing layer is further trained using the text data in the training data set. It can be based on structures such as RNN, LSTM, Transformer, etc. The text processing layer is used to extract key information and features in the text, and the image processing layer is trained using the image data in the training data set. It can be based on a CNN model, and the image processing layer can capture key features and patterns in the image. Supervised training is performed based on the training data set combined with the verification data set, which means that supervised training is performed in combination with the training data set and the verification data set, and during the training process, the loss function is minimized by adjusting the model parameters to improve the performance of the model on the verification data set, thereby completing the construction of the text processing layer and the image processing layer, and the constructed text processing layer and the image processing layer are in a parallel relationship.

[0029] Further, retrieving the historical health assessment file means retrieving the relevant health assessment file from the historical data, and the historical health assessment file may include the patient's medical history, diagnosis information, treatment records, etc. And combining the multimodal feature fusion vector to construct the information evaluation layer means combining the multimodal feature fusion vector with the historical health assessment file by feature splicing, weighted fusion or model fusion methods to construct the information evaluation layer. The information evaluation layer can be used to be responsible for integrating text, images and historical health information, evaluate the patient's rehabilitation status, and communicate and connect the output end of the text processing layer and the output end of the image processing layer with the input end of the information evaluation layer to ensure that the data can be smoothly transmitted and fused, and finally synchronize the test data set to the information evaluation layer for testing. The test data set should include different types of text and image data to comprehensively evaluate the performance of the model. By comparing the evaluation results output by the model with the actual rehabilitation evaluation results, the performance indicators such as the accuracy and generalization ability of the model are evaluated. When the test passes, the output result of the information evaluation layer is the output result of the rehabilitation evaluation large model, that is, when the model shows good performance on the test data set, the model is determined to be a rehabilitation evaluation large model. This model can accurately assess the patient's rehabilitation status based on multimodal data, provide support for subsequent rehabilitation treatment and decision-making, and thus complete the construction of a large rehabilitation assessment model.

[0030] Execute step A400, execute the rehabilitation training program to track the rehabilitation progress of the target user in real time, and obtain rehabilitation effect data; by combining the health status of the target user, rehabilitation goals, doctor's advice and other factors, build a rehabilitation training program that meets the target user, the rehabilitation training program may include detailed information such as specific training movements, training intensity, training duration, training frequency, and may include a multi-stage progressive advancement plan, and at the same time prepare the required rehabilitation training equipment, monitoring instruments, etc., further, wearable devices, sensors, cameras and other technologies can be used to collect physiological data such as heart rate, blood pressure, muscle activity and behavioral data such as movement accuracy and completion of the target user during the rehabilitation training in real time, and the real-time data is processed and analyzed to evaluate the rehabilitation progress and effect of the target user, and the real-time data of the target user is compared with the baseline data, such as the data before rehabilitation training, to evaluate the rehabilitation effect of the target user, and then combined with the professional opinions of doctors or rehabilitation therapists, the user's rehabilitation progress is comprehensively evaluated, thereby achieving real-time tracking of the rehabilitation training program of the target user and obtaining rehabilitation effect data, providing strong support for the user's rehabilitation progress.

[0031] Next, step A500 is executed to synchronize the rehabilitation effect data of the target user to the rehabilitation assessment model to obtain a multi-dimensional assessment result, and feedback optimization is performed on the rehabilitation training program based on the multi-dimensional assessment result to determine a rehabilitation optimization training program.

[0032] In a possible implementation, step A500 further includes step A510, performing data formatting processing based on the rehabilitation effect data of the target user to generate rehabilitation effect text data and rehabilitation effect image data; performing data formatting processing based on the rehabilitation effect data obtained above refers to performing data processing on the rehabilitation effect data according to different data formats, thereby dividing the rehabilitation effect data into text data format and image data format, generating rehabilitation effect text data and rehabilitation effect image data, and the rehabilitation effect text data can be obtained by writing a concise text summary to summarize the user's rehabilitation effect in different time periods or rehabilitation stages, and then describing the user's physiological indicators, sports performance, etc. through text. The changing trend of the surface is pointed out, and the specific aspects of improvement or deterioration are pointed out. Finally, based on the data analysis results, text data of rehabilitation effect is given for reference by users, doctors and rehabilitation therapists. The rehabilitation effect image data can select appropriate chart types according to the data type and analysis requirements, such as line charts (showing trend changes), bar charts (comparing different data points), scatter plots (analyzing correlations), etc., and then the cleaned and pre-processed data are imported into visualization tools (such as Excel, Tableau, R language drawing packages, etc.) to generate intuitive rehabilitation effect image data. The generated rehabilitation effect image data can clearly show the user's rehabilitation progress, and include the changing trends of physiological indicators, the improvement of sports performance, etc.

[0033] Execute step A520 to synchronize the rehabilitation effect text data to the text processing layer of the rehabilitation assessment large model to obtain a text processing result; in a possible implementation, step A520 further includes step A521 to perform word segmentation processing on the rehabilitation effect text data to generate multiple text words; execute step A522 to perform vectorization conversion on the multiple text words to obtain multiple text word vectors; execute step A523 to extract context information based on the rehabilitation effect text data, and perform text understanding based on the multiple text word vectors combined with the context information to obtain text understanding information; execute step A524 to extract the association relationship between the text understanding information and the rehabilitation effect data, and construct an association network; execute step A525 to extract the text understanding information according to the association network, and determine the text processing result according to the extracted information.

[0034] First, the rehabilitation effect text data obtained above is subjected to word segmentation processing, which refers to the process of dividing the continuous text in the rehabilitation effect text data into independent words or phrases, so as to obtain multiple text words in the rehabilitation effect text data, and then the multiple text words can be converted into vector form by using word embedding technology, such as Word2Vec, GloVe or BERT, to obtain multiple text word vectors. The multiple text word vectors not only contain the information of the word itself, but also imply the relationship with other words. Context information is extracted based on the rehabilitation effect text data. The context information may include but is not limited to the background, situation, time, place, people involved (such as patients, therapists) and the interaction between them, and then the multiple text word vectors are combined with the context information for in-depth text understanding. The text understanding information can more accurately grasp the semantic meaning in the text. On the basis of text understanding, key information points in the text are further extracted, such as the patient's symptom description, treatment process, rehabilitation progress, adverse reactions, etc. Then, the correlation between this information and rehabilitation effect data (such as rehabilitation speed, rehabilitation degree, complications, etc.) is analyzed. The correlation can be direct (such as a specific treatment leading to significant improvement) or indirect (such as the impact of changes in the patient's mentality on the rehabilitation process), and the correlation between the extracted key information points and the rehabilitation effect data is sorted out to construct an association network. The association network can be represented by a graph, where nodes represent information points or data items, and edges represent the association between them. Finally, according to the relationship path in the association network, the text understanding information is extracted. The extraction process refers to the screening, integration and interpretation of information to ensure that the final text processing results are accurate, comprehensive and easy to understand. Based on the extracted information, the rehabilitation effect reflected by the text, that is, the text processing results, can be determined, which can include success factors, existing problems and possible improvement directions, so that the information in the text can be accurately identified, interpreted and applied.

[0035] Execute step A530 to synchronize the rehabilitation effect image data to the image processing layer of the rehabilitation assessment large model to obtain an image processing result; in a possible implementation, step A530 further includes step A531 to enhance the rehabilitation effect image data to generate a plurality of enhanced images; execute step A532 to perform standardized conversion based on the plurality of enhanced images to obtain a plurality of standard images; execute step A533 to traverse the plurality of standard images for transfer learning using a convolutional neural network to obtain a plurality of image transfer features; execute step A534 to perform region segmentation detection on the plurality of standard images based on the plurality of image transfer features to generate an image detection result; execute step A535 to perform evaluation based on the image detection result and obtain the image processing result based on the evaluation result.

[0036] First, the original rehabilitation effect image data can be enhanced by adjusting the brightness, contrast, sharpness of the image, or applying noise suppression, image filtering and other technologies. Image enhancement is used to improve the visual effect of the rehabilitation effect image data, or convert the image into a form more suitable for analysis, generate multiple versions of enhanced images, that is, multiple enhanced images, and then perform standardization conversion on the multiple enhanced images, which means adjusting the pixel values ​​of the multiple enhanced images to a uniform range (such as 0 to 1 or -1 to 1) so that the subsequent processing steps can be carried out more efficiently. The process of standardization conversion helps to eliminate the differences in brightness, contrast, etc. between different images in the multiple enhanced images, so that machine learning models such as convolutional neural networks can process these images more fairly.

[0037] Further, a convolutional neural network (CNN) is used to traverse multiple standard images for transfer learning. The transfer learning can apply the knowledge learned on one task in multiple standard images to another related but different task. The image features of multiple standard images can be extracted by using a pre-trained CNN model, and these features are used for new image analysis tasks, and useful image migration features can be quickly extracted from multiple standard images by transfer learning. The extracted multiple image migration features are complex features. Based on the multiple image migration features, multiple standard images are subjected to regional segmentation detection. The regional segmentation detection can divide multiple standard images into different regions or objects for further identification, and determine specific tissues, lesion areas or treatment effects in multiple standard images, etc., and the multiple image migration features can accurately locate and segment the key areas in multiple standard images, thereby generating image detection results, and finally evaluating the image detection results, and obtaining image processing results according to the evaluation results. The evaluation process can be an evaluation of the accuracy, completeness, consistency, etc. of the image detection results. By evaluating the image detection results, it can be determined which areas are correctly identified and which areas have problems such as false detection or missed detection. According to the evaluation results, the parameters or methods in the image processing process can be further adjusted to obtain more accurate image processing results. These results can be used for subsequent medical analysis, diagnosis or treatment decisions, etc., to obtain the image processing results.

[0038] Execute step A540, use the attention mechanism to perform weighted fusion on the text processing result and the image processing result to obtain a weighted fusion result; execute step A550, set an evaluation index based on the multimodal feature fusion vector, and determine multiple evaluation dimensions according to the evaluation index; execute step A560, perform multidimensional evaluation according to the multiple evaluation dimensions based on the weighted fusion result to generate the multidimensional evaluation result.

[0039] First, the attention weights are calculated for the text processing results and the image processing results based on the reliability of the data, the relevance to the current task, and the characteristics of the data itself. The attention weight is used to characterize the importance of each modal data in the final decision. The greater its importance, the higher the attention weight. Then, the calculated attention weight is used to perform weighted fusion on the text processing results and the image processing results. It means multiplying the data of each modality with its corresponding weight, and then adding the weighted data or combining them in other ways to obtain the weighted fusion result. After obtaining the weighted fusion result, it is necessary to set the evaluation index based on the multimodal feature fusion vector. The evaluation index is used to measure the quality of the fusion result and is closely related to the task goal. For example, in the evaluation of rehabilitation effect, the evaluation index may include rehabilitation speed, rehabilitation degree, patient satisfaction, etc. And according to the set evaluation index, multiple evaluation dimensions are determined. The multiple evaluation dimensions refer to a comprehensive evaluation of the fusion results based on different angles. For example, in the evaluation of rehabilitation effects, the evaluation dimensions may include physiological indicators (such as muscle strength, joint mobility), psychological indicators (such as anxiety level, self-confidence) and quality of life indicators (such as daily activity ability, social participation), etc. Then, based on the weighted fusion results and multiple evaluation dimensions, a multidimensional evaluation is performed, which means quantitatively scoring or classifying each evaluation dimension to generate a multidimensional evaluation result. The multidimensional evaluation results can comprehensively reflect the performance of the fused data in different aspects and provide strong support for subsequent decision-making or analysis.

[0040] In one possible implementation, step A500 further includes step A570, determining a feedback cycle based on the execution timing of the rehabilitation training program; executing step A580, determining multiple mining directions based on the multi-dimensional evaluation results, traversing the rehabilitation training program according to the multiple mining directions to perform data mining, and obtaining multiple rehabilitation training data; executing step A590, performing regular evaluations on the multiple rehabilitation training data based on the feedback cycle, and generating a label set of data to be optimized; executing step A5100, synchronizing the label set of data to be optimized to the rehabilitation training program through a feedback mechanism for identification, and determining multiple data to be optimized; executing step A5101, optimizing the rehabilitation training program according to the multiple data to be optimized, and obtaining the rehabilitation optimization training program.

[0041] First, the feedback cycle is determined based on the execution sequence of the rehabilitation training program. The feedback cycle refers to the time interval for evaluating the training effect and adjusting the training program accordingly during the rehabilitation training process. The setting of the feedback cycle can be based on the continuity of the rehabilitation process, the patient's adaptability, and the time when the training effect appears. Further, multiple mining directions are determined based on the multidimensional evaluation results. It means that the multidimensional evaluation results provide comprehensive information about the rehabilitation training effect, that is, the rehabilitation status of different aspects (such as physiological, psychological, social functions, etc.). The multiple mining directions refer to determining the specific goals and paths of data mining based on the potential problems or improvement space pointed out by the evaluation results. Then, the rehabilitation training program is traversed according to multiple mining directions for data mining, and multiple rehabilitation training data points or data, that is, multiple rehabilitation training data, are obtained by analyzing training data, patient feedback, doctor observation and other information. Further, based on the feedback cycle set above, the multiple rehabilitation training data mined are regularly evaluated. It means monitoring the changing trend of the training effect and identifying the training links that perform poorly or need to be adjusted. Generate a set of data labels to be optimized through regular evaluation, and the set of data labels to be optimized may include data points or data sets marked as needing further optimization or adjustment, and synchronize the set of data labels to be optimized to the rehabilitation training program through a feedback mechanism for identification. It means that by adding labels or annotations to the corresponding parts of the training program according to the set of data labels to be optimized, it can be determined which data or links currently need attention and optimization. It helps to locate the problem more accurately in the subsequent training process and provide clear guidance for the optimization work, so as to determine multiple data to be optimized, and finally, optimize the rehabilitation training program according to multiple data to be optimized. The optimization process can be to adjust the training intensity, change the training method, introduce new training tools or technologies, etc. The goal of the optimization is to make the training program more in line with the actual situation and needs of the patient, so as to improve the rehabilitation effect. Through repeated practice and adjustment, a more complete and effective rehabilitation optimization training program is finally obtained, so as to generate the rehabilitation optimization training program, and realize the closed-loop management of personalized rehabilitation treatment for the target user.

[0042] The embodiments of the present application solve the technical problem that traditional rehabilitation assessment relies on professional talents and lacks sufficient professional knowledge and experience, thereby affecting the accuracy and efficiency of the assessment, and achieves the technical effect of conducting intelligent individual assessment of rehabilitation training and improving the accuracy and efficiency of rehabilitation assessment.

[0043] In the above, refer to Figure 1 The multimodal rehabilitation data intelligent evaluation method based on a large model according to an embodiment of the present application is described in detail. Figure 2 A multimodal rehabilitation data intelligent evaluation system based on a large model according to an embodiment of the present application is described.

[0044] The multimodal rehabilitation data intelligent evaluation system based on a large model according to the embodiment of the present application is used to solve the technical problem that traditional rehabilitation evaluation relies on professional talents and lacks sufficient professional knowledge and experience, thereby affecting the accuracy and efficiency of the evaluation, and achieves the technical effect of conducting intelligent individual evaluation of rehabilitation training and improving the accuracy and efficiency of rehabilitation evaluation. The multimodal rehabilitation data intelligent evaluation system based on a large model includes: a data integration module 10, a data retrieval module 20, a training verification module 30, a data tracking module 40, and a feedback optimization module 50.

[0045] A data integration module 10, the data integration module 10 is used to collect data based on multiple data sources to obtain multimodal data collection results, and integrate the multimodal data collection results to obtain a multimodal data set;

[0046] A data retrieval module 20, the data retrieval module 20 is used to synchronize the multimodal data set to the information extraction model for retrieval to determine a target information set;

[0047] A training verification module 30, wherein the training verification module 30 is used to perform training verification based on the target information set to construct a large rehabilitation assessment model;

[0048] A data tracking module 40, which is used to execute a rehabilitation training program to track the rehabilitation progress of the target user in real time and obtain rehabilitation effect data;

[0049] Feedback optimization module 50, the feedback optimization module 50 is used to synchronize the rehabilitation effect data of the target user to the rehabilitation assessment model, obtain multi-dimensional assessment results, perform feedback optimization on the rehabilitation training program based on the multi-dimensional assessment results, and determine the rehabilitation optimization training program.

[0050] The specific configuration of the data retrieval module 20 will be described in detail below. As described above, the multimodal data set is synchronized to the information extraction model for retrieval to determine the target information set. The data retrieval module 20 may further include: a data annotation unit for performing data annotation based on the data type of the multimodal data set to obtain a data annotation result; a first data partitioning unit for dividing the multimodal data set into multiple text data and multiple image data according to the data annotation result; a model data unit for the information extraction model including a text feature extraction layer, an image feature extraction layer, and an extraction feature fusion layer; a first data synchronization unit for synchronizing the multiple text data to the text feature extraction layer to obtain multiple text features; a second data synchronization unit for synchronizing the multiple image data to the image feature extraction layer to obtain multiple image features; a third data synchronization unit for synchronizing the multiple text features and the multiple image features to the extraction feature fusion layer to obtain a multimodal feature fusion vector; and a retrieval unit for traversing the multimodal data set for retrieval according to the multimodal feature fusion vector to determine the target information set.

[0051] The specific configuration of the training verification module 30 will be described in detail below. As described above, training verification is performed based on the target information set to construct a rehabilitation assessment model. The training verification module 30 may further include: a preprocessing unit for performing data preprocessing based on the target information set to generate multiple target information processing results; a second data division unit for dividing the multiple target information processing results into a training data set, a verification data set, and a test data set; a first training unit for performing supervised training based on the training data set combined with the verification data set to construct a text processing layer and an image processing layer, wherein the text processing layer is obtained by training based on the text data in the training data set, and the image processing layer is obtained by training based on the image data in the training data set; a first construction unit is used to retrieve historical health assessment files and construct an information assessment layer in combination with the multimodal feature fusion vector; a communication connection unit is used to communicate and connect the output end of the text processing layer and the output end of the image processing layer with the input end of the information assessment layer; a second construction unit is used to synchronize the test data set to the information assessment layer for testing, and when the test passes, the rehabilitation assessment model is determined.

[0052] The specific configuration of the feedback optimization module 50 will be described in detail below. As described above, the rehabilitation effect data of the target user is synchronized to the rehabilitation assessment model to obtain a multi-dimensional assessment result. The feedback optimization module 50 may further include: a third data division unit for performing data formatting processing based on the rehabilitation effect data of the target user to generate rehabilitation effect text data and rehabilitation effect image data; a fourth data synchronization unit for synchronizing the rehabilitation effect text data to the text processing layer of the rehabilitation assessment model to obtain a text processing result; a fifth data synchronization unit for synchronizing the rehabilitation effect image data to the image processing layer of the rehabilitation assessment model to obtain an image processing result; a weighted fusion unit for weighted fusion of the text processing result and the image processing result using an attention mechanism to obtain a weighted fusion result; a dimension determination unit for setting an assessment index based on the multimodal feature fusion vector and determining multiple assessment dimensions according to the assessment index; and a multi-dimensional assessment unit for performing multi-dimensional assessment according to the multiple assessment dimensions based on the weighted fusion result to generate the multi-dimensional assessment result.

[0053] The specific configuration of the fourth data synchronization unit will be described in detail below. As described above, the rehabilitation effect text data is synchronized to the text processing layer of the rehabilitation assessment large model, and the fourth data synchronization unit may further include: a word segmentation processing unit for performing word segmentation processing on the rehabilitation effect text data to generate multiple text words; a first conversion unit for vectorizing the multiple text words to obtain multiple text word vectors; a text understanding unit for extracting context information based on the rehabilitation effect text data, performing text understanding based on the multiple text word vectors combined with the context information, and obtaining text understanding information; an association unit for extracting the association relationship between the text understanding information and the rehabilitation effect data, and constructing an association network; an extraction unit for extracting the text understanding information according to the association network, and determining the text processing result according to the extracted information.

[0054] The specific configuration of the fifth data synchronization unit will be described in detail below. As described above, the rehabilitation effect image data is synchronized to the image processing layer of the rehabilitation assessment model to obtain the image processing result. The fifth data synchronization unit may further include: an enhancement processing unit for enhancing the rehabilitation effect image data to generate a plurality of enhanced images; a second conversion unit for performing standardized conversion based on the plurality of enhanced images to obtain a plurality of standard images; a transfer learning unit for performing transfer learning by traversing the plurality of standard images using a convolutional neural network to obtain a plurality of image transfer features; a detection unit for performing regional segmentation detection on the plurality of standard images based on the plurality of image transfer features to generate an image detection result; and a first evaluation unit for performing evaluation based on the image detection result to obtain the image processing result based on the evaluation result.

[0055] The specific configuration of the feedback optimization module 50 will be described in detail below. As described above, the rehabilitation training program is feedback optimized based on the multi-dimensional evaluation results to determine the rehabilitation optimization training program. The feedback optimization module 50 may further include: a cycle determination unit for determining a feedback cycle based on the execution timing of the rehabilitation training program; a direction determination unit for determining a plurality of mining directions based on the multi-dimensional evaluation results, traversing the rehabilitation training program according to the plurality of mining directions for data mining, and obtaining a plurality of rehabilitation training data; a second evaluation unit for periodically evaluating the plurality of rehabilitation training data based on the feedback cycle to generate a label set of data to be optimized; an identification unit for synchronizing the label set of data to be optimized to the rehabilitation training program through a feedback mechanism for identification, and determining a plurality of data to be optimized; an optimization unit for optimizing the rehabilitation training program according to the plurality of data to be optimized to obtain the rehabilitation optimization training program.

[0056] The large-model-based multimodal rehabilitation data intelligent evaluation system provided in the embodiments of the present application can execute the large-model-based multimodal rehabilitation data intelligent evaluation method provided in any embodiment of the present application, and has the corresponding functional modules and beneficial effects of the execution method.

[0057] Although the present application makes various references to certain modules in the system according to the embodiments of the present application, any number of different modules may be used and run on the user terminal and / or server, and the various units and modules included are only divided according to functional logic, but are not limited to the above division, as long as the corresponding functions can be achieved; in addition, the specific names of the functional units are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of the present application.

[0058] The above specific implementations do not constitute a limitation on the protection scope of this application. It should be understood by those skilled in the art that various modifications, combinations and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions and improvements made within the spirit and principles of this application should be included in the protection scope of this application.

Claims

1. A multimodal rehabilitation data intelligent evaluation method based on a large model, characterized in that: The method comprises: Collecting data based on multiple data sources to obtain multimodal data collection results, and integrating the multimodal data collection results to obtain a multimodal data set; Synchronizing the multimodal data set to an information extraction model for retrieval to determine a target information set; Conduct training verification based on the target information set to build a large rehabilitation assessment model; Implement rehabilitation training programs to track the rehabilitation progress of target users in real time and obtain rehabilitation effect data; The rehabilitation effect data of the target user is synchronized to the rehabilitation assessment model to obtain a multi-dimensional assessment result, and the rehabilitation training program is feedback-optimized based on the multi-dimensional assessment result to determine a rehabilitation optimization training program.

2. The method for intelligent evaluation of multimodal rehabilitation data based on a large model as claimed in claim 1, characterized in that: Synchronizing the multimodal data set to an information extraction model for retrieval to determine a target information set, the method comprising: Performing data annotation based on the data type of the multimodal data set to obtain a data annotation result; Dividing the multimodal data set into a plurality of text data and a plurality of image data according to the data annotation result; The information extraction model includes a text feature extraction layer, an image feature extraction layer, and an extracted feature fusion layer; Synchronizing the plurality of text data to the text feature extraction layer to obtain a plurality of text features; Synchronizing the plurality of image data to the image feature extraction layer to obtain a plurality of image features; Synchronizing the multiple text features and the multiple image features to an extraction feature fusion layer to obtain a multimodal feature fusion vector; According to the multimodal feature fusion vector, the multimodal data set is traversed for retrieval to determine the target information set.

3. The method for intelligent evaluation of multimodal rehabilitation data based on a large model as claimed in claim 2, characterized in that: Based on the target information set, training verification is performed to construct a large rehabilitation assessment model, and the method includes: Performing data preprocessing based on the target information set to generate multiple target information processing results; Dividing the multiple target information processing results into a training data set, a verification data set, and a test data set; Based on the training data set combined with the verification data set, supervised training is performed to construct a text processing layer and an image processing layer, wherein the text processing layer is obtained by training based on the text data in the training data set, and the image processing layer is obtained by training based on the image data in the training data set; Retrieving historical health assessment files and building an information assessment layer in combination with the multimodal feature fusion vector; Connecting the output end of the text processing layer and the output end of the image processing layer to the input end of the information evaluation layer; The test data set is synchronized to the information evaluation layer for testing, and when the test passes, the rehabilitation evaluation macro model is determined.

4. The method for intelligent evaluation of multimodal rehabilitation data based on a large model as claimed in claim 3, characterized in that: The rehabilitation effect data of the target user is synchronized to the rehabilitation evaluation model to obtain a multi-dimensional evaluation result, the method comprising: Performing data formatting processing based on the rehabilitation effect data of the target user to generate rehabilitation effect text data and rehabilitation effect image data; Synchronizing the rehabilitation effect text data to the text processing layer of the rehabilitation assessment large model to obtain a text processing result; Synchronizing the rehabilitation effect image data to the image processing layer of the rehabilitation assessment large model to obtain an image processing result; Using an attention mechanism to perform weighted fusion on the text processing result and the image processing result to obtain a weighted fusion result; Setting an evaluation index based on the multimodal feature fusion vector, and determining a plurality of evaluation dimensions according to the evaluation index; Based on the weighted fusion result, a multi-dimensional evaluation is performed according to the multiple evaluation dimensions to generate the multi-dimensional evaluation result.

5. The method for intelligent evaluation of multimodal rehabilitation data based on a large model as claimed in claim 4, characterized in that: Synchronizing the rehabilitation effect text data to the text processing layer of the rehabilitation assessment large model to obtain a text processing result, the method comprising: Performing word segmentation processing on the rehabilitation effect text data to generate a plurality of text words; Performing vectorization conversion on the multiple text words to obtain multiple text word vectors; Extracting context information based on the rehabilitation effect text data, performing text understanding based on the multiple text word vectors combined with the context information, and obtaining text understanding information; Extracting the correlation between the text comprehension information and the rehabilitation effect data, and constructing a correlation network; The text comprehension information is extracted according to the association network, and the text processing result is determined according to the extracted information.

6. The method for intelligent evaluation of multimodal rehabilitation data based on a large model as claimed in claim 4, characterized in that: Synchronizing the rehabilitation effect image data to the image processing layer of the rehabilitation assessment large model to obtain an image processing result, the method comprising: Performing enhancement processing on the rehabilitation effect image data to generate a plurality of enhanced images; Performing standardization conversion based on the multiple enhanced images to obtain multiple standard images; Using a convolutional neural network to traverse the multiple standard images for transfer learning, and obtain multiple image transfer features; Performing region segmentation detection on the multiple standard images based on the multiple image migration features to generate image detection results; An evaluation is performed based on the image detection result, and the image processing result is obtained based on the evaluation result.

7. The method for intelligent evaluation of multimodal rehabilitation data based on a large model as claimed in claim 1, characterized in that: Feedback optimization is performed on the rehabilitation training program based on the multi-dimensional evaluation results to determine the rehabilitation optimization training program, the method comprising: Determining a feedback cycle based on the execution timing of the rehabilitation training program; Determine multiple mining directions based on the multi-dimensional evaluation results, traverse the rehabilitation training program according to the multiple mining directions to perform data mining, and obtain multiple rehabilitation training data; Regularly evaluating the plurality of rehabilitation training data based on the feedback cycle to generate a data label set to be optimized; The tag set of the data to be optimized is synchronized to the rehabilitation training program through a feedback mechanism for identification, and a plurality of data to be optimized is determined; The rehabilitation training program is optimized according to the multiple data to be optimized to obtain the rehabilitation optimization training program.

8. A multimodal rehabilitation data intelligent evaluation system based on a large model, characterized by: The system is used to implement the large model-based multimodal rehabilitation data intelligent evaluation method according to any one of claims 1 to 7, and the system comprises: A data integration module, the data integration module is used to collect data based on multiple data sources to obtain multimodal data collection results, and integrate the multimodal data collection results to obtain a multimodal data set; A data retrieval module, the data retrieval module is used to synchronize the multimodal data set to the information extraction model for retrieval, and determine the target information set; A training verification module, the training verification module is used to perform training verification based on the target information set to build a large rehabilitation assessment model; A data tracking module, which is used to execute the rehabilitation training program to track the rehabilitation progress of the target user in real time and obtain rehabilitation effect data; A feedback optimization module is used to synchronize the rehabilitation effect data of the target user to the rehabilitation assessment model to obtain a multi-dimensional assessment result, perform feedback optimization on the rehabilitation training program based on the multi-dimensional assessment result, and determine a rehabilitation optimization training program.

Images