Multi-modal data-based disability intervention quality evaluation method and device and readable storage medium

Abstract

The invention discloses a multimodal data-based disability intervention quality evaluation method and device and a readable storage medium, and the evaluation method comprises the following steps: (1) obtaining multimodal data including image data, time sequence data, text data and table data; (2) extracting features from the image data, the time sequence data and the text data and fusing the features to obtain fused features, (3) extracting key disability indexes related to disability intervention from the table data; (4) constructing and training a disability risk prediction model based on the key disability indexes and the fusion features; and (5) performing disability risk prediction on the new data by using the disability risk prediction model, and evaluating the disability intervention quality. By utilizing the method, the evaluation accuracy of the disability intervention quality can be effectively improved, the evaluation efficiency of the disability intervention quality is improved, and the evaluation stability of the disability intervention quality is improved.

Description

Technical Field

[0001] The present invention relates to the field of medical health, and in particular to a method, device and readable storage medium for evaluating the quality of disability intervention based on multimodal data. Background Art

[0002] Elderly disability refers to a state in which the elderly suffer from impairment of their physical, psychological or social functions due to a combination of factors, thus affecting their daily living ability and self-care ability. There are many existing methods for screening and intervening in elderly disability.

[0003] For example, a Chinese patent document with publication number CN114121272A discloses a system for assessing functional impairment in the elderly. The system can evaluate the functional status of the elderly, identify functional impairment in the elderly, and intervene early, thereby delaying the occurrence and progression of disability, maintaining the functional ability of the elderly, improving the quality of life of the elderly, and achieving healthy aging.

[0004] A Chinese patent document with publication number CN112652397A discloses a method for screening, evaluating and intervening in geriatric syndrome for outpatient and community settings, including: rapid screening of geriatric syndrome in elderly patients; conducting special assessments including degree of frailty, disability, anxiety / depression, cognitive dysfunction, malnutrition and dysphagia; and providing individualized intervention advice based on the assessment conclusions.

[0005] In addition to screening and intervening in elderly disability, how to effectively evaluate the quality of intervention measures has also become an important issue.

[0006] Traditional evaluation methods have problems such as strong subjectivity and low efficiency, and a new data-driven method is urgently needed. Multimodal data contains rich information and can provide more comprehensive data support. How to use multimodal data to evaluate the quality of disability intervention is an urgent problem to be solved. Summary of the invention

[0007] The present invention provides a method, device and readable storage medium for evaluating the quality of disability intervention based on multimodal data, which can effectively improve the evaluation accuracy of the quality of disability intervention, improve the evaluation efficiency of the quality of disability intervention, and improve the evaluation stability of the quality of disability intervention.

[0008] A method for evaluating the quality of disability intervention based on multimodal data comprises the following steps:

[0009] (1) Acquire multimodal data, including image data, time series data, text data, and table data;

[0010] (2) Extract features from image data, time series data, and text data and fuse them to obtain fused features.

[0011] (3) extract key disability indicators related to disability interventions from the tabular data;

[0012] (4) Construct and train a disability risk prediction model based on key disability indicators and fusion features;

[0013] (5) Use the disability risk prediction model to predict the disability risk of new data and evaluate the quality of disability intervention.

[0014] In step (1), the image data includes body images of the elderly when they are moving; the time series data includes data of the elderly from sensors; the text data includes diagnostic descriptions of the elderly and intervention measures taken; and the tabular data includes demographic information, physiological indicators, and behavioral ability levels of the elderly.

[0015] In step (2), features are extracted and fused from image data, time series data, and text data, specifically including:

[0016] Take each data point in the time series data as a point in a line graph, and connect these points in chronological order to form a line graph; splice the obtained line graph with the image data into new multi-channel image data; use the pre-trained image classification model to extract features from the new multi-channel image data to obtain high-level features of the image data;

[0017] Preprocess the text data and segment it into words or phrases; use the pre-trained large language model to extract features from the preprocessed text data; use named entity recognition technology to extract key information related to disability intervention from the text; vectorize the extracted key information and convert it into numerical features;

[0018] The high-level features of image data and the numerical features of text data are combined through machine learning algorithms, and new fusion features are generated through hybrid interpolation methods.

[0019] In step (3), key disability indicators related to disability intervention are extracted from the table data, specifically:

[0020] Based on the attention mechanism, a Feature Transformer model is constructed for automatic feature selection. The structure of the model includes input layer, decision layer and output layer.

[0021] After the tabular data is input through the input layer, it is gradually selected and processed through multiple decision layers. Each decision layer selects some of the most important input features through the sparse attention mechanism to reduce the interference of unimportant features. The outputs of all decision layers are merged to form the final feature representation. The merged features are further processed through a final fully connected layer and finally output through the output layer.

[0022] In step (4), the disability risk prediction model uses a multi-layer perceptron MLP as a basic model, and introduces a multimodal contrastive learning algorithm to train the disability risk prediction model; during the training process, the loss function of the model is set to a cross entropy loss function containing multimodal information;

[0023] Assume that I represents the image feature vector, T represents the text feature vector, and B represents the table feature vector. Calculate the cross entropy loss function loss of the image feature vector and the text feature vector respectively. I,T , the cross entropy loss function loss of the image feature vector and the table feature vector I,B And the cross entropy loss function loss of the text feature vector and the table feature vector T,B ; Finally, the loss I,T 、loss I,B and loss T,B Perform weighted summation to obtain the cross entropy loss of multimodal information.

[0024] In step (5), the quality of disability intervention is evaluated, specifically:

[0025] Comparing the current disability risk with past disability risks, the obvious decrease in risk indicates that the quality of disability intervention is high; high-quality disability intervention measures will be incorporated into the next intervention plan for further evaluation.

[0026] A disability intervention quality evaluation device based on multimodal data, comprising:

[0027] A data acquisition module is used to acquire multimodal data, including image data, time series data, text data, and table data;

[0028] A data processing module for extracting features of multimodal data;

[0029] Data fusion module, used to fuse the features of multimodal data;

[0030] A model building module, used to build a disability risk prediction model based on fusion features;

[0031] The evaluation module is used to obtain the disability risk based on the constructed disability risk prediction model and evaluate the quality of disability intervention.

[0032] A device for evaluating the quality of disability intervention based on multimodal data comprises a memory and a processor, wherein the memory stores a computer program, and the processor implements the above-mentioned method for evaluating the quality of disability intervention when executing the computer program.

[0033] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the above-mentioned disability intervention quality assessment method.

[0034] Compared with the prior art, the present invention has the following beneficial effects:

[0035] The present invention can be applied to the technical field of disability intervention quality evaluation. By extracting and fusing the features of multimodal data, the accuracy of disability intervention quality evaluation can be effectively improved, the efficiency of disability intervention quality evaluation can be improved, and the stability of disability intervention quality evaluation can be improved. The invention has broad application prospects. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 This is a flow chart of a method for evaluating the quality of disability intervention based on multimodal data according to an embodiment of the present invention;

[0037] Figure 2 A flowchart of extracting multimodal data features in an embodiment of the present invention;

[0038] Figure 3 A flowchart of constructing a disability risk prediction model in an embodiment of the present invention;

[0039] Figure 4 The present invention is a block diagram of a device for evaluating the quality of disability intervention based on multimodal data in an embodiment of the present invention. DETAILED DESCRIPTION

[0040] The present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be pointed out that the embodiments described below are intended to facilitate the understanding of the present invention and do not have any limiting effect on the present invention.

[0041] like Figure 1 As shown, a method for evaluating the quality of disability intervention based on multimodal data includes the following steps:

[0042] S101, acquiring multimodal data, including image data, time series data, text data and table data.

[0043] Specifically, image data can be body images of the elderly when they are moving. Time series data can be data about the elderly from sensors, including acceleration counts, etc. Text data can be descriptions of the elderly’s diagnoses and intervention measures taken, etc. Tabular data can be demographic information, physiological indicators, and behavioral ability levels of the elderly, etc.

[0044] S102, extracting features of multimodal data.

[0045] Specifically, the time series data is converted into a line graph and merged with the image data into new image data. Key indicators related to disability intervention are extracted from the table data. Intervention measures are extracted from the text data.

[0046] S103, integrating features of multimodal data.

[0047] As an implementation method, high-level features of image data are extracted through a pre-trained image classification model.

[0048] As an implementation method, high-level features of the text data are extracted by processing the text data through a large language model.

[0049] As an implementation method, the extracted high-level features of the image data are combined with the extracted high-level features of the text data through a machine learning algorithm.

[0050] As an implementation method, new fusion features are generated by a hybrid interpolation method.

[0051] S104, disability risk prediction model based on fusion features.

[0052] As an implementation method, a disability risk prediction model is constructed through comparative learning based on disability indicators and fusion features.

[0053] S105, obtain disability risk and evaluate the quality of disability intervention.

[0054] As an implementation method, the current disability risk is compared with the past disability risk. A significant decrease in risk indicates that the disability intervention is of high quality. The high-quality disability intervention measures are included in the next intervention plan for further evaluation.

[0055] Through the above steps, the quality of disability intervention can be evaluated and the accuracy and efficiency of the evaluation can be improved.

[0056] Based on the above embodiments, Figure 2 The steps of multimodal feature extraction are decomposed and refined. Figure 2 As shown, the implementation process includes the following:

[0057] S201, converting the time series data into a line graph and merging it with the image data into new image data.

[0058] As an implementation method, the specific implementation steps are as follows:

[0059] Time series data preprocessing: De-noise time series data such as accelerometer data and heart rate data to remove outliers and noise to ensure data quality, and perform standardization to convert the data to the same scale for subsequent processing. Treat each data point in the time series data as a point in a line graph, and connect these points in chronological order to form a line graph. Optionally, use different colors or line types to represent different types of time series data for easy distinction.

[0060] Image data preprocessing: adjust the image size, remove the background, etc. to ensure the quality of the image data. Use data enhancement techniques, including rotation, flipping, cropping and other operations to increase the diversity of image data and improve the robustness of the model.

[0061] Merge time series data with image data: Merge the generated line graph with the preprocessed image data to form new image data. Optionally, concatenate the line graph and image data together to form multi-channel image data. Make sure the merged image data has a consistent size and format for subsequent processing.

[0062] Feature extraction: Preprocess the merged image data, such as resizing and normalizing the image, to meet the model's input requirements. Load a pretrained image classification model and set it to evaluation mode. Input the preprocessed image data into the pretrained model and obtain the model's intermediate layer output as the high-level features of the image. Optionally, use Vision Transformer and use the output of the last Transformer encoder layer as features. Optionally, introduce Global Average Pooling to post-process the extracted high-level features to obtain a fixed-length feature vector. Standardize the feature vector to facilitate subsequent feature fusion and model training.

[0063] S202, extract key indicators related to disability intervention from the tabular data.

[0064] As an implementation method, a FeatureTransformer model is constructed based on the attention mechanism for automatic feature selection. The structure of the model includes an input layer, a decision layer, and an output layer.

[0065] In the input layer, the input data is usually a tabular data containing multiple features, each of which can be numerical, categorical, or a combination thereof. For numerical features, standardization is performed (such as a mean of 0 and a variance of 1) to ensure that the data is trained on the same scale. For categorical features, the categorical variables are converted into dense vector representations in the embedding layer. These embedding layers learn the embedding vectors for each category to capture the potential relationship between categorical features.

[0066] The model gradually selects and processes features through multiple decision layers. Each decision layer consists of two important parts: sparse attention mechanism and feature selection. Each decision layer selects some input features through the sparse attention mechanism to form an attention distribution. Unlike traditional neural networks that pay attention to all input features, this mechanism selects the most important features through a mask matrix to reduce the interference of unimportant features. In each decision layer, the model calculates the importance weights of a set of features based on the current context, and uses this weight to determine which features participate in the calculation of the current step. Through the sparse attention mechanism, the model only pays attention to a few features, reducing the computational complexity and improving the model's focus on key features. Each decision layer also transforms the selected features and transforms the features through several fully connected layers to extract more abstract and high-level feature representations. Output of each decision step: After the feature transformation, the model will obtain a new set of feature representations as input for the next step.

[0067] The outputs of all decision layers are combined to form the final feature representation. The outputs of each decision step are combined by weighted averaging, which is similar to the voting process in ensemble learning. The aggregated features are further processed through a final fully connected layer to obtain the output of the model. This can be the class probability in a classification task, or the predicted value in a regression task.

[0068] S203, extract intervention measures from text data.

[0069] Use a large language model to process text data and extract semantic features of the text. Extract key intervention measures, diagnostic descriptions, and other information from the text. Vectorize the extracted features to facilitate subsequent feature fusion.

[0070] As an implementation method, the text data processing process may include text preprocessing, feature extraction, key information extraction, and feature vectorization steps.

[0071] Text preprocessing: Tokenize the text data, splitting the text into words or phrases, and remove stop words (such as meaningless words like "的" and "了") and punctuation marks. Optionally, perform lemmatization or stemming to retain meaningful vocabulary and unify different forms of vocabulary into their basic forms.

[0072] Feature extraction: Use pre-trained large language models (such as BERT, GPT, etc.) to extract features from the preprocessed text data. Input the text data into the large language model to obtain the context semantic vector representation of each vocabulary. For long texts, use the sliding window technique to split the text into multiple segments, perform feature extraction on each segment separately, and then average or concatenate the segment features to obtain the feature representation of the entire text.

[0073] Key information extraction: Use named entity recognition technology to extract key intervention measures, diagnostic descriptions, and other information from the text. Optionally, identify entities (such as disease names, drug names, intervention measures, etc.) in the text through a pre-trained NER model. Classify the identified entities and extract the key information related to disability intervention.

[0074] Feature vectorization: Vectorize the extracted key information and convert it into numerical features. Optionally, use the Word2Vec word embedding technique to convert the vocabulary in the text into vector representations. For the extracted entities, entity embedding techniques can be used to convert the entities into vector representations. Through vectorization, the text data is converted into numerical features, facilitating subsequent feature fusion and model training.

[0075] Based on the above embodiments, Figure 3 The steps of constructing a disability risk prediction model are decomposed and refined. As Figure 3 shown, it includes the following implementation processes:

[0076] S301, Model training:

[0077] As an implementation method, use a multi-layer perceptron (Multi-Layer Perceptron) as the basic model and introduce a multi-modal contrast learning algorithm to train the disability risk prediction model.

[0078] Optionally, set the loss function of the model as a cross-entropy loss function containing multi-modal information to achieve contrast learning of multi-modal data. Assume that I represents the image feature vector, T represents the text feature vector, and B represents the table feature vector. First, normalize the multi-modal feature vectors:

[0079]

[0080] Then the similarity between the image feature vector and the text feature vector, the similarity between the image feature vector and the table feature vector, and the similarity between the text feature vector and the table feature vector are calculated respectively:

[0081]

[0082] in, and They represent the normalized image feature vector, text feature vector, and table feature vector respectively, and τ represents the temperature parameter, which is used to scale the similarity score to control the model's sensitivity to similarity. I,T Represents the similarity between the image feature vector and the text feature vector, logits I,B Represents the similarity between the image feature vector and the table feature vector, logits T,B Indicates the similarity between the text feature vector and the table feature vector.

[0083] Furthermore, the cross entropy loss function between the image feature vector and the text feature vector, the cross entropy loss function between the image feature vector and the table feature vector, and the cross entropy loss function between the text feature vector and the table feature vector are calculated respectively:

[0084] loss I,T = CrossEntropyLoss(logits I,T ,labels)

[0085] loss I,B = CrossEntropyLoss(logits I,B ,labels)

[0086] loss T,B = CrossEntropyLoss(logits T,B ,labels)

[0087] Among them, labels represents labels, CrossEntropyLoss represents the cross entropy loss function, and loss I,T Represents the cross entropy loss function of the image feature vector and the text feature vector, loss I,B Represents the cross entropy loss function of the image feature vector and the table feature vector, loss T,B Represents the cross entropy loss function for text feature vectors and table feature vectors.

[0088] Finally, the loss I,T 、loss I,B and loss T,BPerform weighted summation to obtain the cross entropy loss of multimodal information:

[0089]

[0090] S302, Model Evaluation:

[0091] Evaluate the trained disability risk prediction model.

[0092] Optionally, use cross-validation methods to evaluate the performance of the model.

[0093] Optionally, use accuracy, recall, F1 score, AUC and other evaluation indicators to evaluate the effectiveness of the model.

[0094] Optionally, use visualization tools such as confusion matrix and ROC curve to display the evaluation results of the model.

[0095] S303, Disability Risk Prediction:

[0096] Use the trained model to predict disability risk for new data.

[0097] Compare the predicted results with the actual results to verify the model's predictive ability. Generate a disability risk report for reference by clinicians.

[0098] Through the above steps, the quality of disability intervention can be evaluated and the accuracy and efficiency of the evaluation can be improved.

[0099] Based on the same inventive concept, the embodiment of the present application also provides a disability intervention quality evaluation device for implementing the above-mentioned disability intervention quality evaluation method. Figure 4 As shown, a disability intervention quality evaluation device 4 is provided, which includes: an acquisition module 40, a processing module 41, a fusion module 42, a construction module 43 and an evaluation module 44, wherein:

[0100] An acquisition module 40, for acquiring multimodal data, including image data, time series data, text data and table data;

[0101] A processing module 41, used to extract features of multimodal data;

[0102] A fusion module 42, used for fusing features of multimodal data;

[0103] A construction module 43 is used to construct a disability risk prediction model based on the fusion features;

[0104] Evaluation module 44 is used to evaluate the quality of disability intervention.

[0105] The above-mentioned disability intervention quality evaluation device can be applied to the field of disability intervention quality evaluation technology, and can specifically include modules such as multimodal data acquisition, feature extraction, feature fusion, model construction, and quality evaluation. By extracting the features of multimodal data, fusing the features of multimodal data, selecting appropriate features, training models, and evaluating models, the evaluation of the quality of disability intervention can be achieved. The disability intervention quality evaluation device based on multimodal data provided by the present invention can effectively improve the evaluation accuracy of disability intervention quality, improve the evaluation efficiency of disability intervention quality, and improve the evaluation stability of disability intervention quality. It can be applied to the evaluation of disability intervention quality and has broad application prospects.

[0106] Based on the same inventive concept, an embodiment of the present invention further provides a disability intervention quality assessment device based on multimodal data, comprising a memory and a processor, wherein the memory stores a computer program, and the processor implements the above-mentioned disability intervention quality assessment method when executing the computer program.

[0107] Based on the same inventive concept, an embodiment of the present invention further provides a computer-readable storage medium having a computer program stored thereon, and the computer program implements the above-mentioned disability intervention quality assessment method when executed by a processor.

[0108] The embodiments described above provide a detailed description of the technical solutions and beneficial effects of the present invention. It should be understood that the above are only specific embodiments of the present invention and are not intended to limit the present invention. Any modifications, supplements and equivalent substitutions made within the scope of the principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for evaluating the quality of disability intervention based on multimodal data, characterized in that: The following steps are involved: (1) Acquire multimodal data, including image data, time series data, text data, and table data; (2) Extract features from image data, time series data, and text data and fuse them to obtain fused features. (3) extract key disability indicators related to disability interventions from the tabular data; (4) Construct and train a disability risk prediction model based on key disability indicators and fusion features; (5) Use the disability risk prediction model to predict the disability risk of new data and evaluate the quality of disability intervention.

2. The method for evaluating the quality of disability intervention based on multimodal data according to claim 1, characterized in that: In step (1), the image data includes body images of the elderly when they are moving; the time series data includes data of the elderly from sensors; the text data includes diagnostic descriptions of the elderly and intervention measures taken; and the tabular data includes demographic information, physiological indicators, and behavioral ability levels of the elderly.

3. The method for evaluating the quality of disability intervention based on multimodal data according to claim 1, characterized in that: In step (2), features are extracted and fused from image data, time series data, and text data, specifically including: Take each data point in the time series data as a point in a line graph, and connect these points in chronological order to form a line graph; splice the obtained line graph with the image data into new multi-channel image data; use the pre-trained image classification model to extract features from the new multi-channel image data to obtain high-level features of the image data; Preprocess the text data and segment it into words or phrases; use the pre-trained large language model to extract features from the preprocessed text data; use named entity recognition technology to extract key information related to disability intervention from the text; vectorize the extracted key information and convert it into numerical features; The high-level features of image data and the numerical features of text data are combined through machine learning algorithms, and new fusion features are generated through hybrid interpolation methods.

4. The method for evaluating the quality of disability intervention based on multimodal data according to claim 1, characterized in that: In step (3), key disability indicators related to disability intervention are extracted from the table data, specifically: Based on the attention mechanism, a Feature Transformer model is constructed for automatic feature selection. The structure of the model includes input layer, decision layer and output layer. After the tabular data is input through the input layer, it is gradually selected and processed through multiple decision layers. Each decision layer selects some of the most important input features through the sparse attention mechanism to reduce the interference of unimportant features. The outputs of all decision layers are merged to form the final feature representation. The merged features are further processed through a final fully connected layer and finally output through the output layer.

5. The method for evaluating the quality of disability intervention based on multimodal data according to claim 1, characterized in that: In step (4), the disability risk prediction model uses a multi-layer perceptron MLP as a basic model, and introduces a multimodal contrastive learning algorithm to train the disability risk prediction model; during the training process, the loss function of the model is set to a cross entropy loss function containing multimodal information; Assume that I represents the image feature vector, T represents the text feature vector, and B represents the table feature vector. Calculate the cross entropy loss function loss of the image feature vector and the text feature vector respectively. I,T , the cross entropy loss function loss of the image feature vector and the table feature vector I,B And the cross entropy loss function loss of the text feature vector and the table feature vector T,B ; Finally, the loss I,T 、loss I,B and loss T,B Perform weighted summation to obtain the cross entropy loss of multimodal information.

6. The method for evaluating the quality of disability intervention based on multimodal data according to claim 1, characterized in that: In step (5), the quality of disability intervention is evaluated, specifically: Comparing the current disability risk with past disability risks, the obvious decrease in risk indicates that the quality of disability intervention is high; high-quality disability intervention measures will be incorporated into the next intervention plan for further evaluation.

7. A disability intervention quality assessment device based on multimodal data, characterized in that: include: A data acquisition module is used to acquire multimodal data, including image data, time series data, text data, and table data; A data processing module for extracting features of multimodal data; Data fusion module, used to fuse the features of multimodal data; A model building module, used to build a disability risk prediction model based on fusion features; The evaluation module is used to obtain the disability risk based on the constructed disability risk prediction model and evaluate the quality of disability intervention.

8. A device for evaluating the quality of disability intervention based on multimodal data, comprising a memory and a processor, wherein the memory stores a computer program, characterized in that: When the processor executes the computer program, the method for evaluating the quality of disability intervention according to any one of claims 1 to 6 is implemented.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method for evaluating the quality of disability intervention according to any one of claims 1 to 6 is implemented.

Images