A method and device for evaluating the quality of disability intervention based on multi-modal data, and a readable storage medium

Abstract

The application discloses a kind of based on multimodal data's disablement intervention quality evaluation method, device and readable storage medium, wherein, evaluation method includes following steps: (1) obtain multimodal data, include image data, time series data, text data and table data;(2) feature is extracted from image data, time series data, text data and is fused, obtains fusion feature, (3) from table data, extract the key disability index related to disablement intervention;(4) based on key disability index and fusion feature, construct and train disablement risk prediction model;(5) utilize disablement risk prediction model to carry out disablement risk prediction to new data, and evaluate disablement intervention quality.Utilize the present application, can effectively improve the evaluation accuracy of disablement intervention quality, improve the evaluation efficiency of disablement intervention quality, improve the evaluation stability of disablement intervention quality.

Description

Technical Field

[0001] The present invention relates to the field of medical health, and in particular to a method, device and readable storage medium for evaluating the quality of disability intervention based on multimodal data. Background Art

[0002] Geriatric disability refers to a condition in which a combination of factors impairs the physical, psychological, or social functions of older adults, affecting their ability to live and care for themselves. Numerous methods exist for screening and intervention for geriatric disability.

[0003] For example, Chinese patent document CN114121272A discloses a functional impairment assessment system for the elderly. The system evaluates the functional status of the elderly, identifies functional impairment in the elderly, and intervenes early, thereby delaying the occurrence and progression of disability, maintaining the functional ability of the elderly, improving the quality of life of the elderly, and achieving healthy aging.

[0004] Chinese patent document publication number CN112652397A discloses a method for screening, assessment, and intervention of geriatric syndromes for outpatient and community settings, including: rapid screening of elderly patients for geriatric syndromes; conducting specialized assessments including degree of frailty, disability, anxiety / depression, cognitive impairment, malnutrition, and dysphagia; and providing individualized intervention advice based on the assessment conclusions.

[0005] In addition to screening and intervention for elderly disability, how to effectively evaluate the quality of intervention measures has also become an important issue.

[0006] Traditional evaluation methods suffer from subjectivity and low efficiency, necessitating a new, data-driven approach. Multimodal data, rich in information, can provide more comprehensive data support. Using multimodal data to evaluate the quality of disability interventions is a pressing issue. Summary of the Invention

[0007] The present invention provides a method, device and readable storage medium for evaluating the quality of disability intervention based on multimodal data, which can effectively improve the evaluation accuracy of disability intervention quality, improve the evaluation efficiency of disability intervention quality, and improve the evaluation stability of disability intervention quality.

[0008] A method for evaluating the quality of disability intervention based on multimodal data comprises the following steps:

[0009] (1) Acquire multimodal data, including image data, time series data, text data, and table data;

[0010] (2) Extract features from image data, time series data, and text data and fuse them to obtain fused features.

[0011] (3) extract key disability indicators related to disability intervention from the tabular data;

[0012] (4) Construct and train a disability risk prediction model based on key disability indicators and fusion features;

[0013] (5) Use the disability risk prediction model to predict the disability risk of new data and evaluate the quality of disability intervention.

[0014] In step (1), the image data includes body images of the elderly when they are moving; the time series data includes data of the elderly from sensors; the text data includes diagnostic descriptions of the elderly and intervention measures taken; and the tabular data includes demographic information, physiological indicators, and behavioral ability levels of the elderly.

[0015] In step (2), features are extracted and fused from image data, time series data, and text data, specifically including:

[0016] Treat each data point in the time series data as a point in a line graph, and connect these points in chronological order to form a line graph; splice the obtained line graph with the image data to form new multi-channel image data; use the pre-trained image classification model to extract features from the new multi-channel image data to obtain high-level features of the image data;

[0017] Preprocess the text data and segment it into words or phrases; use a pre-trained large language model to extract features from the preprocessed text data; use named entity recognition technology to extract key information related to disability intervention from the text; vectorize the extracted key information and convert it into numerical features;

[0018] The high-level features of image data and the numerical features of text data are combined through machine learning algorithms, and new fusion features are generated through hybrid interpolation methods.

[0019] In step (3), key disability indicators related to disability intervention are extracted from the table data, specifically:

[0020] A Feature Transformer model is constructed based on the attention mechanism for automatic feature selection. The structure of the model includes input layer, decision layer and output layer.

[0021] After the tabular data is input through the input layer, it is gradually selected and processed through multiple decision layers. Each decision layer selects some of the most important input features through the sparse attention mechanism to reduce the interference of unimportant features; the outputs of all decision layers are merged to form the final feature representation. The merged features are further processed through a final fully connected layer and finally output through the output layer.

[0022] In step (4), the disability risk prediction model uses a multi-layer perceptron (MLP) as the basic model, and introduces a multimodal contrastive learning algorithm to train the disability risk prediction model; during the training process, the loss function of the model is set to a cross-entropy loss function containing multimodal information;

[0023] Assume that I represents the image feature vector, T represents the text feature vector, and B represents the table feature vector. Calculate the cross entropy loss function loss of the image feature vector and the text feature vector respectively. I,T , cross entropy loss function loss of image feature vector and table feature vector I,B and the cross entropy loss function loss of the text feature vector and the table feature vector T,B ; Finally, the loss I,T 、loss I,B and loss T,B Perform weighted summation to obtain the cross entropy loss of multimodal information.

[0024] In step (5), the quality of disability intervention is evaluated, specifically:

[0025] Comparing the current disability risk with past disability risks, the significant decrease in risk indicates that the quality of disability intervention is high; high-quality disability intervention measures will be incorporated into the next intervention plan for further evaluation.

[0026] A device for evaluating the quality of disability intervention based on multimodal data, comprising:

[0027] Data acquisition module, used to acquire multimodal data, including image data, time series data, text data and table data;

[0028] Data processing module, used to extract features of multimodal data;

[0029] Data fusion module, used to fuse the features of multimodal data;

[0030] A model building module is used to build a disability risk prediction model based on fusion features;

[0031] The evaluation module is used to obtain disability risk based on the constructed disability risk prediction model and evaluate the quality of disability intervention.

[0032] A device for evaluating the quality of disability intervention based on multimodal data includes a memory and a processor. The memory stores a computer program, and the processor implements the above-mentioned method for evaluating the quality of disability intervention when executing the computer program.

[0033] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the above-mentioned disability intervention quality assessment method.

[0034] Compared with the prior art, the present invention has the following beneficial effects:

[0035] The present invention can be applied to the field of disability intervention quality evaluation technology. By extracting and fusing the features of multimodal data, it can effectively improve the evaluation accuracy of disability intervention quality, improve the evaluation efficiency of disability intervention quality, and improve the evaluation stability of disability intervention quality. It has broad application prospects. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 This is a flow chart of a method for evaluating disability intervention quality based on multimodal data according to an embodiment of the present invention;

[0037] Figure 2 This is a flowchart of extracting multimodal data features in an embodiment of the present invention;

[0038] Figure 3 This is a flowchart of constructing a disability risk prediction model in an embodiment of the present invention;

[0039] Figure 4 This is a structural block diagram of a disability intervention quality assessment device based on multimodal data in an embodiment of the present invention. DETAILED DESCRIPTION

[0040] The present invention will be described in further detail below with reference to the accompanying drawings and examples. It should be noted that the following examples are intended to facilitate understanding of the present invention and do not have any limiting effect on the present invention.

[0041] like Figure 1 As shown, a method for evaluating the quality of disability intervention based on multimodal data includes the following steps:

[0042] S101, acquiring multimodal data, including image data, time series data, text data, and table data.

[0043] Specifically, image data can include images of an elderly person's body while they're moving. Time series data can include sensor data on the elderly, including accelerometers. Text data can include descriptions of the elderly's diagnosis and interventions. Tabular data can include demographic information, physiological indicators, and behavioral ability levels of the elderly.

[0044] S102: Extract features of multimodal data.

[0045] Specifically, the time series data was converted into a line graph and merged with the image data to form new image data. Key indicators related to disability intervention were extracted from the tabular data. Intervention measures were extracted from the text data.

[0046] S103, integrating features of multimodal data.

[0047] As an implementation method, high-level features of image data are extracted through a pre-trained image classification model.

[0048] As an implementation method, text data is processed by a large language model to extract high-level features of the text data.

[0049] As an implementation method, the extracted high-level features of the image data are combined with the extracted high-level features of the text data through a machine learning algorithm.

[0050] As an implementation method, new fusion features are generated by a hybrid interpolation method.

[0051] S104, disability risk prediction model based on fusion features.

[0052] As an implementation method, a disability risk prediction model is constructed through comparative learning based on disability indicators and fusion features.

[0053] S105, obtain disability risk and evaluate the quality of disability intervention.

[0054] As an implementation method, a significant decrease in the current disability risk compared to past disability risks indicates high-quality disability intervention. High-quality disability intervention measures are then incorporated into the next intervention plan for further evaluation.

[0055] Through the above steps, the quality of disability intervention can be evaluated and the accuracy and efficiency of the evaluation can be improved.

[0056] Based on the above embodiment, Figure 2 The steps of multimodal feature extraction are decomposed and refined. Figure 2 As shown, the implementation process includes the following:

[0057] S201, converting the time series data into a line graph and merging it with the image data into new image data.

[0058] As an implementation method, the specific implementation steps are as follows:

[0059] Time series data preprocessing: De-noise time series data, such as accelerometer data and heart rate data, to remove outliers and noise to ensure data quality. Standardize the data to a consistent scale for easier processing. Each data point in the time series data is treated as a point on a line graph, and these points are connected chronologically to form a line graph. Optionally, use different colors or line types to represent different types of time series data for easier differentiation.

[0060] Image data preprocessing: Resize the image, remove the background, and ensure the quality of the image data. Use data augmentation techniques, including rotation, flipping, and cropping, to increase the diversity of the image data and improve the robustness of the model.

[0061] Merge Time Series Data with Image Data: Merge the generated line graph with the preprocessed image data to form new image data. Optionally, concatenate the line graph and image data together to form multi-channel image data. Ensure that the merged image data has consistent size and format to facilitate subsequent processing.

[0062] Feature extraction: Preprocess the merged image data, such as resizing and normalizing the image, to meet the input requirements of the model. Load the pretrained image classification model and set it to evaluation mode. Input the preprocessed image data into the pretrained model and obtain the output of the model's intermediate layers as high-level features of the image. Optionally, use Vision Transformer and use the output of the last Transformer encoder layer as features. Optionally, introduce Global Average Pooling to post-process the extracted high-level features to obtain a fixed-length feature vector. Standardize the feature vector to facilitate subsequent feature fusion and model training.

[0063] S202, extract key indicators related to disability intervention from the table data.

[0064] As an implementation method, a FeatureTransformer model is constructed based on the attention mechanism for automatic feature selection. The structure of the model includes an input layer, a decision layer, and an output layer.

[0065] In the input layer, the input data is typically tabular data containing multiple features, each of which can be numerical, categorical, or a combination thereof. Numerical features are normalized (e.g., to a mean of 0 and a variance of 1) to ensure that the data is trained on the same scale. For categorical features, the embedding layer converts the categorical variables into dense vector representations. These embedding layers learn an embedding vector for each category, capturing the underlying relationships between categorical features.

[0066] The model progressively selects and processes features through multiple decision layers. Each decision layer consists of two key components: sparse attention and feature selection. Each decision layer uses the sparse attention mechanism to select a subset of input features, forming an attention distribution. Unlike traditional neural networks that focus on all input features, this mechanism uses a mask matrix to select the most important features, reducing the interference of less important features. In each decision layer, the model calculates the importance weights of a set of features based on the current context and uses these weights to determine which features participate in the calculation of the current step. The sparse attention mechanism focuses on only a few features, reducing computational complexity while increasing the model's focus on key features. Each decision layer also transforms the selected features, transforming them through several fully connected layers to extract more abstract and high-level feature representations. The output of each decision step: After feature transformation, the model obtains a new set of feature representations, which serve as input for the next step.

[0067] The outputs of all decision layers are combined to form the final feature representation. The outputs of each decision step are combined using a weighted average, a process similar to the voting process in ensemble learning. The aggregated features are further processed through a final fully connected layer to produce the model's output. This can be a class probability in a classification task or a predicted value in a regression task.

[0068] S203, extract intervention measures from text data.

[0069] Use a large language model to process text data and extract semantic features. Extract key intervention measures, diagnostic descriptions, and other information from the text. Vectorize the extracted features to facilitate subsequent feature fusion.

[0070] As an implementation method, the text data processing process may include text preprocessing, feature extraction, key information extraction, and feature vectorization steps.

[0071] Text preprocessing: Perform word segmentation on the text data, splitting the text into words or phrases, removing stop words (such as meaningless words like "de", "le", etc.) and punctuation marks. Optionally, perform lemmatization or stemming to retain meaningful words and unify different forms of words into their basic forms.

[0072] Feature extraction: Use pre-trained large language models (such as BERT, GPT, etc.) to extract features from the preprocessed text data. Input the text data into the large language model to obtain the context semantic vector representation of each word. For long texts, use the sliding window technique to split the text into multiple segments, perform feature extraction on each segment separately, and then average or splice the segment features to obtain the feature representation of the entire text.

[0073] Key information extraction: Use named entity recognition technology to extract key intervention measures, diagnostic descriptions, etc. from the text. Optionally, identify entities (such as disease names, drug names, intervention measures, etc.) in the text through a pre-trained NER model. Classify the identified entities and extract the key information related to disability intervention.

[0074] Feature vectorization: Represent the extracted key information in vector form and convert it into numerical features. Optionally, use Word2Vec word embedding technology to convert words in the text into vector representations. For the extracted entities, entity embedding technology can be used to convert the entities into vector representations. Through vector representation, the text data is converted into numerical features, facilitating subsequent feature fusion and model training.

[0075] Based on the above embodiments, through Figure 3 the steps of constructing a disability risk prediction model are decomposed and refined. As Figure 3 shown, it includes the following implementation processes:

[0076] S301, Model training:

[0077] As an implementation method, a multi-layer perceptron (Multi-Layer Perceptron) is used as the basic model, and a multi-modal contrast learning algorithm is introduced to train the disability risk prediction model.

[0078] Optionally, set the loss function of the model to a cross-entropy loss function containing multi-modal information to achieve contrast learning of multi-modal data. Assume that I represents the image feature vector, T represents the text feature vector, and B represents the table feature vector. First, normalize the multi-modal feature vectors:

[0079]

[0080] Then calculate the similarity between the image feature vector and the text feature vector, the similarity between the image feature vector and the table feature vector, and the similarity between the text feature vector and the table feature vector:

[0081]

[0082] in, and They represent the normalized image feature vector, text feature vector, and table feature vector respectively, and τ represents the temperature parameter, which is used to scale the similarity score to control the model's sensitivity to similarity. I,T Represents the similarity between the image feature vector and the text feature vector, logits I,B Represents the similarity between the image feature vector and the table feature vector, logits T,B Indicates the similarity between the text feature vector and the table feature vector.

[0083] Furthermore, the cross entropy loss function between the image feature vector and the text feature vector, the cross entropy loss function between the image feature vector and the table feature vector, and the cross entropy loss function between the text feature vector and the table feature vector are calculated respectively:

[0084] loss I,T =CrossEntropyLoss(logits I,T ,labels)

[0085] loss I,B =CrossEntropyLoss(logits I,B ,labels)

[0086] loss T,B =CrossEntropyLoss(logits T,B ,labels)

[0087] Among them, labels represents labels, CrossEntropyLoss represents the cross entropy loss function, and loss I,T Represents the cross entropy loss function of the image feature vector and the text feature vector, loss I,B Represents the cross entropy loss function of the image feature vector and the table feature vector, loss T,B Represents the cross entropy loss function between text feature vectors and table feature vectors.

[0088] Finally, the loss I,T 、loss I,B and loss T,BPerform weighted summation to obtain the cross entropy loss of multimodal information:

[0089]

[0090] S302, Model Evaluation:

[0091] Evaluate the trained disability risk prediction model.

[0092] Optionally, use cross-validation to evaluate the performance of the model.

[0093] Optionally, use evaluation indicators such as accuracy, recall, F1 score, and AUC to evaluate the effectiveness of the model.

[0094] Optionally, use visualization tools such as confusion matrix and ROC curve to display the evaluation results of the model.

[0095] S303, Disability Risk Prediction:

[0096] Use the trained model to predict disability risk for new data.

[0097] Compare the predicted results with the actual results to verify the model's predictive ability. Generate a disability risk report for clinicians' reference.

[0098] Through the above steps, the quality of disability intervention can be evaluated and the accuracy and efficiency of the evaluation can be improved.

[0099] Based on the same inventive concept, the embodiment of the present application also provides a disability intervention quality evaluation device for implementing the above-mentioned disability intervention quality evaluation method. Figure 4 As shown, a disability intervention quality evaluation device 4 is provided, which includes: an acquisition module 40, a processing module 41, a fusion module 42, a construction module 43 and an evaluation module 44, wherein:

[0100] An acquisition module 40 is used to acquire multimodal data, including image data, time series data, text data, and table data;

[0101] A processing module 41 is used to extract features of multimodal data;

[0102] A fusion module 42 for fusing features of multimodal data;

[0103] A construction module 43 is used to construct a disability risk prediction model based on the fusion features;

[0104] Evaluation module 44 is used to evaluate the quality of disability intervention.

[0105] The above-mentioned disability intervention quality evaluation device can be applied to the field of disability intervention quality evaluation technology, and can specifically include modules such as multimodal data acquisition, feature extraction, feature fusion, model construction, and quality evaluation. By extracting the features of multimodal data, fusing the features of multimodal data, selecting appropriate features, training models, and evaluating models, the evaluation of the quality of disability intervention can be achieved. The disability intervention quality evaluation device based on multimodal data provided by the present invention can effectively improve the evaluation accuracy of disability intervention quality, improve the evaluation efficiency of disability intervention quality, and improve the evaluation stability of disability intervention quality. It can be applied to the evaluation of disability intervention quality and has broad application prospects.

[0106] Based on the same inventive concept, an embodiment of the present invention also provides a disability intervention quality evaluation device based on multimodal data, including a memory and a processor, the memory storing a computer program, and the processor implementing the above-mentioned disability intervention quality evaluation method when executing the computer program.

[0107] Based on the same inventive concept, an embodiment of the present invention further provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the above-mentioned disability intervention quality evaluation method is implemented.

[0108] The embodiments described above provide a detailed description of the technical solutions and beneficial effects of the present invention. It should be understood that the above are only specific embodiments of the present invention and are not intended to limit the present invention. Any modifications, supplements and equivalent substitutions made within the scope of the principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A method for evaluating the quality of disability intervention based on multimodal data, characterized in that: The following steps are involved: (1) Acquire multimodal data, including image data, time series data, text data, and tabular data; image data includes body images of the elderly when they are moving; time series data includes data of the elderly from sensors; text data includes descriptions of the elderly’s diagnoses and intervention measures; tabular data includes demographic information, physiological indicators, and behavioral ability levels of the elderly; (2) Extracting and fusing features from image data, time series data, and text data to obtain fused features, specifically including: Treat each data point in the time series data as a point in a line graph, and connect these points in chronological order to form a line graph; splice the obtained line graph with the image data to form new multi-channel image data; use the pre-trained image classification model to extract features from the new multi-channel image data to obtain high-level features of the image data; Preprocess the text data and segment it into words or phrases; use a pre-trained large language model to extract features from the preprocessed text data; use named entity recognition technology to extract key information related to disability intervention from the text; vectorize the extracted key information and convert it into numerical features; The high-level features of image data and the numerical features of text data are combined through machine learning algorithms, and new fusion features are generated through hybrid interpolation methods; (3) Extract key disability indicators related to disability intervention from the tabular data; specifically, construct a Feature Transformer model based on the attention mechanism for automatic feature selection. The model structure includes an input layer, a decision layer, and an output layer; (4) Based on key disability indicators and fusion features, a disability risk prediction model is constructed and trained. The disability risk prediction model uses a multi-layer perceptron (MLP) as the basic model, and introduces a multimodal contrastive learning algorithm to train the disability risk prediction model. During the training process, the model's loss function is set to a cross-entropy loss function that includes multimodal information. Assume that I represents the image feature vector, T represents the text feature vector, and B represents the table feature vector. Calculate the cross entropy loss function loss of the image feature vector and the text feature vector respectively. I,T , cross entropy loss function loss of image feature vector and table feature vector I,B and the cross entropy loss function loss of the text feature vector and the table feature vector T,B ; Finally, the loss I,T 、loss I,B and loss T,B Perform weighted summation to obtain the cross entropy loss of multimodal information; (5) Use the disability risk prediction model to predict the disability risk of new data and evaluate the quality of disability intervention.

2. The method for evaluating the quality of disability intervention based on multimodal data according to claim 1, characterized in that: In step (3), after the tabular data is input through the input layer, it is gradually selected and processed through multiple decision layers. Each decision layer selects some of the most important input features through the sparse attention mechanism to reduce the interference of unimportant features. The outputs of all decision layers are merged to form the final feature representation. The merged features are further processed through a final fully connected layer and finally output through the output layer.

3. The method for evaluating disability intervention quality based on multimodal data according to claim 1, characterized in that: In step (5), the quality of disability intervention is evaluated, specifically: Comparing the current disability risk with past disability risks, the significant decrease in risk indicates that the quality of disability intervention is high; high-quality disability intervention measures will be incorporated into the next intervention plan for further evaluation.

4. A device for evaluating the quality of disability intervention based on multimodal data, comprising a memory and a processor, wherein the memory stores a computer program, characterized in that: When the processor executes the computer program, the method for evaluating the quality of disability intervention according to any one of claims 1 to 3 is implemented.

5. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method for evaluating the quality of disability intervention according to any one of claims 1 to 3 is implemented.

Images