Diabetic foot ulcer risk prediction method and system based on multi-modal information

Abstract

The invention provides a diabetic foot ulcer risk prediction method and system based on multi-modal information, and belongs to the technical field of medical artificial intelligence and clinical decision support. According to the method, multi-modal features are extracted by integrating clinical, image, metabolism, wearable equipment and other multi-source data, feature fusion is performed by using a multi-view factorization machine, risk prediction is performed in combination with a deep learning model, and reasoning and personalized intervention recommendation are performed by means of a diabetic foot ulcer knowledge graph. The method overcomes the defects that an existing method depends on a single data source and lacks multi-modal fusion and knowledge guidance, early-stage, accurate and interpretable risk prediction of the diabetic foot ulcer risk is achieved, and the method has important clinical application value.

Description

Technical Field

[0001] This invention relates to the field of medical information technology and artificial intelligence-assisted diagnosis, and in particular to a method and system for predicting the risk of diabetic foot ulcers based on multimodal information. Background Technology

[0002] Diabetic foot ulcers are one of the most serious and costly complications of diabetes, and a leading cause of non-traumatic lower limb amputations. Early and accurate identification and intervention of high-risk patients are crucial for preventing foot ulcers, reducing amputation rates, and improving patients' quality of life.

[0003] Currently, risk assessment for diabetic foot ulcers mainly relies on clinicians' experience and simple scoring methods, such as the classic Wagner classification and Texas classification. While these methods are simple, they are highly subjective and struggle to capture complex and subtle pathophysiological changes. In recent years, some studies have attempted to use machine learning models to predict risk based on structured data (such as blood glucose, age, and disease duration) from electronic health records, achieving some progress.

[0004] However, existing technical solutions have significant limitations: First, most rely on a single data source (such as only clinical data or only impact data), failing to fully utilize the complementarity between multimodal information (such as medical imaging, malignancy genomics, and continuous wearable device monitoring data); second, they lack effective multimodal information fusion mechanisms, making it difficult to extract stable and biologically meaningful fusion features from high-dimensional, heterogeneous data; finally, existing models are mostly "black boxes," lacking interpretability in their predictions and failing to integrate with rich prior medical knowledge (such as pathological mechanisms and correlations with treatment effectiveness), making it difficult to generate personalized intervention plans with clinical guidance. Due to the lack of mechanisms to effectively fuse heterogeneous multimodal data and the isolation between the model decision-making process and the medical knowledge graph, the predictive accuracy, generalization ability, and clinical interpretability of existing solutions have all reached bottlenecks.

[0005] Therefore, there is an urgent need in this field for an intelligent prediction method and system that can integrate multimodal data, achieve deep feature fusion, and combine medical knowledge for interpretable reasoning.

[0006] The information disclosed in this background section is intended only to enhance the understanding of the overall background of the invention and should not be construed as an admission or in any way implying that the information constitutes prior art known to those skilled in the art. Summary of the Invention

[0007] The purpose of this invention is to provide a method and system for predicting the risk of diabetic foot ulcers based on multimodal information. It aims to achieve high-precision, interpretable prediction of the risk of diabetic foot ulcers and personalized intervention recommendations by integrating multi-source heterogeneous data and combining deep learning and knowledge graph technology.

[0008] To achieve the above objectives, the present invention provides the following solution: A diabetic foot ulcer risk prediction system based on multimodal information includes: The data acquisition and feature extraction module is used to acquire multimodal data and perform preprocessing and feature extraction. The multimodal feature fusion module is used to reduce the dimensionality of features and perform deep fusion using a multi-view factorization machine model. The risk prediction and decision support module is used to predict risks and generate personalized reports based on knowledge graphs.

[0009] A method for predicting the risk of diabetic foot ulcers based on multimodal information includes: Step 1: Acquire multimodal data of diabetic patients, including clinical physiological data, medical imaging data, metabolomics data, and wearable device monitoring data; preprocess the multimodal data, and extract features of each modality using feature extraction methods to construct an initial multimodal feature set; Step 2: Based on the initial multimodal feature set, dimensionality reduction is performed using principal component analysis, and the dimensionality-reduced features are input into a pre-trained multi-view factorization machine model. Through cofactor decomposition, the latent space projections of multiple feature perspectives are learned to obtain latent factors and perspective factors. The latent factors are then used to map each perspective projection to a common low-dimensional latent space to obtain multimodal fusion features. Step 3: Input the multimodal fusion features into the pre-trained risk prediction model to output the probability of diabetic foot ulcer occurrence; based on the probability of occurrence and the pre-constructed knowledge graph of diabetic foot ulcer, perform reasoning to generate personalized risk warnings and intervention suggestions.

[0010] Optionally, in step one, acquiring multimodal data and extracting features includes: The clinical physiological data were structured to extract features such as blood glucose fluctuation, blood pressure variability, and nerve conduction. The medical impact data of the foot were segmented and feature extracted to obtain foot morphological features, skin integrity features, and vascular calcification features. Differential analysis was performed on metabolomics data to extract metabolite expression characteristics; Time-series analysis of wearable device data is performed to extract gait features, plantar pressure distribution features, and temperature change features.

[0011] Optionally, in step one, after constructing the initial multimodal feature set, the feature set is further filtered: Based on the initial multimodal feature set, the regression values ​​and significance of each feature are calculated using a logistic regression model or a Cox risk model, and a subset of statistical features is selected. Statistical feature subsets are mapped to predefined biological pathways or clinical phenotype pathways, and enrichment analysis is used to assess the correlation between pathways and foot ulcer risk, thus constructing pathway feature subsets. Using biological networks or clinical relationship networks, a random walk algorithm is employed to evaluate the importance of features in the network and to purchase a subset of network features. By combining regularization methods with tree models to perform secondary filtering of features, a final multimodal feature set is obtained for subsequent steps.

[0012] Optionally, in step two, the processing procedure of the multi-view factorization machine model includes: A non-linear mapping layer is introduced between each feature perspective and the latent space, and a single-layer neural network is used to map the features of each perspective into a non-linear representation. A multi-view factorization machine objective function is constructed, the model is trained by optimization algorithm, and an attention mechanism is introduced to adaptively weight and fuse the nonlinear representations of multiple views to obtain the multi-view fused representation. The multi-view fusion representation is used as an optimized latent factor to map multimodal features to the common low-dimensional latent space.

[0013] Optionally, in the three steps, the risk prediction model is a deep prediction model based on a multi-layer neural network, and its training and prediction process includes: Construct a deep prediction model and a domain discrimination model, and optimize the model parameters through adversarial training to obtain neighborhood-invariant feature representations; The multimodal fusion features of the patient to be predicted are input into the trained deep prediction model, which outputs the probability of diabetic foot ulcer occurrence.

[0014] Optionally, in step three, the reasoning process based on the knowledge graph includes: A knowledge graph of diabetic foot ulcers was constructed based on hospital guidelines and expert knowledge, with nodes including disease, symptoms, metabolic indicators, treatment methods, and complication entities; The multimodal fusion features are embedded into the corresponding nodes of the knowledge graph to construct a knowledge-enhanced heterogeneous graph; The heterogeneous graph is subjected to graph convolution operation using a heterogeneous graph neural network, and neighbor node information is aggregated through an attention mechanism to update the node feature representation; Based on the updated node feature representation, risk prediction calibration and treatment strategy search are performed to generate personalized reports and interpretable decision support information.

[0015] An electronic device includes a processor, a memory, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the method described above.

[0016] A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the above-described method.

[0017] Compared with the prior art, the present invention has the following beneficial effects: Multimodal information fusion: By integrating data from multiple sources such as clinical, imaging, metabolic, and wearable devices, it comprehensively portrays the patient's condition and overcomes the limitations of a single data source.

[0018] Deep feature learning and fusion: Employing a multi-view factorization machine model, it can effectively learn the complex nonlinear relationships and complementary information between multimodal data, generating more discriminative fused features.

[0019] High-precision and robust prediction: Risk prediction is performed using a deep learning model, and the generalization ability of the model is enhanced through adversarial training, thereby improving the accuracy and stability of the prediction.

[0020] Knowledge-guided and interpretable approach: By constructing a knowledge graph of diabetic foot ulcers and reasoning with the graph based on the prediction results, the decision-making process of the model becomes transparent, generating medically based and interpretable risk assessment reports and personalized intervention recommendations.

[0021] Highly clinically applicable: It can provide specific stratified management and treatment recommendations, directly serving clinical decision-making and helping to achieve precise prevention and control of diabetes. Attached Figure Description

[0022] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0023] Figure 1 This is a flowchart illustrating the method for predicting the risk of diabetic foot ulcers based on multimodal information provided in an embodiment of the present invention.

[0024] Figure 2 This is a schematic diagram illustrating feature fusion in the multi-view factor decomposition machine model provided in an embodiment of the present invention.

[0025] Figure 3 This is a schematic diagram illustrating reasoning and decision support based on knowledge graphs, provided as an embodiment of the present invention.

[0026] Figure 4 This is a structural block diagram of the diabetic foot ulcer risk prediction system based on multimodal information provided in an embodiment of the present invention. Detailed Implementation

[0027] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0028] The purpose of this invention is to provide an intelligent prediction method and system that can integrate multimodal data, achieve deep feature fusion, and combine medical knowledge for interpretable reasoning.

[0029] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0030] Example 1: As Figure 1 As shown in the figure, this embodiment provides a method for predicting the risk of diabetic foot ulcers based on multimodal information. The specific steps are as follows: S101: Multimodal Data Acquisition and Feature Extraction: Data Acquisition: Collect four types of data from diabetic patients.

[0031] 1. Clinical physiological data: obtained from the hospital information system, including glycated hemoglobin, fasting blood glucose, blood lipids, blood pressure, renal function indicators, nerve conduction test reports, etc.; 2. Medical imaging data: including foot X-rays, MRI, skin temperature maps, optical coherence tomography, etc., used to assess skeletal deformities, soft tissue infections, vascular status, and skin integrity.

[0032] 3. Metabolomics data: Detection of small molecule metabolites in blood or urine using mass spectrometry or nuclear magnetic resonance techniques.

[0033] 4. Wearable device monitoring data: from smart insoles, socks or wristbands, continuously monitoring foot pressure distribution, gait parameters, changes in skin temperature / humidity, etc.

[0034] Preprocessing: The data is standardized, denoised, imputed for missing values, and segmented (e.g., using the U-Net network to segment foot bones and soft tissues).

[0035] Feature extraction: Clinical features: Extract statistical features (such as standard deviation of blood glucose fluctuation) and composite indicators defined according to clinical guidelines.

[0036] Image features: Extract morphological features (such as arch index), texture features (such as gray-level co-occurrence matrix), and deep features extracted through pre-trained convolutional neural networks (such as ResNet).

[0037] Metabolic characteristics: Differential metabolites associated with foot ulcers were screened using differential analysis methods (such as PLS-DA) to constitute metabolic expression characteristics.

[0038] Wearable device characteristics: Perform time-series analysis to extract features such as peak pressure, pressure-time integral, gait symmetry, and temperature change trends.

[0039] All features are concatenated to form an initial multimodal feature set.

[0040] S102: Multimodal feature fusion: Dimensionality reduction: Principal component analysis is used to reduce the dimensionality of the initial multimodal feature set in high dimension, removing noise and redundancy while retaining the main variation information.

[0041] Multi-view factorization machine fusion: such as Figure 2 As shown, the dimensionality-reduced multi-omics features are input into a pre-trained multi-view factorization machine model.

[0042] 1. Nonlinear mapping: Set up a single-layer neural network for each modality (view) to map the original features to a nonlinear space, thereby enhancing the model's expressive power.

[0043] 2. Collaborative Decomposition: The model learns a shared latent space matrix (latent factors) and view-specific weight matrices (viewpoint factors) by optimizing the objective function. The objective function comprehensively considers reconstruction error and various regularization terms (such as Frobenius norm and L2 norm) to prevent overfitting.

[0044] 3. Attention Fusion: An attention mechanism is introduced to assign adaptive weights to nonlinear features of different modalities, and then weighted fusion is performed to obtain the final multi-view fusion representation, i.e. the optimized latent factor.

[0045] 4. Common Latent Space Projection: Using this latent factor, the features of all modalities are projected into a unified, low-dimensional common latent space to obtain multimodal fusion features that can comprehensively represent the patient's state.

[0046] S103: Risk Prediction and Personalized Decision Generation Risk prediction: Multimodal fusion features are input into a pre-trained deep prediction model. This model is trained using an adversarial training strategy. Build a deep prediction model (main network) to predict risk.

[0047] Build a domain discrimination model (adversarial network) to determine the data source (e.g., different hospitals).

[0048] Adversarial training guides the main network to learn domain-invariant feature representations, thereby improving the model's generalization ability under different data distributions.

[0049] Finally, the model outputs the results.

[0050] Knowledge graph reasoning and decision generation: such as Figure 3 As shown.

[0051] 1. Knowledge Graph Construction: Based on medical textbooks, clinical guidelines, and research literature, a knowledge graph for diabetic foot ulcers is constructed. Nodes include: diseases (e.g., diabetes, peripheral neuropathy), symptoms (e.g., numbness, tingling), examination indicators (e.g., HbA1c, vibration sensation), treatment methods (e.g., pressure-relieving footwear, debridement), and complications (e.g., infection, gangrene). Edges represent the relationships between them, such as "cause," "treatment," and "constitute risk factors."

[0052] 2. Graph Neural Network Reasoning: The patient's multimodal fusion features are embedded as attributes into the corresponding entity nodes in the knowledge graph (e.g., associating the patient's nerve conduction velocity value with the "peripheral neuropathy" node), forming a knowledge-enhanced heterogeneous graph. Heterogeneous graph neural networks (such as GAT) are used for message propagation and aggregation, updating node representations, thereby fusing the structured knowledge of the graph with the patient's individualized data.

[0053] 3. Multi-task learning and strategy search: Prognostic prediction: The updated key node features are input into a fully connected network to further calibrate the survival risk value or ulcer occurrence probability.

[0054] Treatment recommendation: The current patient status (integrating features and graph information) is input into a policy network based on deep reinforcement learning. The network searches in the treatment path space defined by the knowledge graph, simulates the long-term benefits of different treatment options, and finally recommends the optimal treatment strategy (such as "using customized insoles + weekly specialist follow-up").

[0055] Example 2: As Figure 4 As shown, this embodiment provides a system for implementing the above method, including: Data Acquisition and Feature Extraction Module 401: Responsible for acquiring raw data from various data source interfaces and performing preprocessing and feature extraction tasks.

[0056] Multimodal feature fusion module 402: integrates PCA dimensionality reduction algorithm and multi-view factorization machine model to complete the deep fusion of multimodal features.

[0057] Risk Prediction and Decision Support Module 403: It has a built-in deep prediction model, a knowledge graph of diabetic foot ulcers, a graph neural network inference engine, and a report generator, which is responsible for calculating risk probabilities and making interpretable decision recommendations.

[0058] User interface: Used by clinicians to input queries, view risk prediction results and detailed treatment advice reports.

[0059] Example 3: This example provides an electronic device, including a processor and a memory. The memory stores a computer program, and the processor implements the method described in Example 1 by calling and executing the computer program.

[0060] Validated using retrospective clinical data from 500 patients, this method improved the predictive AUC (area under the curve) by approximately 15% compared to a baseline model using only clinical data; and compared to a model without knowledge graph reasoning, the clinician adoption rate of the generated intervention recommendations increased by 30%. This demonstrates the significant advancements of this method in both predictive accuracy and clinical applicability.

[0061] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the systems disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the descriptions are relatively simple; relevant parts can be referred to the method section.

[0062] This document uses specific examples to illustrate the principles and implementation methods of the present invention. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of the present invention. Furthermore, those skilled in the art will recognize that, based on the ideas of the present invention, there will be changes in the specific implementation methods and application scope. Therefore, the content of this specification should not be construed as a limitation of the present invention.

Claims

1. A diabetic foot ulcer risk prediction system based on multimodal information, characterized in that, include: The data acquisition and feature extraction module is used to acquire multimodal data and perform preprocessing and feature extraction. The multimodal feature fusion module is used to reduce the dimensionality of features and perform deep fusion using a multi-view factorization machine model. The risk prediction and decision support module is used to predict risks and generate personalized reports based on knowledge graphs.

2. A method for predicting the risk of diabetic foot ulcers based on multimodal information, characterized in that, include: Step 1: Acquire multimodal data of diabetic patients, including clinical physiological data, medical imaging data, metabolomics data, and wearable device monitoring data; preprocess the multimodal data, and extract features of each modality using feature extraction methods to construct an initial multimodal feature set; Step 2: Based on the initial multimodal feature set, dimensionality reduction is performed using principal component analysis, and the dimensionality-reduced features are input into a pre-trained multi-view factorization machine model. Through co-factor decomposition, the latent space projections of multiple feature perspectives are learned to obtain latent factors and perspective factors. The latent factors are then used to map each perspective projection to a common low-dimensional latent space to obtain multimodal fusion features. Step 3: Input the multimodal fusion features into the pre-trained risk prediction model and output the probability of diabetic foot ulcer occurrence; Based on the occurrence probability and a pre-constructed knowledge graph of diabetic foot ulcers, inferences are made to generate personalized risk warnings and intervention suggestions.

3. The method for predicting the risk of diabetic foot ulcers based on multimodal information according to claim 2, characterized in that, Step one, which involves acquiring multimodal data and extracting features, includes: The clinical physiological data were structured to extract features such as blood glucose fluctuation, blood pressure variability, and nerve conduction. The medical impact data of the foot were segmented and feature extracted to obtain foot morphological features, skin integrity features, and vascular calcification features. Differential analysis was performed on metabolomics data to extract metabolite expression characteristics; Time-series analysis of wearable device data is performed to extract gait features, plantar pressure distribution features, and temperature change features.

4. The method for predicting the risk of diabetic foot ulcers based on multimodal information according to claim 2, characterized in that, In step one, after constructing the initial multimodal feature set, the process also includes filtering the features: Based on the initial multimodal feature set, the regression values ​​and significance of each feature are calculated using a logistic regression model or a Cox risk model, and a subset of statistical features is selected. Statistical feature subsets are mapped to predefined biological pathways or clinical phenotype pathways, and enrichment analysis is used to assess the correlation between pathways and foot ulcer risk, thus constructing pathway feature subsets. Using biological networks or clinical relationship networks, a random walk algorithm is employed to evaluate the importance of features in the network and to purchase a subset of network features. By combining regularization methods with tree models to perform secondary filtering of features, a final multimodal feature set is obtained for subsequent steps.

5. The method for predicting the risk of diabetic foot ulcers based on multimodal information according to claim 2, characterized in that, In step two, the processing procedure of the multi-view factorization machine model includes: A non-linear mapping layer is introduced between each feature perspective and the latent space, and a single-layer neural network is used to map the features of each perspective into a non-linear representation. A multi-view factorization machine objective function is constructed, the model is trained by optimization algorithm, and an attention mechanism is introduced to adaptively weight and fuse the nonlinear representations of multiple views to obtain the multi-view fused representation. The multi-view fusion representation is used as an optimized latent factor to map multimodal features to the common low-dimensional latent space.

6. The method for predicting the risk of diabetic foot ulcers based on multimodal information according to claim 2, characterized in that, In step three, the risk prediction model is a deep prediction model based on a multi-layer neural network, and its training and prediction process includes: Construct a deep prediction model and a domain discrimination model, and optimize the model parameters through adversarial training to obtain neighborhood-invariant feature representations; The multimodal fusion features of the patient to be predicted are input into the trained deep prediction model, which outputs the probability of diabetic foot ulcer occurrence.

7. The method for predicting the risk of diabetic foot ulcers based on multimodal information according to claim 2, characterized in that, In step three, the reasoning process based on the knowledge graph includes: A knowledge graph of diabetic foot ulcers was constructed based on hospital guidelines and expert knowledge, with nodes including disease, symptoms, metabolic indicators, treatment methods, and complication entities; The multimodal fusion features are embedded into the corresponding nodes of the knowledge graph to construct a knowledge-enhanced heterogeneous graph; The heterogeneous graph is subjected to graph convolution operation using a heterogeneous graph neural network, and neighbor node information is aggregated through an attention mechanism to update the node feature representation; Based on the updated node feature representation, risk prediction calibration and treatment strategy search are performed to generate personalized reports and interpretable decision support information.

8. An electronic device comprising a processor, a memory, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the method as described in any one of claims 2-7.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the method as described in any one of claims 2-7.

Images