An artificial intelligence-based cross-modal chronic disease prediction method

Abstract

The application discloses a kind of cross-modal chronic disease prediction methods based on artificial intelligence, belong to medical artificial intelligence field.The method first acquires four kinds of multi-source heterogeneous information of medical image, biochemical index, behavior data and text data;Second, construct the multi-modal encoder consisting of 3D-CNN, LSTM and BERT, respectively extract spatial, time sequence and semantic features;Subsequently, the cross-modal attention mechanism is used to fuse each modal feature, and the nonlinear law of chronic disease progression is captured by LSTM+attention network;Further introduce causal inference module, distinguish the causal effect and correlation of risk factors;Finally, according to the characteristics of patient population dynamically adjust model weight, realize the personalized risk prediction and progress curve estimation of chronic disease.The application solves the problem of ignoring data multi-modal, time sequence and causal confusion in the prior art, significantly improves the accuracy, explainability and model generalization ability of chronic disease prediction.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence, and more particularly to a cross-modal chronic disease prediction method based on artificial intelligence. Background Technology

[0002] With the increasing aging of the population, chronic diseases have become a major public health issue worldwide. Early and accurate risk prediction for chronic diseases is crucial for slowing disease progression and reducing the burden on healthcare. In recent years, artificial intelligence technology, especially deep learning, has made significant progress in its application in medical prediction models. Chronic diseases are those with long courses, slow progression, and require long-term management. They mainly include cardiovascular and cerebrovascular diseases, metabolic diseases, respiratory diseases, mental illnesses, tumors, chronic kidney disease, and musculoskeletal diseases. These diseases are usually related to genetic, lifestyle, and environmental factors, requiring comprehensive intervention through medication, lifestyle adjustments, and other means.

[0003] However, existing chronic disease prediction models still have many limitations:

[0004] First, data utilization is limited. Most models rely solely on structured data (such as biochemical indicators) from electronic health records, neglecting information-rich medical images (such as CT and MRI scans), patient daily behavior data (such as wearable device monitoring), and unstructured clinical text data (such as physician medical records and diagnostic reports). These multi-source, heterogeneous data have complex complementary relationships, making it difficult for single-modality data to comprehensively depict a patient's health status.

[0005] Secondly, there is a lack of time-series dynamic modeling. The development of chronic diseases is a dynamic and long-term process. Existing methods are mostly based on snapshot data at a single time point for prediction, failing to effectively utilize patients' historical time-series information to capture the nonlinear patterns and long-term dependencies in disease evolution.

[0006] Furthermore, there is the problem of causal confusion. Traditional data-driven predictive models are based on statistical correlation, making it difficult to distinguish between causal and correlational relationships between risk factors and diseases. For example, the model may find a certain indicator highly correlated with a disease, but that indicator may be a consequence of the disease (reverse causation), not a cause. This can lead to biased predictions and potentially erroneous medical insights, misleading clinical decision-making.

[0007] Finally, the model's generalization ability is insufficient. When the trained model is applied to different hospitals and different population groups, its performance often drops significantly due to differences in data distribution, lacking the ability to adaptively adjust.

[0008] Therefore, there is an urgent need in this field for a chronic disease prediction method that can integrate multimodal data, deeply analyze temporal dynamics, identify causal relationships, and has good generalization ability. Summary of the Invention

[0009] To address the aforementioned issues, this invention provides an artificial intelligence-based cross-modal chronic disease prediction method. This method outperforms existing technologies in terms of AUC, C-index, and interpretability metrics on datasets of diabetes, chronic kidney disease, and cardiovascular chronic diseases.

[0010] To achieve the above objectives, the technical solution adopted by this invention is: to provide a cross-modal chronic disease prediction method based on artificial intelligence, comprising:

[0011] Collect multi-source heterogeneous data on chronic diseases, including medical imaging data, biochemical indicator data, behavioral data, and text data. The medical imaging data includes CT, MRI, ultrasound, and digital pathology images; the biochemical indicator data includes complete blood count, comprehensive biochemical tests, urinalysis, and gene sequencing results; the behavioral data includes steps, heart rate, sleep, diet, and exercise logs recorded by wearable devices; and the text data includes medical records, follow-up records, and patient self-reports.

[0012] The multi-source heterogeneous data is preprocessed, and the preprocessing steps include data cleaning, standardization and alignment to obtain a standardized, privacy-de-identified and time-aligned multimodal input sequence.

[0013] A multimodal encoder is constructed. The spatial features of the medical image data are extracted using 3D-CNN, the time dependencies of the biochemical index data and behavioral data are captured using a two-layer LSTM, and the semantic features of the text data are extracted using a BERT pre-trained model to obtain high-dimensional feature representations of each modality.

[0014] The high-dimensional features of each modality are fused by a cross-modal attention mechanism. Image features are used as query vectors, and temporal features and text features are used as key vectors and value vectors, respectively. The cross-modal attention weight matrix is ​​calculated, the inter-modal dependencies are established, and the fused joint representation vector is obtained.

[0015] Based on gated recurrent unit networks and temporal attention mechanisms, the joint representation vector is sequence-modeled to capture the long-term dependencies and nonlinear patterns of chronic diseases, and output preliminary chronic disease risk probability prediction results.

[0016] A causal inference mechanism is introduced to calculate the causal effect between each risk factor and chronic disease outcome, calculate the causal effect value, generate causal enhancement features, and perform weighted fusion of the causal enhancement features and joint representation vector to output the chronic disease risk probability and the chronic disease progression curve within a future preset time window.

[0017] Preferably, the multimodal encoder includes:

[0018] The image modality coding submodule is used to extract spatial and deep semantic features of medical image data using a pre-trained 3D convolutional neural network;

[0019] The temporal modality coding submodule is used to extract dynamic temporal features and forward and backward dependencies from biochemical index data and behavioral data using a bidirectional long short-term memory network.

[0020] The text modality encoding submodule is used to extract semantic features and contextual information of medical terms from clinical texts using a pre-trained medical BERT model based on the Transformer architecture.

[0021] Preferably, the cross-modal attention mechanism is a multi-head attention mechanism, which calculates the attention weight of any source modality to the target modality and enhances the expressive power of the fused features by concatenating the outputs of multiple attention heads.

[0022] More preferably, in the multi-head attention mechanism, the number of heads is set to 8, and the dimension of each head is 64, which is used to improve the intermodal interaction representation capability.

[0023] Preferably, the causal inference mechanism is constructed based on a dual machine learning model, specifically including:

[0024] In the first stage, machine learning models are used to fit the treatment model for risk factors and the conditional expectation model for outcomes.

[0025] In the second stage, residuals are constructed based on the prediction results of the first stage, and the conditional average causal effect of risk factors is estimated using the final model.

[0026] More preferably, the causal inference mechanism is further used to perform counterfactual analysis, by simulating changes in the values ​​of specific risk factors, to calculate the change in the corresponding disease risk, in order to generate an interpretable causal contribution report.

[0027] Preferably, the chronic disease risk probability and the chronic disease progression curve within a future preset time window are output through the SHAP value interpretation model to graphically display the marginal contribution of each risk factor.

[0028] The initial training phase of the chronic disease risk probability prediction model uses a joint loss function:

[0029] ;

[0030] in, Used to regress disease progression curves Used for classifying chronic disease risks Used to constrain the consistency of causal effects, λ1, λ2, and λ3 are adjustable hyperparameters.

[0031] Preferably, the chronic disease risk probability and the chronic disease progression curve within a future preset time window are output through the SHAP value interpretation model to graphically display the marginal contribution of each risk factor.

[0032] The initial training phase of the chronic disease risk probability prediction model uses a joint loss function:

[0033] ;

[0034] in, Used to regress disease progression curves Used for classifying chronic disease risks Used to constrain the consistency of causal effects, λ1, λ2, and λ3 are adjustable hyperparameters.

[0035] The beneficial effects of this invention are as follows:

[0036] Improving prediction accuracy and comprehensiveness: By constructing a multimodal encoder, complementary information from multiple sources such as images, time series, and text is fully utilized to form a more comprehensive and in-depth representation of the patient's health status, fundamentally overcoming the limitations of a single data source and significantly improving the accuracy and robustness of the prediction model.

[0037] Enhanced temporal dynamic capture capability: By using GRU network combined with temporal attention mechanism, it can effectively process long sequence data, capture long-term dependencies and key time nodes in the development of chronic diseases, realize accurate modeling of the non-linear progression of diseases, and make the prediction more consistent with the actual clinical process.

[0038] Overcoming the bottleneck of causal confusion: An innovative causal inference module is introduced, based on dual machine learning and counterfactual analysis, to quantify the causal effects of risk factors and effectively distinguish between causal and correlational relationships. This makes the model's predictions not only more reliable but also generates interpretable causal contribution reports, providing precise and reliable targets for clinical intervention and possessing extremely high clinical guidance value.

[0039] Enhancing Model Generalization and Adaptability: A meta-learning framework enables the model to quickly adapt to the data distribution of new patient groups and dynamically adjust model parameters. This significantly improves the model's generalization ability and practical value across different medical institutions and populations, solving the problem of "difficulty in deploying" machine learning models.

[0040] Optimize clinical decision support: The final output is not just a risk score, but also includes the contribution of each modality feature and the causal analysis results of risk factors, presented in a visual form, providing doctors with comprehensive, transparent and interpretable decision support, and helping to achieve truly personalized medicine. Attached Figure Description

[0041] Figure 1 This is a flowchart of a cross-modal chronic disease prediction method based on artificial intelligence according to the present invention.

[0042] Figure 2 This is a diagram verifying the effect of a preferred embodiment of the present invention. Detailed Implementation

[0043] Please see Figure 1 As shown, this invention provides a cross-modal chronic disease prediction method based on artificial intelligence, comprising:

[0044] S1. Collect multi-source heterogeneous data on chronic diseases, including medical imaging data, biochemical indicator data, behavioral data, and text data. Among them, medical imaging data includes CT, MRI, ultrasound, and digital pathology images; biochemical indicator data includes complete blood count, comprehensive biochemical tests, urine tests, and gene sequencing results; behavioral data includes steps, heart rate, sleep, diet, and exercise logs recorded by wearable devices; and text data includes medical record texts, follow-up records, and patient self-report texts.

[0045] First, retrospective data (multi-source heterogeneous data) of 10,000 patients was extracted from the hospital information system. Each patient had multiple medical records from five consecutive years. The data included:

[0046] Medical imaging data: DICOM file of abdominal CT scan, used to detect visceral fat area.

[0047] Biochemical indicators: Structured data, including fasting plasma glucose (FPG), 2-hour postprandial plasma glucose (2hPG), glycated hemoglobin (HbA1c), total cholesterol (TC), triglycerides (TG), high-density lipoprotein (HDL), low-density lipoprotein (LDL), etc.

[0048] Behavioral data: Wearable device data voluntarily provided by patients, including average daily steps, resting heart rate, and sleep duration, forming a daily time series.

[0049] Clinical text data: Unstructured clinical medical records, including chief complaint, present illness, family history (e.g., "His mother has diabetes"), etc.

[0050] S2. Preprocess the multi-source heterogeneous data. The preprocessing steps include data cleaning, standardization, and alignment to obtain a standardized, privacy-de-identified, and time-aligned multimodal input sequence.

[0051] Preprocessing:

[0052] Image data: CT scan images were uniformly resampled to an isotropic resolution of 1mm × 1mm × 1mm, and then... The window width of HU is normalized to grayscale. The abdominal cavity is automatically segmented using a pre-trained U-Net model, and the visceral fat area is calculated.

[0053] Biochemical and behavioral data: Missing values ​​were filled using the chain equation multivariate imputation method (MICE). All continuous features were Z-score standardized to eliminate the influence of dimensions.

[0054] Text data: De-identification was performed using natural language processing tools to remove all personally identifiable information. Medical entity recognition and standardization were performed using a medical dictionary and BERT model (e.g., standardizing "high blood sugar" to "hyperglycemia").

[0055] Data alignment: Using "Patient ID" and "Examination Date" as keys, data from all modalities are aligned to the same point in time to form a cross-modal time series dataset.

[0056] S3. Construct a multimodal encoder, use 3D-CNN to extract the spatial features of the medical image data from the preprocessed multi-source heterogeneous data, use a two-layer LSTM to process the biochemical index data and behavioral data to capture the time dependency, and use a BERT pre-trained model to extract the semantic features of the text data to obtain high-dimensional feature representations of each modality;

[0057] Multimodal encoders are used to process preprocessed multi-source heterogeneous data, specifically including:

[0058] Image modality encoding: Features of abdominal CT images are extracted using a 3D ResNet-18 model pre-trained on ImageNet. The input is the cropped abdominal cavity volume (64×64×64 voxels), and the output is a 512-dimensional feature vector.

[0059] Temporal modality coding: Standardized biochemical indicators and behavioral data (15 features in total) are treated as a multivariate time series. A two-layer bidirectional LSTM network is used for processing, with 128 hidden layer units. The output of the last time step is taken as the temporal feature vector (256 dimensions) for that visit.

[0060] Text modality encoding: The bert-base-uncased version of the model was used, and domain-adaptive pre-training was performed on the MIMIC-III clinical database. The input was the concatenated clinical text (e.g., "Chief complaint: polydipsia and polyuria. Family history: maternal diabetes. Diagnosis: obesity"), and the output layer state corresponding to the [CLS] tag was taken as the semantic representation vector (768 dimensions) of the entire text.

[0061] S4. The high-dimensional features of each modality are fused through a cross-modal attention mechanism. The image features are used as the query vector, and the temporal features and text features are used as the key vector and value vector, respectively. The cross-modal attention weight matrix is ​​calculated, the inter-modal dependency relationship is established, and the fused joint representation vector is obtained.

[0062] The image feature vector (512 dimensions), time-series feature vector (256 dimensions), and text feature vector (768 dimensions) extracted from each visit of patients with chronic diseases are concatenated to obtain a 1536-dimensional pre-fusion feature.

[0063] The feature is then fed into a cross-modal multi-head attention layer (8 attention heads). This layer calculates the attention weights of text features on image features (e.g., mentioning "obesity" in the text will make the model pay more attention to fat-related regions in CT images), and vice versa. After weighted summation, a 1024-dimensional unified feature representation that reflects intermodal dependencies is output.

[0064] S5. Based on gated recurrent unit network and temporal attention mechanism, sequence modeling is performed on the joint representation vector to capture the long-term dependence and nonlinearity of chronic diseases and output preliminary chronic disease risk probability prediction results.

[0065] The consistent feature representations obtained from multiple patient visits are arranged chronologically and input into a two-layer GRU network (256 hidden units) to capture the long-term evolution trend of the patient's health status. A temporal attention layer is then connected after the GRU network to emphasize the importance of certain key medical events (such as the first abnormality in blood glucose) for prediction.

[0066] Finally, the output of the attention layer is connected to a fully connected layer, and the Sigmoid activation function is used to output a value between 0 and 1 as the preliminary risk probability P0 of the patient developing T2D in the next five years.

[0067] The fused features (512 dimensions / time) of all patient visits are input sequentially into a two-layer gated recurrent unit (GRU) network with 256 hidden units. GRU networks are good at capturing long-term temporal dependencies.

[0068] After the hidden state of the last time step of GRU, we added an additional temporal attention layer, which can assign different importance weights to all visits in history, thereby focusing on critical periods of rapid changes in the condition.

[0069] Finally, the weighted features are input into a fully connected classifier (layer structure: 512 -> 128 -> 64 -> 1), and the Sigmoid activation function is used to output a value between 0 and 1, which is the patient's initial risk probability P0.

[0070] S6. Introduce a causal inference mechanism to calculate the causal effect between each risk factor and the chronic disease outcome, calculate the causal effect value, generate causal enhancement features, and perform weighted fusion of the causal enhancement features and the joint representation vector to output the chronic disease risk probability and the chronic disease progression curve within the future preset time window.

[0071] The specific principles of the causal inference mechanism are as follows:

[0072] Causal inference using dual machine learning:

[0073] Treatment variable (T): Focus on the causal effect of a particular risk factor (e.g., low-density lipoprotein LDL) on the outcome.

[0074] Outcome variable (Y): Whether or not one has T2D.

[0075] Confounding variable (X): all other characteristics (age, gender, other indicators, etc.).

[0076] Step 1: Fit the model of the treatment variable T predicted by the confounding variable X and the model of the outcome variable Y predicted by the confounding variable X using the random forest regression model.

[0077] Step 2: Calculate the residuals of the treatment variable and the outcome variable:

[0078] ;

[0079] Step 3: Fit using linear regression The obtained θ is the conditional average causal effect (CATE) of LDL on T2D.

[0080] This process is repeated for multiple key risk factors to obtain their causal effect value vector C. These values ​​are then used to refine the initial probabilities.

[0081] ;

[0082] Where k is the learned scaling factor, and the final result is the base probability value after causal purification. .

[0083] We then employed a model-agnostic meta-learning (MAML) framework to enable the model to dynamically adjust. During the training phase, we divided the data into multiple tasks based on hospitals from different sources (simulating different population distributions).

[0084] To enable the model to quickly adapt to the data distribution of different hospitals (out-of-distribution generalization), we use the Model-Agnostic Meta-Learning (MAML) framework for training.

[0085] Meta-training is used to treat data from different hospitals as different "tasks". During training, "support sets" (a small number of samples) and "query sets" are simulated.

[0086] Meta-test (dynamic adjustment): When the model is deployed to a new hospital, anonymized data from the hospital's top 100 patients is used as a "support set" to update the last layer of the prediction network (step S104) with five gradient descent iterations. This process takes only a few seconds, allowing the model weights φ to adaptively adjust to the distribution φ' of the new hospital.

[0087] When predicting patient outcomes for a new hospital, only a few steps (e.g., 5 steps) of gradient descent fine-tuning are needed on the final layer (classifier) ​​of the prediction model using a small amount of historical data from that hospital. This allows the model to quickly adapt to the data distribution of the new patient population. Ultimately, the system outputs a personalized final T2D risk probability value for that patient group. The final result is a report in JSON format.

[0088] When the probability of chronic disease risk is greater than or equal to the preset threshold of 0.3, the system marks the patient as high-risk and triggers a clinical intervention reminder; when the probability of chronic disease risk is greater than or equal to 0.7, it marks the patient as extremely high-risk and simultaneously pushes a SHAP interpretation report to highlight key causal risk factors.

[0089] In this embodiment, extremely high risk ( When the level is ≥0.7, an OGTT request form will be automatically generated, and the hospital nutritionist will be notified for an emergency consultation; for high risk (0.3≤ <0.7), the report recommends that patients be followed up by telephone for 3 months and that personalized exercise prescriptions be sent to regulate bodily mechanisms; at low risk ( <0.3), only annual physical examination reminders will be sent.

[0090] This includes automatically generated gradient-weighted class activation map (Grad-CAM) heatmaps, which are overlaid on the original CT images to highlight image regions (such as the pancreas) that the model focuses on when making decisions. It also generates feature contribution bar charts.

[0091] Example of effectiveness verification Figure 2As shown, the results demonstrate that this invention, through multimodal fusion, causal inference, and meta-learning dynamic adjustment, not only significantly improves traditional indicators such as AUC, but more importantly, its high sensitivity (ability to detect true positives) and high specificity (ability to exclude true negatives) make it more suitable for clinical screening scenarios, while its excellent interpretability provides direct decision support for physicians. This invention provides predictive services to clinicians' workstations. After uploading anonymized patient data to the workstation, physicians can obtain a detailed risk assessment report within seconds to minutes.

[0092] The above embodiments are merely descriptions of preferred embodiments of the present invention and are not intended to limit the scope of the present invention. Various modifications and improvements made by those skilled in the art to the technical solutions of the present invention without departing from the spirit of the present invention should fall within the protection scope defined by the claims of the present invention.

Claims

1. A cross-modal chronic disease prediction method based on artificial intelligence, characterized in that, include: Collect multi-source heterogeneous data on chronic diseases, including medical imaging data, biochemical indicator data, behavioral data, and text data. The medical imaging data includes CT, MRI, ultrasound, and digital pathology images; the biochemical indicator data includes complete blood count, comprehensive biochemical tests, urinalysis, and gene sequencing results; the behavioral data includes steps, heart rate, sleep, diet, and exercise logs recorded by wearable devices; and the text data includes medical records, follow-up records, and patient self-reports. The multi-source heterogeneous data is preprocessed, and the preprocessing steps include data cleaning, standardization and alignment to obtain a standardized, privacy-de-identified and time-aligned multimodal input sequence. A multimodal encoder is constructed. The spatial features of the medical image data are extracted using 3D-CNN, the time dependencies of the biochemical index data and behavioral data are captured using a two-layer LSTM, and the semantic features of the text data are extracted using a BERT pre-trained model to obtain high-dimensional feature representations of each modality. The high-dimensional features of each modality are fused by a cross-modal attention mechanism. Image features are used as query vectors, and temporal features and text features are used as key vectors and value vectors, respectively. The cross-modal attention weight matrix is ​​calculated, the inter-modal dependencies are established, and the fused joint representation vector is obtained. Based on gated recurrent unit networks and temporal attention mechanisms, the joint representation vector is sequence-modeled to capture the long-term dependencies and nonlinear patterns of chronic diseases, and output preliminary chronic disease risk probability prediction results. A causal inference mechanism is introduced to calculate the causal effect between each risk factor and chronic disease outcome, calculate the causal effect value, generate causal enhancement features, and perform weighted fusion of the causal enhancement features and joint representation vector to output the chronic disease risk probability and the chronic disease progression curve within a future preset time window. The causal inference mechanism is built on a dual machine learning model, specifically including: In the first stage, machine learning models are used to fit the treatment model for risk factors and the conditional expectation model for outcomes. In the second stage, the residuals are constructed based on the prediction results of the first stage, and the conditional average causal effect of the risk factors is estimated using the final model. The causal inference mechanism is further used to perform counterfactual analysis, which calculates the change in the risk of the corresponding disease by simulating changes in the value of specific risk factors, in order to generate an interpretable causal contribution report. The probability of chronic disease risk and the progression curve of chronic disease within a future preset time window are output by the SHAP value interpretation model, and the marginal contribution of each risk factor is displayed in a graphical way. The initial training phase of the chronic disease risk probability prediction model uses a joint loss function: ； in, Used to regress disease progression curves Used for classifying chronic disease risks Used to constrain the consistency of causal effects, λ1, λ2, and λ3 are adjustable hyperparameters.

2. The cross-modal chronic disease prediction method based on artificial intelligence according to claim 1, characterized in that, The multimodal encoder includes: The image modality coding submodule is used to extract spatial and deep semantic features of medical image data using a pre-trained 3D convolutional neural network; The temporal modality coding submodule is used to extract dynamic temporal features and forward and backward dependencies from biochemical index data and behavioral data using a bidirectional long short-term memory network. The text modality encoding submodule is used to extract semantic features and contextual information of medical terms from clinical texts using a pre-trained medical BERT model based on the Transformer architecture.

3. The cross-modal chronic disease prediction method based on artificial intelligence according to claim 1, characterized in that, The cross-modal attention mechanism is a multi-head attention mechanism, which calculates the attention weight of any source modality to the target modality and enhances the expressive power of the fused features by concatenating the outputs of multiple attention heads.

4. The cross-modal chronic disease prediction method based on artificial intelligence according to claim 3, characterized in that, In the multi-head attention mechanism, the number of heads is set to 8, and the dimension of each head is 64, which is used to improve the intermodal interaction representation capability.

5. The cross-modal chronic disease prediction method based on artificial intelligence according to claim 1, characterized in that, The steps for obtaining the probability of chronic disease risk include: Obtain the final joint representation vector and causal enhancement features; The final joint representation vector is input into the trained Softmax classifier head, and the probability of chronic disease risk P(disease)∈[0,1] at the current time t0 is output. The final joint representation vector is input into a two-layer LSTM decoder in parallel, and the risk-time curve S(t) within the future time window T is output, where T is a configurable parameter with a value range of 1-5 years; When the probability of chronic disease risk is greater than or equal to the preset threshold of 0.3, the system marks the patient as high-risk and triggers a clinical intervention reminder; when the probability of chronic disease risk is greater than or equal to 0.7, it marks the patient as extremely high-risk and simultaneously pushes a SHAP interpretation report to highlight key causal risk factors. The probability of chronic disease risk and the risk-time curve are written into the electronic medical record timeline in real time, allowing doctors to view and review them interactively on a visual interface.

Images