Multi-label chronic disease risk prediction device based on multi-mode and graph neural network

Abstract

The invention discloses a multi-label chronic disease risk prediction device based on multi-modality and a graph neural network, and belongs to the technical field of intelligent medical treatment, and the device comprises a data processing unit which is used for obtaining electronic case data and carrying out multi-modality data screening, cleaning and preprocessing, wherein the multi-modal data comprises numerical value type inspection result data and text type inspection result data; the model construction unit is used for constructing a multi-label chronic disease risk prediction model comprising a multi-modal feature learning module, a multi-modal fusion module, a multi-disease correlation extraction module and a prediction module, and the application prediction unit is used for performing multi-label chronic disease risk prediction based on the constructed multi-label chronic disease risk prediction model. In this way, more effective features are mined from multi-source heterogeneous multi-modal data, and the correlation among chronic diseases is considered, so that on one hand, the risk of suffering from various chronic diseases is predicted more comprehensively, accurately and highly interpretably;

Description

Technical Field

[0001] The present invention belongs to the field of smart medical technology, and specifically relates to a multi-label chronic disease risk prediction device based on multimodality and graph neural network. Background Art

[0002] As people pay more attention to and prevent chronic diseases, chronic disease management is gradually shifting from treatment-centered to health-centered, moving the site of health intervention forward. Chronic diseases can be effectively controlled through active prevention and early intervention. Prevention is better than cure, but if the prediction results are inaccurate, it may have an adverse effect on the patient's mental state. Therefore, improving the accuracy of the prediction model is crucial for disease prediction.

[0003] For grassroots people, limited resources are needed to manage their condition and timely intervene in the disease to avoid the risk of serious illness through early warning of chronic diseases. Therefore, it is extremely necessary to establish an efficient and low-cost chronic disease (referred to as chronic disease) risk prediction model.

[0004] Since chronic diseases promote each other, it is common for multiple diseases to co-occur. Studies have shown that diabetes is an independent risk factor for recurrent ischemic stroke. ‎ . Hypertension is associated with the recurrence of small vessel subtypes of ischemic stroke. When the elderly suffer from two or more chronic diseases, the clinical symptoms are complex and the risk prognosis needs to be considered comprehensively. Therefore, multi-label chronic disease prediction models and consideration of the correlation between different chronic diseases are of great significance. Although many scholars have applied machine learning technology to the predictive diagnosis of a single disease, there is not much related work on the prediction of multiple diseases, and no research has considered the mutual influence between different diseases.

[0005] With the rapid development of computer technology and its widespread application in the medical field, a large amount of data resources are stored in various hospital information systems. How to make good use of these medical data to provide data support for doctors' decision-making is particularly important. Electronic medical records contain comprehensive information about the patient's health status. They are heterogeneous multimodal data, including structured data (such as demographics, vital signs, and test results) and unstructured data (such as clinical diagnosis and examination reports). Nowadays, the use of multimodal medical data for research and clinical practice has become a major trend in the medical field, bringing far-reaching impacts on medical diagnosis and treatment. However, how to effectively extract features from highly heterogeneous modalities and capture the complex interactions between modalities is still a considerable challenge. Summary of the invention

[0006] In view of the above, the object of the present invention is to provide a multi-label chronic disease risk prediction device based on multimodal and graph neural networks, which provides a more accurate, efficient and low-cost solution for chronic disease prevention.

[0007] To achieve the above-mentioned purpose of the invention, the embodiment provides a multi-label chronic disease risk prediction device based on multimodal and graph neural network, comprising: A data processing unit, which is used to obtain electronic medical record data and perform multimodal data screening, cleaning and preprocessing, wherein the multimodal data includes numerical inspection result data and text inspection result data; A model construction unit, which is used to construct a multi-label chronic disease risk prediction model including a multimodal feature learning module, a multimodal fusion module, a multi-disease correlation extraction module and a prediction module, wherein the numerical test result data and the textual diagnosis result data are respectively subjected to the multimodal feature learning module to learn the test features and the multi-class chronic disease risk features, the test features and the multi-class chronic disease risk features are fused through the multimodal fusion module through the cross-modal attention mechanism to obtain the multimodal fusion features of each class of chronic diseases, the chronic disease graph constructed based on the co-occurrence correlation of multiple classes of chronic diseases is subjected to the multi-disease correlation extraction module to extract the multi-disease correlation features of each class of chronic diseases, the multimodal fusion features and the multi-disease correlation features of each class of chronic diseases are fused through the prediction module to predict the risk probability of each class of chronic diseases; A prediction unit is applied, which is used to perform multi-label chronic disease risk prediction based on the constructed multi-label chronic disease risk prediction model.

[0008] Preferably, the numerical test result data includes continuous features and categorical features. For the continuous features, the categorical features are converted into categorical features by setting a threshold, wherein the categorical features are distinguished and represented by numerical values; Demographic information is also added to the numerical test result data to supplement the classification features, and the supplemented classification features are also input into the multimodal feature learning module to learn the test features.

[0009] Preferably, in the multimodal feature learning module, the Transformer model is used as an encoder, the numerical test result data is represented by a time series sequence and then input into the Transformer model, and the Transformer model is used to extract the feature correlation between the time series of the numerical test result data to obtain the test features.

[0010] Preferably, in the multimodal feature learning module, a denoising medical text encoder is designed based on a large language model, and a series of prompt templates are specifically designed, and the denoising medical text encoder is constructed by interactive learning based on the prompt templates using the large language model; The denoising medical text encoder is used to predict the interpretability of the risk of multiple chronic diseases based on the prompt template and the corresponding text-based examination result data. The predictive results of the interpretability of the risk of disease are scored, and the higher the score, the higher the risk of disease. The predictive results of the interpretability of the risk of disease are used as the risk characteristics of multiple chronic diseases. The prompt template includes task commands, the output format of the disease risk interpretable prediction results and reasons.

[0011] Preferably, in the multimodal fusion module, the test features and the risk features of multiple types of chronic diseases are fused through a cross-modal attention mechanism to obtain multimodal fusion features of each type of chronic disease, including: The test features and multiple chronic disease risk characteristics The query vector is calculated by linear projection of the average value of ,in, Represents the query vector weight parameter; Calculate the key vector corresponding to each type of chronic disease , ,in c Represents the chronic disease category index, represents the feature dimension, represents the feature space, Indicates c The key vector weight parameter corresponding to the chronic disease class, Represents based on test features and multiple chronic disease risk characteristics The constructed concatenated features are taken as value vectors, and the superscript T indicates transposition; For each chronic disease type, based on the query vector and key vector Generating Attention Weights ,in, Represents the key vector Dimensions; Calculate multimodal fusion features for each type of chronic disease based on attention weights ,in, A vector representing the value weight parameters.

[0012] Preferably, a chronic disease graph is constructed based on the co-occurrence correlation of multiple types of chronic diseases, including: The word embedding representation of each type of chronic disease is used as the node of the chronic disease graph, and the co-occurrence count of chronic diseases in the sample set is used to measure the chronic disease related dependencies. Based on this, the edges between the nodes are constructed to obtain the adjacency matrix of the chronic disease graph. Specifically, the chronic disease related dependencies are modeled in the form of conditional probability. , When chronic disease i Chronic disease when presentj The probability of occurrence, and Not equal to , the adjacency matrix is ​​asymmetric, and then the threshold is set Conditional probability Screen to get the value of each element in the adjacency matrix : .

[0013] Preferably, in the multi-disease correlation extraction module, the multi-disease correlation features of each type of chronic disease are extracted based on the chronic disease graph constructed based on the co-occurrence correlation of multiple types of chronic diseases, including: The node features and adjacency matrix of the chronic disease graph are input into the graph convolutional network or graph attention network to extract multi-disease correlation features.

[0014] Preferably, in the prediction module, the multimodal fusion features and multi-disease correlation features of each type of chronic disease are fused to predict the risk probability of each type of chronic disease, including: For each chronic disease , the multi-disease correlation features obtained in the multi-disease correlation extraction module And the multimodal fusion features obtained in the multimodal fusion module Inner product multiplication to obtain fusion features , and predict the risk of disease ,in Represents the sigmoid function.

[0015] Preferably, the multi-label chronic disease risk prediction model is trained to learn the optimal parameters before being applied. During training, a multi-label-based cross entropy loss function is used. And backpropagation to update the parameters: ; in, Represents the chronic disease category index, represents the total number of chronic disease categories, represents the predicted disease risk probability output by the model, represents the true classification label of chronic diseases, Represents the sigmoid function.

[0016] To achieve the above-mentioned purpose of the invention, an embodiment further provides a computing device, including a memory and one or more processors, wherein the memory stores executable code, and when the one or more processors execute the executable code, the steps of using the above-mentioned multi-label chronic disease risk prediction device to perform multi-label chronic disease risk prediction are implemented: Using a data processing unit to obtain electronic medical record data and perform multimodal data screening, cleaning and preprocessing, wherein the multimodal data includes numerical test result data and text test result data; A multi-label chronic disease risk prediction model including a multimodal feature learning module, a multimodal fusion module, a multi-disease correlation extraction module and a prediction module is constructed using a model construction unit, wherein the numerical test result data and the textual diagnosis result data are respectively subjected to the multimodal feature learning module to learn the test features and the multi-class chronic disease risk features, and the test features and the multi-class chronic disease risk features are fused through the multimodal fusion module through a cross-modal attention mechanism to obtain the multimodal fusion features of each class of chronic disease, and the chronic disease graph constructed based on the co-occurrence correlation of multiple classes of chronic diseases is subjected to the multi-disease correlation extraction module to extract the multi-disease correlation features of each class of chronic disease, and the multimodal fusion features and multi-disease correlation features of each class of chronic disease are fused through the prediction module to predict the risk probability of each class of chronic disease; Multi-label chronic disease risk prediction is performed using the application prediction unit based on the constructed multi-label chronic disease risk prediction model.

[0017] Compared with the prior art, the present invention has the following beneficial effects: This invention uses advanced data analysis technology, large language models, attention mechanisms and graph neural networks to design a new structure that combines multimodality and graph neural networks to mine more effective features from multi-source heterogeneous multimodal data, and considers the correlation between chronic diseases. On the one hand, it can more comprehensively, accurately and highly explainably predict the risk of suffering from multiple chronic diseases, and better manage, prevent and intervene in diseases; on the other hand, it makes chronic disease screening more universal, simple and low-cost. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0019] Figure 1 is a schematic structural diagram of a multi-label chronic disease risk prediction device based on multimodality and graph neural network provided in an embodiment; Figure 2 is a schematic diagram of the structure of a multi-label chronic disease risk prediction model provided in an embodiment; Figure 3 is a schematic diagram of a chronic disease map provided in an embodiment. DETAILED DESCRIPTION

[0020] To make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific implementation methods described herein are only used to explain the present invention and do not limit the scope of protection of the present invention.

[0021] The inventive concept of the present invention is to provide a multi-label chronic disease risk prediction device based on multimodality and graph neural network, which makes technical innovations in the following aspects: 1. There is a lot of noise in the text of electronic medical records, which makes it difficult to extract effective information. This paper uses the excellent text evaluation ability of the large language model to propose a denoising medical text encoder, which can not only skillfully capture salient information from a large amount of medical text, but also absorb external knowledge from LLM to enrich the representation; 2. The importance of each modality (time series, text) varies from patient to patient and prediction target, and incorrect fusion of multiple modalities will lead to inconsistent predictions and poor model performance. This invention solves the problem of modality inconsistency through a cross-modal attention mechanism, generating appropriate modality fusion weights and multimodal fusion features for different patients and different diseases; 3. The co-occurrence of multiple diseases is a very important feature of chronic diseases. The pathogenesis and progression of different chronic diseases are often intertwined, but the existing chronic disease risk prediction methods lack the use of this important feature. The present invention considers the mutual influence between five chronic diseases (diabetes, hypertension, chronic kidney disease, cardiovascular disease, cerebrovascular disease), learns the correlation features of multiple chronic diseases, and then fuses them with multimodal fusion features to obtain the final fusion features, based on which the risk of five chronic diseases is generated.

[0022] Based on the above-mentioned inventive concept, an embodiment provides a multi-label chronic disease risk prediction device 10 based on multimodality and graph neural network, including a data processing unit 11, a model building unit 12, and an application prediction unit 13. Based on these three units, the accuracy and interpretability of multi-label chronic disease risk prediction are improved.

[0023] In the embodiment, the data processing unit 11 is used to obtain electronic medical record data and perform multimodal data screening, cleaning and preprocessing, wherein the multimodal data includes numerical inspection result data and text inspection result data.

[0024] Specifically, the patient demographic information, numerical test results, and text test results of patients who visited the emergency department, were hospitalized, and had physical examinations were screened from the database and preprocessed as follows: For demographic information, consider age, gender, marital status, and insurance status. Except for age, all others are categorical items. After binning age into 10-year intervals and converting it into a categorical feature, all demographic features become categorical features.

[0025] The test results of numerical types include continuous features and categorical features. For continuous features, such as blood pressure and blood sugar, they are divided into four categories: normal, low, high and abnormal according to the given normal value range, so that all continuous test indicators are also converted into categorical features. 20-40 test indicators are selected from the massive test indicators. For each patient, all target tests performed within half a year are converted into continuous features to form data in the form of time series. It should also be noted that demographic information is also regarded as categorical features and is also added to the time series. Furthermore, the categorical features here are distinguished and represented by numerical values. For example, the above four categories of normal, low, high and abnormal can be represented by 0, 1, 2, and 3 as categorical features.

[0026] Regarding text-based examination results, imaging analysis is a very important method for chronic disease screening. For the five predicted chronic diseases, text examination result data of ultrasound, CT, MR and other related examinations of important parts such as the heart, brain, blood vessels, and kidneys were selected, and the data was represented in text form.

[0027] In order to address the common problem of missing modalities in medical data, missing data is also filled in. Specifically, missing data for numerical inspections are filled with -1, and missing data for text inspections are not filled in.

[0028] During the preprocessing stage, experts from the Medical School were asked to classify the diagnostic results in the database into six categories: diabetes, hypertension, cardiovascular disease, cerebrovascular disease, chronic kidney disease and others. In the entire risk prediction, only the first five chronic diseases were considered as labels for subsequent training and learning.

[0029] In the embodiment, the model building unit 12 is used to build a multi-label chronic disease risk prediction model, such as Figure 2 As shown in the figure, the multi-label chronic disease risk prediction model includes a multimodal feature learning module, a multimodal fusion module, a multi-disease correlation extraction module and a prediction module.

[0030] The multimodal feature learning module is used to perform multimodal learning on numerical test result data and text diagnosis result data to obtain test features and multi-type chronic disease risk features. Numerical test results and text test results are two highly heterogeneous modalities that require separate ways to encode input data.

[0031] Specifically, for numerical test result data, the Transformer model is used as an encoder, the numerical test result data is represented by a time series and then input into the Transformer model, and the Transformer model is used to extract the feature correlation between the time series of the numerical test result data to obtain the test feature Transformer ,in, represents the input time series, Represents the numerical test result data at time T, express The corresponding time position code.

[0032] Specifically, for text-based diagnostic result data, a denoising medical text encoder is designed based on a large language model to filter redundant information and extract useful and accurate information. A series of prompt templates are specifically designed, and a denoising medical text encoder is constructed by interactive learning based on the prompt template using a large language model.

[0033] The prompt template includes task commands, the output format of disease risk interpretability prediction results and reasons. Specifically: You are an experienced clinician. Please make a professional assessment of the patient's chronic disease risk from the following five dimensions based on the patient's CT and B-ultrasound examination results (each item has a full score of 10 points, and the higher the score, the higher the risk): 1. **Diabetes risk**: assessed based on pancreatic morphology (such as atrophy / fatty infiltration), liver fatty degeneration, and increased renal cortical echogenicity; 2. **Risk of hypertension**: assessed based on characteristics such as left ventricular hypertrophy, aortic sclerosis, and renal artery stenosis; 3. **Cardiovascular disease risk**: assessed based on characteristics such as coronary artery calcification, myocardial ischemia, cardiac chamber enlargement, and valve function; 4. **Risk of cerebrovascular disease**: assessed based on characteristics such as carotid artery plaques, intracranial vascular stenosis, and white matter lesions; 5. **Risk of chronic kidney disease**: assessed based on characteristics such as reduced kidney size, thinning of the renal cortex, and hydronephrosis.

[0034] Please strictly follow the following format: - Diabetes risk: X / 10, Rationale: [Concise explanation with specific imaging findings] - Hypertension risk: X / 10, Rationale: [Concise explanation with specific imaging findings] - Cardiovascular disease risk: X / 10, Rationale: [Concise explanation with specific imaging findings] - Risk of cerebrovascular disease: X / 10, Rationale: [Concise explanation with specific imaging findings] - Risk of chronic kidney disease: X / 10, Rationale: [Concise explanation with specific imaging findings] Inspection result text: [Enter specific inspection report text] The denoising medical text encoder is used to predict the risk of multiple chronic diseases based on the prompt template and the corresponding text-based examination result data. The prediction result of the risk of multiple chronic diseases is scored, with a full score of 10. The higher the score, the higher the risk of disease. The text-based examination result data of each patient is represented as a five-dimensional vector as a multi-class chronic disease risk feature. This can effectively reduce the significant noise in the original text data.

[0035] The interpretability of the prediction results of the risk of disease using the scoring system is highly interpretable for the following reasons: 1) It utilizes the powerful natural language generation capabilities unique to the large language model. The large language model can not only output the disease risk score, but also generate the corresponding explanatory text. This dual output mode of "prediction + reason" is more intuitive than traditional black box models (such as random forests and neural networks). 2) Simulate doctor thinking: Through the prompt design ("playing the role of a doctor"), the model will generate a chain of thought explanations that conform to medical logic, which is easy for doctors and patients to understand. 3) Knowledge distillation: The large language model learns a large amount of medical literature, guidelines, and cases during pre-training, and can perform professional deductions on the input text.

[0036] For example, diabetes risk: 6 / 10, reason: B-ultrasound shows fatty infiltration in the body of the pancreas (line 3 of the original text), and thickened liver echo suggests fatty liver.

[0037] Risk of hypertension: 7 / 10, Reason: CT showed left ventricular wall thickness of 12mm (normal <11mm) and mild renal artery stenosis.

[0038] Cardiovascular disease risk: 5 / 10, Reason: Coronary artery calcium score Agatston 120, but no evidence of cardiac chamber enlargement.

[0039] Risk of cerebrovascular disease: 4 / 10, Reason: Carotid artery IMT 1.0mm (critical value), no obvious plaques were found.

[0040] Risk of chronic kidney disease: 3 / 10, Reason: Both kidneys are normal in size, with mild cortical thinning in the right kidney (line 7 of the original text).

[0041] The multimodal fusion module is used to obtain the multimodal fusion features of each type of chronic disease through the cross-modal attention mechanism based on the test features and the risk features of multiple types of chronic diseases. Inspired by the fact that clinicians use different diagnostic index standards according to the patient's health status and specific diseases, the present invention is designed to use a cross-modal attention mechanism to fuse two modal features to learn the importance of each modality in predicting different diseases for different patients, so that the importance of each modality can be dynamically adjusted for different prediction targets. The specific process is: The test features and multiple chronic disease risk characteristics The query vector is calculated by linear projection of the average value of ,in, Represents the query vector weight parameter; In order to capture the importance of different patterns, the key vector corresponding to each type of chronic disease is calculated , ,in c Represents the chronic disease category index, represents the feature dimension, represents the feature space, Indicates c The key vector weight parameter corresponding to the chronic disease class, Represents based on test features and multiple chronic disease risk characteristics The constructed concatenated features are taken as value vectors, and the superscript T indicates transposition; For each chronic disease type, based on the query vector and key vector Generating Attention Weights ,in, Represents the key vector Dimensions; Calculate multimodal fusion features for each type of chronic disease based on attention weights ,in, A vector representing the value weight parameters.

[0042] The multi-disease correlation extraction module is used to extract the multi-disease correlation features of each chronic disease based on the chronic disease graph constructed based on the co-occurrence correlation of multiple chronic diseases. Specifically, a directed graph is constructed for five chronic diseases, such as Figure 3 As shown in the figure, the word embedding representation of each type of chronic disease is used as the node of the chronic disease graph, and the co-occurrence count of chronic diseases in the sample set is used to measure the chronic disease related dependencies. Based on this, the edges between the nodes are constructed to obtain the adjacency matrix of the chronic disease graph. Specifically, the chronic disease related dependencies are modeled in the form of conditional probability. , When chronic disease i Chronic disease when present j The probability of occurrence, and Not equal to , the adjacency matrix is ​​asymmetric, which is consistent with the relationship between chronic diseases in reality and can better simulate the intertwined pathogenesis between different chronic diseases.

[0043] However, the simple correlation above may have two disadvantages. First, the co-occurrence pattern between a chronic disease and other chronic diseases may show a long-tail distribution, and some rare co-occurrences may be noise. Second, the absolute number of co-occurrences in training and testing may not be exactly the same. Overfitting the correlation matrix of the training set may harm the generalization ability. Therefore, setting the threshold Conditional probability Screening is performed to reduce the so-called noise and obtain the value of each element in a 2-value adjacency matrix : .

[0044] Among them, the threshold Preferably [0.5, 0.7], more preferably 0.6, due to the conditional probability Greater than 0.6 indicates chronic disease i Chronic disease when present j The probability of occurrence is higher if More than 0.6 is considered a chronic disease i There is a potential relationship with the pathogenesis of chronic diseases.

[0045] The process of extracting the multi-disease correlation features of each type of chronic disease based on the above chronic disease graph is as follows: input the node features and adjacency matrix of the chronic disease graph into the graph convolutional network (GCN) or graph attention network (GAT) to extract the multi-disease correlation features. Specifically, the first layer of the network input is the node features , is the number of chronic diseases, is the dimension of chronic disease word embedding, and the last layer outputs multi-disease correlation features , is the dimension of the final feature in the multi-disease correlation extraction module.

[0046] The prediction module is used to predict the risk probability of each type of chronic disease based on the fusion of multimodal fusion features and multi-disease correlation features of each type of chronic disease. The specific process is as follows: for each type of chronic disease , the multi-disease correlation features obtained in the multi-disease correlation extraction module And the multimodal fusion features obtained in the multimodal fusion module Inner product multiplication to obtain fusion features , and predict the risk of disease ,in Represents the sigmoid function.

[0047] The above multi-label chronic disease risk prediction model is trained on a large amount of historical data to learn the optimal parameters before being applied. During training, a multi-label based cross entropy loss function is used. And backpropagation to update the parameters: ; in, Represents the chronic disease category index, represents the total number of chronic disease categories, represents the predicted disease risk probability output by the model, represents the true classification label of chronic diseases, Represents the sigmoid function.

[0048] After training, the disease risk prediction performance of the model is evaluated on a new test data set (out-of-sample). Various evaluation indicators can be used to objectively evaluate the prediction accuracy of the model. Multi-label classification evaluation indicators such as precision, recall, and macro-average of F1 score can be used, that is, the indicators are calculated for each label separately and then the average is taken. These indicators can help understand the overall prediction accuracy of the model, thereby providing a reference for further optimization and improvement.

[0049] In the embodiment, the application prediction unit 13 is used to perform multi-label chronic disease risk prediction based on the constructed multi-label chronic disease risk prediction model. When the above-mentioned multi-label chronic disease risk prediction model is constructed, the multimodal data of the patient to be predicted can be preprocessed and then input into the multi-label chronic disease risk prediction model. The test features and multi-class chronic disease risk features are learned through the multimodal feature learning module, and then the multimodal fusion features are extracted based on the test features and multi-class chronic disease risk features through the multi-disease correlation extraction module, and the multimodal fusion features and multi-disease correlation features are input into the prediction module to predict the risk probability of each type of chronic disease after fusion.

[0050] Based on the same inventive concept, an embodiment further provides a computing device, including a memory and one or more processors, wherein executable code is stored in the memory, and when the one or more processors execute the executable code, the steps of using the above-mentioned multi-label chronic disease risk prediction device to predict the risk of multi-label chronic diseases are implemented: S1, using a data processing unit to obtain electronic medical record data and perform multimodal data screening, cleaning and preprocessing, wherein the multimodal data includes numerical inspection result data and text inspection result data; S2, using the model building unit to build a multi-label chronic disease risk prediction model including a multimodal feature learning module, a multimodal fusion module, a multi-disease correlation extraction module and a prediction module, wherein the numerical test result data and the textual diagnosis result data are respectively subjected to the multimodal feature learning module to learn the test features and the multi-class chronic disease risk features, the test features and the multi-class chronic disease risk features are fused through the multimodal fusion module through the cross-modal attention mechanism to obtain the multimodal fusion features of each class of chronic disease, the chronic disease graph constructed based on the co-occurrence correlation of multiple classes of chronic diseases is subjected to the multi-disease correlation extraction module to extract the multi-disease correlation features of each class of chronic disease, the multimodal fusion features and the multi-disease correlation features of each class of chronic disease are fused through the prediction module to predict the risk probability of each class of chronic disease; S3, using the application prediction unit to perform multi-label chronic disease risk prediction based on the constructed multi-label chronic disease risk prediction model.

[0051] The computing device provided in the embodiment, at the hardware level, includes not only a processor and a memory, but also hardware required for other services such as an internal bus, a network interface, and a memory. The memory is a non-volatile memory, and the processor reads the corresponding computer program from the non-volatile memory into the memory and then runs it to implement the steps of multi-label chronic disease risk prediction described in S1-S3 above. Of course, in addition to software implementations, the present invention does not exclude other implementation methods, such as logic devices or a combination of software and hardware, etc., that is to say, the execution subject of the following processing flow is not limited to each logic unit, but can also be hardware or logic devices.

[0052] The specific implementation methods described above provide a detailed description of the technical solutions and beneficial effects of the present invention. It should be understood that the above is only the most preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, supplements and equivalent substitutions made within the scope of the principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A multi-label chronic disease risk prediction device based on multimodal and graph neural network, characterized in that: include: A data processing unit, which is used to obtain electronic medical record data and perform multimodal data screening, cleaning and preprocessing, wherein the multimodal data includes numerical test result data and text test result data; A model construction unit, which is used to construct a multi-label chronic disease risk prediction model including a multimodal feature learning module, a multimodal fusion module, a multi-disease correlation extraction module and a prediction module, wherein the numerical test result data and the textual diagnosis result data are respectively subjected to the multimodal feature learning module to learn the test features and the multi-class chronic disease risk features, the test features and the multi-class chronic disease risk features are fused through the multimodal fusion module through the cross-modal attention mechanism to obtain the multimodal fusion features of each class of chronic diseases, the chronic disease graph constructed based on the co-occurrence correlation of multiple classes of chronic diseases is subjected to the multi-disease correlation extraction module to extract the multi-disease correlation features of each class of chronic diseases, the multimodal fusion features and the multi-disease correlation features of each class of chronic diseases are fused through the prediction module to predict the risk probability of each class of chronic diseases; A prediction unit is applied, which is used to perform multi-label chronic disease risk prediction based on the constructed multi-label chronic disease risk prediction model.

2. The multi-label chronic disease risk prediction device based on multimodal and graph neural network according to claim 1 is characterized in that: The numerical test result data includes continuous features and categorical features. For continuous features, they are converted into categorical features by setting thresholds, where categorical features are distinguished and represented by numerical values. Demographic information is also added to the numerical test result data to supplement the classification features, and the supplemented classification features are also input into the multimodal feature learning module to learn the test features.

3. The multi-label chronic disease risk prediction device based on multimodal and graph neural network according to claim 1 or 2, characterized in that: In the multimodal feature learning module, the Transformer model is used as the encoder, the numerical test result data is represented by a time series sequence and then input into the Transformer model, and the Transformer model is used to extract the feature correlation between the time series of the numerical test result data to obtain the test features.

4. The multi-label chronic disease risk prediction device based on multimodal and graph neural network according to claim 1, characterized in that: In the multimodal feature learning module, a denoising medical text encoder is designed based on a large language model. Specifically, a series of prompt templates are designed, and the denoising medical text encoder is constructed by interactive learning based on the prompt templates using the large language model. The denoising medical text encoder is used to predict the interpretability of the risk of multiple chronic diseases based on the prompt template and the corresponding text-based examination result data. The predictive results of the interpretability of the risk of disease are scored, and the higher the score, the higher the risk of disease. The predictive results of the interpretability of the risk of disease are used as the risk characteristics of multiple chronic diseases. The prompt template includes task commands, the output format of the disease risk interpretable prediction results and reasons.

5. The multi-label chronic disease risk prediction device based on multimodal and graph neural network according to claim 1, characterized in that: In the multimodal fusion module, the test features and multi-type chronic disease risk features are fused through the cross-modal attention mechanism to obtain the multimodal fusion features of each type of chronic disease, including: The test features and multiple chronic disease risk characteristics The query vector is calculated by linear projection of the average value of ,in, Represents the query vector weight parameter; Calculate the key vector corresponding to each type of chronic disease , ,in c Represents the chronic disease category index, represents the feature dimension, represents the feature space, Indicates c The key vector weight parameter corresponding to the chronic disease class, Represents based on test features and multiple chronic disease risk characteristics The constructed concatenated features are taken as value vectors, and the superscript T indicates transposition; For each chronic disease type, based on the query vector and key vector Generating Attention Weights ,in, Represents the key vector Dimensions; Calculate multimodal fusion features for each type of chronic disease based on attention weights ,in, A vector representing the value weight parameters.

6. The multi-label chronic disease risk prediction device based on multimodal and graph neural network according to claim 1, characterized in that: Construct a chronic disease graph based on the co-occurrence correlation of multiple chronic diseases, including: The word embedding representation of each type of chronic disease is used as the node of the chronic disease graph, and the co-occurrence count of chronic diseases in the sample set is used to measure the chronic disease related dependencies. Based on this, the edges between the nodes are constructed to obtain the adjacency matrix of the chronic disease graph. Specifically, the chronic disease related dependencies are modeled in the form of conditional probability. , When chronic disease i Chronic disease when present j The probability of occurrence, and Not equal to , the adjacency matrix is ​​asymmetric, and then the threshold is set Conditional probability Screen to get the value of each element in the adjacency matrix : 。 7. The multi-label chronic disease risk prediction device based on multimodal and graph neural network according to claim 1 is characterized in that: In the multi-disease correlation extraction module, the multi-disease correlation features of each chronic disease are extracted based on the chronic disease graph constructed based on the co-occurrence correlation of multiple chronic diseases, including: The node features and adjacency matrix of the chronic disease graph are input into the graph convolutional network or graph attention network to extract multi-disease correlation features.

8. The multi-label chronic disease risk prediction device based on multimodal and graph neural network according to claim 1, characterized in that: In the prediction module, the multimodal fusion features and multi-disease correlation features of each type of chronic disease are fused to predict the risk probability of each type of chronic disease, including: For each chronic disease , the multi-disease correlation features obtained in the multi-disease correlation extraction module And the multimodal fusion features obtained in the multimodal fusion module Inner product multiplication to obtain fusion features , and predict the risk of disease ,in Represents the sigmoid function.

9. The multi-label chronic disease risk prediction device based on multimodal and graph neural network according to claim 1, characterized in that: The multi-label chronic disease risk prediction model is also trained to learn the optimal parameters before being applied. During training, a multi-label based cross entropy loss function is used. And backpropagation to update the parameters: ； in, Represents the chronic disease category index, represents the total number of chronic disease categories, represents the predicted disease risk probability output by the model, represents the true classification label of chronic diseases, Represents the sigmoid function.

10. A computing device comprising a memory and one or more processors, wherein the memory stores executable code, characterized in that: When the one or more processors execute the executable code, the steps of using the multi-label chronic disease risk prediction device according to any one of claims 1 to 9 to perform multi-label chronic disease risk prediction are implemented: Using a data processing unit to obtain electronic medical record data and perform multimodal data screening, cleaning and preprocessing, wherein the multimodal data includes numerical test result data and text test result data; A multi-label chronic disease risk prediction model including a multimodal feature learning module, a multimodal fusion module, a multi-disease correlation extraction module and a prediction module is constructed using a model construction unit, wherein the numerical test result data and the textual diagnosis result data are respectively subjected to the multimodal feature learning module to learn the test features and the multi-class chronic disease risk features, and the test features and the multi-class chronic disease risk features are fused through the multimodal fusion module through a cross-modal attention mechanism to obtain the multimodal fusion features of each class of chronic disease, and the chronic disease graph constructed based on the co-occurrence correlation of multiple classes of chronic diseases is subjected to the multi-disease correlation extraction module to extract the multi-disease correlation features of each class of chronic disease, and the multimodal fusion features and multi-disease correlation features of each class of chronic disease are fused through the prediction module to predict the risk probability of each class of chronic disease; Multi-label chronic disease risk prediction is performed using the application prediction unit based on the constructed multi-label chronic disease risk prediction model.

Images