Method and system for generating medical suggestions based on multi-modal data fusion

Abstract

The embodiment of the invention provides a method and system for generating medical suggestions based on multi-modal data fusion, and the method comprises the steps: integrating a medical image, a physical examination report and dynamic physiological parameters of a patient through a multi-source data fusion module, generating a multi-modal data set, and synchronously inputting the multi-modal data set into a hybrid reasoning module and a dynamic knowledge graph engine. And the dynamic knowledge graph engine accurately recall a target diagnosis and treatment guide associated with the current multi-modal data set. The rule reasoning sub-module generates a first diagnosis suggestion containing a diagnosis conclusion, a treatment scheme and an evidence level based on a guide structured rule, and meanwhile, the neural network reasoning sub-module analyzes a multi-modal data set by relying on a triple topological structure and an edge weight; and generating a second diagnosis suggestion comprising the disease risk probability, the differentiated treatment suggestion and the evidence source. And finally, the interactive output module fuses the two suggestions to generate a medical suggestion report covering the diagnosis basis, the evidence level and the treatment scheme, so that the diagnosis and treatment precision of chronic disease management and health risk assessment is remarkably improved.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a method and system for generating medical advice based on multimodal data fusion. Background Art

[0002] With the in-depth application of artificial intelligence technology in the medical field, the medical landscape is undergoing an unprecedented data transformation. Multimodal data such as CT images, electronic medical records, and continuous physiological time-series signals collected by smartwatches are growing exponentially. This heterogeneous data records patients' health status in different dimensions.

[0003] Currently, traditional medical systems mainly rely on single-modality data for disease diagnosis and health assessment, such as making judgments based only on text reports of static CT images or isolated blood pressure values.

[0004] However, traditional medical systems rely on single-modality data for disease diagnosis and health assessment, lack the ability to cross-modally fuse multi-source heterogeneous data such as medical image pixel matrices and wearable device timing signals, and lack the correlation analysis between medical image pathological characteristics and physiological indicators, resulting in low diagnostic and treatment accuracy. Summary of the Invention

[0005] The embodiments of the present invention provide a method and system for generating medical advice based on multimodal data fusion, which can improve the accuracy of diagnosis and treatment.

[0006] In a first aspect, an embodiment of the present invention provides a method for generating medical advice based on multimodal data fusion, the method comprising:

[0007] The multi-source data fusion module is used to collect the current patient's medical images, physical examination report data and dynamic physiological parameters, fuse them to generate a multimodal data set and synchronously input it into the hybrid reasoning module and the dynamic knowledge graph engine, wherein the hybrid reasoning module includes a rule reasoning submodule and a neural network reasoning submodule;

[0008] Recalling the target diagnosis and treatment guideline corresponding to the multimodal dataset using the dynamic knowledge graph engine, and generating a first current diagnosis suggestion based on the rule reasoning submodule, wherein the target diagnosis and treatment guideline is associated with a corresponding level of evidence, and the first current diagnosis suggestion includes: a diagnosis conclusion, a treatment plan, and a level of evidence;

[0009] Based on the triple topology structure and corresponding edge weights constructed by the dynamic knowledge graph engine, the neural network reasoning submodule is used to reason on the multimodal dataset to generate a second current diagnosis suggestion including disease risk probability, differentiated treatment suggestions, and evidence sources, wherein the triple topology structure includes: symptoms, diseases, and treatment options;

[0010] Based on the first current diagnostic suggestion and the second current diagnostic suggestion, a medical advice report for the current patient is generated using a multi-level report generator of an interactive output module, wherein the medical advice report includes a diagnostic basis, an evidence level, and a treatment plan.

[0011] Preferably,

[0012] After the multi-source data fusion module is used to collect the medical images, physical examination report data and dynamic physiological parameters of the current patient, and before the fusion generates a multimodal data set and synchronously inputs it into the hybrid reasoning module and the dynamic knowledge graph engine, it further includes:

[0013] Based on a deep learning algorithm, a medical image feature extraction unit is used to extract pathological features of the medical image and generate a structured image feature vector;

[0014] The report feature extraction unit is used to extract medical entities from the physical examination report and generate a structured text feature vector;

[0015] Using a wearable device data adapter to collect the dynamic physiological parameters from the wearable device, and performing a data standardization operation on the dynamic physiological parameters to generate a time series feature sequence;

[0016] The structured image feature vector, the structured text feature vector, and the temporal feature sequence are unified into a cross-modal semantic space by using feature-level fusion technology.

[0017] Preferably,

[0018] The step of using the dynamic knowledge graph engine to recall the target diagnosis and treatment guideline corresponding to the multimodal dataset and generating a first current diagnosis suggestion based on the rule reasoning submodule includes:

[0019] Utilize the real-time crawling module of the dynamic knowledge graph engine to obtain the latest medical diagnosis and treatment guidelines and store them in a cloud-based think tank, wherein the medical diagnosis and treatment guidelines include literature impact factors and trial stages;

[0020] Using the BERT-Med entity linking technology of the knowledge disambiguation unit, semantic normalization is performed on ambiguous terms in the medical diagnosis and treatment guidelines;

[0021] The first formula is used to determine the evidence score based on the literature impact factor of the medical diagnosis and treatment guideline, the trial stage, and the authoritative literature pre-annotated by the local expert database;

[0022] The first formula is:

[0023] Among them, Y is the evidence score value, is the literature impact factor weight, is the literature impact factor, is the experimental stage weight, and is the experimental stage coefficient;

[0024] Based on the evidence score, using an evidence weight calculator to mark the evidence level of the current medical diagnosis and treatment guideline and determine the target diagnosis and treatment guideline corresponding to the multimodal dataset, wherein the target diagnosis and treatment guideline includes: a diagnostic pathway and medication recommendations;

[0025] Based on the target diagnosis and treatment guideline, the rule reasoning submodule is used to generate a first current diagnosis suggestion including a diagnosis conclusion, a treatment plan and an evidence level.

[0026] Preferably,

[0027] The triple topology structure and corresponding edge weights constructed by the dynamic knowledge graph engine are used to reason on the multimodal dataset using the neural network submodule to generate a second current diagnosis recommendation including disease risk probability, differentiated treatment recommendations, and evidence sources, including:

[0028] Extracting disease entities, symptom entities, treatment plan entities, complication entities, and timestamp features from the multimodal dataset using a large language model;

[0029] Based on the timestamp features, entity features in the multimodal dataset are mapped to nodes of the dynamic knowledge graph using entity linking technology, and the extracted entity features and the nodes of the dynamic knowledge graph are semantically aligned using cross-modal alignment technology;

[0030] Based on the edge weights of the dynamic knowledge graph, a graph neural network is used to calculate the association probability between nodes to generate a second current diagnosis recommendation that includes disease risk probability, differentiated treatment recommendations, and evidence sources.

[0031] Preferably,

[0032] The hybrid reasoning module further includes: a risk prediction submodule;

[0033] After generating the medical advice report for the current patient based on the first current diagnosis suggestion and the second current diagnosis suggestion by using the multi-level report generator of the interactive output module, the method further includes:

[0034] Based on the historical time series data in the multimodal dataset, the risk prediction sub-model in the hybrid reasoning module is used to determine the probability of complications within a preset time, and the prediction result is embedded in the medical advice report.

[0035] In a second aspect, an embodiment of the present invention provides a system for generating medical advice based on multimodal data fusion, the system comprising:

[0036] Multi-source data fusion module: used to collect the current patient's medical images, physical examination report data and dynamic physiological parameters, fuse them to generate a multimodal data set and synchronously input it into the hybrid reasoning module and dynamic knowledge graph engine, wherein the hybrid reasoning module includes a rule reasoning submodule and a neural network reasoning submodule;

[0037] A rule reasoning submodule is configured to use the dynamic knowledge graph engine to recall the target diagnosis and treatment guideline corresponding to the multimodal dataset and generate a first current diagnosis suggestion, wherein the target diagnosis and treatment guideline is associated with a corresponding level of evidence, and the first current diagnosis suggestion includes: a diagnosis conclusion, a treatment plan, and a level of evidence;

[0038] A neural network reasoning submodule is configured to reason about the multimodal dataset based on the triple topology structure and corresponding edge weights constructed by the dynamic knowledge graph engine to generate a second current diagnosis recommendation including disease risk probability, differentiated treatment recommendations, and evidence sources, wherein the triple topology structure includes symptoms, diseases, and treatment options;

[0039] Interactive output module: used to generate a medical advice report for the current patient based on the first current diagnostic suggestion and the second current diagnostic suggestion using a multi-level report generator, wherein the medical advice report includes a diagnostic basis, evidence level and treatment plan.

[0040] Preferably,

[0041] The multi-source data fusion module is further configured to, after the multi-source data fusion module is used to collect the medical images, physical examination report data, and dynamic physiological parameters of the current patient, and before the fusion generates a multimodal dataset and synchronously inputs it into the hybrid reasoning module and the dynamic knowledge graph engine, execute the following:

[0042] Based on a deep learning algorithm, a medical image feature extraction unit is used to extract pathological features of the medical image and generate a structured image feature vector;

[0043] The report feature extraction unit is used to extract medical entities from the physical examination report and generate a structured text feature vector;

[0044] Using a wearable device data adapter to collect the dynamic physiological parameters from the wearable device, and performing a data standardization operation on the dynamic physiological parameters to generate a time series feature sequence;

[0045] The structured image feature vector, the structured text feature vector, and the temporal feature sequence are unified into a cross-modal semantic space by using feature-level fusion technology.

[0046] Preferably,

[0047] The rule reasoning submodule is further used to execute:

[0048] Utilize the real-time crawling module of the dynamic knowledge graph engine to obtain the latest medical diagnosis and treatment guidelines and store them in a cloud-based think tank, wherein the medical diagnosis and treatment guidelines include literature impact factors and trial stages;

[0049] Using the BERT-Med entity linking technology of the knowledge disambiguation unit, semantic normalization is performed on ambiguous terms in the medical diagnosis and treatment guidelines;

[0050] The first formula is used to determine the evidence score based on the literature impact factor of the medical diagnosis and treatment guideline, the trial stage, and the authoritative literature pre-annotated by the local expert database;

[0051] The first formula is:

[0052] Among them, Y is the evidence score value, is the literature impact factor weight, is the literature impact factor, is the experimental stage weight, and is the experimental stage coefficient;

[0053] Based on the evidence score, using an evidence weight calculator to mark the evidence level of the current medical diagnosis and treatment guideline and determine the target diagnosis and treatment guideline corresponding to the multimodal dataset, wherein the target diagnosis and treatment guideline includes: a diagnostic pathway and medication recommendations;

[0054] Based on the target diagnosis and treatment guideline, a first current diagnosis recommendation including a diagnosis conclusion, a treatment plan and a level of evidence is generated.

[0055] Preferably,

[0056] The neural network submodule is further used to perform:

[0057] Extracting disease entities, symptom entities, treatment plan entities, complication entities, and timestamp features from the multimodal dataset using a large language model;

[0058] Based on the timestamp features, entity features in the multimodal dataset are mapped to nodes of the dynamic knowledge graph using entity linking technology, and the extracted entity features and the nodes of the dynamic knowledge graph are semantically aligned using cross-modal alignment technology;

[0059] Based on the edge weights of the dynamic knowledge graph, a graph neural network is used to calculate the association probability between nodes to generate a second current diagnosis recommendation that includes disease risk probability, differentiated treatment recommendations, and evidence sources.

[0060] Preferably,

[0061] When the hybrid reasoning module includes a risk prediction submodule,

[0062] The risk prediction submodule is used to determine the probability of complications occurring within a preset time based on the historical time series data in the multimodal dataset after the interactive output module, and embed the prediction results into the medical advice report.

[0063] In a third aspect, an embodiment of the present invention provides a system for generating medical advice based on multimodal data fusion, comprising: at least one memory and at least one processor;

[0064] The at least one memory is configured to store a machine-readable program;

[0065] The at least one processor is configured to call the machine-readable program to execute any one of the methods described in the first aspect.

[0066] In a fourth aspect, an embodiment of the present invention provides a computer-readable medium having computer instructions stored thereon. When the computer instructions are executed by a processor, the processor is caused to execute any of the methods described in the first aspect.

[0067] Embodiments of the present invention provide a method and system for generating medical recommendations based on multimodal data fusion. This method integrates patient medical images, physical examination reports, and dynamic physiological parameters from wearable devices through a multi-source data fusion module to generate a multimodal dataset, which is then simultaneously input into a hybrid inference module and a dynamic knowledge graph engine. The dynamic knowledge graph engine crawls the latest literature daily to accurately retrieve target diagnosis and treatment guidelines relevant to the current multimodal data. The rule-based inference submodule generates a primary diagnosis recommendation based on the structured rules of the guidelines, including the diagnosis conclusion, treatment plan, and level of evidence. Simultaneously, the neural network inference submodule analyzes the multimodal data based on the "symptom-disease-treatment plan" triple topology and edge weights to generate a secondary diagnosis recommendation based on disease risk probability, differentiated treatment recommendations, and evidence sources. Finally, the interactive output module integrates these two types of recommendations to generate a stratified medical report encompassing the diagnosis basis, level of evidence, and treatment plan. This achieves technological breakthroughs in deep multimodal data correlation, real-time updating of diagnosis and treatment knowledge, and interpretability of the inference process, significantly improving the accuracy of chronic disease management and health risk assessment. BRIEF DESCRIPTION OF THE DRAWINGS

[0068] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0069] Figure 1 This is a flowchart of a method for generating medical advice based on multimodal data fusion provided by one embodiment of the present invention;

[0070] Figure 2 is a flowchart of another method for generating medical advice based on multimodal data fusion provided by one embodiment of the present invention;

[0071] Figure 3 is a schematic diagram of a system for generating medical advice based on multimodal data fusion provided by one embodiment of the present invention;

[0072] Figure 4 2 is a schematic diagram of another system for generating medical advice based on multimodal data fusion provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0073] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0074] like Figure 1 As shown, an embodiment of the present invention provides a method for generating medical advice based on multimodal data fusion, which may include the following steps:

[0075] Step 101: Utilize a multi-source data fusion module to collect the current patient's medical images, physical examination report data, and dynamic physiological parameters, fuse them to generate a multimodal dataset, and simultaneously input them into a hybrid reasoning module and a dynamic knowledge graph engine. The hybrid reasoning module includes a rule reasoning submodule and a neural network reasoning submodule.

[0076] Step 102: Using a dynamic knowledge graph engine to recall target diagnosis and treatment guidelines corresponding to the multimodal dataset, and generating a first current diagnosis recommendation based on a rule reasoning submodule, wherein the target diagnosis and treatment guidelines are associated with a corresponding level of evidence, and the first current diagnosis recommendation includes: a diagnosis conclusion, a treatment plan, and a level of evidence;

[0077] Step 103: Based on the triple topology structure and corresponding edge weights constructed by the dynamic knowledge graph engine, the neural network reasoning submodule is used to reason on the multimodal dataset to generate a second current diagnosis recommendation including disease risk probability, differentiated treatment recommendations, and evidence sources. The triple topology structure includes symptoms, diseases, and treatment options.

[0078] Step 104: Based on the first current diagnostic suggestion and the second current diagnostic suggestion, a medical suggestion report for the current patient is generated using the multi-level report generator of the interactive output module, wherein the medical suggestion report includes a diagnostic basis, an evidence level, and a treatment plan.

[0079] In an embodiment of the present invention, a method for generating medical recommendations based on multimodal data fusion is provided. This method integrates patient medical images, physical examination reports, and dynamic physiological parameters from wearable devices through a multi-source data fusion module to generate a multimodal dataset, which is then simultaneously input into a hybrid reasoning module and a dynamic knowledge graph engine. The dynamic knowledge graph engine crawls the latest literature daily to accurately recall target diagnosis and treatment guidelines associated with the current multimodal dataset. The rule reasoning submodule generates a first current diagnosis recommendation based on the structured rules of the guidelines, including the diagnosis conclusion, treatment plan, and level of evidence. Simultaneously, the neural network reasoning submodule analyzes the multimodal dataset using the "symptom-disease-treatment plan" triple topology and edge weights to generate a second current diagnosis recommendation, including disease risk probability, differentiated treatment recommendations, and evidence sources. Finally, the interactive output module fuses these two types of recommendations to generate a medical recommendation report covering the diagnostic basis, level of evidence, and treatment plan. This achieves technological breakthroughs in deep multimodal data correlation, real-time updating of diagnosis and treatment knowledge, and interpretability of the reasoning process, significantly improving the diagnostic and treatment accuracy of chronic disease management and health risk assessment.

[0080] In order to obtain full-dimensional data and improve diagnosis and treatment accuracy, in one embodiment of the present invention, after using the multi-source data fusion module to collect the current patient's medical images, physical examination report data, and dynamic physiological parameters in step 101 of the above embodiment, and before the fusion generates a multimodal data set and synchronously inputs it into the hybrid reasoning module and the dynamic knowledge graph engine, it further includes:

[0081] Based on a deep learning algorithm, a medical image feature extraction unit is used to extract pathological features of the medical image and generate a structured image feature vector;

[0082] The report feature extraction unit is used to extract medical entities from the physical examination report and generate a structured text feature vector;

[0083] Using a wearable device data adapter to collect the dynamic physiological parameters from the wearable device, and performing a data standardization operation on the dynamic physiological parameters to generate a time series feature sequence;

[0084] The structured image feature vector, the structured text feature vector, and the temporal feature sequence are unified into a cross-modal semantic space by using feature-level fusion technology.

[0085] In an embodiment of the present invention, a medical image feature extraction unit driven by a deep learning algorithm is used to analyze the pathological features of medical images such as CT (such as the morphology of fundus microaneurysms) to generate a structured image feature vector; a report feature extraction unit is used to identify medical entities such as disease entities and treatment strategies (such as "diabetes" and "ACE inhibitors") from physical examination reports and convert them into structured text feature vectors; dynamic physiological parameters such as heart rate and blood pressure are collected in real time through a wearable device data adapter, and a time series feature sequence (such as a continuous 72-hour heart rate variability curve) is generated after standardized processing. Finally, through feature-level fusion technology, the image pixel matrix, text semantic features and physiological time series signals are unified into a cross-modal semantic space, solving the problem of multimodal data fragmentation in traditional systems, enabling doctors to analyze the condition from a multi-dimensional correlation perspective (such as the temporal correlation between fundus image features and blood sugar fluctuations in diabetic patients), and providing full-dimensional data support for the precise construction of dynamic knowledge graphs and in-depth analysis of hybrid reasoning modules.

[0086] To ensure that the diagnosis and treatment recommendations comply with the latest medical consensus, in one embodiment of the present invention, the use of the dynamic knowledge graph engine in step 102 of the above embodiment to recall the target diagnosis and treatment guidelines corresponding to the multimodal dataset and generate the first current diagnosis recommendation based on the rule reasoning submodule may specifically include the following steps:

[0087] Utilize the real-time crawling module of the dynamic knowledge graph engine to obtain the latest medical diagnosis and treatment guidelines and store them in a cloud-based think tank, wherein the medical diagnosis and treatment guidelines include literature impact factors and trial stages;

[0088] Using the BERT-Med entity linking technology of the knowledge disambiguation unit, semantic normalization is performed on ambiguous terms in the medical diagnosis and treatment guidelines;

[0089] The first formula is used to determine the evidence score based on the literature impact factor of the medical diagnosis and treatment guideline, the trial stage, and the authoritative literature pre-annotated by the local expert database;

[0090] The first formula is:

[0091] Among them, Y is the evidence score, ω IF is the weight of the literature impact factor, IF is the literature impact factor, ω i is the experimental stage weight, α is the experimental stage coefficient;

[0092] Based on the evidence score, using an evidence weight calculator to mark the evidence level of the current medical diagnosis and treatment guideline and determine the target diagnosis and treatment guideline corresponding to the multimodal dataset, wherein the target diagnosis and treatment guideline includes: a diagnostic pathway and medication recommendations;

[0093] Based on the target diagnosis and treatment guideline, the rule reasoning submodule is used to generate a first current diagnosis suggestion including a diagnosis conclusion, a treatment plan and an evidence level.

[0094] In an embodiment of the present invention, the BERT-Med entity linking technology of the knowledge disambiguation unit is used to perform semantic normalization on ambiguous terms such as "ACE inhibitors" and "angiotensin-converting enzyme inhibitors" in the medical diagnosis and treatment guidelines: first, the BERT-Med model is used to identify medical entities in the text (for example, the "ACEI" entity is extracted from "patients taking ACEI drugs"), and then candidate mapping items are generated from the knowledge base (such as UMLS), and semantic disambiguation is completed by calculating the similarity between the entity context vector and the candidate entity description vector. At the same time, based on the literature impact factor (obtained in real time through the Scopus API, ranging from 0 to 10), the trial stage (Phase I to Phase IV correspond to coefficients of 0.3-0.7 respectively) and the authoritative literature pre-annotated by the local expert database, the first formula is used to calculate the evidence score (for example, the evidence score of a Phase III study with a high impact factor of 8.5 is 0.83). The evidence weight calculator labels the evidence level according to the score range (for example, 0.7-1.0 is a high evidence level), and accurately screens out the target diagnosis and treatment guidelines (such as the 2023ADA diabetes guidelines) that match the multimodal dataset. Finally, the structured rules embedded in the guidelines through the rule reasoning submodule are used to generate the first current diagnostic recommendation including diagnostic conclusions, treatment plans and evidence levels, to ensure that the diagnosis and treatment recommendations are in line with the latest medical consensus and have traceable evidence support.

[0095] To achieve accurate risk assessment, in one embodiment of the present invention, step 103 in the above embodiment uses the triple topology structure and corresponding edge weights constructed by the dynamic knowledge graph engine to reason on the multimodal dataset using the neural network submodule to generate a second current diagnosis recommendation including disease risk probability, differentiated treatment recommendations, and evidence sources. Specifically, the following steps may be included:

[0096] Extracting disease entities, symptom entities, treatment plan entities, complication entities, and timestamp features from the multimodal dataset using a large language model;

[0097] Based on the timestamp features, entity features in the multimodal dataset are mapped to nodes of the dynamic knowledge graph using entity linking technology, and the extracted entity features and the nodes of the dynamic knowledge graph are semantically aligned using cross-modal alignment technology;

[0098] Based on the edge weights of the dynamic knowledge graph, a graph neural network is used to calculate the association probability between nodes to generate a second current diagnosis recommendation that includes disease risk probability, differentiated treatment recommendations, and evidence sources.

[0099] In an embodiment of the present invention, large language models (LLMs) are used to extract disease entities (such as "diabetes"), symptom entities (such as "dizziness"), treatment plan entities (such as "ACE inhibitors"), complication entities (such as "retinopathy") and timestamp features (such as wearable device monitoring time) from a multimodal dataset. The entity linking technology is used to map the entity features of each modality (such as medical image pathological features, text report diagnostic conclusions) to triple nodes in the dynamic knowledge graph (such as the "diabetes" node is associated with the "polydipsia" symptom node and the "insulin injection" treatment node). The cross-modal alignment technology (such as contrastive learning) is used to unify the image pixel matrix, text semantic vector and time-series physiological signal into the graph semantic space to ensure the semantic consistency of heterogeneous data. Based on the edge weights in the graph, the graph neural network (GNN) is used to analyze the association strength between nodes, calculate the probability of disease progression (for example, LSTM combined with timestamps predicts the probability of complications in the next three months), and generate a second current diagnostic recommendation that includes disease risk probability, differentiated treatment recommendations (such as drug dosage adjustments based on individual patient characteristics), and evidence sources (such as PubMed document IDs), realizing dynamic reasoning and accurate risk assessment driven by multimodal data.

[0100] In order to formulate a preventive intervention plan, in one embodiment of the present invention, the hybrid reasoning module in the above embodiment further includes: a risk prediction submodule;

[0101] After generating the medical advice report for the current patient based on the first current diagnosis suggestion and the second current diagnosis suggestion by using the multi-level report generator of the interactive output module, the method further includes:

[0102] Based on the historical time series data in the multimodal dataset, the risk prediction sub-model in the hybrid reasoning module is used to determine the probability of complications within a preset time, and the prediction result is embedded in the medical advice report.

[0103] In an embodiment of the present invention, the risk prediction sub-model in the hybrid reasoning module can perform time series feature analysis and pattern recognition on historical data, predict the probability of complications within a preset time (such as the next 3 months) (such as the risk probability of diabetic retinopathy), and embed the prediction results in the form of quantitative indicators in the medical advice report, forming a complete report including diagnostic basis, evidence level, treatment plan and complication risk prediction, providing dynamic data support for doctors to formulate preventive intervention plans and patients to understand the development of the disease, further improving the foresight of chronic disease management.

[0104] In one embodiment of the present invention, the interactive output module is a module that interacts directly with the user, and includes a 3D visualization unit and a report display. The 3D visualization unit uses advanced graphics processing technology to accurately construct and display a 3D model of the heart, while simulating the dynamic changes of the lesion site, providing doctors with an intuitive and comprehensive view of the lesion. Doctors can observe the heart structure and its pathological characteristics from different angles by rotating, scaling, and moving the model, thereby improving the accuracy and efficiency of diagnosis. The multi-level report generator automatically generates a medical report that complies with the FHIR standard based on the patient's specific condition and treatment plan. The report not only contains detailed professional information for internal communication and decision-making by doctors, but also can automatically generate a patient-friendly simplified guide to explain the condition, treatment plan and expected results to patients in plain language, thereby enhancing communication and understanding between doctors and patients.

[0105] like Figure 2 To more clearly illustrate the technical solutions and advantages of the present invention, a method for generating medical advice based on multimodal data fusion is described in detail below. The method may include the following steps:

[0106] Step 201: using a multi-source data fusion module to collect the current patient's medical images, physical examination report data, and dynamic physiological parameters;

[0107] Step 202: Based on a deep learning algorithm, a medical image feature extraction unit is used to extract pathological features of the medical image and generate a structured image feature vector;

[0108] Step 203: extracting medical entities from the physical examination report using a report feature extraction unit to generate a structured text feature vector;

[0109] Step 204: using the wearable device data adapter to collect dynamic physiological parameters from the wearable device, and performing data standardization operations on the dynamic physiological parameters to generate a time series feature sequence;

[0110] Step 205: Using feature-level fusion technology, the structured image feature vector, the structured text feature vector, and the temporal feature sequence are unified into a cross-modal semantic space, fused to generate a multimodal dataset, and simultaneously input into the hybrid reasoning module and the dynamic knowledge graph engine. The hybrid reasoning module includes a rule reasoning submodule, a neural network reasoning submodule, and a risk prediction submodule.

[0111] Step 206: Using the real-time crawling module of the dynamic knowledge graph engine, the latest medical diagnosis and treatment guidelines are obtained and stored in the cloud think tank. The medical diagnosis and treatment guidelines include literature impact factors and experimental stages. Step 207: Using the BERT-Med entity linking technology of the knowledge disambiguation unit, semantic normalization operations are performed on ambiguous terms in the medical diagnosis and treatment guidelines.

[0112] Step 208: Determine the evidence score using the first formula based on the literature impact factor of the medical diagnosis and treatment guidelines, the trial stage, and the authoritative literature pre-annotated by the local expert database;

[0113] Specifically, the first formula is:

[0114] Among them, Y is the evidence score, ω IF is the weight of the literature impact factor, IF is the literature impact factor, ω i is the experimental stage weight, α is the experimental stage coefficient;

[0115] Step 209: Based on the evidence score, use an evidence weight calculator to label the evidence level of the current medical diagnosis and treatment guideline and determine the target diagnosis and treatment guideline corresponding to the multimodal dataset, wherein the target diagnosis and treatment guideline includes: diagnostic pathway and medication recommendation;

[0116] Step 210: Based on the target diagnosis and treatment guideline, a rule reasoning submodule is used to generate a first current diagnosis suggestion including a diagnosis conclusion, a treatment plan, and a level of evidence;

[0117] Step 211: extracting disease entities, symptom entities, treatment plan entities, complication entities, and timestamp features from the multimodal dataset using a large language model;

[0118] Step 212: Based on the timestamp feature, entity features in the multimodal dataset are mapped to nodes of the dynamic knowledge graph using entity linking technology, and the extracted entity features and nodes of the dynamic knowledge graph are semantically aligned using cross-modal alignment technology;

[0119] Step 213: Based on the edge weights of the dynamic knowledge graph, the graph neural network is used to calculate the association probability between nodes, and a second current diagnosis recommendation including disease risk probability, differentiated treatment recommendations, and evidence sources is generated;

[0120] Step 214: Based on the first current diagnostic suggestion and the second current diagnostic suggestion, a medical suggestion report for the current patient is generated using the multi-level report generator of the interactive output module, wherein the medical suggestion report includes a diagnostic basis, an evidence level, and a treatment plan.

[0121] Step 215: Based on the historical time series data in the multimodal dataset, the risk prediction sub-model in the hybrid reasoning module is used to determine the probability of complications within a preset time, and the prediction results are embedded in the medical advice report.

[0122] like Figure 3 As shown, an embodiment of the present invention provides a system for generating medical advice based on multimodal data fusion, the system comprising:

[0123] Multi-source data fusion module 301: used to collect the current patient's medical images, physical examination report data and dynamic physiological parameters, fuse them to generate a multimodal data set and synchronously input it into the hybrid reasoning module and dynamic knowledge graph engine, wherein the hybrid reasoning module includes a rule reasoning submodule and a neural network reasoning submodule;

[0124] Rule reasoning submodule 302: configured to use the dynamic knowledge graph engine to recall the target diagnosis and treatment guideline corresponding to the multimodal data set collected by the multi-source data fusion module 301, and generate a first current diagnosis suggestion, wherein the target diagnosis and treatment guideline is associated with a corresponding level of evidence, and the first current diagnosis suggestion includes: a diagnosis conclusion, a treatment plan, and a level of evidence;

[0125] Neural network reasoning submodule 303: configured to perform reasoning on the multimodal dataset collected by the multi-source data fusion module 301 based on the triple topology structure and corresponding edge weights constructed by the dynamic knowledge graph engine, and generate a second current diagnosis recommendation including disease risk probability, differentiated treatment recommendations, and evidence sources, wherein the triple topology structure includes: symptoms, diseases, and treatment options;

[0126] Interactive output module 303: used to generate a medical advice report for the current patient based on the first current diagnostic suggestion inferred by the rule reasoning submodule 302 and the second current diagnostic suggestion inferred by the neural network reasoning submodule 303, using a multi-level report generator, wherein the medical advice report includes a diagnostic basis, evidence level and treatment plan.

[0127] like Figure 3 As shown, the multi-source data fusion module 301 is further configured to, after the multi-source data fusion module is used to collect the medical images, physical examination report data, and dynamic physiological parameters of the current patient, and before the fusion generates a multimodal data set and synchronously inputs it into the hybrid reasoning module and the dynamic knowledge graph engine, execute the following:

[0128] Based on a deep learning algorithm, a medical image feature extraction unit is used to extract pathological features of the medical image and generate a structured image feature vector;

[0129] The report feature extraction unit is used to extract medical entities from the physical examination report and generate a structured text feature vector;

[0130] Using a wearable device data adapter to collect the dynamic physiological parameters from the wearable device, and performing a data standardization operation on the dynamic physiological parameters to generate a time series feature sequence;

[0131] The structured image feature vector, the structured text feature vector, and the temporal feature sequence are unified into a cross-modal semantic space by using feature-level fusion technology.

[0132] like Figure 3 As shown, the rule reasoning submodule 302 is further used to execute:

[0133] Utilize the real-time crawling module of the dynamic knowledge graph engine to obtain the latest medical diagnosis and treatment guidelines and store them in a cloud-based think tank, wherein the medical diagnosis and treatment guidelines include literature impact factors and trial stages;

[0134] Using the BERT-Med entity linking technology of the knowledge disambiguation unit, semantic normalization is performed on ambiguous terms in the medical diagnosis and treatment guidelines;

[0135] The first formula is used to determine the evidence score based on the literature impact factor of the medical diagnosis and treatment guideline, the trial stage, and the authoritative literature pre-annotated by the local expert database;

[0136] The first formula is:

[0137] Among them, Y is the evidence score, ω IF is the weight of the literature impact factor, IF is the literature impact factor, ω i is the experimental stage weight, α is the experimental stage coefficient;

[0138] Based on the evidence score, using an evidence weight calculator to mark the evidence level of the current medical diagnosis and treatment guideline and determine the target diagnosis and treatment guideline corresponding to the multimodal dataset, wherein the target diagnosis and treatment guideline includes: a diagnostic pathway and medication recommendations;

[0139] Based on the target diagnosis and treatment guideline, a first current diagnosis recommendation including a diagnosis conclusion, a treatment plan and a level of evidence is generated.

[0140] like Figure 3 As shown, the neural network submodule 303 is further configured to execute:

[0141] Extracting disease entities, symptom entities, treatment plan entities, complication entities, and timestamp features from the multimodal dataset using a large language model;

[0142] Based on the timestamp features, entity features in the multimodal dataset are mapped to nodes of the dynamic knowledge graph using entity linking technology, and the extracted entity features and the nodes of the dynamic knowledge graph are semantically aligned using cross-modal alignment technology;

[0143] Based on the edge weights of the dynamic knowledge graph, a graph neural network is used to calculate the association probability between nodes to generate a second current diagnosis recommendation that includes disease risk probability, differentiated treatment recommendations, and evidence sources.

[0144] based on Figure 3 A system for generating medical advice based on multi-source data fusion is shown in Figure 4 As shown,

[0145] When the hybrid reasoning module includes the risk prediction submodule 305,

[0146] The risk prediction submodule 305 is used to determine the probability of complications occurring within a preset time period based on the historical time series data in the multimodal dataset after the interactive output module 304, and embed the prediction result into the medical advice report.

[0147] It should be understood that the structure illustrated in the embodiments of the present invention does not constitute a specific limitation on the system for generating medical advice based on multi-source data fusion. In other embodiments of the present invention, the system for generating medical advice based on multi-source data fusion may include more or fewer components than illustrated, or may combine or separate certain components, or arrange the components differently. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0148] The information interaction, execution process, etc. between the units in the above-mentioned device are based on the same concept as the embodiment of the method of the present invention. For specific contents, please refer to the description in the embodiment of the method of the present invention and will not be repeated here.

[0149] An embodiment of the present invention further provides a system for generating medical advice based on multi-source data fusion, comprising: at least one memory and at least one processor;

[0150] at least one memory for storing a machine-readable program;

[0151] At least one processor is configured to call a machine-readable program to execute a method for generating medical advice based on multi-source data fusion according to any embodiment of the present invention.

[0152] An embodiment of the present invention further provides a computer-readable medium having computer instructions stored thereon. When the computer instructions are executed by a processor, the processor executes a method for generating medical advice based on multi-source data fusion according to any embodiment of the present invention.

[0153] Specifically, a system or device equipped with a storage medium can be provided, on which software program codes that implement the functions of any of the above-mentioned embodiments are stored, and a computer (or CPU or MPU) of the system or device can be enabled to read and execute the program codes stored in the storage medium.

[0154] In this case, the program code itself read from the storage medium can realize the function of any one of the above-mentioned embodiments, and thus the program code and the storage medium storing the program code constitute part of the present invention.

[0155] Examples of storage media for providing program code include floppy disks, hard disks, magneto-optical disks, optical disks (such as CD-ROM, CD-R, CD-RW, DVD-ROM, DVD-RAM, DVD-RW, DVD+RW), magnetic tapes, non-volatile memory cards, and ROMs. Alternatively, the program code can be downloaded from a server computer via a communication network.

[0156] In addition, it should be clear that the functions of any of the above embodiments can be achieved not only by executing the program code read by the computer, but also by enabling the operating system operating on the computer to complete part or all of the actual operations based on the instructions of the program code.

[0157] In addition, it can be understood that the program code read from the storage medium is written into the memory provided in the expansion board inserted into the computer or into the memory provided in the expansion unit connected to the computer, and then based on the instructions of the program code, the CPU installed on the expansion board or expansion unit is enabled to perform part or all of the actual operations, thereby realizing the functions of any of the above embodiments.

[0158] Each embodiment of the present invention has at least the following beneficial effects:

[0159] 1. In an embodiment of the present invention, a method for generating medical advice based on multimodal data fusion is provided. The multi-source data fusion module integrates the patient's medical images, physical examination reports and dynamic physiological parameters of wearable devices to generate a multimodal data set, and simultaneously inputs the hybrid reasoning module and the dynamic knowledge graph engine. The dynamic knowledge graph engine captures the latest literature every day and accurately recalls the target diagnosis and treatment guidelines related to the current multimodal data. The rule reasoning submodule generates a first current diagnosis suggestion containing a diagnosis conclusion, treatment plan and evidence level based on the structured rules of the guideline. At the same time, the neural network reasoning submodule relies on the "symptom-disease-treatment plan" triple topology structure and edge weights to analyze the multimodal data set to generate a second current diagnosis suggestion of disease risk probability, differentiated treatment suggestions and evidence source. Finally, the interactive output module fuses the two types of suggestions to generate a stratified medical report covering diagnostic basis, evidence level and treatment plan, achieving a technical breakthrough in deep association of multimodal data, real-time updating of diagnosis and treatment knowledge and interpretability of the reasoning process, thereby significantly improving the diagnosis and treatment accuracy of chronic disease management and health risk assessment;

[0160] 2. In an embodiment of the present invention, a medical image feature extraction unit driven by a deep learning algorithm is used to analyze the pathological features of medical images such as CT and generate a structured image feature vector; a report feature extraction unit is used to identify medical entities such as disease entities and treatment strategies from physical examination reports and convert them into structured text feature vectors; dynamic physiological parameters such as heart rate and blood pressure are collected in real time through a wearable device data adapter, and a time series feature sequence is generated after standardized processing. Finally, through feature-level fusion technology, the image pixel matrix, text semantic features and physiological time series signals are unified into a cross-modal semantic space, solving the problem of multimodal data fragmentation in traditional systems, enabling doctors to analyze the condition from a multi-dimensional correlation perspective, and providing full-dimensional data support for the precise construction of dynamic knowledge graphs and in-depth analysis of hybrid reasoning modules;

[0161] 3. In an embodiment of the present invention, semantic normalization is performed on ambiguous terms such as "ACE inhibitors" and "angiotensin-converting enzyme inhibitors" in medical diagnosis and treatment guidelines through the BERT-Med entity linking technology of the knowledge disambiguation unit: first, the BERT-Med model is used to identify medical entities in the text, and then candidate mapping items are generated from the knowledge base. The semantic disambiguation is completed by calculating the similarity between the entity context vector and the candidate entity description vector. At the same time, based on the literature impact factor, the experimental stage and the authoritative literature pre-annotated by the local expert database, the first formula is used to calculate the evidence score, and the evidence weight calculator annotates the evidence level according to the score range to accurately screen out the target diagnosis and treatment guidelines that match the multimodal data set. Finally, the structured rules embedded in the guidelines through the rule reasoning submodule are used to generate the first current diagnostic recommendation containing the diagnostic conclusion, treatment plan and evidence level, to ensure that the diagnosis and treatment recommendation conforms to the latest medical consensus and has traceable evidence support.

[0162] It should be noted that not all steps and modules in the above processes and system structure diagrams are required, and certain steps or modules can be omitted according to actual needs. The execution order of each step is not fixed and can be adjusted as needed. The system structure described in the above embodiments can be a physical structure or a logical structure, that is, some modules may be implemented by the same physical entity, or some modules may be implemented by multiple physical entities, or may be implemented by certain components in multiple independent devices.

[0163] In the above embodiments, the hardware unit can be realized by mechanical means or electrical means. For example, a hardware unit can include permanent dedicated circuits or logic (such as special processors, FPGA or ASIC) to complete the corresponding operations. The hardware unit can also include programmable logic or circuits (such as general-purpose processors or other programmable processors), which can be temporarily set up by software to complete the corresponding operations. Concrete implementation (mechanical means or dedicated permanent circuits or temporarily set circuits) can be determined based on the consideration on cost and time.

[0164] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A method for generating medical advice based on multimodal data fusion, characterized in that: The method includes: The multi-source data fusion module is used to collect the current patient's medical images, physical examination report data and dynamic physiological parameters, fuse them to generate a multimodal data set and synchronously input it into the hybrid reasoning module and the dynamic knowledge graph engine, wherein the hybrid reasoning module includes a rule reasoning submodule and a neural network reasoning submodule; Recalling the target diagnosis and treatment guideline corresponding to the multimodal dataset using the dynamic knowledge graph engine, and generating a first current diagnosis suggestion based on the rule reasoning submodule, wherein the target diagnosis and treatment guideline is associated with a corresponding level of evidence, and the first current diagnosis suggestion includes: a diagnosis conclusion, a treatment plan, and a level of evidence; Based on the triple topology structure and corresponding edge weights constructed by the dynamic knowledge graph engine, the neural network reasoning submodule is used to reason on the multimodal dataset to generate a second current diagnosis suggestion including disease risk probability, differentiated treatment suggestions, and evidence sources, wherein the triple topology structure includes: symptoms, diseases, and treatment options; Based on the first current diagnostic suggestion and the second current diagnostic suggestion, a medical advice report for the current patient is generated using a multi-level report generator of an interactive output module, wherein the medical advice report includes a diagnostic basis, an evidence level, and a treatment plan.

2. The method according to claim 1, characterized in that After the multi-source data fusion module is used to collect the medical images, physical examination report data and dynamic physiological parameters of the current patient, and before the fusion generates a multimodal data set and synchronously inputs it into the hybrid reasoning module and the dynamic knowledge graph engine, it further includes: Based on a deep learning algorithm, a medical image feature extraction unit is used to extract pathological features of the medical image and generate a structured image feature vector; The report feature extraction unit is used to extract medical entities from the physical examination report and generate a structured text feature vector; Using a wearable device data adapter to collect the dynamic physiological parameters from the wearable device, and performing a data standardization operation on the dynamic physiological parameters to generate a time series feature sequence; The structured image feature vector, the structured text feature vector, and the temporal feature sequence are unified into a cross-modal semantic space by using feature-level fusion technology.

3. The method according to claim 1, characterized in that The step of using the dynamic knowledge graph engine to recall the target diagnosis and treatment guideline corresponding to the multimodal dataset and generating a first current diagnosis suggestion based on the rule reasoning submodule includes: Utilize the real-time crawling module of the dynamic knowledge graph engine to obtain the latest medical diagnosis and treatment guidelines and store them in a cloud-based think tank, wherein the medical diagnosis and treatment guidelines include literature impact factors and trial stages; Using the BERT-Med entity linking technology of the knowledge disambiguation unit, semantic normalization is performed on ambiguous terms in the medical diagnosis and treatment guidelines; The first formula is used to determine the evidence score based on the literature impact factor of the medical diagnosis and treatment guideline, the trial stage, and the authoritative literature pre-annotated by the local expert database; The first formula is: Among them, Y is the evidence score, ω IF is the weight of the literature impact factor, IF is the literature impact factor, ω i is the experimental stage weight, α is the experimental stage coefficient; Based on the evidence score, using an evidence weight calculator to mark the evidence level of the current medical diagnosis and treatment guideline and determine the target diagnosis and treatment guideline corresponding to the multimodal dataset, wherein the target diagnosis and treatment guideline includes: a diagnostic pathway and medication recommendations; Based on the target diagnosis and treatment guideline, the rule reasoning submodule is used to generate the first current diagnosis suggestion including a diagnosis conclusion, a treatment plan and a level of evidence.

4. The method according to any one of claims 1 to 3, characterized in that The triple topology structure and corresponding edge weights constructed by the dynamic knowledge graph engine are used to reason on the multimodal dataset using the neural network submodule to generate a second current diagnosis recommendation including disease risk probability, differentiated treatment recommendations, and evidence sources, including: Extracting disease entities, symptom entities, treatment plan entities, complication entities, and timestamp features from the multimodal dataset using a large language model; Based on the timestamp features, entity features in the multimodal dataset are mapped to nodes of the dynamic knowledge graph using entity linking technology, and the extracted entity features and the nodes of the dynamic knowledge graph are semantically aligned using cross-modal alignment technology; Based on the edge weights of the dynamic knowledge graph, a graph neural network is used to calculate the association probability between nodes, and the second current diagnosis recommendation is generated, which includes the disease risk probability, differentiated treatment recommendations, and evidence sources; and / or, The hybrid reasoning module further includes: a risk prediction submodule; After generating the medical advice report for the current patient based on the first current diagnosis suggestion and the second current diagnosis suggestion by using the multi-level report generator of the interactive output module, the method further includes: Based on the historical time series data in the multimodal dataset, the risk prediction sub-model in the hybrid reasoning module is used to determine the probability of complications within a preset time, and the prediction result is embedded in the medical advice report.

5. A system for generating medical advice based on multimodal data fusion, characterized in that: The system includes: Multi-source data fusion module: used to collect the current patient's medical images, physical examination report data and dynamic physiological parameters, fuse them to generate a multimodal data set and synchronously input it into the hybrid reasoning module and dynamic knowledge graph engine, wherein the hybrid reasoning module includes a rule reasoning submodule and a neural network reasoning submodule; A rule reasoning submodule is configured to use the dynamic knowledge graph engine to recall the target diagnosis and treatment guideline corresponding to the multimodal dataset and generate a first current diagnosis suggestion, wherein the target diagnosis and treatment guideline is associated with a corresponding level of evidence, and the first current diagnosis suggestion includes: a diagnosis conclusion, a treatment plan, and a level of evidence; A neural network reasoning submodule is configured to reason about the multimodal dataset based on the triple topology structure and corresponding edge weights constructed by the dynamic knowledge graph engine to generate a second current diagnosis recommendation including disease risk probability, differentiated treatment recommendations, and evidence sources, wherein the triple topology structure includes symptoms, diseases, and treatment options; Interactive output module: used to generate a medical advice report for the current patient based on the first current diagnostic suggestion and the second current diagnostic suggestion using a multi-level report generator, wherein the medical advice report includes a diagnostic basis, evidence level and treatment plan.

6. The system according to claim 5, characterized in that The multi-source data fusion module is further configured to, after the multi-source data fusion module is used to collect the medical images, physical examination report data, and dynamic physiological parameters of the current patient, and before the fusion generates a multimodal dataset and synchronously inputs it into the hybrid reasoning module and the dynamic knowledge graph engine, execute the following: Based on a deep learning algorithm, a medical image feature extraction unit is used to extract pathological features of the medical image and generate a structured image feature vector; The report feature extraction unit is used to extract medical entities from the physical examination report and generate a structured text feature vector; Using a wearable device data adapter to collect the dynamic physiological parameters from the wearable device, and performing a data standardization operation on the dynamic physiological parameters to generate a time series feature sequence; The structured image feature vector, the structured text feature vector, and the temporal feature sequence are unified into a cross-modal semantic space by using feature-level fusion technology.

7. The system according to claim 5, characterized in that The rule reasoning submodule is further used to execute: Utilize the real-time crawling module of the dynamic knowledge graph engine to obtain the latest medical diagnosis and treatment guidelines and store them in a cloud-based think tank, wherein the medical diagnosis and treatment guidelines include literature impact factors and trial stages; Using the BERT-Med entity linking technology of the knowledge disambiguation unit, semantic normalization is performed on ambiguous terms in the medical diagnosis and treatment guidelines; The first formula is used to determine the evidence score based on the literature impact factor of the medical diagnosis and treatment guideline, the trial stage, and the authoritative literature pre-annotated by the local expert database; The first formula is: Among them, Y is the evidence score, ω IF is the weight of the literature impact factor, IF is the literature impact factor, ω i is the experimental stage weight, α is the experimental stage coefficient; Based on the evidence score, using an evidence weight calculator to mark the evidence level of the current medical diagnosis and treatment guideline and determine the target diagnosis and treatment guideline corresponding to the multimodal dataset, wherein the target diagnosis and treatment guideline includes: a diagnostic pathway and medication recommendations; Based on the target diagnosis and treatment guideline, a first current diagnosis recommendation including a diagnosis conclusion, a treatment plan and a level of evidence is generated.

8. The system according to any one of claims 5 to 7, characterized in that: The neural network submodule is further used to perform: Extracting disease entities, symptom entities, treatment plan entities, complication entities, and timestamp features from the multimodal dataset using a large language model; Based on the timestamp features, entity features in the multimodal dataset are mapped to nodes of the dynamic knowledge graph using entity linking technology, and the extracted entity features and the nodes of the dynamic knowledge graph are semantically aligned using cross-modal alignment technology; Based on the edge weights of the dynamic knowledge graph, a graph neural network is used to calculate the association probability between nodes, and a second current diagnosis recommendation including disease risk probability, differentiated treatment recommendations, and evidence sources is generated; and / or, When the hybrid reasoning module includes a risk prediction submodule, The risk prediction submodule is used to determine the probability of complications occurring within a preset time based on the historical time series data in the multimodal dataset after the interactive output module, and embed the prediction results into the medical advice report.

9. A system for generating medical advice based on multimodal data fusion, characterized in that: include: at least one memory and at least one processor; The at least one memory is configured to store a machine-readable program; The at least one processor is configured to call the machine-readable program to execute the method according to any one of claims 1 to 4.

10. A computer-readable medium, characterized in that The computer-readable medium stores computer instructions, which, when executed by a processor, cause the processor to perform the method according to any one of claims 1 to 4.

Images