Large model agent driven medical multi-modal knowledge fusion method

Abstract

The invention discloses a medical multi-modal knowledge fusion method driven by a large model agent, and relates to the technical field of medical treatment, and the method comprises the following steps: obtaining medical field multi-modal data, and a medical knowledge labeling rule and a cross-modal association mapping relation corresponding to each modal data; processing the multi-modal data based on a medical knowledge labeling rule to obtain a standardized medical multi-modal data set; establishing an inter-modal association index of the standardized medical multi-modal data set based on a cross-modal association mapping relationship; performing knowledge unit splitting on the standardized medical multi-modal data set through a modal analysis module of the large model agent to obtain a medical single-modal knowledge unit set; performing association matching on the medical single-modal knowledge unit set based on an inter-modal association index to obtain a medical cross-modal knowledge association set; the method has the effect of continuously improving the precision and reliability of medical multi-modal knowledge fusion.

Description

Technical Field

[0001] This invention relates to the field of medical technology, and more specifically, to a method for medical multimodal knowledge fusion driven by large-scale intelligent agents. Background Technology

[0002] Traditional methods of medical knowledge fusion struggle to meet the ever-increasing demands of healthcare. Medical data encompasses multiple modalities, including text, images, and sensor data. These data come from diverse sources and are presented in various formats, making it difficult for traditional methods to deeply mine the complex clinical knowledge contained within them. For text modalities, it is challenging to extract key descriptions of patient conditions and diagnostic conclusions; for image modalities, there is a lack of ability to analyze visual features from multiple dimensions and correlate them with clinical semantics; and for sensor modalities, it is difficult to dynamically capture the connection between changes in physiological indicators and clinical intent. The inability to form a comprehensive medical knowledge system makes it difficult for physicians to obtain integrated multimodal information for support during diagnosis and treatment, thus affecting diagnostic accuracy and treatment planning. Furthermore, traditional methods lack robust verification and optimization mechanisms, making it difficult to effectively assess the quality of the fused knowledge. Even if knowledge conflicts or gaps exist, they are difficult to detect and correct in a timely manner, thus failing to guarantee the effectiveness and clinical suitability of the knowledge. Summary of the Invention

[0003] To address the shortcomings of existing technologies, the present invention aims to provide a medical multimodal knowledge fusion method driven by a large-scale intelligent agent.

[0004] To achieve the above objectives, the present invention provides the following technical solution: A medical multimodal knowledge fusion method driven by a large-model intelligent agent, comprising the following steps: Acquire multimodal data in the medical field, as well as the medical knowledge annotation rules and cross-modal association mapping relationships corresponding to each modality of data; A standardized medical multimodal dataset is obtained by processing multimodal data based on medical knowledge annotation rules; an intermodal association index for the standardized medical multimodal dataset is established based on cross-modal association mapping relationships; The standardized medical multimodal dataset is decomposed into knowledge units using the modality parsing module of the large model agent to obtain a medical single-modal knowledge unit set; Based on the intermodal association index, the medical single-modal knowledge unit set is associated and matched to obtain the medical cross-modal knowledge association set; The medical single-modal knowledge unit set and the medical cross-modal knowledge association set are input into the fusion model to obtain the preliminary medical multimodal knowledge fusion results; A medical knowledge verification index system is set up, and the knowledge fusion verification result set is obtained after verifying and evaluating the preliminary medical multimodal knowledge fusion results based on the index system. After identifying knowledge conflict points and fusion vulnerabilities in the preliminary medical multimodal knowledge fusion results based on the knowledge fusion verification result set, a structured identification result set is formed. The parameters of the fusion model are iteratively optimized through the feedback adjustment module of the large model agent until a medical multimodal interactive knowledge fusion result that meets the verification index requirements is obtained.

[0005] Preferably, the multimodal data includes text modal feature data, image modal feature data, and sensor modal feature data.

[0006] Preferably, a standardized medical multimodal dataset is obtained by processing the multimodal data based on medical knowledge annotation rules, specifically including the following steps: Text modal feature data, image modal feature data, and sensor modal feature data are mapped to the feature space through linear transformation. During the mapping process, the corresponding modal weight coefficients are obtained according to the contribution of each modal data to the fusion of medical knowledge. The text modal feature data, image modal feature data, and sensor modal feature data are adjusted and processed according to modal weight coefficients to obtain a standardized medical multimodal dataset; wherein, the standardized medical multimodal dataset includes standardized text modal feature data, standardized image modal feature data, and standardized sensor modal feature data.

[0007] Preferably, establishing an inter-modal association index for a standardized medical multimodal dataset based on cross-modal association mapping relationships specifically includes the following steps: Configure an identifier for clinical semantic attributes for each data point in a standardized medical multimodal dataset; wherein the identifier includes the sample collection source, collection time, clinical scenario label, and data association priority; Clinical semantics are extracted from standardized text modal feature data as semantic anchors. Visual features of clinical semantics are retrieved from standardized image modal feature data. Visual semantics are extracted from standardized image modal feature data as feature anchors. Corresponding descriptive information is matched in standardized text modal feature data. Based on the association results of standardized text modal feature data and standardized image modal feature data, monitoring indicator data related to clinical scenarios are retrieved from standardized sensor modal feature data, thus forming a three-dimensional association chain. Generate an intermodal association index based on the identifier and the three-dimensional association chain.

[0008] Preferably, the standardized medical multimodal dataset is decomposed into knowledge units using the modality parsing module of the large model agent to obtain a medical single-modal knowledge unit set, specifically including the following steps: The modality parsing module of the large model agent captures the clinical diagnosis and treatment intentions corresponding to each data in the standardized medical multimodal dataset; Based on clinical diagnosis and treatment intentions, the standardized medical multimodal dataset is divided into different clinical intention clusters. Each clinical intention cluster is further divided into text sub-modules, image sub-modules, and sensor sub-modules according to modality type. Based on the text submodule, image submodule, and sensor submodule, the corresponding standardized medical multimodal dataset is processed to obtain text knowledge fragments, image feature knowledge fragments, and indicator change knowledge fragments. By calling a pre-defined clinical intent rule base in the medical field through a large model agent, text knowledge fragments, image feature knowledge fragments, and indicator change knowledge fragments are verified. The corresponding knowledge fragments that pass the verification are classified and integrated according to modality type, clinical diagnosis and treatment intent category, and supporting attributes. After configuring clinical intent association tags and data traceability identifiers for each knowledge fragment, a medical single-modal knowledge unit set is formed. The medical single-modal knowledge unit set includes a subset of text single-modal knowledge units, a subset of image single-modal knowledge units, and a subset of sensor single-modal knowledge units.

[0009] Preferably, the text submodule, image submodule, and sensing submodule process the corresponding standardized medical multimodal dataset to obtain text knowledge fragments, image feature knowledge fragments, and indicator change knowledge fragments, specifically including the following steps: For the text sub-module, key paragraphs of standardized text modal feature data are identified based on the needs of corresponding clinical intent, and text knowledge fragments are extracted based on the key paragraphs. For the imaging sub-module, the core observation object associated with the corresponding clinical intent is the focus, and standardized image modal feature data is extracted from different visual analysis dimensions to obtain image feature knowledge fragments; For the sensing submodule, the corresponding key physiological monitoring time intervals related to clinical intent are matched, and standardized sensing modality feature data are dynamically extracted from the time intervals according to clinical intent requirements to obtain knowledge fragments of indicator changes.

[0010] Preferably, a medical knowledge verification indicator system is established, specifically including the following steps: Establish a multi-dimensional indicator framework that includes knowledge effectiveness, clinical suitability, semantic relevance, and dynamic adaptability; Based on the types of clinical diagnosis and treatment scenarios, the indicators in the multi-dimensional indicator framework are further refined into scenario-specific indicator systems. A medical knowledge verification indicator system is obtained by dynamically evaluating each indicator in the scenario-specific indicator system.

[0011] Preferably, after verifying and evaluating the preliminary medical multimodal knowledge fusion results based on the indicator system, a knowledge fusion verification result set is obtained, which specifically includes the following steps: Based on the preliminary results of medical multimodal knowledge fusion, the corresponding clinical diagnosis and treatment scenario types are matched with the relevant scenario-specific detailed indicators in the indicator system; The preliminary medical multimodal knowledge fusion results were verified through multi-dimensional linkage based on scenario-specific and refined indicators. During the multi-dimensional linkage verification process, the verification data is organized to form a knowledge fusion verification result set.

[0012] Preferably, the identification of knowledge conflict points and fusion vulnerabilities in the preliminary medical multimodal knowledge fusion results based on the knowledge fusion verification result set specifically includes the following steps: Construct an identification framework that includes knowledge conflict types, vulnerability dimensions, and clinical impact correlations; Based on the recognition framework, the verification data of the knowledge fusion verification result set is subjected to multi-dimensional linkage recognition. The conflict and vulnerability information of the multi-dimensional linkage identification process is integrated and classified and labeled according to conflict type, vulnerability dimension and clinical impact to form a structured identification result set.

[0013] Preferably, the parameters of the fusion model are iteratively optimized through the feedback adjustment module of the large model agent until a medical multimodal interactive knowledge fusion result that meets the verification index requirements is obtained. Specifically, this includes the following steps: Establish a parameter regulation framework that includes parameter correlation mapping, regulation priority classification, and clinical impact assessment; Phased iterative parameter optimization is performed based on the conflict and vulnerability information of the parameter tuning framework and knowledge fusion verification result set. After applying the phased parameter iteration optimization fusion model to the standardized medical multimodal dataset, the process continues until a medical multimodal knowledge fusion result that meets the validation criteria is generated.

[0014] Compared with the prior art, the present invention has the following beneficial effects: This invention acquires multimodal data in the medical field and generates a standardized medical multimodal dataset based on medical knowledge annotation rules. Simultaneously, it establishes an intermodal association index, effectively addressing the issues of inconsistent data formats and unclear intermodal relationships in medical multimodal data. This lays a solid and standardized data foundation for subsequent knowledge fusion. Utilizing the modality parsing module of a large-scale intelligent model, knowledge units are decomposed to capture the clinical treatment intentions corresponding to each data point. Multimodal data is divided into different clinical intention clusters and corresponding sub-modules, resulting in various knowledge fragments. Through verification and classification, these fragments are integrated to form a medical single-modal knowledge unit set, improving the accuracy and effectiveness of knowledge extraction. This ensures that each knowledge unit closely aligns with clinical treatment needs. After inputting the single-modal knowledge unit set and cross-modal knowledge association set into the fusion model to obtain preliminary results, a multi-dimensional medical knowledge verification index system is established to comprehensively verify and evaluate the preliminary results, identifying knowledge conflicts and fusion vulnerabilities. The feedback adjustment module of the large-scale intelligent model iteratively optimizes the fusion model parameters until results meeting the verification indicators are obtained. This continuously improves the accuracy and reliability of medical multimodal knowledge fusion, providing knowledge support for clinical diagnosis and treatment and promoting the advancement of medical standards. Attached Figure Description

[0015] Figure 1 This is a schematic diagram of the medical multimodal knowledge fusion method driven by a large-model intelligent agent proposed in this invention; Figure 2 This diagram illustrates the steps involved in obtaining the intermodal association index using a large-model intelligent agent-driven medical multimodal knowledge fusion method proposed in this invention. Detailed Implementation

[0016] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

[0017] Many specific details are set forth in the following description in order to provide a full understanding of the invention. However, the invention may also be practiced in other ways different from those described herein, and those skilled in the art can make similar extensions without departing from the spirit of the invention. Therefore, the invention is not limited to the specific embodiments disclosed below.

[0018] Secondly, the term "an embodiment" or "embodiment" as used herein refers to a specific feature, structure, or characteristic that may be included in at least one implementation of the present invention. The phrase "in one embodiment" appearing in different places throughout this specification does not necessarily refer to the same embodiment, nor is it a single embodiment or an embodiment selectively excluded from other embodiments.

[0019] Reference Figures 1-2 As shown.

[0020] The embodiments further illustrate the medical multimodal knowledge fusion method driven by a large model intelligent agent proposed in this invention.

[0021] A medical multimodal knowledge fusion method driven by a large-model intelligent agent, comprising the following steps: Acquire multimodal data in the medical field, as well as the medical knowledge annotation rules and cross-modal association mapping relationships corresponding to each modality of data; A standardized medical multimodal dataset is obtained by processing multimodal data based on medical knowledge annotation rules; an intermodal association index for the standardized medical multimodal dataset is established based on cross-modal association mapping relationships; The standardized medical multimodal dataset is decomposed into knowledge units using the modality parsing module of the large model agent to obtain a medical single-modal knowledge unit set; Based on the intermodal association index, the medical single-modal knowledge unit set is associated and matched to obtain the medical cross-modal knowledge association set; The medical single-modal knowledge unit set and the medical cross-modal knowledge association set are input into the fusion model to obtain the preliminary medical multimodal knowledge fusion results; A medical knowledge verification index system is set up, and the knowledge fusion verification result set is obtained after verifying and evaluating the preliminary medical multimodal knowledge fusion results based on the index system. After identifying knowledge conflict points and fusion vulnerabilities in the preliminary medical multimodal knowledge fusion results based on the knowledge fusion verification result set, a structured identification result set is formed. The parameters of the fusion model are iteratively optimized through the feedback adjustment module of the large model agent until a medical multimodal interactive knowledge fusion result that meets the verification index requirements is obtained.

[0022] Multimodal data includes text modal feature data, image modal feature data, and sensor modal feature data.

[0023] Textual modal data comprises patient medical records, which contain textual information such as descriptions of symptoms, medical history, and medical orders. Imaging modal data includes medical images such as CT and MRI scans, which can present visual features such as the morphology and structure of internal organs and tissues. For example, lung CT images can show visual signs of inflammation, tumors, or other lesions in the lungs. Sensor modal data covers monitoring data of the patient's physiological indicators, such as changes in heart rate, blood pressure, and blood oxygen saturation at different time points. These data reflect the dynamic state of the patient's physiological condition.

[0024] A standardized medical multimodal dataset is obtained by processing multimodal data based on medical knowledge annotation rules, specifically including the following steps: Text modal feature data, image modal feature data, and sensor modal feature data are mapped to the feature space through linear transformation. During the mapping process, the corresponding modal weight coefficients are obtained according to the contribution of each modal data to the fusion of medical knowledge. The text modal feature data, image modal feature data, and sensor modal feature data are adjusted and processed according to modal weight coefficients to obtain a standardized medical multimodal dataset; wherein, the standardized medical multimodal dataset includes standardized text modal feature data, standardized image modal feature data, and standardized sensor modal feature data.

[0025] First, a linear transformation is performed on the text modal feature data, image modal feature data, and sensor modal feature data to map them into a common feature space. For example, text data consists of symptom descriptions and diagnostic conclusions from medical records; image data can be the visual content of organ morphology and lesion areas presented by CT and MRI medical images; and sensor data consists of monitored values ​​of patients' heart rate and blood pressure. In this mapping process, the corresponding modal weight coefficients are determined based on the contribution of different modal data to the fusion of medical knowledge. For the diagnosis of cardiovascular diseases, echocardiography images in the image modality can visually display the structure and function of the heart, contributing significantly to the diagnosis, so its weight coefficient is higher; while the descriptions of palpitations in the text modality contribute less, resulting in a lower weight coefficient. The text modal feature data, image modal feature data, and sensor modal feature data are then adjusted based on these modal weight coefficients. The final standardized medical multimodal dataset obtained after this processing includes standardized text modal feature data, standardized image modal feature data, and standardized sensor modal feature data.

[0026] The process of establishing a standardized intermodal association index for a medical multimodal dataset based on cross-modal association mapping includes the following steps: Configure clinical semantic attribute identifiers for each data point in the standardized medical multimodal dataset; the identifiers include the sample collection source, collection time, clinical scenario label, and data association priority; Clinical semantics are extracted from standardized text modal feature data as semantic anchors. Visual features of clinical semantics are retrieved from standardized image modal feature data. Visual semantics are extracted from standardized image modal feature data as feature anchors. Corresponding descriptive information is matched in standardized text modal feature data. Based on the association results of standardized text modal feature data and standardized image modal feature data, monitoring indicator data related to clinical scenarios are retrieved from standardized sensor modal feature data, thus forming a three-dimensional association chain. Generate an intermodal association index based on the identifier and the three-dimensional association chain.

[0027] When establishing an intermodal association index for a standardized medical multimodal dataset, the first step is to configure a clinical semantic attribute identifier for each data point in the dataset. This identifier includes the sample collection source, collection time, clinical scenario label, and data association priority. For example, a patient data set from the cardiology department of a top-tier hospital, collected at 9:00 AM, and targeting the coronary heart disease treatment scenario, would have its data association priority set to a higher level based on clinical needs.

[0028] Clinical semantics are extracted from standardized text modal feature data as semantic anchors, such as extracting the clinical semantics of chest pain lasting 30 minutes from a patient's medical record. Then, visual features corresponding to this semantics are retrieved from standardized image modal feature data, such as searching for visual manifestations of myocardial ischemia in cardiac CT images. Visual semantics are extracted from image modal feature data as feature anchors, such as extracting the visual semantics of decreased myocardial density in the anterior wall of the left ventricle from images showing myocardial ischemia, and matching the corresponding descriptive information in standardized text modal feature data to check for relevant textual records in the medical record. Based on the association results of text and image modal feature data, monitoring indicator data related to the clinical scenario are retrieved from standardized sensor modal feature data, such as retrieving sensor monitoring data of the patient's electrocardiogram and myocardial enzymes in the context of coronary heart disease diagnosis and treatment. This forms a three-dimensional association chain of text, images, and sensors.

[0029] Based on the identifiers and the resulting three-dimensional association chains, an intermodal association index is generated, enabling data from different modalities to establish close and orderly associations based on the index, facilitating subsequent operations such as the fusion of multimodal medical knowledge.

[0030] The standardized medical multimodal dataset is decomposed into knowledge units using the modality parsing module of the large model agent to obtain a medical single-modal knowledge unit set. The specific steps include: The modality parsing module of the large model agent captures the clinical diagnosis and treatment intentions corresponding to each data in the standardized medical multimodal dataset; Based on clinical diagnosis and treatment intentions, the standardized medical multimodal dataset is divided into different clinical intention clusters. Each clinical intention cluster is further divided into text sub-modules, image sub-modules, and sensor sub-modules according to modality type. Based on the text submodule, image submodule, and sensor submodule, the corresponding standardized medical multimodal dataset is processed to obtain text knowledge fragments, image feature knowledge fragments, and indicator change knowledge fragments. By calling a pre-defined clinical intent rule base in the medical field through a large model agent, text knowledge fragments, image feature knowledge fragments, and indicator change knowledge fragments are validated. The corresponding knowledge fragments that pass the validation are classified and integrated according to modality type, clinical diagnosis and treatment intent category, and supporting attributes. After configuring clinical intent association tags and data traceability identifiers for each knowledge fragment, a medical monomodal knowledge unit set is formed. The medical monomodal knowledge unit set includes a subset of text monomodal knowledge units, a subset of image monomodal knowledge units, and a subset of sensor monomodal knowledge units.

[0031] When decomposing a standardized multimodal medical dataset into knowledge units to obtain a single-modal medical knowledge unit set, the modality parsing module of a large model agent is first used to capture the clinical diagnostic intent corresponding to each data point in the dataset. For example, in a multimodal dataset containing a patient's chest pain symptom description, chest CT images, and heart rate monitoring data, the modality parsing module identifies that the clinical diagnostic intent may be to screen for coronary heart disease.

[0032] Based on clinical diagnostic and treatment intentions, the standardized medical multimodal dataset is divided into different clinical intention clusters. Each clinical intention cluster is further subdivided into text submodules, image submodules, and sensor submodules according to modality type. Taking the intention cluster for screening coronary heart disease as an example, the text submodule corresponds to the textual records of chest pain in the medical record, the image submodule corresponds to chest CT images, and the sensor submodule corresponds to the monitoring data of heart rate and physiological indicators.

[0033] Based on the text, image, and sensor submodules, the corresponding standardized medical multimodal datasets are processed to obtain different knowledge fragments. The text submodule identifies key paragraphs corresponding to clinical intent needs, such as paragraphs in medical records describing the time and frequency of chest pain attacks, and extracts information from these paragraphs to obtain textual knowledge fragments. The image submodule focuses on the core observation object, extracting information from chest CT images from different visual resolution dimensions to obtain image feature knowledge fragments. The sensor submodule matches key physiological monitoring time intervals and dynamically extracts heart rate data within those intervals to obtain knowledge fragments on indicator changes.

[0034] The large-scale intelligent agent calls a pre-defined clinical intent rule base in the medical field to validate these knowledge fragments. For example, it validates whether the description of chest pain in the text knowledge fragment matches the common expressions of coronary heart disease symptoms, whether the changes in heart structure in the imaging feature knowledge fragment are consistent with the pathological manifestations of coronary heart disease, and whether the changes in heart rate in the indicator change knowledge fragment are within the typical range of a coronary heart disease attack. The validated knowledge fragments are classified and integrated according to modality type, clinical diagnosis and treatment intent category, and supporting attributes, and each knowledge fragment is assigned a clinical intent association tag and a data traceability identifier, ultimately forming a medical monomodal knowledge unit set, including a subset of text monomodal knowledge units, a subset of imaging monomodal knowledge units, and a subset of sensor monomodal knowledge units.

[0035] The text, image, and sensor submodules process the corresponding standardized medical multimodal datasets to obtain text knowledge fragments, image feature knowledge fragments, and indicator change knowledge fragments. Specifically, this includes the following steps: For the text sub-module, key paragraphs of standardized text modal feature data are identified based on the needs of corresponding clinical intent, and text knowledge fragments are extracted based on the key paragraphs. For the imaging sub-module, the core observation object associated with the corresponding clinical intent is the focus, and standardized image modal feature data is extracted from different visual analysis dimensions to obtain image feature knowledge fragments; For the sensing submodule, the corresponding key physiological monitoring time intervals related to clinical intent are matched, and standardized sensing modality feature data are dynamically extracted from the time intervals according to clinical intent requirements to obtain knowledge fragments of indicator changes.

[0036] Different sub-modules operate differently when extracting knowledge fragments from standardized medical multimodal datasets. For the text sub-module, key paragraphs in the standardized text modality feature data are identified based on the corresponding clinical intent. Text information is then extracted from these key paragraphs to obtain textual knowledge fragments. For example, when the clinical intent is to diagnose pneumonia, the text sub-module will identify paragraphs in the patient's medical record describing symptoms such as cough and fever, as well as a history of lung disease, and extract the content of these paragraphs to form textual knowledge fragments.

[0037] The imaging submodule focuses on the core observation object associated with the corresponding clinical intent, extracting standardized image modal feature data from different visual resolution dimensions to obtain image feature knowledge fragments. If the clinical intent is to examine a brain tumor, the core observation object is the brain. The imaging submodule will extract from different visual resolution dimensions such as tissue density, tumor morphology, and mass effect of brain CT images to obtain image feature knowledge fragments such as a round high-density shadow on the left side of the brain with unclear boundaries and edema in the surrounding brain tissue.

[0038] The sensing submodule needs to match key physiological monitoring time intervals related to the corresponding clinical intent, and then dynamically extract standardized sensing modality feature data from this time interval according to the needs of the clinical intent, thereby obtaining knowledge fragments of indicator changes. For example, if the clinical intent is to monitor the changes in the condition of a patient with heart failure, and the key physiological monitoring time interval is one hour after the patient's activity, the sensing submodule dynamically extracts data such as heart rate, blood pressure, and blood oxygen saturation from this time period to form knowledge fragments of indicator changes.

[0039] The steps involved in setting up an indicator system for verifying medical knowledge are as follows: Establish a multi-dimensional indicator framework that includes knowledge effectiveness, clinical suitability, semantic relevance, and dynamic adaptability; Based on the types of clinical diagnosis and treatment scenarios, the indicators in the multi-dimensional indicator framework are further refined into scenario-specific indicator systems. A medical knowledge verification indicator system is obtained by dynamically evaluating each indicator in the scenario-specific indicator system.

[0040] When setting up an indicator system for medical knowledge verification, the first step is to establish a multi-dimensional indicator framework, which includes knowledge effectiveness, clinical suitability, semantic relevance, and dynamic adaptability. Knowledge effectiveness focuses on whether the integrated knowledge is accurate and reliable; for example, knowledge about diagnosing pneumonia should correctly distinguish the characteristics of bacterial and viral pneumonia. Clinical suitability examines whether the knowledge aligns with actual clinical diagnostic and treatment procedures; for example, in emergency situations, can the knowledge quickly assist doctors in making preliminary judgments? Semantic relevance requires that the semantics of different modalities of knowledge be interconnected; for example, the manifestations of lung inflammation in imaging should semantically match the symptom descriptions in text. Dynamic adaptability refers to the ability of knowledge to adapt to the dynamic changes in a patient's condition; for example, as treatment progresses, the knowledge should reflect the improvement or deterioration of the condition.

[0041] Based on the types of clinical diagnosis and treatment scenarios, the indicators in the multi-dimensional indicator framework are further refined into scenario-specific indicator systems. Taking emergency and chronic disease management scenarios as examples, in emergency scenarios, the indicators of knowledge effectiveness focus more on the ability to quickly identify critical and severe illnesses, such as whether it is possible to determine whether a patient has severe respiratory failure from multimodal data in a short time; clinical suitability emphasizes the support of knowledge for rapid decision-making in emergency situations, such as whether it can provide a basis for emergency examinations or treatments. In chronic disease management scenarios, knowledge effectiveness focuses more on the accurate judgment of the long-term development trend of chronic diseases, such as predicting the occurrence of complications in diabetic patients through multimodal data; clinical suitability focuses on the guidance of knowledge for adjusting long-term treatment plans, such as adjusting medication based on changes in the patient's imaging and sensor data.

[0042] The medical knowledge verification indicator system is obtained by dynamically evaluating each indicator in the scenario-specific indicator system. This means that it will be adjusted according to new situations and needs in clinical practice. For example, as medical technology develops and new imaging diagnostic technologies emerge, the evaluation indicators for the effectiveness of imaging modal knowledge will be updated accordingly to adapt to new diagnostic methods, ensuring that the entire medical knowledge verification indicator system can continuously and effectively verify the results of medical multimodal knowledge fusion.

[0043] After verifying and evaluating the preliminary medical multimodal knowledge fusion results based on the indicator system, a knowledge fusion verification result set is obtained, which specifically includes the following steps: Based on the preliminary results of medical multimodal knowledge fusion, the corresponding clinical diagnosis and treatment scenario types are matched with the relevant scenario-specific detailed indicators in the indicator system; The preliminary medical multimodal knowledge fusion results were verified through multi-dimensional linkage based on scenario-specific and refined indicators. During the multi-dimensional linkage verification process, the verification data is organized to form a knowledge fusion verification result set.

[0044] When verifying and evaluating the preliminary medical multimodal knowledge fusion results, the first step is to match the corresponding scenario-specific detailed indicators in the indicator system based on the clinical diagnosis and treatment scenario type corresponding to the results. If the preliminary fusion results correspond to the diagnosis and treatment scenario of acute myocardial infarction, then scenario-specific detailed indicators for acute myocardial infarction will be matched, such as the assessment indicators of changes in myocardial enzyme spectrum and the judgment indicators of electrocardiogram characteristics.

[0045] Based on these scenario-specific and detailed indicators, the preliminary medical multimodal knowledge fusion results were validated through multi-dimensional linkage. Taking the acute myocardial infarction scenario as an example, the validity of the fused knowledge was verified from the perspective of knowledge effectiveness, to determine whether it could accurately identify the typical symptoms and signs of myocardial infarction; the clinical adaptability was assessed to determine whether the knowledge conformed to the emergency treatment process for acute myocardial infarction; the semantic relevance was examined to check whether the description of chest pain in the text, the pathological manifestations of the heart in the imaging, and the changes in data rate and blood pressure of the sensor data center were semantically related and consistent; and the dynamic adaptability was assessed to evaluate whether the knowledge could adapt to the dynamic changes in the patient's electrocardiogram and enzyme indicators during the emergency treatment process.

[0046] During the multi-dimensional linkage verification process, the data generated during the verification process are organized, such as the scores of various indicators and the matching of different modal knowledge in the verification. These data are integrated to form a knowledge fusion verification result set, so that knowledge conflict points and fusion vulnerabilities in the preliminary fusion results can be identified based on this result set.

[0047] Identify knowledge conflicts and vulnerabilities in the preliminary medical multimodal knowledge fusion results based on the knowledge fusion verification result set, specifically including the following steps: Construct an identification framework that includes knowledge conflict types, vulnerability dimensions, and clinical impact correlations; Based on the recognition framework, the verification data of the knowledge fusion verification result set is subjected to multi-dimensional linkage recognition. The conflict and vulnerability information of the multi-dimensional linkage identification process is integrated and classified and labeled according to conflict type, vulnerability dimension and clinical impact to form a structured identification result set.

[0048] When identifying knowledge conflicts and vulnerabilities in the preliminary medical multimodal knowledge fusion results, an identification framework is first constructed. This framework includes knowledge conflict types, vulnerability dimensions, and clinical impact correlations. For example, knowledge conflict types refer to contradictions between different modalities of knowledge, such as a text stating that a patient has a bacterial infection, while the features shown in the images are more consistent with a viral infection. Vulnerability dimensions involve the completeness of knowledge, such as a lack of diagnostic and treatment knowledge for a specific type of case, or an incorrect description of the diagnostic criteria for a certain disease. Clinical impact correlations refer to the potential impact of these conflicts or vulnerabilities on clinical diagnosis and treatment, such as whether they will lead to misdiagnosis or delayed treatment.

[0049] Based on the recognition framework, the verification data in the knowledge fusion verification result set is subjected to multi-dimensional linkage recognition. If the verification data contains text-modal bacterial infection diagnosis knowledge, image-modal viral infection feature knowledge, and sensor-modal inflammatory index data, the contradictions between text and image knowledge are identified from the knowledge conflict type dimension; the existence of missing knowledge about the diagnosis and treatment of different stages of such infectious diseases is checked from the vulnerability dimension; and the impact of such conflicts and vulnerabilities on the patient's treatment plan selection is determined by combining clinical impact correlation.

[0050] The conflict and vulnerability information discovered during the multi-dimensional linkage identification process is integrated and categorized and labeled according to conflict type, vulnerability dimension, and clinical impact to form a structured identification result set. This structured result set clearly reveals the problems existing in the preliminary fusion results, providing a basis for subsequent optimization of the fusion model.

[0051] The parameters of the fusion model are iteratively optimized through the feedback adjustment module of the large model agent until the medical multimodal interactive knowledge fusion result that meets the validation index requirements is obtained. The specific steps include: Establish a parameter regulation framework that includes parameter correlation mapping, regulation priority classification, and clinical impact assessment; Phased iterative parameter optimization is performed based on the conflict and vulnerability information of the parameter tuning framework and knowledge fusion verification result set. After applying the phased parameter iteration optimization fusion model to the standardized medical multimodal dataset, the process continues until a medical multimodal knowledge fusion result that meets the validation criteria is generated.

[0052] When iteratively optimizing the parameters of the fusion model to obtain medical multimodal knowledge fusion results that meet the validation criteria, the first step is to build a parameter adjustment framework, which includes parameter association mapping, adjustment priority division, and clinical impact assessment.

[0053] The parameter tuning framework implements mapping, partitioning, and iteration. Taking the tuning of parameters related to image and text modalities as an example, firstly, a parameter association mapping map is constructed to clarify the association paths between image feature extraction parameters and text semantic parsing parameters. For example, there is a strong correlation between the lesion edge clarity parameter in CT images and the lesion morphology description parameter in medical record text, and the association weight can be quantified. Next, the tuning priority is divided, with core parameters that directly affect the diagnostic conclusion (such as image texture analysis parameters for determining the benignity or malignancy of tumors) assigned high priority, and auxiliary descriptive parameters assigned low priority. In the iterative optimization phase, a non-global parameter fine-tuning strategy based on conflict feedback is adopted. For example, when verification reveals a conflict between the image feature knowledge and the pathology report text knowledge of a lung cancer patient, fine-tuning is only performed on the nodule boundary recognition sub-parameter in the image segmentation algorithm and the pathology description keyword extraction sub-parameter in the text parsing. Through five rounds of such targeted iterations, the resolution rate of this type of conflict is increased from the initial 60% to 90%, while avoiding blind adjustments to global parameters. This ensures that each parameter tuning affects key aspects of clinical diagnosis and treatment, making the entire parameter tuning process clear and traceable from algorithm to effect.

[0054] For example, parameter association mapping can clarify the relationship between different parameters in the fusion model; adjustment priority division determines the order of adjustment based on the importance of parameters to the fusion results and their clinical impact. If a change in a parameter will directly lead to serious clinical misjudgment, its adjustment priority will be very high; clinical impact assessment evaluates whether the impact of parameter adjustment on clinical diagnosis and treatment is positive or negative.

[0055] Originally, the dietary recommendations module in the model's diabetes diagnosis and treatment knowledge output only matched 65% of the patients' individual dietary preferences and metabolic indicators. After we specifically adjusted the parameters generated in the dietary recommendations output branch, a follow-up evaluation of 100 diabetic patients revealed that the consistency between the dietary plan based on multimodal knowledge fusion in the diagnostic process and the patients' actual metabolic status improved to 82%. Following this plan, the glycemic control achievement rate increased from 70% to 85%, and the incidence of hypoglycemic events due to improper diet decreased by 40%. This quantitative example clearly demonstrates the positive impact of this parameter adjustment on clinical diagnosis and treatment. It improves the quality of diabetes chronic disease management through more accurate knowledge output, helps patients achieve better treatment outcomes, and provides clinicians with more reliable knowledge support for developing personalized treatment plans.

[0056] Based on the parameter tuning framework and the conflict and vulnerability information in the knowledge fusion verification result set, phased iterative parameter optimization is performed. If the verification results show a conflict of inaccurate association between text and image knowledge, and parameter association mapping reveals that this is caused by unreasonable image feature matching parameter settings, and this parameter has a high priority for adjustment according to the adjustment priority classification, and the clinical impact assessment shows that the adjustment can effectively improve the accuracy of knowledge fusion and reduce the possibility of clinical misdiagnosis, then the image feature matching parameter will be adjusted first. After optimizing this parameter in the first stage, the process moves to the next stage to check other potentially affected parameters or other types of conflicts and vulnerabilities, and continues parameter optimization.

[0057] The fusion model, optimized through phased parameter iterations, is applied to a standardized medical multimodal dataset. This process is repeated until a medical multimodal knowledge fusion result that meets the validation criteria is generated. For example, after multiple parameter optimizations, the fusion model meets the requirements for knowledge effectiveness and clinical suitability validation metrics when processing standardized data, and the resulting knowledge fusion result is then satisfactory.

[0058] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0059] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0060] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A medical multimodal knowledge fusion method driven by a large-scale intelligent agent, characterized in that, The method includes the following steps: Acquire multimodal data in the medical field, as well as the medical knowledge annotation rules and cross-modal association mapping relationships corresponding to each modality of data; A standardized medical multimodal dataset is obtained by processing multimodal data based on medical knowledge annotation rules; an intermodal association index for the standardized medical multimodal dataset is established based on cross-modal association mapping relationships; The standardized medical multimodal dataset is decomposed into knowledge units using the modality parsing module of the large model agent to obtain a medical single-modal knowledge unit set; Based on the intermodal association index, the medical single-modal knowledge unit set is associated and matched to obtain the medical cross-modal knowledge association set; The medical single-modal knowledge unit set and the medical cross-modal knowledge association set are input into the fusion model to obtain the preliminary medical multimodal knowledge fusion results; A medical knowledge verification index system is set up, and the knowledge fusion verification result set is obtained after verifying and evaluating the preliminary medical multimodal knowledge fusion results based on the index system. After identifying knowledge conflict points and fusion vulnerabilities in the preliminary medical multimodal knowledge fusion results based on the knowledge fusion verification result set, a structured identification result set is formed. The parameters of the fusion model are iteratively optimized through the feedback adjustment module of the large model agent until a medical multimodal interactive knowledge fusion result that meets the verification index requirements is obtained.

2. The medical multimodal knowledge fusion method driven by a large-scale intelligent agent according to claim 1, characterized in that, The multimodal data includes text modal feature data, image modal feature data, and sensor modal feature data.

3. The medical multimodal knowledge fusion method driven by a large-scale intelligent agent according to claim 2, characterized in that, A standardized medical multimodal dataset is obtained by processing multimodal data based on medical knowledge annotation rules, specifically including the following steps: Text modal feature data, image modal feature data, and sensor modal feature data are mapped to the feature space through linear transformation. During the mapping process, the corresponding modal weight coefficients are obtained according to the contribution of each modal data to the fusion of medical knowledge. The text modal feature data, image modal feature data, and sensor modal feature data are adjusted and processed according to modal weight coefficients to obtain a standardized medical multimodal dataset; wherein, the standardized medical multimodal dataset includes standardized text modal feature data, standardized image modal feature data, and standardized sensor modal feature data.

4. The medical multimodal knowledge fusion method driven by a large-model intelligent agent according to claim 3, characterized in that, The process of establishing a standardized intermodal association index for a medical multimodal dataset based on cross-modal association mapping includes the following steps: Configure an identifier for clinical semantic attributes for each data point in a standardized medical multimodal dataset; wherein the identifier includes the sample collection source, collection time, clinical scenario label, and data association priority; Clinical semantics are extracted from standardized text modal feature data as semantic anchors. Visual features of clinical semantics are retrieved from standardized image modal feature data. Visual semantics are extracted from standardized image modal feature data as feature anchors. Corresponding descriptive information is matched in standardized text modal feature data. Based on the association results of standardized text modal feature data and standardized image modal feature data, monitoring indicator data related to clinical scenarios are retrieved from standardized sensor modal feature data, thus forming a three-dimensional association chain. Generate an intermodal association index based on the identifier and the three-dimensional association chain.

5. The medical multimodal knowledge fusion method driven by a large-model intelligent agent according to claim 4, characterized in that, The standardized medical multimodal dataset is decomposed into knowledge units using the modality parsing module of the large model agent to obtain a medical single-modal knowledge unit set. The specific steps include: The modality parsing module of the large model agent captures the clinical diagnosis and treatment intentions corresponding to each data in the standardized medical multimodal dataset; Based on clinical diagnosis and treatment intentions, the standardized medical multimodal dataset is divided into different clinical intention clusters. Each clinical intention cluster is further divided into text sub-modules, image sub-modules, and sensor sub-modules according to modality type. Based on the text submodule, image submodule, and sensor submodule, the corresponding standardized medical multimodal dataset is processed to obtain text knowledge fragments, image feature knowledge fragments, and indicator change knowledge fragments. By calling a pre-defined clinical intent rule base in the medical field through a large model agent, text knowledge fragments, image feature knowledge fragments, and indicator change knowledge fragments are verified. The corresponding knowledge fragments that pass the verification are classified and integrated according to modality type, clinical diagnosis and treatment intent category, and supporting attributes. After configuring clinical intent association tags and data traceability identifiers for each knowledge fragment, a medical single-modal knowledge unit set is formed. The medical single-modal knowledge unit set includes a subset of text single-modal knowledge units, a subset of image single-modal knowledge units, and a subset of sensor single-modal knowledge units.

6. The medical multimodal knowledge fusion method driven by a large-model intelligent agent according to claim 5, characterized in that, The text, image, and sensor submodules process the corresponding standardized medical multimodal datasets to obtain text knowledge fragments, image feature knowledge fragments, and indicator change knowledge fragments. Specifically, this includes the following steps: For the text sub-module, key paragraphs of standardized text modal feature data are identified based on the needs of corresponding clinical intent, and text knowledge fragments are extracted based on the key paragraphs. For the imaging sub-module, the core observation object associated with the corresponding clinical intent is the focus, and standardized image modal feature data is extracted from different visual analysis dimensions to obtain image feature knowledge fragments; For the sensing submodule, the corresponding key physiological monitoring time intervals related to clinical intent are matched, and standardized sensing modality feature data are dynamically extracted from the time intervals according to clinical intent requirements to obtain knowledge fragments of indicator changes.

7. The medical multimodal knowledge fusion method driven by a large-model intelligent agent according to claim 1, characterized in that, The steps involved in setting up an indicator system for verifying medical knowledge are as follows: Establish a multi-dimensional indicator framework that includes knowledge effectiveness, clinical suitability, semantic relevance, and dynamic adaptability; Based on the types of clinical diagnosis and treatment scenarios, the indicators in the multi-dimensional indicator framework are further refined into scenario-specific indicator systems. A medical knowledge verification indicator system is obtained by dynamically evaluating each indicator in the scenario-specific indicator system.

8. The medical multimodal knowledge fusion method driven by a large-scale intelligent agent according to claim 6, characterized in that, After verifying and evaluating the preliminary medical multimodal knowledge fusion results based on the indicator system, a knowledge fusion verification result set is obtained, which specifically includes the following steps: Based on the preliminary results of medical multimodal knowledge fusion, the corresponding clinical diagnosis and treatment scenario types are matched with the relevant scenario-specific detailed indicators in the indicator system; The preliminary medical multimodal knowledge fusion results were verified through multi-dimensional linkage based on scenario-specific and refined indicators. During the multi-dimensional linkage verification process, the verification data is organized to form a knowledge fusion verification result set.

9. The medical multimodal knowledge fusion method driven by a large-scale intelligent agent according to claim 8, characterized in that, Identify knowledge conflicts and vulnerabilities in the preliminary medical multimodal knowledge fusion results based on the knowledge fusion verification result set, specifically including the following steps: Construct an identification framework that includes knowledge conflict types, vulnerability dimensions, and clinical impact correlations; Based on the recognition framework, the verification data of the knowledge fusion verification result set is subjected to multi-dimensional linkage recognition. The conflict and vulnerability information of the multi-dimensional linkage identification process is integrated and classified and labeled according to conflict type, vulnerability dimension and clinical impact to form a structured identification result set.

10. The medical multimodal knowledge fusion method driven by a large-model intelligent agent according to claim 9, characterized in that, The parameters of the fusion model are iteratively optimized through the feedback adjustment module of the large model agent until the medical multimodal interactive knowledge fusion result that meets the validation index requirements is obtained. The specific steps include: Establish a parameter regulation framework that includes parameter correlation mapping, regulation priority classification, and clinical impact assessment; Phased iterative parameter optimization is performed based on the conflict and vulnerability information of the parameter tuning framework and knowledge fusion verification result set. After applying the phased parameter iteration optimization fusion model to the standardized medical multimodal dataset, the process continues until a medical multimodal knowledge fusion result that meets the validation criteria is generated.

Images