A meta-system for developing medical multimodal data knowledge graphs
By establishing a meta-system for developing a medical multimodal data knowledge graph, the problem of the lack of standardization in multimodal data knowledge graphs was solved, the accuracy and consistency of multimodal data were achieved, and the accuracy of disease diagnosis and treatment and the capabilities of AI were improved.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- GENERAL HOSPITAL OF SOUTHERN THEATRE COMMAND OF PLA
- Filing Date
- 2025-09-10
- Publication Date
- 2026-07-31
AI Technical Summary
Existing medical knowledge graphs are mainly unimodal, lacking standards and guidelines for constructing multimodal medical data knowledge graphs. This results in a lack of consistency, reusability, and interoperability among multimodal data knowledge graphs developed by different institutions, leading to redundant resource consumption.
A meta-system for developing medical multimodal data knowledge graphs is established, including a medical multimodal metamodal architecture, a multimodal framework, a multimodal spectrum, and a multimodal knowledge graph module. Following international and domestic standards, machine learning and deep learning technologies are used to construct a multi-dimensional multimodal knowledge graph.
It achieves accuracy, consistency, robustness, and scalability of medical multimodal data knowledge graphs, improves the accuracy of disease diagnosis and treatment, supports the fusion and alignment of multimodal data, and enhances AI's capabilities in disease prevention, diagnosis, and rehabilitation.
Smart Images
Figure CN121144531B_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of medical artificial intelligence technology and is geared towards machine learning applications of medical multimodal data. It describes the structure, scope, rules, and elements of medical multimodal data knowledge graphs to standardize the development and construction of medical multimodal knowledge graphs. Background Technology
[0002] Modality refers to the way in which natural phenomena are perceived or expressed. Real-world objects represent their state of existence in multiple different modalities. Through various sensors, modalities are converted into data, such as electrical signals, images, text, video, and sound. By integrating this multi-source, heterogeneous data, multimodal data can more comprehensively and meticulously reveal the characteristics of things, thus providing a richer and more complete information representation.
[0003] Medical data is inherently multimodal, encompassing various data types such as medical images, electrophysiological signals, examination and test results, medical records, pathological images, and omics information. Supported by artificial intelligence (AI) technology, this multimodal data is efficiently utilized through techniques like machine learning and deep learning, providing doctors with precise decision support in diagnosing diseases, developing treatment plans, and assessing patient prognosis.
[0004] Medical knowledge graphs play a crucial role in enhancing data feature representation and semantic understanding in medical data AI applications. However, existing medical knowledge graphs are primarily unimodal, consisting of triples (entity, attribute, and relation) of medical data entities within the same modality. With the development of AI technology, especially the emergence of large-scale multimodal models, multimodal medical knowledge graphs are increasingly attracting industry attention. In the medical field, the importance of multimodal knowledge graphs is mainly reflected in their knowledge spectrum... Figure 3 On the relational elements of tuples, new fusion relationships are generated when one modal data entity interacts with another. These new features arising from the fusion of different modal data significantly enhance AI's capabilities in disease prevention, diagnosis, and rehabilitation. Currently, the application of medical multimodal data knowledge graphs has received widespread attention; however, the lack of guidelines and standards for constructing medical multimodal data knowledge graphs hinders their development and application. Summary of the Invention
[0005] To address the aforementioned challenges in developing and constructing medical multimodal data knowledge graphs, we have established a meta-system for this purpose. This meta-system describes the structure, scope, rules, and elements of medical multimodal data knowledge graphs from the perspectives of concept, framework, logic, and expression, providing guidance, standards, and constraints for their development and construction.
[0006] This system complies with the relevant standards and specifications issued by the International Organization for Standardization (ISO), the International Electrotechnical Commission (IEC), and my country's national standards (GB) and industry standards (such as the health industry standard WS).
[0007] This system does not impose specific requirements on the specific technologies, tools, and products used in the development and construction of the medical multimodal data knowledge graph.
[0008] The technical solution of the present invention is as follows: a meta-system for developing medical multimodal data knowledge graphs, comprising four modules: medical multimodal metamodal architecture (4MA), medical multimodal framework (MMF), medical multimodal spectrum (MMS), and medical multimodal knowledge graph (MMGK).
[0009] The Medical Multimodal Metamodal Architecture (4MA) module, based on the data modality ontology and conceptual level of medical multimodal machine learning, standardizes and constrains the structure, logic, and expression of medical multimodal data knowledge graphs, establishing a high-top architecture model.
[0010] The Medical Multimodal Framework (MMF) module is used to set cross-domain and interdisciplinary rules for medical multimodal data. It categorizes medical multimodal data according to disciplinary standards, generating subject types for the data. First, medical multimodal data is collected, its entity and data types are analyzed, and stored as data modality tuples. Next, the medical multimodal data is categorized and hierarchically structured based on the content of the data modality tuples. Then, according to the categorization and hierarchical structure, a medical multimodal data framework is formed and transmitted to the Medical Multimodal Spectrum module.
[0011] The Medical Multimodal Spectrum (MMS) module, based on the output of the Medical Multimodal Framework (MMF) module, connects different modal relationships, labels the types of multimodal data relationships, and forms medical multimodal spectra for different disciplines, diseases, and disease types. These spectra are then transmitted to the Medical Multimodal Knowledge Graph module. The medical multimodal spectrum reflects the modality types of medical multimodal data, including heterogeneous modalities, classificatory modalities, homogeneous modalities, and homogeneous-heterogeneous modalities.
[0012] The Medical Multimodal Knowledge Graph (MMKG) module constructs a multimodal knowledge graph (target multimodal knowledge graph) for disciplines, diseases, and disease types, based on the output of the medical multimodal framework and medical multimodal spectrum, which is required for machine learning on medical multimodal data. Within the medical multimodal spectrum framework, it uses machine learning and deep learning techniques to analyze and describe the interaction relationships of the data entities included in the target multimodal knowledge graph and establish relational connections.
[0013] Within the scope of the data entities in the target multimodal knowledge graph, the names, attributes, and interaction relationships (relationships between different modal data) of the data entities are described according to the triple rules of the knowledge graph, and the target multimodal knowledge graph is constructed.
[0014] First, based on the application scenario, determine the medical multimodal knowledge graph to be constructed (target multimodal knowledge graph).
[0015] Furthermore, within the medical domain of the Medical Multimodal Metamodal Architecture (4MA) module, ontologies, terms, disciplines, and information models related to the target multimodal knowledge graph are selected. This process determines the terminology for the multimodal data included (or to be included) in the target multimodal knowledge graph, including its identifiers, names, and definitions. For data not included in the terminology standard, attribute filling or mapping processing is required. This step ensures the scientific rigor and accuracy of the target multimodal knowledge graph.
[0016] Furthermore, within the standard domain of the Medical Multimodal Metamodal Architecture (4MA) module, the information standards to be adopted will be determined from existing international / domestic medical information-related standards for target multimodal knowledge graphs, serving as guidelines, norms, and constraints.
[0017] Furthermore, following the modeling domain conventions of the Medical Multimodal Metamodal Architecture (4MA), object-oriented and data-driven information modeling methods are employed to perform relevant modeling for the construction of the target multimodal knowledge graph. This step ensures the reusability and robustness of the target multimodal knowledge graph.
[0018] Furthermore, in the AI domain of the Medical Multimodal Metamodal Architecture (4MA) module, the proposed machine learning techniques are selected based on the characteristics and relationships of the relevant modal data of the target multimodal knowledge graph. This step ensures the accuracy and scalability of the target multimodal knowledge graph.
[0019] Different machine learning algorithms or different parameters of the same model should be considered. It is also recommended to use a multimodal large model for post-training (such as prompt words, RAG, fine-tuning, API, AGENT, etc.).
[0020] Furthermore, the entity types and data types of the multimodal data in the target multimodal knowledge graph are analyzed and stored as data modality tuples. Based on the content of the data modality tuples, the medical multimodal data is classified and layered. The medical multimodal data is classified and layered according to its type (heterogeneous modality, classificatory modality, homogeneous modality, and homogeneous-heterogeneous modality) to form a medical multimodal data framework.
[0021] Furthermore, it is used to set semantic rules for cross-domain and interdisciplinary medical multimodal data, classify medical multimodal data into disciplines according to discipline standards, and generate discipline types for medical multimodal data.
[0022] Furthermore, we will connect the relationships between different modal data, label the relationship types of multimodal data, form a multimodal data set for the target multimodal knowledge graph, and establish a modality spectrum of medical multimodal data for different disciplines, diseases, and disease types.
[0023] Furthermore, based on the multimodal spectrum of the target multimodal knowledge graph, machine learning and deep learning technologies are used to analyze and describe the interaction relationships of data entities in the target multimodal knowledge graph, and establish relationship connections.
[0024] Furthermore, following the triplet rules of knowledge graphs, the names, attributes, and interaction relationships (relationships between different modal data) of data entities are described to construct the target multimodal knowledge graph.
[0025] Furthermore, a target multimodal knowledge graph is generated and stored in a graph database.
[0026] The above approach has the following beneficial effects:
[0027] 1. Medical knowledge graphs play a crucial role in medical research, clinical applications, public health, and new drug development. Existing medical knowledge graphs primarily describe the attributes and relationships of data entities with similar patterns, similar to describing and representing related data on a single plane. With the development of AI technology, especially the increasingly widespread application of large-scale models, machine learning and deep learning have continuously enhanced their data processing capabilities. Coupled with the emergence of multimodal large-scale models, multimodal medical data has received significant attention. Compared to unimodal knowledge graphs, multimodal knowledge graphs describe and represent related data in a multidimensional space. Multimodal knowledge graphs can provide more accurate support for disease diagnosis and treatment. For example, in the diagnosis of heart disease patients, the patient's electrocardiogram (time-series signal modality) and the changes in the patient's condition recorded by medical staff (natural language document modality) can be fused and aligned using machine learning, significantly improving the accuracy of disease diagnosis. Multimodal knowledge graphs play a vital role in the fusion and alignment process of data from different modalities.
[0028] 2. The construction of a medical multimodal knowledge graph is a complex system process, involving multiple key steps such as multimodal data classification and layering, establishing modality data spectra, and generating the target multimodal knowledge graph. The technologies and methods involved in these steps encompass multiple fields, including medicine, artificial intelligence (AI), information modeling, and standardization. Therefore, the construction of a medical multimodal knowledge graph requires the establishment of a meta-system to describe and standardize the structure, behavior, rules, boundaries, and elements of the system combining these fields. The establishment and application of the meta-system ensures that the constructed medical multimodal knowledge graph possesses good consistency, accuracy, robustness, reusability, and scalability.
[0029] 3. The construction of a medical multimodal data knowledge graph requires handling complex data. Medical data is among the most complex in nature, encompassing unified medical language, medical terminology, diagnostic codes, surgical codes, drug codes, clinical guidelines, electronic medical records, and more, within the medical field alone. Constructing a medical multimodal knowledge graph first requires feature analysis and clustering analysis of the complex medical data to establish modal categories and hierarchical relationships. Second, it requires analyzing the interaction relationships connecting the modal data, which is the core feature of a multimodal data knowledge graph. These analysis and construction processes require AI technology support. Commonly used machine algorithms include: statistical machine learning for data modality processing, convolutional neural networks (CNNs) for image modality processing, graph neural networks (GNNs) for interface modality processing, and recurrent neural networks (RNNs) or Tansformer architecture models for language modality processing.
[0030] 4. With the rapid development of AI technology, especially the launch and application of multimodal large-scale models such as Alibaba HumanOmniV2, OpenAI GPT-4o, and Gemini 2.5, the importance of multimodal data knowledge graphs for model training and inference is becoming increasingly prominent. Currently, the medical industry lacks unified standards for the development of medical multimodal data knowledge graphs, resulting in a lack of consistency, reusability, and interoperability among multimodal data knowledge graphs developed by different institutions or groups, and also leading to redundant development and resource consumption. This invention provides an effective method to solve this problem.
[0031] Additional aspects and advantages of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. Attached Figure Description
[0032] Figure 1 This is a schematic diagram of the overall architecture of the meta-system for developing a medical multimodal data knowledge graph according to the present invention.
[0033] Figure 2 This is a schematic diagram of the medical multimodal metamodal architecture (4MA) of the metasystem for developing a medical multimodal data knowledge graph according to the present invention.
[0034] Figure 3 This is a schematic diagram of the medical multimodal framework (MMF) of the meta-system for developing a medical multimodal data knowledge graph according to the present invention.
[0035] Figure 4 This is a schematic diagram of the medical multimodal spectrum (MMS) of the meta-system for developing a medical multimodal data knowledge graph according to the present invention.
[0036] Figure 5 This is a schematic diagram of the medical multimodal knowledge graph (MMGK) of the meta-system developed for the medical multimodal data knowledge graph of this invention. Detailed Implementation
[0037] The following detailed description illustrates the specific implementation method:
[0038] For example Figure 1 As shown, the present invention (a meta-system for developing a medical multimodal data knowledge graph) includes four modules:
[0039] The Medical Multimodal Meta-Modal Architecture (4MA) module is used to set semantic rules and constraint rules for medical multimodal data and generate the Medical Multimodal Meta-Modal Architecture (4MA).
[0040] The Medical Multimodal Framework (MMF) module is used to classify medical multimodal data and generate a multimodal framework (MMF).
[0041] Medical Multimodal Spectrum (MMS) module: used to analyze the interrelationships of medical multimodal data and generate a multimodal spectrum (MMS).
[0042] The Medical Multimodal Knowledge Graph (MMKG) module is used to output modality-related data and generate a multimodal knowledge graph (MMKG).
[0043] Figure 1 The diagram shows the relationships between the four modules, with the top-down view showing the normative and constraint relationships, and the bottom-up view showing the dependency and compliance relationships.
[0044] The specific functions of the four modules are as follows:
[0045] Medical Multimodal Metamodal Architecture (4MA) Module
[0046] like Figure 2 As shown, the Medical Multimodal Metamodal Architecture (4MA) is the top-level model (metamodel) of this system. It consists of the medical domain, standard domain, modeling domain, and AI domain related to the field of medical multimodal machine learning. At the knowledge ontology and system concept level, it establishes a top-level model that guides and standardizes the description of medical multimodal data entities and constructs a multimodal data knowledge graph.
[0047] The medical domain describes and standardizes the medical knowledge system that needs to be referenced in the development of medical multimodal data knowledge graphs, defining the content and boundaries of medical multimodal data knowledge graphs.
[0048] The standard domain describes and standardizes the international / domestic medical information-related standards that need to be referenced and followed when developing medical multimodal data knowledge graphs. It defines the identification and structure of various types of medical multimodal data, as well as their interoperability.
[0049] The modeling domain describes and standardizes the modeling techniques used in developing medical multimodal data knowledge graphs, and defines object-oriented and data-driven information modeling methods.
[0050] The AI domain describes and standardizes the AI technologies used in the development of medical multimodal data knowledge graphs, and recommends AI technologies such as machine learning and deep learning that can be adopted in the development process.
[0051] The above four domains describe the system scope for constructing a medical multimodal data knowledge graph. From the dimensions of knowledge resources, standards and specifications, information models and AI technology, a general architecture for constructing a medical multimodal data knowledge graph is formed.
[0052] Medical Multimodal Framework (MMF) Module
[0053] like Figure 3 As shown, the Medical Multimodal Framework (MMF) module classifies and layers various modalities of medical data. Referring to the Meta-Object Facility (MOF) standard (GB / T28391-2009 / ISO / IEC 19502-2005), it establishes a multimodal framework layered by heterogeneity, classificatory, isomorphic, and isomorphic-heterogeneous based on the entity attributes of medical multimodal data. Heterogeneous modalities refer to modalities with significant structural differences that cannot be consistently expressed through direct mapping relationships; isomorphic modalities refer to modalities with similar structures and features that can be consistently expressed through mapping relationships; classificatory modalities refer to modal states between heterogeneous and isomorphic modalities; and isomorphic-heterogeneous modalities are isomorphic native modalities, i.e., modalities that cannot be further specialized (no submodalities). Moving upwards from the isomorphic-heterogeneous modality layer, it represents a progressive generalization from entities to entity classes, including attributes, relationships, and interactions.
[0054] Medical Multimodal Spectrum (MMS) Module
[0055] like Figure 4 As shown, this paper collects and organizes medical data modalities related to a medical multimodal knowledge graph. Referring to the AI knowledge graph technology framework standard (GB / T42131-2022), and based on the multimodal framework (MMF), within the scope of medical data, through the analysis of multimodal entities, attributes, relationships, fusion, and interaction elements, the collected and organized medical data modalities are generated into a complete multimodal spectrum (MMS). The multimodal spectrum (MMS) consists of heterogeneous, class-based, isomorphic, and isomorphic-heterogeneous multimodal data combinations. The medical multimodal spectrum (MMS) provides the scope of multimodal data entities for constructing a medical multimodal knowledge graph.
[0056] The Medical Multimodal Spectrum (MMS) in the medical field is a very large system. In practice, the required Medical Multimodal Spectrum (MMS) can be established by constructing a target Medical Multimodal Knowledge Graph as needed. Figure 4 This is an example of multimodal spectroscopy (MMS), which describes the layer-by-layer specialization of medical image data → nuclear medicine image data → positron emission tomography (PET) image data → positron emission tomography (PET) sequence image data.
[0057] Medical Multimodal Knowledge Graph (MMKG) Module
[0058] The focus of constructing MMKG is to study and express the interaction relationships between adjacent modalities, that is, the new features and information generated after the combination of adjacent modalities. These new features and information do not exist (or are not expressed) in single-modal data applications, but are expressed or emerge in multimodal data applications.
[0059] Studying the interaction relationships between data entities of different modalities requires the use of machine learning techniques. For example, data feature extraction, classification, and aggregation analyses typically employ machine learning algorithms such as XGBoost, Support Vector Machines, Bayesian classifiers, and Logistic Regression. For the discovery, extraction, and fusion alignment of relationships between different modalities, deep learning algorithms can be used, including large language models, graph neural networks (GNNs), relational graph convolutional networks (RGNNs), and multimodal fusion models. The generated medical multimodal knowledge graph is stored in a graph database (such as Neo4j), providing knowledge support for medical AI applications.
[0060] Figure 5 The diagram shows a medical multimodal knowledge graph with N data entities in three data modalities. Solid lines between entities represent the interaction relationships between data entities of different modalities (this relationship needs to be studied and expressed), while dashed lines represent the relationships between data entities of the same modality (this relationship is the same as that in a unimodal knowledge graph).
[0061] Obviously, the above embodiments are merely illustrative examples for clear explanation and are not intended to limit the implementation. Those skilled in the art will recognize that other variations or modifications can be made based on the above description. It is neither necessary nor possible to exhaustively list all possible implementations here. However, obvious variations or modifications derived therefrom are still within the scope of protection of this invention.
Claims
1. A meta-system for developing a medical multimodal data knowledge graph, characterized in that, It consists of four modules: Medical Multimodal Metamodal Architecture (4MA), Medical Multimodal Framework (MMF), Medical Multimodal Spectrum (MMS), and Medical Multimodal Knowledge Graph (MMKG). It provides guidance and specifications for the development and construction of medical multimodal knowledge graphs for medical multimodal data machine learning research and applications. The Medical Multimodal Metamodal Architecture (4MA) module establishes a top-level architectural model at the data modality ontology and conceptual level of medical multimodal machine learning, and standardizes and constrains the structure, logic and expression of medical multimodal data knowledge graphs. The Medical Multimodal Framework (MMF) module is used to set cross-domain and interdisciplinary rules for medical multimodal data, classify medical multimodal data according to disciplinary standards, and generate disciplinary types for medical multimodal data. The Medical Multimodal Spectrum (MMS) module, based on the output of the Medical Multimodal Framework (MMF) module, connects different modal relationships, labels the types of multimodal data relationships, forms medical multimodal spectra of different disciplines, diseases and types of medical multimodal data, and transmits them to the Medical Multimodal Knowledge Graph (MMKG) module; The Medical Multimodal Knowledge Graph (MMKG) module, based on the output of the medical multimodal framework and medical multimodal spectrum, constructs a multimodal knowledge graph of disciplines, diseases, and disease types required for machine learning on medical multimodal data. Within the framework of medical multimodal spectrum, interactive analysis is performed on the relationship types of multimodal data, describing and defining the interactive relationships between different modal data, establishing relational connections, and describing the relational elements of the medical multimodal knowledge graph; In the process of interactive analysis of multimodal data relationship types, machine learning technology is used to perform interactive analysis on different types of multimodal data to generate features, semantics and connection relationships between multimodal data.
2. The meta-system for developing medical multimodal data knowledge graphs according to claim 1, characterized in that, The Medical Multimodal Metamodal Architecture (4MA) module is used to establish a top-level framework architecture for describing medical multimodal data entities and constructing a multimodal data knowledge graph.
3. The meta-system for developing medical multimodal data knowledge graphs according to claim 1, characterized in that, The Medical Multimodal Metamodal Architecture (4MA) module consists of the medical domain, standard domain, modeling domain, and AI domain related to the field of medical multimodal machine learning. It establishes top-level semantic rules for standardizing medical multimodal data entities and constructing a multimodal data knowledge graph.
4. The meta-system for developing a medical multimodal data knowledge graph according to claim 1, characterized in that, The Medical Multimodal Framework (MMF) module is used to establish a hierarchical medical multimodal data framework for heterogeneous modalities, class-based modalities, isomorphic modalities, and isomorphic-heterogeneous modalities, and to transfer the Medical Multimodal Framework (MMF) module to the Medical Multimodal Spectrum (MMS) module.
5. The meta-system for developing a medical multimodal data knowledge graph according to claim 1, characterized in that, The Medical Multimodal Spectrum (MMS) module, based on the hierarchical structure of the Medical Multimodal Framework (MMF) module, analyzes the entity types and data types of medical multimodal data according to the data modalities of the target medical multimodal knowledge graph, and stores them as data modal tuples.
6. The meta-system for developing medical multimodal data knowledge graphs according to claim 5, characterized in that, The Medical Multimodal Spectrum (MMS) module collects medical multimodal data based on target multimodal knowledge graph data modal tuples. Through the analysis of multimodal entities, attributes, relationships, fusion and interaction elements, it generates a complete multimodal spectrum MMS.
7. The meta-system for developing medical multimodal data knowledge graphs according to claim 1, characterized in that, The Medical Multimodal Knowledge Graph (MMKG) module is used to describe the names, attributes, and interaction relationships of data entities within the target medical multimodal knowledge graph, according to the triple rules of the knowledge graph.
8. The meta-system for developing medical multimodal data knowledge graphs according to claim 7, characterized in that, The Medical Multimodal Knowledge Graph (MMKG) module is used to generate a target medical multimodal knowledge graph based on the names, attributes, and interaction relationships of the data entities in the target multimodal knowledge graph.
9. The meta-system for developing a medical multimodal data knowledge graph according to claim 1, characterized in that, The research and application of machine learning in medical multimodal data includes the development of medical multimodal knowledge graphs. The machine learning techniques used are statistical machine learning techniques and neural network-based deep learning techniques.