Metasystem for medical multi-modal data knowledge graph development

Abstract

The invention belongs to the technical field of medical artificial intelligence, and relates to a medical multi-modal data machine learning standardization system. Comprising the following modules and functions: a multi-modal element modal architecture (4MA) module is used for describing a data modal ontology concept of medical multi-modal machine learning and establishing a multi-modal data top-layer architecture model; a multi-modal framework (MMF) module is used for expressing core elements such as entities, data and relation types of medical multi-modal data and defining a semantic framework structure of the multi-modal data; the MMS module is used for describing a modal relation and an interaction relation of medical multi-modal data entities in various medical data categories and establishing relation logic among the multi-modal data; the MMKG module is based on the medical multi-modal spectrum, carries out deep analysis on multi-modal data modals and interaction relations, describes and defines relation attributes among different modal data, establishes relation connection, and outputs a medical multi-modal knowledge graph.

Description

Technical Field

[0001] This invention belongs to the field of medical artificial intelligence technology and is geared towards machine learning applications of medical multimodal data. It describes the structure, scope, rules, and elements of medical multimodal data knowledge graphs to standardize the development and construction of medical multimodal knowledge graphs. Background Technology

[0002] Modality refers to the way in which natural phenomena are perceived or expressed. Real-world objects represent their state of existence in multiple different modalities. Through various sensors, modalities are converted into data, such as electrical signals, images, text, video, and sound. By integrating this multi-source, heterogeneous data, multimodal data can more comprehensively and meticulously reveal the characteristics of things, thus providing a richer and more complete information representation.

[0003] Medical data is inherently multimodal, encompassing various data types such as medical images, electrophysiological signals, examination and test results, medical records, pathological images, and omics information. Supported by artificial intelligence (AI) technology, this multimodal data is efficiently utilized through techniques like machine learning and deep learning, providing doctors with precise decision support in diagnosing diseases, developing treatment plans, and assessing patient prognosis.

[0004] Medical knowledge graphs play a crucial role in enhancing data feature representation and semantic understanding in medical data AI applications. However, existing medical knowledge graphs are primarily unimodal, consisting of triples (entity, attribute, and relation) of medical data entities within the same modality. With the development of AI technology, especially the emergence of large-scale multimodal models, multimodal medical knowledge graphs are increasingly attracting industry attention. In the medical field, the importance of multimodal knowledge graphs is mainly reflected in their knowledge spectrum... Figure Three On the relational elements of tuples, new fusion relationships are generated when one modal data entity interacts with another. These new features arising from the fusion of different modal data significantly enhance AI's capabilities in disease prevention, diagnosis, and rehabilitation. Currently, the application of medical multimodal data knowledge graphs has received widespread attention; however, the lack of guidelines and standards for constructing medical multimodal data knowledge graphs hinders their development and application. Summary of the Invention

[0005] To address the aforementioned challenges in developing and constructing medical multimodal data knowledge graphs, we have established a meta-system for this purpose. This meta-system describes the structure, scope, rules, and elements of medical multimodal data knowledge graphs from the perspectives of concept, framework, logic, and expression, providing guidance, standards, and constraints for their development and construction.

[0006] The meta system follows the relevant standards and specifications published by the International Organization for Standardization (ISO), the International Electrotechnical Commission (IEC), and the national standards (GB) and industry standards (such as the health industry standard WS) of our country.

[0007] The meta system does not make specific requirements for the specific technologies, tools and products used in the development and construction process of the medical multi-modal data knowledge graph.

[0008] The technical scheme of the present application is as follows: a meta system for developing a medical multi-modal data knowledge graph, comprising a medical multi-modal meta-architecture (4MA), a medical multi-modal framework (MMF), a medical multi-modal spectrum (MMS), and a medical multi-modal knowledge graph (MMGK).

[0009] The medical multi-modal meta-architecture (4MA) module, based on the data modal ontology and concept level of medical multi-modal machine learning, makes specifications and constraints on the structure, logic and expression of the medical multi-modal data knowledge graph, and establishes a high-top architecture model.

[0010] The medical multi-modal framework (MMF) module is used to set the medical multi-modal data rules across domains and disciplines, classify the medical multi-modal data according to the discipline standards, and generate the discipline types of the medical multi-modal data. First, medical multi-modal data is collected, the entity types and data types of the medical multi-modal data are analyzed, and the data modal tuples are stored. Then, the medical multi-modal data is classified and layered according to the content of the data modal tuples. Then, the medical multi-modal data framework is formed according to the classification and layering of the medical multi-modal data, and the medical multi-modal framework is transmitted to the medical multi-modal spectrum module.

[0011] The medical multi-modal spectrum (MMS) module, based on the output of the medical multi-modal framework (MMF) module, connects different modal relationships, labels the multi-modal data relationship types, forms the medical multi-modal spectrum of different disciplines, diseases and disease types of the medical multi-modal data, and transmits it to the medical multi-modal knowledge graph module. The medical multi-modal spectrum is used to reflect the modal types of the medical multi-modal data, including heterogeneous modalities, similar modalities, homogeneous modalities and homogeneous heterogeneous modalities.

[0012] The medical multi-modal knowledge graph (MMKG) module, according to the output of the medical multi-modal framework and the medical multi-modal spectrum, constructs the multi-modal knowledge graph (target multi-modal knowledge graph) of disciplines, diseases and disease types required for medical multi-modal data machine learning. Within the framework of the medical multi-modal spectrum, the interactive relationships of the data entities included in the target multi-modal knowledge graph are analyzed and described using machine learning and deep learning technologies, and the relationship connections are established.

[0013] In the data entity range of the target multi-modal knowledge graph, according to the triple rule of the knowledge graph, the name, attribute and interaction relationship (the relationship between different modal data) of the data entity are described, and the target multi-modal knowledge graph is constructed.

[0014] Firstly, according to the application scene, the medical multi-modal knowledge graph to be constructed (target multi-modal knowledge graph) is determined.

[0015] Further, in the medical multi-modal meta-modal architecture (4MA) module, the ontology, terminology, discipline and information model related to the target multi-modal knowledge graph are selected, and the terminology of the multi-modal data included (or to be included) in the target multi-modal knowledge graph is determined, including identification, naming and definition. For data not included in the terminology standard, attribute or mapping processing needs to be done. This step guarantees the scientificity and accuracy of the target multi-modal knowledge graph.

[0016] Further, in the standard domain of the medical multi-modal meta-modal architecture (4MA) module, the information standards adopted in the existing international / domestic medical information related standards related to the target multi-modal knowledge graph are determined as guidance, specification and constraint.

[0017] Further, according to the modeling domain convention of the medical multi-modal meta-modal architecture (4MA) module, the object-oriented and data-driven information modeling method is adopted to carry out related modeling of the target multi-modal knowledge graph construction. This step guarantees the reusability and robustness of the target multi-modal knowledge graph.

[0018] Further, in the AI domain of the medical multi-modal meta-modal architecture (4MA) module, according to the characteristics and relationships of the modal data related to the target multi-modal knowledge graph, the machine learning technology to be adopted is selected. This step guarantees the accuracy and scalability of the target multi-modal knowledge graph.

[0019] Different machine learning algorithm models or different parameters of the same model should be considered, and multi-modal large models are also recommended for post-training (such as prompt word, RAG, fine-tuning, API, AGENT, etc.).

[0020] Further, the entity type and data type of the multi-modal data of the target multi-modal knowledge graph are analyzed and stored as data modal tuples. Based on the content of the data modal tuples, the medical multi-modal data is classified and layered. According to the type of medical multi-modal data (heterogeneous modal, similar modal, homogeneous modal and homogeneous heterogeneous modal), the medical multi-modal data framework is formed.

[0021] Further, the semantic rules of the medical multi-modal data across domains and disciplines are set, the medical multi-modal data is classified according to the discipline standard, and the discipline type of the medical multi-modal data is generated.

[0022] Further, the relationship connection of different modal data is performed, the multi-modal data relationship type is marked, the multi-modal data set of the target multi-modal knowledge graph is formed, and the modal spectrum of medical multi-modal data of different disciplines, diseases and disease types is established.

[0023] Further, based on the multi-modal spectrum of the target multi-modal knowledge graph, the interaction relationship of the data entities of the target multi-modal knowledge graph is analyzed and described by using machine learning and deep learning technologies, and the relationship connection is established.

[0024] Further, according to the triple rule of the knowledge graph, the name, attribute and interaction relationship (relationship between different modal data) of the data entity are described, and the target multi-modal knowledge graph is constructed.

[0025] Further, the target multi-modal knowledge graph is generated and stored in a graph database.

[0026] The above scheme has the following beneficial effects:

[0027] 1. Medical knowledge graph plays an important role in medical research, clinical application, public health and new drug development. The existing medical knowledge graph mainly describes the attributes and relationships of the same mode data entities, which is similar to describing and characterizing related data on a plane. With the development of AI technology, especially the increasingly widespread application of large models, the data processing capability of machine learning and deep learning is continuously enhanced, and the multi-modal large model is put forward, and the medical multi-modal data mode is highly concerned. Compared with single-modal knowledge graph, multi-modal knowledge graph describes and characterizes related data in multi-dimensional space. Multi-modal knowledge graph can provide more accurate support for disease diagnosis and treatment, such as diagnosis of heart disease patients, ECG examination (time series signal mode) of patients and disease changes recorded by medical staff (natural language document mode), and through machine learning, the fusion and alignment processing of data of the two modes will significantly improve the accuracy of disease diagnosis. In the process of fusion and alignment processing of different modal data, multi-modal knowledge graph plays an important role.

[0028] 2. The construction of medical multi-modal knowledge graph is a complex system process, which includes multi-modal data classification and stratification, establishment of modal data spectrum and generation of target multi-modal knowledge graph, etc. The technology and method of these links involve multiple fields such as medicine, artificial intelligence AI, information modeling and standardization. Therefore, in the process of constructing medical multi-modal knowledge graph, a meta-system needs to be established to describe and standardize the structure, behavior, rule, boundary and element of the system composed of the above fields. The establishment and application of the meta-system ensure that the constructed medical multi-modal knowledge graph has good consistency, accuracy, robustness, reusability and scalability.

[0029] 3. The construction process of medical multi-modal data knowledge graph needs to process complex data. Medical data is one of the most complex data in nature, including unified medical language, medical terminology, diagnosis code, surgery code, drug code, clinical guidelines, electronic medical record, etc. in the medical field. To construct a medical multi-modal knowledge graph, first, feature analysis and clustering analysis of complex medical data are needed to establish the modal category and hierarchical relationship of medical data. Second, the interaction relationship connecting the modal data needs to be analyzed, which is the core feature of multi-modal data knowledge graph. These analysis and establishment processes need AI technology support, and common machine algorithms include: statistical machine learning for data modal processing, convolutional neural network CNN for image modal processing, graph neural network GNN for interface modal processing, recurrent neural network RNN or Tansformer architecture model for language modal processing, etc.

[0030] 4. With the rapid development of AI technology, especially the release and application of multi-modal large models such as Ali HumanOmniV2, OpenAI GPT-4o, Gemini 2.5, the importance of multi-modal data knowledge graph for model training and inference is increasingly prominent. At present, the medical industry still lacks unified specifications for the development of medical multi-modal data knowledge graph, resulting in a lack of consistency, reusability and interoperability of multi-modal data knowledge graphs developed by different institutions or groups, and also causing resource consumption of repeated development. The present application provides an effective method to solve this problem.

[0031] Additional aspects and advantages of the application will be in part apparent and in part pointed out hereinafter. BRIEF DESCRIPTION OF DRAWINGS

[0032] Figure 1 The overall architecture of a medical multi-modal data knowledge graph development meta-system of the present application is shown in the figure.

[0033] Figure 2 The medical multi-modal meta-modal architecture (4MA) of the medical multi-modal data knowledge graph development meta-system of the present application is shown in the figure.

[0034] Figure 3 The medical multi-modal framework (MMF) of the medical multi-modal data knowledge graph development meta-system of the present application is shown in the figure.

[0035] Figure 4 The medical multi-modal spectrum (MMS) of the medical multi-modal data knowledge graph development meta-system of the present application is shown in the figure.

[0036] Figure 5 The medical multi-modal knowledge graph (MMGK) of the medical multi-modal data knowledge graph development meta-system of the present application is shown in the figure. DETAILED DESCRIPTION

[0037] The following is further described in detail through a specific embodiment:

[0038] The application (a medical multi-modal data knowledge graph development meta-system) includes four modules as shown in the following figure: Figure 1 Medical multi-modal meta-modality architecture (4MA) module: used to set semantic rules and constraint rules of medical multi-modal data, and generate medical multi-modal meta-modality architecture (4MA).

[0039] Medical multi-modal framework (MMF) module: used to classify medical multi-modal data, and generate multi-modal framework (MMF).

[0040] Medical multi-modal spectrum (MMS) module: used to analyze the mutual relationship of medical multi-modal data, and generate multi-modal spectrum (MMS).

[0041] Medical multi-modal knowledge graph (MMKG) module: used to output modal related data, and generate multi-modal knowledge graph (MMKG).

[0042]

[0043] The above four modules are shown to have mutual relationship, and the specification and constraint relationship is realized from top to bottom, and the dependency and following relationship is realized from bottom to top. Figure 1 The specific functions of the four modules are as follows:

[0044] Medical multi-modal meta-modality architecture (4MA) module

[0045] As shown in the following figure, the medical multi-modal meta-modality architecture (4MA) is the top model (meta-model) of the system, which is composed of medical domain, standard domain, modeling domain and AI domain related to medical multi-modal machine learning field, and establishes the top model for guiding and standardizing the description of medical multi-modal data entities and the construction of multi-modal data knowledge graph at the level of knowledge ontology and system concept.

[0046] Figure 2 Medical domain: describes and standardizes the medical knowledge system needed to develop medical multi-modal data knowledge graph, and defines the content and boundary of medical multi-modal data knowledge graph.

[0047] Medical domain: describes and standardizes the medical knowledge system needed to develop medical multi-modal data knowledge graph, and defines the content and boundary of medical multi-modal data knowledge graph.

[0048] ​Standard domain, describes and specifies the international / domestic medical information related standards that need to be followed for the development of medical multi-modal data knowledge graph, which defines the identification and structure of various medical multi-modal data, as well as their interoperability.

[0049] Modeling domain, describes and specifies the modeling techniques used to develop medical multi-modal data knowledge graph, which stipulates object-oriented and data-driven information modeling methods.

[0050] AI domain, describes and specifies the AI techniques used in the development of medical multi-modal data knowledge graph, and recommends machine learning, deep learning and other AI techniques that can be used in the development process.

[0051] The above four domains describe the system category of building medical multi-modal data knowledge graph, forming the overall architecture of building medical multi-modal data knowledge graph from the dimensions of knowledge resources, standards and specifications, information models, and AI techniques.

[0052] Medical multi-modal framework (MMF) module

[0053] As shown in Figure 3 , the medical multi-modal framework (MMF) module classifies and layers various modal data of medical data, refers to the meta-object facility (MOF) standard (GB / T28391-2009 / ISO / IEC 19502-2005), and establishes a multi-modal framework based on the entity attributes of medical multi-modal data, which is layered according to heterogeneity, class structure, isomorphism, and isomorphism. Among them, the heterogeneous modal refers to the significant structural difference between modalities, which cannot be expressed consistently through direct mapping relationship; the isomorphic modal refers to the modal state between the heterogeneous modal and the isomorphic modal, which can be expressed consistently through the mapping relationship; the class structure modal is between the heterogeneous modal and the isomorphic modal; the isomorphic heterogeneous modal is the isomorphic original modal, which cannot be specialized (without sub-modal) modal. From the isomorphic heterogeneous modal layer upwards, it is the layer-by-layer generalization of entities to entity classes, including attributes, relationships, and interactions.

[0054] Medical multi-modal spectrum (MMS) module

[0055] As shown in Figure 4 , collect and organize medical data modal types related to medical multi-modal knowledge graph, refer to the artificial intelligence knowledge graph technical framework standard (GB / T42131-2022), based on the multi-modal framework (MMF), analyze the collected and organized medical data modal through multi-modal entities, attributes, relationships, fusion, and interaction elements, and generate a complete multi-modal spectrum MMS. The multi-modal spectrum (MMS) is composed of heterogeneous, class structure, isomorphic, and isomorphic heterogeneous multi-modal data. The medical multi-modal spectrum (MMS) provides the scope of multi-modal data entities for the construction of medical multi-modal knowledge graph.

[0056] Medical multi-modal spectrum (MMS) in the medical field is a very large system, and in practice, the target medical multi-modal knowledge graph can be constructed according to the needs, and the required medical multi-modal spectrum (MMS) can be established. Figure 4 is an example of multi-modal spectrum (MMS), which describes the layer-by-layer specialization of medical image data → nuclear medicine image data → positron (PET) image data → positron (PET) sequence image data.

[0057] Medical multi-modal knowledge graph (MMKG) module

[0058] The focus of constructing MMKG is to study and express the interaction relationship between adjacent modalities, that is, the new features and new information generated after the combination of adjacent modalities. Such new features and information do not exist (or are not expressed) in single-modal data applications, but emerge or emerge when multi-modal data applications are performed.

[0059] The study of the interaction relationship between different modal data entities requires the use of machine learning techniques, for example, data feature extraction, classification and aggregation analysis usually use machine learning algorithms such as XGBoost, support vector machine, Bayesian classifier and Logistic regression. For relationship discovery, extraction and fusion alignment between different modal data, deep learning algorithms can be used, including large language models, graph neural networks GNN, relationship graph convolutional networks RGNN, multi-modal fusion models, etc. The generated medical multi-modal knowledge graph is stored in a graph database (such as Neo4j), providing knowledge support for medical AI applications.

[0060] Figure 5 The medical multi-modal knowledge graph of N data entities of 3 data modalities is shown, and the solid lines between the entities in the figure represent the interaction relationship between different modal data entities (which needs to be studied and expressed), and the dashed lines represent the relationship between the same modal data entities (which is the same as the single-modal knowledge graph).

[0061] Obviously, the above embodiments are only examples for the purpose of clear illustration, and are not intended to limit the embodiments. Based on the above description, other different forms of changes or variations can also be made by those skilled in the art. Here, it is not necessary and impossible to exhaust all embodiments. The obvious changes or variations derived therefrom are still within the protection scope of the present application.

Claims

1. A meta-system for medical multi-modal data knowledge graph development, characterized in that, The medical multi-modal meta-modal architecture (4MA) module, the medical multi-modal framework (MMF) module, the medical multi-modal spectrum (MMS) module, and the medical multi-modal knowledge graph (MMKG) module are four component modules for providing guidance and specifications for the development and construction of medical multi-modal knowledge graphs for medical multi-modal data machine learning research and application. The medical multi-modal meta-modal architecture (4MA) module establishes a top-level architecture model at the data modality ontology and concept level of medical multi-modal machine learning, and makes specifications and constraints on the structure, logic, and expression of medical multi-modal data knowledge graphs. The medical multi-modal framework (MMF) module is used to set up cross-domain and interdisciplinary medical multi-modal data rules, classify medical multi-modal data according to discipline standards, and generate discipline types of medical multi-modal data. The medical multi-modal spectrum (MMS) module, based on the output of the medical multi-modal framework (MMF) module, connects different modal relationships, labels multi-modal data relationship types, forms medical multi-modal spectra of different disciplines, diseases, and disease types, and transmits them to the medical multi-modal knowledge graph (MMKG) module. The medical multi-modal knowledge graph (MMKG) module, based on the output of the medical multi-modal framework and the medical multi-modal spectrum, constructs the multi-modal knowledge graph of disciplines, diseases, and disease types required for medical multi-modal data machine learning. In the medical multi-modal spectrum framework, the interaction between multi-modal data relationship types is analyzed, the interaction between different modal data is described and defined, and the relationship connection is established to describe the relationship elements of the medical multi-modal knowledge graph. In the process of interactive analysis of multi-modal data relationship types, machine learning (including deep learning) techniques are used to analyze different types of multi-modal data, generating features, semantics, and connection relationships between multi-modal data.

2. The meta-system for medical multi-modal data knowledge graph development of claim 1, wherein, The medical multi-modal meta-modal architecture (4MA) module is used to establish a top-level framework architecture for describing medical multi-modal data entities and constructing multi-modal data knowledge graphs.

3. The meta-system for medical multi-modal data knowledge graph development of claim 1, wherein, The medical multi-modal meta-modal architecture (4MA) module is composed of medical domains, standard domains, modeling domains, and AI domains related to the field of medical multi-modal machine learning, and establishes top-level semantic rules for standardizing medical multi-modal data entities and constructing multi-modal data knowledge graphs.

4. The meta-system for medical multi-modal data knowledge graph development of claim 1, wherein, The medical multi-modal framework (MMF) module is used to establish a hierarchical medical multi-modal data framework for heterogeneous modalities, similar modalities, homogeneous modalities, and homogeneous heterogeneous modalities, and transmits the medical multi-modal framework (MMF) module to the medical multi-modal spectrum (MMS) module.

5. The meta-system for medical multi-modal data knowledge graph development of claim 1, wherein, The medical multi-modal spectrum (MMS) module, based on the hierarchical structure of the medical multi-modal framework (MMF) module, analyzes the entity types and data types of medical multi-modal data according to the data modalities of the target medical multi-modal knowledge graph, and stores them as data modality tuples.

6. The meta-system for medical multi-modal data knowledge graph development of claim 5, wherein, The medical multi-modal spectrum (MMS) module, based on the data modality tuples of the target multi-modal knowledge graph, collects medical multi-modal data, and generates a complete multi-modal spectrum MMS through multi-modal entity, attribute, relationship, fusion, and interaction element analysis.

7. The meta-system for medical multi-modal data knowledge graph development of claim 1, wherein, The medical multi-modal knowledge graph (MMKG) module is configured to describe the name, attribute and interaction relationship (relationship between different modal data) of the data entity according to the triple rule of the knowledge graph within the data entity range of the target medical multi-modal knowledge graph.

8. The meta-system for medical multi-modal data knowledge graph development of claim 7, wherein, The medical multi-modal knowledge graph (MMKG) module is configured to generate the target medical multi-modal knowledge graph according to the name, attribute and interaction relationship of the data entity of the target multi-modal knowledge graph.

9. The meta-system for medical multi-modal data knowledge graph development of claim 1, wherein, The medical multi-modal knowledge graph developed by the medical multi-modal data machine learning research and application is based on statistical machine learning technology and neural network deep learning technology.

Images