A multimodal large language model knowledge graph construction method and system

Abstract

The application discloses a kind of multimodal large language model knowledge graph construction method and system, it is related to electric digital data processing technical field;Including: in traditional Chinese medicine multi-source multimodal data is collected, after desensitization, missing value is completed in uniform format, generates standardization dataset;Fusion multimodal feature, extract knowledge element candidate;Fine-tuning model and combined semantic alignment, realize knowledge element accurate identification and normalization;Excavate entity relationship and construct triple;Correlation multi-source entity resolves conflict, integrates unified knowledge set;Design schema imports graph database, constructs structured knowledge graph;Establish increment channel, real-time update graph and periodically optimize performance.The knowledge graph constructed by the application is accurate in entity recognition, deep in relationship mining;Support multi-scene multimodal retrieval and clinical assistance, improve inheritance efficiency and diagnosis and treatment standardization, provide strong support for traditional Chinese medicine experience digitization inheritance, scientific research analysis, help traditional Chinese medicine modernization development.

Description

Technical Field

[0001] This invention relates to the field of electronic digital data processing technology, and in particular to a method and system for constructing a multimodal large language model knowledge graph. Background Technology

[0002] The diagnostic and treatment experiences of various schools of thought in Traditional Chinese Medicine (TCM) are a core treasure of traditional medicine. The wisdom accumulated by renowned veteran TCM doctors through long-term clinical practice, linking disease, syndrome, prescription, and medication, has irreplaceable value for clinical diagnosis and treatment, academic inheritance, and the development of the discipline. However, the current inheritance of the experience of renowned veteran TCM doctors faces many practical difficulties. Traditional inheritance models primarily rely on oral instruction and observation, depending on the learner's comprehension ability. The accuracy and completeness of experience transmission are difficult to guarantee, and a large amount of valuable diagnostic and treatment wisdom is at risk of being lost due to a lack of systematic recording. With the development of digital technology, some medical institutions have attempted to collect relevant diagnostic and treatment data, but a standardized inheritance system has not yet been formed, making it difficult to achieve large-scale and precise inheritance of experience.

[0003] The fragmentation and integration of multimodal data has become a core obstacle to its transmission. Data generated during the clinical practice of renowned traditional Chinese medicine (TCM) practitioners encompasses various types of information, including TCM diagnostic methods, audio and video recordings of consultations, clinical medical records, ancient texts, and laboratory reports. These data sources are scattered and vary in format. The diagnostic information is mostly qualitative, audio and video data lacks structured labeling, and medical records and literature suffer from inconsistent terminology and unclear relationships. Existing data collection methods lack unified standards, data labeling and association methods are incomplete, and audio / video segmentation and text extraction are inefficient, making effective integration of multimodal data difficult, forming "data silos" that fail to provide a high-quality data foundation for experience mining. Furthermore, the lack of standardized de-identification mechanisms for sensitive data poses challenges to data security and privacy protection, further limiting data sharing and application.

[0004] The application of knowledge graph construction technology in the field of Traditional Chinese Medicine (TCM) has significant shortcomings. Existing technologies mostly focus on knowledge extraction from single-type data, making it difficult to achieve deep integration of multimodal data. This results in knowledge graphs with weak entity associations and insufficient semantic mining, failing to fully present the complex relationships between "disease-syndrome-prescription-medicine" and elements such as symptoms and constitution. Furthermore, existing knowledge graphs tend to emphasize the organization of theoretical knowledge, lacking integration with the actual clinical experience of renowned veteran TCM doctors, thus failing to meet practical needs such as clinical decision support and immersive teaching. The functional design of inheritance platforms is also relatively simple, lacking core functions such as multimodal archive retrieval, knowledge visualization analysis, and personalized learning path planning. This prevents the formation of an interactive "observation-training-practice-evaluation" inheritance model, leading to low inheritance efficiency and hindering the effective improvement of learners' clinical skills. In the context of the modernization and intelligent development of TCM, there is an urgent need for an innovative technological solution that can integrate multimodal data, construct accurate knowledge graphs, and support multi-scenario applications to overcome the bottlenecks of traditional inheritance. Summary of the Invention

[0005] This invention proposes a method and system for constructing a multimodal large language model knowledge graph to solve the problems mentioned in the prior art.

[0006] To achieve the above objectives, the present invention adopts the following technical solution: a method for constructing a multimodal large language model knowledge graph, comprising: The multimodal data acquisition and standardization process involves: acquiring multimodal data, desensitizing sensitive data, standardizing text, image, and audio / video data formats, using similarity interpolation to fill in missing values, segmenting audio / video data by patient visit ID and transcribing it into text, extracting text information from ancient books and medical records using optical character recognition technology, and generating a standardized multimodal dataset with a unified structure and complete content. The steps of multimodal feature fusion and knowledge element candidate extraction are as follows: deep semantic encoding of text data is performed using a pre-trained large language model; visual features such as tongue image are extracted through convolutional neural network; audio features are extracted using speech sentiment analysis technology; a cross-modal attention mechanism is introduced to fuse multimodal feature vectors; and knowledge element candidates are initially extracted by combining a custom rule base and a hidden Markov model. The steps of knowledge element accurate identification and entity normalization are as follows: a large language model is fine-tuned based on the corpus of traditional Chinese medicine, and the entity boundary recognition accuracy is optimized by combining conditional random fields and bidirectional long short-term memory networks. A thesaurus and ontology library of traditional Chinese medicine are constructed. The knowledge element candidates are normalized by string similarity matching and semantic alignment technology to eliminate entity ambiguity and assign unique entity identifiers. The entity relationship mining and triple construction steps are based on multimodal contextual semantic information, utilize the relationship extraction capability of large language models, combine graph neural networks to capture potential associations between entities, construct core relationship triples, and label relationship types, confidence levels and supporting data sources. The knowledge fusion and conflict resolution steps involve using entity linking technology to associate the same entity in multi-source data, establishing an entity attribute mapping table, detecting attribute and relationship conflicts through rule matching and machine learning algorithms, resolving conflicts using weighted voting mechanisms and semantic similarity calculations, integrating multi-dimensional knowledge, and generating a unified knowledge set without contradictions. The steps for constructing and storing a structured knowledge graph include designing a graph database schema for the field of Traditional Chinese Medicine, defining entity types, attribute fields, and relation types, importing normalized entity, triple, and attribute information into the graph database, establishing a structured knowledge graph containing nodes, edges, and attributes, optimizing the graph database index structure, and supporting efficient querying and traversal of entities, relations, and attributes. The knowledge graph dynamic optimization and incremental update steps involve establishing an incremental data acquisition channel to receive new diagnosis and treatment data and literature data in real time, repeating the above steps to extract new knowledge elements and relationships, updating the graph node attributes and edge information through entity similarity calculation and relationship confidence evaluation, periodically checking the graph integrity and consistency, and using graph structure optimization algorithms to improve the graph retrieval and reasoning performance.

[0007] Furthermore, it also includes a multimodal data quality assessment step, constructing a data quality assessment index system, designing quantitative indicators from four dimensions: completeness, accuracy, consistency, and timeliness, calculating data quality scores through weighted index calculation, and triggering a secondary processing flow for data below a set threshold, re-standardizing and completing operations to improve the quality of the input dataset and provide reliable data support for subsequent knowledge extraction and graph construction.

[0008] Furthermore, it also includes a knowledge element priority ranking step. Based on the importance of knowledge elements in the diagnosis and treatment scenario, data support, and expert recognition, a priority evaluation model is built to rank the extracted knowledge elements. High-priority knowledge elements are prioritized for inclusion in the core nodes of the graph, while low-priority knowledge elements are stored as extension nodes associated with the core nodes. This preserves the complete knowledge system without affecting the core query efficiency, optimizes the graph storage structure and query efficiency, and meets the knowledge access needs in different scenarios.

[0009] Furthermore, a knowledge element confidence calculation mechanism is introduced into the process of precise knowledge element identification. The final identification result is output through multi-model fusion, and the calculation expression is as follows:

[0010] in For knowledge elements The final confidence level ranges from 0 to 1; The number of models participating in the fusion; For the first The weight coefficients of each model are dynamically calibrated based on the historical accuracy of the model in the identification of knowledge elements in the field of traditional Chinese medicine. The sum of the weights of all models is 1. For the first Each model identifies knowledge elements. The original score; For the first Each model in knowledge element Accuracy correction factor for the category; For knowledge elements Frequency of occurrence in multi-source data; For knowledge elements The domain relevance coefficient is determined by domain experts and ranges from 0.1 to 1.0.

[0011] Furthermore, a cross-modal context enhancement strategy is adopted in the entity relationship mining process. Visual features in the image are fused with text semantic features and speech-to-text features to generate multi-dimensional context vectors. These vectors are then input into a large language model for relationship extraction. At the same time, a domain relationship template library is constructed, which covers common disease-syndrome-prescription-drug association patterns in traditional Chinese medicine. This library filters and supplements the model extraction results, achieving full coverage of core relationships and improving the recall and accuracy of relationship extraction. This provides high-quality relationship data for triple construction.

[0012] Furthermore, a dynamic adjustment formula for entity relationship weights is introduced during the knowledge fusion process to optimize the fusion effect of multi-source relationships. The calculation expression is as follows:

[0013] in For entities and Relationship between The final weight; Based on the weighting coefficient, Contribution coefficients for multi-source data The relationship link length influence coefficient, the sum of the three is 1; For relationship The initial base weights; The number of data sources supporting this relationship; For the first Relationships in data sources Confidence level; For the first The credibility weight of each data source; For entities and The shortest path length between them; The maximum value of the shortest path length for all entity pairs in the knowledge graph is obtained through a dynamic balance of basic weights, contributions from multiple data sources, and relational link features.

[0014] Furthermore, a two-stage update mechanism is adopted in the dynamic optimization process of the knowledge graph. The first stage is incremental update, which processes new data in real time, extracts knowledge elements and relationships, and judges new nodes or updated attributes through entity similarity threshold. The second stage is full optimization, which periodically performs a global scan of the graph to detect isolated nodes, redundant edges and inconsistent relationships, and uses graph clustering algorithm to optimize node grouping.

[0015] A multimodal large language model knowledge graph construction system includes: The multimodal data acquisition module integrates a traditional Chinese medicine diagnostic instrument, an audio and video recording terminal, an electronic medical record interface, and an ancient book scanning device. It supports both real-time acquisition and batch import modes, adapts to the data acquisition needs of different diagnosis and treatment scenarios, collects multi-source data, and has an automatic data format recognition function. The data standardization processing module has built-in desensitization algorithms, format conversion tools, missing value completion modules, and audio and video cutting and transcription units to achieve sensitive data anonymization, multi-format data unification, missing information completion, audio and video to text conversion, and generate standardized multimodal datasets to adapt to subsequent knowledge extraction processes. The multimodal feature fusion module is equipped with a large language model semantic coding unit, a computer vision feature extractor, and a speech feature analyzer. It adopts a cross-modal attention mechanism to fuse text, image, and audio features and outputs a unified-dimensional multimodal feature vector to support knowledge element extraction. The knowledge element extraction and recognition module includes a rule base, a statistical model, and a pre-trained large language model fine-tuning unit. It combines conditional random fields and convolutional neural networks to achieve knowledge element candidate extraction, accurate recognition, and entity normalization, and establishes a unified entity identification system to ensure the consistency of knowledge elements. The entity relationship mining module integrates a large language model relationship extraction unit and a graph neural network association capture unit. It supports relationship extraction and potential association mining based on multimodal context, constructs core triples, and labels relationship confidence to provide relationship data for graph construction. The knowledge fusion and conflict resolution module has entity linking tools, conflict detection rule base and weighted voting unit, which realizes multi-source entity association, attribute and relationship conflict resolution, integrates multi-dimensional knowledge, generates a unified knowledge set and eliminates knowledge redundancy and contradiction; The knowledge graph construction and storage module includes a graph database schema design tool, a data import unit, and an index optimization component. It supports structured storage of entities, relations, and attributes, and uses a graph database to achieve efficient querying and traversal, ensuring stable storage and fast access to the graph. The dynamic optimization and update module has a built-in incremental data processing unit, similarity calculation tool, and graph detection component, which enables real-time processing of new data, dynamic updating of graph nodes and edges, and regular optimization of graph structure to improve graph integrity and consistency. The multimodal retrieval and visualization module supports text keyword retrieval, image feature retrieval, and multi-condition combination retrieval. It is equipped with a graph visualization tool to display the relationship between disease, syndrome, prescription, and medicine in the form of nodes and edges, and provides graph zooming, traversal, and detailed viewing functions.

[0016] Furthermore, it also includes a knowledge quality assessment module, which has built-in integrity assessment unit, accuracy detection unit, and consistency verification unit. By calculating entity coverage rate, mean relation confidence score, and attribute consistency ratio, it quantifies the quality of the knowledge graph, generates a quality assessment report, provides direction for graph optimization, supports user-defined quality thresholds, and triggers an automatic optimization process when the quality does not meet the standards, continuously improving the quality of the knowledge graph.

[0017] Furthermore, it also includes a domain adaptation and extension module, with a built-in TCM domain ontology library, thesaurus, and relation template library. It supports user-defined entity types, attributes, and relation types, adapts to the knowledge graph construction needs of different TCM schools or medical fields, and provides standardized interfaces to support docking with external clinical systems, teaching platforms, and scientific research databases, enabling cross-system application and data sharing of knowledge graphs.

[0018] Compared with existing technologies, the beneficial effects of this invention are: In terms of data integration and standardization, the invention establishes a unified multimodal data acquisition and processing system, effectively solving the problems of inconsistent data formats and severe fragmentation in traditional data. By standardizing the acquisition process of data such as diagnostic information, audio and video recordings, and medical records, and optimizing data labeling and association methods and audio and video segmentation methods, the structured integration of multi-source data is achieved, improving data integrity and consistency. The sensitive data anonymization mechanism and secure storage design not only ensure data privacy and security but also support the legal sharing and long-term traceability of data, providing a high-quality and highly reliable data foundation for experience mining and scientific research analysis, completely changing the traditional situation of disorganized and unusable data.

[0019] In terms of knowledge graph construction and association depth, the invention achieves deep association of multiple elements such as "disease-symptom-prescription-medicine," as well as symptoms, constitution, and ancient literature, overcoming the pain point of weak knowledge association in existing technologies. Through multimodal feature fusion, precise knowledge element recognition, and entity normalization techniques, combined with the advantages of large language models and graph neural networks, the constructed knowledge graph accurately presents the diagnostic and treatment logic and experiential patterns of renowned traditional Chinese medicine practitioners, with more accurate entity recognition, deeper relationship mining, and tighter semantic association. The knowledge fusion and conflict resolution mechanisms effectively eliminate contradictions and redundancies in multi-source data, ensuring the accuracy and consistency of the knowledge graph and providing solid knowledge support for subsequent applications.

[0020] In terms of application scenario adaptation and transmission efficiency, the invention significantly expands the application boundaries of TCM experience transmission and improves the intelligence level of transmission and treatment. The constructed transmission innovation platform supports diverse functions such as multimodal archive retrieval, knowledge graph visualization analysis, and clinical decision support, meeting the needs of various scenarios including showcasing the experience of renowned veteran TCM doctors, immersive learning for learners, precise diagnosis and treatment for clinicians, and data analysis for researchers. The multimodal retrieval function allows users to quickly obtain relevant treatment cases and experiences through text, images, and other methods. The knowledge graph visualization intuitively presents complex knowledge relationships, helping learners quickly understand the treatment logic. The clinical support model can provide personalized treatment suggestions based on the knowledge graph, improving the scientific rigor and accuracy of clinical decision-making. Simultaneously, the platform supports an interactive transmission model of "observation-training-operation-evaluation," effectively lowering the learning threshold, improving transmission efficiency, and promoting the transformation of TCM experience from the traditional oral transmission model to a data-driven intelligent model.

[0021] In terms of scientific research and disciplinary development, the invention provides new support for the scientific research of traditional Chinese medicine (TCM) experience. Standardized multimodal datasets and structured knowledge graphs offer reliable tools for uncovering the correlation patterns between "disease-syndrome-prescription-medicine" and analyzing the diagnostic and treatment approaches of renowned TCM doctors, helping to reveal the core mechanisms of TCM diagnosis and treatment and enhancing the scientific rigor and persuasiveness of TCM. The long-term data storage and traceability functions support large-sample, long-term scientific research analysis, providing rich data resources for the modernization of TCM research. Furthermore, the domain adaptation and extension modules support personalized adjustments for different TCM schools or medical fields, possessing good scalability and reusability, which can promote the large-scale application of TCM experience inheritance techniques and contribute to the innovative development and continuation of traditional medicine. Attached Figure Description

[0022] Figure 1 This is a schematic block diagram of a method for constructing a multimodal large language model knowledge graph proposed in this invention; Figure 2 This is a schematic block diagram of a multimodal large language model knowledge graph construction system proposed in this invention; Figure 3 A bar chart comparing the accuracy of knowledge element identification for different schools of traditional Chinese medicine. Figure 4 Construct a line chart comparing the time consumption of each step in the knowledge graph process; Figure 5 Radar chart of the effects of primary care clinical decision support across various dimensions; Figure 6 A bar chart comparing the incremental update efficiency of knowledge graphs. Detailed Implementation

[0023] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0024] In the description of this invention, it should be understood that the terms "center," "longitudinal," "lateral," "length," "width," "thickness," "upper," "lower," "front," "rear," "left," "right," "vertical," "horizontal," "top," "bottom," "inner," "outer," "clockwise," and "counterclockwise," etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are only for the convenience of describing this invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on this invention.

[0025] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Thus, features defined with "first" and "second" may explicitly or implicitly include one or more of the stated features. In the description of this invention, "a plurality of" means two or more, unless otherwise explicitly specified. Furthermore, the terms "installed," "connected," and "linked" should be interpreted broadly; for example, they may refer to a fixed connection, a detachable connection, or an integral connection; they may refer to a mechanical connection or an electrical connection; they may refer to a direct connection or an indirect connection through an intermediate medium; and they may refer to the internal connection of two components. Those skilled in the art can understand the specific meaning of the above terms in this invention based on the specific circumstances. The invention will now be described in further detail with reference to the accompanying drawings.

[0026] Reference Figures 1 to 6 A method for constructing a knowledge graph for a multimodal large language model. The multimodal data collection and standardization process involves collecting multi-source, multimodal data, including information from the four diagnostic methods of traditional Chinese medicine, audio and video consultations, clinical medical records, ancient texts, and laboratory test reports. Sensitive data is desensitized, and the formats of text, images, audio, and video data are standardized. Missing values ​​are filled using similarity interpolation. Audio and video data are segmented and transcribed into text according to the patient's consultation ID. Text information from ancient texts and medical records is extracted using optical character recognition technology, resulting in a standardized multimodal dataset with a unified structure and complete content. The multimodal feature fusion and knowledge element candidate extraction steps utilize a pre-trained large language model to perform deep semantic encoding on text data, extract visual features such as tongue images through convolutional neural networks, extract audio features using speech sentiment analysis technology, introduce a cross-modal attention mechanism to fuse multimodal feature vectors, and combine a custom rule base and a hidden Markov model to initially extract knowledge element candidates such as diseases, syndromes, prescriptions, drugs, and symptoms. The steps of knowledge element accurate identification and entity normalization are as follows: a large language model is fine-tuned based on the corpus of traditional Chinese medicine, and the entity boundary recognition accuracy is optimized by combining conditional random fields and bidirectional long short-term memory networks. A thesaurus and ontology library of traditional Chinese medicine are constructed. The knowledge element candidates are normalized by string similarity matching and semantic alignment technology to eliminate entity ambiguity and assign unique entity identifiers. The entity relationship mining and triplet construction steps are based on multimodal contextual semantic information, utilize the relationship extraction capability of large language models, and combine graph neural networks to capture potential associations between entities. Core relationship triplets such as disease-syndrome-prescription-drug, symptom-syndrome, and drug-prescription are constructed, and the relationship type, confidence level and supporting data source are labeled. The knowledge fusion and conflict resolution steps involve using entity linking technology to associate the same entity in multi-source data, establishing an entity attribute mapping table, detecting attribute and relationship conflicts through rule matching and machine learning algorithms, resolving conflicts using weighted voting mechanisms and semantic similarity calculations, integrating multi-dimensional knowledge, and generating a unified knowledge set without contradictions. The steps for constructing and storing a structured knowledge graph include designing a graph database schema for the field of Traditional Chinese Medicine, defining entity types, attribute fields, and relation types, importing normalized entity, triple, and attribute information into the graph database, establishing a structured knowledge graph containing nodes, edges, and attributes, optimizing the graph database index structure, and supporting efficient querying and traversal of entities, relations, and attributes. The knowledge graph dynamic optimization and incremental update steps involve establishing an incremental data acquisition channel to receive new diagnosis and treatment data and literature data in real time, repeating the above steps to extract new knowledge elements and relationships, updating the graph node attributes and edge information through entity similarity calculation and relationship confidence evaluation, periodically checking the graph integrity and consistency, and using graph structure optimization algorithms to improve the graph retrieval and reasoning performance.

[0027] This invention also includes a multimodal data quality assessment step, which constructs a data quality assessment index system and designs quantitative indicators from four dimensions: completeness, accuracy, consistency, and timeliness. The data quality score is calculated by weighting the indicators. Data below the set threshold triggers a secondary processing flow, which performs standardization and completion operations again to improve the quality of the input dataset and provide reliable data support for subsequent knowledge extraction and graph construction.

[0028] This invention also includes a knowledge element priority ranking step. Based on the importance of knowledge elements in the diagnosis and treatment scenario, data support, and expert recognition, a priority evaluation model is constructed to rank the extracted knowledge elements. High-priority knowledge elements are prioritized for inclusion in the core nodes of the graph, while low-priority knowledge elements are stored as extension nodes associated with the core nodes. This preserves the complete knowledge system without affecting the core query efficiency, optimizes the graph storage structure and query efficiency, and meets the knowledge retrieval needs in different scenarios.

[0029] In this invention, a knowledge element confidence calculation mechanism is introduced during the accurate knowledge element identification process. The final identification result is output through multi-model fusion, and the calculation expression is as follows:

[0030] in For knowledge elements The final confidence level ranges from 0 to 1; The number of models participating in the fusion; For the first The weight coefficients of each model are dynamically calibrated based on the historical accuracy of the model in the identification of knowledge elements in the field of traditional Chinese medicine. The sum of the weights of all models is 1. For the first Each model identifies knowledge elements. The original score; For the first Each model in knowledge element Accuracy correction factor for the category; For knowledge elements Frequency of occurrence in multi-source data; For knowledge elements The domain relevance coefficient is determined by domain experts and ranges from 0.1 to 1.0. Through comprehensive calculation of multi-model performance and data features, the reliability of knowledge element recognition is improved.

[0031] In this invention, a cross-modal context enhancement strategy is adopted in the entity relationship mining process. Visual features in the image are fused with text semantic features and speech-to-text features to generate a multi-dimensional context vector, which is then input into a large language model for relationship extraction. At the same time, a domain relationship template library is constructed, which covers common disease-syndrome-prescription-drug association patterns in traditional Chinese medicine. The template library filters and supplements the model extraction results to achieve full coverage of core relationships, improve the recall and accuracy of relationship extraction, and provide high-quality relationship data for triple construction.

[0032] In this invention, a dynamic adjustment formula for entity relationship weights is introduced during the knowledge fusion process to optimize the fusion effect of multi-source relationships. The calculation expression is as follows:

[0033] in For entities and Relationship between The final weight; Based on the weighting coefficient, Contribution coefficients for multi-source data The relationship link length influence coefficient, the sum of the three is 1; For relationship The initial base weights; The number of data sources supporting this relationship; For the first Relationships in data sources Confidence level; For the first The credibility weight of each data source; For entities and The shortest path length between them; The maximum value of the shortest path length for all entity pairs in the knowledge graph is used to improve the rationality of knowledge fusion through a dynamic balance of basic weights, contributions from multiple data sources, and relational link features.

[0034] In this invention, a two-stage update mechanism is adopted in the dynamic optimization process of the knowledge graph. The first stage is incremental update, which processes new data in real time, extracts knowledge elements and relationships, and judges new nodes or updated attributes through entity similarity threshold. The second stage is full optimization, which performs a global scan of the graph periodically to detect isolated nodes, redundant edges and inconsistent relationships. Graph clustering algorithm is used to optimize node grouping, and low-confidence relationships are eliminated by edge confidence screening, thereby improving the compactness and accuracy of the graph and ensuring that the graph can effectively serve clinical assistance, teaching and inheritance and other scenarios in the long term.

[0035] This invention also discloses a multimodal large language model knowledge graph construction system, comprising: The multimodal data acquisition module integrates a traditional Chinese medicine four diagnostic instruments, an audio and video recording terminal, an electronic medical record interface, and an ancient book scanning device. It supports both real-time acquisition and batch import modes, adapting to the data acquisition needs of different diagnosis and treatment scenarios. It can collect multi-source data such as traditional Chinese medicine four diagnostic information, consultation audio and video, clinical medical records, and ancient books and has an automatic data format recognition function. The data standardization processing module has built-in desensitization algorithms, format conversion tools, missing value completion modules, and audio and video cutting and transcription units to achieve sensitive data anonymization, multi-format data unification, missing information completion, audio and video to text conversion, and generate standardized multimodal datasets to adapt to subsequent knowledge extraction processes. The multimodal feature fusion module is equipped with a large language model semantic coding unit, a computer vision feature extractor, and a speech feature analyzer. It adopts a cross-modal attention mechanism to fuse text, image, and audio features and outputs a unified-dimensional multimodal feature vector to support knowledge element extraction. The knowledge element extraction and recognition module includes a rule base, a statistical model, and a pre-trained large language model fine-tuning unit. It combines conditional random fields and convolutional neural networks to achieve knowledge element candidate extraction, accurate recognition, and entity normalization, and establishes a unified entity identification system to ensure the consistency of knowledge elements. The entity relationship mining module integrates a large language model relationship extraction unit and a graph neural network association capture unit. It supports relationship extraction and potential association mining based on multimodal context, constructs core triples such as disease-symptom-prescription-drug, labels relationship confidence, and provides relationship data for graph construction. The knowledge fusion and conflict resolution module has entity linking tools, conflict detection rule base and weighted voting unit, which realizes multi-source entity association, attribute and relationship conflict resolution, integrates multi-dimensional knowledge, generates a unified knowledge set and eliminates knowledge redundancy and contradiction; The knowledge graph construction and storage module includes a graph database schema design tool, a data import unit, and an index optimization component. It supports structured storage of entities, relations, and attributes, and uses a graph database to achieve efficient querying and traversal, ensuring stable storage and fast access to the graph. The dynamic optimization and update module has a built-in incremental data processing unit, similarity calculation tool, and graph detection component, which enables real-time processing of new data, dynamic updating of graph nodes and edges, and regular optimization of graph structure to improve graph integrity and consistency. The multimodal retrieval and visualization module supports text keyword retrieval, image feature retrieval, and multi-condition combination retrieval. It is equipped with a graph visualization tool that displays the relationship between disease, syndrome, prescription, and medicine in the form of nodes and edges. It provides graph zooming, traversal, and detail viewing functions to facilitate users to intuitively obtain knowledge.

[0036] This invention also includes a knowledge quality assessment module, which incorporates an integrity assessment unit, an accuracy detection unit, and a consistency verification unit. By calculating indicators such as entity coverage, mean relation confidence, and attribute consistency ratio, it quantifies the quality of the knowledge graph, generates a quality assessment report, provides direction for graph optimization, supports user-defined quality thresholds, triggers an automatic optimization process when quality is not up to standard, and continuously improves the quality of the knowledge graph.

[0037] This invention also includes a domain adaptation and extension module, which has a built-in ontology library, thesaurus, and relation template library for the traditional Chinese medicine domain. It supports users to customize entity types, attributes, and relation types, adapts to the knowledge graph construction needs of different schools of traditional Chinese medicine or medical fields, and provides standardized interfaces to support docking with external clinical systems, teaching platforms, and scientific research databases. This enables cross-system application and data sharing of knowledge graphs, expands the service scenarios and scope of use of the graphs, and enhances the reuse value of knowledge.

[0038] The following two examples further illustrate specific embodiments of the present invention:

[0039] Example 1 Construction and Application of the Experience Inheritance Platform of Famous Veteran Traditional Chinese Medicine Practitioners of the Menghe School of Medicine This embodiment is applied to the digital inheritance scenario of the experience of famous veteran TCM doctors in the Menghe School of Medicine. The core requirement is to integrate multimodal data of diagnosis and treatment of famous veteran TCM doctors, construct an accurate disease-syndrome-prescription-medicine knowledge graph, support immersive learning and experience retrieval for the trainees, form an interactive inheritance model of observation-training-practice-evaluation, and solve the problems of experience loss and inaccurate inheritance in traditional oral transmission.

[0040] The specific implementation process is as follows: In the multimodal data acquisition and standardization stage, a TCM four-diagnosis instrument and an audio-visual recording terminal are integrated into the renowned TCM doctor workstation. Before the consultation, the four-diagnosis instrument collects the patient's facial appearance, tongue appearance, pulse appearance, and constitution identification data to generate digital four-diagnosis information. During the consultation, the terminal simultaneously records the consultation process, automatically segments video clips according to the patient's consultation ID, and transcribes them into text data. At the same time, electronic medical records, laboratory test reports, and ancient books of the Menghe School of Medicine are collected in a targeted manner. Sensitive data such as patient identity are anonymized and desensitized. The similarity interpolation method is used to fill in missing values ​​in the four-diagnosis information. The text content in the ancient books is extracted through optical character recognition technology. The data format is unified into a structured dataset to improve data integrity and consistency.

[0041] In the multimodal feature fusion and knowledge element candidate extraction stage, a pre-trained large language model is used to perform deep semantic encoding on medical record texts, transcribed consultation texts, and ancient literature. Visual features of tongue and facial features are extracted using a convolutional neural network, and features such as tone and speech rate in consultation audio are extracted using speech sentiment analysis technology. A cross-modal attention mechanism is introduced to fuse multimodal feature vectors. Combining a custom-designed rule base specific to the Menghe School of Medicine with a Hidden Markov Model, candidate knowledge elements such as disease syndromes, prescriptions, drugs, and symptoms are initially extracted. The rule base covers exclusive content such as commonly used terminology of the Menghe School of Medicine and classic prescription compatibility rules.

[0042] In the stage of precise knowledge element identification and entity normalization, a large language model is fine-tuned based on the clinical corpus of the Menghe School of Medicine, and the accuracy of entity boundary recognition is optimized by combining conditional random fields and bidirectional long short-term memory networks. A thesaurus and ontology of the Menghe School of Medicine are constructed, including disease names, drug aliases, and formula variations unique to this school. String similarity matching and semantic alignment techniques are used to normalize candidate knowledge elements, eliminating ambiguities such as different names for the same disease and different diseases with the same name. A unique identifier is assigned to each normalized entity to ensure the consistency of knowledge elements.

[0043] In the entity relation mining and triple construction stage, based on multimodal contextual semantic information, the relation extraction capabilities of large language models are utilized, combined with graph neural networks to capture potential associations between entities. The focus is on constructing core relation triples such as disease-syndrome-prescription-drug, symptom-syndrome, and drug-prescription. Each triple is labeled with its relation type, confidence level, and data source, including clinical records of renowned traditional Chinese medicine practitioners, ancient literature, and clinical cases, ensuring the traceability of the relations.

[0044] In the knowledge fusion and conflict resolution stage, entity linking technology is used to associate the same entity in multi-source data, establishing an entity attribute mapping table that covers attributes such as the entity's properties, flavors, meridian tropism, efficacy, and indications. Attribute and relationship conflicts are detected through rule matching and machine learning algorithms, such as differences in the drug composition of the same prescription in different ancient books. Conflicts are resolved using a weighted voting mechanism and semantic similarity calculation, integrating multi-dimensional knowledge to generate a unified knowledge set free of contradictions.

[0045] In the structured construction and storage phase of the knowledge graph, a schema for a graph database in the field of Traditional Chinese Medicine (TCM) was designed. Entity types, attribute fields, and relation types were defined. Entity types include diseases, syndromes, prescriptions, drugs, symptoms, and constitutions. Attribute fields cover basic entity information, clinical application scenarios, and relevant cases. Normalized entities, triples, and attribute information were imported into the graph database. The index structure was optimized to support efficient querying and traversal of entity, relation, and attribute information, facilitating the retrieval of specific knowledge associations by students.

[0046] During the dynamic optimization and incremental update phase of the knowledge graph, an incremental data acquisition channel is established to receive new diagnostic and treatment data from renowned traditional Chinese medicine practitioners and the latest research literature in real time. The above steps are repeated to extract new knowledge elements and relationships. Entity similarity calculations and relationship confidence assessments are used to update the graph node attributes and edge information. The completeness and consistency of the graph are periodically checked, and graph structure optimization algorithms are used to remove redundant edges and isolated nodes, improving graph retrieval and reasoning performance and ensuring the timeliness and accuracy of inherited knowledge.

[0047] Table 1 Comparison of the Effects of Experience Inheritance in the Menghe School of Medicine

[0048] Table 1 clearly demonstrates the core advantages of this invention in the inheritance of the experience of renowned traditional Chinese medicine practitioners. Traditional oral transmission relies on human memory and comprehension, which is prone to errors in knowledge element identification, incomplete relationship mining, and inefficient retrieval of inheritance archives, relying entirely on literature review. Furthermore, knowledge updates lag behind clinical practice. This invention, through multimodal data integration and intelligent algorithms, significantly improves the accuracy of knowledge element identification and the completeness of relationship mining. The structured knowledge graph significantly enhances retrieval efficiency, experience reuse does not require expert presence, and the incremental update mechanism ensures knowledge keeps pace with clinical progress. This effectively addresses the pain points of experience loss, inaccurate transmission, and difficulty in reuse in traditional inheritance, providing efficient and precise learning support for trainees.

[0049] Example 2 Application of Knowledge Graphs in Traditional Chinese Medicine Clinical Decision Support Systems This embodiment is applied to the clinical decision support scenario of TCM in primary hospitals. The core requirement is to provide primary physicians with disease-syndrome-prescription-drug association recommendations, treatment plan references and literature support based on multimodal knowledge graphs, so as to improve the accuracy and standardization of TCM diagnosis and treatment at the primary level and solve the problems of insufficient experience and limited treatment ideas of primary physicians.

[0050] The specific implementation process is as follows: In the multimodal data acquisition and standardization stage, the system integrates TCM diagnostic instruments, electronic medical record interfaces, and laboratory equipment data ports to collect data such as outpatient diagnostic information, electronic medical records, and laboratory test reports. Simultaneously, it includes classical TCM literature and clinical treatment guidelines. The collected data undergoes anonymization to remove patient privacy information. Missing values ​​in the diagnostic data are filled using similarity interpolation. The text, image, and audio data formats are standardized. Key indicators from laboratory reports are extracted using optical character recognition technology to generate a standardized multimodal dataset suitable for subsequent knowledge processing.

[0051] In the multimodal feature fusion and knowledge element candidate extraction stage, a pre-trained large language model is used to semantically encode electronic medical records and treatment guidelines. Visual features such as tongue and pulse images are extracted using a convolutional neural network. Combined with the numerical features of the test data, a cross-modal attention mechanism is used to fuse multi-dimensional feature vectors. A rule base and a hidden Markov model commonly used in primary care clinical practice are constructed. The rule base covers the diagnosis and treatment guidelines and drug incompatibilities for common diseases. Initial extraction of knowledge element candidates, such as disease syndromes, prescriptions, drugs, symptoms, and test indicators, is performed to ensure that the knowledge elements meet the needs of primary care clinical practice.

[0052] In the stage of precise knowledge element identification and entity normalization, a large language model is fine-tuned based on primary clinical corpora. Entity boundary recognition is optimized by combining conditional random fields and bidirectional long short-term memory networks, with a focus on improving the accuracy of knowledge element identification for common and frequently occurring diseases. A TCM clinical thesaurus and ontology are constructed, including commonly used primary care terms, generic and alternative names of drugs, and standard names of prescriptions. Knowledge element candidates are normalized using string similarity matching and semantic alignment techniques to eliminate entity ambiguity, assign unique entity identifiers, and ensure consistency with entities in the knowledge graph.

[0053] In the entity relation mining and triple construction stage, based on multimodal contextual information, the relation extraction capabilities of large language models and graph neural networks are utilized to mine core relations such as disease-symptom, symptom-syndrome, syndrome-prescription, prescription-drug, and drug-lab indicator, constructing clinically oriented triples. Relationship types, confidence levels, and supporting evidence are labeled, including treatment guidelines, classic literature, and clinical cases from renowned traditional Chinese medicine practitioners, providing traceable support for primary care physicians' diagnostic and treatment references.

[0054] In the knowledge fusion and conflict resolution phase, entity linking technology is used to connect patient data with entities in the knowledge graph, establishing an attribute mapping table that covers key diagnostic and treatment points, drug dosages, contraindications, and other information for each entity. Attribute and relationship conflicts are detected through rule matching and machine learning algorithms, such as differences in prescription recommendations for the same syndrome across different guidelines. Conflicts are resolved using weighted voting mechanisms and semantic similarity calculations, integrating the knowledge into a unified set that aligns with the realities of primary care clinical practice, thus avoiding misleading primary care physicians.

[0055] In the structured construction and storage phase of the knowledge graph, a graph database schema adapted to primary care clinical practice was designed. Complex attribute fields were simplified, core diagnostic and treatment information was highlighted, and entity types were defined to include diseases, syndromes, prescriptions, drugs, symptoms, and test indicators. Relationship types focused on direct clinical associations. Normalized entities, triples, and attribute information were imported into the graph database, and the index structure was optimized to support rapid querying of disease-related syndromes, prescriptions, drugs, and precautions, thus meeting the efficient diagnostic and treatment needs of primary care clinical practice.

[0056] During the dynamic optimization and incremental update phase of the knowledge graph, an incremental data collection channel is established to receive new diagnosis and treatment data and the latest clinical guidelines updates from primary hospitals in real time. The above steps are repeated to extract new knowledge elements and relationships. Entity similarity calculations and relationship confidence assessments are used to update the graph node attributes and edge information, with a focus on optimizing the diagnosis and treatment associations for common diseases. The accuracy of the graph in primary clinical applications is regularly tested, and graph structure optimization algorithms are used to adjust the graph structure based on physician feedback, improving the practicality and accuracy of decision support.

[0057] Table 2 Comparison of Auxiliary Effects of Traditional Chinese Medicine Clinical Diagnosis and Treatment in Primary Care

[0058] Table 2 data highlights the application value of this invention in primary care clinical decision support. In traditional treatment methods, primary care physicians rely on personal experience, resulting in low diagnostic accuracy and poor prescription compatibility. Treatment plans lack standardization, and obtaining literature support requires manual research, leading to low efficiency and slow learning progress. This invention integrates massive amounts of clinical knowledge through a multimodal knowledge graph, providing primary care physicians with accurate diagnostic references and prescription recommendations. This ensures that treatment plans comply with standards, and literature support is readily available, significantly improving the convenience and accuracy of diagnosis and treatment. Simultaneously, the dynamic updating of the graph, combined with clinical practice, helps physicians quickly accumulate experience and improve their professional level, effectively addressing the pain points of insufficient experience and lack of standardization in primary care TCM diagnosis and treatment, and providing strong support for improving the quality of primary healthcare.

[0059] rely on Figure 3 This bar chart visually demonstrates the core advantages of this invention in identifying knowledge elements from different schools of traditional Chinese medicine. Traditional identification methods rely on manual annotation and general models, failing to consider the unique terminology systems and disease naming habits of each school, resulting in an accuracy rate generally below 70%. Furthermore, the differences between schools are irregular and cannot meet the need for precise transmission of school experience. This invention fine-tunes a large language model for each school's specific corpus, constructs a thesaurus and ontology for each school, and eliminates ambiguities such as "different names for the same disease" through semantic alignment, increasing the accuracy of knowledge element identification for each school to over 90%. This adaptive identification method overcomes the limitations of general models in processing school-specific knowledge, ensuring that the core diagnostic and treatment knowledge of different schools can be accurately extracted, thus laying a solid data foundation for the digital transmission of experience from each school.

[0060] rely on Figure 4 The line graph clearly demonstrates the significant improvement in knowledge graph construction efficiency achieved by this invention. Traditional construction processes are fragmented, with data standardization relying on manual format conversion, and feature fusion and knowledge element recognition requiring multiple manual interventions. This results in each step taking over 20 minutes, leading to overall inefficiency and an inability to meet the rapid knowledge updates required in clinical and traditional settings. This invention, through an automated data standardization module, a cross-modal attention mechanism feature fusion algorithm, and a multi-model fusion knowledge element recognition strategy, automates and parallelizes each step, reducing the time required for each step to 5-8 minutes. This efficient construction process significantly shortens the time from data collection to usability of the knowledge graph, enabling rapid responses to newly added clinical experience from renowned traditional Chinese medicine practitioners and the latest needs of grassroots clinical practice, ensuring the timeliness of the knowledge graph.

[0061] rely on Figure 5This radar chart comprehensively demonstrates the integrated advantages of this invention in primary care clinical decision support scenarios. Traditional diagnosis and treatment methods rely on the personal experience of primary care physicians, achieving a diagnostic accuracy rate of over 70%. However, they lack sufficient adaptability of prescriptions and standardization of treatment plans. Furthermore, literature support requires manual review, resulting in extremely low efficiency in experience reuse. Performance across various dimensions is uneven, and the overall level is low. This invention integrates massive amounts of clinical knowledge based on a multimodal knowledge graph, providing primary care physicians with accurate diagnostic references, suitable prescription recommendations, and standardized treatment plans. It also allows for one-click retrieval of relevant literature support, significantly improving effectiveness across all dimensions. In particular, the convenience of literature support and the efficiency of experience reuse are improved to approximately 95%, addressing the pain points of insufficient resources and difficulty in knowledge acquisition for primary care physicians, and achieving comprehensive optimization of effectiveness across all dimensions.

[0062] rely on Figure 6 This bar chart highlights the core value of this invention in incremental knowledge graph updates. Traditional update methods require a complete reconstruction of the entire graph, with update time increasing linearly and rapidly with the amount of data; updating 5000 data entries takes up to 65 minutes, which cannot meet the needs of continuously adding clinical data and rapid knowledge iteration. This invention adopts an incremental update mechanism, extracting knowledge elements, mining relationships, and updating the graph only for newly added data, without needing to reconstruct the entire graph. The time increase is gradual, with updating 5000 data entries taking only 11 minutes. This efficient update mode ensures that the knowledge graph can absorb new clinical experience from renowned traditional Chinese medicine practitioners, the latest clinical cases from grassroots levels, and the latest research results in traditional Chinese medicine in real time, always maintaining the cutting-edge nature and practicality of knowledge.

[0063] The above are merely preferred embodiments of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.

Claims

1. A method for constructing a knowledge graph of a multimodal large language model, characterized in that, include: The multimodal data acquisition and standardization process involves: acquiring multimodal data, desensitizing sensitive data, standardizing text, image, and audio / video data formats, using similarity interpolation to fill in missing values, segmenting audio / video data by patient visit ID and transcribing it into text, extracting text information from ancient books and medical records using optical character recognition technology, and generating a standardized multimodal dataset with a unified structure and complete content. The steps of multimodal feature fusion and knowledge element candidate extraction are as follows: deep semantic encoding of text data is performed using a pre-trained large language model; visual features such as tongue image are extracted through convolutional neural network; audio features are extracted using speech sentiment analysis technology; a cross-modal attention mechanism is introduced to fuse multimodal feature vectors; and knowledge element candidates are initially extracted by combining a custom rule base and a hidden Markov model. The steps of knowledge element accurate identification and entity normalization are as follows: a large language model is fine-tuned based on the corpus of traditional Chinese medicine, and the entity boundary recognition accuracy is optimized by combining conditional random fields and bidirectional long short-term memory networks. A thesaurus and ontology library of traditional Chinese medicine are constructed. The knowledge element candidates are normalized by string similarity matching and semantic alignment technology to eliminate entity ambiguity and assign unique entity identifiers. The entity relationship mining and triple construction steps are based on multimodal contextual semantic information, utilize the relationship extraction capability of large language models, combine graph neural networks to capture potential associations between entities, construct core relationship triples, and label relationship types, confidence levels and supporting data sources. The knowledge fusion and conflict resolution steps involve using entity linking technology to associate the same entity in multi-source data, establishing an entity attribute mapping table, detecting attribute and relationship conflicts through rule matching and machine learning algorithms, resolving conflicts using weighted voting mechanisms and semantic similarity calculations, integrating multi-dimensional knowledge, and generating a unified knowledge set without contradictions. The steps for constructing and storing a structured knowledge graph include designing a graph database schema for the field of Traditional Chinese Medicine, defining entity types, attribute fields, and relation types, importing normalized entity, triple, and attribute information into the graph database, establishing a structured knowledge graph containing nodes, edges, and attributes, optimizing the graph database index structure, and supporting efficient querying and traversal of entities, relations, and attributes. The knowledge graph dynamic optimization and incremental update steps involve establishing an incremental data acquisition channel to receive new diagnosis and treatment data and literature data in real time, repeating the above steps to extract new knowledge elements and relationships, updating the graph node attributes and edge information through entity similarity calculation and relationship confidence evaluation, periodically checking the graph integrity and consistency, and using graph structure optimization algorithms to improve the graph retrieval and reasoning performance.

2. The method for constructing a multimodal large language model knowledge graph according to claim 1, characterized in that, It also includes a multimodal data quality assessment step, constructing a data quality assessment index system, designing quantitative indicators from four dimensions: completeness, accuracy, consistency, and timeliness, calculating data quality scores by weighting the indicators, and triggering a secondary processing flow for data below the set threshold, re-standardizing and completing the data to improve the quality of the input dataset and provide reliable data support for subsequent knowledge extraction and graph construction.

3. The method for constructing a multimodal large language model knowledge graph according to claim 1, characterized in that, It also includes a knowledge element priority sorting step. Based on the importance of knowledge elements in the diagnosis and treatment scenario, data support, and expert recognition, a priority evaluation model is built to sort the extracted knowledge elements. High-priority knowledge elements are prioritized to be included in the core nodes of the graph, while low-priority knowledge elements are stored as extension nodes associated with the core nodes. This preserves the complete knowledge system without affecting the core query efficiency, optimizes the graph storage structure and query efficiency, and meets the knowledge access needs in different scenarios.

4. The method for constructing a multimodal large language model knowledge graph according to claim 1, characterized in that, The knowledge element confidence calculation mechanism is introduced into the accurate knowledge element identification process. The final identification result is output through multi-model fusion, and the calculation expression is as follows: in For knowledge elements The final confidence level, ranging from 0 to 1; The number of models participating in the fusion; For the first The weight coefficients of each model are dynamically calibrated based on the historical accuracy of the model in the identification of knowledge elements in the field of traditional Chinese medicine. The sum of the weights of all models is 1. For the first Each model identifies knowledge elements. The original score; For the first Each model in knowledge element Accuracy correction factor for the category; For knowledge elements Frequency of occurrence in multi-source data; For knowledge elements The domain relevance coefficient is determined by domain experts and ranges from 0.1 to 1.

0.

5. The method for constructing a multimodal large language model knowledge graph according to claim 1, characterized in that, In the entity relationship mining process, a cross-modal context enhancement strategy is adopted to fuse visual features in images with text semantic features and speech-to-text features to generate multi-dimensional context vectors. These vectors are then input into a large language model for relationship extraction. Simultaneously, a domain relationship template library is constructed, which covers common disease-syndrome-prescription-drug association patterns in traditional Chinese medicine. This library filters and supplements the model's extraction results, achieving full coverage of core relationships and improving the recall and accuracy of relationship extraction. This provides high-quality relationship data for triple construction.

6. The method for constructing a multimodal large language model knowledge graph according to claim 1, characterized in that, A dynamic adjustment formula for entity relationship weights is introduced during knowledge fusion to optimize the fusion effect of multi-source relationships. The calculation expression is as follows: in For entities and Relationship between The final weight; Based on the weighting coefficient, Contribution coefficients for multi-source data The relationship link length influence coefficient, the sum of the three is 1; For relationship The initial base weights; The number of data sources supporting this relationship; For the first Relationships in data sources Confidence level; For the first The credibility weight of each data source; For entities and The shortest path length between them; The maximum value of the shortest path length for all entity pairs in the knowledge graph is obtained through a dynamic balance of basic weights, contributions from multiple data sources, and relational link features.

7. The method for constructing a multimodal large language model knowledge graph according to claim 1, characterized in that, The knowledge graph dynamic optimization process adopts a two-stage update mechanism. The first stage is incremental update, which processes new data in real time, extracts knowledge elements and relationships, and determines whether to add nodes or update attributes through entity similarity threshold. The second stage is full optimization, which periodically performs a global scan of the graph to detect isolated nodes, redundant edges and inconsistent relationships, and uses graph clustering algorithm to optimize node grouping.

8. A system for constructing a multimodal large language model knowledge graph according to any one of claims 1-7, characterized in that, include: The multimodal data acquisition module integrates a traditional Chinese medicine diagnostic instrument, an audio and video recording terminal, an electronic medical record interface, and an ancient book scanning device. It supports both real-time acquisition and batch import modes, adapts to the data acquisition needs of different diagnosis and treatment scenarios, collects multi-source data, and has an automatic data format recognition function. The data standardization processing module has built-in desensitization algorithms, format conversion tools, missing value completion modules, and audio and video cutting and transcription units to achieve sensitive data anonymization, multi-format data unification, missing information completion, audio and video to text conversion, and generate standardized multimodal datasets to adapt to subsequent knowledge extraction processes. The multimodal feature fusion module is equipped with a large language model semantic coding unit, a computer vision feature extractor, and a speech feature analyzer. It adopts a cross-modal attention mechanism to fuse text, image, and audio features and outputs a unified-dimensional multimodal feature vector to support knowledge element extraction. The knowledge element extraction and recognition module includes a rule base, a statistical model, and a pre-trained large language model fine-tuning unit. It combines conditional random fields and convolutional neural networks to achieve knowledge element candidate extraction, accurate recognition, and entity normalization, and establishes a unified entity identification system to ensure the consistency of knowledge elements. The entity relationship mining module integrates a large language model relationship extraction unit and a graph neural network association capture unit. It supports relationship extraction and potential association mining based on multimodal context, constructs core triples, and labels relationship confidence to provide relationship data for graph construction. The knowledge fusion and conflict resolution module has entity linking tools, conflict detection rule base and weighted voting unit, which realizes multi-source entity association, attribute and relationship conflict resolution, integrates multi-dimensional knowledge, generates a unified knowledge set and eliminates knowledge redundancy and contradiction; The knowledge graph construction and storage module includes a graph database schema design tool, a data import unit, and an index optimization component. It supports structured storage of entities, relations, and attributes, and uses a graph database to achieve efficient querying and traversal, ensuring stable storage and fast access to the graph. The dynamic optimization and update module has a built-in incremental data processing unit, similarity calculation tool, and graph detection component, which enables real-time processing of new data, dynamic updating of graph nodes and edges, and regular optimization of graph structure to improve graph integrity and consistency. The multimodal retrieval and visualization module supports text keyword retrieval, image feature retrieval, and multi-condition combination retrieval. It is equipped with a graph visualization tool to display the relationship between disease, syndrome, prescription, and medicine in the form of nodes and edges, and provides graph zooming, traversal, and detailed viewing functions.

9. A multimodal large language model knowledge graph construction system according to claim 8, characterized in that, It also includes a knowledge quality assessment module, which has built-in integrity assessment unit, accuracy detection unit, and consistency verification unit. By calculating entity coverage, mean relationship confidence, and attribute consistency ratio indicators, it quantifies the quality of the knowledge graph, generates a quality assessment report, provides direction for graph optimization, supports user-defined quality thresholds, and triggers an automatic optimization process when the quality is not up to standard, so as to continuously improve the quality of the graph.

10. A multimodal large language model knowledge graph construction system according to claim 8, characterized in that, It also includes domain adaptation and extension modules, with built-in TCM domain ontology, thesaurus, and relation template library. It supports user-defined entity types, attributes, and relation types, adapts to the knowledge graph construction needs of different TCM schools or medical fields, and provides standardized interfaces to support docking with external clinical systems, teaching platforms, and scientific research databases, enabling cross-system application and data sharing of knowledge graphs.

Images