Multimodal information extraction and cross-domain fusion method based on dual-domain knowledge sub-graph

Abstract

The application provides a multimodal information extraction and cross-domain fusion method based on a dual-domain knowledge subgraph, belongs to the technical field of knowledge graph construction and multimodal information processing, and collects and pre-processes multimodal data, including cleaning, standardization, enhancement and feature extraction, generates modality embedding by using a pre-training model, and stores the modality embedding in a vector database; a shared semantic space is constructed, the Milvus database is used to store the aligned embedding, a user text or image query is processed, enhanced embedding is generated by analyzing the limitation condition of the text query, and embedding is generated by fusing the text extracted by OCR and the image features through the image query; Top-K candidates are retrieved by condition screening and semantic similarity calculation in the database; and the candidate results are sorted according to multimodal correlation and the results are returned. According to the user query, the application combines natural language processing and OCR technology, generates semantic enhanced query embedding, and accurately matches the candidate results in the shared semantic space through condition screening and similarity calculation.

Description

Technical Field

[0001] This invention belongs to the field of knowledge graph construction and multimodal information processing technology, specifically, it relates to a method for multimodal information extraction and cross-domain fusion based on dual-domain knowledge subgraphs. Background Technology

[0002] With the rapid development of digital humanities and intelligent information processing technologies, the digitization of ancient Chinese medicine texts faces a significant challenge in integrating multi-source, heterogeneous data. The knowledge system of traditional Chinese medicine possesses unique complexity: on the one hand, its knowledge carriers are diverse, including ancient texts, meridian and acupoint diagrams, drug atlases, tongue and pulse diagrams, and audio recordings of medical cases; on the other hand, its knowledge content is rich and layered, covering core TCM relationships such as disease-symptom-treatment and prescription-drug-efficacy, as well as information on textual evolution and academic transmission. Existing technologies struggle to effectively integrate this heterogeneous, multimodal information, resulting in low knowledge retrieval efficiency and limited semantic understanding capabilities.

[0003] In the field of knowledge graph construction, traditional methods typically employ a single knowledge domain modeling approach, storing all knowledge elements uniformly in a flat graph structure. This approach ignores the essential differences between the content domain (core TCM knowledge) and the system domain (document metadata) within the TCM knowledge system, failing to effectively express the complex temporal relationships such as the evolution of ancient text versions and the transmission of academic schools. For example, the *Shanghan Lun*, as a classic of TCM, contains TCM content (such as the formulation of Guizhi Tang) and documentary information (such as the version changes from the Song Dynasty to the Kangping era), which belong to different knowledge dimensions and require differentiated modeling strategies. Furthermore, existing multimodal information extraction technologies suffer from insufficient recognition accuracy and incomplete semantic extraction when handling special content such as traditional Chinese characters and illustrations in ancient texts.

[0004] In the field of multimodal information fusion, existing research mainly focuses on general scenarios such as image-text and video-text, lacking customized fusion strategies tailored to the specific characteristics of specialized domains. For highly specialized and historically significant knowledge resources like ancient Chinese medicine texts, it is necessary to comprehensively consider multiple dimensions such as text semantics and image visual features to construct a more refined knowledge representation and fusion mechanism. Ancient Chinese medicine texts contain a rich variety of illustrations, including drug atlases (depicting the morphology and origin of medicinal materials), acupoint and meridian diagrams (marking the location of acupoints and the direction of meridians), and tongue and pulse diagnosis diagrams (recording diagnostic features). Each image type carries unique TCM knowledge, requiring targeted information extraction methods.

[0005] Feature representation is a fundamental step in knowledge graph construction, aiming to map heterogeneous modal data to a unified semantic space to achieve semantic alignment between different modalities. Considering the multimodal characteristics of data in the field of Traditional Chinese Medicine (TCM), text modalities typically employ pre-trained language models (such as BERT and Qwen) trained on large-scale corpora to capture contextual semantics and generate high-dimensional embedding vectors. In image modalities, visual question-answering models (such as Qwen2.5-VL) are widely used to extract semantic information from images, enabling them to understand image content and answer questions about images. Feature representation of audio modalities relies on speech recognition technology, converting audio signals into text for unified processing. However, modal heterogeneity, semantic gaps, and data sparsity remain major challenges in feature representation. To address these issues, self-supervised learning further enhances the robustness and generalization ability of feature representation by mining semantic features from unlabeled data.

[0006] After completing the feature representation of multimodal data, the next step is to match and fuse features from different modalities. Feature matching methods are generally divided into two types: global feature matching and local feature matching. The goal of global feature matching methods is to embed the overall content into a shared semantic space and calculate cross-modal similarity through similarity metrics. Local feature matching methods, on the other hand, compensate for the shortcomings of global feature matching methods in capturing local details and semantic relationships by matching local regions of an image with words or phrases in text. For traditional Chinese medicine knowledge, it is necessary to consider both global semantics (such as the overall efficacy of a prescription) and local details (such as the compatibility of individual herbs), thus requiring the design of a matching strategy that fuses global and local features.

[0007] In knowledge storage and retrieval, the development of vector database technology has provided effective support for large-scale semantic retrieval. The HNSW (Hierarchical Navigable Small World) index structure, by constructing a multi-layered nearest neighbor graph, achieves efficient approximate nearest neighbor search, capable of retrieving the similarity of millions of vectors within milliseconds. Meanwhile, relational databases (such as PostgreSQL) have advantages in storing structured knowledge and supporting complex queries. How to organically combine vector retrieval with structured queries to build an efficient hybrid storage and retrieval mechanism is a key technical issue in knowledge graph applications.

[0008] Furthermore, Traditional Chinese Medicine (TCM) knowledge exhibits significant temporal evolution. From the *Huangdi Neijing* to the *Shanghan Lun*, from the Four Great Masters of the Jin and Yuan Dynasties to the Wenbing School of the Ming and Qing Dynasties, the theoretical system of TCM has undergone continuous development and evolution. The same concept (such as "wind evil") may have different connotations in different historical periods, and the same prescription (such as "Guizhi Tang") may also have multiple versions. How to effectively represent this temporal evolutionary relationship in a knowledge graph and achieve knowledge traceability is an important requirement for the digitization of TCM knowledge.

[0009] Therefore, designing a knowledge graph construction method that can effectively integrate multimodal information, distinguish different knowledge levels, and achieve cross-domain semantic alignment has become a key technological bottleneck for the digitization and intelligent application of TCM knowledge. Summary of the Invention

[0010] To address the shortcomings of existing TCM knowledge graph construction technologies, such as insufficient multimodal data processing capabilities, unclear knowledge hierarchy distinctions, and difficulties in cross-domain semantic alignment, this invention proposes a method for multimodal information extraction and cross-domain fusion based on a dual-domain knowledge subgraph. By designing a dual-domain knowledge subgraph architecture comprising the TCM knowledge domain and the literature tracing domain, and combining a multimodal large model with vectorized storage technology, this method achieves efficient extraction and cross-domain fusion of multimodal content from ancient TCM texts, thereby constructing a semantically rich and structurally complete TCM knowledge graph.

[0011] This invention is achieved through the following technical solution: A method for multimodal information extraction and cross-domain fusion based on dual-domain knowledge subgraphs: The method specifically includes the following steps: Step 1: Multimodal data acquisition and preprocessing, collecting and standardizing multi-source heterogeneous data from ancient Chinese medicine books; Step 2: Extract multimodal information from the TCM knowledge domain, and conduct in-depth semantic analysis on the core medical entities and relationships in TCM; Step 3: Vectorized storage and index construction of TCM knowledge domain. The extracted medical knowledge is semantically vectorized, a TCM knowledge domain subgraph is constructed, and an index is created. Step 4: Extract structured features from the source domain of ancient books and collect metadata and version information; Step 5: Unstructured feature processing and vectorization of the source domain of literature: Information extraction and semantic representation of the unstructured content of ancient books are performed to construct a subgraph of the source domain of literature. Step 6: Cross-domain fusion alignment and knowledge graph storage, linking and fusing the two knowledge subgraphs from Step 3 and Step 5 and storing them persistently; Step 7: Retrieve Top-K candidates through conditional filtering and semantic similarity calculation; sort the candidate results by multimodal relevance and return the results.

[0012] Further, step 1 includes: Step 1.1: Data source selection and collection. Obtain multimodal TCM ancient book resources from public channels, including ancient book texts, drug atlases, meridian and acupoint diagrams, tongue diagnosis and pulse diagrams, and medical case audio. Step 1.2: Ancient text preprocessing, cleaning the original text to remove scanning noise and watermark interference, and using OpenCC tool to perform traditional and simplified Chinese conversion; Step 1.3: Ancient and Modern Character Comparison Processing: Construct a comparison table of ancient and modern characters in the field of Traditional Chinese Medicine, mapping ancient characters to modern standard characters; Step 1.4: Image data preprocessing, uniformly adjust image size, perform image pixel normalization, and enhance low-quality images; Step 1.5: Audio data preprocessing, including noise removal from the audio signal, and standardization of sampling rate and audio format; Step 1.6: Data quality verification. Establish a data quality assessment index system to verify the completeness, accuracy, and consistency of the preprocessed data.

[0013] Further, step 2 includes: Step 2.1: Extracting triples from ancient texts. Using the Qwen2.5-VL multimodal large model, we perform deep semantic analysis on the content of ancient texts, extract entity types and relation types, and construct core medical relation triples such as disease-symptom-treatment and prescription-drug-efficacy. Step 2.2: Extraction of drug image information. PaddleOCR is used to recognize text information in the image and extract text labels including drug name, place of origin, and properties. Qwen2.5-VL is used for visual question answering to identify drug morphological features. Step 2.3: Extract information from acupoint and meridian diagrams. Use Qwen2.5-VL for visual question answering to extract structured information including acupoint location, meridian affiliation, therapeutic effects, acupuncture depth, and contraindications. Step 2.4: Extracting information from the tongue diagnosis and pulse image, including the following diagnostic features: tongue color, tongue shape, tongue coating, tongue posture, pulse position, pulse rate, pulse strength, and pulse shape. Step 2.5: Extract audio information from medical records. Use the Vosk speech recognition model to convert the audio content into text, and then extract the triples. Step 2.6: Multimodal information fusion, which integrates information extracted from different modalities to establish cross-modal entity alignment relationships.

[0014] Furthermore, in step 2.1, the extracted entity types include: disease name, symptoms, prescription, medicine, acupoint, meridian, and physician. The types of relationships include: treatment method, compatibility, meridian tropism, indications, contraindications, etiology and pathogenesis, symptom presentation and composition.

[0015] Furthermore, step 3 includes: Step 3.1: Text semantic vectorization. Use the Bge-Embedding model to semantically vectorize all extracted text content, generating a 1024-dimensional dense vector representation; Step 3.2: Entity Vector Representation. Generate an independent vector representation for each medical entity, integrating multi-dimensional information such as entity name, attribute description, and relational context. Step 3.3: Relation vector representation, which involves vectorizing the extracted relation types; Step 3.4: Vector normalization processing, perform L2 normalization processing on all generated vectors; Step 3.5: HNSW index construction. The vector is organized using the HNSW index structure to achieve approximate nearest neighbor search. Step 3.6: Storage of TCM knowledge domain subgraph. Entities, relations and their vector representations are stored in the TCM knowledge domain subgraph to establish an entity-relation-entity graph structure.

[0016] Furthermore, step 4 includes: Step 4.1: Extract version information, extract scalar fields, including version type, printing year, location of the copy, and version number; Step 4.2: Extract document attributes, extract categorization fields, including author, year of publication, medical school, book category, and number of volumes and articles; Step 4.3: Numerical data extraction, extracting statistical information, including the number of chapters, the number of prescriptions, the number of drugs, the frequency of citations, and the number of times a document is cited; Step 4.4: Extract time-series data, including the date of publication, timeline of version evolution, and timeline of academic inheritance; Step 4.5: Version lineage construction, analyze the source and flow relationships between different versions of the same work, and construct a version evolution lineage diagram; Step 4.6: Academic School Affiliation. Based on the characteristics of the document's content and the author's mentorship, determine the academic school affiliation of the document.

[0017] Furthermore, step 5 includes: Step 5.1: Processing of prefaces and epigraphs: Use Qwen2.5-VL to extract semantic information, which includes the author's thoughts, the background of the book's creation, the writing motivation, and the academic origins. Step 5.2: Commentary and evaluation process: extract the viewpoints of later physicians, analyze the stance of the annotations, and construct an academic evaluation network; Step 5.3: Version difference description processing, extract the version differences between different versions, the version differences include textual proofreading information, record variant texts, errors, redundant texts, missing texts and reversed texts; Step 5.4: Citation relationship processing, analyzing the citation types between ancient books, including the relationship of quoting classics, and identifying direct citation, indirect citation, adaptation, criticism and elaboration; Step 5.5: Vectorize the document content. Use the Bge model to generate semantic vectors for unstructured text, which includes prefaces, postscripts, annotations, and collation notes. Step 5.6: Document source domain subgraph storage, storing document entities, version relationships, citation relationships and their vector representations into the document source domain subgraph.

[0018] Furthermore, step 6 includes: Step 6.1: Entity linking, establishing entity correspondence between the two knowledge subgraphs: the TCM knowledge domain subgraph and the literature tracing domain subgraph, including candidate generation based on string matching, candidate ranking based on semantic vectors, and comprehensive scoring by fusing multiple features; Step 6.2: Temporal correlation, establishing a temporal dimension correlation between TCM knowledge and literature versions, including constructing a timestamp index and a temporal evolution diagram; Step 6.3: Entity ambiguity resolution, addressing entity ambiguity issues in the cross-domain fusion process by employing a context-aware entity disambiguation method; Step 6.4: Relationship Conflict Handling. This step addresses conflicts between knowledge sources from different sources by employing a credibility-based knowledge fusion strategy. Step 6.5: Knowledge graph schema design. Design the overall schema structure of the knowledge graph, including core data tables such as entity table, relation table, cross-domain link table, version table, time series table, and variant text table. Step 6.6: Persistent storage of the knowledge graph. The merged dual-domain knowledge graph is stored in a PostgreSQL database, and pgvector is used to extend support for vector storage and retrieval.

[0019] A multimodal information extraction and cross-domain fusion system based on dual-domain knowledge subgraphs; The system includes a multimodal data acquisition and preprocessing module, a multimodal information extraction module for the TCM knowledge domain, a vectorized storage and index construction module for the TCM knowledge domain, a structured feature extraction module for the literature tracing domain, an unstructured feature processing and vectorization module for the literature tracing domain, a cross-domain fusion alignment and knowledge graph storage module, and a filtering and output module. The multimodal data acquisition and preprocessing module collects and standardizes multi-source heterogeneous data from ancient Chinese medicine books. The TCM knowledge domain multimodal information extraction module performs in-depth semantic analysis on core TCM medical entities and relationships. The TCM knowledge domain vectorization storage and indexing construction module performs semantic vectorization on the extracted medical knowledge, constructs a TCM knowledge domain subgraph, and establishes an index. The document tracing domain structured feature extraction module is used to collect ancient book metadata and version information; The document tracing domain unstructured feature processing and vectorization module extracts information and semantically represents the unstructured content of ancient books, and constructs a document tracing domain subgraph. The cross-domain fusion alignment and knowledge graph storage module associates and fuses the two knowledge subgraphs from steps 3 and 5 and stores them persistently. The filtering output module retrieves Top-K candidates through conditional filtering and semantic similarity calculation; it then sorts the candidate results by multimodal relevance and returns the results.

[0020] A computer device system includes a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of the above-described method. A computer-readable storage medium having a computer program / instructions stored thereon, which, when executed by a processor, implement the steps of the above-described method.

[0021] A computer program product includes a computer program / instructions that, when executed by a processor, implement the steps of the method described above.

[0022] Compared with the prior art, the beneficial effects of the present invention are as follows: This invention provides a method for multimodal information extraction and cross-domain fusion based on a dual-domain knowledge subgraph. It achieves dual-domain hierarchical modeling of Traditional Chinese Medicine (TCM) knowledge, effectively expressing the hierarchical structure of the TCM knowledge system by distinguishing between the TCM knowledge domain and the literature source domain, thus solving the problem that traditional flat knowledge graphs struggle to represent complex knowledge relationships. Simultaneously, this invention integrates a multimodal large-scale model with traditional OCR technology, enabling high-precision information extraction from various modalities such as ancient texts, illustrations, and audio, significantly improving the coverage and accuracy of digitized TCM ancient texts. Furthermore, this invention designs a complete cross-domain fusion alignment mechanism, achieving the organic integration of TCM knowledge and literature metadata through entity linking, temporal association, and conflict resolution methods, supporting knowledge source tracing and evolutionary analysis. In terms of storage and retrieval, this invention adopts a storage method combining semantic vectorization and HNSW indexing, significantly improving the retrieval efficiency of large-scale knowledge graphs while ensuring semantic retrieval accuracy, providing effective technical support for the intelligent management and knowledge services of TCM ancient texts. Attached Figure Description

[0023] Figure 1 A flowchart illustrating a unified indexing and semantic alignment method for multimodal data; Figure 2 A unified indexing and semantic alignment framework diagram for multimodal data; Figure 3 The flowchart shows a method for multimodal information extraction and cross-domain fusion based on dual-domain knowledge subgraphs. Figure 4 It is a framework diagram of a multi-modal information extraction and cross-domain fusion method based on a dual-domain knowledge subgraph. Specific implementation manners

[0024] Next, the technical solutions in the embodiments of the present invention will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without making creative efforts belong to the scope of protection of the present invention.

[0025] The experimental methods used in the following embodiments are all conventional methods unless otherwise specified. The materials, reagents, methods, and instruments used are all conventional materials, reagents, methods, and instruments in this field unless otherwise specified, and those skilled in the art can obtain them through commercial channels.

[0026] This method is divided into three core parts: the construction method of the traditional Chinese medicine knowledge domain subgraph, the construction method of the literature tracing domain subgraph, and the cross-domain fusion alignment method As Figures 1 to 4 shown, the present invention provides a multi-modal embedding unified index and semantic alignment method, which specifically includes the following steps: Step 1: Multi-modal (text, image, video, audio) data collection and preprocessing (cleaning, standardization, enhancement, and feature extraction), and collect and standardize the multi-source heterogeneous data of traditional Chinese medicine ancient books. Step 1.1: Data source selection and collection. Obtain traditional Chinese medicine ancient book resources from channels such as public data sets, ancient book digitization platforms, and medical school libraries, including multi-modal data such as ancient book texts, drug atlases, meridian acupoint maps, tongue diagnosis and pulse condition maps, and medical case audios, ensuring that the data sources are legal and representative. In this embodiment, the data collection scope includes: "Treatise on Febrile Diseases" and its main annotations (Cheng Wuji's "Annotations on Treatise on Febrile Diseases", Ke Yunbo's "Collection of Treatises on Febrile Diseases" etc.), the drug atlas of "Compendium of Materia Medica", the acupoint map of "A-B Classic of Acupuncture and Moxibustion", the pulse condition map of "Pulse Theory of Binhu" etc. Let the collected data set be D, which includes three subsets: a text data set, an image data set, and an audio data set.

[0027] Step 1.2: Preprocessing of ancient book texts. Clean the original text to remove irrelevant information such as scanning noise and watermark interference; standardize traditional Chinese characters and variant characters, and use the OpenCC tool for traditional-to-simplified conversion, and retain the original traditional form as metadata. Taking the original text of "Treatise on Febrile Diseases" as an example, the conversion process is as follows: "傷寒論" is converted to "伤寒论", "風寒" is converted to "风寒", and "營衛" is converted to "营卫".

[0028] Step 1.3: Processing of ancient and modern Chinese character comparison. Construct a comparison table of ancient and modern Chinese characters in the field of traditional Chinese medicine, map ancient characters to modern standard characters, and retain the original characters for special traditional Chinese medicine characters (such as "癥瘕", "痃癖", etc.) and add modern medical comparison annotations. The ancient and modern Chinese character comparison dictionary contains the following mapping relationships: "反" corresponds to "返", "閒" corresponds to "间", "采" corresponds to "採", "泄" corresponds to "瀉", "欬" corresponds to "咳", "鬲" corresponds to "膈", "胷" corresponds to "胸", "脣" corresponds to "唇", "齗" corresponds to "龈", "痓" corresponds to "痉", etc.

[0029] For special traditional Chinese medicine terms, establish an annotation structure containing four fields: the original character, pinyin, modern equivalent term, and definition. Taking "癥瘕" as an example: the original character is "癥瘕", the pinyin is "zhēng jiǎ", the modern equivalent term is "abdominal mass", and the definition is "a disease syndrome of internal abdominal lumps, where 癥 is tangible and can be identified, and 瘕 is characterized by being scattered and irregular".

[0030] Step 1.4: Preprocessing of image data. Uniformly adjust the image size according to the model requirements, perform image pixel normalization processing, and perform enhancement operations on low-quality images, including denoising, contrast adjustment, sharpening, etc. Let the original image size be , and the target size be . The pixel normalization processing formula is:

[0031] where, is the pixel mean, is the pixel standard deviation. Gaussian filtering is used for denoising:

[0032] where, is the Gaussian kernel function, is the filtering window radius. Step 1.5: Preprocessing of audio data. Remove noise from audio signals such as medical case explanations and famous doctors' medical theories, clean background noise through a filtering denoising algorithm, unify the sampling rate and audio format, and improve the quality of the speech signal. The target sampling rate is set to 16000Hz. Spectral subtraction is used for noise removal:

[0033] where, is the noisy signal spectrum, is the estimated noise spectrum, is the over-subtraction factor, is the estimated clean signal spectrum. The time-domain signal is restored as:

[0034] where, is the phase spectrum of the original signal. Step 1.6: Data Quality Verification. Establish a data quality assessment index system to verify the completeness, accuracy, and consistency of the preprocessed data, remove unqualified data, and ensure the data quality for subsequent processing. The data quality scoring function is defined as:

[0035] in, This is a weighting parameter. In this embodiment, it is set as follows: , , A quality threshold of 0.7 is set; data with a quality score below this threshold is discarded. The completeness metric is calculated as the ratio of filled fields to required fields, while the consistency metric is calculated by comparing data across sources to determine the proportion of consistent attributes to all attributes.

[0036] Step 2: Extract multimodal information from the TCM knowledge domain, process user text or image queries, parse the constraints of text queries to generate enhanced embeddings, and extract text from image queries using OCR and fuse it with image features to generate embeddings. Deep semantic analysis is performed on core TCM medical entities and relationships, specifically including the following steps: Step 2.1: Extraction of Triples from Ancient Texts. Using the Qwen2.5-VL multimodal model, deep semantic analysis is performed on the content of ancient texts to extract entity types and relation types, constructing core TCM relation triples such as disease-symptom-treatment and prescription-drug-effect. A TCM-customized prompt template is designed for the model: "Please extract triples in the form of (entity, relation, entity) from the following TCM text. Entity types include: disease name, symptom, prescription, drug, acupoint, meridian, physician, etc.; relation types include: treatment method, compatibility, meridian tropism, indications, contraindications, etiology and pathogenesis, symptom manifestation, composition, etc." The extracted set of triples is represented as follows:

[0037] in, For the head entity, For tail entities, For relation types, For a collection of entities, It is a set of relations. Taking the original text of "Treatise on Cold Damage" as an example, "The disease of Taiyang is characterized by a floating pulse, stiffness and pain in the head and neck, and aversion to cold," we can extract the following three-element groups: (Taiyang disease, symptoms, floating pulse), (Taiyang disease, symptoms, stiffness and pain in the head and neck), and (Taiyang disease, symptoms, aversion to cold).

[0038] Taking the "Guizhi Tang formula from the Treatise on Cold Damage: Guizhi 3 liang, Shaoyao 3 liang, Gancao 2 liang (prepared), Shengjiang 3 liang (sliced), Dazao 12 pieces (split)" as an example, the following three-element groups were extracted: (Guizhi Tang, composition, Guizhi), (Guizhi Tang, composition, Shaoyao), (Guizhi Tang, composition, Gancao), (Guizhi Tang, composition, Shengjiang), (Guizhi Tang, composition, Dazao), (Guizhi, dosage, 3 liang), (Shaoyao, dosage, 3 liang), (Gancao, dosage, 2 liang), (Gancao, preparation, roasted), (Shengjiang, dosage, 3 liang), (Shengjiang, preparation, sliced), (Dazao, dosage, 12 pieces), (Dazao, preparation, split).

[0039] Step 2.2: Extraction of Drug Image Information. First, PaddleOCR is used to recognize text information in the image, extracting text labels such as drug name, origin, and properties. Then, Qwen2.5-VL is used for visual question answering, recognizing visual information such as drug morphological features (roots, stems, leaves, flowers, fruits, etc.) and processing characteristics. Finally, the OCR results are fused with the visual question answering results. The fusion process is represented as follows:

[0040] in, Image of the drug. This is a template for questions and answers about drug characteristics. The drug characteristic Q&A template includes the following predefined questions: "What are the morphological characteristics of this drug?", "From which part of the body is this drug used in medicine?", "What are the color characteristics of this drug?", "What are the surface texture characteristics of this drug?", and "What are the cross-sectional characteristics of this drug?" Taking the "ginseng" drug atlas in the "Compendium of Materia Medica" as an example, the OCR recognition results include: drug name "ginseng" (confidence 0.98), place of origin "Shangdang, Liaodong" (confidence 0.95), and properties "sweet, slightly bitter, and warm" (confidence 0.93). The visual question-and-answer results include: morphology "the root is cylindrical or spindle-shaped, with many fibrous roots, about 10-20cm long", medicinal part "the root is used", color "the surface is light yellow or light brown", surface characteristics "with longitudinal wrinkles and transverse lenticel-like protrusions", and cross-sectional characteristics "the cross-section is light yellowish-white, powdery, and the cambium rings are obvious".

[0041] Step 2.3: Extraction of acupoint and meridian information. Using Qwen2.5-VL for visual question answering, structured information such as acupoint location, meridian affiliation, therapeutic effects, acupuncture depth, and contraindications is extracted to construct an association network of acupoints, meridians, and therapeutic effects.

[0042] The acupoint information Q&A template includes the following predefined questions: "Where is this acupoint located anatomically?", "Which meridian does this acupoint belong to?", "What are the main indications for this acupoint?", "What are the appropriate needling depth and angle for this acupoint?", and "What are the contraindications or precautions for this acupoint?" The structured representation of acupoint information includes fields such as name, location, meridian, indications, needling depth, and contraindications. Taking the "Zusanli" acupoint as an example, the extracted results are: Name "Zusanli", Location "Three cun below Dubi, one finger-width lateral to the anterior crest of the tibia", Meridian "Stomach Meridian of Foot Yangming", Indications "Stomach pain, vomiting, abdominal distension, diarrhea, dysentery, constipation, mastitis, lower limb weakness and paralysis, edema, mania, and general weakness and emaciation", Needling depth "Perpendicular insertion 1-2 cun", Contraindications "Use with caution in pregnant women".

[0043] Construct a three-element group of acupoint-meridian-treatment association: (Zusanli, meridian, Stomach Meridian of Foot Yangming), (Zusanli, treatment, stomach pain), (Zusanli, treatment, abdominal distension), (Zusanli, treatment, diarrhea), (Stomach Meridian of Foot Yangming, including acupoint, Zusanli).

[0044] Step 2.4: Extraction of Tongue and Pulse Diagnosis Information. Diagnostic features are extracted, including tongue color, shape, coating, and posture; pulse features include pulse location, rate, strength, and shape. A correspondence between diagnostic features and disease patterns is established.

[0045] The tongue diagnosis feature question and answer template includes the following predefined questions: "What is the color of the tongue?", "What are the morphological characteristics of the tongue?", "What is the color and thickness of the tongue coating?", "Are there any abnormalities in the tongue's appearance?", and "What disease or syndrome does this tongue appearance indicate?" The structured representation of tongue diagnosis information includes fields such as tongue color, tongue shape, coating color, coating texture, tongue state, and diagnosis. Taking "pale white tongue with thin white coating" as an example: the tongue color is "pale white", the tongue shape is "plump and tender with teeth marks on the sides", the coating color is "white", the coating texture is "thin", the tongue state is "normal", and the diagnosis is "deficiency of both qi and blood or yang deficiency".

[0046] The pulse characteristic Q&A template includes the following predefined questions: "Is the pulse superficial or deep?", "Is the pulse rate rapid or slow?", "Is the pulse forceful or weak?", "What are the characteristics of the pulse shape?", and "What disease or syndrome does this pulse indicate?" The structured representation of pulse information includes fields such as pulse location, pulse rate, pulse strength, pulse shape, pulse rhythm, and diagnosis. Taking "floating pulse" as an example: the pulse location is "easily felt with light pressure, slightly reduced with heavy pressure", the pulse rate is "normal", the pulse strength is "strong or weak", the pulse shape is "like wood floating on water", the pulse rhythm is "regular", and the diagnosis is "exterior syndrome, or deficiency of yang rising outward".

[0047] Establish a three-factor correspondence between diagnostic features and disease symptoms: (pale white tongue, suggesting deficiency of both qi and blood), (thin white coating, suggesting mild superficial or internal syndrome), (floating pulse, suggesting superficial syndrome), (floating and forceful pulse, suggesting excess syndrome), (floating and weak pulse, suggesting deficiency syndrome).

[0048] Step 2.5: Extraction of audio information from medical records. The Vosk speech recognition model is used to convert audio content such as explanations of medical records and medical theories by famous doctors into text. The recognition results are post-processed and corrected, and then triplet extraction is performed according to the method in Step 2.1.

[0049] Vosk employs an end-to-end acoustic model, with the recognition process based on CTC (Connectionist Temporal Classification) decoding:

[0050] in, For word sequence, To align the path, For mapping to The set of all paths. Post-processing correction includes TCM terminology correction and sentence segmentation optimization, using a TCM terminology dictionary (containing professional terms such as prescription names, drug names, and disease names) for correction. The corrected text is then processed for triple extraction according to the method in step 2.1.

[0051] Step 2.6: Multimodal Information Fusion. Information extracted from different modalities such as text, images, and audio will be integrated to establish cross-modal entity alignment relationships, eliminate duplicate information, and form a unified representation of TCM knowledge.

[0052] Cross-modal entity alignment employs a semantic similarity-based matching method, establishing alignment relationships when the entity similarity exceeds an alignment threshold of 0.85. Entity similarity calculation combines name similarity and attribute similarity.

[0053] in, Here are the weighting parameters. Name similarity is calculated using edit distance:

[0054] Attribute similarity is calculated using the Jaccard coefficient:

[0055] The duplicate information elimination adopts an entity merging strategy to retain the entity representation with the most complete attributes.

[0056] Step 3: Vectorized storage and index construction of TCM knowledge domain. The extracted TCM knowledge is semantically vectorized and an efficient index is built. Modal embeddings are generated using pre-trained models (Sentence-BERT, CLIP, VideoCLIP, OpenL3) and stored in a vector database; a shared semantic space is constructed, cross-modal vector alignment is optimized through contrastive learning, and a modal mapping network is designed to further enhance the embedding projection effect; Specifically, it includes the following steps: Step 3.1: Text Semantic Vectorization. The Bge model is used to semantically vectorize all extracted text content, generating a 1024-dimensional dense vector representation to capture the deep semantic features of the text. The Bge model employs a pre-trained language model with a dual-tower structure. The encoding process is as follows:

[0057] in, The hidden state sequence output by the BERT encoder. This represents the number of tokens. Taking the description of the "Guizhi Tang" formula as an example, the input text is "Guizhi Tang: Guizhi 3 liang, Shaoyao 3 liang, Gancao 2 liang, Shengjiang 3 liang, Dazao 12 pieces, mainly treats Taiyang stroke, relieves muscle tension and releases exterior pathogens, and harmonizes Ying and Wei", generating a 1024-dimensional semantic vector representation.

[0058] Step 3.2: Entity Vector Representation. An independent vector representation is generated for each TCM entity, integrating multi-dimensional information such as entity name, attribute description, and relational context to construct a comprehensive semantic embedding of the entity. The entity vector generation calculation method is as follows:

[0059] in, , , These are the weight parameters. For name vectors, For attribute vectors, This is the context vector (the average of all context text vectors in which the entity appears). Taking the drug entity "Guizhi" as an example, the name vector is generated from "Guizhi", the attribute vector is generated from "its nature is pungent, sweet and warm, it enters the heart, lung and bladder meridians, it induces sweating and relieves muscle tension, and it warms and unblocks the meridians", and the context vector is obtained by averaging the context text vectors of the entity in various documents.

[0060] Step 3.3: Relation Vector Representation. The extracted relation types are vectorized and encoded to support relation reasoning and completion based on semantic similarity. A descriptive text is constructed for each relation type, and a vector is generated: the treatment method relation is described as "representing the treatment method for a disease, such as clearing heat, relieving exterior symptoms, warming the interior, etc."; the composition relation is described as "representing the drug components of a prescription"; the meridian tropism relation is described as "representing the meridian to which the drug or acupoint belongs"; and the main indication relation is described as "representing the main therapeutic symptoms of the drug, prescription, or acupoint."

[0061] Relational reasoning employs the idea of ​​the TransE model for triples. ,satisfy Relation completion is achieved through vector operations, predicting the tail entity as... The closest entity vector. Step 3.4: Vector Normalization. All generated vectors are subjected to L2 normalization to ensure the consistency of the vector space and improve the accuracy of subsequent similarity calculations. The normalization formula is:

[0062] in, Let be the vector dimension. The normalized vector satisfies... Cosine similarity calculation is simplified to inner product:

[0063] Step 3.5: HNSW Index Construction. The aligned embeddings are stored using a Milvus database to construct an HNSW index, supporting efficient querying and dynamic updates; The vectors are organized using the HNSW (Hierarchical Navigable Small World) index structure, and appropriate index parameters are set to achieve efficient approximate nearest neighbor search. In this embodiment, the maximum number of connections per layer is set. Search width during construction . The hierarchical structure of HNSW is defined as follows:

[0064] in, This is the hierarchical multiplier factor. Each node is at the [level]. The number of neighbors of the layer satisfies the constraint: the bottom layer ( Maximum number of connections is The maximum number of connections in other layers is . During retrieval, given the query vector and the number of results returned... The search begins from the entrance point at the highest level, with the exploration factor set to [value missing]. The search complexity is ,in This represents the number of vectors in the index. Step 3.6: Storage of the TCM Knowledge Domain Subgraph. Entities, relations, and their vector representations are stored in the TCM knowledge domain subgraph, establishing an entity-relation-entity graph structure that supports hybrid access of graph traversal queries and semantic retrieval.

[0065] Traditional Chinese Medicine knowledge domain subgraph defined as ,in For a set of entity nodes, Let be the set of relation edges. It is a collection of attributes. The node data structure includes fields such as: Entity ID (primary key), Name, Type, Vector Embedding, Attributes (JSON format), and Creation Time. The edge data structure includes fields such as: Relationship ID (primary key), Source Entity, Target Entity, Relationship Type, Confidence, and Evidence Text.

[0066] Step 4: Extract structured features from the source domain of ancient books, and collect metadata and version information. This includes the following steps: Step 4.1: Version Information Extraction. Extract scalar fields such as version type (printed edition, manuscript, facsimile, movable type, etc.), printing year, location of the copy holder, and version number to establish a unique version identification system. The structured representation of version information includes fields such as version ID, version type, printing year, holding institution, version number, and physical characteristics. Version types include printed edition, manuscript, facsimile, movable type, lithographic, letterpress, and typeset editions.

[0067] Taking the major versions of *Shanghan Lun* as examples: the Song edition has the version ID "SHL_Song_001", is a "printed edition", was printed in 1065, is held by the National Library of China, and is numbered "Ming Dynasty Zhao Kaimei reprinted the Song edition". Its physical characteristics are "ten lines per half page, eighteen characters per line"; the Kangping edition has the version ID "SHL_Kangping_001", is a "manuscript", was printed in 1060, is held by the Cabinet Library of Japan, and is numbered "Kangping Third Year Manuscript". Its physical characteristics are "scroll binding"; the Guilin edition has the version ID "SHL_Guilin_001", is a "manuscript", was printed during the Qing Dynasty, is held by Guangxi University of Chinese Medicine, and is numbered "Guilin Ancient Edition". Its physical characteristics are "thread-bound".

[0068] Step 4.2: Document Attribute Extraction. Extract categorized fields such as author, year of publication, medical school of thought, book category, volume number, and article list to construct a basic attribute profile for the document. The structured representation of document attributes includes fields such as author, year of publication, medical school of thought, book category, volume number, and article list.

[0069] Traditional Chinese medicine schools of thought include the Typhoid Fever School, the Warm Disease School, the Tonifying Earth School, the Nourishing Yin School, the Attacking Pathogen School, the Warming and Tonifying School, and the Integrating School. Books are classified using a four-part system combined with TCM specialty classifications, including categories such as Medical Classics, Materia Medica, Prescriptions, Acupuncture and Moxibustion, Diagnostic Methods, Medical Cases, Health Preservation, and General Knowledge.

[0070] Taking the Treatise on Febrile Diseases as an example: the author is "Zhang Zhongjing", the year of its completion is "around 200-210", the medical school is "the Febrile Disease School", the book is classified as "prescription book", the number of volumes is 10, and the list of chapters includes "Differentiation of Taiyang Disease Pulse and Symptoms and Treatment, Part 1" and "Differentiation of Taiyang Disease Pulse and Symptoms and Treatment, Part 2", etc.

[0071] Step 4.3: Numerical Data Extraction. Extract statistical information such as the number of chapters, prescriptions, drugs, citation frequency, and number of citations to support quantitative analysis and comparative studies of the literature. Taking *Shang Han Lun* as an example: 22 chapters, 113 prescriptions, 82 drugs, 45 citations to other literature, and 15,680 citations to other literature. Taking *Jin Gui Yao Lue* as an example: 25 chapters, 262 prescriptions, 156 drugs, 38 citations to other literature, and 8,920 citations to other literature.

[0072] Step 4.4: Extraction of Time-Series Data. Extract the time of composition, version evolution timeline, and academic transmission sequence, and construct a time-dimensional index of the documents using a combination of timestamps and interval representation. The time representation uses interval format to handle cases where the year is uncertain; for example, the composition time of *Shanghan Lun* is represented as [196, 220] (AD).

[0073] The timeline of version evolution is represented as an ordered sequence. Taking the Treatise on Febrile Diseases as an example: (200 AD, original version) → (280 AD, Wang Shuhe's revised version) → (1060 AD, Kangping version) → (1065 AD, Song version) → (Qing Dynasty, Guilin version).

[0074] The academic lineage is represented as follows: Zhang Zhongjing (200 years) → Wang Shuhe (280 years) → Cheng Wuji (1144 years) → Ke Yunbo (1674 years) → Chen Xiuyuan (1803 years).

[0075] Step 4.5: Version Genealogy Construction. Analyze the origins and development relationships between different versions of the same work, construct a version evolution genealogy, and record the inheritance, differentiation, and merging processes of the versions. A version genealogy is defined as follows: ,in For version node set, This is a set of edges representing version evolution. The types of edges include inheritance, differentiation, merging, copying, and correction. Taking the textual lineage of the *Shanghan Lun* as an example, the textual nodes include: the original text, the Wang Shuhe edited version, the Tang version, the Song version, the Kangping version, the Guilin version, and the Cheng Zhu version. The textual evolution includes: the original text was edited to form the Wang Shuhe edited version; the Wang Shuhe edited version was copied to form the Tang version; the Tang version was reprinted to form the Song version; the Wang Shuhe edited version was copied to form the Kangping version; and the Song version was annotated to form the Cheng Zhu version.

[0076] Step 4.6: Academic School Affiliation. Based on the characteristics of the document's content and the author's mentorship, determine the academic school affiliation of the document and establish a network linking school, document, and physician. Academic school classification is based on content feature matching; calculate the matching degree between the document and the feature sets of each school, and select the school with the highest matching degree.

[0077] Taking the Typhoid Fever School as an example, its characteristic set includes: the Six Channels Differentiation, the Eight Principles Differentiation, the application of classical formulas, and the equal emphasis on cold and heat. A three-element group is established linking the school, its literature, and its physicians: (Typhoid Fever School, representative work, Treatise on Typhoid Fever), (Typhoid Fever School, representative physician, Zhang Zhongjing), (Zhang Zhongjing, author, Treatise on Typhoid Fever), (Cheng Wuji, belongs to, Typhoid Fever School), (Cheng Wuji, annotation, Treatise on Typhoid Fever).

[0078] Step 5: Unstructured feature processing and vectorization of the source domain of ancient books. This involves information extraction and semantic representation of the unstructured content of ancient books, specifically including the following steps: Step 5.1: Processing of prefaces and epigraphs. Use Qwen2.5-VL to extract semantic information such as the author's thoughts, the background of the book's creation, the motivation for writing, and the academic origins, to understand the creative background and academic positioning of the document.

[0079] The prompt for extracting information from prefaces and postscripts is: "Please extract the following information from the following prefaces and postscripts: 1. Author's writing motivation; 2. Historical background of the book's creation; 3. Academic origins and mentorship; 4. Main academic viewpoints; 5. Evaluation of previous works." The structured representation of prefaces and postscripts includes fields such as author's intention, historical background, academic origin, main viewpoints, and evaluation. Taking Cheng Wuji's preface to "Annotations on the Treatise on Febrile Diseases" as an example, the extracted results are: author's intention is "to elucidate the meaning of Zhang Zhongjing so that later scholars can easily understand it"; historical background is "during the period of medical innovation in the Jin and Yuan dynasties, medical theories were actively debated"; academic origin is "inheriting Wang Shuhe's revised version and referring to the annotated versions of various scholars in the Tang and Song dynasties"; main viewpoint is "annotating the Treatise on Febrile Diseases with the theory of the Inner Canon of Medicine, emphasizing the Qi transformation of the Six Channels"; and evaluation is "honoring Zhang Zhongjing as the Sage of Medicine and regarding the Treatise on Febrile Diseases as the standard of medicine".

[0080] Step 5.2: Annotation and Commentary Processing. Extract the annotation viewpoints of later physicians, analyze the stances of the annotations (agreement, questioning, elaboration, supplementation, etc.), construct an academic evaluation network, and track the transmission and debate of academic viewpoints. The structured representation of annotation information includes fields such as annotator, annotated text, viewpoint, stance, and arguments. Annotation stances are categorized into agreement, questioning, elaboration, supplementation, correction, and criticism.

[0081] Take Ke Yunbo's annotation of "Taiyang disease" as an example: the annotator is "Ke Yunbo", the text being annotated is "Taiyang disease is characterized by a floating pulse, headache and neck stiffness and aversion to cold", the viewpoint is "Taiyang governs the exterior and controls Ying and Wei, Taiyang disease is the general term for exterior syndromes", the stance is "elaborate", and the argument is "Taiyang is the first of the six channels, governs the exterior of the body, and Ying and Wei Qi circulate on the exterior...".

[0082] Academic evaluation network is defined as ,in For medical family nodes, To evaluate the relationship edges. Step 5.3: Version Difference Processing. Extract textual proofreading information from different versions, record variant texts, errors, redundant texts, missing texts, and inverted texts, and establish a version comparison table. The structured representation of version differences includes fields such as Difference ID, Version A, Version B, Difference Type, Location, Version A Content, Version B Content, and Proofreading Description. Difference types include variant texts, errors, redundant texts, missing texts, inverted texts, and phonetic loan characters.

[0083] Taking the difference in the dosage of "Gui Zhi" in the Treatise on Cold Damage as an example: the difference ID is "VAR_001", version A is the "Song edition", version B is the "Guilin edition", the difference type is "variant text", the location is "Gui Zhi Tang formula", version A contains "Gui Zhi three liang", version B contains "Gui Zhi four liang", and the collation explanation is "the Song edition and the Kangping edition are consistent, the Guilin edition is uniquely different".

[0084] Step 5.4: Citation Relationship Processing. Analyze the citation relationships between ancient texts, identify different citation types such as direct citation, indirect citation, adaptation, criticism, and elaboration, and construct a bibliographic citation network. The structured representation of citation relationships includes fields such as citing document, cited document, citation type, cited text, context, and location. Citation types include direct citation, indirect citation, adaptation, criticism, elaboration, and annotation.

[0085] Taking the citation of the Treatise on Cold Damage in the Treatise on Warm Diseases as an example: the cited literature is the Treatise on Warm Diseases, the cited literature is the Treatise on Cold Damage, the citation type is "elaboration", the cited text is the Guizhi Tang formula, the context is "Zhongjing's Guizhi Tang, for external symptoms it is to relieve muscle tension and harmonize Ying and Wei, for internal symptoms it is to transform Qi and regulate Yin and Yang", the position is "Article 16 of the Upper Jiao Chapter".

[0086] Literature citation network is defined as Citation network analysis metrics include in-degree (number of citations), out-degree (number of references), and PageRank value:

[0087] in, is the damping coefficient. Step 5.5: Vectorization of Document Content. The Bge model is used to generate semantic vectors from unstructured texts such as prefaces, postscripts, annotations, and collation notes, which are then stored in the knowledge subgraph of the document tracing domain.

[0088] For long texts, a chunked vectorization strategy is used, with a chunk length of 512 characters. Average pooling is employed.

[0089] Step 5.6: Document Source Tracing Domain Subgraph Storage. Document entities, version relationships, citation relationships, and their vector representations are stored in the document source tracing domain knowledge subgraph, establishing a complete document metadata knowledge base.

[0090] The literature tracing domain subgraph is defined as ,in For a set of document entity nodes, Let be the set of relation edges. This is a set of attributes. Node types include literature, version, physician, school of thought, preface, postscript, annotation, etc. Edge types include author, annotation, citation, version evolution, lineage, school of thought, etc. Step 6: Cross-domain fusion alignment and knowledge graph storage. This involves associating and fusing the two knowledge subgraphs and persistently storing them. Specifically, this includes the following steps: Step 6.1: Entity Linking. Establish entity correspondences between the two knowledge subgraphs, associating entities in the TCM knowledge domain with literature sources in the literature tracing domain. The entity linking process includes three steps: candidate generation based on string matching, candidate ranking based on semantic vectors, and comprehensive scoring integrating multiple features.

[0091] Step 6.1.1: Candidate Generation. Generate candidate entity pairs based on string matching and calculate edit distance similarity:

[0092] The candidate threshold is set to 0.6. When the edit distance similarity is greater than this threshold, the entity pair is added to the candidate set.

[0093] Step 6.1.2: Candidate Ranking. Calculate the semantic similarity of candidate entities based on semantic vectors:

[0094] Step 6.1.3: Contextual Similarity Calculation. Compare the context windows of the entities in the original text, with a window radius set to 50 characters:

[0095] Step 6.1.4: Comprehensive Scoring. The final link score is calculated by fusing multiple features:

[0096] in, , , Set the link threshold to 0.8. When the overall score is greater than this threshold, a link relationship will be established. Taking the entity link "Guizhi Tang" as an example, the edit distance similarity is 0.95, the semantic similarity is 0.97, the context similarity is 0.94, and the overall score is [missing information]. The link results are: Link type "Source", Link score 0.957, Confidence "High". Step 6.2: Temporal Correlation. Establish temporal correlations between TCM knowledge and literature versions, track the historical evolution of knowledge, including constructing timestamp indexes and temporal evolution diagrams to support knowledge tracing and evolutionary analysis.

[0097] Step 6.2.1: Timestamp Index Construction. Add timestamps to each knowledge entity, including the earliest mention time, most recent mention time, frequency by time period, and source timeline information. Taking "Guizhi Tang" as an example: the earliest mention time is "around 200 years (Eastern Han Dynasty)," the most recent mention time is "2024 (modern times)," and the frequency by time period is: 1 time in the Han Dynasty, 3 times in the Wei and Jin Dynasties, 8 times in the Sui and Tang Dynasties, 25 times in the Song, Jin, and Yuan Dynasties, 48 ​​times in the Ming Dynasty, 86 times in the Qing Dynasty, and 350 times in modern times.

[0098] Step 6.2.2: Constructing the Time-Series Evolution Diagram. This represents the changes in knowledge concepts across different historical periods. The time-series evolution diagram is defined as follows: ,in For entity nodes with timestamps, This refers to the temporal evolution edge. Evolution types include conceptual development, theoretical expansion, application expansion, name change, and divergence of viewpoints. Taking the academic evolution of "Guizhi Tang" as an example: (Guizhi Tang, Eastern Han Dynasty) evolved into (Guizhi Tang_Relieves Muscle and Releases Exterior, Jin Dynasty) through "theoretical expansion", (Guizhi Tang_Relieves Muscle and Releases Exterior, Jin Dynasty) evolved into (Guizhi Tang_The Ancestor of Harmonizing Methods, Ming Dynasty) through "conceptual development", (Guizhi Tang_The Ancestor of Harmonizing Methods, Ming Dynasty) evolved into (Guizhi Tang_Harmonizes Yin and Yang, Qing Dynasty) through "theoretical expansion".

[0099] Step 6.3: Entity Ambiguity Resolution. To address entity ambiguity issues during cross-domain fusion, a context-aware entity disambiguation method is employed, combining multi-dimensional information such as source literature, contextual information, and domain knowledge to determine the accurate referent of each entity.

[0100] Let the candidate referent set of ambiguous entities be . The disambiguation objective is to select the candidate with the highest posterior probability.

[0101] Using a Bayesian framework:

[0102] Among them, prior probability Name likelihood is calculated based on candidate occurrence frequency. Based on name similarity calculation, contextual likelihood Source likelihood is calculated based on semantic similarity between context and candidate descriptions. Determine whether the source document is in the candidate set of typical sources. Taking the disambiguation of "Baihu" as an example, the candidate set includes Baihu Tang (formula), Baihu Xue (acupoint), and Baihu (four symbols). In the context of "Shanghan Lun" (Treatise on Cold Damage), "In cases of cold damage, if the pulse is slippery and the patient is cold, there is internal heat, and Baihu Tang is the main treatment," the posterior probability of Baihu Tang is 0.92, the posterior probability of Baihu Xue is 0.05, and the posterior probability of Baihu (four symbols) is 0.03. The disambiguation result is Baihu Tang.

[0103] Step 6.4: Relationship Conflict Handling. Relationship conflicts between knowledge from different sources are addressed by employing a credibility-based knowledge fusion strategy, comprehensively considering factors such as source authority, citation frequency, and time relevance to determine the final knowledge representation.

[0104] Credibility calculation:

[0105] in, , , . The authority rating of the source is as follows: classic works such as "Huangdi Neijing", "Shanghan Lun", "Jinkui Yaolue", and "Shennong Bencao Jing" are rated 1.0; authoritative annotated editions such as "Annotated Shanghan Lun", "Compendium of Materia Medica", and "Wenbing Tiaobian" are rated 0.9; other authoritative annotated editions are rated 0.8; famous medical case studies are rated 0.6; and general literature is rated 0.4.

[0106] Recency factor:

[0107] in, This is the attenuation coefficient. Taking the contradictory dosage of "cinnamon twigs" as an example, the Song Dynasty edition records "three liang of cinnamon twigs," while the Guilin edition records "four liang of cinnamon twigs." The reliability of the Song Dynasty edition is calculated as follows: The credibility of Guilin is calculated as follows: The integration strategy is to primarily record the "three liang" in the Song edition, while marking the "four liang" in the Guilin edition as a variant. Step 6.5: Knowledge Graph Schema Design. Design the overall schema structure of the knowledge graph, including core data tables such as entity tables, relationship tables, cross-domain link tables, version tables, time-series tables, and variant text tables, and define the constraints between the tables.

[0108] The Entity table includes the following fields: entity_id (primary key), name (name), type (type), domain (domain), embedding (1024-dimensional vector), attributes (JSON attributes), created_at (creation time), and updated_at (update time).

[0109] The Relation table fields include: relation_id (primary key), head_entity (head entity foreign key), tail_entity (tail entity foreign key), relation_type (relation type), confidence (confidence level), source_doc (source document), and evidence_text (evidence text).

[0110] The CrossDomain_Link table includes the following fields: link_id (primary key), medical_entity (foreign key for TCM entity), doc_entity (foreign key for document entity), link_type (link type), link_score (link score), and temporal_info (time series information JSON).

[0111] The Version table includes the following fields: version_id (primary key), doc_id (foreign key), type (version type), publish_year (publication year), repository (collection institution), and physical_features (physical features).

[0112] The Timeline table fields include: timeline_id (primary key), entity_id (entity foreign key), time_point (time point), event_type (event type), description (description), and source_doc (source document).

[0113] The Variant table fields include: variant_id (primary key), version_a (foreign key for version A), version_b (foreign key for version B), diff_type (difference type), position (position), content_a (content for version A), content_b (content for version B), and note (proofreading notes).

[0114] Step 6.6: Persistent Storage of the Knowledge Graph. The merged dual-domain knowledge graph is stored in a PostgreSQL database. pgvector is used to extend support for vector storage and retrieval, establishing a hybrid storage architecture of graph data model and vector index, supporting unified access for structured queries and semantic retrieval.

[0115] Step 7: Retrieve Top-K candidates from the database through conditional filtering and semantic similarity calculation; sort the candidate results by multimodal relevance and return the results.

[0116] Here is an example of a mixed query: Structured query (querying the formula information of Guizhi Tang): Perform a joint query from the entity table and the relation table, filter records with the name "Guizhi Tang" and relation types of "composition", "dosage" and "processing", and return the names of the constituent drugs, relation types and confidence levels.

[0117] Semantic retrieval (querying similar prescriptions): Given a query vector, filter entities of the TCM domain prescription type from the entity table, sort them by vector cosine distance, and return the top 10 most similar prescription names and similarity scores.

[0118] Cross-domain source tracing query (querying the literature source of Guizhi Tang): Combine the entity table, cross-domain link table, and version table to filter records with the TCM entity name "Guizhi Tang" and return information such as the source literature name, link type, link score, and publication year.

[0119] The core data table structure of the knowledge graph is summarized in Table 1: Table 1. Structure of the core data table for knowledge graphs

[0120] A computer device system includes a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of the above-described method. A computer-readable storage medium having a computer program / instructions stored thereon, which, when executed by a processor, implement the steps of the above-described method.

[0121] A computer program product includes a computer program / instructions that, when executed by a processor, implement the steps of the method described above.

[0122] The above provides a detailed description of the multimodal information extraction and cross-domain fusion method based on dual-domain knowledge subgraphs proposed in this invention, and elucidates the principles and implementation methods of this invention. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of this invention. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this invention. Therefore, the content of this specification should not be construed as a limitation of this invention.

Claims

1. A method for multimodal information extraction and cross-domain fusion based on dual-domain knowledge subgraphs, characterized in that: The method specifically includes the following steps: Step 1: Multimodal data acquisition and preprocessing, collecting and standardizing multi-source heterogeneous data from ancient Chinese medicine books; Step 2: Extract multimodal information from the TCM knowledge domain, and conduct in-depth semantic analysis on the core medical entities and relationships in TCM; Step 3: Vectorized storage and index construction of TCM knowledge domain. The extracted medical knowledge is semantically vectorized, a TCM knowledge domain subgraph is constructed, and an index is created. Step 4: Extract structured features from the source domain of ancient books and collect metadata and version information; Step 5: Unstructured feature processing and vectorization of the source domain of literature: Information extraction and semantic representation of the unstructured content of ancient books are performed to construct a subgraph of the source domain of literature. Step 6: Cross-domain fusion alignment and knowledge graph storage, linking and fusing the two knowledge subgraphs from Step 3 and Step 5 and storing them persistently; Step 7: Retrieve Top-K candidates through conditional filtering and semantic similarity calculation; sort the candidate results by multimodal relevance and return the results.

2. The method according to claim 1, characterized in that: Step 1 includes: Step 1.1: Data source selection and collection. Obtain multimodal TCM ancient book resources from public channels, including ancient book texts, drug atlases, meridian and acupoint diagrams, tongue diagnosis and pulse diagrams, and medical case audio. Step 1.2: Ancient text preprocessing, cleaning the original text to remove scanning noise and watermark interference, and using OpenCC tool to perform traditional and simplified Chinese conversion; Step 1.3: Ancient and Modern Character Comparison Processing: Construct a comparison table of ancient and modern characters in the field of Traditional Chinese Medicine, mapping ancient characters to modern standard characters; Step 1.4: Image data preprocessing, uniformly adjust image size, perform image pixel normalization, and enhance low-quality images; Step 1.5: Audio data preprocessing, including noise removal from the audio signal, and standardization of sampling rate and audio format; Step 1.6: Data quality verification. Establish a data quality assessment index system to verify the completeness, accuracy, and consistency of the preprocessed data.

3. The method according to claim 2, characterized in that: Step 2 includes: Step 2.1: Extracting triples from ancient texts. Using the Qwen2.5-VL multimodal large model, we perform deep semantic analysis on the content of ancient texts, extract entity types and relation types, and construct core medical relation triples such as disease-symptom-treatment and prescription-drug-efficacy. Step 2.2: Extraction of drug image information. PaddleOCR is used to recognize text information in the image and extract text labels including drug name, place of origin, and properties. Qwen2.5-VL is used for visual question answering to identify drug morphological features. Step 2.3: Extract information from acupoint and meridian diagrams. Use Qwen2.5-VL for visual question answering to extract structured information including acupoint location, meridian affiliation, therapeutic effects, acupuncture depth, and contraindications. Step 2.4: Extracting information from the tongue diagnosis and pulse image, including the following diagnostic features: tongue color, tongue shape, tongue coating, tongue posture, pulse position, pulse rate, pulse strength, and pulse shape. Step 2.5: Extract audio information from medical records. Use the Vosk speech recognition model to convert the audio content into text, and then extract the triples. Step 2.6: Multimodal information fusion, which integrates information extracted from different modalities to establish cross-modal entity alignment relationships.

4. The method according to claim 3, characterized in that: In step 2.1, the extracted entity types include: disease name, symptoms, prescription, medicine, acupoint, meridian, and physician. The types of relationships include: treatment method, compatibility, meridian tropism, indications, contraindications, etiology and pathogenesis, symptom presentation and composition.

5. The method according to claim 4, characterized in that: Step 3 includes: Step 3.1: Text semantic vectorization. Use the Bge-Embedding model to semantically vectorize all extracted text content, generating a 1024-dimensional dense vector representation; Step 3.2: Entity Vector Representation. Generate an independent vector representation for each medical entity, integrating multi-dimensional information such as entity name, attribute description, and relational context. Step 3.3: Relation vector representation, which involves vectorizing the extracted relation types; Step 3.4: Vector normalization processing, perform L2 normalization processing on all generated vectors; Step 3.5: HNSW index construction. The vector is organized using the HNSW index structure to achieve approximate nearest neighbor search. Step 3.6: Storage of TCM knowledge domain subgraph. Entities, relations and their vector representations are stored in the TCM knowledge domain subgraph to establish an entity-relation-entity graph structure.

6. The method according to claim 5, characterized in that: Step 4 includes: Step 4.1: Extract version information, extract scalar fields, including version type, printing year, location of the copy, and version number; Step 4.2: Extract document attributes, extract categorization fields, including author, year of publication, medical school, book category, and number of volumes and articles; Step 4.3: Numerical data extraction, extracting statistical information, including the number of chapters, the number of prescriptions, the number of drugs, the frequency of citations, and the number of times a document is cited; Step 4.4: Extract time-series data, including the date of publication, timeline of version evolution, and timeline of academic inheritance; Step 4.5: Version lineage construction, analyze the source and flow relationships between different versions of the same work, and construct a version evolution lineage diagram; Step 4.6: Academic School Affiliation. Based on the characteristics of the document's content and the author's mentorship, determine the academic school affiliation of the document.

7. The method according to claim 6, characterized in that: Step 5 includes: Step 5.1: Processing of prefaces and epigraphs: Use Qwen2.5-VL to extract semantic information, which includes the author's thoughts, the background of the book's creation, the writing motivation, and the academic origins. Step 5.2: Commentary and evaluation process: extract the viewpoints of later physicians, analyze the stance of the annotations, and construct an academic evaluation network; Step 5.3: Version difference description processing, extract the version differences between different versions, the version differences include textual proofreading information, record variant texts, errors, redundant texts, missing texts and reversed texts; Step 5.4: Citation relationship processing, analyzing the citation types between ancient books, including the relationship of quoting classics, and identifying direct citation, indirect citation, adaptation, criticism and elaboration; Step 5.5: Vectorize the document content. Use the Bge model to generate semantic vectors for unstructured text, which includes prefaces, postscripts, annotations, and collation notes. Step 5.6: Document source domain subgraph storage, storing document entities, version relationships, citation relationships and their vector representations into the document source domain subgraph.

8. The method according to claim 7, characterized in that: Step 6 includes: Step 6.1: Entity linking, establishing entity correspondence between the two knowledge subgraphs: the TCM knowledge domain subgraph and the literature tracing domain subgraph, including candidate generation based on string matching, candidate ranking based on semantic vectors, and comprehensive scoring by fusing multiple features; Step 6.2: Temporal correlation, establishing a temporal dimension correlation between TCM knowledge and literature versions, including constructing a timestamp index and a temporal evolution diagram; Step 6.3: Entity ambiguity resolution, addressing entity ambiguity issues in the cross-domain fusion process by employing a context-aware entity disambiguation method; Step 6.4: Relationship Conflict Handling. This step addresses conflicts between knowledge sources from different sources by employing a credibility-based knowledge fusion strategy. Step 6.5: Knowledge graph schema design. Design the overall schema structure of the knowledge graph, including core data tables such as entity table, relation table, cross-domain link table, version table, time series table, and variant text table. Step 6.6: Persistent storage of the knowledge graph. The merged dual-domain knowledge graph is stored in a PostgreSQL database, and pgvector is used to extend support for vector storage and retrieval.

9. A multimodal information extraction and cross-domain fusion system based on dual-domain knowledge subgraphs, characterized in that: The system is used to execute the multimodal information extraction and cross-domain fusion method based on dual-domain knowledge subgraphs as described in any one of claims 1 to 8: The system includes a multimodal data acquisition and preprocessing module, a multimodal information extraction module for the TCM knowledge domain, a vectorized storage and index construction module for the TCM knowledge domain, a structured feature extraction module for the literature tracing domain, an unstructured feature processing and vectorization module for the literature tracing domain, a cross-domain fusion alignment and knowledge graph storage module, and a filtering and output module. The multimodal data acquisition and preprocessing module collects and standardizes multi-source heterogeneous data from ancient Chinese medicine books. The TCM knowledge domain multimodal information extraction module performs in-depth semantic analysis on core TCM medical entities and relationships. The TCM knowledge domain vectorization storage and indexing construction module performs semantic vectorization on the extracted medical knowledge, constructs a TCM knowledge domain subgraph, and establishes an index. The document tracing domain structured feature extraction module is used to collect ancient book metadata and version information; The document tracing domain unstructured feature processing and vectorization module extracts information and semantically represents the unstructured content of ancient books, and constructs a document tracing domain subgraph. The cross-domain fusion alignment and knowledge graph storage module associates and fuses the two knowledge subgraphs from steps 3 and 5 and stores them persistently. The filtering output module retrieves Top-K candidates through conditional filtering and semantic similarity calculation; it then sorts the candidate results by multimodal relevance and returns the results.

10. A computer device system, comprising a memory, a processor, and a computer program stored in the memory, characterized in that, The processor executes the computer program to implement the steps of the method according to any one of claims 1 to 8.

Images