Rare disease knowledge graph construction method based on modal injection and multi-modal fusion

Abstract

The present application relates to the medical artificial intelligence and knowledge graph construction technical field, especially in rare disease knowledge graph construction method based on modal injection and multi-modal fusion, including the following steps: S1, collect including text, image and gene multi-modal medical information;S2, standardization processing, construct three layer metadata structure;S3, complete missing modal data, respectively to modal data feature extraction and uniform dimension conversion, realize the representation alignment in shared semantic space;S4, multi-level semantic fusion, obtain uniform fusion semantic vector;S5, construct including ontology layer and instance layer double layer structure system rare disease knowledge graph.The present application selects text, image and gene multi-modal medical information to construct rare disease knowledge graph, improves its knowledge graph application range, coverage and accuracy, can solve the corresponding adaptation of rare disease clinical diagnosis and treatment when the few cases, has strong recognition ability.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of medical artificial intelligence and knowledge graph construction, and particularly relates to a rare disease knowledge graph construction method based on modal injection and multi-modal fusion. BACKGROUND

[0002] With the development of medical artificial intelligence and knowledge engineering, medical knowledge graphs have gradually become an important support means for clinical auxiliary diagnosis, scientific research knowledge discovery and medical resource integration. However, in the field of rare diseases, the construction and application of knowledge graphs still face many technical challenges, mainly in the aspects of data scarcity, high heterogeneity and semantic fragmentation, which seriously restrict the promotion of the graph in the intelligent diagnosis and treatment scene.

[0003] Rare diseases usually have the characteristics of extremely small number of clinical samples, long diagnosis cycle and significant phenotypic heterogeneity, which makes it difficult for traditional graph construction methods relying on large-scale structured corpus and single modal information to be applicable. Especially in real clinical scenarios, patients' medical data usually exist in the form of multi-modal, covering structured clinical text (such as diagnosis records, symptom descriptions), medical images (such as CT, MRI) and genomic sequencing data (such as mutation sites in VCF format). These modalities have significant differences in structure, dimension and semantic system, showing the complex characteristics of "multi-source heterogeneity".

[0004] Current research attempts to model text and image or text and gene in a dual-modal manner, but has not yet formed a knowledge graph construction scheme that can simultaneously fuse text, image and genomic three modalities. This is mainly due to the fact that the following technical difficulties have not been effectively broken through:

[0005] Firstly, the three types of modalities are highly heterogeneous in terms of data organization method, semantic representation form and structural granularity. Text is a time-series semantic structure, image is spatial distribution information, and gene data is high-sparse, strongly structured encoded data, which are difficult to establish equivalent relationships or share semantic representation spaces.

[0006] Secondly, existing modal fusion methods mostly use simple concatenation or linear weighting methods, which cannot fully model the deep semantic complementarity and linkage mechanism between modalities. The lack of a unified modal alignment mechanism makes it difficult to maintain semantic consistency in entity relationships in the graph, especially in applications such as diagnosis and reasoning that require high semantic requirements, which shows obvious shortcomings.

[0007] Thirdly, the problem of missing modal data is particularly serious in the field of rare diseases. Most rare disease patients only have partial modal information, such as missing image examination or incomplete gene sequencing. Existing multi-modal graph construction schemes generally rely on complete modal data input, cannot handle incomplete modal samples, and lack an automatic evaluation mechanism for the credibility of synthetic completed data, thereby affecting the rigor and explainability of the graph.

[0008] In addition, in terms of knowledge graph structure construction, the traditional scheme relies on static rules and structured ontology extraction, and it is difficult to automatically generate supplementary semantic edges or enhanced semantic connections according to the semantic linkage relationship between different modalities, so that the adaptability of the graph in reasoning, path retrieval and clinical application is weak.

[0009] In summary, there is an urgent need for a graph construction method that can target the characteristics of rare disease multi-modal medical data, has the ability to enhance the semantic consistency between modalities, the ability to complete the missing, and the structure traceable mechanism, to break through the application bottleneck of existing technology in the environment of high sparsity, multi-heterogeneous and weak annotation, and to improve the intelligent diagnosis ability and knowledge expression efficiency in the rare disease scenario. SUMMARY

[0010] The present application provides a rare disease knowledge graph construction method based on modality injection and multi-modal fusion, which aims to simultaneously process multi-modal medical data and realize fusion and enhancement of modality features, adapt to rare disease knowledge graph, to solve the problems of weak fusion ability, unclear structure and poor clinical adaptability of existing methods in multi-modal medical data processing, modality missing completion, semantic consistency alignment and graph structure expression.

[0011] In order to achieve the above purpose, the present application provides a rare disease knowledge graph construction method based on modality injection and multi-modal fusion, comprising the following steps:

[0012] S1, collecting multi-modal medical information including text, image and gene;

[0013] S2, standardizing the multi-modal medical information and constructing a three-layer metadata structure;

[0014] S3, completing the missing modality data, extracting and converting the modality data to a unified dimension respectively, obtaining a unified format global semantic modality representation vector, processing the modality representation vector using a modality injection mechanism, enhancing cross-modality semantic perception, obtaining an enhanced representation vector, and realizing cross-modality representation alignment in a shared semantic space through a cross-modality alignment module;

[0015] S4, multi-level semantic fusion of the modality features after cross-modality representation alignment to obtain a unified fusion semantic vector;

[0016] S5, based on the unified fusion semantic vector, constructing a double-layer structure system rare disease knowledge graph including ontology layer and instance layer, and establishing modality reference, semantic enhancement and traceability path edge relationships in the graph.

[0017] Preferably, the text, image and gene include structured clinical text, medical image and genomic data from rare disease patients, the genomic data including low-frequency mutation sites and their pathogenicity annotation information.

[0018] Rare diseases have some special requirements for knowledge graph construction due to sample shortage, special onset conditions and other reasons: on the one hand, rare disease data itself is very scarce, and often modalities are missing; on the other hand, its diagnosis largely depends on gene mutation information, unlike common diseases that can be determined by image or symptoms alone. Therefore, mechanisms such as modality completion, modality injection and fusion representation are introduced in our invention to better solve the problems of data structure and semantic fusion of rare diseases, improve the accuracy of the entire process, and have core advantages in the field of rare disease diagnosis.

[0019] More preferably, the multi-modal medical information is derived from hospital clinical information system (HIS), picture archiving and communication system (PACS) and gene sequencing center. Structured clinical text data includes patient basic information, diagnosis record, disease course description, diagnosis description, treatment description, treatment plan and follow-up record, etc.; medical image data includes typical examination results such as chest CT and brain MRI, stored in DICOM format; genomic data includes variant site information of target region or whole genome, usually represented in VCF file form. After the above multi-modal data is collected, irrelevant information is cleaned, named entity is desensitized, unified file format is converted, and physical and semantic mapping relationships are established through path index and semantic label, and stored in multi-modal metadata framework for subsequent cross-modal analysis and calling.

[0020] Preferably, the three-layer metadata structure in S2 includes disease layer, case layer and modality layer, the disease layer is used to record standard disease codes, core symptom terms and related pathogenic gene identifiers; the case layer is used to record patient basic information, diagnosis and treatment timeline and family medical history; the modality layer is used to record modality data path, semantic label, feature vector and quality score.

[0021] All standardized data will be uniformly organized and stored in the following "three-layer metadata structure": the Schema layer defines standard medical entities such as disease ontology, symptom terminology, pathogenic gene, treatment plan, etc., to achieve standardized expression and unified reference of upper-layer medical knowledge; the Instance layer carries specific individual cases, including patient information, timeline, family history, etc., and builds a bridge from standard knowledge to actual cases; the Evidence layer records the multi-modal raw data and semantic labels, quality scores corresponding to each case, providing a basis for graph reasoning and traceability. This structure effectively avoids the problem of "flat structure and flat entity" in traditional graphs, enabling the graph to have three-layer semantic capabilities of ontology reusability, instance extensibility, and modality traceability.

[0022] The Evidence layer explicitly models each data path, supporting path mapping between raw text, images, gene files, and semantic entities; combined with the modality alignment module, it generates semantic enhanced edges (such as semantically_related_to, is_similar_to, textual_equivalent_to) to automatically generate semantic links between heterogeneous data. It supports doctors or systems to query the graph nodes and trace the original data sources and quality indicators in reverse.

[0023] Preferably, the complementing missing modality data in S3 specifically includes: for the case of missing modality, complementing missing modality data based on RareGAN-VAE conditional generation model, and using a three-level verification mechanism including automatic evaluation, semantic rule verification, and expert scoring to mark its credibility;

[0024] The model takes as input the segment features of existing modalities and noise vectors, and as output the synthesized data of missing modalities. The automatic evaluation uses Frechet Inception Distance to evaluate image similarity, and uses ClinVar database to verify gene mutation and UMLS definition matching between gene and disease phenotype. The semantic rule verification includes gene-disease association rules and image-pathological feature matching rules. The gene-disease association rule is that the synthesized gene mutation should have a dominant association with the target disease in the ClinVar / HGNC database. The image-pathological feature matching rule is that if a disease clearly indicates a certain pathological sign in Radiopaedia (a comprehensive medical imaging encyclopedia and teaching platform) or UMLS (Unified Medical Language System), then the synthesized image should have the corresponding features in the target area. The expert scoring is double-blind scored by experts with rare disease clinical experience, and the average score should not be lower than the preset threshold.

[0025] The RareGAN-VAE model addresses the serious problem of missing modalities in rare disease cases by using a conditional generation strategy to complete the missing modalities. The input consists of a random noise vector and existing modality embedding fragments (such as gene sequences and text descriptions), and the output is synthetic data for the corresponding modality. The RareGAN-VAE model used in this invention is based on the known variational autoencoder (VAE) and generative adversarial network (GAN) combined architecture in the field, combined with a conditional generation mechanism, to solve the problem of missing modalities in rare disease cases. The model structure itself is not the innovative content of this invention, but as a supporting completion module in the multi-modal atlas construction process, it is adapted for use to address the actual problem of incomplete image or gene data in the rare disease scenario. The model input includes a random noise vector and existing modality embedding fragments (such as gene sequences and text descriptions), and the output is synthetic data (such as images or missing text) for the target modality, which is used to improve the cross-modal alignment and the integrity of the atlas node information.

[0026] To ensure the quality of generation and clinical reliability, a three-level verification mechanism is introduced: (1) For the image modality, use FID (Frechet Inception Distance) to calculate the similarity between the synthetic image and the real case distribution; Frechet Inception Distance (FID) is an index used to evaluate the quality of generated images by a generation model (such as GANs, VAEs). It compares the distribution difference between generated images and real images in the feature space to measure the similarity between them. The lower the FID, the closer the quality and diversity of the generated images to the real data. (2) For the synthetic mutation sites of the gene modality, match the known pathogenic mutations in authoritative databases such as ClinVar to verify their clinical reasonableness; (3) Perform semantic consistency detection on the synthetic case text and structural description, and conduct blind review scoring by three specialists with clinical experience in rare diseases. According to the five-point scoring system, only when the average score is greater than 4 points is the data determined to be usable. All verification results and generation parameters are stored in the metadata to form a traceable synthetic data meta record.

[0027] In the field of rare diseases, not only are the samples extremely small, but often the modalities are incomplete, such as missing medical images, missing gene sequencing, and incomplete clinical text descriptions. In this case, it is necessary to use a generation model (such as RareGAN-VAE) to complete the missing modalities, but it also carries a higher risk: once the quality of the synthetic data is not high, it may mislead the subsequent knowledge graph construction, relationship extraction, and even diagnosis reasoning results.

[0028] Therefore, the application puts forward higher reliability requirements for the completion result, not only requiring "like" in structure, but also "reasonable" in semantics and "usable" in clinic. Under this high requirement background, a single verification method (such as using only automatic indicators) is far from enough. Therefore, the present application designs a three-level verification mechanism to ensure the quality of generated data from three angles:

[0029] 1. Statistical evaluation (such as FID) - ensure that the generated results are similar in structure to the real distribution;

[0030] 2. Knowledge rule comparison (such as ClinVar matching) - ensure that the generated content meets medical semantics;

[0031] 3. Expert artificial blind review - finally judged by clinicians whether it has real usability.

[0032] The advantages of this mechanism are:

[0033] Comprehensive coverage of the structure, semantics and actual application layer of the generated data;

[0034] Three barriers superimposed, greatly reducing the risk of false information entering the atlas;

[0035] The verification result is quantifiable, recordable and traceable, which is conducive to data governance and continuous optimization.

[0036] The feature extraction and unified dimension conversion of the modality data specifically include:

[0037] Text, image and gene data are extracted respectively to obtain the original representation of the three types of modalities, which are uniformly mapped to the same dimension, and a modal representation head is designed to extract a global semantic representation vector in a unified format;

[0038] Text modality (Text-MRH): Bio_ClinicalBERT encoder is used to obtain the semantic representation of clinical text, and a 256-dimensional vector is output after aggregating the context vector; Bio_ClinicalBERT is a BERT model specially pre-trained for biomedical and clinical text, trained by Stanford University and Google Research team based on clinical notes (such as MIMIC-III) and PubMed literature. It optimizes medical entity recognition, relationship extraction and question answering tasks, and is an important tool for clinical natural language processing (NLP).

[0039] Img-MRH: Based on 3D ResNet to extract global anatomical features, combined with U-Net to segment the lesion area, and finally output a 256-dimensional vector through light-weight convolution layer dimensionality reduction processing; 3D ResNet is a three-dimensional extension of traditional ResNet (Residual Network), which is specially used to process video, medical image (such as CT / MRI), spatio-temporal sequence data and other 3D input. Its core idea is to solve the gradient vanishing problem of deep network through residual connection (Skip Connection), and capture spatio-temporal features through 3D convolution. U-Net is a classic convolutional neural network (CNN) for medical image segmentation.

[0040] Gene-MRH: The ClinVar annotated pathogenic mutation information is fused with the HGNC encoded gene site embedding vector, encoded through a multilayer perceptron (MLP), and a 256-dimensional representation vector is output; HGNC (Human Genome Organization Nomenclature Committee) is an authoritative gene naming organization, which provides unique and standardized names for genes, gene families and non-coding RNAs in the human genome.

[0041] Respectively denoted as:

[0042] Text modality representation vector : ;

[0043] Image modality representation vector : ;

[0044] Gene modality representation vector : ;

[0045] Among them, , , respectively represent the text, image and gene three modalities.

[0046] Preferably, the modality injection mechanism in S3 specifically includes:

[0047] Introducing the global semantic representation of other modalities to each modality representation vector to enhance the cross-modal semantic perception ability;

[0048] For the original representation vector of the modality , its enhanced representation vector is defined by the following formula:

[0049] ;

[0050] Among them, a modal injection adjustment coefficient, a global vector representing other modalities.

[0051] Preferably, the cross-modality alignment module in S3 specifically comprises:

[0052] The representation mapping method based on the bidirectional attention mechanism is adopted to map the enhanced representation vectors of the three types of modalities of text, image and gene to a shared semantic space, so as to realize the feature alignment between modalities. The inputs of the cross-modality alignment module for the three types of modalities of text, image and gene are respectively:

[0053] the enhanced representation vector of the text modality : ;

[0054] the enhanced representation vector of the image modality : ;

[0055] the enhanced representation vector of the gene modality : ;

[0056] wherein, , , are the number of text , the number of image , and the number of gene mutation sites respectively, is the unified embedding dimension;

[0057] For any modality pair , the attention weight mapping from modality to modality is defined by the following formula:

[0058] ;

[0059] ;

[0060] ;

[0061] wherein, are learnable projection parameters, represents a real matrix space of ; is the query matrix of modality , which has the same dimension as ; is the projection parameter of ; is the key matrix of modality , which has the same dimension as ; is The projection parameters; In the attention mechanism, T represents the transpose of the matrix. yes The transpose of is used to calculate and dot product between; Represents a unified embedding dimension; and Representing modes and modality The enhanced representation vector;

[0062] softmax represents the normalization function. It is the attention weight matrix;

[0063] Further compute the cross-modal alignment representation under this mapping:

[0064] ;

[0065] ;

[0066] in, These are learnable projection parameters; From modality To mode The weighted sum of the value vectors; It is the attention weight matrix; It is modal The value vector; Representing modes The enhanced representation vector; It is the projection matrix of the value vector; Represent a The space of real matrices;

[0067] To enhance the consistency and reversibility of alignment, a path-independent consistency loss is introduced. :

[0068] ;

[0069] in, It is modal The indivual Mapping to mode The representation of; The corresponding reverse mapping is then returned to the mode. The alignment result; Measuring modes and modality Consistency loss due to alignment and The superscript represents the generalization variable of the modal, which is a generalization of the three modalities 、 、 , and are selected from 、 、 ;

[0070] Finally, sum all the modalities to get the overall alignment consistency loss :

[0071] ;

[0072] wherein, is the overall alignment consistency loss;

[0073] is the alignment consistency loss of modal and modal ;

[0074] is the alignment consistency loss of modal and modal ;

[0075] is the alignment consistency loss of modal and modal .

[0076] Preferably, the multi-level semantic fusion in step S4 specifically includes:

[0077] Using a multi-level fusion module FusionBLK to perform deep semantic fusion on the aligned modal features; the module is composed of fusion blocks (Fusion Block), denotes the total number of fusion modules or the number of fusion blocks, and each layer includes four steps of modal quality weighting, sequence alignment, encoder fusion and residual connection.

[0078] The aligned representation of each modality is dynamically adjusted according to its metadata quality weight:

[0079] ;

[0080] ;

[0081] ;

[0082] wherein, is the aligned modal feature sequence, is the modality quality score;

[0083] is the aligned modality feature sequence, is the modality quality score;

[0084] is the aligned modality feature sequence, is the modality quality score;

[0085] is the enhanced text modality representation vector;

[0086] is the enhanced image modality representation vector;

[0087] is the enhanced gene modality representation vector;

[0088] Adjust the modality sequences of different lengths to a consistent size, and unify them to a length :

[0089] ;

[0090] is the enhanced text modality representation vector;

[0091] is the enhanced image modality representation vector;

[0092] is the enhanced gene modality representation vector;

[0093] represents a real matrix space, the meaning is to unify the sequence length, representing the unified embedding dimension;

[0094] The features of the three modalities are spliced and sent to the fusion encoder:

[0095] ;

[0096] wherein, “ ” represents splicing in dimension; is the joint modality representation vector of the layer fusion; is the index of the fusion layer; represents a real matrix space, denotes the unified sequence length, denotes the unified embedding dimension;

[0097] is the enhanced text modality representation vector;

[0098] is the enhanced image modality representation vector;

[0099] is the enhanced gene modality representation vector;

[0100] The concatenated multi-modal representation is fed into a fusion encoder module (which can be Transformer, Restormer, or Gated MLP):

[0101] ;

[0102] ;

[0103] wherein, is the output of the previous layer, initially a zero vector ; =0, i.e., a zero vector with dimension ; denotes the unified sequence length, denotes the unified embedding dimension; denotes the fusion encoder; is the joint modality representation vector of the th fusion layer; is the index of the fusion layer;

[0104] After layers of iteration, the fusion representation of the th layer is output , which is aggregated into a unified fusion semantic vector using Mean Pooling or Attention Pooling:

[0105] ;

[0106] wherein, is the fusion semantic vector; Pooling is the pooling operation; is the real space with dimension vector set, denotes the unified embedding dimension; is the fusion representation of the th layer, is the index of the fusion layer; Mean Pooling is the mean pooling, and Attention Pooling is the attention pooling.

[0107] Preferably, the construction of the double-layer structure system including the ontology layer and the instance layer in step S5 specifically includes:

[0108] The ontology layer is used to represent standard medical knowledge in the field of rare diseases, including disease concepts, clinical symptoms, pathogenic genes and treatment plans, and the nodes come from authoritative medical ontology libraries, including Orphanet, UMLS and HGNC; the instance layer is used to store specific case data and its corresponding multi-modal information, and the nodes include case instance nodes, text nodes, image nodes and gene nodes, all of which have unique identifiers and modal attributes; the ontology layer and the instance layer are connected through semantic edges including instance_of, aligned_to, has_symptom and has_gene, forming a hierarchical tree graph structure.

[0109] Among them, instance_of indicates that a specific entity is a specific instance of a general category, aligned_to indicates the corresponding relationship between two entities in different systems or standards, has_symptom indicates the relationship between diseases and symptoms, and has_gene indicates the relationship between diseases, traits or organisms and related genes.

[0110] Preferably, the edge relationship of modality reference, semantic enhancement and traceability path established in the graph in step S5 specifically includes:

[0111] Using the multi-modal fusion semantic vector generated in step S4 , embedding representation is performed on the instance nodes in the graph, and the structural connection relationship between multi-modalities is constructed; the original modality data nodes are explicitly modeled, and a traceable path between the modality ontology and the instance is constructed.

[0112] The structural connection relationship between multi-modalities is constructed by using the inter-modal attention weight learned by the MultiAlign module to judge the semantic coupling strength between modalities; MultiAlign is an algorithm or software module for multiple sequence alignment (MSA) and is widely used in the fields of bioinformatics, genomics, protein structure prediction, etc. Its core goal is to align multiple biological sequences (DNA, RNA or protein), identify conserved regions, variant sites and evolutionary relationships.

[0113] If the weight exceeds the set threshold, a cross-modal semantic edge is automatically generated, which specifically includes:

[0114] semantically_related_to: automatically establish semantic association edges between modality pairs when the attention weight or fusion correlation between modality pairs exceeds a set threshold; is_similar_to: when the cosine similarity of the fusion representation between two case nodes is lower than a preset distance threshold , it is determined as a semantically similar case, and an is_similar_to edge is generated; textual_equivalent_to: when the alignment confidence between the original text and the standardized UMLS term is higher than a set confidence threshold, a textual_equivalent_to edge is generated; the above edge relationships are automatically calculated and generated by the weight matrix and fusion vector space distance in the modality alignment module.

[0115] Preferably, the knowledge graph system supports multi-source heterogeneous query methods, respectively facing structured queries and semantic vector retrieval, and the deployment structure comprises:

[0116] A graph database module is used to store structured entity nodes, semantic edge relationships and modality reference paths, and supports graph query, path reasoning and subgraph extraction operations based on the Cypher query language; Cypher is a declarative query language for Neo4j graph databases;

[0117] A vector database module is used to store the fusion semantic vectors of each instance node , and supports vector retrieval and case recall based on semantic similarity;

[0118] An object storage system module is used to store original modality data files, including structured clinical text, medical images and genomic data of rare disease patients, and provides path backtracking capability, supports mapping and calling with graph node attributes.

[0119] The three cooperatively constitute the storage, indexing, retrieval and calling infrastructure of the multi-modal knowledge graph, and support downstream applications facing intelligent diagnosis, semantic question answering and clinical reasoning.

[0120] The knowledge graph system constructed by the application combines structured graph query and vectorized semantic retrieval two types of heterogeneous query mechanisms, which are respectively deployed in the graph database and the vector database module, and have the following advantages:

[0121] (1) Graph database (structured path query) - supports semantic rule reasoning and relationship path search:

[0122] Based on Neo4j + Cypher query language, support multi-hop query, instance tracing, subgraph extraction; Can realize complex rule query, such as: Rare disease X → has_gene → GENE_Y → is_similar_to → Patient P; Case B → has_image → CT image → textual_equivalent_to → Symptom S.

[0123] Corresponding to the habit of "rule diagnosis logic" of doctors, it is suitable for etiology tracking, gene candidate screening, disease clustering and other scenes.

[0124] (2) Vector database (semantic similarity query) - support intelligent retrieval and recommendation in semantic vector space:

[0125] Based on Milvus or Faiss, support vector Top-K retrieval, similar case recall; Query mode can be: input case fusion vector → retrieve similar disease; Only input text / image / gene fragment → fast modal collaborative completion of similar graph node.

[0126] The vector database improves the accessibility of the graph under unstructured input, and is suitable for intelligent question answering, auxiliary diagnosis, data-driven discovery and other tasks.

[0127] The above scheme of the present application has the following advantages:

[0128] (1) The present application selects multi-modal medical information of text, image and gene to construct rare disease knowledge graph, improves the application range, coverage and accuracy of the knowledge graph, can solve the corresponding adaptation of rare disease clinical diagnosis and treatment when the case is rare, and has strong recognition ability;

[0129] (2) When the present application extracts and modality fusion of complex multi-modal medical information, the modality injection means and cross-modal alignment module are selected, the representation alignment of different information is fully realized, and the unified fusion semantic vector is constructed through the multi-level semantic fusion step, the rare disease knowledge graph can be stably constructed, and the knowledge graph data constructed has strong discrimination ability;

[0130] (3) The knowledge graph construction of the present application includes a double-layer structure system of ontology layer and instance layer, and establishes modality reference, semantic enhancement and traceability path and other edge relationships in the graph, which brings great convenience to graph query application, has high accessibility and strong practicality, and can be widely applied in clinical assistance of rare disease treatment;

[0131] (4) The three-layer metadata structure system composed of disease type layer, case layer and modality layer is adopted, full-link mapping and management from disease ontology, individual case to original modality data are effectively supported, the clear organization of atlas structure is realized, meanwhile, the traceability and quality control ability of modality data are enhanced, and good expansibility and iterative training support are provided;

[0132] (5) The knowledge graph system constructed in the application has a dual-channel query mechanism of combining a graph database and a vector database, can support structured semantic path reasoning and similarity retrieval based on modality fusion vectors, can meet the diversified needs of complex logical query and intelligent semantic recall, and has high flexibility and practicality in application scenarios such as clinical auxiliary diagnosis, semantic question answering and case recommendation.

[0133] Other beneficial effects of the application will be described in detail in the subsequent specific embodiment part. BRIEF DESCRIPTION OF DRAWINGS

[0134] Figure 1 The overall flowchart of the rare disease knowledge graph construction method of the application is shown in the figure.

[0135] Figure 2 The detailed flowchart of the multi-modality data acquisition and standardization processing in S1 step of the application is shown in the figure. DETAILED DESCRIPTION

[0136] In order to make the technical problems, technical solutions and advantages of the application clearer, specific embodiments will be described in detail below with reference to the drawings and specific embodiments. Obviously, the described embodiments are part of the embodiments of the application, not all the embodiments. Based on the embodiments in the application, all other embodiments obtained by those skilled in the art without creative labor belong to the protection scope of the application.

[0137] In the description of the application, it should be noted that the terms "center", "upper", "lower", "left", "right", "vertical", "horizontal", "inner", "outer" and the like indicate the orientation or positional relationship shown in the drawings, and are only for the convenience of describing the application and simplifying the description, and do not indicate or imply that the device or element referred to must have a particular orientation, be constructed and operated in a particular orientation, therefore it cannot be understood as a limitation of the application. In addition, the terms "first", "second", "third" are only for the purpose of description, and cannot be understood as indicating or implying relative importance.

[0138] In the description of the present application, it should be noted that unless otherwise explicitly specified and limited, the terms "mounting", "connection", "connecting" should be understood broadly, for example, it can be a locking connection, or a detachable connection, or an integral connection; it can be a mechanical connection, or an electrical connection; it can be a direct connection, or an indirect connection through an intermediate medium, or a communication within two elements. For those skilled in the art, the specific meanings of the above terms in the present application can be understood according to the specific circumstances.

[0139] In addition, the technical features involved in the different embodiments of the application described below can be combined with each other as long as there is no conflict between them.

[0140] The rare disease knowledge graph construction method based on modal injection and multi-modal fusion comprises the following steps:

[0141] (1) Obtain multi-source heterogeneous medical data, collect structured clinical text of rare disease patients from a hospital information system, medical image data from a PACS system, and whole genome or targeted mutation data from a genomic sequencing platform, and perform desensitization processing and modal format standardization;

[0142] (2) Based on the constructed three-layer metadata structure (disease layer Schema, case layer Instance, and modal layer Evidence), the above data is uniformly organized and semantically modeled to form a standardized multi-modal data system that can be parsed;

[0143] (3) To solve the problem of missing modal data in the collected data in the three-layer metadata structure, an improved RareGAN-VAE generative model is used for conditional modal completion, the input is the embedding fragment feature of the existing modal in the three-layer metadata structure and the noise vector, and the output is the synthesized modal data, and a three-way verification mechanism is constructed to evaluate its credibility;

[0144] The original feature representations of the three types of modalities of text, image and gene are extracted respectively, Bio_ClinicalBERT, 3D ResNet + U-Net and MLP models are used to map them to global semantic modal representation vectors in a unified format, and a modal injection mechanism is introduced to improve the semantic interaction ability between modalities; The enhanced representation vector after modal injection is input into the MultiAlign alignment module, a bidirectional path-independent attention mapping mechanism is constructed, and a path consistency loss function is introduced for alignment learning in the shared semantic space;

[0145] (4) Use the multi-level residual fusion module FusionBLK to fuse the three types of aligned features, and construct the final unified fusion semantic vector through modal quality weighting, sequence length alignment, encoder fusion and residual connection;

[0146] (5) Constructing the multi-modal medical knowledge graph structure composed of ontology layer and instance layer, relying on Orphanet, UMLS, HGNC and other authoritative knowledge bases to establish standard semantic nodes and their structural relationships, and using modal alignment output vector to embed the instance layer; based on the attention weight and semantic similarity of modal fusion, automatically generating semantic enhanced edges such as semantically_related_to, textual_equivalent_to, is_similar_to and other edge relationships; deploying the graph on the heterogeneous platform of graph database, vector database and object storage system, supporting dual-channel retrieval and reasoning services based on structural path and semantic vector.

[0147] Further, step (1) is specifically as follows:

[0148] Figure 2 A detailed flowchart of the multi-modal data acquisition and standardization processing in S1 of the application;

[0149] The multi-modal medical data is derived from hospital clinical information system (HIS), picture archiving and communication system (PACS) and gene sequencing center. Structured clinical text data includes patient basic information, diagnosis record, disease course description, diagnosis description, treatment description, disposal scheme and follow-up record and other contents; medical image data includes typical examination results such as chest CT and brain MRI, stored in DICOM format; genomic data includes variant site information of target region or whole genome, usually represented in VCF file form. After the above multi-modal data is collected, irrelevant information is cleaned, named entity is desensitized, unified file format is converted, and the mapping relationship between physical and semantic is established through path index and semantic label, and stored in the multi-modal metadata framework, which is convenient for cross-modal analysis and calling in subsequent tasks.

[0150] All standardized data will be uniformly organized and stored in the following "three-layer metadata structure": the disease layer (Schema) defines the disease ontology, standard disease coding, core symptom terminology, related pathogenic genes, treatment schemes and other standard medical entities, realizes the standardized expression and unified reference of the upper layer of medical knowledge; the case layer (Instance) carries specific individual cases, including patient basic information, diagnosis and treatment timeline, family medical history, etc., and builds a bridge from standard knowledge to actual cases; the modal layer (Evidence) records the multi-modal original data path and semantic label, feature vector and quality score corresponding to each case, providing a basis for graph reasoning and traceability. This structure effectively avoids the problem of "flat structure and flat entity" in traditional graphs, making the graph have three-layer semantic capabilities of ontology reusability, instance extensibility and modal traceability.

[0151] The modal layer explicitly models each data path, supporting path mapping between raw text, images, gene files, and semantic entities; combined with the modal alignment module, it generates semantic enhancement edges (such as semantically_related_to, is_similar_to, textual_equivalent_to), enabling automatic generation of semantic links between heterogeneous data. When doctors or systems query the graph nodes, it supports reverse tracking of the original data source and quality indicators.

[0152] Further, in step (3):

[0153] The RareGAN-VAE model addresses the serious problem of missing modalities in rare disease cases by using a conditional generative model to complete the missing modalities. The input consists of a random noise vector and existing modality embedding fragments (such as gene sequences and text descriptions) from a three-layer metadata structure, and the output is synthetic data for the corresponding modality. For missing modalities, the RareGAN-VAE conditional generative model is used to complete the missing modality data, and a three-level validation mechanism including automatic evaluation, semantic rule verification, and expert scoring is used to mark its credibility;

[0154] The model takes as input the fragment features of existing modalities and noise vectors, and as output the synthetic data for missing modalities. Automatic evaluation uses Frechet Inception Distance to evaluate image similarity, and uses the ClinVar database to verify gene mutations and the matching of UMLS definitions between genes and disease phenotypes; expert scoring is performed by double-blind scoring by experts with clinical experience in rare diseases, with an average score not less than the pre-set threshold;

[0155] To ensure the quality of the generated data and its clinical credibility, a three-level validation mechanism is introduced: (1) For the image modality, FID (Frechet Inception Distance) is used to calculate the similarity between the synthetic image and the true case distribution; (2) For the synthetic mutation site of the gene modality, match the known pathogenic mutations in authoritative databases such as ClinVar to verify its clinical reasonableness; (3) For the synthetic case text and structural description, perform semantic consistency detection, and have three specialist doctors with clinical experience in rare diseases perform blind review scoring according to a five-point scoring system. Only when the average score is greater than 4 is the data considered usable. All validation results and generation parameters are stored in the metadata, forming a traceable synthetic data metadata record.

[0156] The setting of the three-level verification mechanism is mainly based on the high complexity of the rare disease modal missing problem and the key influence of the completion result on the accuracy and explainability of the knowledge graph construction. Compared with other diseases, rare disease case data is generally characterized by sample scarcity, incomplete modal and strong heterogeneity, and genomic or image information often plays a decisive role in diagnosis. Therefore, when using a generative model for modal completion, more stringent quality assurance methods are needed to prevent low-quality synthetic data from having a negative impact on the graph structure and diagnosis reasoning.

[0157] The three-level verification mechanism adopted by the present application starts from three dimensions of structural quality, semantic compliance and clinical rationality, and constructs a verification system of "statistical evaluation + knowledge rules + manual judgment", effectively realizing the comprehensive screening and reliable control of the generated samples. The specific advantages are: 1. Enhance the clinical safety of generated data and avoid semantic bias; 2. Improve the semantic consistency and structural matching degree of the completed samples; 3. Realize the quantifiable and traceable management of synthetic data quality; 4. Fully adapt to different modal characteristics and ensure the scientificity and effectiveness of the evaluation method.

[0158] Further, in step (3):

[0159] To alleviate the problem of dimensional and semantic space deviation of multi-modal feature representation, first, the original representations of text, image and gene modalities are uniformly mapped in dimension, and a modality representation head (Modality Representation Head, MRH) is designed to extract a global semantic modality representation vector in a unified format.

[0160] Text modality (Text-MRH): Bio_ClinicalBERT encoder is used to obtain the semantic representation of clinical text, and a 256-dimensional vector is output after aggregating the context vector;

[0161] Image modality (Img-MRH): Based on 3D ResNet, the global anatomical features are extracted, combined with U-Net to segment the lesion area, and finally a 256-dimensional vector is output through dimension reduction processing by a lightweight convolutional layer;

[0162] Gene modality (Gene-MRH): The ClinVar annotated pathogenic mutation information is fused with the HGNC encoded gene site embedding vector, and a 256-dimensional representation vector is output through a multilayer perceptron (MLP).

[0163] Let them be respectively:

[0164] Text modality representation vector : ;

[0165] Image modality representation vector : ;

[0166] Gene modality representation vector : ;

[0167] wherein, , , represent the text, image and gene modalities respectively.

[0168] To improve the perception ability of each modality to the semantics of other modalities, a modality injection mechanism is introduced. For any modality , its original representation vector , the enhanced representation vector is defined as:

[0169] ;

[0170] wherein, is a modality injection adjustment coefficient, which regulates the cross-modality semantic injection strength, represents the global vector of other modalities.

[0171] Although the three types of modalities of text, image and gene are all mapped into 256-dimensional vectors, due to the significant differences in their original structures, information density and semantic expression methods, there are still many challenges in the joint training process. First, the semantic space distribution among the three types of modalities is offset, and direct fusion can easily cause semantic mismatch. Second, the model structures used by each type are quite different. Bio_ClinicalBERT has complex parameters and slow training, 3D ResNet has dense features and fast convergence, and Gene-MLP model is simple and its gradient is easily suppressed, resulting in different training paces and unstable fusion effects. In addition, the gene modality itself has sparse semantics, which is easily "overwhelmed" by other modalities in the fusion, affecting the expression of its diagnostic value.

[0172] To solve the above problems, the present application makes targeted optimization in the training strategy. First, a normalization module (LayerNorm + ) is introduced at the output end of each modality representation head to unify the feature scale; second, a learnable coefficient is set in the modality injection mechanism to dynamically regulate the semantic injection strength; third, dynamic weighting of modality loss is used in joint training to keep the gradient balance of each modality; fourth, the freeze-fine-tuning strategy is used for Bio_ClinicalBERT and 3D ResNet, only updating the high-level parameters in the fusion stage to slow down the training dominant bias.

[0173] Further, in step (3):

[0174] After the completion of the modal injection, each mode obtains the global semantic hints of other modes;

[0175] Further proposed is a cross-modal alignment module MultiAlign for mapping enhanced representation vectors of three modalities of text, image and gene to a shared semantic space, realizing feature alignment between modalities, and realizing semantic consistency and interoperability between modalities.

[0176] The inputs of the cross-modal alignment module for the features of three modalities of text, image and gene are respectively:

[0177] The enhanced representation vector of the text modality ;

[0178] The enhanced representation vector of the image modality ;

[0179] The enhanced representation vector of the gene modality ;

[0180] wherein, , , are the number of text words, the number of image words and the number of gene mutation sites respectively, is the unified embedding dimension;

[0181] For any modality pair , the attention weight mapping from the modality to the modality is defined by the following formula:

[0182] ;

[0183] ;

[0184] ;

[0185] wherein, are learnable projection parameters, represents a real matrix space of ; is the query matrix of the modality , and the dimension is consistent with ; is the projection parameter of ; is the key matrix of the modality , and the dimension is consistent with ; is​​​ The projection parameters; In the attention mechanism, T represents the transpose of the matrix. yes The transpose of is used to calculate and dot product between; Represents a unified embedding dimension; and Representing modes and modality The enhanced representation vector;

[0186] softmax represents the normalization function. It is the attention weight matrix;

[0187] Further calculation of cross-modal representation alignment under this mapping:

[0188] ;

[0189] ;

[0190] in, These are learnable projection parameters; From modality To mode The weighted sum of the value vectors; It is the attention weight matrix; It is modal The value vector; Representing modes The enhanced representation vector; It is the projection matrix of the value vector; Represent a The space of real matrices;

[0191] To improve the reversibility and stability of the aligned structure, a "path independence" loss term is introduced, requiring that the mode 𝑚, after being aligned and then back-projected, should approximate the original representation. The loss is defined as follows:

[0192] ;

[0193] in, It is modal The indivual Mapping to mode The representation of; The corresponding reverse mapping is then returned to the modality. The alignment result; Measuring modes and modality Consistency loss due to alignment and The superscript represents the generalization variable of the modal, which is a generalization of three modalities , , , and are respectively selected from , , ;

[0194] Finally, sum all the modalities to get the overall alignment consistency loss :

[0195] ;

[0196] Wherein, is the overall alignment consistency loss;

[0197] is the alignment consistency loss of modal and modal ;

[0198] is the alignment consistency loss of modal and modal ;

[0199] is the alignment consistency loss of modal and modal .

[0200] Further, in step (4):

[0201] The application adopts a multi-level fusion module FusionBLK to perform deep semantic fusion on the aligned modal features. The module is composed of fusion blocks, denoting the total number of fusion modules or the number of fusion blocks, and each layer contains four steps of modal quality weighting, sequence alignment, encoder fusion and residual connection.

[0202] The aligned representation of each modality is dynamically adjusted according to its metadata quality weight:

[0203] ;

[0204] ;

[0205] ;

[0206] Wherein, is the aligned modal feature sequence, is the modality quality score (derived from metadata);

[0207] is the aligned modality feature sequence, is the modality quality score (derived from metadata);

[0208] is the aligned modality feature sequence, is the modality quality score (derived from metadata);

[0209] is the enhanced text modality representation vector;

[0210] is the enhanced image modality representation vector;

[0211] is the enhanced gene modality representation vector;

[0212] Adjust the modality sequences of different lengths to a consistent size (such as by average pooling, upsampling, or zero padding), and unify them to a length :

[0213] ;

[0214] is the enhanced text modality representation vector;

[0215] is the enhanced image modality representation vector;

[0216] is the enhanced gene modality representation vector;

[0217] represents a real matrix space of ; the meaning is to unify the sequence length, represents the unified embedding dimension;

[0218] The features of the three modalities are concatenated and sent to the fusion encoder:

[0219] ;

[0220] wherein, " indicates concatenation in dimension; is the joint modality representation vector of the layer fusion; is the index of the fusion layer; represents a real matrix space, represents the uniform sequence length, represents the uniform embedding dimension;

[0221] is the enhanced text modality representation vector;

[0222] is the enhanced image modality representation vector;

[0223] is the enhanced gene modality representation vector;

[0224] The spliced multi-modal representation is sent to a fusion encoder module (which can be a Transformer, Restormer or Gated MLP):

[0225] ;

[0226] ;

[0227] wherein, is the output of the previous layer, initially a zero vector ; =0, i.e. a zero vector with a dimension of ; represents the uniform sequence length, represents the uniform embedding dimension; represents the fusion encoder; is the joint modality representation vector of the layer fusion; is the index of the fusion layer;

[0228] After layer iterations, the fusion representation of the layer is output , which is aggregated into a unified fusion semantic vector using Mean Pooling or Attention Pooling:

[0229] ;

[0230] wherein, is the fusion semantic vector; Pooling is the pooling operation; is a real space with a dimension of vector set, represents the uniform embedding dimension; is the fusion representation of the layer, is the index of the fusion layer; Mean Pooling is mean pooling, and Attention Pooling is attention pooling.

[0231] The multi-level fusion module (FusionBLK) proposed in the present application has obvious advantages compared with the existing conventional modal splicing or linear weighted fusion method. First, FusionBLK is based on a multi-layer fusion unit, which clearly divides the modal fusion process into four sub-steps: quality weighting, sequence alignment, splicing encoding, and unified aggregation. This step-by-step strategy not only ensures the clarity of the fusion process structure, but also improves the ability to perceive and adapt to modal differences at each step.

[0232] Second, the modal quality weight is introduced in the weighting stage , which realizes the adaptive adjustment of the contribution proportion of modal expression according to the quality of the original data, avoiding the interference of low-quality modal to the overall expression. In the sequence alignment stage, the length of each modal sequence is unified through pooling or padding, solving the semantic misplacement problem caused by inconsistent information length. In the encoding stage, the three-modal features are spliced and sent to the optional deep fusion network (such as Transformer, Restormer or Gated MLP), and are equipped with a residual connection structure, effectively alleviating the gradient vanishing and information loss problems in deep fusion, enhancing the stability of expression and the depth of fusion.

[0233] Finally, the fusion output is aggregated into a unified vector through Mean Pooling or Attention Pooling, not only improving the compactness of semantic expression, but also preserving the complementary semantic relationship between modalities.

[0234] Further, in step (5):

[0235] A two-layer structure system of rare disease knowledge graph is constructed, including an ontology layer and an instance layer. The ontology layer is used to represent standard medical knowledge in the field of rare diseases, including disease concepts, clinical symptoms, pathogenic genes, and treatment plans. The nodes come from authoritative medical ontology libraries, including Orphanet, UMLS, and HGNC. The instance layer is used to store specific case data and its corresponding multi-modal information. The nodes include case instance nodes, text nodes, image nodes, and gene nodes, all of which have unique identifiers and modal attributes. The ontology layer and the instance layer are connected by semantic edges including instance_of, aligned_to, has_symptom, and has_gene, forming a hierarchical tree graph structure. Among them, instance_of represents that a specific entity is a specific instance of a general category, aligned_to represents the corresponding relationship between two entities in different systems or standards, has_symptom represents the relationship between disease and symptom, and has_gene represents the relationship between disease, trait, or organism and related genes.

[0236] The establishment of modal reference, semantic enhancement, and traceability path edge relationships specifically includes:

[0237] The knowledge graph supports the generation of semantic enhancement edge relationships based on modal fusion representation, specifically including:

[0238] semantically_related_to: When the attention weight or fusion correlation between modal pairs exceeds the set threshold, a semantic association edge is automatically established between modal nodes.

[0239] is_similar_to: When the cosine similarity of the fusion representation between two case nodes is lower than the pre-set distance threshold , it is determined as a semantically similar case, and an is_similar_to edge is generated.

[0240] textual_equivalent_to: When the alignment confidence between the original text and the standardized UMLS term is higher than the set confidence threshold, a textual_equivalent_to edge is generated. The above edge relationships are automatically calculated and generated by the weight matrix and fusion vector space distance in the modal alignment module.

[0241] The knowledge graph system supports dual-channel query services for structural relationships and semantic spaces, deployed on a heterogeneous system architecture:

[0242] A graph database (Neo4j) is used to store ontology layer and instance layer nodes, entity attributes, and semantic edge relationships, supporting graph queries, path reasoning, and subgraph extraction operations based on the Cypher query language.

[0243] Vector database (Milvus) is used to store all the fusion vectors , which realizes semantic similarity calculation and vector neighbor retrieval, and can be used for recommendation of similar cases or modal collaborative retrieval.

[0244] Object storage system (Ceph) is used to save original modal data files, including medical images (such as DICOM), genomic files (such as VCF), and text data (such as electronic medical record XML / JSON), which can be quickly located and data retrieved through the path index in the atlas node attribute.

[0245] The three cooperatively constitute the storage, indexing, retrieval and calling infrastructure of the multi-modal knowledge graph, supporting downstream applications for intelligent diagnosis, semantic question answering and clinical reasoning.

[0246] Embodiment 1

[0247] As shown in Figure 1 , the embodiment of the present application provides a rare disease knowledge graph construction method based on modal injection and multi-modal fusion, comprising the following steps:

[0248] S1 Multi-modal data acquisition and standardization processing For the construction of the rare disease multi-modal knowledge graph, the present embodiment first acquires and standardizes the multi-source heterogeneous medical data, establishes a unified data expression system, and ensures the accuracy and consistency of subsequent graph construction.

[0249] S11 Multi-modal medical data acquisition In this step, by interfacing with various medical information systems, structured and unstructured data related to rare diseases are collected, mainly including the following three types of modal information:

[0250] (1) Structured clinical text data

[0251] Through the hospital information system (HIS), the core medical record information of the patient is obtained, including but not limited to: patient basic demographic information (gender, age, etc.); diagnosis record (initial diagnosis time, diagnosis conclusion, diagnosis code); treatment process record (treatment plan, drug use, follow-up summary, etc.).

[0252] (2) Medical image data

[0253] Through the picture archiving and communication system (PACS), the patient's relevant medical image data is collected, mainly including: CT images; MRI images; all data are uniformly kept in DICOM format.

[0254] (3) Genomic sequencing data

[0255] Patient's genomic information is obtained from the gene sequencing center, and the data sources include: whole genome sequencing (WGS); targeted region sequencing (such as rare disease related gene Panel); annotation content includes HGNC standard naming, ClinVar pathogenicity annotation, low frequency mutation site information, etc.

[0256] S12 Multimodal data standardization processing To ensure that data from different sources can be integrated and calculated, the above-mentioned multimodal data is uniformly standardized in this embodiment, including the following aspects:

[0257] (1) Text data processing: using regular expressions to clean up messy format and irrelevant semantic redundant content; desensitizing sensitive information such as patient name and ID number; retaining core clinical semantic content to ensure professionalism and data security.

[0258] (2) Image and gene data format standardization: medical image data is uniformly converted to a storage format that meets the DICOM protocol; gene data is converted to standard VCF / FASTA format, with HGNC naming and ClinVar annotation.

[0259] (3) Unified three-layer metadata structure storage: all standardized data will be stored in the following "three-layer metadata structure": disease layer (Schema Layer): used to define the standard medical knowledge system, covering disease ontology, clinical symptom terminology, pathogenic gene and treatment plan, etc. It includes standard disease name (ORPHA code, ICD-10 code), standardized symptom terminology (based on UMLS, HPO), pathogenic gene identification (HGNC standard symbol, ClinVar pathogenic annotation) and standard treatment plan (ATC classified drugs, operation code, etc.). This layer serves as the knowledge baseline for case layer and modality layer, ensuring data consistency and scalability across institutions and data sources.

[0260] Case layer (Instance Layer): carries specific individual case instances, records patient desensitized basic demographic information (gender, age, family history), diagnosis and treatment timeline (initial diagnosis date, diagnosis date, follow-up record) and medical events (such as examination items, treatment behavior, drug use, etc.). The case layer serves as a bridge between standard knowledge and actual data, supporting dynamic expansion, asynchronous supplement and time series modeling of cases, ensuring traceability of the association between instance data and standard knowledge.

[0261] Evidence Layer: Detailed records of each case corresponding to the multi-modal original data and its derived features, including structured text data (such as medical record description file path and entity labels), medical image data (DICOM image path, lesion mask file, image metadata such as resolution and device model), genomic data (VCF / FASTA format file, mutation annotation table), and feature vector representation extracted from each modality; The modality layer synchronously maintains data quality indicators (such as text term coverage, image FID score, and gene mutation pathogenicity score), and supports bidirectional mapping between original data path and graph node, meeting the traceability and quality control requirements.

[0262] S2: Synthesis and verification mechanism for missing modal data, for the common problem of multi-modal information missing in rare disease actual clinical data (such as lack of image or gene sequencing data), this embodiment proposes a multi-modal data synthesis method based on improved RareGAN-VAE, and designs a multi-level verification mechanism to ensure the authenticity and clinical usability of the synthesized data.

[0263] S21 Conditional generation of multi-modal data, due to the scarcity of case samples in actual rare disease data sets, there are often cases of missing some modalities; To improve data integrity and model training effect, this embodiment uses a conditional generation model RareGAN-VAE to complete the data based on a small number of labeled cases.

[0264] (1) Generator input design: The synthesis model takes random noise variable z and part of the existing modal data as input, for example:

[0265] Text description + partial gene sequence → synthesized image;

[0266] Text description + image features → synthesized gene fragments.

[0267] (2) Cross-modal attention mechanism:

[0268] The model uses a cross-modal attention module to fully utilize the semantic information carried in the remaining modalities when generating the target modality. For example:

[0269] When a case is missing a chest image, the model automatically synthesizes a CT image with characteristic rib cartilage lesions based on the gene mutation information (such as UBA1 p.Met41Val) and clinical description (such as "recurrent chondritis") it carries.

[0270] S22 Three-level synthesized data verification mechanism, to ensure the clinical credibility of the generated data, this embodiment designs a three-level verification mechanism including automatic evaluation and manual review:

[0271] (1) Automated similarity and consistency evaluation: Image quality evaluation: use Frechet Inception Distance (FID) index to evaluate the distribution difference of synthetic images and real images in feature space; Text semantic consistency detection: use medical pre-training language model (such as Med-BERT) to calculate the embedding similarity of original text and synthetic text, and eliminate samples with semantic deviation; Gene mutation rationality evaluation: analyze the generated gene mutation site, verify whether it exists in ClinVar or OMIM database, and ensure that the generated variation has known clinical relevance or pathogenic potential.

[0272] (2) Cross-modal semantic rule checking mechanism: Gene-disease association rule: the synthetic gene mutation should have explicit association (through mutation site or gene pathway) with the target disease in ClinVar / HGNC database; Image-pathological feature matching rule: if the disease clearly indicates a certain pathological sign (such as "costal cartilage thickening") in Radiopaedia or UMLS, the synthetic image should appear corresponding features in the target area (judged by deep model segmentation).

[0273] (3) Expert manual evaluation mechanism: randomly select 20% of synthetic samples; invite at least 3 rare disease doctors with clinical diagnosis experience to independently score the medical rationality and mutual coordination of text, image and gene data in synthetic samples; use 5-point scale score, get the average score, if the average is ≥ 4.0, mark as "clinically credible".

[0274] S23 Credibility labeling and iterative optimization: After the synthetic data is verified, it will be labeled and archived according to the quality grading mechanism, and will participate in the subsequent iterative optimization process.

[0275] Data labeling structure: Each synthetic data record is attached with metadata, including: synthetic_rank: quality level (A / B / C); fid_score: image generation quality index; expert_consensus: expert score average.

[0276] Model iteration mechanism: add the verified synthetic data to the training set; update the RareGAN-VAE model parameters; re-execute the synthesis and verification process after each iteration to gradually improve the generation quality.

[0277] Data archiving and storage: all synthetic data and verification metadata are stored in object storage system (such as Ceph); stored separately from real data, and through labeling to achieve traceability and controllability.

[0278] S31 Multi-modal data analysis and feature representation extraction;

[0279] This step parses the text, image, and gene three types of modal data from the standardized three-layer metadata structure respectively, and extracts their semantic features, represented as fixed-length vectors.

[0280] (1) Text modal processing: Extract structured symptom description and clinical text from the Schema layer, use the medical pre-training language model Bio_ClinicalBERT for context encoding, filter non-medical entities and retain standard terms using the UMLS clinical term library. To further enhance the medical expression ability of text semantics, introduce the symptom keyword positioning mechanism and synonym expansion strategy to improve the term recognition rate and semantic clarity. Finally, the text is represented as a 256-dimensional semantic vector, combining context information and standardized medical term aggregation characteristics.

[0281] (2) Medical image modal processing: Load DICOM format medical image data according to the path recorded in the Evidence layer, use 3D ResNet-50 to extract global anatomical features, and perform potential lesion region segmentation based on the U-Net model to generate a saliency mask. The mask guides the network to focus on the key lesion area and generates a local enhanced representation, while introducing the FID score as a quality score indicator to dynamically adjust the feature credibility. After fusion, a 512-dimensional image vector is obtained.

[0282] (3) Gene modal processing: Analyze the HGNC code and ClinVar annotation recorded in the Evidence layer, use a multilayer perceptron (MLP) to embed and encode the mutation site, and fuse the pathogenic grade weight and KEGG pathway context representation to improve the biological semantic interpretation ability of gene representation. To meet privacy protection requirements, a light Gaussian perturbation is added to the output vector. The final gene representation is a 128-dimensional embedded vector.

[0283] (4) Modal representation unification and standardization: Different dimensions of modal vectors are introduced into the modal representation head (MRH) for unified conversion:

[0284] Text modal → Text-MRH (BERT feature aggregation)

[0285] Image modal → Img-MRH (ResNet+U-Net feature compression)

[0286] Gene modal → Gene-MRH (MLP embedding)

[0287] After mapping, the three modalities are unified into 256-dimensional representation vectors, serving as cross-modal alignment input.

[0288] S32 Cross-modal semantic alignment and fusion mechanism design;

[0289] Modality representation unification and injection mechanism:

[0290] To alleviate the dimension and space offset problem of each modality representation, first, the different modalities are unified dimension mapping and modality semantic injection:

[0291] (1) Modality representation unification design

[0292] Text modality (Text-MRH): Bio_ClinicalBERT encoder is used to obtain the semantic representation of clinical text, and a 256-dimensional vector is output after aggregating the context vector;

[0293] Image modality (Img-MRH): Based on 3D ResNet, the global anatomical features are extracted, combined with U-Net to segment the lesion area, and finally output a 256-dimensional vector through light convolution layer dimension reduction processing;

[0294] Gene modality (Gene-MRH): The ClinVar annotated pathogenic mutation information is fused with the HGNC encoded gene site embedding vector, and the multi-layer perceptron (MLP) is used for encoding, and a 256-dimensional representation vector is output.

[0295] Let

[0296] Text modality representation vector : ;

[0297] Image modality representation vector : ;

[0298] Gene modality representation vector : ;

[0299] Where , , represent text, image and gene three modalities respectively.

[0300] (2) Modality injection mechanism

[0301] To improve the perception ability of each modality to other modalities, the modality injection mechanism is introduced. For the original representation vector of modality , its enhanced representation vector is defined as follows:

[0302] ;

[0303] where is the modal injection coefficient, regulating the strength of cross-modal semantic injection, represents the global vector of other modalities.

[0304] MultiAlign

[0305] After the completion of the modal injection mechanism, each modal feature has obtained the global semantic hint of other modalities, forming the enhanced representation vectors of text, image, and gene modalities , , ;

[0306] Further, the MultiAlign module is proposed, which aims to map the enhanced representations of the three modalities into a unified shared semantic space, thereby realizing the semantic consistency and interoperability between modalities. This module adopts a bidirectional path-independent attention mechanism and introduces an alignment consistency loss to ensure that the representations between different modalities tend to be symmetric in structure and semantically matched.

[0307] (1) The input of the three modalities is respectively:

[0308] Text modality enhanced representation vector : ;

[0309] Image modality enhanced representation vector : ;

[0310] Gene modality enhanced representation vector : ;

[0311] where , , are the number of text , image , and gene mutation sites, respectively, is the unified embedding dimension;

[0312] For any modal pair , the attention weight mapping from modality to modality is denoted as , which is defined by the following formula:

[0313] ;

[0314] ;

[0315] ;

[0316] where, is a learnable projection parameter, denotes a real matrix space; is a query matrix of modal with the same dimension as ; is a projection parameter of ; is a key matrix of modal with the same dimension as ; is a projection parameter of ; T in denotes the transpose of a matrix, in the attention mechanism, is the transpose of , used to calculate the dot product between and ; represents the unified embedding dimension; and represent the enhanced representation vectors of modal and modal respectively;

[0317] softmax denotes a normalization function, is an attention weight matrix;

[0318] Further calculate the cross-modal alignment representation under this mapping:

[0319] ;

[0320] ;

[0321] where, is a learnable projection parameter; is a weighted sum of value vectors from modal to modal ; is an attention weight matrix; is a value vector of modal ; denotes an enhanced representation vector of modal ; is a projection matrix of value vectors; denotes a real matrix space;

[0322] (3) To enhance the stability and reversibility of the alignment process, this module introduces the "path independence" constraint, which requires that the bidirectional alignment between the two modalities has consistency. The specific loss function is defined as follows:

[0323] ;

[0324] in, It is modal The indivual Mapping to mode The representation of; The corresponding reverse mapping is then returned to the mode. The alignment result; Measuring modes and modality Consistency loss due to alignment and Subscripts represent modality generalization variables, which are applied to the three modes. , , A general term, and Selected from , , One of them;

[0325] Summing over all modal combinations yields the overall alignment consistency loss. :

[0326] ;

[0327] in, It is a loss of overall alignment consistency;

[0328] It is modal and modality Consistency loss due to alignment;

[0329] It is modal and modality Consistency loss due to alignment;

[0330] It is modal and modality Consistency loss due to alignment.

[0331] Meanwhile, to enhance the targeted modeling of semantics for each modality during the alignment phase, the following strategies are introduced:

[0332] Text modality: Highly matched with medical terminology Allocate more attention to improve the accuracy of semantic alignment between symptoms and genes, and symptoms and images;

[0333] Image modality: Local guiding vectors are generated using saliency masks of lesion regions and participate in the construction of attention matrices;

[0334] Gene modality: Adjust the attention distribution by combining the pathogenic grade of mutation sites and the participation of KEGG pathways, so that disease-related genes are preferentially matched to corresponding clinical manifestations and imaging signs.

[0335] Multi-level fusion module (FusionBLK)

[0336] After completing the semantic alignment of multi-modal features, the present application further introduces a multi-level fusion module FusionBLK for integrating aligned features from three modalities of text, image and gene to generate a structured unified representation of fusion semantic vector to support subsequent entity embedding and reasoning tasks of knowledge graph. This module adopts a multi-level residual aggregation structure and combines the modality quality score in the metadata to realize weight regulation, ensuring that the fusion result has both structural integrity and semantic consistency.

[0337] The FusionBLK module is composed of Fusion blocks (Fusion Block), representing the total number of layers or the number of fusion blocks of the fusion module. Layer by layer, modalities are aggregated and semantic enhanced. Each layer includes the following steps:

[0338] (1) Adjust each modality's aligned representation according to its metadata quality weight dynamically:

[0339] ;

[0340] ;

[0341] ;

[0342] wherein, is the aligned modality feature sequence, is the modality quality score; (derived from metadata)

[0343] is the aligned modality feature sequence, is the modality quality score; (derived from metadata)

[0344] is the aligned modality feature sequence, is the modality quality score; (derived from metadata)

[0345] is the enhanced text modality representation vector;

[0346] is the enhanced image modality representation vector;

[0347] is the enhanced gene modality representation vector;

[0348] (2) Adjust the modality sequences of different lengths to a consistent size (such as by average pooling, upsampling, or zero padding), and unify them to a length :

[0349] ;

[0350] is the enhanced text modality representation vector;

[0351] is the enhanced image modality representation vector;

[0352] is the enhanced gene modality representation vector;

[0353] represents a real matrix space , and has the meaning of unifying the sequence length, represents the unified embedding dimension;

[0354] (3) Concatenate the features of the three modalities and send them to the fusion encoder:

[0355] ;

[0356] wherein, represents concatenation in the dimension; is the joint modality representation vector of the layer fusion; is the index of the fusion layer; represents a real matrix space , and represents the unified sequence length, represents the unified embedding dimension;

[0357] is the enhanced text modality representation vector;

[0358] is the enhanced image modality representation vector;

[0359] is the enhanced gene modality representation vector;

[0360] (4) The spliced multi-modal representation is sent to a fusion encoder module (which can be a Transformer, Restormer or Gated MLP):

[0361] ;

[0362] ;

[0363] wherein, is the output of the previous layer, initially a zero vector ; =0, i.e. a zero vector of dimension ; denotes a uniform sequence length, represents a uniform embedding dimension; denotes the fusion encoder; is the joint modal representation vector of the layer fusion; is the index of the fusion layer;

[0364] After layer iterations, the fusion representation of the layer is output , which is aggregated into a unified fusion semantic vector using Mean Pooling or Attention Pooling:

[0365] ;

[0366] wherein, is the fusion semantic vector; Pooling is the pooling operation; is a real space, a vector set of dimension , represents a uniform embedding dimension; is the fusion representation of the layer, is the index of the fusion layer; Mean Pooling is mean pooling, and Attention Pooling is attention pooling.

[0367] At the same time, to enhance the individualized difference perception of modal fusion expression, the following optimization mechanism is designed:

[0368] Text modality: structured term coverage and paragraph semantic confidence are introduced as attention factors during fusion;

[0369] Image modality: dynamic weight correction is performed based on lesion area distribution density and FID quality score;

[0370] Gene modality: fuse the information of pathway involvement, pathogenic grade, and site density, to improve the semantic proportion of disease explanation feature vector.

[0371] S4 Construction method of multi-modal medical knowledge graph, construct a multi-modal medical knowledge graph for rare disease field, adopt a double-layer hierarchical structure design, including schema layer and instance layer, realize the structured semantic mapping from general medical knowledge to specific case data.

[0372] S41 Hierarchical structure design of schema layer and instance layer

[0373] (1) Schema layer (Schema layer), the schema layer is used to describe the general medical knowledge structure related to rare diseases, which has high stability and standardization characteristics, and the node types include:

[0374] (2) Disease concept node: cited from Orphanet coding system, records standard disease name and ORPHA code;

[0375] Clinical symptom node: cited from UMLS clinical terminology system, standardized representation of disease performance;

[0376] Pathogenic gene node: cited from HGNC naming system, combined with ClinVar annotation to represent known pathogenic mutations;

[0377] Therapeutic means node: such as drugs, surgical methods, reference WHO ATC classification;

[0378] Relationship between nodes: including has_symptom (disease-symptom), related_gene (disease-gene), treated_by (disease-treatment), etc.

[0379] (2) Instance layer (Instance layer), the instance layer is used to express the specific case information collected or generated, which is independent of the structure of the schema layer and dynamically expanded, each node has a unique identifier, and records its source modality, data quality level, path information, and whether it is a synthetic sample and other key metadata in the attribute. Node types include:

[0380] Case instance node: represents a single patient sample, carries attributes such as gender, age, and visit time;

[0381] Text modality node: structured medical record description, symptom text;

[0382] Imaging modality node: index node of CT, MRI and other medical image files;

[0383] Gene modality node: gene mutation record, such as VCF file or HGVS expression;

[0384] Structural symptom node: parsed standardized UMLS term entry.

[0385] S42 fusion vector embedding and instance mapping mechanism, using the modal fusion semantic vector generated in S3 step , the instance nodes in the atlas are embedded and the structural connection relationship between multiple modalities is constructed:

[0386] (1) Vector embedding identification: the fusion semantic vector is taken as the semantic representation of the case node; the vector is bound to the vector database (such as Milvus) through a unique identifier, supporting vectorized retrieval; when writing the atlas, the vector ID is taken as an instance node attribute field.

[0387] (2) Automatic generation of semantic enhancement edge relationship: using the inter-modal attention weight learned by the MultiAlign module to judge the semantic coupling strength between modalities; if the weight exceeds the set threshold, a cross-modal semantic edge is automatically generated, such as:

[0388] semantically_related_to: text symptom gene mutation;

[0389] is_similar_to: high semantic similarity relationship between case nodes.

[0390] S43 structure rule driven and semantic perception driven relationship construction, the strategy of the present application for constructing the edge relationship of the atlas is divided into two categories:

[0391] (1) Structure rule driven path

[0392] Based on the entity relationship template predefined by the medical knowledge base, the static semantic connection edge is constructed, such as:

[0393] has_symptom: disease → symptom (according to the Orphanet symptom mapping table);

[0394] has_gene: disease → gene (according to ClinVar mutation attribution);

[0395] instance_of: case node → disease ontology node;

[0396] All such relationships can be automatically generated by templates during the atlas writing stage, with fixed structure and stable logic.

[0397] (2) Semantic perception driven path

[0398] Combining the semantic information in the modal fusion and alignment process, dynamically determine the potential semantic relevance, generate the following edge types:

[0399] is_similar_to: the fusion vector distance is less than the threshold , determine that it is a similar case;

[0400] aligns_with_image: the maximum weight path between gene features and image features;

[0401] textual_equivalent_to: when the semantic alignment degree between unstructured text and standard UMLS terms is higher than the set confidence threshold, it is automatically generated.

[0402] S44 modality node and traceability path design, in order to enhance the interpretability and data traceability of the graph, the original modality data node is explicitly modeled, and the "traceable path" between modality entities and instances is constructed:

[0403] (1) The modality node has the following characteristics:

[0404] Has an independent identifier (such as text_id, image_id, gene_id);

[0405] Node attributes record: physical path, modality category, quality score, whether it is synthetic;

[0406] Establish modality edges with corresponding case nodes, such as has_image, has_gene, and has_symptom.

[0407] (3) Path organization logic:

[0408] Modality traceability path: such as case node → has_image → DICOM file node, which can be used for data retrieval;

[0409] Symptom-gene-image path: through has_symptom → textual_equivalent_to → related_gene → aligns_with_image to realize modality collaborative semantic reasoning;

[0410] Modality aggregation path: calculate the semantic aggregation degree of multiple modality nodes under the same instance to generate a fused_representation edge, realize modality question and answer and intelligent retrieval.

[0411] S45 knowledge graph deployment and service interface, the multi-modal knowledge graph constructed in this embodiment supports two types of heterogeneous query methods, respectively facing the structure logic and the vector space, and the structure is deployed as follows:

[0412] Graph database (Neo4j): Store structured nodes and semantic relationship and modal reference path; Support Cypher query, path reasoning, subgraph extraction and other operations.

[0413] Vector database (Milvus): Store semantic vector fusion Support vector-based similar case recall and semantic association reasoning.

[0414] Object storage system (Ceph): Store raw modal data files (CT images, gene sequences, structured text); Provide the ability to quickly locate underlying data through graph path.

[0415] S46 The query and auxiliary diagnosis method of the multi-modal rare disease knowledge graph constructed by the application is suitable for the case that the user input forms are different in actual clinical scenarios, supports two paths of structured query and semantic vector retrieval, and has good adaptability and interpretability.

[0416] In practical applications, users may input standardized structured medical information, such as specific symptom terms, rare disease names, gene sites, etc., at which time the system enters the structured query mode. This mode accesses the graph based on the Neo4j graph database, and the system first performs entity recognition and standardization mapping on the user input, mapping the keywords to entity node types in the graph, such as Disease, Gene, or Symptom. The system then automatically generates a Cypher query statement based on the graph relationship structure and calls the graph database interface to perform retrieval operations. For example, if the user input keyword is "IKBKG", the system will construct the query instruction: "MATCH (d:Disease)-[:ASSOCIATED_WITH]->(g:Gene{name:'IKBKG'}) RETURN d.name, g.name", and return all disease information associated with the gene in the database. The query result includes the basic attributes of the hit nodes, the relationship type between the nodes and its semantic weight, and the front-end interface supports graph structure visualization, path highlighting, node detail expansion, edge attribute viewing and other interactive functions, making it easy for professional users to trace and verify and provide clinical knowledge support.

[0417] When the user input is an unstructured text described in natural language, the system automatically switches to the semantic vector retrieval mode. The system first performs semantic analysis and embedding coding on the input text, and uses Bio_ClinicalBERT and other medical field pre-training models to map the text into a fixed-length semantic vector representation. During the graph construction phase, the system has pre-generated embedding vector representations for all key nodes such as diseases, symptoms, and genes, and stored them. The query vector will be compared with the embedding vectors of all entity nodes in the graph, and the cosine similarity index is usually used as the similarity measurement function. The Top-K similar nodes are returned as the matching results. Subsequently, the system will perform path backtracking based on the hit nodes, extract their upstream and downstream knowledge structures, construct multi-hop semantic chains containing symptoms-diseases-genes, and provide visual display and explanation of the path graph.

[0418] In the query result output stage of the system, regardless of the path used, a structured result set is output, including: matching entity name, entity type, semantic matching confidence, path structure relationship, node attribute description, data source reference (such as ClinVar, Orphanet ID), etc. Information, supporting path export, result download and original data traceable access.

[0419] To evaluate the adaptability and effectiveness of the method in actual clinical conditions, a test set containing 50 real or simulated rare disease cases is constructed, covering text input, gene input, natural language questions and other input forms. The system achieves a Top-1 query accuracy of 86% and a Top-3 accuracy of 94%. At the same time, doctors with experience in diagnosing rare diseases are invited to conduct blind review, and the rationality of the output results, semantic explanation ability and clinical reference value are scored, with an average score of more than 4.6 / 5, verifying the high adaptability and application prospect of the system in the rare disease auxiliary query task.

[0420] The above is the preferred embodiment of the present application. It should be noted that for those skilled in the art, without departing from the principles of the present application, several improvements and refinements can be made, which should also be considered within the scope of protection of the present application.

Claims

1. A rare disease knowledge graph construction method based on modal injection and multi-modal fusion, characterized in that, The method comprises the following steps: S1, collecting multi-modal medical information including text, image and gene; S2, standardizing the multi-modal medical information and constructing a three-layer metadata structure; S3, completing missing modal data, extracting features and converting the dimensions of the modal data respectively, obtaining a unified format global semantic modal representation vector, processing the modal representation vector using a modal injection mechanism to enhance cross-modal semantic perception, obtaining an enhanced representation vector, and realizing cross-modal representation alignment in a shared semantic space through a cross-modal alignment module; The completion of the missing modal data specifically includes: based on the RareGAN-VAE conditional generation model, the missing modal data is completed for the modal missing condition, and a three-level verification mechanism including automatic evaluation, semantic rule verification and expert scoring is used to mark the reliability; RareGAN-VAE is a hybrid generation model combining variational autoencoder and generative adversarial network; The feature extraction and dimension conversion of the modal data specifically include: The features of the text, image and gene data are extracted respectively, the original representations of the three types of modalities are mapped to the same dimension, and a modal representation head is designed to extract a unified format global semantic modal representation vector; Text modality: Bio_ClinicalBERT encoder is used to obtain clinical text semantic representation, and a 256-dimensional vector is output after aggregating the context vector; Bio_ClinicalBERT is a BERT model specially pre-trained for biomedical and clinical text; Image modality: 3D ResNet is used to extract global anatomical features, U-Net is used to segment the lesion area, and a lightweight convolution layer is used for dimension reduction processing, finally outputting a 256-dimensional vector; 3D ResNet is a three-dimensional extension of the traditional residual network, and U-Net is a classic convolutional neural network for medical image segmentation; Gene modality: The ClinVar annotated pathogenic mutation information is fused with the HGNC encoded gene site embedding vector, and a multi-layer perceptron is used for encoding to output a 256-dimensional representation vector; S4, multi-level semantic fusion is performed on the modal features after cross-modal representation alignment to obtain a unified fusion semantic vector; S5, based on the unified fusion semantic vector, a double-layer structure system of rare disease knowledge graph including ontology layer and instance layer is constructed, and edge relationships such as modal reference, semantic enhancement and traceability path are established in the graph.

2. The method of claim 1, wherein, The text, image and gene include structured clinical text, medical image and genomic data derived from rare disease patients, and the genomic data includes low-frequency mutation sites and their pathogenicity annotation information.

3. The method of claim 1, wherein, The three-layer metadata structure in S2 includes disease layer, case layer and modal layer, the disease layer is used to record standard disease codes, core symptom terms and related pathogenic gene identifiers; the case layer is used to record patient basic information, treatment timeline and family medical history; and the modal layer is used to record modal data path, semantic label, feature vector and quality score.

4. The method of claim 3, wherein, The model in S3 takes the fragment features of the existing modalities and the noise vector as input, and the synthesized data of the missing modalities as output. The automatic evaluation uses Frechet Inception Distance to evaluate the similarity of images, which is an indicator for evaluating the quality of images generated by a generation model. The ClinVar database is used to verify the matching of gene mutations and the UMLS definition of gene and disease phenotypes. UMLS is the Unified Medical Language System. The semantic rule verification includes gene-disease association rules and image-pathological feature matching rules. The gene-disease association rule is that the synthesized gene mutation should have a clear association with the target disease in the ClinVar / HGNC database. The image-pathological feature matching rule is that if a disease clearly indicates a certain pathological sign in Radiopaedia or UMLS, the synthesized image should have the corresponding features in the target area. Radiopaedia is a medical imaging encyclopedia and teaching platform. The expert score is double-blind scored by experts with clinical experience in rare diseases, and the average score should not be lower than the preset threshold. The 256-dimensional representation vectors of the text modality, image modality, and gene modality are denoted as: Textual modality representation vector : ; Image modality representation vector : ; Gene modality representation vector : ; wherein, , , respectively represent text, image and gene three modalities.

5. The method of claim 1, wherein, The modality injection mechanism in S3 specifically includes: For each modality representation vector, introduce the global semantic representation of other modalities to enhance cross-modal semantic perception ability; For a modal original representation vector enhanced representation vector is defined by the following equation: ； wherein, is a modal injection adjustment coefficient, represents a global vector of other modes.

6. The method of claim 5, wherein, The cross-modal alignment module in S3 specifically includes: Using a representation mapping method based on a bidirectional attention mechanism, the enhanced representation vectors of the text, image, and gene modalities are mapped to a shared semantic space to realize feature alignment between modalities. The inputs of the cross-modal alignment module for the text, image, and gene modalities are: Textual modality augmented representation vector : ; Image modality augmented representation vector : ; Gene modality augmented representation vector : ; wherein, , , are the number of text , image , gene mutation sites, respectively, is the unified embedding dimension; For any modality pair the attention weight mapping from modality to modality is defined by ； ； ； wherein, are learnable projection parameters, denotes a real matrix space; is a query matrix of modal with the same dimension as ; is a projection parameter of ; is a key matrix of modal with the same dimension as ; is a projection parameter of ; T in the above equations denotes the transpose of a matrix, in the attention mechanism, is the transpose of , used to calculate the dot product between and ; represents the unified embedding dimension; and represent the enhanced representation vectors of modal and modal , respectively; softmax represents a normalization function, is an attention weight matrix; Calculate the cross-modal alignment representation under this mapping: ； ； wherein, is a learnable projection parameter; is a weighted sum of value vectors from modalities to modality ; is an attention weight matrix; is a value vector of modality ; denotes an enhanced representation vector of modality ; is a projection matrix of value vectors; denotes a real matrix space of one ; To enhance the consistency and reversibility of alignment, a path-agnostic consistency loss is introduced : ； wherein, is a modality of the th mapping to a representation of a modality ; the alignment result after the corresponding reverse mapping back to the modality ; a consistency loss measuring the alignment of the modality and the modality , and the superscript denotes the generalized variable of the modality, which is a collective term for the three modalities , , , and are each selected from one of , , . The final alignment consistency loss is summed over all modalities : ； wherein, is the overall alignment consistency loss; is a modality and a modality alignment consistency loss; is a modality and a modality alignment consistency loss; is the modality and the modality alignment consistency loss.

7. The method of claim 1, wherein, The multi-level semantic fusion in step S4 specifically includes: A multi-level fusion module FusionBLK is adopted to perform deep semantic fusion on the aligned modal features. The module is composed of a fusion block, N represents the total number of fusion modules or the number of fusion blocks, and each layer includes four steps of modal quality weighting, sequence alignment, encoder fusion, and residual connection. Adjust the alignment representation of each modality according to its metadata quality weight: ； ； ； wherein, is the aligned modality feature sequence, is the modality quality score; is the aligned modality feature sequence, is the modality quality score; is the aligned modality feature sequence, is the modality quality score; is the enhanced text modality representation vector; is the enhanced image modality representation vector; is the enhanced gene modality representation vector; Adjust the different length of modal sequences to a consistent size, uniform to length : ； is the enhanced text modality representation vector; is the enhanced image modality representation vector; is the enhanced gene modality representation vector; represents a real matrix space, means uniform sequence length, represents uniform embedding dimension; Concatenate the features of the three modalities and send them to a fusion encoder: ； wherein, represents the concatenation of the dimensions; represents the concatenation of the dimensions; is the joint modality representation vector of the layer fusion; is the index of the fusion layer; represents a real matrix space, represents a uniform sequence length, represents a uniform embedding dimension; is the enhanced text modality representation vector; is the enhanced image modality representation vector; is the enhanced gene modality representation vector; Send the concatenated multi-modal representation to a fusion encoder module: ； ； where, is the output of the previous layer, initially a zero vector ; = 0, i.e., a zero vector of dimension ; denotes the uniform sequence length, represents the uniform embedding dimension; denotes the fusion encoder; is the joint modality representation vector of the th fusion layer; is the index of the fusion layer; After layer iteration, output the fusion representation of the layer, which is aggregated into a unified fusion semantic vector using Mean Pooling or AttentionPooling: ​ ； wherein, is the fused semantic vector; Pooling is the pooling operation; is the real space with dimension is the set of vectors, represents the unified embedding dimension; is the fused representation of the layer, is the index of the fused layer; Mean Pooling is the mean pooling and Attention Pooling is the attention pooling.

8. The method of claim 1, wherein, The construction of the two-layer structure system of the rare disease knowledge graph in step S5 specifically includes: The ontology layer is used to represent standard medical knowledge in the field of rare diseases, including disease concepts, clinical symptoms, pathogenic genes, and treatment plans. The nodes come from authoritative medical ontology libraries, including Orphanet, UMLS, and HGNC. The instance layer is used to store specific case data and its corresponding multi-modal information. The nodes include case instance nodes, text nodes, image nodes, and gene nodes, all of which have unique identifiers and modality attributes. The ontology layer and the instance layer are connected through semantic edges including instance_of, aligned_to, has_symptom, and has_gene, forming a hierarchical tree graph structure. Among them, instance_of indicates that a specific entity is a specific instance of a certain general category, aligned_to indicates the correspondence of two entities in different systems or standards, has_symptom indicates the relationship between disease and symptoms, and has_gene indicates the relationship between disease, trait or organism and related genes.

9. The method of claim 1, wherein, The edge relationships such as modal reference, semantic enhancement and traceability path in the graph established in step S5 specifically include: Utilize the modal fusion semantic vector generated in the S4 step The instance nodes in the graph are embedded and represented, and the structural connection relationship between the multiple modalities is constructed. The original modal data nodes are explicitly modeled, and a traceable path between the modal ontology and the instance is constructed. The construction of the structural connection relationship between multiple modes uses the attention weight between modes learned by the MultiAlign module to judge the semantic coupling strength between modes; MultiAlign is a multi-sequence alignment algorithm or software module; if the weight exceeds the set threshold, a cross-modal semantic edge is automatically generated, which specifically includes: semantically_related_to: when the attention weight or fusion correlation between modal pairs exceeds the set threshold, a semantic association edge is automatically established between modal nodes; is_similar_to: when the cosine similarity of the fusion representation between two case nodes is lower than a pre-set distance threshold then determine as a semantically similar case and generate an is_similar_to edge; textual_equivalent_to: when the alignment confidence between the original text and the standardized UMLS term is higher than the set threshold, a textual_equivalent_to edge is generated; the above edge relationship is automatically calculated and generated by the weight matrix and fusion vector space distance in the modal alignment module.

10. The method of claim 1, wherein, The knowledge graph supports multi-source heterogeneous query methods, respectively facing structured query and semantic vector retrieval, and the deployment structure includes: a graph database module for storing structured entity nodes, semantic edge relationships and modal reference paths, supporting graph query, path reasoning and subgraph extraction operations based on the Cypher query language; Cypher is a declarative query language for Neo4j graph database; a vector database module for storing the fused semantic vectors of the respective instance nodes supporting vector retrieval and case recall based on semantic similarity an object storage system module for storing original modal data files, including structured clinical text, medical images and genomic data of rare disease patients, and providing path backtracking capability, supporting mapping and retrieval with graph node attributes.

Images