Brain tumor survival prediction method and system based on multi-modal medical knowledge graph

Abstract

The invention provides a brain tumor survival prediction method and system based on a multi-modal medical knowledge graph, and belongs to the technical field of brain tumor survival prediction. The multi-modal medical knowledge graph based on third-party knowledge base fusion is constructed; performing feature extraction on the brain tumor multi-modal data; searching an entity corresponding to the brain tumor related data in the multi-modal medical knowledge graph, and converting the entity into feature representation by using an entity representation learning method; the learned feature representation related to the brain tumor type complements the missing data mode, and finally the complemented features are input into a pre-trained survival prediction model to achieve brain tumor survival prediction. According to the multi-modal medical knowledge graph, comprehensive medical knowledge support meeting clinical requirements is provided; the multi-modal mapping knowledge domain is used for missing modal completion of brain tumor survival prediction, and a completion feature is generated by querying an associated entity through the mapping knowledge domain, so that the problem of weak modal missing processing capability in the prior art is solved.

Description

Technical Field

[0001] This invention relates to the field of brain tumor survival prediction technology, specifically to a brain tumor survival prediction method and system based on a multimodal medical knowledge graph. Background Technology

[0002] Current research mainly revolves around two directions: "construction of multimodal knowledge graphs" and "multimodal data fusion prediction." Research on multimodal knowledge graph construction largely focuses on entity recognition and relation extraction in the biomedical field, providing a methodological foundation for knowledge graph construction. For example, Liu et al. applied BERT to extract low-level text features for named entity recognition in the biomedical field, which can assist in identifying brain tumor-related entities such as "glioblastoma" and "IDH1 gene." Zheng et al. defined multimodal relation extraction (MRE) as classifying the textual relationship between two entities using visual information, providing a methodological reference for mining cross-modal relationships such as "MRI enhancement features - brain tumor subtypes." Jha et al. used the Graph-BERT model to encode the protein-protein interaction (PPI) network graph; its node feature learning logic can be borrowed from the representation learning of "gene-disease" nodes in the brain tumor knowledge graph, providing ideas for the vector representation of entities in multimodal knowledge graphs. These research findings in biomedical entity recognition, cross-modal relationship extraction, and node feature learning provide technical support for the construction of multimodal knowledge graphs in brain tumor scenarios.

[0003] In studies on multimodal data fusion for predicting brain tumor survival, existing findings have largely improved prediction results by integrating data from different modalities, providing a reference for knowledge-guided fusion. In pan-cancer research, Tong et al. used the principles of complementarity and consensus to integrate multi-omics data such as gene expression, DNA methylation, miRNA expression, and copy number variation for breast cancer survival prediction; their multi-source information integration approach provides a reference for multimodal data fusion in brain tumors. In the field of brain tumors, Pei et al. proposed a context-aware deep learning method that uses structural multimodal magnetic resonance imaging (MRI) for brain tumor segmentation, subtype classification, and overall survival prediction, effectively mining anatomical information from imaging modalities. Zhou et al. proposed an end-to-end overall survival prediction model (multimodal multichannel network) that extracts latent and high-level features from different MR scans through a specific modal network for brain tumor survival prediction. Zhuang et al. proposed a weakly supervised, interpretable multimodal deep learning model that integrates histological, radiological, and genomic features for glioma survival prediction, achieving the synergistic utilization of multimodal data.

[0004] Multimodal data fusion is a core research direction in the field of brain tumor survival prediction. Different modalities of data (such as MRI images, pathological slides, genomic data, and demographic information) can supplement patient information from multiple dimensions, including anatomical structure, cell morphology, molecular mechanisms, and clinical background, providing support for accurate prognostic assessment. Knowledge graphs, as structured knowledge carriers, can integrate prior knowledge in the medical field (such as gene-disease-survival associations and imaging features-pathological grade correspondences), providing logical constraints for multimodal data association and having significant application potential in medical prediction tasks.

[0005] Although multimodal data fusion provides a key technical path for predicting brain tumor survival, and knowledge graphs lay the foundation for integrating domain priors and strengthening modal associations, a series of core challenges remain to be addressed in actual clinical applications and model development. Specifically, these challenges manifest in three main aspects: First, multimodal heterogeneous information is difficult to integrate effectively. Existing multimodal fusion methods largely rely on data-driven feature interactions, failing to incorporate prior knowledge in the brain tumor domain and thus failing to capture clinically meaningful modal associations, resulting in limited predictive performance. Second, the problem of missing modalities in clinical data is prominent. Patients often experience partial modal loss due to limitations in examination conditions. Existing processing methods either directly discard samples containing missing modalities, exacerbating the scarcity of brain tumor samples and further reducing the model's generalization ability, or randomly reconstruct missing features using the maximum mean difference, ignoring medical logic, and the generated features may contradict the actual condition. Third, the medical interpretability of the model's predictive logic is insufficient, failing to provide traceable predictive evidence for clinical practice, which is detrimental to doctors' trust in the prediction results and their reference for clinical decision-making.

[0006] How to incorporate medical knowledge to guide the fusion of multimodal heterogeneous data in brain tumor survival prediction, while simultaneously achieving reasonable medical completion of missing modalities and improving model interpretability, is a current research challenge. Summary of the Invention

[0007] The purpose of this invention is to provide a brain tumor survival prediction method and system based on a multimodal medical knowledge graph, to solve at least one of the technical problems existing in the background art. This invention considers the characteristics of multimodal data and the supporting role of medical knowledge graphs, integrating multi-source modal information and domain priors. In model construction, it achieves effective interaction of multimodal heterogeneous features and reasonable medical completion of missing modalities, thereby mitigating the impact of low multimodal fusion efficiency, poor data integrity, and insufficient interpretability, thus obtaining a more accurate brain tumor survival prediction model that meets clinical needs. This model utilizes knowledge graphs to integrate domain prior knowledge, improving the fusion efficiency of multimodal heterogeneous data, achieving reasonable medical completion of missing modalities, reducing the dependence of traditional multimodal prediction models on complete datasets, and further enhancing the model's predictive performance and interpretability in real-world clinical scenarios.

[0008] To achieve the above objectives, the present invention adopts the following technical solution:

[0009] In a first aspect, the present invention provides a brain tumor survival prediction method based on a multimodal medical knowledge graph, comprising:

[0010] Construct a multimodal medical knowledge graph based on the fusion of third-party knowledge bases;

[0011] Feature extraction was performed on multimodal brain tumor data. For the radiological modality, tumor and edema regions were extracted from MRI, and slices containing the largest lesion area were selected. Image features and manual features were extracted and concatenated to form a radiological feature vector. For the pathological modality, deep image features were extracted from pathological slices, and a cell interaction graph was constructed using a graph convolutional network to extract graph features. The two were then fused as pathological features. For the genomics modality, a feature set including copy number variation (CNV) features and IDH1 gene mutation status was constructed, and genomic features were obtained using a self-normalization network combined with regularization. For the demographic modality, one-hot encoding was performed on different attribute features, and feature embedding was carried out to maintain data normalization characteristics.

[0012] In a multimodal medical knowledge graph, the corresponding entities for brain tumor-related data are found, and entity representation learning methods are used to convert them into feature representations.

[0013] The learned features related to brain tumor types are used to complete the missing data modalities, and finally the completed features are input into a pre-trained survival prediction model to achieve brain tumor survival prediction.

[0014] As a further limitation of the first aspect of this invention, the completion of missing genomic modalities is supported by structured medical priors of a multimodal medical knowledge graph, and is completed through gene entity association, vector representation, and feature extraction. This includes: firstly, retrieving a list of core genes directly associated with brain tumors from the multimodal medical knowledge graph. The associations of these genes are derived from entity alignment results from a third-party knowledge base, ensuring that gene selection conforms to molecular biology principles; loading a pre-trained Word2Vec model to convert each gene name into a low-dimensional dense vector. To ensure that the vectors reflect the semantic associations between genes, cosine similarity is used to calculate vector similarity; inputting the gene vectors into a deep survival model containing three fully connected layers, introducing nonlinearity using the ELU activation function, and preventing overfitting through an Alpha Dropout layer. The training of the deep survival model uses a Cox partial likelihood loss function to optimize the association between features and survival time, and finally outputs the completed genomic features.

[0015] As a further limitation of the first aspect of the present invention, the completion of missing radiological modalities relies on the BraTs2020 dataset associated with a knowledge graph, reusing the core process of single-modal feature embedding and optimizing the details; including: firstly, using the nnUNet segmentation model to automatically segment MRI images, extracting regions of interest, and achieving high-precision segmentation through an encoder-decoder architecture; selecting 2D key slices containing the largest tumor region, adjusting the size to obtain standardized images; extracting deep features through a pre-trained ResNet-18 model, and extracting handcrafted features using PyRadiomics to form a handcrafted feature vector; and integrating the deep features and handcrafted features using an average vector fusion strategy to obtain the completed radiological modal features.

[0016] As a further limitation of the first aspect of the present invention, the features are first L2 normalized before fusing the deep features and the handcrafted features:

[0017]

[0018] To verify the reliability of the completed features, a consistency index was used for evaluation:

[0019]

[0020] Where M is the number of comparable sample pairs. To complete the risk score corresponding to the feature, I( ) is an indicator function; Indicates deep features, Indicates handcrafted characteristics.

[0021] As a further limitation of the first aspect of the present invention, a multimodal fusion and brain tumor survival prediction module is used to integrate the completed multimodal features, construct a survival prediction model and output the results, including: integrating single-modal features of radiology, pathology, genomics and demographics using an average vector fusion strategy, inputting the fused feature vector into a three-layer multilayer perceptron to avoid overfitting and enhance feature interaction capabilities; and optimizing model training with a Cox partial likelihood loss function.

[0022] As a further limitation of the first aspect of the present invention, the loss calculation formula for optimizing the training model using the Cox partial likelihood loss function is as follows:

[0023]

[0024] Where n is the number of samples; The risk score predicted by the model; This is an indicator function; it takes the value 1 when the observation time of sample j is greater than or equal to that of sample i, and 0 otherwise. The event indicator is set to 1 for sample i that observes a death event and 0 for censored events.

[0025] Secondly, the present invention provides a brain tumor survival prediction system based on a multimodal medical knowledge graph, comprising:

[0026] The building module is used to construct a multimodal medical knowledge graph based on the fusion of third-party knowledge bases;

[0027] The extraction module is used to extract features from multimodal brain tumor data. For the radiological modality, tumor and edema regions are extracted from MRI, slices containing the largest lesion region are selected, image features are extracted, and manual features are extracted from tumor blocks using PyRadiomics, which are then concatenated to form a radiological feature vector. For the pathological modality, deep image features are extracted from pathological slices, a cell interaction graph is constructed using a graph convolutional network, and graph features are extracted. The two are then fused as pathological features. For the genomics modality, a feature set including copy number variation (CNV) features and IDH1 gene mutation status is constructed, and genomic features are obtained using a self-normalization network combined with regularization. For the demographic modality, one-hot encoding is performed on different attribute features, and feature embedding is performed to maintain data normalization characteristics. The module searches for corresponding entities related to brain tumor data in a multimodal medical knowledge graph and converts them into feature representations using entity representation learning methods.

[0028] The completion prediction module is used to complete the missing data modal by learning the feature representations related to brain tumor type. Finally, the completed features are input into the pre-trained survival prediction model to achieve brain tumor survival prediction.

[0029] Thirdly, the present invention provides a non-transitory computer-readable storage medium for storing computer instructions, which, when executed by a processor, implement the brain tumor survival prediction method based on a multimodal medical knowledge graph as described in the first aspect.

[0030] Fourthly, the present invention provides a computer device including a memory and a processor, wherein the processor and the memory communicate with each other, the memory stores program instructions that can be executed by the processor, and the processor calls the program instructions to execute the brain tumor survival prediction method based on multimodal medical knowledge graph as described in the first aspect.

[0031] Fifthly, the present invention provides an electronic device, comprising: a processor, a memory, and a computer program; wherein the processor is connected to the memory, the computer program is stored in the memory, and when the electronic device is running, the processor executes the computer program stored in the memory to cause the electronic device to execute instructions for implementing the brain tumor survival prediction method based on multimodal medical knowledge graph as described in the first aspect.

[0032] The beneficial effects of this invention are as follows: For brain tumor survival prediction, a multimodal medical knowledge graph integrating multi-source information from demographics, pathology, genes, and MRI images is constructed. This graph, combined with clinical guidelines, real clinical data, and third-party knowledge bases, forms a knowledge carrier covering multi-dimensional medical information, providing comprehensive and clinically relevant medical knowledge support for subsequent tasks such as missing modality completion and survival prediction for brain tumor patients. Applying this multimodal knowledge graph to missing modality completion in brain tumor survival prediction, by querying related entities through the graph and generating completion features using Word2Vec / ResNet-18, effectively solves the problem of weak modality missing processing capabilities in existing technologies and is more suitable for real-world clinical data scenarios.

[0033] The advantages of additional aspects of the invention will be set forth more clearly in the following description or will be learned by practice of the invention. Attached Figure Description

[0034] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the following description of the embodiments will be briefly introduced. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0035] Figure 1 This is a flowchart of the brain tumor survival prediction method based on multimodal medical knowledge graph as described in an embodiment of the present invention. Detailed Implementation

[0036] Embodiments of the present invention are described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and should not be construed as limiting the present invention.

[0037] It will be understood by those skilled in the art that, unless otherwise defined, all terms used herein (including technical and scientific terms) have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains.

[0038] It should also be understood that terms such as those defined in general dictionaries should be understood to have meanings consistent with their meanings in the context of the prior art, and should not be interpreted in an idealized or overly formal sense unless defined as here.

[0039] Those skilled in the art will understand that, unless specifically stated otherwise, the singular forms “a,” “an,” “the,” and “the” used herein may also include the plural forms. It should be further understood that the term “comprising” as used in this specification means the presence of the stated features, integers, steps, operations, elements, and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, and / or groups thereof.

[0040] In the description of this specification, references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the present invention. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of those different embodiments or examples.

[0041] To facilitate understanding of the present invention, the present invention will be further explained and described below with reference to the accompanying drawings and specific embodiments. However, the specific embodiments do not constitute a limitation on the embodiments of the present invention.

[0042] Those skilled in the art should understand that the accompanying drawings are merely schematic diagrams of embodiments, and the components in the drawings are not necessarily essential for implementing the present invention.

[0043] Current research on brain tumor survival prediction largely employs data-driven fusion strategies, simply splicing or weighting together multimodal features from imaging, clinical data, and genomics. However, it fails to leverage domain knowledge to optimize the fusion logic, resulting in inefficient correlation mining of heterogeneous data. Furthermore, model performance is highly dependent on the quality and scale of the complete multimodal dataset, performing poorly in common clinical scenarios involving missing modalities. While some studies have attempted to complete missing data using generative models, the completion process lacks medical knowledge constraints, making it difficult to guarantee the clinical relevance of the features.

[0044] Therefore, considering the above-mentioned problems, this invention constructs a multimodal medical knowledge graph based on brain tumors. Addressing the issue of missing gene data and radiological imaging data—two key modalities—in clinical scenarios for brain tumor patients, this invention completes the gene modality by using the Word2Vec representation method combined with a deep survival model containing regularization mechanisms to extract features for missing gene data. For missing radiological imaging data, this invention integrates extracted features using a tumor region segmentation model, a pre-trained ResNet18 feature extraction model, and manual feature extraction tools to complete the radiological imaging modality, improving data completeness and usability. Based on the constructed multimodal medical knowledge graph, this invention uses an average vector fusion method to integrate multimodal features for brain tumor survival prediction, solving the problem of insufficient interpretability in traditional models and increasing clinical confidence in the prediction results.

[0045] Example 1

[0046] In this embodiment 1, a brain tumor survival prediction system based on a multimodal medical knowledge graph is first provided, including: a construction module for constructing a multimodal medical knowledge graph based on a third-party knowledge base; an extraction module for extracting features from multimodal brain tumor data; for the radiological modality, tumor and edema regions are extracted from MRI, slices containing the largest lesion region are selected, image features are extracted, and manual features are extracted from tumor blocks using PyRadiomics, and then concatenated to form a radiological feature vector; for the pathological modality, deep image features are extracted from pathological slices, a cell interaction graph is constructed using a graph convolutional network and graph features are extracted, and the two are fused as pathological features; for the genomics modality, a feature graph including copy number variation (CNV) and IDH1 is constructed. The feature set of gene mutation status is obtained by using a self-normalized network combined with regularization to obtain genomic features. For the demographic modality, one-hot encoding is performed on different attribute features for feature embedding to maintain data normalization characteristics. Corresponding entities related to brain tumor data are found in the multimodal medical knowledge graph, and entity representation learning methods are used to convert them into feature representations. The completion prediction module is used to complete the missing data modality with the learned feature representations related to brain tumor types. Finally, the completed features are input into a pre-trained survival prediction model to achieve brain tumor survival prediction.

[0047] In this embodiment, the above-described system is used to implement a brain tumor survival prediction method based on a multimodal medical knowledge graph, including: constructing a multimodal medical knowledge graph based on a third-party knowledge base using a construction module; extracting features from the multimodal brain tumor data using an extraction module; for the radiological modality, extracting tumor and edema regions from MRI, selecting slices containing the largest lesion region, extracting image features, extracting manual features from tumor blocks using PyRadiomics, and splicing them to form a radiological feature vector; for the pathological modality, extracting deep image features from pathological slices, constructing a cell interaction graph through a graph convolutional network and extracting graph features, and fusing the two as pathological features; for the genomics modality, constructing a feature set including copy number variation (CNV) features and IDH1 gene mutation status, and obtaining genomic features using a self-normalization network combined with regularization; for the demographic modality, performing one-hot encoding on different attribute features, embedding features, and maintaining data normalization characteristics; searching for corresponding entities related to brain tumor data in the multimodal medical knowledge graph, and converting them into feature representations using entity representation learning methods. The completion prediction module uses learned features related to brain tumor type to complete missing data modalities. Finally, the completed features are input into a pre-trained survival prediction model to achieve brain tumor survival prediction.

[0048] The completion of missing genomic modalities is supported by structured medical priors from a multimodal medical knowledge graph, and is accomplished through gene entity association, vector representation, and feature extraction. This includes: first, retrieving a list of core genes directly associated with brain tumors from the multimodal medical knowledge graph; these gene associations are derived from entity alignment results from a third-party knowledge base to ensure gene selection conforms to molecular biology principles; loading a pre-trained Word2Vec model to convert each gene name into a low-dimensional dense vector; and using cosine similarity to calculate vector similarity to ensure the vectors reflect semantic relationships between genes; inputting the gene vectors into a deep survival model with three fully connected layers; introducing non-linearity using the ELU activation function; and preventing overfitting using an Alpha Dropout layer. The deep survival model is trained using a Cox partial likelihood loss function to optimize the correlation between features and survival time, and finally outputting the completed genomic features.

[0049] The completion of missing radiological modalities relies on the BraTs2020 dataset associated with a knowledge graph, reusing the core process of single-modal feature embedding and optimizing the details. This includes: firstly, using the nnUNet segmentation model to automatically segment MRI images, extracting regions of interest, and achieving high-precision segmentation through an encoder-decoder architecture; secondly, selecting 2D key slices containing the largest tumor region and adjusting their size to obtain standardized images; thirdly, extracting deep features using a pre-trained ResNet-18 model, and extracting handcrafted features using PyRadiomics to form handcrafted feature vectors; and finally, integrating the deep features and handcrafted features using an average vector fusion strategy to obtain the completed radiological modal features.

[0050] Before fusing deep features and handcrafted features, L2 normalization is performed on the features:

[0051]

[0052] To verify the reliability of the completed features, a consistency index was used for evaluation:

[0053]

[0054] Where M is the number of comparable sample pairs. To complete the risk score corresponding to the feature, I( ) is an indicator function; Indicates deep features, Indicates handcrafted characteristics.

[0055] The multimodal fusion and brain tumor survival prediction module is responsible for integrating the completed multimodal features, constructing a survival prediction model and outputting results. This includes: integrating single-modal features from radiology, pathology, genomics and demographics using an average vector fusion strategy, inputting the fused feature vectors into a three-layer multilayer perceptron to avoid overfitting and enhance feature interaction capabilities; and optimizing model training with the Cox partial likelihood loss function.

[0056] The loss calculation formula for optimizing model training using the Cox partial likelihood loss function is as follows:

[0057]

[0058] Where n is the number of samples; The risk score predicted by the model; This is an indicator function; it takes the value 1 when the observation time of sample j is greater than or equal to that of sample i, and 0 otherwise. The event indicator is set to 1 for sample i that observes a death event and 0 for censored events.

[0059] Example 2

[0060] This embodiment first proposes a brain tumor survival prediction method based on a multimodal medical knowledge graph. This method first constructs a multimodal medical knowledge graph based on a third-party knowledge base fusion, and then uses the multimodal medical knowledge graph to complete missing data modalities. Specifically, it searches for corresponding entities related to the brain tumor in the multimodal knowledge graph and converts them into feature representations using entity representation learning methods. Subsequently, the learned feature representations related to the brain tumor type are added to the multimodal fusion model, and finally, the completed features are used for brain tumor survival prediction, demonstrating excellent performance on existing survival prediction tasks.

[0061] like Figure 1 As shown, the method described in this embodiment relies on a brain cancer survival prediction framework based on a multimodal medical knowledge graph (3MKGBraSurv).

[0062] Specifically, in this embodiment, a multimodal medical knowledge graph construction module is used to construct a multimodal medical knowledge graph based on third-party knowledge base fusion. This module primarily integrates and structures multimodal information related to craniocerebral diseases, providing knowledge support for subsequent missing modality completion and survival prediction. The core process unfolds in four steps:

[0063] The first step was to construct a text knowledge graph based on medical imaging guidelines. Using the "Central Nervous System" chapter from "Medical Imaging (8th Edition)" as the source text, entities and relationships related to intracranial diseases were extracted—a total of 57 disease entities (such as diffuse astrocytoma grade II and glioblastoma grade IV) and 343 symptom entities (such as masses and punctate calcifications) were obtained. 440 "disease-symptom" type relationships were also extracted and stored in a triplet format to form the basic text knowledge graph.

[0064] The second step is to expand the atlas by incorporating real clinical information. Neurological clinical examination reports are collected, focusing on the imaging findings and conclusions, to supplement the atlas with 53 entity relationships closely related to clinical practice (such as rare manifestations of specific diseases), thus expanding the atlas's clinical applicability.

[0065] The third step is to expand the graph dimensions by integrating third-party knowledge bases. Through entity alignment technology, the basic graph is connected to third-party knowledge bases (including DrugBank, PrimeKG Precision Medicine Graph, SympGAN Symptom-Drug Association Database, etc.) with "disease" as the bridge, supplementing entity types such as drugs and genes, and finally forming a structured graph containing 15,033 entities (3,369 diseases, 846 symptoms, 2,998 drugs, and 7,820 genes) and 64,351 relationships.

[0066] The fourth step involves introducing medical imaging modalities to achieve multimodalization. Brain MRI images are embedded into the atlas as attributes of disease entities in the form of heatmaps (intuitively displaying the lesion area and size). A visual web system is built using Flask, Vue, and Neo4j to support the association query between image modalities and text / structured information, providing multi-source knowledge for subsequent modal completion.

[0067] The framework uses a unimodal feature embedding module to extract features from multimodal brain tumor data (radiology, pathology, genomics, demographics). The unimodal feature embedding module designs a dedicated network for different data types, and the specific process is as follows:

[0068] Radiographic modality: The nnUNet segmentation model (including 2DU-Net, 3DU-Net and cascaded 3DU-Net) was used to automatically extract tumor and edema regions from MRI. A 120×120 four-channel slice containing the largest lesion region was selected. Image features were extracted by pre-trained ResNet-18 and 318 handcrafted 2D / 3D features (shape, position, texture) were extracted by PyRadiomics and then stitched together to form a radiographic feature vector.

[0069] Pathological modality: For 512×512 ROI pathological sections (20x magnification), deep image features were extracted using a pre-trained VGG19. At the same time, a cell interaction graph was constructed and graph features were extracted using a graph convolutional network (GCN). The two were fused as pathological features.

[0070] Genomics Modalities: A feature set containing 79 CNV features and 1 IDH1 mutation status was constructed. A self-normalized network (SNN, activation function SeLU) + Alpha Dropout regularization (containing 4 fully connected layers + ELU activation) was used to avoid overfitting in small samples and output genomic features.

[0071] Demographic modality: Four features, including age and gender, are one-hot encoded and embedded using an SNN to maintain data normalization.

[0072] The framework utilizes a missing modality completion module, which is a core component in ensuring the quality of multimodal survival prediction data for brain tumors. Its design logic closely addresses the real-world pain points of clinical data acquisition—genomic data is generally scarce due to its reliance on specialized testing equipment, high cost, and long processing time; radiological data is prone to missing data due to equipment limitations and patient cooperation. Both are key modalities for brain tumor survival prediction and require reasonable medical completion based on a multimodal medical knowledge graph. The specific process is as follows:

[0073] To fill in the gaps in genomic modalities, a structured medical prior based on knowledge graphs is required, which can be accomplished through a three-step process: "gene entity association - vector representation - feature extraction." First, a list of core genes directly associated with brain tumors (such as gliomas) is retrieved from the multimodal medical knowledge graph. The associations of these genes are derived from entity alignment results from third-party knowledge bases (such as DrugBank and PrimeKG), ensuring that gene selection conforms to molecular biology principles. Next, a pre-trained Word2Vec model is loaded to associate each gene name... Convert to a low-dimensional dense vector (d is the vector dimension). To ensure that the vectors reflect the semantic relationships between genes, cosine similarity is used to calculate vector similarity. The formula is:

[0074]

[0075] in The dot product of gene vectors, Using the L2 norm of the vector, this method ensures that functionally similar genes are closer together in the vector space. Finally, the gene vectors are input into a deep survival model with three fully connected layers. The model uses the ELU activation function to introduce non-linearity to avoid the "dead neuron" problem. The formula is:

[0076]

[0077] Meanwhile, overfitting is prevented by using an Alpha Dropout layer, and the model training optimizes the correlation between features and survival time using the Cox partial likelihood loss function. Finally, the completed genomic features are output. .

[0078] To address the missing radiological modalities, the BraTs2020 dataset (containing 1698 annotated MRI samples of brain tumors) associated with a knowledge graph is utilized. The core workflow of single-modal feature embedding is reused and its details optimized. First, the nnUNet segmentation model is used to automatically segment the MRI images, extracting regions of interest (ROIs) such as the tumor core (ET) and edema area (ED). This model achieves high-precision segmentation through an encoder-decoder architecture. Then, 2D key slices containing the largest tumor region are selected and resized to 120 pixels. 120 obtains standardized images :

[0079]

[0080] Where S is the set of slices, The volume of the tumor core region in slice s is given. Then, a dual-path extraction strategy of "deep features + handcrafted features" is employed: deep features are extracted using a pre-trained ResNet-18 model (with fine-tuning of the last 3 layers). Using the PyRadiomics tool, 318 handcrafted features (including shape, texture, etc.) were extracted and combined to form a handcrafted feature vector. Finally, an average vector fusion strategy is used to integrate the two types of features. Before fusion, the features are first L2 normalized.

[0081]

[0082] To verify the reliability of the completed features, the consistency index (C-index) is used for evaluation:

[0083]

[0084] Where M is the number of comparable sample pairs. To complete the risk score corresponding to the feature, I( ) is the indicator function. Experiments show that the C-index of the 3MKGBraSurv model after genomic feature completion and the C-index after radiological feature completion are both significantly higher than those of traditional randomized completion methods (such as the MMD model), which fully demonstrates the effectiveness and medical rationality of the completion scheme.

[0085] The framework uses a multimodal fusion and brain tumor survival prediction module to integrate the completed multimodal features, construct a survival prediction model, and output the results. The specific process is as follows:

[0086] First, an "average vector fusion" strategy is adopted to integrate single-modal features from radiology, pathology, genomics (including completion), and demographics. The fused feature vector is then input into a three-layer multilayer perceptron (MLP) containing ReLU activation function and Dropout layer to avoid overfitting and enhance feature interaction capabilities.

[0087] Secondly, the model training is optimized using the Cox partial likelihood loss function, and the loss calculation formula is as follows:

[0088]

[0089] Where n is the number of samples. The risk score predicted by the model. This is an indicator function (1 when the observation time of sample j is ≥ that of sample i, otherwise 0). This is the event indicator (1 for sample i that observes a death event, 0 for censored events).

[0090] In this embodiment, the benchmark dataset used is a large-scale glioma dataset combining TCGA (Cancer Genome Atlas), TCIA (Cancer Imaging Archive), and BraTs (Brain Tumor Segmentation Challenge) as a standard dataset to validate the effectiveness of the brain tumor survival prediction framework (3MKGBraSurv) based on a multimodal medical knowledge graph. This dataset includes 1698 samples from 962 brain tumor patients. Each patient contains at least one usable modality, and all patients are accompanied by survival time and censoring status labels, supporting survival prediction research in multimodal missing scenarios. The specific modal distribution and patient number information of the dataset are shown in Table 1.

[0091] Table 1 Dataset Details

[0092]

[0093] Improving the accuracy and interpretability of survival predictions in modal loss scenarios is the core objective of this technology. To comprehensively evaluate the performance of the 3MKGBraSurv framework, the experiment used the consistency index (C-index) as the core evaluation indicator (measuring the consistency between the model's predicted risk score and the patient's actual survival time, with a value ranging from 0 to 1, the closer to 1, the better the predictive performance). The statistical significance of the prediction results was verified using Kaplan-Meier survival curves and p-values. Simultaneously, 3MKGBraSurv was compared with various baseline methods in terms of "modal loss completion effect" and "multimodal fusion performance." Specific results and analyses are as follows.

[0094] Comparison of survival prediction performance in modality-deficient scenarios:

[0095] The loss of genomic and radiological modalities in clinical data poses a major challenge to survival prediction in brain tumors. The experiment simulated two typical scenarios: "loss of genomic modality" and "loss of radiological modality". The 3MKGBraSurv model was compared with the following two baseline methods: Omic_missing / Rado_missing: a basic model that directly discards the missing modalities and uses only the remaining three modalities for prediction; MMD: a traditional method that completes the missing information by random reconstruction (generating missing modal features based on data statistical attributes) with the addition of random reconstruction loss constraints.

[0096] Table 2. Comparison of consistency indices among brain tumor survival prediction models with different modalities missing.

[0097]

[0098] As shown in Table 2, 3MKGBraSurv achieved a C-index of 0.7803±0.031 in the genomic missing scenario, a 2.2% improvement over the basic model Omic_missing and a 0.86% improvement over the random reconstruction method MMD. This advantage stems from the fact that 3MKGBraSurv does not rely on statistical assumptions to generate missing features. Instead, it queries glioma-related gene entities (such as STAT3, ST7-AS1, and other key genes regulating brain tumor progression) through a multimodal medical knowledge graph, converts gene names into semantic vectors using Word2Vec, and then generates biologically meaningful genomic features through a deep survival model. The completed features have a stronger correlation with the actual gene data, thereby improving prediction accuracy.

[0099] Overall performance comparison with multimodal baseline methods:

[0100] To verify the overall competitiveness of 3MKGBraSurv in the field of brain tumor survival prediction, the experiment compared it with five mainstream multimodal survival prediction methods: Multimodal Prognosis (a classic method based on multimodal neural networks fusing clinical and omics data), CAMR (a method that improves prediction accuracy through cross-aligned multimodal representations), MultiCoFusion (a multimodal fusion framework based on multi-task related learning), Deep Orthogonal Fusion (a fusion method that reduces modal redundancy through orthogonal loss), and Pathomic Fusion (an interpretable framework that fuses histological and genomic features). The overall performance comparison results are shown in Table 3.

[0101] Table 3 Comparison of Survival Prediction Performance of Different Multimodal Methods

[0102]

[0103] As shown in Table 3, the C-index of 3MKGBraSurv is 0.7881±0.027, significantly higher than all baseline methods, and 2.39% higher than Pathomic Fusion (C-index 0.7697±0.047), which ranks second in performance. Its key advantages are mainly reflected in two aspects: First, other methods often assume that the data is "modal complete," and their performance will drop significantly in scenarios where modalities are missing in actual clinical practice. However, 3MKGBraSurv can complete the missing modal data through a multimodal medical knowledge graph, making it more suitable for real clinical data environments. Second, the interpretability of most baseline methods relies only on feature visualization and lacks clear logical traceability. However, 3MKGBraSurv can trace the prediction basis through the knowledge graph (e.g., "a patient is predicted to be high-risk" can correspond to the entity relationship of "IDH1 mutation + MRI edema area expansion"), making the prediction results more acceptable to clinical practice.

[0104] Example 3

[0105] This embodiment 3 provides a non-transitory computer-readable storage medium for storing computer instructions. When executed by a processor, the computer instructions implement the brain tumor survival prediction method based on a multimodal medical knowledge graph as described above. The method includes:

[0106] Construct a multimodal medical knowledge graph based on the fusion of third-party knowledge bases;

[0107] Feature extraction was performed on multimodal brain tumor data. For the radiological modality, tumor and edema regions were extracted from MRI, and slices containing the largest lesion region were selected. Image features were extracted from tumor blocks using PyRadiomics, and hand-crafted features were extracted and concatenated to form a radiological feature vector. For the pathological modality, deep image features were extracted from pathological slices, and a cell interaction graph was constructed using a graph convolutional network to extract graph features. The two were then fused as pathological features. For the genomics modality, a feature set including copy number variation (CNV) features and IDH1 gene mutation status was constructed, and genomic features were obtained using a self-normalization network combined with regularization. For the demographic modality, one-hot encoding was performed on different attribute features, and feature embedding was performed to maintain data normalization characteristics.

[0108] In a multimodal medical knowledge graph, the corresponding entities for brain tumor-related data are found, and entity representation learning methods are used to convert them into feature representations.

[0109] The learned features related to brain tumor types are used to complete the missing data modalities, and finally the completed features are input into a pre-trained survival prediction model to achieve brain tumor survival prediction.

[0110] Example 4

[0111] This embodiment 4 provides a computer device, including a memory and a processor, wherein the processor and the memory communicate with each other, and the memory stores program instructions that can be executed by the processor. The processor calls the program instructions to execute the brain tumor survival prediction method based on a multimodal medical knowledge graph as described above, the method including:

[0112] Construct a multimodal medical knowledge graph based on the fusion of third-party knowledge bases;

[0113] Feature extraction was performed on multimodal brain tumor data. For the radiological modality, tumor and edema regions were extracted from MRI, and slices containing the largest lesion area were selected. Image features were extracted, and manual features were extracted from tumor blocks using PyRadiomics, which were then concatenated to form a radiological feature vector. For the pathological modality, deep image features were extracted from pathological slices, and a cell interaction graph was constructed using a graph convolutional network to extract graph features. The two were then fused as pathological features. For the genomics modality, a feature set including copy number variation (CNV) features and IDH1 gene mutation status was constructed, and genomic features were obtained using a self-normalization network combined with regularization. For the demographic modality, one-hot encoding was performed on different attribute features, and feature embedding was carried out to maintain data normalization characteristics.

[0114] In a multimodal medical knowledge graph, the corresponding entities for brain tumor-related data are found, and entity representation learning methods are used to convert them into feature representations.

[0115] The learned features related to brain tumor types are used to complete the missing data modalities, and finally the completed features are input into a pre-trained survival prediction model to achieve brain tumor survival prediction.

[0116] Example 5

[0117] This embodiment 5 provides an electronic device, including: a processor, a memory, and a computer program; wherein, the processor is connected to the memory, and the computer program is stored in the memory. When the electronic device is running, the processor executes the computer program stored in the memory, so that the electronic device executes instructions to implement the brain tumor survival prediction method based on multimodal medical knowledge graph as described above, the method including:

[0118] Construct a multimodal medical knowledge graph based on the fusion of third-party knowledge bases;

[0119] Feature extraction was performed on multimodal brain tumor data. For the radiological modality, tumor and edema regions were extracted from MRI, and slices containing the largest lesion area were selected. Image features were extracted, and manual features were extracted from tumor blocks using PyRadiomics, which were then concatenated to form a radiological feature vector. For the pathological modality, deep image features were extracted from pathological slices, and a cell interaction graph was constructed using a graph convolutional network to extract graph features. The two were then fused as pathological features. For the genomics modality, a feature set including copy number variation (CNV) features and IDH1 gene mutation status was constructed, and genomic features were obtained using a self-normalization network combined with regularization. For the demographic modality, one-hot encoding was performed on different attribute features, and feature embedding was carried out to maintain data normalization characteristics.

[0120] In a multimodal medical knowledge graph, the corresponding entities for brain tumor-related data are found, and entity representation learning methods are used to convert them into feature representations.

[0121] The learned features related to brain tumor types are used to complete the missing data modalities, and finally the completed features are input into a pre-trained survival prediction model to achieve brain tumor survival prediction.

[0122] In summary, the brain tumor survival prediction method based on a multimodal medical knowledge graph proposed in this invention realizes a method for constructing a multimodal medical knowledge graph based on the fusion of third-party knowledge bases. It integrates medical imaging guideline texts, real clinical data, third-party libraries such as DrugBank / PrimeKG, and MRI image heatmaps to form a structured atlas covering multiple entities including diseases, symptoms, drugs, and genes, providing knowledge support for modality completion. A missing modality completion strategy based on the multimodal medical knowledge graph is proposed. For scenarios with missing genomics / radiology modalities, it queries related entities (such as brain tumor-related genes and radiological features of similar cases) through the graph, and generates effective completion features by combining Word2Vec / ResNet-18, avoiding the blindness of traditional random reconstruction methods. A brain tumor survival prediction framework that integrates completed features is proposed. It uses average vector fusion of completed multimodal features, combines Cox partial likelihood loss to optimize a three-layer MLP model, and traces the prediction basis through knowledge graph-related entities, balancing prediction accuracy and interpretability.

[0123] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0124] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0125] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0126] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment, whereby a series of operational steps are performed to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0127] While the specific embodiments of the present invention have been described above in conjunction with the accompanying drawings, this is not intended to limit the scope of protection of the present invention. Those skilled in the art should understand that, based on the technical solutions disclosed in the present invention, various modifications or variations that can be made by those skilled in the art without creative effort should be included within the scope of protection of the present invention.

Claims

1. A brain tumor survival prediction method based on multimodal medical knowledge graph, characterized in that, include: Construct a multimodal medical knowledge graph based on the fusion of third-party knowledge bases; Feature extraction from multimodal brain tumor data; For the radiology modality, tumor and edema regions were extracted from MRI, and slices containing the largest lesion area were selected. Image features were extracted, and PyRadiomics was used to extract handcrafted features from tumor blocks, which were then concatenated to form a radiology feature vector. For the pathology modality, deep image features were extracted from pathology slices, and a cell interaction graph was constructed using a graph convolutional network to extract graph features. The two were then fused as pathology features. For the genomics modality, a feature set containing copy number variation (CNV) features and IDH1 gene mutation status was constructed, and a self-normalization network combined with regularization was used to obtain genomic features. For the demographic modality, one-hot encoding was performed on different attribute features, and feature embedding was performed to maintain data normalization characteristics. In a multimodal medical knowledge graph, we search for corresponding entities related to brain tumor data and convert them into feature representations using entity representation learning methods. The learned features related to brain tumor types are used to complete the missing data modalities, and finally the completed features are input into a pre-trained survival prediction model to achieve brain tumor survival prediction.

2. The brain tumor survival prediction method based on multimodal medical knowledge graph according to claim 1, characterized in that, To address the missing genomic modalities, a structured medical prior based on a multimodal medical knowledge graph is employed, involving gene entity association, vector representation, and feature extraction. The process includes: first, retrieving a list of core genes directly associated with brain tumors from the multimodal medical knowledge graph. These gene associations are derived from entity alignment results from a third-party knowledge base, ensuring gene selection aligns with molecular biology principles; loading a pre-trained Word2Vec model to convert each gene name into a low-dimensional dense vector. Cosine similarity is used to calculate vector similarity to ensure the vectors reflect semantic relationships between genes; inputting the gene vectors into a deep survival model with three fully connected layers, introducing non-linearity using the ELU activation function, and preventing overfitting with an AlphaDropout layer. The deep survival model is trained using a Cox partial likelihood loss function to optimize the correlation between features and survival time. Finally, the completed genomic features are output.

3. The brain tumor survival prediction method based on multimodal medical knowledge graph according to claim 1, characterized in that, The completion of missing radiological modalities relies on the BraTs2020 dataset associated with a knowledge graph, reusing the core process of single-modal feature embedding and optimizing the details. This includes: firstly, using the nnUNet segmentation model to automatically segment MRI images, extracting regions of interest, and achieving high-precision segmentation through an encoder-decoder architecture; secondly, selecting 2D key slices containing the largest tumor region and adjusting their size to obtain standardized images; thirdly, extracting deep features using a pre-trained ResNet-18 model, and extracting handcrafted features using PyRadiomics to form handcrafted feature vectors; and finally, integrating the deep features and handcrafted features using an average vector fusion strategy to obtain the completed radiological modal features.

4. The brain tumor survival prediction method based on multimodal medical knowledge graph according to claim 3, characterized in that, Before fusing deep features and handcrafted features, L2 normalization is performed on the features: ； To verify the reliability of the completed features, a consistency index was used for evaluation: ； Where M is the number of comparable sample pairs. To complete the risk score corresponding to the feature, I( ) is an indicator function; Indicates deep features, Indicates handcrafted characteristics; This is an event indicator.

5. The brain tumor survival prediction method based on multimodal medical knowledge graph according to claim 1, characterized in that, The multimodal fusion and brain tumor survival prediction module is responsible for integrating the completed multimodal features, constructing a survival prediction model and outputting results. This includes: integrating single-modal features from radiology, pathology, genomics and demographics using an average vector fusion strategy, inputting the fused feature vectors into a three-layer multilayer perceptron to avoid overfitting and enhance feature interaction capabilities; and optimizing model training with the Cox partial likelihood loss function.

6. The brain tumor survival prediction method based on multimodal medical knowledge graph according to claim 5, characterized in that, The loss calculation formula for optimizing model training using the Cox partial likelihood loss function is as follows: ； Where n is the number of samples; The risk score predicted by the model; This is an indicator function; it takes the value 1 when the observation time of sample j is greater than or equal to that of sample i, and 0 otherwise. The event indicator is set to 1 for sample i that observes a death event and 0 for censored events.

7. A brain tumor survival prediction system based on a multimodal medical knowledge graph, characterized in that, include: The building module is used to construct a multimodal medical knowledge graph based on the fusion of third-party knowledge bases; The feature extraction module is used to extract features from multimodal brain tumor data. For the radiology modality, tumor and edema regions were extracted from MRI, and slices containing the largest lesion area were selected. Image features were extracted, and PyRadiomics was used to extract handcrafted features from tumor blocks, which were then concatenated to form a radiology feature vector. For the pathology modality, deep image features were extracted from pathology slices, and a cell interaction graph was constructed using a graph convolutional network to extract graph features. The two were then fused as pathology features. For the genomics modality, a feature set containing copy number variation (CNV) features and IDH1 gene mutation status was constructed, and a self-normalized network combined with regularization was used to obtain genomic features. For the demographic modality, one-hot encoding was performed on different attribute features, and feature embedding was performed to maintain data normalization characteristics. Corresponding entities related to brain tumor data were found in the multimodal medical knowledge graph, and entity representation learning methods were used to convert them into feature representations. The completion prediction module is used to complete the missing data modal by learning the feature representations related to brain tumor type. Finally, the completed features are input into the pre-trained survival prediction model to achieve brain tumor survival prediction.

8. A non-transitory computer-readable storage medium, characterized in that, The non-transitory computer-readable storage medium is used to store computer instructions, which, when executed by a processor, implement the brain tumor survival prediction method based on a multimodal medical knowledge graph as described in any one of claims 1-6.

9. A computer device, characterized in that, The method includes a memory and a processor, the processor and the memory communicating with each other, the memory storing program instructions executable by the processor, and the processor calling the program instructions to execute the brain tumor survival prediction method based on a multimodal medical knowledge graph as described in any one of claims 1-6.

10. An electronic device, characterized in that, include: The device includes a processor, a memory, and a computer program; wherein the processor is connected to the memory, the computer program is stored in the memory, and when the electronic device is running, the processor executes the computer program stored in the memory to cause the electronic device to execute instructions to implement the brain tumor survival prediction method based on a multimodal medical knowledge graph as described in any one of claims 1-6.

Images