Multi-modal data fusion-based interpretable cancer survival prediction method

Abstract

The invention discloses an interpretable cancer survival prediction method based on multi-modal data fusion, and the method comprises the following steps: carrying out the preprocessing of a full-section pathological image, and obtaining an image feature matrix; preprocessing a pathological report text corresponding to the pathological image, and constructing a text feature matrix; preprocessing the high-dimensional gene expression data of the patient to generate a plurality of survival-related gene modules and feature vectors thereof; performing dynamic fusion on the obtained multi-modal features through a self-adaptive multi-modal expert hybrid module to obtain final fusion feature representation; and training a deep learning model by using fusion feature representation in combination with a negative log-likelihood loss function and a Cox proportional risk model, and performing prognosis analysis on the cancer patient. According to the explainable cancer survival prediction method based on multi-modal data fusion, the prediction accuracy is improved, good model interpretability is achieved, and biomarkers closely related to cancer prognosis can be automatically recognized.

Description

Technical Field

[0001] The present invention relates to the technical field of bioinformatics, and particularly to an interpretable cancer survival prediction method based on multimodal data fusion. Background Art

[0002] Cancer is one of the major global health challenges, and it is crucial to conduct cancer prognosis analysis for patients. Existing cancer survival prediction methods are mostly based on single-modal data, such as pathological images, genomic data, or pathological reports. However, a single data type often fails to comprehensively reflect the patient's condition. In recent years, the integrated analysis of multimodal data has gradually attracted attention, but existing methods still have deficiencies: most multimodal survival prediction models only combine pathological images and genomic data, ignoring the unstructured text data in pathological reports, which may contain tumor-specific descriptions highly relevant to patient survival; the high dimensionality of genomic data may lead to overfitting problems, and existing dimensionality reduction methods may affect the accuracy of module generation due to data noise; in multimodal fusion models, sometimes the fusion performance is even worse than that of a single modality, which may be caused by factors such as differences between multimodal data.

[0003] Therefore, in view of the limitations of the above-mentioned existing technologies, the present invention proposes an interpretable cancer survival prediction method based on multimodal data fusion. Summary of the Invention

[0004] The purpose of the present invention is to provide an interpretable cancer survival prediction method based on multimodal data fusion, introducing a multimodal fusion strategy of whole-slide pathological images (WSIs), pathological report texts, and gene expression data, and combining an end-to-end gene module identification (GMI) algorithm with an adaptive multimodal expert mixture (AMMEM) module, which can effectively solve the problems of limitations of single-modal data, improper processing of high-dimensional gene data, and imperfect multimodal fusion strategies in the existing technologies, and achieve the prognosis analysis of cancer patients.

[0005] To achieve the above purpose, the present invention provides an interpretable cancer survival prediction method based on multimodal data fusion, including the following steps: Step S1, preprocess the whole-slide pathological image WSI to obtain an image feature matrix; Step S2, preprocess the pathological report text corresponding to the pathological image to construct a text feature matrix; Step S3, preprocess the high-dimensional gene expression data of the patient to generate multiple survival-related gene modules and their feature vectors; Step S4, through the adaptive multimodal expert mixture AMMEM module, dynamically fuse the multimodal features obtained in the above steps to obtain the final fused feature representation; Step S5: Use the fused feature representation, combine the negative log-likelihood loss function with the Cox proportional hazard model to train a deep learning model, and perform prognostic analysis on cancer patients.

[0006] Preferably, in step S1, the whole slice pathology image WSI is preprocessed to obtain an image feature matrix, and the specific process is as follows: Step S11, perform tissue region detection and segmentation on the full-slice pathological image; The tissue area is divided into several non-overlapping image patches, as shown below: ; in, Indicates A single image patch for each patient; is an index variable, indicating the position of the current image block in the set; For patients The total number of patches; Step S12, input each image block into the image encoder of the pathological vision-language basic model PLIP to perform feature extraction; The PLIP model extracts fine-grained features from pathological images and then passes them through a linear layer to obtain the feature vector representation of each image patch. , as shown below: ; in, For the The feature representation of an image patch has a dimension of 512; Step S13: concatenate the feature vectors of all image patches into a matrix to obtain the patient The image feature matrix , as shown below: ; in, For patients The image feature matrix.

[0007] Preferably, in step S2, the pathology report text corresponding to the pathology image is preprocessed to construct a text feature matrix, and the specific process is as follows: Step S21, segment the pathology report corresponding to the pathology image into Text unit , as shown below: ; Step S22: Each text unit After passing through the PLIP text encoder, the feature representation of the text is extracted , as follows: ; Among them, is the feature representation of the th text unit, with a dimension of 512; Step S23: Concatenate the feature vectors of all text units into a text feature matrix , as follows: ; Among them, is the text feature matrix of patient .

[0008] Preferably, in step S3, the high-dimensional gene expression data of the patient is preprocessed to generate multiple survival-related gene modules and their feature vectors. The specific process is as follows: Step S31: Screen the genes with high variance in the gene expression data, and select the top genes with the largest variance in expression values. Then the gene expression data of patient is as follows: ; Among them, is the gene expression data of patient , which contains genes; Step S32: Combine the gene expression data and the prior gene embedding information, and use the trainable projection matrices and to transform the gene expression data to generate the final gene feature representation , as follows: ; Among them, is the prior gene embedding matrix from Gene2Vec; is the fused gene feature representation; represents the transformation matrix; Step S33: Measure the functional similarity between gene pairs through the Pearson correlation coefficient to generate a weight matrix , as follows: ; Among them, represents the similarity between the features of gene pair and , and the diagonal elements are set to 0, indicating that the similarity of a gene to itself is not calculated; Step S34: Identify gene modules using the local maximum algorithm; By traversing all local maximum edges, select the strongest connection based on edge weights, and gradually add genes related to the increase in module density into the module; among them, the density of the module The calculation formula is as follows: ; Among them, is the number of genes in the module ; is the weight of the edges within the module; Through this process, multiple gene modules are generated, and similar modules are merged according to the overlap rate threshold ; Finally, the gene modules of the patient are obtained, as follows: ; ; Step S35: Extract characteristic genes for each gene module , use SVD decomposition to extract the principal components of each module, and obtain the gene embedding representation of the patient, as follows: ; ; Among them, is the characteristic gene of the th gene module, coming from the first principal component of the gene expression matrix of module .

[0009] Preferably, in step S3, a loss function is introduced to optimize the quality of gene modules, as follows: ; Among them, represents the loss function; represents the number of all observation objects; is the clustering coefficient of gene module ; The first term represents the compactness of the module; the second term measures the difference between the embeddings of different gene modules.

[0010] Preferably, in step S3, through the Adaptive Multi-modal Mixture of Experts (AMMEM) module, the multi-modal features obtained in the above steps are dynamically fused to obtain the final fused feature representation. The specific process is as follows: Step S41: Gating network; The gating network takes the image feature matrix , the text feature matrix , and the characteristic genes of the gene module The multi-modal embedding is used as the input, as follows: ; Among them, , is a learnable matrix for image, gene, and pathology report embeddings; represents the Gaussian error linear unit, represents the root mean square normalization layer; The output of the gating network , as follows: ; Among them, is the fusion weight, which determines whether the final prediction result is based on the complete multi-modal information or partial or single-modal embeddings; Step S42, the expert network; The module contains four expert networks, and each expert network represents a different combination method of image, gene, and pathology report data; among them, the specific calculation method of the expert network is as follows: ; Among them, and respectively represent the cross-attention and self-attention mechanisms, represents a filtering operation, which is used to screen out the corresponding pathology information from the self-attention results; After selecting the expert network, according to the selected expert network, the multi-modal fusion result of the patient is obtained.

[0011] Preferably, in step S5, the fusion feature representation is used to train the deep learning model by combining the negative log-likelihood loss function and the Cox proportional hazards model for prognostic analysis of cancer patients. The specific process is as follows: Step S51, in the survival prediction process, use the proportional hazards model to estimate the risk and survival time of the patient; (1) The risk function is used to describe the instantaneous death risk of the patient at time , given its survival status , then the definition of the risk function is as follows: ; Among them, is the probability that the patient survives to after time t; represents the historical information or covariates; (2) The survival function represents the cumulative probability of a patient's survival after time as follows: ; where represents the time point, from to ; Step S52, Parameterize the proportional hazards model; In the model, the hazard function of the survival data is parameterized as: ; where is the baseline hazard function; is the regression coefficient; is the feature vector of the patient, including the fused features of the image, text, and gene modules; Step S53, Optimize the model parameters by minimizing the negative log-likelihood loss function as follows: ; where is the censoring indicator variable, indicating whether an event has occurred; During the optimization process, the goal is to maximize the likelihood of the survival data under the model, so as to achieve accurate prediction of the survival time.

[0012] Therefore, the present invention adopts the above-mentioned interpretable cancer survival prediction method based on multimodal data fusion, introduces a multimodal fusion strategy of whole-slide pathology images (WSIs), pathology report texts, and gene expression data, and combines the end-to-end gene module identification (GMI) algorithm with the adaptive multimodal expert mixture (AMMEM) module, which can effectively solve the problems of the limitations of single-modal data, improper processing of high-dimensional gene data, and imperfect multimodal fusion strategies in the prior art, and realize the prognostic analysis of cancer patients; not only improves the prediction accuracy, but also has good model interpretability, and can automatically identify the biomarkers closely related to cancer prognosis.

[0013] The technical solution of the present invention will be further described in detail below with reference to the drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] Figure 1 is a schematic flowchart of an interpretable cancer survival prediction method based on multimodal data fusion according to the present invention; Figure 2 is a detailed model of an interpretable cancer survival prediction method based on multimodal data fusion according to the present invention; Figure 3Visual components of the AMMEM of the present invention; among them, (a) is the gating network; (b) is the expert network. Detailed implementation manners

[0015] The technical solution of the present invention will be further described below with reference to the accompanying drawings and embodiments.

[0016] As Figure 1 and Figure 2 shown, a method for interpretable cancer survival prediction based on multi-modal data fusion of the present invention includes the following steps: Step S1: Preprocess the whole-slide pathology image (WSI) to obtain an image feature matrix; Step S2: Preprocess the pathology report text corresponding to the pathology image to construct a text feature matrix; Step S3: Preprocess the high-dimensional gene expression data of the patient to generate multiple survival-related gene modules and their feature vectors; Step S4: Through the adaptive multi-modal expert mixture (AMMEM) module, dynamically fuse the multi-modal features obtained in the above steps to obtain the final fused feature representation; Step S5: Use the fused feature representation, combine the negative log-likelihood loss function with the Cox proportional hazards model to train the deep learning model for prognostic analysis of cancer patients.

[0017] Embodiment Step S1: Preprocess the whole-slide pathology image (WSI) to obtain an image feature matrix.

[0018] Step S11: Detect and segment the tissue region of the whole-slide pathology image. Divide the tissue region of each patient into several non-overlapping image patches (patches), as follows: ; Among them, represents the single image patch of the th patient; is the index variable, representing the position of the current image patch in the set ( = 1, 2,..., ); is the total number of patches of patient .

[0019] Step S12: Input each image patch into the image encoder of the pathology vision-language foundation model (PLIP) for feature extraction.

[0020] The PLIP model extracts fine-grained features from pathological images and then obtains the feature vector representation of each image patch through a linear layer , as follows: ; Among them, is the feature representation of the th image patch, with a dimension of 512.

[0021] Step S13: Concatenate the feature vectors of all image patches into a matrix to obtain the image feature matrix of the patient , as follows: , as follows: ; Among them, is the image feature matrix of the patient .

[0022] Step S2: Preprocess the pathological report text corresponding to the pathological image and construct a text feature matrix.

[0023] Step S21: Perform sentence splitting on the pathological report corresponding to the pathological image, and divide the pathological report into text units , as follows: ; Step S22: After each text unit passes through the PLIP text encoder, extract the feature representation of the text , as follows: ; Among them, is the feature representation of the th text unit, with a dimension of 512.

[0024] Step S23: Concatenate the feature vectors of all text units into a text feature matrix , as follows: ; Among them, is the text feature matrix of the patient .

[0025] Step S3: Preprocess the high-dimensional gene expression data of the patient to generate multiple survival-related gene modules and their feature vectors.

[0026] Step S31: Perform high-variance gene screening on the gene expression data, and screen out the top genes to reduce redundant information and ensure the validity of the data.

[0027] patient 's gene expression data is as follows: ; Among them, is the gene expression data of patient and contains genes.

[0028] Step S32: Combine the gene expression data and the prior gene embedding information, and use the trainable projection matrices and to transform the gene expression data and generate the final gene feature representation , as follows: ; Among them, is the prior gene embedding matrix from Gene2Vec; is the fused gene feature representation; represents the transformation matrix.

[0029] Step S33: Measure the functional similarity between gene pairs through the Pearson correlation coefficient to generate the weight matrix , as follows: ; Among them, represents the similarity between the features of gene pair and , and the diagonal elements are set to 0, indicating that the similarity of a gene to itself is not calculated.

[0030] Step S34: Adopt the local maximum algorithm to identify gene modules.

[0031] By traversing all local maximum edges, select the strongest connections based on edge weights, and gradually add genes related to the increase in module density into the module. Among them, the density of the module is calculated as follows: ; Among them, is the number of genes in module ; is the weight of the edges within the module.

[0032] Through this process, multiple gene modules are generated, and similar modules are merged according to the overlap rate threshold . Finally, patient 's ; Step S35: For each gene module , extract the characteristic genes, use SVD decomposition to extract the principal components of each module, and obtain the gene embedding representation of the patient, as follows: ; ; Among them, is the characteristic gene of the th gene module, which comes from the first principal component of the gene expression matrix of module .

[0033] Step S36: In order to further improve the compactness of the identified gene modules and the difference between different gene embeddings, a loss function is introduced to optimize the quality of the gene modules. The loss function is as follows: ; Among them, represents the loss function; represents the number of all patients or subjects; is the clustering coefficient of gene module ; The first term represents the compactness of the module; the second term measures the difference between different gene module embeddings. By minimizing this loss function, more compact and well-distinguishable gene modules and their embeddings can be obtained.

[0034] Step S4: As shown in the Adaptive Multimodal Mixture of Experts (AMMEM) module Figure 3 , dynamically fuse the multimodal features obtained in the above steps to obtain the final fused feature representation.

[0035] Step S41: Gating network.

[0036] The gating network takes the multimodal embeddings of the image feature matrix , text feature matrix , and characteristic genes of gene modules as inputs, as follows: ; Among them, , is the learnable matrix of image, gene, and pathology report embeddings; represents the Gaussian Error Linear Unit, represents the Root Mean Square Normalization layer.

[0037] The output of the gating network is as follows: ; Among them, is the fusion weight, which determines whether the final prediction result is based on complete multimodal information or partial (single) modal embeddings.

[0038] Step S42, Expert network.

[0039] To better integrate multimodal features, it contains four expert networks, and each expert network represents a different combination method of image, gene, and pathological report data. Among them, the specific calculation method of the expert network is as follows: ; Among them, and respectively represent the cross-attention and self-attention mechanisms, represents a filtering operation, which is used to screen out the corresponding pathological information from the self-attention results.

[0040] After selecting the appropriate expert network, the multimodal fusion result of the patient can be obtained according to the selected expert network .

[0041] Step S5, Using the fused feature representation, combined with the negative log-likelihood loss function and the Cox proportional hazards model to train a deep learning model for prognostic analysis of cancer patients.

[0042] Step S51, In the survival prediction process, use the proportional hazards model to estimate the risk and survival time of the patient.

[0043] (1) Hazard function is used to describe the instantaneous death risk of the patient at time , given its survival status , then the definition of the hazard function is as follows: ; Among them, is the probability that the patient survives to after time t; represents historical information or covariates, such as the patient's age, gender, disease status, etc.

[0044] (2) Survival function represents the cumulative probability that the patient survives after time , as follows: ; Among them, represents the time point, starting from to 。

[0045] Step S52, Parametrize the proportional hazards model.

[0046] In the model, the hazard function of the survival data is parametrized as: ; where, is the baseline hazard function; are the regression coefficients; is the feature vector of the patient, which contains the fused features of the image, text, and gene modules.

[0047] Optimize the model parameters by minimizing the negative log-likelihood loss function as follows: ; where, is the censoring indicator variable, indicating whether an event (such as death) has occurred.

[0048] In the optimization process, aim to maximize the likelihood of the survival data under the model, so as to achieve accurate prediction of the survival time.

[0049] Therefore, the present invention adopts the above-mentioned interpretable cancer survival prediction method based on multimodal data fusion, introduces a multimodal fusion strategy of whole-slide pathology images (WSIs), pathology report texts, and gene expression data, and combines the end-to-end gene module identification (GMI) algorithm with the adaptive multimodal expert mixture (AMMEM) module, which can effectively solve the problems of the limitations of single-modal data, improper processing of high-dimensional gene data, and imperfect multimodal fusion strategies in the prior art, and realize the prognostic analysis of cancer patients; not only improves the prediction accuracy, but also has good model interpretability, and can automatically identify the biomarkers closely related to cancer prognosis.

[0050] Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit them. Although the present invention has been described in detail with reference to the preferred embodiments, those of ordinary skill in the art should understand that: they can still modify or equivalently replace the technical solutions of the present invention, and these modifications or equivalent replacements cannot make the modified technical solutions deviate from the spirit and scope of the technical solutions of the present invention.

Claims

1. An interpretable cancer survival prediction method based on multi-modal data fusion, characterized in that, It includes the following steps: Step S1: Preprocess the whole-slide pathology image (WSI) to obtain an image feature matrix; Step S2: Preprocess the pathology report text corresponding to the pathology image to construct a text feature matrix; Step S3: Preprocess the high-dimensional gene expression data of the patient to generate multiple survival-related gene modules and their feature vectors; Step S4: Dynamically fuse the multi-modal features obtained in the above steps through an Adaptive Multi-modal Mixture of Experts (AMMEM) module to obtain the final fused feature representation; Step S5: Use the fused feature representation, combine the negative log-likelihood loss function with the Cox proportional hazards model to train a deep learning model for prognostic analysis of cancer patients.

2. The interpretable cancer survival prediction method based on multi-modal data fusion according to claim 1, wherein In Step S1, preprocess the whole-slide pathology image (WSI) to obtain an image feature matrix. The specific process is as follows: Step S11: Detect and segment the tissue regions in the whole-slide pathology image; for each patient divide the tissue regions into a number of non-overlapping image patches (patches), as follows: ； Among them, represents the single image patch of the th patient; is an index variable indicating the position of the current image patch in the set; is the patient 's total number of patches; Step S12: Input each image patch into the image encoder of the pathology vision-language foundation model (PLIP) for feature extraction; The PLIP model extracts fine-grained features from pathological images and then obtains the feature vector representation of each image patch through a linear layer , as follows: ； Among them, is the feature representation of the th image patch, with a dimension of 512; Step S13: Concatenate the feature vectors of all image patches into a matrix to obtain the image feature matrix of the patient , as shown below: ； Among them, is the image feature matrix of the patient .

3. The interpretable cancer survival prediction method based on multi-modal data fusion according to claim 1, characterized in that, In Step S2, preprocess the pathology report text corresponding to the pathology image to construct a text feature matrix. The specific process is as follows: Step S21, segment the pathology report corresponding to the pathology image into Text unit , as shown below: ； Step S22, each text unit After passing through the PLIP text encoder, extract the feature representation of the text , as follows: ； Among them, is the feature representation of the th text unit, with a dimension of 512; Step S23: Concatenate the feature vectors of all text units into a text feature matrix , as shown below: ； Among them, is the text feature matrix of the patient .

4. An interpretable cancer survival prediction method based on multi-modal data fusion according to claim 1, characterized in that, In Step S3, preprocess the high-dimensional gene expression data of the patient to generate multiple survival-related gene modules and their feature vectors. The specific process is as follows: Step S31: Screen for genes with high variance in gene expression data, and select the top genes with the largest variance in expression values. Then, the gene expression data of patient is as follows: ； Among them, is the gene expression data of the patient and contains genes; Step S32: Combine gene expression data and prior gene embedding information, and use a trainable projection matrix and to transform the gene expression data and generate the final gene feature representation , as follows: ； Among them, is the prior gene embedding matrix from Gene2Vec; is the fused gene feature representation; represents the transformation matrix; Step S33: Measure the functional similarity between gene pairs using the Pearson correlation coefficient to generate a weight matrix , as follows: ； Among them, represents the similarity between gene pairs and features, and the diagonal elements are set to 0, indicating that the similarity of a gene to itself is not calculated; Step S34: Use the local maximum algorithm to identify gene modules; By traversing all local maximum edges, the strongest connection is selected based on edge weights, and genes related to the increase in module density are gradually added to the module; among them, the density of the module The calculation formula is as follows: ； Among them, is the number of genes in the module ; is the weight of the edges within the module Multiple gene modules are generated through this process, and similar modules are merged according to the overlap rate threshold ; finally, the patient's of gene modules are obtained, as follows: ； Step S35: For each gene module perform feature gene extraction, use SVD decomposition to extract the principal components of each module, and obtain the gene embedding representation of the patient as follows: ； Among them, is the characteristic gene of the th gene module, which comes from the first principal component of the gene expression matrix of module .

5. The interpretable cancer survival prediction method based on multi-modal data fusion according to claim 4, wherein, In Step S3, introduce a loss function to optimize the quality of gene modules as follows: ； Among them, represents the loss function; represents the number of all observed objects; is the gene module clustering coefficient; The first item represents the compactness of the module; the second metric measures the difference between the embeddings of different gene modules.

6. The interpretable cancer survival prediction method based on multimodal data fusion according to claim 1, wherein, In Step S3, dynamically fuse the multi-modal features obtained in the above steps through an Adaptive Multi-modal Mixture of Experts (AMMEM) module to obtain the final fused feature representation. The specific process is as follows: Step S41: Gating network; Gated network using the image feature matrix , the text feature matrix , and the characteristic genes of the gene module as the multi-modal embedding input, as shown below: ； Among them, , is a learnable matrix for embedding images, genes, and pathological reports; represents the Gaussian error linear unit, represents the root mean square normalization layer; Output of the gating network , as follows: ； Among them, is the fusion weight, which determines whether the final prediction result is based on complete multimodal information or partial or single-modal embeddings; Step S42: Expert network; The module contains four expert networks, and each expert network represents a different combination method of image, gene, and pathological report data; among them, the specific calculation method of the expert network is as follows: ； Among them, and respectively represent the cross-attention and self-attention mechanisms, represents a filtering operation for screening out the corresponding pathological information from the self-attention results; After selecting the expert network, the multi-modal fusion result of the patient is obtained according to the selected expert network .​ 7. The interpretable cancer survival prediction method based on multi-modal data fusion according to claim 1, wherein, In Step S5, use the fused feature representation, combine the negative log-likelihood loss function with the Cox proportional hazards model to train a deep learning model for prognostic analysis of cancer patients. The specific process is as follows: Step S51. During the survival prediction process, use the proportional hazards model to estimate the risk and survival time of the patient; (1) Hazard function used to describe the instantaneous death risk of a patient at time given its survival status then the definition of the hazard function is as follows: ； wherein, is the probability that the patient survives to after time t; represents historical information or covariates; (2)Survival function which represents the cumulative probability that a patient survives after time as follows: ； Among them, represents a time point, from to ; Step S52, Parametrize the proportional hazards model; In the model, the hazard function of the survival data is parameterized as: ； Among them, is the baseline risk function; is the regression coefficient; is the feature vector of the patient, including the fusion features of the image, text, and gene modules; Step S53: Optimize the model parameters by minimizing the negative log-likelihood loss function as follows: ； Among them, is a truncation indicator variable indicating whether an event has occurred; During the optimization process, aim to maximize the likelihood of survival data under the model to achieve accurate survival time prediction.

Images