An interpretable cancer survival prediction method based on multimodal data fusion

Abstract

The present invention discloses an interpretable cancer survival prediction method based on multimodal data fusion, comprising the following steps: preprocessing full-slice pathology images to obtain an image feature matrix; preprocessing the pathology report text corresponding to the pathology image to construct a text feature matrix; preprocessing the patient's high-dimensional gene expression data to generate multiple survival-related gene modules and their feature vectors; dynamically fusing the obtained multimodal features through an adaptive multimodal expert hybrid module to obtain a final fusion feature representation; using the fusion feature representation, combining the negative log-likelihood loss function with the Cox proportional hazard model to train a deep learning model to perform prognosis analysis on cancer patients. The present invention adopts the above-mentioned interpretable cancer survival prediction method based on multimodal data fusion, which not only improves the prediction accuracy, but also has good model interpretability and can automatically identify biomarkers closely related to cancer prognosis.

Description

Technical Field

[0001] The present invention relates to the technical field of bioinformatics, and in particular to an interpretable cancer survival prediction method based on multimodal data fusion. Background Art

[0002] Cancer is one of the major global health challenges, and it is crucial to conduct cancer prognosis analysis for patients. Existing cancer survival prediction methods are mostly based on single-modality data, such as pathology images, genomic data, or pathology reports, but a single data type often cannot fully reflect the patient's condition. In recent years, multimodal data integration analysis has gradually attracted attention, but existing methods still have shortcomings: most multimodal survival prediction models only combine pathology images and genomic data, ignoring the unstructured text data in pathology reports, which may contain tumor-specific descriptions that are highly relevant to patient survival; the high dimensionality of genomic data may lead to overfitting problems, and existing dimensionality reduction methods may affect the accuracy of module generation due to data noise; in multimodal fusion models, the fusion performance is sometimes even worse than that of a single modality, which may be due to factors such as the differences between multimodal data.

[0003] Therefore, in response to the limitations of the above-mentioned existing technologies, the present invention proposes an interpretable cancer survival prediction method based on multimodal data fusion. Summary of the Invention

[0004] The purpose of this invention is to provide an interpretable cancer survival prediction method based on multimodal data fusion. It introduces a multimodal fusion strategy of full-field pathology images (WSI), pathology report text and gene expression data, and combines the end-to-end gene module identification (GMI) algorithm with the adaptive multimodal mixture of experts (AMMEM) module. It can effectively solve the problems of single modality data limitations, improper processing of high-dimensional gene data and imperfect multimodal fusion strategies in the existing technology, and realize prognostic analysis of cancer patients.

[0005] To achieve the above objectives, the present invention provides an interpretable cancer survival prediction method based on multimodal data fusion, comprising the following steps:

[0006] Step S1, preprocessing the whole-slice pathology image WSI to obtain an image feature matrix;

[0007] Step S2: pre-processing the pathology report text corresponding to the pathology image to construct a text feature matrix;

[0008] Step S3: preprocessing the patient's high-dimensional gene expression data to generate multiple survival-related gene modules and their feature vectors;

[0009] Step S4: Dynamically fuse the multimodal features obtained in the above steps through the adaptive multimodal expert mixture AMMEM module to obtain the final fused feature representation;

[0010] Step S5: Using the fused feature representation, combining the negative log-likelihood loss function with the Cox proportional hazards model to train a deep learning model to perform prognostic analysis on cancer patients.

[0011] Preferably, in step S1, the whole-slice pathology image WSI is preprocessed to obtain an image feature matrix, and the specific process is as follows:

[0012] Step S11, perform tissue region detection and segmentation on the full-slice pathology image; The tissue area is divided into several non-overlapping image patches, as shown below:

[0013] ;

[0014] in, Indicates the A single image patch for each patient; is an index variable, indicating the position of the current image block in the set; For patients The total number of patches;

[0015] Step S12: input each image block into the image encoder of the pathological vision-language basic model PLIP to perform feature extraction;

[0016] The PLIP model extracts fine-grained features from pathological images and then passes a linear layer to obtain the feature vector representation of each image patch. , as shown below:

[0017] ;

[0018] in, For the The feature representation of an image patch has a dimension of 512;

[0019] Step S13: Concatenate the eigenvectors of all image patches into a matrix to obtain the patient The image feature matrix , as shown below:

[0020] ;

[0021] in, For patients The image feature matrix.

[0022] Preferably, in step S2, the pathology report text corresponding to the pathology image is preprocessed to construct a text feature matrix. The specific process is as follows:

[0023] Step S21: Segment the pathology report corresponding to the pathology image and divide the pathology report into Text units , as shown below:

[0024] ;

[0025] Step S22: Each text unit After passing through the PLIP text encoder, the feature representation of the text is extracted , as shown below:

[0026] ;

[0027] in, For the Feature representation of text units, with a dimension of 512;

[0028] Step S23: Concatenate the feature vectors of all text units into a text feature matrix , as shown below:

[0029] ;

[0030] in, For patients The text feature matrix.

[0031] Preferably, in step S3, the high-dimensional gene expression data of the patient is preprocessed to generate multiple survival-related gene modules and their feature vectors. The specific process is as follows:

[0032] Step S31: Screen the gene expression data for high variance genes and select the genes with the largest variance in expression values. genes, then the patient The gene expression data are as follows:

[0033] ;

[0034] in, For patients Gene expression data, including genes;

[0035] Step S32: Combine gene expression data and prior gene embedding information and use a trainable projection matrix and Transform gene expression data to generate the final gene feature representation , as shown below:

[0036] ;

[0037] in, is the prior gene embedding matrix from Gene2Vec; is the fused gene signature representation; represents the transformation matrix;

[0038] Step S33: Use the Pearson correlation coefficient to measure the functional similarity between gene pairs and generate a weight matrix , as shown below:

[0039] ;

[0040] in, Represents gene pairs and The similarity between features, and the diagonal elements are set to 0, indicating that the similarity between the gene and itself is not calculated;

[0041] Step S34: using a local maximum algorithm to identify gene modules;

[0042] By traversing all local maximum edges and selecting the strongest connection based on edge weight, genes associated with increased module density are gradually added to the module; the density of the module The calculation formula is as follows:

[0043] ;

[0044] in, It is a module the number of genes in is the weight of the edge within the module;

[0045] This process generates multiple gene modules and the overlap rate threshold is used to Merge similar modules; finally get patients of The gene modules are as follows:

[0046] ;

[0047] Step S35: For each gene module Perform feature gene extraction and use SVD decomposition to extract the principal components of each module to obtain the patient Gene embedding representation , as shown below:

[0048] ;

[0049] in, For the The characteristic genes of the gene module come from the module The first principal component of the gene expression matrix.

[0050] Preferably, in step S3, a loss function is introduced to optimize the quality of the gene module, as shown below:

[0051] ;

[0052] in, represents the loss function; represents the number of all observed objects; It is a gene module The clustering coefficient of

[0053] Item 1 represents the compactness of the module; the second measures the difference between the embeddings of different gene modules.

[0054] Preferably, in step S3, the multimodal features obtained in the above steps are dynamically fused through the adaptive multimodal expert mixture AMMEM module to obtain the final fused feature representation. The specific process is as follows:

[0055] Step S41, gate control network;

[0056] Gated Network Image feature matrix , text feature matrix , characteristic genes of gene modules The multimodal embedding of is taken as input as follows:

[0057] ;

[0058] in, , is a learnable matrix of image, gene, and pathology report embeddings; represents the Gaussian error linear unit, represents the root mean square normalization layer;

[0059] Output of the gating network , as shown below:

[0060] ;

[0061] in, To fuse weights, decide whether the final prediction result is based on complete multimodal information or partial or single modality embedding;

[0062] Step S42: expert network;

[0063] The module contains four expert networks, each representing a different combination of image, gene, and pathology report data. The specific expert network calculation method is as follows:

[0064] ;

[0065] in, and Represent the cross attention and self-attention mechanisms respectively, Represents the filtering operation, which is used to filter out the corresponding pathological information from the self-attention results;

[0066] After selecting the expert network, patients are obtained according to the selected expert network. Multimodal fusion results .

[0067] Preferably, in step S5, the fusion feature representation is used to combine the negative log-likelihood loss function with the Cox proportional hazard model to train a deep learning model to perform prognostic analysis on cancer patients. The specific process is as follows:

[0068] Step S51: In the survival prediction process, use Proportional hazards models were used to estimate patient risk and survival time;

[0069] (1) Risk function Used to describe the patient's The immediate death risk at , then the definition of the risk function is as follows:

[0070] ;

[0071] in, is the number of patients who survive to probability; represents historical information or covariates;

[0072] (2) Survival function Indicates that the patient is at time The cumulative probability of survival after that is as follows:

[0073] ;

[0074] in, Indicates a point in time, from arrive ;

[0075] Step S52: proportional hazards model parameterization;

[0076] exist In the model, the hazard function of survival data is parameterized as:

[0077] ;

[0078] in, is the benchmark risk function; is the regression coefficient; is the patient's feature vector, which contains the fusion features of image, text and gene modules;

[0079] Step S53: Optimize the model parameters by minimizing the negative log-likelihood loss function, as shown below:

[0080] ;

[0081] in, is a censored indicator variable, indicating whether the event occurred;

[0082] During the optimization process, the goal is to maximize the likelihood of survival data under the model, thereby achieving accurate survival time prediction.

[0083] Therefore, the present invention adopts the above-mentioned interpretable cancer survival prediction method based on multimodal data fusion, introduces a multimodal fusion strategy of full-field pathology images (WSI), pathology report text and gene expression data, and combines the end-to-end gene module identification (GMI) algorithm with the adaptive multimodal mixture of experts (AMMEM) module. It can effectively solve the problems of single modality data limitations, improper processing of high-dimensional gene data and imperfect multimodal fusion strategies in the existing technology, and realize the prognosis analysis of cancer patients; it not only improves the prediction accuracy, but also has good model interpretability, and can automatically identify biomarkers closely related to cancer prognosis.

[0084] The technical solution of the present invention is further described in detail below through the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0085] Figure 1 Schematic diagram of a flow chart of an interpretable cancer survival prediction method based on multimodal data fusion according to the present invention;

[0086] Figure 2 A detailed model explanation of the present invention's interpretable cancer survival prediction method based on multimodal data fusion;

[0087] Figure 3 The visualization composition of AMMEM of the present invention; (a) is the gating network; (b) is the expert network. DETAILED DESCRIPTION

[0088] The technical solution of the present invention is further described below with reference to the accompanying drawings and embodiments.

[0089] like Figure 1 and Figure 2 As shown, the present invention provides an interpretable cancer survival prediction method based on multimodal data fusion, comprising the following steps:

[0090] Step S1: preprocess the whole slice pathology image (WSI) to obtain an image feature matrix;

[0091] Step S2: pre-processing the pathology report text corresponding to the pathology image to construct a text feature matrix;

[0092] Step S3: preprocessing the patient's high-dimensional gene expression data to generate multiple survival-related gene modules and their feature vectors;

[0093] Step S4: Dynamically fuse the multimodal features obtained in the above steps through the adaptive multimodal mixture of experts (AMMEM) module to obtain the final fused feature representation;

[0094] Step S5: Using the fused feature representation, combining the negative log-likelihood loss function with the Cox proportional hazards model to train a deep learning model to perform prognostic analysis on cancer patients.

[0095] Example

[0096] Step S1: Preprocess the whole slice pathology image (WSI) to obtain an image feature matrix.

[0097] Step S11, perform tissue region detection and segmentation on the full-slice pathology image. The tissue area is divided into several non-overlapping image patches, as shown below:

[0098] ;

[0099] in, Indicates the A single image patch for each patient; is an index variable, indicating the position of the current image block in the set ( =1,2,…, ); For patients The total number of patches.

[0100] Step S12: Input each image block into the image encoder of the Pathology Vision-Language Based Model (PLIP) for feature extraction.

[0101] The PLIP model extracts fine-grained features from pathological images and then passes a linear layer to obtain the feature vector representation of each image patch. , as shown below:

[0102] ;

[0103] in, For the The feature representation of an image patch has a dimension of 512.

[0104] Step S13: Concatenate the eigenvectors of all image patches into a matrix to obtain the patient The image feature matrix , as shown below:

[0105] ;

[0106] in, For patients The image feature matrix.

[0107] Step S2: pre-process the pathology report text corresponding to the pathology image to construct a text feature matrix.

[0108] Step S21: Segment the pathology report corresponding to the pathology image and divide the pathology report into Text units , as shown below:

[0109] ;

[0110] Step S22: Each text unit After passing through the PLIP text encoder, the feature representation of the text is extracted , as shown below:

[0111] ;

[0112] in, For the The feature representation of a text unit has a dimension of 512.

[0113] Step S23: Concatenate the feature vectors of all text units into a text feature matrix , as shown below:

[0114] ;

[0115] in, For patients The text feature matrix.

[0116] Step S3: preprocess the patient's high-dimensional gene expression data to generate multiple survival-related gene modules and their feature vectors.

[0117] Step S31: Screen the gene expression data for high variance genes and select the genes with the largest variance in expression values. genes, thereby reducing redundant information and ensuring the validity of the data.

[0118] patient The gene expression data are as follows:

[0119] ;

[0120] in, For patients Gene expression data, including A gene.

[0121] Step S32: Combine gene expression data and prior gene embedding information and use a trainable projection matrix and Transform gene expression data to generate the final gene feature representation , as shown below:

[0122] ;

[0123] in, is the prior gene embedding matrix from Gene2Vec; is the fused gene signature representation; Represents the transformation matrix.

[0124] Step S33: Use the Pearson correlation coefficient to measure the functional similarity between gene pairs and generate a weight matrix , as shown below:

[0125] ;

[0126] in, Represents gene pairs and The similarity between features, and the diagonal elements are set to 0, indicating that the similarity between the gene and itself is not calculated.

[0127] Step S34: using the local maximum algorithm to identify gene modules.

[0128] By traversing all local maximum edges and selecting the strongest connection based on edge weight, genes associated with increased module density are gradually added to the module. The calculation formula is as follows:

[0129] ;

[0130] in, It is a module the number of genes in is the weight of the edge within the module.

[0131] This process generates multiple gene modules and the overlap rate threshold is used to Merge similar modules. Finally, get the patient of The gene modules are as follows:

[0132] ;

[0133] Step S35: For each gene module Perform feature gene extraction and use SVD decomposition to extract the principal components of each module to obtain the patient Gene embedding representation , as shown below:

[0134] ;

[0135] in, For the The characteristic genes of the gene module come from the module The first principal component of the gene expression matrix.

[0136] Step S36: In order to further improve the compactness of the identified gene modules and the differences between different gene embeddings, a loss function is introduced to optimize the quality of the gene modules. The loss function is as follows:

[0137] ;

[0138] in, represents the loss function; represents the number of all patients or observation subjects; It is a gene module The clustering coefficient of

[0139] Item 1 The first term represents the compactness of the module; the second term measures the difference between the embeddings of different gene modules. By minimizing this loss function, more compact and well-distinguished gene modules and their embeddings can be obtained.

[0140] Step S4, through the adaptive multimodal mixture of experts (AMMEM) module such as Figure 3 As shown in FIG, the multimodal features obtained in the above steps are dynamically fused to obtain the final fused feature representation.

[0141] Step S41: Gating the network.

[0142] Gated Network Image feature matrix , text feature matrix , characteristic genes of gene modules The multimodal embedding of is taken as input as follows:

[0143] ;

[0144] in, , is a learnable matrix of image, gene, and pathology report embeddings; represents the Gaussian error linear unit, represents a root mean square normalization layer.

[0145] Output of the gating network , as shown below:

[0146] ;

[0147] in, To fuse the weights, it is decided whether the final prediction result is based on the complete multimodal information or the partial (single) modality embedding.

[0148] Step S42: Expert network.

[0149] In order to better integrate multimodal features, It contains four expert networks, each representing a different combination of image, gene, and pathology report data. The specific expert network calculation method is as follows:

[0150] ;

[0151] in, and Represent the cross attention and self-attention mechanisms respectively, Represents the filtering operation, which is used to filter out the corresponding pathological information from the self-attention results.

[0152] After selecting the appropriate expert network, patients can be Multimodal fusion results .

[0153] Step S5: Using the fused feature representation, combining the negative log-likelihood loss function with the Cox proportional hazards model to train a deep learning model to perform prognostic analysis on cancer patients.

[0154] Step S51: In the survival prediction process, use Proportional hazards models were used to estimate patient risk and survival time.

[0155] (1) Risk function Used to describe the patient's The immediate death risk at , then the definition of the risk function is as follows:

[0156] ;

[0157] in, is the number of patients who survive to probability; Represents historical information or covariates, such as the patient's age, gender, disease status, etc.

[0158] (2) Survival function Indicates that the patient is at time The cumulative probability of survival after that is as follows:

[0159] ;

[0160] in, Indicates a point in time, from arrive .

[0161] Step S52: Proportional hazards model parameterization.

[0162] exist In the model, the hazard function of survival data is parameterized as:

[0163] ;

[0164] in, is the benchmark risk function; is the regression coefficient; is the patient’s feature vector, which contains the fusion features of image, text and gene modules.

[0165] The model parameters are optimized by minimizing the negative log-likelihood loss function as follows:

[0166] ;

[0167] in, is a censored indicator variable that indicates whether an event (such as death) occurred.

[0168] During the optimization process, the goal is to maximize the likelihood of survival data under the model, thereby achieving accurate survival time prediction.

[0169] Therefore, the present invention adopts the above-mentioned interpretable cancer survival prediction method based on multimodal data fusion, introduces a multimodal fusion strategy of full-field pathology images (WSI), pathology report text and gene expression data, and combines the end-to-end gene module identification (GMI) algorithm with the adaptive multimodal mixture of experts (AMMEM) module. It can effectively solve the problems of single modality data limitations, improper processing of high-dimensional gene data and imperfect multimodal fusion strategies in the existing technology, and realize the prognosis analysis of cancer patients; it not only improves the prediction accuracy, but also has good model interpretability, and can automatically identify biomarkers closely related to cancer prognosis.

[0170] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit the same. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that they can still modify or replace the technical solutions of the present invention with equivalents, and these modifications or equivalent replacements cannot cause the modified technical solutions to deviate from the spirit and scope of the technical solutions of the present invention.

Claims

1. An interpretable cancer survival prediction method based on multimodal data fusion, characterized by: The steps include: Step S1, preprocessing the whole-slice pathology image WSI to obtain an image feature matrix; Step S2: pre-processing the pathology report text corresponding to the pathology image to construct a text feature matrix; Step S3: preprocessing the patient's high-dimensional gene expression data to generate multiple survival-related gene modules and their feature vectors; Step S4: Dynamically fuse the multimodal features obtained in the above steps through the adaptive multimodal expert mixture AMMEM module to obtain the final fused feature representation. The specific process is as follows: Step S41, gate control network; Gated Network Image feature matrix , text feature matrix , characteristic genes of gene modules The multimodal embedding of is taken as input as follows: ； in, , is a learnable matrix of image, gene, and pathology report embeddings; represents the Gaussian error linear unit, represents the root mean square normalization layer; Output of the gating network , as shown below: ； Step S42: expert network; The module contains four expert networks, each representing a different combination of image, gene, and pathology report data. The specific expert network calculation method is as follows: ； in, and Represent the cross attention and self-attention mechanisms respectively, Represents the filtering operation, which is used to filter out the corresponding pathological information from the self-attention results; After selecting the expert network, patients are obtained according to the selected expert network. Multimodal fusion results ; Step S5: Using the fused feature representation, combining the negative log-likelihood loss function with the Cox proportional hazards model to train a deep learning model to perform prognostic analysis on cancer patients.

2. The interpretable cancer survival prediction method based on multimodal data fusion according to claim 1, characterized in that: In step S1, the whole-slice pathology image WSI is preprocessed to obtain the image feature matrix. The specific process is as follows: Step S11, perform tissue region detection and segmentation on the full-slice pathology image; The tissue area is divided into several non-overlapping image patches, as shown below: ； in, Indicates the A single image patch for each patient; is an index variable, indicating the position of the current image block in the set; For patients The total number of patches; Step S12: input each image block into the image encoder of the pathological vision-language basic model PLIP to perform feature extraction; The PLIP model extracts fine-grained features from pathological images and then passes a linear layer to obtain the feature vector representation of each image patch. , as shown below: ； in, For the The feature representation of an image patch has a dimension of 512; Step S13: Concatenate the eigenvectors of all image patches into a matrix to obtain the patient The image feature matrix , as shown below: ； in, For patients The image feature matrix.

3. The interpretable cancer survival prediction method based on multimodal data fusion according to claim 1, characterized in that: In step S2, the pathology report text corresponding to the pathology image is preprocessed to construct a text feature matrix. The specific process is as follows: Step S21: Segment the pathology report corresponding to the pathology image and divide the pathology report into Text units , as shown below: ； Step S22: Each text unit After passing through the PLIP text encoder, the feature representation of the text is extracted , as shown below: ； in, For the Feature representation of text units, with a dimension of 512; Step S23: Concatenate the feature vectors of all text units into a text feature matrix , as shown below: ； in, For patients The text feature matrix.

4. The interpretable cancer survival prediction method based on multimodal data fusion according to claim 1, characterized in that: In step S3, the high-dimensional gene expression data of the patients are preprocessed to generate multiple survival-related gene modules and their feature vectors. The specific process is as follows: Step S31: Screen the gene expression data for high variance genes and select the genes with the largest variance in expression values. genes, then the patient The gene expression data are as follows: ； in, For patients Gene expression data, including genes; Step S32: Combine gene expression data and prior gene embedding information and use a trainable projection matrix and Transform gene expression data to generate the final gene feature representation , as shown below: ； in, is the prior gene embedding matrix from Gene2Vec; is the fused gene signature representation; represents the transformation matrix; Step S33: Use the Pearson correlation coefficient to measure the functional similarity between gene pairs and generate a weight matrix , as shown below: ； in, Represents gene pairs and The similarity between features, and the diagonal elements are set to 0, indicating that the similarity between the gene and itself is not calculated; Step S34: using a local maximum algorithm to identify gene modules; By traversing all local maximum edges and selecting the strongest connection based on edge weight, genes associated with increased module density are gradually added to the module; the density of the module The calculation formula is as follows: ； in, It is a module the number of genes in is the weight of the edge within the module; This process generates multiple gene modules and the overlap rate threshold is used to Merge similar modules; finally get patients of The gene modules are as follows: ； Step S35: For each gene module Perform feature gene extraction and use SVD decomposition to extract the principal components of each module to obtain the patient Gene embedding representation , as shown below: ； in, For the The characteristic genes of the gene module come from the module The first principal component of the gene expression matrix.

5. The interpretable cancer survival prediction method based on multimodal data fusion according to claim 4, characterized in that: In step S3, a loss function is introduced to optimize the quality of the gene module as follows: ； in, represents the loss function; represents the number of all observed objects; It is a gene module The clustering coefficient of Item 1 represents the compactness of the module; the second measures the difference between the embeddings of different gene modules.

6. The interpretable cancer survival prediction method based on multimodal data fusion according to claim 1, characterized in that: In step S5, the fused feature representation is used to train a deep learning model in combination with the negative log-likelihood loss function and the Cox proportional hazards model to perform prognostic analysis on cancer patients. The specific process is as follows: Step S51: In the survival prediction process, use Proportional hazards models were used to estimate patient risk and survival time; (1) Risk function Used to describe the patient's The immediate death risk at , then the definition of the risk function is as follows: ； in, is the number of patients who survive to probability; represents historical information or covariates; (2) Survival function Indicates that the patient is at time The cumulative probability of survival after that is as follows: ； in, Indicates a point in time, from arrive ; Step S52: proportional hazards model parameterization; exist In the model, the hazard function of survival data is parameterized as: ； in, is the benchmark risk function; is the regression coefficient; is the patient's feature vector, which contains the fusion features of image, text and gene modules; Step S53: Optimize the model parameters by minimizing the negative log-likelihood loss function, as shown below: ； in, is a censored indicator variable, indicating whether the event occurred; During the optimization process, the goal is to maximize the likelihood of survival data under the model, thereby achieving accurate survival time prediction.

Images