A method for predicting patient survival prognosis applicable to multiple cancers

Abstract

The present invention discloses a method for predicting patient survival prognosis applicable to a variety of cancers, comprising the following steps: step 1, obtaining the patient's pathological image and molecular data, the molecular data at least including gene sequence copy number variation data and gene sequence mutation data; step 2, extracting information from the pathological image; step 3, obtaining the features of the molecular data through a mixed attention encoder; step 4, using a triple mask encoder to reconstruct the missing feature information; step 5, constructing a multimodal cancer survival prognosis prediction model, after training, importing all the acquired patient feature information into the prediction model, and finally obtaining the patient's survival prediction. The present invention achieves more accurate survival prediction by adopting a targeted feature extraction method for different modal data and effectively processing the missing modal data.

Description

Technical Field

[0001] The present invention belongs to the technical field of artificial intelligence and bioinformatics, and specifically relates to a method for predicting the survival prognosis of patients with various cancers, especially bladder cancer, breast cancer, lung adenocarcinoma and endometrial cancer. Background Art

[0002] The survival prognosis of cancer patients plays an important role in clinical decision-making, policy making, and planning for patients and their families. Survival prognosis is also a basic task in computational pathology, which uses whole-slide images of tissues for automatic risk assessment, patient stratification and triage, and prediction of treatment response. The Cox hazard ratio model is a classic survival prediction model for cancer patients, but this method fails to explain the role of pathology and genetic material, limiting the effectiveness of survival prediction. Secondly, survival prediction analysis is significantly different from the tasks of cancer staging and grading. It relies on a large amount of data under different factors for analysis, modeling, and prediction. Therefore, high-quality acquisition of data in multiple modalities is very important for developing patient survival prediction models.

[0003] Existing multimodal patient survival prognosis prediction methods often ignore the data relationship within and between modalities, and only obtain better feature representation on a single modality and then use simple feature fusion schemes, such as vector concatenation, modal data aggregation, Kronecker product and other fusion methods, to process different modal data, which greatly limits the prediction performance of the model in practical applications. In addition, certain reasons in real life often cause patients to ignore certain aspects of examinations or are unwilling to pay for examinations, resulting in the loss of multimodal data, thus affecting the accuracy of patient survival prognosis prediction.

[0004] Therefore, it is necessary to design a new method to predict patient survival prognosis. Summary of the invention

[0005] The purpose of the present invention is to provide a method for predicting the survival prognosis of patients with bladder cancer, breast cancer, lung adenocarcinoma and endometrial cancer, so as to solve the problem that the existing multimodal patient survival prognosis prediction method proposed in the background technology often ignores the data relationship within and between modalities, only obtains a better feature representation on a single modality and then uses a simple feature fusion scheme, such as vector splicing, modal data aggregation, Kronecker product and other fusion methods to process different modal data, so that the prediction performance of the model is greatly limited in practical applications; and certain reasons in real life often cause patients to ignore certain aspects of examinations or are unwilling to pay for examinations, resulting in the loss of multimodal data, thereby affecting the accuracy of patient survival prognosis prediction.

[0006] To achieve the above object, the present invention provides a method for predicting the survival prognosis of patients with various cancers, comprising the following steps:

[0007] Step 1: Obtain the patient's pathological images and molecular data, where the molecular data at least includes gene sequence, copy number variation data, and gene sequence mutation data;

[0008] Step 2: extracting information from pathological images;

[0009] Specifically include:

[0010] Step 2.1, construct the spatial topological structure of large, medium and small fields of view of the pathological image to obtain the characteristics at the cellular level, tissue level and tumor heterogeneity level;

[0011] Step 2.2, cross-fusion learning of pathological image structural features under large, medium and small fields of view and obtaining cross-scale contextual relationships between image features under different fields of view;

[0012] Step 3: Obtain the features of molecular data through a mixed attention encoder;

[0013] Step 4: Use a triple mask encoder to reconstruct the missing feature information;

[0014] Step 5: Build a multimodal cancer survival prognosis prediction model. After training, import all the characteristic information of the patients into the prediction model to finally obtain the survival prediction of the patients.

[0015] In a specific embodiment, the method for predicting patient survival prognosis applicable to multiple cancers is applicable to predicting survival prognosis of patients diagnosed with bladder cancer, breast cancer, lung adenocarcinoma and endometrial cancer.

[0016] In a specific implementation, the step 2.1 includes the following specific steps:

[0017] First, the pathological images were segmented into patches of 256×256, 512×512, and 1024×1024, i.e., small, medium, and large fields of view, and then the background was removed to keep the patches containing image information;

[0018] Each patch embedding obtained is unified into a 1024-dimensional vector representation, and these vectorized feature representations are connected in groups of 8 adjacent features to obtain a graphical representation of pathological images at different scales, which is G P ={v,e}, where v and e represent the set of points and the set of edges in the graph.

[0019] In a specific implementation, the step 2.2 includes the following specific steps:

[0020] First, we use the graph convolutional neural network to extract the image features under small, medium and large fields of view respectively, and then save the extracted features into G P ′ and passed to Transformer; Transformer encoder from G P ′ to obtain rich semantic information, namely G P ″; Set the residual connection to feature G P ′ and G P ″Perform information transmission and feature fusion;

[0021] Then the features obtained for the large field of view and the features obtained in the middle field of view Fusion, centering the field of view to obtain features and features obtained with a small field of view Fusion is performed, and then the features of large, medium and small fields of view are integrated.

[0022] In a specific implementation, in step 3, the hybrid attention encoder includes a context attention module, a channel attention module and a global attention module.

[0023] In a specific embodiment, the contextual attention module includes discrete wavelet transform, long short-term neural network and multi-head attention mechanism.

[0024] In a specific implementation, the discrete wavelet transform uses Dobesh wavelet.

[0025] In a specific implementation, in step 4, the triple mask encoder includes an asymmetric encoder and decoder, wherein the encoder includes three Transformers in parallel, and the decoder includes a cross-attention layer and a self-attention layer.

[0026] In a specific embodiment, in step 5, the constructed multimodal cancer survival prognosis prediction model includes a modal feature mixing module, the modal feature mixing module includes two layers of perceptrons and a normalization layer, and each perceptron includes two fully connected layers.

[0027] In a specific embodiment, the hyperparameters in the multimodal cancer survival prognosis prediction model are set as follows: learning rate is 0.005, batch size is 50, optimizer is Adam, epoch is 40, and Dropout is 0.2.

[0028] Compared with the prior art, the present invention has the following beneficial effects:

[0029] The present invention achieves more accurate survival prediction by adopting targeted feature extraction methods for different modal data and effectively processing missing modal data.

[0030] The cross-fusion Transformer adopted in the present invention can effectively fuse the local information and global information of images at different magnifications, and improve the survival prediction accuracy by obtaining cross-scale context-related information.

[0031] The triple mask encoder adopted in the present invention is an asymmetric masked triple mask encoder, and its mask decoder based on the attention mechanism can reconstruct multimodal data features from the latent feature representation and masked tokens.

[0032] Benefiting from the extraction of different modality data by the cross-fusion TranSformer and the hybrid encoder respectively, the triple mask encoder can reconstruct the missing modality data. The present invention can predict the survival of patients in the case of missing modality data and missing intra-modality data.

[0033] The present invention can obtain survival-related features not only from the cellular level, but also from the tissue level and tumor heterogeneity.

[0034] In addition to the above-described purposes, features and advantages, the present invention has other purposes, features and advantages. The present invention will be further described in detail with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] The drawings constituting a part of this application are used to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings:

[0036] Figure 1 It is a flow chart of the patient survival prognosis prediction method of the present invention; the flow chart includes a multi-view feature extraction module, a cross fusion module, a hybrid attention mechanism module, a triple mask autoencoder and a multimodal prediction survival module.

[0037] Figure 2 The figure is a comparison chart of the survival prognosis of patients with bladder cancer, breast cancer, lung adenocarcinoma and endometrial cancer using the existing prediction method and the prediction method of the present invention.

[0038] Figure 3 The figure is an ablation experiment diagram of evaluating the survival prediction performance (C-index) of the present invention on four cancer datasets in the case of modality loss. P, R, C&M represent pathological images, mRNA, CNV, and MUT, respectively. They correspond to pathological patterns, mRNA patterns, and other patterns, respectively.

[0039] Figure 4 Graph of ablation experiments evaluating the C-index performance of TriMAE on four cancer datasets in the absence of intra-modality information.

[0040] Figure 5 It is an ablation experiment diagram evaluating the C-index performance of the hybrid attention encoder of the present invention on four cancer datasets. DETAILED DESCRIPTION

[0041] The embodiments of the present invention are described in detail below in conjunction with the accompanying drawings. The specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0042] Example 1

[0043] The present invention proposes a method for predicting the survival prognosis of patients with various cancers, comprising the following steps:

[0044] Step 1: Obtain the patient's pathological images and molecular data, where the molecular data at least includes gene sequence copy number variation data and gene sequence mutation data.

[0045] Step 2: extracting information from pathological images;

[0046] The step 2 specifically includes:

[0047] Step 2.1, construct the spatial topological structure of large, medium and small fields of view of the pathological image to obtain the characteristics at the cellular level, tissue level and tumor heterogeneity level;

[0048] Step 2.2: Cross-integrate the structural features of pathological images under large, medium and small fields of view and obtain the cross-scale contextual relationship between image features under different fields of view.

[0049] The cross-fusion Transformer adopted in the present invention can effectively fuse the local information and global information of images at different magnifications, and improve the survival prediction accuracy by obtaining cross-scale context-related information.

[0050] Step 3, obtain the characteristics of molecular data through a hybrid attention encoder. The present invention uses a hybrid attention encoder to learn the characteristics of gene sequences, copy number variations and mutations. The hybrid attention encoder includes a contextual attention module, a channel attention module and a global attention module. The contextual attention module is constructed by combining discrete wavelet transform, long-short time neural network and multi-head attention mechanism.

[0051] Step 4: Use a triple mask encoder to reconstruct the missing feature information. The triple mask encoder used in the present invention is an asymmetric mask triple mask encoder. The mask decoder based on the attention mechanism can reconstruct multimodal data features from the potential feature representation and masked tokens.

[0052] Step 5: Build a multimodal cancer survival prognosis prediction model. After training, import all the characteristic information of the patients into the prediction model to finally obtain the survival prediction of the patients.

[0053] The present invention uses the diagnostic pathology data, gene sequence, copy number variation and mutation data of the cancer genome atlas to form the multimodal data D = {P, R, C & M} of the present invention, and its corresponding observation time and state can be expressed as (t i ,β i ), meeting the data needs of multi-omics data to predict patient survival.

[0054] Furthermore, the step 2.1 is specifically as follows:

[0055] Pathological images at low magnification mainly show image information at the tissue level, while at high magnification, mainly show information at the cellular level. Therefore, the whole slide image uses a sliding window strategy to divide the image into 256×256, 512×512, and 1024×1024 patches, i.e., small, medium, and large fields of view. The background is then removed and the patches containing image information are retained. On average, more than 20,000 patches can be obtained for each high-resolution whole slide image. Patch embedding representations are extracted before model training to reduce the demand for hardware equipment. In extracting patch embedding representations, KimiaNet is preferably used to extract patch feature embeddings. Each patch embedding obtained is unified into a 1024-dimensional vectorized representation, and these vectorized feature representations are connected in groups of 8 adjacent features to obtain a graphical representation of pathological images at different scales, which is G. P = {v, e}. v and e represent the set of points and edges in the graph. That is, the feature of the whole slide image after processing is G Ps , G Pm and G Pl The adjacency matrix A of size M×M defines the similarity between graph nodes. Then, the copy number variation and mutation (C, M) data are concatenated together as an input modality data using the concat method. Through the above preprocessing operations on pathology image (Path) data and molecular data, multimodal data of P, R, C&M are constructed for each patient.

[0056] Furthermore, in step 2.2, when preprocessing the whole slide image data, the resolution of the Transformer patch input is fixed, so the field of view of the learned features is also limited. Ps , G Pm and G PlIt can reflect different local information under different fields of view. A common method for extracting multi-scale features is to use parallel Transformers to extract small, medium and large fields of view respectively. Each Transformer contains the same number of encoders. However, this parallel Transformer has a high computational complexity and ultra-large-scale parameters, which will cause an explosive growth in hardware costs. In order to solve these problems, the present invention adopts a cross-fusion Transformer method, which can efficiently learn the cross-scale contextual correlation between encoders of transformers with different fields of view. Compared with the parallel transformer, it reduces the reasoning complexity and the consumption of hardware resources while reducing the parameters of the model.

[0057] The process of processing features using the cross-fusion Transformer method is as follows.

[0058] Preprocessed field of view G P ={V,E} is a set of nodes V = {v1,v2,…,v n} and the edges E between these nodes to construct the spatial topological spectrum of all images in the field of view. Set (v m ,v n )∈E represents node v m ∈V to v n ∈V, N(v m )={v n ∈V|(v m , v n )∈E} represents the node v m areas, and N(v m )∪{v m}. The edge G designed by the present invention P is undirected, that is Set N(S) = {v∈V|(v m , v n )} represents the domain of the node set S, and N(S) represents N(S)∪S. Indicates dimension d x The node v m Characteristics of h m ∈R d Represented as node v m The d-dimensional embedding of and H = (h1, h2, ..., h n )∈R d×n The present invention uses To represent a node set Embedding. is a p×q matrix A∈R p×q The vector representation of A m，n ∈A m+(n-1)p The present invention uses A n Denotes the nth column of A. For a positive integer L, [L] = {1, 2, ..., L}. The graph convolutional neural network updates the node embedding H by minimizing the objective function. The present invention first uses a graph convolutional neural network with L = 2 layers of graph convolution to extract image features under small, medium and large fields of view respectively to generate the final node embedding H = H L ,like:

[0059] When H 0 =X, is the learnable parameter W l The l-th layer feature transfer function. Strictly follow the node aggregation and update method, the process is:

[0060]

[0061]

[0062] in, A function that generates a separate feature for each adjacent value of vm in the lth feature transfer iteration, It maps a set of features to the final feature The aggregation function of It uses the previous node embedding features Mapped features The optimization function for updating node embedding is used with input feature xm. In short, in the present invention, graph sampling and aggregation are used, and the channel of each node is mapped to the hidden dimension C, which is set to 500. The use of graph convolutional neural network establishes the mutual communication between nodes in the feature graph GP under each field of view, and then saves the extracted features to G P ′ and passed it to Transformer.

[0063] Each Transformer learns representative features through a multi-head self-attention mechanism. The Transformer encoder consists of a multi-head self-attention layer, a normalization layer, and a forward transmission (FF) layer. First, the Transformer encoder uses a multi-head self-attention mechanism to extract the representative features from G P ′ to obtain rich semantic information, namely G P ″. In order to prevent the gradient from disappearing and improve the performance of the model, a residual connection is set to connect the feature G P ′ and G P″ to transfer information and fuse features. Then use the normalization layer to standardize the output, and the previous transmission network will perform nonlinear transformation on the complex semantic features to capture the complex patterns and features in the input data. The process is as follows:

[0064] G P =FFN(LN(MSA(G P ′)+G P ′))

[0065] FFN(x))=max(0,xW1+b1)W2+b2

[0066]

[0067] In order to use the cross-fusion Transformer to better represent the semantic information and spatial relationship of these modules, the present invention uses two Transformer encoders to learn features for different fields of view. Through this design, the present invention can reduce the number of Transformer encoders and learn the features of different fields of view fairly, which is conducive to reducing hardware equipment requirements and improving the reasoning speed of the model.

[0068] In order to discover the cross-scale contextual correlation between different regions of different histopathological images and different fields of view, the present invention uses a cross-fusion module to cross the features of different fields of view. In this module, first let and Fusion, and Fusion, and then let the features of large, medium and small fields of view merge. In this way, the feature information of the large field of view can locally guide the learning of the features of the small field of view, and further explore the contextual relationship of the cross-scale feature map. The fusion of two different fields of view, such as the fusion of large and medium fields of view and the fusion of medium and small fields of view, is regarded as the first type of cross fusion, and the fusion of three different fields of view, such as large, medium and small fields of view, is regarded as the second type of cross fusion.

[0069] like Figure 1 As shown, the purpose of the first type of cross-fusion module is to convert the first layer of Transformer encoder processed and In order to ensure that the Transformer encoder can learn cross-scale contextual correlations, it is necessary to cross-fuse one-dimensional features. and Remap and integrate into the same feature space To achieve this goal, the present invention uses a reshaping function R(·) to map the one-dimensional feature and Convert it into a two-dimensional feature map and perform a splicing operation, and then use two-dimensional convolution to perform multi-channel attention nonlinear mapping and fusion on the spliced ​​features. In addition, in order to prevent the loss of some information during the crossover, the present invention weights and sums the fused features with the features of the nonlinear mapping to fully integrate the semantic information of the large field of view and the medium field of view. The process is:

[0070]

[0071] is the feature concatenation operation, and Fusion(·) is the function of fusing large and small fields of view. G is the output of feature fusion. Finally, the two-dimensional fusion features are nonlinearly transformed and mapped to the one-dimensional feature space. Then, and The first type of cross-fusion module is also applicable to and feature fusion.

[0072] In order to integrate the features of all fields of view, the purpose of the second type of cross fusion is to process the second layer of Transformer and Cross-fusion is performed. These cross-fusion modules generate cross-scale embeddings that help the cross-fusion Transformer learn the cross-scale contextual correlations between different fields of view, retrieve the most representative features under different fields of view and magnifications, and help patients make survival prognoses. In detail, the present invention simultaneously maps three one-dimensional feature maps to a two-dimensional feature space, and performs feature splicing on them, and uses the multi-channel characteristics of a two-dimensional convolutional neural network to perform local feature extraction on the weighted features. The fused features are then subjected to an upsampling operation of the convolutional neural network to keep the output features consistent with the input features. Finally, the cross-fused features are passed to the channel global average pooling and the fully connected layer to optimize the output results.

[0073]

[0074] Furthermore, in step 3, the context attention module is specifically as follows.

[0075] In view of the large amount of noise in gene sequence, mutation and copy number variation data, the present invention first uses discrete wavelet transform in the context attention module to reduce the noise of these data. In discrete wavelet transform, it is preferred to use Dobeshi wavelet to preprocess the input molecular data. Dobeshi wavelet is a type of compact support wavelet basis, which is suitable for processing signals with mutation characteristics and can effectively process some gene expression changes or mutant genes in genomic data. Dobeshi wavelet consists of a pair of filters, respectively called low-pass filter and high-pass filter. Through these two filters, the signal is decomposed into approximate coefficients and detail coefficients of different scales (or frequencies), namely:

[0076]

[0077] Where u(t) is the unit function. The detail coefficients after wavelet transform are soft-thresholded. The purpose of soft-thresholding is to set the detail coefficients with smaller amplitudes to zero, thereby eliminating noise. The wavelet coefficients after soft-thresholding are inversely transformed to reconstruct the denoised signal. For feature extraction of molecular sequencing data, the main advantage of long-short-time neural networks is that they can process and capture long-term dependencies in R, C, and M data. In the long-short-time neural network, the input data of each time step is converted into a vector of fixed length, i.e., the extracted features. The long-short-time neural network uses a gating mechanism that can selectively retain or forget past information, so it can better handle long-term dependencies, and also has a certain robust effect on molecular data noise, improving the accuracy and robustness of feature extraction. Then, the data is normalized to standardize the input data.

[0078] At the same time, in order to obtain the local attention features of molecular data, the present invention uses the self-attention mechanism to extract their features. The multiple heads in the self-attention mechanism divide the molecular data into segments, and the heads map the segment inputs to a high-dimensional space to generate feature weights W Q ,W K and W V . Use W Q ,W K Calculate the attention score to get the attention weight, and then use the weight and W V The weighted sum of attention is calculated, i.e., the local attention feature. Each head obtains local features separately and the local features extracted by the heads are concatenated together at the end. The Dropout function is then used to prevent the model from overfitting the training data.

[0079] Furthermore, in step 3, the channel attention module is specifically as follows.

[0080] Channel attention introduces the global information of molecular data to calculate the weight of channel attention. The attention mechanism can use global information to weight features, thereby activating more features. The CAN gate module consists of several standard one-dimensional convolutional layers with GELU activation function and another one for channel attention. Figure 1 As shown in the figure, the present invention compresses the number of channels of the two convolutional layers to a constant β, and the channels of the output features after the first convolution are compressed to Cβ, where C is the number of channels of the first convolution. Then, the second convolution expands the features to C channels, and then uses channel attention to adaptively adjust the channel features. This can reduce the computational cost of directly using fixed-width convolution.

[0081] Finally, the global attention module uses a multi-head self-attention mechanism to extract features from the merged features.

[0082] The workflow of the entire hybrid attention encoder is as follows:

[0083] x CAT =MSA(LN(LSTM(DWT(LN(x R,C&M ))))

[0084] x CNA =αCAB(LN(x R,C&M ))

[0085] x HAE =MSA(LN(x R，C&M )+x CAT +x CNA )

[0086] x CNA =CNA(LayerNorm(x in ))

[0087] x out =MSA(LayerNorm(x in )+x CTA +x CNA ).

[0088] Furthermore, the step 4 is specifically as follows:

[0089] In order to make the model not affected by feature loss during reasoning, the present invention designs a triple mask encoder to reconstruct the missing feature information in multimodality. The triple mask encoder can reconstruct the detailed features of the masked frames in the video. Through the analysis of multimodal data and features, it is found that different modalities have missing features. Therefore, the method used in the present invention is that when the triple mask encoder processes multimodal data, each branch only masks one of the modal features, and the masking rate is as high as more than 90%. In this way, it is difficult for the features to reconstruct the masked pixels through the surrounding features, forcing the encoder to learn meaningful features.

[0090] The triple mask encoder consists of an asymmetric encoder-decoder. The encoder is composed of three Transformers in parallel as the basic backbone network. It only operates on those unmasked feature patches, which greatly reduces the inference cost of the encoder. The decoder is a lightweight module composed of a cross-attention layer and a self-attention layer. The feature information obtained by the encoders from different branches pays attention to each other through the cross-attention layer, and then pays attention to each other through the self-attention layer to complement the missing information. Its input is mainly the unmasked feature patches and the masked tokens. The masked features are vectors shared and learnable by all missing positions. The triple mask encoder takes features that mask different modal information as input. Through the learning of the encoder and decoder, the contextual relationship of the masked tokens and the semantic information of the unmasked features can be used to reconstruct the missing feature details. The triple mask encoder uses the attention mechanism principle of the cross-attention layer to perform local attention calculations on the reconstructed features from the three branches. The process is:

[0091]

[0092] Where x1, x2, and x3 are the features extracted by different branches on the triple mask encoder, m Indicates different modes, and Respectively represent the learned projection matrix. k for and The dimension size of the set is used to scale the attention size. After feature cross-fusion, the reconstructed features are subjected to self-attention feature learning to obtain the best feature representation for each modality data.

[0093] Furthermore, the step 5 is specifically as follows:

[0094] In order to establish the interaction between multimodal data, the present invention designs a module for modal feature mixing, including two layers of perceptrons and a normalization layer. Each perceptron contains two fully connected layers to mix information of different modalities and edge information of channels to promote comprehensive interaction between multimodalities. Then, the present invention separates the features after interaction again and combines the previous features to obtain the final modal output. The process is expressed as:

[0095] F=MLP(LayerNorm(MLP(x)+x)+x).

[0096] Finally, the survival output of mode m can be calculated as: W m =MLP(Readout(F m )).

[0097] Therefore, the Coxloss of mode m can be defined as:

[0098]

[0099] Among them, O m (i) and O m (j) represent the survival output of the i-th and j-th patients in mode m, respectively. In summary, the loss of the entire model can be defined as:

[0100]

[0101] Among them, λ0 and λ m are adjustable parameters. Considering the complexity of multi-modality, it is preferred that λ0 and λ m They are set to 5 and 1 respectively. The hyperparameters of the optimal model are set as follows: learning rate is 0.005, batch size is 50, optimizer is Adam, epoch is 40, and Dropout is 0.2.

[0102] In the process of survival prognosis, all modalities are fused for decision making, and finally the patient's survival prediction is obtained. Benefiting from the extraction of different modal data by cross-fusion Tranformer and hybrid encoder respectively, the triple mask encoder can reconstruct the missing modal data. The present invention can predict the patient's survival in the case of missing modal data and missing intra-modal data.

[0103] The present invention is compared with the most advanced single-modal and multi-modal methods: For single-modal data, the present invention compares Cox, SNNTrans, XGBLC, DeepAttnMISL, CLAM-SB, CLAM-MB, TransMIL, DTFD-MIL. For genomic data, the present invention compares SNN, AttnMIL, GS convolutional neural network, Pathomic, Metric learning, MMF, Porpoise, MultiSurv, MCAT, Outer product, HGCN, MOTCat and other methods. The results are shown in Table 1. Since different experiments use different cancer data sets, - is used to indicate that the results have not been reported yet.

[0104] Table 1. Performance of C-Index (mean ± std) on four cancer datasets. P, R, C&M represent pathological images, m-gene sequences, copy number variations and mutations, respectively. They correspond to pathological patterns, m-gene sequence patterns, and other patterns, respectively. FORESEE is the solution of the present invention.

[0105] method P R C&M BLCA BRCA LUAD UCEC Cox √ - - 0.621±0.039 0.68±0.025 SNN √ √ 0.618±0.022 0.624±0.06 0.611±0.047 0.679±0.04 SNNTrans √ 0.659±0.032 0.647±0.063 0.638±0.022 0.656±0.038 XGBLC √ - 0.6811±0.003 0.673±0.006 0.675± AttnMIL √ √ 0.599±0.048 0.609±0.065 0.567±0.01 0.67±0.01 DeepAttnMISL √ 0.504±0.042 0.524±0.043 0.548±0.05 0.597±0.059 CLAM-SB √ 0.559±0.034 0.573±0.044 0.594±0.063 0.644±0.061 CLAM-MB √ 0.565±0.027 0.578±0.032 0.582±0.072 0.609±0.082 TransMIL √ 0.575±0.034 0.666±0.029 0.642±0.046 0.655±0.046 DTFD-MIL √ 0.546±0.021 0.609±0.059 0.585±0.066 0.656±0.045 GS Convolutional Neural Network √ √ 0.596±0.014 0.636±0.021 0.617±0.012 0.693±0.031 Pathomic √ √ 0.586±0.062 - 0.826±0.009 - Metriclearning √ √ 0.584±0.012 0.661±0.043 0.613±0.026 0.69±0.011 MMF √ √ √ 0.636±0.012 0.674±0.023 0.600±0.057 0.634±0.032 Porpoise √ √ √ 0.636±0.024 0.652±0.042 0.647±0.031 0.695±0.032 MultiSurv √ √ - - 0.626±0.031 0.69±0.015 MCAT √ √ 0.672±0.032 0.659±0.031 0.659±0.027 0.649±0.043 Outer product √ √ - - 0.602±0.014 0.676±0.027 HGCN √ √ - - 0.651±0.008 0.682±0.014 MOTCat √ √ 0.683±0.026 0.673±0.006 0.67±0.038 0.675±0.004 FORESEE √ √ √ 0.686±0.008 0.679±0.013 0.672±0.013 0.730±0.005

[0106] Single modality VS the present invention: The present invention showed the best results on bladder cancer (BLCA), breast cancer (BRCA), and endometrial cancer (UCEC) cancer cohorts. This shows that the present invention can effectively improve the accuracy of survival prediction and has generalization. The experiments of the present invention on the lung adenocarcinoma (LUAD) dataset also achieved results that are better than most methods. Among them, the survival prediction using genetic data is significantly better than the survival prediction using pathological images. Because there is a strong correlation between genes and phenotypes.

[0107] In the multi-modal patient cohort experiment, the present invention is clearly in an advantageous position, which indicates that the multi-modal features can effectively improve the performance of survival prediction in the absence of modalities and the absence of intra-modal information.

[0108] Multimodal cutting-edge method vs. the present invention: Through a large number of experimental verifications, the present invention has improved the prediction performance of the cutting-edge MOTCat by 0.3%, 0.6% and 3.7% respectively in the bladder cancer (BLCA), breast cancer (BRCA) and endometrial cancer (UCEC) cohorts. This shows that the present invention can improve the performance of multimodal prediction by adding multimodal data of copy number variation and mutation, because the expression of copy number variation and mutation directly indicates the cause of cancer. Secondly, the strategy of extracting features from multiple fields of view can effectively assist the present invention in survival prediction.

[0109] Ablation experiment.

[0110] The influence of large and small fields of view in pathological modalities.

[0111] In order to study the effect of feature information in large and small fields of view in pathological modality on the prognosis of cancer patients, the fusion strategies of different fields of view are continuously changed by controlling the modality data R and C&M. The results are shown in Table 2. s , P m and P l When P s It can provide features closely related to prognosis for multimodal data. The present invention experiments in independent R and C&M modalities and found that the prediction performance of using multiple fields of view is better than that of using a single field of view. s and P m In the case of combined use, the present invention achieves better prediction performance on 4 cancer cohorts. However, the poor prediction of survival using only C&M and multimodal data composed of different independent fields of view may be due to the lack of some complementary modality information to help multimodal data for survival analysis. The performance of the present invention in performing survival analysis using feature information in all fields of view in R and C&M modalities is significantly better than other cases where field of view features are missing.

[0112] In summary, after a large number of experiments, it was found that the features of different fields of view in the fusion pathological images are very important for survival prediction. The present invention can not only obtain survival-related features from the cellular level, but also obtain survival-related features from the tissue level and tumor heterogeneity.

[0113] Table 2. Performance of C-Index (average) on four cancer datasets. s , P m , P l , R, C&M represent the small field of view, medium field of view, large field of view, m gene sequence, copy number variation and mutation of pathological images, respectively. They correspond to pathological mode, m gene sequence mode and other modes, respectively.

[0114]

[0115] In order to evaluate the importance of modality multi-modality data for predicting survival in the present invention, a set of ablation experiments were designed to explore the impact of modality data missing. Figure 3, on four cancer data sets, the performance of predicting patient survival using only R and C&M data is poor, probably because the amount of data is relatively small, which affects the mapping relationship with survival-related data. The performance of the present invention when using P data independently for prediction is about 2% to 3% higher than that of using R and C&M data independently. Under the two modalities of P and C&M, the present invention obtains better performance. What is most noteworthy is that the present invention uses P, R and C&M multimodal data to obtain better performance on four cancer cohorts. Therefore, although the lack of multimodal data can predict patient survival, making use of multimodal data as much as possible can significantly improve the accuracy of survival prediction.

[0116] The impact of missing information within the modality. In order to evaluate the effectiveness of the triple mask encoder, a set of ablation experiments were also conducted to explore the impact of different mask encoders on the prediction performance of the present invention. The compared methods include simple mask encoder (MAE), Conv mask encoder (ConvMAE), FC mask encoder (FCMAE), D mask encoder (DMAE), PUT and MAGE strategies. Figure 4 In the paper, it was found that the technical solution without using mask encoder (None) was affected by the missing data within the modality and showed poor prediction performance. Conv mask encoder (ConvMAE), MAGE, simple mask encoder (MAE), PUT and D mask encoder (DMAE) all achieved good prediction performance. FC mask encoder (FCMAE) also performed poorly when the information within the modality was missing. However, the triple mask encoder (TriMAE) successfully tested on four cancer cohorts and achieved the best prediction performance. This may be because the structural design of the asymmetric mask in the triple mask encoder can obtain potential feature representations from the unmasked parts and masked tokens on different branches, making up for the problem of missing information in multimodal data. In short, the triple mask encoder is indispensable in multimodal prediction of survival.

[0117] Impact of molecular feature extraction methods. Finally, the hybrid attention encoder (HAE) of the present invention is compared with various better molecular feature extraction methods, including convolutional neural network (CNN), long short-term neural network (LSTM), AutoEncoder (AutoEncoder), DAE, VAE and Transformer strategies. Figure 5The comparative results of the present invention for survival prediction on four cancer patient cohorts are described. The long-short time neural network obtains the best prediction performance on the BLCA patient cohort. It is important that the hybrid attention encoder (HAE) of the present invention obtains the best results on BLCA, BRCA and LUAD. This is because the context attention module in the hybrid attention encoder can obtain the long-range dependencies of molecular features and obtain local feature information. The channel attention module in the hybrid attention encoder introduces the global information of the molecular data. Finally, the global information can be used to weight the features through the attention mechanism, thereby activating more features. In short, the hybrid attention encoder has great advantages in extracting molecular data.

[0118] The above contents are further detailed descriptions of the present invention in combination with specific preferred embodiments, and it cannot be determined that the specific implementation of the present invention is limited to these descriptions. For ordinary technicians in the technical field to which the present invention belongs, several simple deductions and substitutions can be made without departing from the concept of the present invention, which should be regarded as belonging to the protection scope of the present invention.

Claims

1. A method for predicting the survival prognosis of patients with various cancers, characterized in that: The following steps are involved: Step 1: Obtain the patient's pathological images and molecular data, where the molecular data at least includes gene sequence, copy number variation data, and gene sequence mutation data; Step 2: extracting information from pathological images; Specifically include: Step 2.1, construct the spatial topological structure of large, medium and small fields of view of the pathological image to obtain the characteristics at the cellular level, tissue level and tumor heterogeneity level; Step 2.2, cross-fusion learning of pathological image structural features under large, medium and small fields of view and obtaining cross-scale contextual relationships between image features under different fields of view; The step 2.2 includes the following specific steps: First, we use the graph convolutional neural network to extract the image features under small, medium and large fields of view respectively, and then save the extracted features into G P ′ and passed to Transformer; Transformer encoder from G P ′ to obtain rich semantic information, namely G P ″; Set the residual connection to feature G P ′ and G P ″Perform information transmission and feature fusion; Then the features obtained for the large field of view and the features obtained in the middle field of view Fusion, centering the field of view to obtain features and features obtained with a small field of view Perform fusion, and then integrate the features of large, medium and small fields of view; Step 3: Obtain the features of molecular data through a mixed attention encoder; In step 3, the hybrid attention encoder includes a context attention module, a channel attention module and a global attention module; the context attention module includes a discrete wavelet transform, a long-short time neural network and a multi-head attention mechanism; the discrete wavelet transform adopts a multi-bech wavelet; In the context attention module, the Dobesh wavelet is first used to preprocess the input molecular data; then the long-short time neural network is used to extract the features of the molecular sequencing data, and the multi-head attention mechanism is used to obtain the local attention features of the molecular data; Then, the global information of the molecular data is introduced through the channel attention module to calculate the weight of the channel attention; Finally, the combined features are extracted using a multi-head self-attention mechanism through the global attention module; Step 4: Use a triple mask encoder to reconstruct the missing feature information; In step 4, the triple mask encoder includes an asymmetric encoder and decoder, wherein the encoder includes three Transformers in parallel, and the decoder includes a cross attention layer and a self-attention layer; The triple mask encoder takes features that mask different modal information as input, and through the learning of the encoder and decoder, it uses the contextual relationship of the masked tokens and the semantic information of the unmasked features to reconstruct the missing feature details; Step 5: Build a multimodal cancer survival prognosis prediction model. After training, import all the characteristic information of the patient into the prediction model to finally obtain the patient's survival prediction; In step 5, the constructed multimodal cancer survival prognosis prediction model includes a modal feature mixing module, which includes two layers of perceptrons and a normalization layer. Each perceptron contains two fully connected layers to mix information from different modalities and edge information of channels to promote comprehensive interaction between multiple modalities; then the interacted features are separated again and combined with the previous features to obtain the final modal output.

2. The method for predicting survival prognosis of patients with various cancers according to claim 1, characterized in that: The method for predicting the survival prognosis of patients with various cancers is applicable to predicting the survival prognosis of patients diagnosed with bladder cancer, breast cancer, lung adenocarcinoma and endometrial cancer.

3. The method for predicting survival prognosis of patients with various cancers according to claim 1, characterized in that: The step 2.1 includes the following specific steps: First, the pathological images were segmented into patches of 256×256, 512×512, and 1024×1024, i.e., small, medium, and large fields of view, and then the background was removed to keep the patches containing image information; Each patch embedding obtained is unified into a 1024-dimensional vector representation, and these vectorized feature representations are connected in groups of 8 adjacent features to obtain a graphical representation of pathological images at different scales, which is G P ={v,e}, where v and e represent the set of points and the set of edges in the graph.

4. The method for predicting survival prognosis of patients with various cancers according to claim 1, characterized in that: The hyperparameters in the multimodal cancer survival prognosis prediction model are set as follows: learning rate is 0.005, batch size is 50, optimizer is Adam, epoch is 40, and Dropout is 0.2.

Images