Transform-based multi-modal cancer prognosis evaluation method

Abstract

The invention aims to provide a multi-modal cancer prognosis evaluation method based on Transform, which comprises the following steps: S1, pre-processing multi-modal data, carrying out foreground segmentation and patch segmentation on pathology to obtain patch features, and clustering to obtain a phenotype cluster expressed by morphological specificity; performing expression significance analysis on the multi-omics data, and screening significant expression features; s2, single-mode modeling is carried out, strong pathological representation is obtained based on Transform and a two-stage attention convergence layer, and multi-omics modeling is realized based on a self-normalization network (SNN); s3, cross-modal interaction is carried out, and dense omics-phenotype cluster common attention mapping is realized by using a cross-modal Transform; s4, performing multi-modal fusion, and integrating the cross-modal interaction feature, each single-modal feature and clinical information by using a unified Transform coding layer to obtain a final multi-modal representation; and S5, risk prediction: inputting the final multi-modal representation into a risk prediction module to obtain a risk prediction result. Pathology and multi-omics characteristics are effectively fused, and clinical decision making is promoted.

Description

Technical field:

[0001] The present invention relates to a multimodal deep learning technology, and in particular to a cancer prognosis assessment method based on multimodal data fusion Background technology:

[0002] Prognostic analysis provides a basis for doctors to develop personalized treatment plans by evaluating the possible evolution of a patient's future health status. It can also identify high-risk patients and conduct early intervention to reduce the probability of disease progression. Compared with classification, survival analysis is much more complicated because risks are usually reflected in a series of histological patterns corresponding to different degrees of disease progression. In addition, factors such as the ultra-high resolution of histopathological images, high data complexity, and weak supervisory signals bring greater challenges to prognostic analysis.

[0003] Tumor heterogeneity is an issue that cannot be ignored in cancer research. Histopathological images can present tumor growth and morphology at extremely detailed billion-pixel resolution, while also showing highly complex structural and morphological features. The mixed tissue structures and cell populations within them strengthen the intratumor heterogeneity, posing a huge challenge to modeling. Most existing methods are based on the MIL framework, usually using local ROI features or directly using all patch features. They have limitations in considering patch correlation and reducing intra-model heterogeneity, and are unable to capture complex histological patterns associated with cancer progression.

[0004] Cancer is usually heterogeneous in etiology, and multi-perspective diagnosis can help improve the reliability of prognostic assessment. Genomic analysis of tissue biopsies can provide quantitative information on genomic expression and changes, and histological sections provide rich phenotypic information. Single-modality technology cannot use this valuable information to complement each other. In addition, in the past, multimodal technologies mostly used late fusion, which could not capture the interpretable genotype-phenotype interactions in the tumor microenvironment. For a small number of technologies that use early fusion, they have limitations in interpretability and can usually only visualize high- and low-risk areas of pathological images. This obviously cannot fully tap the potential value of multimodal features. Summary of the invention:

[0005] The main purpose of the present invention is to overcome the limitations of the prior art and provide a Transformer-based multimodal cancer prognosis assessment method. For the first time, it attempts to incorporate the concept of phenotypic clusters into multimodal prognostic research to achieve reliable pathological modeling and achieve more fine-grained dense omics-phenotype cluster co-attention mapping, which can not only improve performance, but also help locate important patterns related to cancer prognosis in tissue pathology images and observe the tendency of each omics modality to focus on different phenotypic clusters. Compared with traditional cancer prognosis methods, it is fast and efficient, portable between different cancers, and highly interpretable. It greatly reduces the burden on pathologists and promotes clinical decision-making. The technical problem solved by the present invention is achieved by the following steps:

[0006] To achieve the above object, the present invention provides a Transformer-based multimodal cancer prognosis assessment method, comprising the following steps:

[0007] S1. Preprocessing of multimodal data: segmenting the pathology foreground to obtain ROI, which is then divided into patches to obtain patch features. Finally, clustering is performed to obtain phenotypic clusters with morphological specificity, and expression significance analysis is performed on multi-omics data to screen for significant expression features.

[0008] S2, unimodal modeling, based on Transformer and two-stage attention aggregation layer to obtain powerful pathological representation, and multi-omics modeling based on self-normalizing network (SNN);

[0009] S3, cross-modal interaction, using cross-modal Transformer to achieve dense omics-phenotype cluster co-attention mapping;

[0010] S4, multimodal fusion, using a unified Transformer encoding layer to integrate cross-modal interaction features, single modality features and clinical information to obtain the final multimodal representation, achieving sufficient fine-grained fusion;

[0011] S5. Risk prediction: input the final multimodal representation into the risk prediction module to obtain the final risk prediction result.

[0012] The above-mentioned Transformer-based multimodal cancer prognosis assessment method specifically includes the following steps in step S1:

[0013] S10. Preprocessing of pathological tissue image foreground segmentation

[0014] The original pathological images were downsampled 32 times to obtain a low-resolution thumbnail that was easier to process. The thumbnail was then converted from the RGB color space to the HSV color space; the HSV color image was subjected to median blur on the saturation channel to smooth the image edges. Subsequently, the saturation channel of the smoothed image was thresholded to obtain a binary mask of the tissue area, and a morphological closing operation was used to fill the holes and gaps in the binary mask. The connected components with smaller areas in the binary mask were discarded to obtain the final mask of the tissue area.

[0015] S11. Pathological tissue image feature extraction and clustering preprocessing

[0016] The extracted tissue foreground is divided into N non-overlapping 256×256 patches, and the ResNet50 pre-trained on ImageNet is used as the feature extractor to extract a feature vector of length 1024 for each image. Finally, each pathological image is converted into a vector representation of N×1024.

[0017] Pathological images present tumor growth and morphology in extremely detailed gigapixel resolution, reflecting the underlying molecular processes and disease progression. The phenotype of pathological sections is a combination of various observable characteristics of tissues, providing a convenient visual representation of disease aggressiveness. We use K-means to cluster patch features and then construct phenotypic clusters, with different clusters representing different tissue types with morphologically specific characteristics.

[0018] S12. Multi-omics data preprocessing

[0019] Since most features in high-dimensional omics data modes are rarely or not expressed, the features in each modality are ranked according to the variance of the patient, and then the upper quartile is taken as the screening threshold, and the features below the threshold are discarded. The continuous feature values ​​in gene expression (mRNA), microRNA (miRNA), and DNA methylation (DNAm) are scaled to between 0 and 1 for input into the model for learning. Finally, mRNA contains 1,000 features, miRNA contains 1,881 features, and DNAm contains 5,000 features.

[0020] S2 specifically includes the following steps:

[0021] S20, capturing intra-cluster correlation

[0022] Transformer is used for each phenotypic cluster to capture the correlation between patches within the cluster, which can not only eliminate the heterogeneity within the cluster but also obtain a more robust patch representation. in Indicates the nth cluster, containing m ipatches, n represents the number of clusters. For a specific cluster, the internal feature extraction process is expressed as:

[0023]

[0024]

[0025] Among them, MSA means multi-head attention, LayerNorm means normalizing the output of each layer of the network, and the output is the representation of this phenotype cluster.

[0026] S21. Phenotypic clusters based on two-stage attention aggregation

[0027] For tissue pathology images, following the MIL framework, a two-stage attention aggregation module is introduced at this stage. The principle of each attention aggregation module is the same, and all use the attention-based MLP pooling layer for trainable weighted aggregation. Compared with the maximum pooling operator and the average pooling operator, it has better interpretability and performance. The first attention module realizes aggregation within each phenotypic cluster, and the second attention module realizes aggregation between phenotypic clusters. The specific description can be described using the following paradigm:

[0028]

[0029] in

[0030]

[0031] Where P wsi Represents the overall representation of the final tissue pathology image

[0032] S22. Modeling multi-omics data based on self-normalizing networks

[0033] The self-normalizing network (SNN) is introduced to model multi-omics data. The network is not prone to overfitting in high-dimensional and low-sample scenarios, and can realize deep semantic information mining of each omics modality. It is implemented using two fully connected layers, followed by ELU activation and Alpha Dropout, and finally the multi-omics data is mapped to P o ∈R k×256 A vector representation of , where k is the number of omics modalities, here k = 3.

[0034] S3 specifically is:

[0035] The introduction of cross-modal Transformer realizes dense co-attention mapping between phenotypic clusters and multi-omics features in pathological images, which can not only bridge the heterogeneity between tissue pathological images and molecular data, but also capture the interpretable gene-phenotype cluster interactions in the tumor microenvironment at a finer granularity, and realize efficient and reliable information reduction and selection. The specific cross-modal Transformer is shown in the following formula:

[0036] Q=Linear(P o )=P o W Q

[0037]

[0038]

[0039] Where W Q ,W K ,W V is the weight matrix of the corresponding linear layer.

[0040]

[0041] V i '=S i ×V i i=1,2,....n

[0042]

[0043] Where V i '∈R k×256 represents the new representation of the i-th cluster after fusion guided by omics modality, P wo Represents a cross-modal interactive representation that aggregates omics modalities and tissue pathology modalities.

[0044] S4 specifically:

[0045] A unified Transformer encoding layer is used to encode cross-modal interaction features, single-modal features, and clinical information including age and gender. clinical Integrate to obtain the final multimodal representation, achieve sufficient fine-grained fusion, and define a learnable class label P cls To aggregate the input of the entire sequence, the unified Transformer encoding layer is expressed as:

[0046] P = concat(P cls ;P wsi ;P o ;P wo ;P clinical )

[0047] P'=MSA(LayerNorm(P))+P

[0049] P"=MLP(L(')+P'

[0050] The P in the final output P" cls ” as a multimodal representation of the whole.

[0051] S5 specifically:

[0052] A multi-layer perceptron was used as the downstream prediction module to explore the relationship between multimodal features and survival time. The resulting multimodal representation was input into the risk prediction module, and the predicted survival time was output. The patient's actual survival time was used as a supervisory signal to train the model. The L2 regularized average negative log-partial likelihood was used as the objective function to train the model, the back-propagation algorithm was used to optimize the model parameters, and the consistency index (c-index) was used to evaluate the model performance on the test set. Description of the drawings:

[0053] Figure 1 This is a flowchart of a Transformer-based multimodal cancer prognosis assessment method according to an embodiment of the present invention;

[0054] Figure 2 A schematic diagram of a method for evaluating multimodal cancer prognosis based on Transformer according to an embodiment of the present invention;

[0055] Figure 3 The present invention is a Transformer-based multimodal cancer prognosis evaluation method in the risk area heat map and important pattern evaluation heat map of three cancer datasets (selecting one patient each), BLCA, BRCA, and UCEC;

[0056] Figure 4 This is a heat map of the attention patterns of omics patterns on tissue pathology images in three cancer datasets (one patient each) of BLCA, BRCA, and UCEC in a Transformer-based multimodal cancer prognosis assessment method according to an embodiment of the present invention;

[0057] Figure 5 A schematic diagram of multimodal data fusion of a Transformer-based multimodal cancer prognosis evaluation method according to an embodiment of the present invention; Specific implementation method:

[0058] The present invention will be further described below in conjunction with the accompanying drawings. It should be noted that this embodiment is based on the technical solution and provides a detailed implementation method and specific operation process, but the protection scope of the present invention is not limited to this embodiment.

[0059] Figure 1 This is a flowchart of a Transformer-based multimodal cancer prognosis assessment method according to an embodiment of the present invention; Figure 2 A schematic diagram of a method for evaluating multimodal cancer prognosis based on Transformer according to an embodiment of the present invention; Figure 3 The present invention is a Transformer-based multimodal cancer prognosis evaluation method in the risk area heat map and important pattern evaluation heat map of three cancer datasets (selecting one patient each), BLCA, BRCA, and UCEC; Figure 4 This is a heat map of the attention patterns of omics patterns on tissue pathology images in three cancer datasets (one patient each) of BLCA, BRCA, and UCEC in a Transformer-based multimodal cancer prognosis assessment method according to an embodiment of the present invention; Figure 5 The present invention is a schematic diagram of multimodal data fusion of a Transformer-based multimodal cancer prognosis evaluation method according to an embodiment of the present invention; the invention comprises the following steps:

[0060] S1. Preprocessing of multimodal data: segmenting the pathology foreground to obtain ROI, which is then divided into patches to obtain patch features. Finally, clustering is performed to obtain phenotypic clusters with morphological specificity, and expression significance analysis is performed on multi-omics data to screen for significant expression features.

[0061] S2, unimodal modeling, based on Transformer and two-stage attention aggregation layer to obtain powerful pathological representation, and multi-omics modeling based on self-normalizing network (SNN);

[0062] S3, cross-modal interaction, using cross-modal Transformer to achieve dense omics-phenotype cluster co-attention mapping;

[0063] S4, multimodal fusion, using a unified Transformer encoding layer to integrate cross-modal interaction features, single modality features and clinical information to obtain the final multimodal representation, achieving sufficient fine-grained fusion;

[0064] S5. Risk prediction: input the final multimodal representation into the risk prediction module to obtain the final risk prediction result.

[0065] The above-mentioned Transformer-based multimodal cancer prognosis assessment method specifically includes the following steps in step S1:

[0066] S10. Preprocessing of pathological tissue image foreground segmentation

[0067] The original pathological images were downsampled 32 times to obtain a low-resolution thumbnail that was easier to process. The thumbnail was then converted from the RGB color space to the HSV color space; the HSV color image was subjected to median blur on the saturation channel to smooth the image edges. Subsequently, the saturation channel of the smoothed image was thresholded to obtain a binary mask of the tissue area, and a morphological closing operation was used to fill the holes and gaps in the binary mask. The connected components with smaller areas in the binary mask were discarded to obtain the final mask of the tissue area.

[0068] S11. Pathological tissue image feature extraction and clustering preprocessing

[0069] The extracted tissue foreground is divided into N non-overlapping 256×256 patches, and the ResNet50 pre-trained on ImageNet is used as the feature extractor to extract a feature vector of length 1024 for each image. Finally, each pathological image is converted into a vector representation of N×1024.

[0070] Pathological images present tumor growth and morphology in extremely detailed billion-pixel resolution, reflecting the underlying molecular processes and disease progression. The phenotype of pathological sections is a combination of various observable characteristics of the tissue. This provides a convenient visual representation of the aggressiveness of the disease. Traditional modeling of tissue pathology images using ROI and all patches directly has a certain effect in capturing the heterogeneity between different tumors and tumor subtypes, but has obvious limitations in capturing histological patterns related to cancer progression. We use K-means to cluster patch features and then construct phenotypic clusters. Different clusters represent different tissue types with morphological specific characteristics. This modeling at the granularity of tissue type and cell type can effectively make up for this limitation.

[0071] S12. Multi-omics data preprocessing

[0072] Since most features in high-dimensional omics data modes are rarely or not expressed, the features in each modality are ranked according to the variance of the patient, and then the upper quartile is taken as the screening threshold, and the features below the threshold are discarded. The continuous feature values ​​in gene expression (mRNA), microRNA (miRNA), and DNA methylation (DNAm) are scaled to between 0 and 1 for input into the model for learning. Finally, mRNA contains 1,000 features, miRNA contains 1,881 features, and DNAm contains 5,000 features.

[0073] S2 specifically includes the following steps:

[0074] S20, capturing intra-cluster correlation

[0075] Transformer is used for each phenotypic cluster to capture the correlation between patches within the cluster, which can not only eliminate the heterogeneity within the cluster but also obtain a more robust patch representation. in Indicates the nth cluster, containing m i patches, n represents the number of clusters. For a specific cluster, the internal feature extraction process is expressed as:

[0076]

[0077]

[0078] Among them, MSA means multi-head attention, LayerNorm means normalizing the output of each layer of the network, and the output is the representation of this phenotype cluster.

[0079] S21. Phenotypic clusters based on two-stage attention aggregation

[0080] For tissue pathology images, following the MIL framework, a two-stage attention aggregation module is introduced at this stage. The principle of each attention aggregation module is the same, and all use the attention-based MLP pooling layer for trainable weighted aggregation. Compared with the maximum pooling operator and the average pooling operator, it has better interpretability and performance. The first attention module realizes aggregation within each phenotypic cluster, and the second attention module realizes aggregation between phenotypic clusters. The specific description can be described using the following paradigm:

[0081]

[0082] in

[0083]

[0084] Where P wsi Represents the overall representation of the final tissue pathology image.

[0085] for Figure 3 The visual effect is to normalize each patch obtained in the first attention stage to between 0 and 1, and then superimpose the scores on the tissue pathology image according to the corresponding positions, so as to obtain a heat map of high- and low-risk areas of the tissue pathology image, providing a direct visual assessment for the clinician. The second attention stage obtains the score of each phenotypic cluster, and obtains a heat map of the importance of phenotypic clusters, which directly locates high-risk cell types and histological patterns.

[0086] S22. Modeling multi-omics data based on self-normalizing networks

[0087] The self-normalizing network (SNN) is introduced to model multi-omics data. The network is not prone to overfitting in high-dimensional and low-sample scenarios, and can realize deep semantic information mining of each omics modality. It is implemented using two fully connected layers, followed by ELU activation and Alpha Dropout, and finally the multi-omics data is mapped to P o ∈R k×256 A vector representation of , where k is the number of omics modalities, here k = 3.

[0088] S3 reference Figure 2 and Figure 4 Specifically:

[0089] The cross-modal Transformer is introduced, using omics features as queries and histopathological image features as keys and values ​​to achieve dense co-attention mapping between phenotypic clusters in pathological images and multi-omics features. This can not only bridge the heterogeneity between histopathological images and molecular data, but also capture more fine-grained interpretable gene-phenotype cluster interactions in tumor microenvironments, and achieve efficient and reliable information reduction and selection. The specific cross-modal Transformer is shown in the following formula:

[0090] Q=Linear(P o )=P o W Q

[0091]

[0092]

[0093] Where W Q ,W K ,W V is the weight matrix of the corresponding linear layer.

[0094]

[0095] V i '=S i ×V i i=1,2,....n

[0096]

[0097] Where V i '∈R k×256 represents the new representation of the i-th cluster after fusion guided by omics modality, P wo represents a cross-modal interactive representation that aggregates omics modalities and tissue pathology modalities,

[0098] for Figure 4 The visual effect is to normalize the attention scores of the omics-phenotype clusters to between 0 and 1, and then superimpose the scores on the tissue pathology image according to the corresponding positions to obtain a heat map. In addition, the attention heat map of omics and phenotype clusters can reveal the potential relationship between molecular features and morphological features, thereby providing valuable knowledge for diagnosis and prognosis.

[0099] S4 specifically:

[0100] A unified Transformer encoding layer is used to encode cross-modal interaction features, single-modal features, and clinical information including age and gender. clinical Integrate to obtain the final multimodal representation, achieve sufficient fine-grained fusion, and define a learnable class label P cls To aggregate the input of the entire sequence, the unified Transformer encoding layer is expressed as:

[0101] P = concat(P cls ;P wsi ;P o ;P wo ;P clinical )

[0102] P'=MSA(LayerNorm(P))+P

[0104] P"=MLP(L(')+P'

[0105] The P in the final output P" cls ” as a multimodal representation of the whole.

[0106] S5 specifically:

[0107] A multi-layer perceptron was used as the downstream prediction module to explore the relationship between multimodal features and survival time. The resulting multimodal representation was input into the risk prediction module, and the predicted survival time was output. The patient's actual survival time was used as a supervisory signal to train the model. The L2 regularized average negative log-partial likelihood was used as the objective function to train the model, the back-propagation algorithm was used to optimize the model parameters, and the consistency index (c-index) was used to evaluate the model performance on the test set. The selected training scheme and hyperparameter settings are as follows:

[0108] 1) Implemented using the Pytorch framework;

[0109] 2) Use the Adam optimizer to update model parameters;

[0110] 3) The learning rate is set to 2×10 -4,, use weight decay, the decay rate is 1×10 -5 ;

[0111] 4) Considering that the number of patches divided from each patient ranges from thousands to tens of thousands, a batch size of 1 and a gradient accumulation strategy of 32 are used;

[0112] 5) The epoch is set to 100, and the early stopping mechanism is used to avoid overfitting;

[0113] 6) Add L2 regularization to the loss function to avoid overfitting;

[0114] 7) Five-fold cross validation was used to obtain a more reliable performance evaluation.

Claims

1. A Transformer-based multimodal cancer prognosis assessment method, characterized in that: The following steps are involved: S1. Preprocessing of multimodal data: segmenting the pathology foreground to obtain ROI, which is then divided into patches to obtain patch features. Finally, clustering is performed to obtain phenotypic clusters with morphological specificity, and expression significance analysis is performed on multi-omics data to screen for significant expression features. S2, unimodal modeling, based on Transformer and two-stage attention aggregation layer to obtain powerful pathological representation, and multi-omics modeling based on self-normalizing network (SNN); S3, cross-modal interaction, using cross-modal Transformer to achieve dense omics-phenotype cluster co-attention mapping; S4, multimodal fusion, using a unified Transformer encoding layer to integrate cross-modal interaction features, single modality features and clinical information to obtain the final multimodal representation, achieving sufficient fine-grained fusion; S5. Risk prediction: input the final multimodal representation into the risk prediction module to obtain the final risk prediction result.

2. According to claim 1, a Transformer-based multimodal cancer prognosis assessment method is characterized in that: The step S1 specifically includes: S10 downsamples the acquired original pathological image by 32 times to obtain a thumbnail. Then the thumbnail is converted from RGB color space to HSV color space; A median blur is applied on the saturation channel to smooth the image edges. Then, threshold processing is performed to obtain a binary mask of the tissue area, and a morphological closing operation is used to fill small holes and gaps in the binary mask. The connected components with smaller areas in the binary mask are discarded to obtain the final mask of the tissue area. S11 divides the extracted tissue foreground into N non-overlapping 256×256 patches and uses the ResNet50 pre-trained on ImageNet. A feature vector of length 1024 was extracted for each image, and each pathological image was finally converted into a vector representation of N × 1024. K-means was used to cluster the patch features and then construct phenotypic clusters with morphological specific characteristics. S12 ranks the features in each omics modality according to the variance of the patient, then takes the upper quartile as the screening threshold, discards the features below the threshold, and finally scales them to between 0 and 1. The final mRNA contains 1000 features, miRNA contains 1881 features, and DNAm Contains 5000 features.

3. A Transformer-based multimodal cancer prognosis assessment method according to claim 1, characterized in that In, S2 The specific steps include: S20, capturing intra-cluster correlation Transformer is used for each phenotypic cluster to capture the correlation between patches within the cluster, which can not only eliminate the heterogeneity within the cluster but also obtain a more robust patch representation. in Indicates the nth cluster, containing m i patches, n represents the number of clusters. For a specific cluster, the internal feature extraction process is expressed as: Among them, MSA means multi-head attention, LayerNorm means normalizing the output of each layer of the network, and the output is the representation of this phenotype cluster. S21. Phenotypic clusters based on two-stage attention aggregation For tissue pathology images, following the MIL framework, a two-stage attention aggregation module is introduced at this stage. The principle of each attention aggregation module is the same, and all use the attention-based MLP pooling layer for trainable weighted aggregation. The first attention module realizes aggregation within each phenotypic cluster, and the second attention module realizes aggregation between phenotypic clusters. The specific description can be used as follows: in Where P wsi Represents the overall representation of the final tissue pathology image. S22. Modeling multi-omics data based on self-normalizing networks The self-normalizing network (SNN) is introduced to model multi-omics data. The network is not prone to overfitting in high-dimensional and low-sample scenarios, and can realize deep semantic information mining of each omics modality. It is implemented using two fully connected layers, followed by ELU activation and Alpha Dropout, and finally the multi-omics data is mapped to P o ∈R k×256 A vector representation of , where k is the number of omics modalities, here k = 3.

4. The Transformer-based multimodal cancer prognosis assessment method according to claim 1, characterized in that: The S3 Specifically: The cross-modal Transformer is introduced to achieve dense co-attention mapping between phenotypic clusters and multi-omics features in pathological images, bridge the heterogeneity between tissue pathology images and molecular data, and capture more fine-grained interpretable gene-phenotype cluster interactions in the tumor microenvironment; the specific cross-modal Transformer is shown in the following formula: Q=Linear(P o )=P o W Q Where W Q ,W K ,W V is the weight matrix of the corresponding linear layer. V i '=S i ×V i i=1,2,....n Where V i '∈R k×256 represents the new representation of the i-th cluster after fusion guided by omics modality, P wo Represents a cross-modal interactive representation that aggregates omics modalities and tissue pathology modalities.

5. The Transformer-based multimodal cancer prognosis assessment method according to claim 1, characterized in that: The S4 Specifically: A unified Transformer encoding layer is used to encode cross-modal interaction features, single-modal features, and clinical information including age and gender. clinical Integrate to obtain the final multimodal representation, achieve sufficient fine-grained fusion, and define a learnable class label P cls To aggregate the input of the entire sequence, the unified Transformer encoding layer is expressed as: P=concat(P cls ;P wsi ;P o ;P wo ;P clinical ) P'=MSA(LayerNorm(P))+P P"=MLP(L(')+P' The P in the final output P" cls ” as a multimodal representation of the whole.

6. The Transformer-based multimodal cancer prognosis assessment method according to claim 1, characterized in that: The S5 Specifically: A multi-layer perceptron was used as the downstream prediction module to explore the relationship between multimodal features and survival time. The resulting multimodal representation was input into the risk prediction module, and the predicted survival time was output. The patient's actual survival time was used as a supervisory signal to train the model. The L2 regularized average negative log-partial likelihood was used as the objective function to train the model, the back-propagation algorithm was used to optimize the model parameters, and the consistency index (c-index) was used to evaluate the model performance on the test set.

7. The Transformer-based multimodal cancer prognosis assessment method according to claim 1, characterized in that: The Transformer-based multimodal cancer prognosis assessment method described in any one of claims 1-6 includes a multimodal data preprocessing module, a single modal modeling module, a cross-modal interaction module, a multimodal fusion module, and a risk prediction module.

Images