A gastric cancer prognosis detection method and system based on a Transformer deep learning

Abstract

This invention provides a gastric cancer prognostic detection method based on Transformer deep learning. The method constructs a multimodal heterogeneous dataset of gastric cancer containing pathological images, transcriptomic data, and clinical follow-up data. Through prior-guided collaborative preprocessing, a pathological ROI dataset, a prognostic gene expression matrix, and a structured clinical time-series feature set are obtained. A three-branch customized Transformer network extracts deep features from the three modalities, and multimodal fusion features are obtained through modality alignment and adaptive bidirectional cross-attention fusion. A multi-task joint prognostic prediction head is used to calculate a gastric cancer-specific comprehensive prognostic index, complete prognostic risk grading, and generate an interpretable attribution report. This method solves the problems of poor generalization, insufficient clinical adaptability, and lack of interpretability in existing technologies, significantly improving the accuracy, generalization, and practicality of gastric cancer prognostic detection.

Description

Technical Field

[0001] This invention relates to the field of precision prognostic detection technology for tumors, and in particular to a method and system for prognostic detection of gastric cancer based on Transformer deep learning. Background Technology

[0002] Gastric cancer is a prevalent malignant tumor of the digestive system worldwide. Postoperative gastric cancer patients face a high risk of recurrence and metastasis, and their prognosis varies greatly due to multiple factors, including tumor pathological heterogeneity, molecular biological characteristics, and differences in clinical phenotype. Currently, the gold standard for clinical prognostic assessment of gastric cancer is the TNM staging system. However, this system only achieves macroscopic risk stratification and cannot consider the synergistic influence of tumor morphology, molecular levels, and clinical follow-up information, making it difficult to achieve personalized and accurate prognostic assessment. With the rapid development of precision oncology, comprehensive prognostic detection combining pathological imaging, multi-omics data, and clinical information has become a core clinical need in the diagnosis and treatment of gastric cancer. Deep learning technology, especially the Transformer model, with its powerful long-sequence feature modeling, global feature extraction, and multimodal fusion capabilities, provides a new technical approach for the precise quantitative assessment of gastric cancer prognosis.

[0003] Currently, deep learning-based prognostic technologies for gastric cancer have formed several mainstream technical approaches. Early studies mostly focused on prognostic analysis of single-modality data, such as using convolutional neural networks to extract morphological features from whole-section images of gastric cancer pathology to construct overall survival prediction models; or using transcriptome sequencing data to screen prognostic-related genes and construct molecular prognostic feature models. However, these methods can only utilize single-dimensional biological information and are difficult to comprehensively cover the multidimensional influencing factors of gastric cancer prognosis, resulting in significant limitations in prediction accuracy and generalization. Some solutions introduce the Transformer architecture to model long sequence features; however, they still have many core shortcomings and cannot meet the application needs of real-world clinical scenarios. First, the data preprocessing stage lacks gastric cancer-specific prior guidance. Existing solutions often use random block processing for pathological images without specifically screening ROI regions strongly correlated with gastric cancer prognosis, such as the tumor invasion front, resulting in a large amount of redundant noise in the input data, which directly affects the effectiveness of subsequent model feature extraction. Second, existing solutions often directly apply the general Transformer structure without designing dedicated feature extraction branches for different modal characteristics or incorporating gastric cancer prognostic priors to constrain the attention mechanism. This makes it easy to learn redundant features unrelated to prognosis and fail to accurately capture the core biological features related to gastric cancer prognosis. Third, existing solutions often use simple feature splicing or fixed-weight fusion, which cannot solve the heterogeneity problem of different modal data, nor can they dynamically adjust the fusion weights according to the quality of single-modal data, resulting in insufficient robustness. Summary of the Invention

[0004] The purpose of this invention is to provide a gastric cancer prognostic detection method and system based on Transformer deep learning. This method constructs a gastric cancer-specific comprehensive prognostic index and improves the accuracy, generalization and practicality of gastric cancer prognostic detection by using three-branch customized Transformer deep feature extraction, adaptive cross-modal fusion and multi-task joint prognostic prediction.

[0005] To achieve the above objectives, the present invention provides the following solution: A gastric cancer prognostic detection method based on Transformer deep learning includes the following steps: A multimodal heterogeneous dataset of gastric cancer was constructed, and prior-guided co-processing was performed on whole-slice pathological images, transcriptome sequencing data and clinical follow-up data in the multimodal heterogeneous dataset of gastric cancer to obtain a pathological ROI dataset, a prognostic gene expression matrix and a structured clinical time series feature set. A three-branch Transformer feature extraction network was constructed, and the pathological ROI dataset, prognosis-related gene expression matrix, and structured clinical time-series feature set were respectively input into different branches of the three-branch Transformer feature extraction network for deep feature extraction, resulting in pathological morphological deep features, transcriptomic molecular deep features, and clinical time-series deep features; Modality alignment projection, adaptive bidirectional cross-attention fusion, and dynamic adjustment of modality weights were applied to pathological morphological depth features, transcriptomic molecular depth features, and clinical temporal depth features to obtain multimodal fused depth features. The multi-modal fusion deep features are predicted and analyzed by a multi-task joint prognostic prediction head, and the gastric cancer-specific comprehensive prognostic index is calculated based on the results of overall survival (OS) risk prediction, disease-free survival (DFS) risk prediction, and postoperative recurrence and metastasis binary classification prediction. Prognostic risk stratification was performed based on the gastric cancer-specific comprehensive prognostic index, and a prognostic attribution report was generated.

[0006] Optionally, a multimodal heterogeneous dataset of gastric cancer is constructed, and prior-guided co-processing is performed on whole-slice pathological images, transcriptome sequencing data, and clinical follow-up data in the multimodal heterogeneous dataset of gastric cancer to obtain a pathological ROI dataset, a prognostic gene expression matrix, and a structured clinical time-series feature set, including: Pixel-level semantic segmentation was performed on the tumor core area, tumor invasion front area and stroma area in the whole slice pathological image. Background and irrelevant tissue areas were removed to obtain the ROI area. After obtaining the ROI area, non-overlapping blocks were formed to generate pathological blocks. The ROI area labels were marked in the pathological blocks to obtain the pathological ROI dataset. LASSO-Cox regression analysis was performed on transcriptome sequencing data to screen for the core gene set of gastric cancer prognosis. The gene expression data in the core gene set of gastric cancer prognosis were then subjected to multicenter batch effect elimination using the ComBat-seq function to obtain the prognosis-related gene expression matrix. The clinical follow-up data were subjected to ordered coding and one-hot coding to obtain a structured clinical time series feature set.

[0007] Optionally, features are extracted from the pathological ROI dataset using the pathological image branch of the three-branch Transformer feature extraction network, including: Linear projection is performed on the pathological ROI dataset to obtain block embedding vectors, which are then superimposed with ROI region-aware location codes; the ROI region-aware location codes are a weighted sum of absolute location codes and ROI region label codes. The block embedding vector is input into the first Transformer encoder to constrain the attention weights of the block embedding vector through a gastric cancer prognostic prior attention mask; The output features of different depth layers in the first Transformer encoder are aggregated at multiple scales, and then pathological morphological depth features are obtained through global average pooling and linear projection.

[0008] Optionally, features of the prognosis-related gene expression matrix are extracted using the transcriptome branch of a three-branch Transformer feature extraction network, including: Based on the prior knowledge of gastric cancer oncogenic pathways, the prognostic gene expression matrix was divided into multiple pathway genomes, and the expression value of each gene was linearly projected into a gene token embedding vector. The gene token embedding vector is input into the second Transformer encoder. The second Transformer encoder includes a local branch and a global branch. The local branch is used to perform self-attention calculation on the gene token embedding vector within the same pathway, and the global branch is used to perform self-attention calculation on the gene token embedding vector of all pathways. The output features of local and global branches are concatenated, global average pooling is performed, and linear projection is applied to obtain the deep molecular features of the transcriptome.

[0009] Optionally, feature extraction is performed on the structured clinical time-series feature set through the clinical feature branch in the three-branch Transformer feature extraction network, including: The feature vectors of each follow-up time window in the structured clinical time-series feature set are linearly projected and superimposed with time-series position codes to obtain the clinical token embedding vector; The clinical token embedding vector is input into the third Transformer encoder to adjust the attention weights of gastric cancer prognostic strongly correlated features in the clinical token embedding vector through a clinical prior attention mask; the gastric cancer prognostic strongly correlated features include: TNM stage, tumor differentiation degree, lymph node metastasis status and vascular invasion status; Temporal global average pooling and linear projection are performed on the output features of the third Transformer encoder to obtain clinical temporal depth features.

[0010] Optionally, modality-aligned projection, adaptive bidirectional cross-attention fusion, and dynamic adjustment of modality weights are applied to pathological morphological depth features, transcriptomic molecular depth features, and clinical temporal depth features to obtain multimodal fused depth features, including: By using linear projection, the pathological morphological depth features, transcriptomic molecular depth features, and clinical temporal depth features are mapped to a common feature space of the same dimension, resulting in pathological features, transcriptomic features, and clinical features. By using pathological features as queries and transcriptomic features as keys and values, cross-attention calculation is performed to obtain the first attention feature. By using transcriptome features as queries and pathological features as keys and values, cross-attention calculation is performed to obtain the second attention feature. The first attention feature and the second attention feature are concatenated and linearly projected to obtain the dual-modal fusion feature; The quality confidence scores of pathological features, transcriptomic features, and clinical features were calculated based on the missing data rate, signal-to-noise ratio, and percentage of valid data, respectively. Using the dual-modal fusion feature as the query and the clinical feature as the key and value, cross-attention calculation is performed to obtain the third attention feature; By using clinical features as queries and bimodal fusion features as keys and values, cross-attention calculation is performed to obtain the fourth attention feature. Adaptive fusion weights are determined based on quality confidence to perform weighted fusion of the third and fourth attention features, resulting in multimodal fusion deep features.

[0011] Optionally, a multi-task joint prognostic prediction head is used to predict and analyze the multimodal fusion deep features, and a gastric cancer-specific comprehensive prognostic index is calculated based on the obtained overall survival (OS) risk prediction results, disease-free survival (DFS) risk prediction results, and postoperative recurrence and metastasis binary classification prediction results, including: The multimodal fusion deep features were predicted in parallel by using the OS risk prediction branch, DFS risk prediction branch and postoperative recurrence and metastasis binary prediction branch of the multi-task joint prognostic prediction head to obtain the OS risk ratio, DFS risk ratio and postoperative 5-year recurrence and metastasis probability. Calculate the risk collaborative correction coefficient based on the contribution vector of the core features in the output process of the three prediction branches; Based on the preset prognostic endpoint weight mapping rule, basic weights are assigned to the OS hazard ratio, DFS hazard ratio, and 5-year postoperative recurrence and metastasis probability, and the basic weights are corrected by the risk co-correction coefficient to obtain a dynamic weight set. The gastric cancer-specific comprehensive prognostic index is obtained by weighting and summing the OS hazard ratio, DFS hazard ratio, and 5-year recurrence and metastasis probability based on the dynamic weight set.

[0012] Optionally, prognostic risk stratification is performed based on a gastric cancer-specific comprehensive prognostic index, and a prognostic attribution report is generated, including: The optimal risk cutoff value was determined using the X-tile algorithm. Samples with a gastric cancer-specific comprehensive prognostic index greater than the optimal risk cutoff value were classified as high-prognostic-risk group, while samples with a gastric cancer-specific comprehensive prognostic index less than the optimal risk cutoff value were classified as low-prognostic-risk group. Attribution analysis of pathological, transcriptomic, and clinical features was performed using attention weighted backtracking and SAP values ​​to identify key influencing factors. A prognostic attribution report is generated based on the prognostic risk grouping results, key influencing factors, and a gastric cancer-specific comprehensive prognostic index.

[0013] Optionally, attribution analysis of pathological, transcriptomic, and clinical features is performed using attention weighted backtracking and SAP values ​​to identify key influencing factors, including: Attribution analysis of pathological features yielded the top-N pathological regions of interest (ROIs) that contributed most to prognostic outcomes. Attribution analysis of transcriptome features identified the top-N prognostic-related core genes that contributed the most to the prognostic results. Attribution analysis of clinical features revealed the top-N clinical features that contributed most to prognostic outcomes.

[0014] A gastric cancer prognostic detection system based on Transformer deep learning, comprising: The data preprocessing module is used to construct a multimodal heterogeneous dataset of gastric cancer and to perform prior-guided collaborative preprocessing on whole-slice pathological images, transcriptome sequencing data and clinical follow-up data in the multimodal heterogeneous dataset of gastric cancer to obtain a pathological ROI dataset, a prognosis-related gene expression matrix and a structured clinical time series feature set. The Transformer feature extraction module is used to construct a three-branch Transformer feature extraction network. The pathological ROI dataset, the prognosis-related gene expression matrix, and the structured clinical time-series feature set are respectively input into different branches of the three-branch Transformer feature extraction network for deep feature extraction, resulting in pathological morphological deep features, transcriptomic molecular deep features, and clinical time-series deep features. The multimodal fusion module is used to perform modal alignment projection, adaptive bidirectional cross-attention fusion, and dynamic adjustment of modal weights on pathological morphological deep features, transcriptomic molecular deep features, and clinical temporal deep features to obtain multimodal fused deep features. The prognostic index calculation module is used to predict and analyze multimodal fusion deep features through a multi-task joint prognostic prediction head, and calculate the gastric cancer-specific comprehensive prognostic index based on the results of overall survival (OS) risk prediction, disease-free survival (DFS) risk prediction, and postoperative recurrence and metastasis binary classification prediction. The prognostic grading and outcome visualization module is used to classify prognostic risks based on the gastric cancer-specific comprehensive prognostic index and generate a prognostic attribution report.

[0015] According to specific embodiments provided by the present invention, the following technical effects are disclosed: The present invention provides a gastric cancer prognostic detection method and system based on Transformer deep learning. The method includes: constructing a multimodal heterogeneous dataset of gastric cancer, and performing prior-guided collaborative preprocessing on whole-slice pathological images, transcriptome sequencing data, and clinical follow-up data in the multimodal heterogeneous dataset of gastric cancer to obtain a pathological ROI dataset, a prognostic gene expression matrix, and a structured clinical time-series feature set; constructing a three-branch Transformer feature extraction network, and inputting the pathological ROI dataset, the prognostic gene expression matrix, and the structured clinical time-series feature set into the three-branch Transformer feature extraction network respectively. Deep feature extraction is performed in different branches of the network to obtain pathological morphological deep features, transcriptomic molecular deep features, and clinical temporal deep features. Modality alignment projection, adaptive bidirectional cross-attention fusion, and dynamic adjustment of modality weights are then applied to these features to obtain multimodal fused deep features. These multimodal fused deep features are then used for predictive analysis via a multi-task joint prognostic prediction head. A gastric cancer-specific comprehensive prognostic index is calculated based on the predicted overall survival (OS), disease-free survival (DFS), and postoperative recurrence / metastasis. Prognostic risk is stratified based on the gastric cancer-specific comprehensive prognostic index, and a prognostic attribution report is generated. This method, through prior-guided multimodal heterogeneous data preprocessing, three-branch customized Transformer deep feature extraction, adaptive cross-modal fusion, and multi-task joint prognostic prediction, constructs a clinically adapted gastric cancer-specific comprehensive prognostic index and enables interpretable attribution of multidimensional prognostic influencing factors, improving the accuracy, generalization, and practicality of gastric cancer prognostic detection. Attached Figure Description

[0016] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0017] Figure 1 This is a flowchart of the gastric cancer prognostic detection method based on Transformer deep learning according to the present invention; Figure 2 This is a schematic diagram of the gastric cancer prognosis detection system based on Transformer deep learning according to the present invention. Detailed Implementation

[0018] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0019] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0020] like Figure 1 As shown, this invention provides a gastric cancer prognostic detection method based on Transformer deep learning, comprising the following steps: Step 100: Construct a multimodal heterogeneous dataset of gastric cancer, and perform prior-guided co-processing on whole-slice pathological images, transcriptome sequencing data and clinical follow-up data in the multimodal heterogeneous dataset of gastric cancer to obtain a pathological ROI dataset, a prognostic gene expression matrix and a structured clinical time series feature set. Step 200: Construct a three-branch Transformer feature extraction network, and input the pathological ROI dataset, prognosis-related gene expression matrix and structured clinical time series feature set into different branches of the three-branch Transformer feature extraction network for deep feature extraction to obtain pathological morphological deep features, transcriptomic molecular deep features and clinical time series deep features; Step 300: Modality alignment projection, adaptive bidirectional cross-attention fusion, and dynamic adjustment of modality weights are performed on pathological morphological deep features, transcriptomic molecular deep features, and clinical temporal deep features to obtain multimodal fused deep features; Step 400: Perform predictive analysis on multimodal fusion deep features using a multi-task joint prognostic prediction head, and calculate the gastric cancer-specific comprehensive prognostic index based on the obtained overall survival (OS) risk prediction results, disease-free survival (DFS) risk prediction results, and postoperative recurrence and metastasis binary classification prediction results. Step 500: Perform prognostic risk stratification based on the gastric cancer-specific comprehensive prognostic index and generate a prognostic attribution report.

[0021] In the specific implementation process, step 100 first sets strict inclusion and exclusion criteria for samples: included samples are patients with primary gastric cancer diagnosed by pathological biopsy, and excluded samples are those that received neoadjuvant chemoradiotherapy before surgery, those with other systemic malignancies, those with more than 30% missing clinical follow-up endpoint data, and those with mismatched pathological or transcriptomic data. The dataset covers samples from multiple clinical sources to ensure the cross-center generalization of the subsequent model, with a median follow-up time of no less than 36 months to ensure the completeness of prognostic endpoint data. The constructed dataset contains three core heterogeneous datasets with single-sample pairings: First, whole-slice pathological images (WSI) of gastric cancer tissue after surgery stained with hematoxylin and eosin (HE), with a scanning magnification of no less than 20×, a pixel accuracy of no less than 0.5 μm / pixel, and no obvious staining artifacts, blurring of focus, or other quality issues; second, transcriptome sequencing data of paired gastric cancer tumor tissues, which are raw gene counting matrices obtained from RNA-seq sequencing, with no less than 60M clean reads per sample, covering the entire coding region of human genes; and third, complete clinical follow-up data of patients, covering baseline clinical characteristics, tumor pathological characteristics, diagnosis and treatment information, and regular postoperative follow-up data, clearly recording the occurrence time and status of three prognostic endpoints: overall survival (OS), disease-free survival (DFS), and postoperative recurrence and metastasis events.

[0022] Then, guided by prior prognostic information about gastric cancer, pathological images are denoised, effective regions are selected, and standardized, outputting a pathological ROI dataset that can be directly input into the Transformer network. Specifically, the first step is to normalize the colors of the original WSI images by using the Macenko normalization algorithm, addressing color deviations caused by staining batches and scanning equipment. The original RGB image is first converted to optical density space, and the calculation formula is as follows: ,in, Let I be the optical density matrix, and let I be the RGB pixel value matrix of the original WSI image. The minimum constant (in this embodiment, it is taken as 1e) -6The process involves several steps. First, low-density background pixels are removed, a staining matrix is ​​calculated and mapped to a standard HE staining reference space to obtain a color-normalized WSI image. Then, Otsu's method is used to perform threshold segmentation on the normalized image to distinguish between tissue foreground and blank background, eliminating invalid slices where the background area accounts for more than 90%. Second, based on prior clinical knowledge of gastric cancer, three core ROI regions strongly correlated with gastric cancer prognosis are identified: the tumor core region, the tumor invasion front region, and the stromal region. The tumor invasion front region is defined as the tissue region extending 500 μm outward from the tumor core region. A pixel-level semantic segmentation network based on U-Net++ is constructed. Using masks of the three ROI regions double-blindly annotated by senior gastric cancer pathologists as the gold standard, the network is trained, validated, and tested, ensuring that the model's Intersection over Union (IoU) is no less than 0.85. The foreground tissue image is input into the trained segmentation network, which outputs pixel-level segmentation masks for three types of ROI regions. Simultaneously, irrelevant tissue regions such as necrotic tissue, normal mucosal tissue, and adipose tissue are automatically removed, retaining only the effective tissue masks for the three prognostic-related ROIs. In the third step, at a WSI magnification of 20×, the three types of ROI regions are divided into 256×256 pixel non-overlapping sliding window blocks based on the segmentation masks, generating single pathological patches. A second validity screening is performed on the generated pathological patches, removing invalid patches with less than 50% effective tissue, staining artifacts, or blurred focus. Simultaneously, each effective pathological patch is labeled with its corresponding ROI region label: 0 for the tumor core region, 1 for the tumor invasion front region, and 2 for the mesenchymal region. Finally, all effective pathological patches and their corresponding labels are integrated to construct the pathological ROI dataset.

[0023] Next, transcriptome data standardization, prognostic gene screening, and batch effect elimination were performed to output a prognostic gene expression matrix adapted to the Transformer network input. Specifically, in the first step, the raw RNA-seq sequencing data was processed using Trimmomatic software to remove sequencing adapter sequences, low-quality reads, and primer contamination sequences, resulting in clean reads. The clean reads were then aligned to the human reference genome GRCh38 version using STAR alignment software, and gene-level expression quantification was performed using HTSeq-count software to obtain the raw gene counting matrix. The raw counting matrix was then quality filtered to remove low-expression genes whose sum of expression counts was less than 10 across all samples and those expressed in less than 5% of the samples, retaining only effective protein-coding genes. Subsequently, TPM (Transcripts Per Million) was used for standardization to eliminate the influence of gene length and sequencing depth on expression levels. The second step involves constructing a LASSO-Cox proportional hazards regression model with patient overall survival (OS) endpoints (survival time, survival status) as the dependent variable and standardized gene expression values ​​as the independent variables. The optimal regularization parameter is determined through 10-fold cross-validation, and the minimum value of the optimal regularization parameter corresponding to the minimum cross-validation error is selected as the final parameter. Model training is then completed based on this parameter. LASSO penalties are used to compress the regression coefficients of genes not significantly related to gastric cancer prognosis to 0, retaining genes with non-zero regression coefficients to obtain a preliminary prognostic gene set. Univariate Cox regression analysis is performed on the initially screened genes to remove genes with statistically insignificant p-values ​​greater than 0.05, resulting in the core gene set for gastric cancer prognosis. The third step involves using the ComBat-seq method to correct for batch effects in the expression data of the core gene set for gastric cancer prognosis, based on multi-center transcriptome data. This method adapts the distribution characteristics of RNA-seq counting data to a negative binomial distribution model, eliminating technical biases caused by different sequencing centers, sequencing platforms, and sequencing batches while preserving inter-group biological differences. The corrected core gene expression data are integrated according to the sample dimension. Each row of the matrix corresponds to a patient sample, each column corresponds to a prognostic core gene, and the matrix elements are the corrected and standardized expression values ​​of the corresponding genes in the corresponding samples, thus constructing a prognostic related gene expression matrix.

[0024] Finally, the unstructured clinical follow-up data is cleaned, encoded, and transformed into a time-series structured dataset, outputting a structured clinical time-series feature set that can be directly input into the Transformer network. Specifically, the first step involves processing the original clinical follow-up data for missing and outlier values, removing feature fields with more than 30% missing values. For features with less than 30% missing values, categorical features are filled with the mode, and continuous features are filled with the median. Outliers in continuous features are also removed using the 3σ principle. Subsequently, the clinical data is dimensionally split into four main feature domains: baseline clinical features (including age, gender, BMI, etc.), tumor pathology features (including TNM stage, tumor differentiation, Lauren classification, lymph node metastasis status, vascular invasion status, and nerve invasion status), treatment features (including surgical procedures and postoperative adjuvant therapy plans), and time-series follow-up features (including postoperative re-examination indicators recorded at fixed time windows, recurrence and metastasis events, and survival status). The second step involves using 0-1 binary encoding for binary classification features (i.e., gender, vascular invasion status, neurological invasion status, and recurrence / metastasis status), assigning a value of 1 to positive risk status and 0 to negative risk status; for ordered classification features (i.e., TNM stage, tumor differentiation degree, etc., which have clear prognostic risk levels), ordinal encoding is used, assigning values ​​in ascending order of risk level; for unordered classification features (i.e., Lauren classification, surgical method, adjuvant therapy regimen, etc., which do not have a clear hierarchical order), one-hot encoding is used, converting the different categories of each feature into mutually independent binary vectors; and for continuous features (i.e., age, BMI, follow-up time), Z-score standardization is used to eliminate dimensional differences. The third step involves using a fixed 3-month time window for postoperative follow-up time series data. The encoded features, such as re-examination indicators and event status, within each window are integrated to generate a feature vector corresponding to each follow-up time window. The feature vectors of all time windows are arranged in chronological order of follow-up time, aligning the number of follow-up windows for all samples. For samples with insufficient follow-up time, zero vectors are used to fill the missing windows. Finally, a three-dimensional structured clinical time series feature set is constructed, with the row dimension representing patient samples, the column dimension representing follow-up time windows, and each window corresponding to a set of encoded features.

[0025] It should be noted that step 100 first achieves precise enrichment of prognostic information at the data source. Through a preprocessing workflow guided by prior clinical observations in gastric cancer, it accurately identifies prognostically relevant ROIs, such as the tumor invasion front zone, in the pathological image stage, eliminating over 90% of background noise and redundant information from irrelevant tissues. This addresses the core defects of feature redundancy and loss of core prognostic information caused by random segmentation. Secondly, it standardizes and homogenizes the three types of heterogeneous data. LASSO-Cox screening and ComBat-seq correction identify the core prognostic gene set for gastric cancer and eliminate multi-center batch effects. A differentiated coding strategy achieves the structured transformation of clinical data, resolving the issue of strong heterogeneity among pathological, transcriptomic, and clinical data, which cannot be directly adapted to deep learning model input. Finally, it lays a high-quality data foundation for subsequent model training. Strict sample inclusion and exclusion, quality control, and multi-center data coverage ensure the clinical representativeness and reliability of the input data, significantly reducing the risk of overfitting in subsequent models from a data perspective.

[0026] In the specific implementation process, feature extraction of the pathological ROI dataset is performed through the pathological image branch of the three-branch Transformer feature extraction network, specifically including: Each 256×256 pixel RGB pathological block in the pathological ROI dataset is divided into fixed-size sub-blocks. A 16×16 pixel non-overlapping sliding window is used to divide a single pathological block into sub-blocks, with each sub-block corresponding to a basic "token" for subsequent Transformer processing. Then, each 16×16×3 RGB sub-block is flattened into a one-dimensional vector, resulting in an original sub-block feature vector with a dimension of 768. , This is the index of the sub-block. Next, the original sub-block feature vectors are mapped to a predefined model dimension using a learnable linear projection matrix. This yields the block embedding vectors corresponding to the sub-blocks, completing the initial mapping from the original image pixels to the high-dimensional semantic feature space. The block embedding vectors of all sub-blocks together constitute the block embedding sequence.

[0027] The i-th sub-block is encoded using a standard sine-cosine positional encoding formula to generate an absolute positional code. Then, the ROI region label *l* of the current pathological block is obtained, where *l=0* corresponds to the tumor core region, *l=1* corresponds to the tumor invasion front region, and *l=2* corresponds to the stroma region. A learnable region label embedding layer maps label *l* to a dimension of... The vector is then used to weight and fuse the absolute position encoding and the ROI region label encoding to obtain the ROI region-aware position encoding. The weight coefficient is set to 0.7 to balance the contribution of absolute position information and ROI region prior information. Finally, the block embedding vector is added element by element to the ROI region-aware position encoding to obtain the final token sequence input to the first Transformer encoder.

[0028] The first Transformer encoder is constructed, which consists of N layers (6 layers in this embodiment) of identical encoder sublayers stacked together. Each encoder sublayer contains two core modules: a multi-head self-attention module and a feedforward neural network module. Both modules adopt a residual connection + layer normalization structure. A gastric cancer prognostic prior attention mask is introduced into the multi-head self-attention module. Based on prior clinical knowledge of gastric cancer, the tumor core area and the tumor invasion front area are key areas strongly correlated with prognosis, while the prognostic value of the mesenchymal area is relatively low. The matrix element M[i][j] (where i is the query sub-block index and j is the key sub-block index) of the mask matrix takes the following value: if the label l of the current pathological block is 0 or 1 (tumor core area or invasion front area), and the sub-block corresponding to j belongs to the mesenchymal area (determined by the sub-block's position in the original pathological block corresponding to the segmentation mask), then M[i][j] = -1e 9 Otherwise, M[i][j] = 0. During attention calculation, the mask matrix is ​​added to the input of the softmax function to suppress excessive attention to regions with low prognostic value. The feedforward neural network module consists of two linear layers and a GELU activation function, as shown in the formula: ,in These are the input features of the feedforward neural network. This is the first linear layer, used to map the input features to an intermediate dimension. Let Gaussian error be the activation function of the linear unit. The second linear layer maps intermediate-dimensional features back to the original dimensions. The input token sequence is fed into the stacked first Transformer encoder to obtain the output features of each layer. Finally, the output features from different depth layers in the first Transformer encoder are aggregated at multiple scales, and then global average pooling and linear projection are used to obtain the pathological morphological depth features.

[0029] It should be noted that by using ROI-aware location encoding, the spatial location information of sub-blocks and prior prognostic region information of gastric cancer are integrated. This not only solves the inherent defect of Transformer models lacking location awareness, but also guides the model to focus on prognostically relevant regions such as the tumor core area and the invasion front area from the input level. By constraining the multi-head self-attention mechanism through gastric cancer prognostic prior attention mask, the interference of low prognostic value regions such as the interstitial area on attention weights is actively suppressed, enabling the model to accurately capture core pathological features related to gastric cancer prognosis and avoid the problem of indiscriminate learning of redundant features by the general Transformer. Through a multi-scale feature aggregation strategy, low, medium and high-dimensional features from different depth layers of the Transformer encoder are integrated, comprehensively covering multi-scale pathological information from edge texture to tumor atypia. The final output pathological morphological depth features have high prognostic specificity and representativeness.

[0030] In the specific implementation process, the transcriptome branch of the three-branch Transformer feature extraction network is used to extract features from the prognostic gene expression matrix, specifically including: Based on the KEGG database and the prior knowledge of core oncogenic pathways in the TCGA gastric cancer study, a set of pathways strongly associated with the occurrence, development, and prognosis of gastric cancer was screened. These pathways include, but are not limited to, cell cycle pathways, PI3K-Akt signaling pathways, Wnt signaling pathways, MAPK signaling pathways, TGF-β signaling pathways, and cell adhesion pathways, totaling m core pathways. Each gene in the prognostic gene expression matrix was mapped to its corresponding core pathway, forming m pathway genomes. For prognostic core genes not explicitly mapped to known core pathways, a separate (m+1)th pathway genome was constructed to ensure that all prognostic core genes were included in the grouping. The normalized expression value of the g-th gene was mapped to a dimension of [dimensionality missing] using a learnable linear projection. Gene token embedding vectors are generated. Simultaneously, learnable pathway location codes are generated for each pathway genome to preserve prior information about the pathway to which the gene belongs. The gene token embedding vector is added element-wise to the corresponding pathway location code to obtain the final input token. Then, the input tokens of all genes are arranged in pathway order to form the input sequence of the transcriptome branch.

[0031] The local branches of the second Transformer encoder are constructed by stacking N identical encoder sublayers (6 layers in this embodiment). Each sublayer contains a multi-head self-attention module and a feedforward neural network module, both employing a residual connection + layer normalization structure. The local branches introduce intra-pathway attention masks, focusing only on gene interactions within the same pathway. The mask matrix elements follow the rule: if two genes belong to the same pathway genome, the matrix element is 0; otherwise, it is -1e. 9During attention calculation, a mask matrix is ​​added to the input of the softmax function to suppress ineffective attention between genes in different pathways. The outputs of multiple attention heads are concatenated and then passed through a linear projection to obtain the final output of the local branch multi-head self-attention. The feedforward neural network module of the local branch consists of two linear layers and a GELU activation function, as shown in the formula: .

[0032] Similarly, the global branch of the second Transformer encoder is constructed by stacking N layers (the same number as the local branch layers). Its structure is identical to the local branch, but it does not use intra-pathway attention masks, allowing the model to learn interactions between all genes. The global branch shares the same input sequence as the local branch, but generates its query, key, and value matrices through independent learnable linear projections. The feedforward neural network module of the global branch has the same structure as the local branch, but its parameters are independent. It inputs the input sequence into the stacked N-layer global branch encoder, ultimately obtaining the output features of the global branch. Finally, the output features of the local and global branches are concatenated, globally averaged, and linearly projected to obtain the deep molecular features of the transcriptome.

[0033] It should be noted that the pathway genome partitioning guided by prior knowledge of gastric cancer oncogenic pathways incorporates prior knowledge of gastric cancer biology at the input level, avoiding the shortcomings of general Transformer indiscriminate gene modeling and making the model more focused on known core regulatory pathways of gastric cancer. Through the dual-pathway design of local and global branches, the model captures the co-expression relationships of genes within the same pathway using local branches and the cross-regulatory relationships between different pathways using global branches, comprehensively covering molecular interaction information related to gastric cancer prognosis. Through the constraint of attention masking within pathways and the fusion of dual-pathway features, the final output transcriptome molecular depth features have high prognostic specificity, overcoming the limitations of existing technologies that only focus on single-pathway or global modeling.

[0034] In the specific implementation process, the structured clinical time-series feature set is extracted through the clinical feature branch of the three-branch Transformer feature extraction network, specifically including: For a single patient sample in a structured clinical time-series feature set, feature vectors from multiple follow-up time windows are extracted and linearly projected onto a predefined model dimension. The initial time window embedding vector is obtained. To preserve the temporal sequence information of clinical follow-up, the standard sine and cosine position coding formula is used to generate the temporal position code. The initial time window embedding vector and the temporal position code are added element by element to obtain the final clinical token embedding vector. Then, the clinical token embedding vectors of all time windows are arranged in chronological order to form the input sequence of the clinical feature branch.

[0035] A third Transformer encoder is constructed, consisting of N layers (6 layers in this embodiment) of identical encoder sublayers stacked together. Each encoder sublayer contains two core modules: a multi-head self-attention module and a feedforward neural network module, both of which adopt a residual connection + layer normalization structure. Based on prior clinical knowledge of gastric cancer, TNM staging, tumor differentiation degree, lymph node metastasis status, and vascular invasion status are identified as four features strongly correlated with gastric cancer prognosis. These features are mostly baseline clinical characteristics, recorded in all follow-up time windows or exhibiting time invariance. A mask matrix is ​​then constructed, where the matrix elements are set as follows: if all four strongly correlated features are present in the feature vector corresponding to the key time window, the matrix element is 0; if any one of the strongly correlated features is missing in the key time window, the matrix element is -1e. 9 Meanwhile, to further enhance the attention weights of strongly correlated features, a learnable enhancement coefficient is introduced, with an initial value of 1.2, which is dynamically updated as the model trains. This coefficient is used to amplify the weights of key time windows that are highly correlated with strongly correlated features after the initial attention weights are obtained.

[0036] In multi-head self-attention computation, the input sequence is first passed through three independent learnable linear projections to obtain the query matrix. Key matrix The value matrix is ​​then calculated. Subsequently, the initial attention weight matrix is ​​calculated, and a clinical prior attention mask is added. The calculation formula is as follows: ,in For the dimension of a single attention head, The softmax function is calculated row-wise. For each query time window, the top 30% of time windows with the highest contribution of strongly correlated features in the key time window are identified (determined by multiplying the encoded value of the strongly correlated feature in the key time window feature vector by its weight). The attention weights at these positions are multiplied by the enhancement coefficient and re-normalized to obtain the final attention weight matrix. The value matrix is ​​weighted and summed using the final attention weight matrix, and the outputs of multiple attention heads are concatenated and passed through a linear projection to obtain the output of the multi-head self-attention module. The feedforward neural network module consists of two linear layers and a GELU activation function. The input sequence is fed into a stacked N-layer third Transformer encoder to obtain the final encoder output features. Finally, temporal global average pooling and linear projection are performed on the output features of the third Transformer encoder to obtain the clinical temporal depth features.

[0037] It should be noted that by using temporal position encoding, the temporal sequence information of clinical follow-up is effectively preserved, solving the problem that the Transformer model itself lacks temporal awareness, enabling the model to capture the dynamic changes of patients' clinical characteristics over time. By constraining the clinical prior attention mask and adjusting the learnable enhancement coefficient, the model is actively guided to focus on four types of prognostic features of gastric cancer, such as TNM stage and tumor differentiation degree, suppressing the interference of irrelevant clinical features, enabling the model to accurately capture core clinical information related to gastric cancer prognosis. By integrating the information of all follow-up time windows through temporal global average pooling, the final output of clinical temporal deep features comprehensively covers the patient's baseline features and follow-up dynamics, and has high prognostic specificity and representativeness.

[0038] In the specific implementation process, the input of step 300 is the three types of deep features output by the three-branch Transformer network: pathological morphological deep features. Transcriptome molecular depth features Clinical time-series depth features By using three independent learnable linear projection layers, the three types of features are mapped to a unified common feature dimension, resulting in aligned pathological features. Transcriptome characteristics Clinical features .

[0039] Aligned pathological features Generate the query matrix using learnable linear projection. Aligned transcriptome features The key matrix is ​​generated by two independent learnable linear projections. Sum matrix The first attention feature is calculated using the standard cross-attention mechanism. The calculation formula is: , The dot product of the query matrix and the key matrix is ​​used to measure the correlation between pathological features and transcriptomic features. This is used to aggregate transcriptome feature information based on attention weights, ultimately resulting in... It integrates transcriptomic complementary information from a pathological perspective. Similarly, it integrates aligned transcriptomic features. Generate the query matrix using learnable linear projection. Aligned pathological features The key matrix is ​​generated by two independent learnable linear projections. Sum matrix The second attention feature is calculated using the standard cross-attention mechanism. The calculation of the second attention feature and the first attention feature are mutually complementary, and together they realize the bidirectional interaction between pathological and transcriptomic features.

[0040] The first attention feature and the second attention feature are concatenated along the channel dimension (i.e., the feature dimension) to obtain the concatenated feature. The calculation formula is as follows: Then, a learnable linear projection layer maps the concatenated features back to the common feature dimension, resulting in dual-modal fusion features. The calculation formula is: ,in The linear projection weight matrix is ​​learnable. This is a learnable bias vector.

[0041] Regarding pathological features, the missing rate Defined as the proportion of invalid pathological blocks in the pathological ROI dataset out of the total number of pathological blocks; signal-to-noise ratio. Defined as the ratio of the pixel variance of the tumor region to the pixel variance of the background region in a pathological block; effective data percentage. Defined as the proportion of samples with fully annotated ROI regions out of the total samples. For transcriptome features, the missing rate... Defined as the proportion of missing gene expression values ​​in the prognosis-related gene expression matrix; signal-to-noise ratio. Defined as the ratio of the expression variance of the core prognostic gene among different prognostic risk groups to the expression variance within the group; percentage of valid data. Defined as the proportion of samples with acceptable gene expression quality after batch effect correction out of the total sample. For clinical characteristics, the deletion rate... Defined as the proportion of follow-up windows missing in the structured clinical time series features; signal-to-noise ratio Defined as the ratio of the variance among different prognostic risk groups to the variance within different groups for strongly correlated features such as TNM staging; percentage of valid data. Defined as the proportion of samples with complete data at the follow-up endpoint out of the total sample. The formula for calculating the combined quality confidence of the three features is: ,in, For modal identification, = p Corresponding pathology, = t Corresponding transcriptome, = c Corresponding to clinical practice, For the first m The quality confidence score for each class feature is calculated; a higher value indicates higher data quality and more reliable prognostic information for that modality. The three quality confidence scores are then normalized to obtain the normalized quality confidence scores.

[0042] Dual-modal fusion features Generate the query matrix using learnable linear projection. Aligned clinical features The key matrix is ​​generated by two independent learnable linear projections. Sum matrix The third attention feature is calculated using the standard cross-attention mechanism. Then align the clinical features. Generate the query matrix using learnable linear projection. fusion features of dual modes The key matrix is ​​generated by two independent learnable linear projections. Sum matrix The third attention feature is calculated using the standard cross-attention mechanism. .

[0043] The average of the normalized quality confidence scores of pathology and transcriptomics was taken as the comprehensive quality confidence score of the bimodal fusion feature. The weights of the third attention feature corresponding to the bimodal fusion feature are calculated using the following formula: , Normalized quality confidence score for clinical features. Weights of the fourth attention features corresponding to the clinical features. Finally, multimodal fusion deep features are obtained through weighted fusion.

[0044] It should be noted that modality-aligned projection addresses the heterogeneity issue among pathological, transcriptomic, and clinical data, mapping features from different dimensions and semantic spaces to a unified common space, laying the foundation for cross-modal interaction. Through two rounds of adaptive bidirectional cross-attention fusion, bidirectional collaboration between pathological and transcriptomic data is first achieved, followed by bidirectional complementarity between bimodal and clinical features, fully capturing prognostic correlations between different modalities and avoiding information redundancy and interaction loss caused by simple feature splicing. By calculating quality confidence based on missing rate, signal-to-noise ratio, and effective data percentage, and through adaptive weight adjustment, dynamic allocation of fusion weights based on the quality of single-modal data is achieved. When the quality of a particular modality's data is poor, its contribution is automatically reduced, significantly improving the robustness and clinical adaptability of multimodal fusion. The final output multimodal fusion deep features comprehensively cover multidimensional information on gastric cancer prognosis.

[0045] In the specific implementation process, the multi-task joint prognostic prediction head in step 400 adopts an end-to-end architecture of a shared feature layer and parallel task branches, consisting of a shared feature mapping layer and three parallel prognostic prediction branches. The shared feature mapping layer is shared by all task branches and consists of two fully connected network layers, a GELU activation function, and a Dropout layer. It first extracts and transforms the dimensions of the input multimodal fusion deep features to obtain shared prognostic features. The role of the shared feature mapping layer is to extract the core semantics that are generally relevant to the prognosis of gastric cancer from the three types of modality fusion features, thereby achieving information complementarity and collaborative learning among multiple tasks, while reducing feature dimensionality and providing a unified high-quality input for the three parallel branches.

[0046] The three branches share only the parameters of the preceding shared feature mapping layer; the branch layer parameters are independent of each other to avoid interference between tasks. The three branches are: overall survival (OS) risk prediction branch, disease-free survival (DFS) risk prediction branch, and postoperative recurrence and metastasis binary prediction branch. The specific prediction process includes: The OS risk prediction branch, adapted to the characteristics of survival analysis tasks, employs a Cox proportional hazards regression layer structure. It shares prognostic features. After the input branch, a single-dimensional logarithmic risk score is first output through a fully connected layer, and then the OS risk ratio is obtained through exponential transformation. The calculation formula is as follows: ,in This represents the OS risk ratio of the sample relative to the baseline population in the training set. A value greater than 1 indicates that the sample's OS prognostic risk is higher than that of the baseline population, while a value less than 1 indicates that the risk is lower than that of the baseline population.

[0047] The DFS risk prediction branch has the same structure as the OS branch, employing a Cox proportional hazards regression layer. It outputs the DFS risk ratio after inputting shared prognostic features. .

[0048] A binary classification prediction branch for postoperative recurrence and metastasis is included. This branch is adapted for binary event prediction tasks and employs a logistic regression layer structure. After inputting shared prognostic features into the branch, a fully connected layer and a sigmoid activation function are passed through it to output the 5-year postoperative recurrence and metastasis probability. The calculation formula is as follows: , The value range is [0,1]. The closer the value is to 1, the higher the probability of recurrence and metastasis 5 years after the operation.

[0049] Then, based on the model's backpropagation gradient, the contribution weight of each dimension in the shared prognostic features to the prediction result of the corresponding branch is calculated, resulting in a feature contribution vector. The pairwise cosine similarity between the three vectors is then calculated as the feature contribution similarity, thereby quantifying the synergy of feature contributions between branches. Finally, the risk synergy correction coefficient is calculated using the following formula: ,in It contributes similarity to the features of the OS and DFS branches. The similarity between OS and recurrent metastatic branches is contributed. This contributes similarity to the characteristics of DFS and recurrent metastatic branches. Simultaneously, it sets... The upper and lower limits of the value are [0.8, 1.2]. If the calculated value exceeds the range, it is truncated to the boundary value to avoid excessive weight adjustment caused by extreme values. The closer to 1, the higher the synergy of the feature contributions of the three branches, and the stronger the consistency of the prediction results. The greater the deviation from 1, the more obvious the conflict between the prediction results of the branches, and the more powerful the correction and adjustment of the basic weights are required.

[0050] Next, the baseline weighting for general clinical scenarios was determined: Overall survival (OS) is the primary clinical endpoint for prognostic assessment of gastric cancer, with the highest level of evidence-based medicine, and is therefore assigned a baseline weight. =0.5; Disease-free survival (DFS) is a core secondary endpoint for postoperative prognostic assessment of gastric cancer, and is directly related to patients' postoperative quality of life and recurrence risk, and is assigned a basic weight. =0.3; Recurrence and metastasis 5 years after surgery is the most critical clinical event after gastric cancer surgery and an important predictive indicator of overall survival (OS) and disease-free survival (DFS). The baseline weight is assigned as follows: =0.2. Simultaneously, a differentiated weighting mapping rule is preset. For stage I early-stage gastric cancer patients, the risk of postoperative recurrence and metastasis is extremely low, and prognostic assessment focuses on long-term disease-free survival (DFS) and overall survival (OS). The base weight is adjusted to... =0.55、 =0.35、 =0.1; For patients with stage III locally advanced gastric cancer, the risk of postoperative recurrence and metastasis is high. Prognostic assessment focuses on short-term recurrence events and long-term survival, and the baseline weight is adjusted to 0.1. =0.45、 =0.3、 =0.25; For patients with stage IV advanced gastric cancer, the core prognostic assessment focuses on overall survival (OS) and disease progression, and the baseline weight is adjusted to 0.25; =0.6、 =0.25、 =0.15.

[0051] Finally, using OS as the core benchmark endpoint, the weight adjustment logic is designed: when the risk collaborative adjustment coefficient... When <1, it indicates poor inter-branch prediction consistency. In this case, the weight of the core endpoint OS should be increased, and the weight of branches with poor consistency should be decreased to ensure the reliability of the prognostic assessment. A score >1 indicates strong predictive consistency among branches. Appropriately increasing the weights of DFS and recurrence / metastasis branches allows for full utilization of complementary information from multiple endpoints, enhancing the comprehensiveness of prognostic assessment. The formula for calculating the initial adjusted weights is: , , Then, a quantifiable comprehensive prognostic index is obtained by dynamically weighting and summing the results. Since the OS hazard ratio and DFS hazard ratio range from [0, +∞), while the postoperative recurrence and metastasis probability ranges from [0, 1], direct weighting would lead to the hazard ratio dominating the index calculation. Therefore, the three prediction results are first Z-score standardized to map them to the same dimension space. Subsequently, based on the dynamic weight set, the standardized three indicators are weighted and summed to obtain the final gastric cancer-specific comprehensive prognostic index. .

[0052] It should be noted that, through the parallel learning architecture of a multi-task joint prognostic prediction head, the collaborative prediction of three core prognostic endpoints of gastric cancer—OS, DFS, and postoperative recurrence and metastasis—was achieved. This not only improved the prediction accuracy of a single branch by leveraging the complementary information between multiple tasks, but also comprehensively covered the multi-dimensional assessment needs of short-term recurrence risk and long-term survival prognosis after gastric cancer surgery, solving the problems of one-sided information and insufficient generalization of single-task prediction in existing technologies. Through risk collaborative correction coefficients and dynamic weight adjustment mechanisms, the weight allocation is adaptively corrected based on the collaborative consistency of the prediction results of the three branches. This ensures the dominant position of OS as a core clinical endpoint, while fully adapting to the weight requirements under different clinical scenarios and different prediction consistency conditions, avoiding the limitations of fixed weight fusion, and significantly improving the clinical adaptability and robustness of the prognostic index. The gastric cancer-specific comprehensive prognostic index obtained through standardized weighted calculation realizes the single-value quantitative assessment of gastric cancer prognostic risk, solving the problem of the difficulty in uniformly interpreting multi-dimensional prognostic results in existing technologies.

[0053] In the specific implementation process, step 500 sorts all GCPI values ​​of the samples in ascending order, excluding extreme values ​​within the top and bottom 5% to avoid cutoff bias caused by outliers, and iterates through all candidate cutoff values ​​within the remaining GCPI value range with a step size of 0.01. For each candidate cutoff value, the samples are divided into a high-risk group (values ​​greater than the candidate cutoff value) and a low-risk group (values ​​not exceeding the candidate cutoff value). Based on the OS survival data of the two groups, a log-rank test is performed to calculate the corresponding chi-square statistic and p-value. The candidate cutoff value with the largest chi-square statistic and the smallest corresponding p-value is selected as the preliminary optimal cutoff value. Five-fold cross-validation is used to verify the stability of the preliminary optimal cutoff value to ensure that it can achieve significant prognostic stratification effects in different data subsets. At the same time, the Bonferroni method is used to correct the p-value of the multiple tests. The statistical significance of the cutoff value is confirmed only when the corrected p-value is <0.05. Finally, the optimal risk cutoff value of GCPI that conforms to clinical standards is obtained. For each patient sample, those with a GCPI value greater than the optimal risk cutoff were classified into the high-prognostic-risk group; those with a GCPI value not exceeding the optimal risk cutoff were classified into the low-prognostic-risk group. OS and DFS survival curves were plotted for the high- and low-risk groups using the Kaplan-Meier method. The significance of the difference between the two survival curves was verified using the log-rank test, requiring a corrected P-value < 0.001 to ensure strong statistical discrimination of the risk stratification. Univariate and multivariate Cox proportional hazards regression models were used to verify the independent prognostic value of GCPI risk stratification. After including routine clinical prognostic factors such as age, sex, TNM stage, and tumor differentiation, GCPI risk stratification was confirmed as an independent prognostic factor for gastric cancer, requiring a 95% confidence interval of HR not including 1 and a P-value < 0.05.

[0054] Then, a dual-track attribution strategy combining attention weight backtracking and SHAP values ​​is employed. Attention weight backtracking, based on the attention weight distribution during model training, traces the contribution paths of different modalities and features to the final GCPI result from the model's decision-making mechanism level, identifying the core dependent features of the model's decisions. SHAP values, based on a game-theoretic interpretability framework, quantify the marginal contribution of each feature to the GCPI prediction result, outputting the statistically significant feature contribution that conforms to clinical research standards. The attribution analysis comprehensively covers the pathology, transcriptomics, and clinical modalities, ultimately outputting the top-N key influencing factors with the highest contribution to prognostic results within each modality. The N value is fixed at 3-5 according to clinical interpretation needs, avoiding difficulties in clinical interpretation caused by too many factors.

[0055] Finally, from the bidirectional cross-attention layer of the multimodal fusion module, the attention weight matrix for the final fusion is extracted. The global contributions of pathological features, transcriptomic features, and clinical features to the final multimodal fusion features are calculated to obtain the global weight ratio of the three modalities. For the pathological image branch, the self-attention weights of the first Transformer encoder are backtracked, and combined with ROI region label encoding, the contribution of each pathological ROI region to the pathological morphological depth features is calculated. Then, combined with the modal global weights, the contribution of each ROI region to the final GCPI result is obtained. For the transcriptomic branch, the attention weights of the local and global branches of the second Transformer encoder are backtracked, and the contribution of each prognostic core gene to the transcriptomic molecular depth features is calculated. Combined with the modal global weights, the contribution of each gene to the GCPI is obtained. For the clinical feature branch, the self-attention weights of the third Transformer encoder are backtracked, and combined with the clinical prior attention mask, the contribution of each clinical feature to the clinical temporal depth features is calculated. Combined with the modal global weights, the contribution of each clinical feature to the GCPI is obtained.

[0056] Furthermore, the specific implementation process of SHAP value attribution analysis is as follows: 100 representative samples are randomly selected from the dataset as a background dataset to calculate the baseline expected output of the features, avoiding SHAP value calculation errors caused by background dataset bias. For the GCPI prediction results of each patient sample, the SHAP value of each input feature is calculated based on the KernelSHAP framework. Then, for the features of the pathology, transcriptomics, and clinical modalities, the mean absolute value of the SHAP value of each feature in the entire sample set is calculated. The larger the value, the higher the global influence of the feature on GCPI prediction.

[0057] For each feature, the arithmetic mean of the standardized attention contribution and the SHAP contribution is taken to obtain the feature's overall contribution. The closer the overall contribution is to 1, the greater the impact of the feature on the patient's prognostic outcome. For the pathological modality, all pathological ROI regions are sorted in descending order by overall contribution, and the top-N pathological ROI regions with the highest contribution to the prognostic outcome are extracted, while the contribution direction of each ROI region is output. For the transcriptome modality, all core prognostic genes are sorted in descending order by overall contribution, and the top-N core genes with the highest contribution to the prognostic outcome are extracted, while the gastric cancer oncogenic pathway to which the genes belong is labeled. For the clinical modality, all clinical features are sorted in descending order by overall contribution, and the top-N clinical features with the highest contribution to the prognostic outcome are extracted. The top-N key influencing factors from the three modalities are then integrated to form a complete set of prognostic influencing factors.

[0058] Finally, based on the comprehensive prognostic assessment results, the specific GCPI value, optimal risk cutoff value, and final prognostic risk classification (high / low risk) for each patient are clearly indicated. Individualized predicted values ​​for 3-year and 5-year postoperative OS rate, DFS rate, and recurrence / metastasis probability are also provided, along with a reference median survival time for the corresponding risk group. Furthermore, the top-N key influencing factors, their corresponding comprehensive contributions, and prognostic impact directions are listed, accompanied by visualization charts, including a heatmap of pathological ROI region contribution, a beehive plot of characteristic SHAP values, and a pie chart of the contribution percentages of the three modalities. Combined with clinical guidelines for gastric cancer diagnosis and treatment, individualized follow-up strategies, references for adjuvant therapy plans, and intervention suggestions for risk factors are provided, thus generating a prognostic attribution report.

[0059] like Figure 2 As shown, the present invention also provides a gastric cancer prognostic detection system based on Transformer deep learning, comprising: The data preprocessing module is used to construct a multimodal heterogeneous dataset of gastric cancer and to perform prior-guided collaborative preprocessing on whole-slice pathological images, transcriptome sequencing data and clinical follow-up data in the multimodal heterogeneous dataset of gastric cancer to obtain a pathological ROI dataset, a prognosis-related gene expression matrix and a structured clinical time series feature set. The Transformer feature extraction module is used to construct a three-branch Transformer feature extraction network. The pathological ROI dataset, the prognosis-related gene expression matrix, and the structured clinical time-series feature set are respectively input into different branches of the three-branch Transformer feature extraction network for deep feature extraction, resulting in pathological morphological deep features, transcriptomic molecular deep features, and clinical time-series deep features. The multimodal fusion module is used to perform modal alignment projection, adaptive bidirectional cross-attention fusion, and dynamic adjustment of modal weights on pathological morphological deep features, transcriptomic molecular deep features, and clinical temporal deep features to obtain multimodal fused deep features. The prognostic index calculation module is used to predict and analyze multimodal fusion deep features through a multi-task joint prognostic prediction head, and calculate the gastric cancer-specific comprehensive prognostic index based on the results of overall survival (OS) risk prediction, disease-free survival (DFS) risk prediction, and postoperative recurrence and metastasis binary classification prediction. The prognostic grading and outcome visualization module is used to classify prognostic risks based on the gastric cancer-specific comprehensive prognostic index and generate a prognostic attribution report.

[0060] The beneficial effects of this invention are as follows: 1) A multimodal heterogeneous dataset for gastric cancer, consisting of paired pathological images, transcriptome sequencing data, and clinical follow-up data, was constructed. This dataset simultaneously covers prognostic factors across three dimensions: tumor morphology, molecular biology, and clinical phenotype. This addresses the core limitation of relying solely on single-dimensional biological information and failing to comprehensively cover all prognostic factors in gastric cancer. Furthermore, through multi-task joint prediction, the dataset simultaneously assesses three core clinical endpoints: overall survival (OS), disease-free survival (DFS), and postoperative recurrence and metastasis. This comprehensively covers the full range of needs for short-term recurrence risk and long-term survival prognosis after gastric cancer surgery, significantly improving the comprehensiveness and accuracy of prognostic prediction. 2) For the data characteristics of the three modalities, a dedicated Transformer feature extraction branch was designed instead of directly applying the general Transformer structure: the pathological image branch guides the model to focus on pathological regions with strong prognostic correlation through ROI region-aware location encoding and gastric cancer prognostic prior attention mask; the transcriptome branch uses the prior gene grouping of gastric cancer oncogenic pathways, combined with a dual-path design of local branches within the pathway and global branches across the entire pathway, to simultaneously capture gene synergy within the pathway and cross-regulation information between pathways; the clinical feature branch strengthens the weight of prognostic features with strong correlation such as TNM stage and tumor differentiation degree through temporal location encoding and clinical prior attention mask, which solves the defects of the general Transformer indiscriminately learning redundant features and failing to accurately capture the core biological features of gastric cancer prognosis, and greatly improves the specificity and effectiveness of feature extraction; 3) A two-round adaptive bidirectional cross-attention fusion strategy was designed. First, bidirectional cross-attention was used to achieve deep synergy between pathological and transcriptomic features. Then, bidirectional complementarity between bimodal features and clinical features was achieved, which fully captured the prognostic correlation information between different modalities and avoided information redundancy and interaction loss caused by simple feature splicing. At the same time, the single-modal quality confidence was calculated based on the data missing rate, signal-to-noise ratio, and effective data ratio, and the fusion weight was dynamically adjusted. When the data quality of a certain modality was poor or there were missing data, its contribution ratio was automatically reduced. This solved the problem that fixed weight fusion could not adapt to data quality fluctuations and lacked robustness, and significantly improved the model's generalization ability and anti-interference ability in multi-center and real clinical scenarios. 4) Through a multi-task learning architecture with a shared feature layer and parallel task branches, collaborative prediction of three types of prognostic endpoints was achieved. The information complementarity between multiple tasks further improved the prediction accuracy of a single branch. At the same time, the risk collaborative correction coefficient was calculated based on the collaborative contribution of the features of the three prediction branches. Combined with the weight mapping rules preset by the clinical evidence-based medicine evidence for gastric cancer, the weight of the prognostic endpoint was dynamically corrected. This not only ensured the dominant position of OS, the core clinical endpoint, but also adapted to different TNM stages and different clinical scenarios with different prediction consistency. Finally, a single-value quantitative assessment of the prognostic risk of gastric cancer was achieved, which solved the problems of difficulty in unifying the clinical interpretation of multi-dimensional prognostic results and poor adaptability of fixed weights. 5) By combining the X-tile algorithm with log-rank test and Cox regression validation, the optimal risk cutoff value with strict statistical significance was determined, achieving standardized prognostic risk grading. At the same time, through the dual-track attribution strategy of attention weight backtracking and SHAP value, the pathological, transcriptomic, and clinical key factors affecting patient prognosis were accurately located from two dimensions: model decision-making mechanism and clinical statistics, breaking the "black box" dilemma of deep learning models. The final standardized prognostic attribution report transforms the complex model prediction results into interpretable and applicable diagnostic and treatment reference information for clinicians, realizing a closed loop from accurate prognostic prediction to clinical diagnosis and treatment guidance, and solving the core problems of lack of interpretability and disconnection from clinical diagnosis and treatment needs.

[0061] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other.

[0062] Specific examples have been used to illustrate the principles and implementation methods of this invention. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of this invention. Furthermore, those skilled in the art will recognize that, based on the ideas of this invention, there will be changes in the specific implementation methods and application scope. Therefore, the content of this specification should not be construed as a limitation of this invention.

Claims

1. A gastric cancer prognostic detection method based on Transformer deep learning, characterized in that, Includes the following steps: A multimodal heterogeneous dataset of gastric cancer was constructed, and prior-guided co-processing was performed on the whole-slice pathological images, transcriptome sequencing data and clinical follow-up data in the multimodal heterogeneous dataset of gastric cancer to obtain a pathological ROI dataset, a prognosis-related gene expression matrix and a structured clinical time series feature set. A three-branch Transformer feature extraction network was constructed, and the pathological ROI dataset, the prognosis-related gene expression matrix, and the structured clinical time-series feature set were respectively input into different branches of the three-branch Transformer feature extraction network for deep feature extraction, so as to obtain pathological morphological deep features, transcriptomic molecular deep features, and clinical time-series deep features; Modality alignment projection, adaptive bidirectional cross-attention fusion, and dynamic adjustment of modality weights are performed on the pathological morphological depth features, the transcriptomic molecular depth features, and the clinical temporal depth features to obtain multimodal fused depth features; The multi-modal fusion deep features are predicted and analyzed by a multi-task joint prognostic prediction head, and a gastric cancer-specific comprehensive prognostic index is calculated based on the results of overall survival (OS) risk prediction, disease-free survival (DFS) risk prediction, and postoperative recurrence and metastasis binary classification prediction. Prognostic risk stratification is performed based on the gastric cancer-specific comprehensive prognostic index, and a prognostic attribution report is generated.

2. The gastric cancer prognostic detection method based on Transformer deep learning according to claim 1, characterized in that, A multimodal heterogeneous dataset for gastric cancer was constructed, and prior-guided co-processing was performed on the whole-slice pathological images, transcriptome sequencing data, and clinical follow-up data in the dataset to obtain a pathological ROI dataset, a prognostic gene expression matrix, and a structured clinical time-series feature set, including: Pixel-level semantic segmentation is performed on the tumor core area, tumor invasion front area and stroma area in the whole slice pathological image. After removing the background and irrelevant tissue areas, the ROI area is obtained and then non-overlapping blocks are formed to generate pathological blocks. The ROI area is labeled in the pathological blocks to obtain the pathological ROI dataset. The transcriptome sequencing data were screened using LASSO-Cox regression analysis to obtain the core gene set for gastric cancer prognosis. The gene expression data in the core gene set for gastric cancer prognosis were then subjected to multicenter batch effect elimination using the ComBat-seq function to obtain the expression matrix of the prognosis-related genes. The clinical follow-up data are subjected to ordered encoding and unique-hot encoding to obtain the structured clinical time-series feature set.

3. The gastric cancer prognostic detection method based on Transformer deep learning according to claim 1, characterized in that, Feature extraction of the pathological ROI dataset is performed using the pathological image branch of the three-branch Transformer feature extraction network, including: Linear projection is performed on the pathological ROI dataset to obtain block embedding vectors, which are then superimposed with ROI region-aware location codes; the ROI region-aware location codes are a weighted sum of absolute location codes and ROI region label codes. The block embedding vector is input into the first Transformer encoder to constrain the attention weights of the block embedding vector through a gastric cancer prognostic prior attention mask; The output features of different depth layers in the first Transformer encoder are aggregated at multiple scales, and the pathological morphological depth features are obtained by global average pooling and linear projection.

4. The gastric cancer prognostic detection method based on Transformer deep learning according to claim 1, characterized in that, Feature extraction of the prognosis-related gene expression matrix is ​​performed using the transcriptome branch of the three-branch Transformer feature extraction network, including: Based on the prior knowledge of the gastric cancer carcinogenesis pathway, the prognosis-related gene expression matrix is ​​divided into multiple pathway genomes, and the expression value of each gene is linearly projected into a gene token embedding vector. The gene token embedding vector is input into the second Transformer encoder; the second Transformer encoder includes a local branch and a global branch, the local branch is used to perform self-attention calculation on the gene token embedding vector within the same pathway, and the global branch is used to perform self-attention calculation on the gene token embedding vector of all pathways; The output features of the local branch and the global branch are concatenated, subjected to global average pooling, and linear projection to obtain the deep molecular features of the transcriptome.

5. The gastric cancer prognostic detection method based on Transformer deep learning according to claim 1, characterized in that, Feature extraction is performed on the structured clinical time-series feature set using the clinical feature branch of the three-branch Transformer feature extraction network, including: The feature vectors of each follow-up time window in the structured clinical time-series feature set are linearly projected and superimposed with time-series position codes to obtain the clinical token embedding vector; The clinical token embedding vector is input into the third Transformer encoder to adjust the attention weights of the gastric cancer prognostic strongly correlated features in the clinical token embedding vector through a clinical prior attention mask; the gastric cancer prognostic strongly correlated features include: TNM stage, tumor differentiation degree, lymph node metastasis status and vascular invasion status; The clinical temporal depth features are obtained by performing temporal global average pooling and linear projection on the output features of the third Transformer encoder.

6. The gastric cancer prognostic detection method based on Transformer deep learning according to claim 1, characterized in that, Modality alignment projection, adaptive bidirectional cross-attention fusion, and dynamic adjustment of modality weights are performed on the pathological morphological depth features, the transcriptomic molecular depth features, and the clinical temporal depth features to obtain multimodal fused depth features, including: The pathological morphological depth features, transcriptomic molecular depth features, and clinical temporal depth features are mapped to a common feature space of the same dimension by linear projection, resulting in pathological features, transcriptomic features, and clinical features. Using the pathological features as queries and the transcriptomic features as keys and values, cross-attention calculation is performed to obtain the first attention feature; Using the transcriptome features as queries and the pathological features as keys and values, cross-attention calculation is performed to obtain the second attention feature; The first attention feature and the second attention feature are concatenated and linearly projected to obtain the dual-modal fusion feature; The quality confidence levels of the pathological features, transcriptomic features, and clinical features were calculated based on the missing data rate, signal-to-noise ratio, and percentage of valid data, respectively. Using the dual-modal fusion feature as the query and the clinical feature as the key and value, cross-attention calculation is performed to obtain the third attention feature; Using the clinical features as queries and the bimodal fusion features as keys and values, cross-attention calculation is performed to obtain the fourth attention feature; Based on the quality confidence, an adaptive fusion weight is determined to perform weighted fusion of the third attention feature and the fourth attention feature to obtain the multimodal fusion deep feature.

7. The gastric cancer prognostic detection method based on Transformer deep learning according to claim 1, characterized in that, The multimodal fusion deep features are predicted and analyzed using a multi-task joint prognostic prediction head. Based on the obtained overall survival (OS) risk prediction results, disease-free survival (DFS) risk prediction results, and postoperative recurrence and metastasis binary classification prediction results, a gastric cancer-specific comprehensive prognostic index is calculated, including: The multimodal fusion deep features are predicted in parallel using the OS risk prediction branch, DFS risk prediction branch, and postoperative recurrence and metastasis binary prediction branch of the multi-task joint prognostic prediction head to obtain the OS risk ratio, DFS risk ratio, and postoperative 5-year recurrence and metastasis probability. Calculate the risk collaborative correction coefficient based on the contribution vector of the core features in the output process of the three prediction branches; Based on the preset prognostic endpoint weight mapping rule, basic weights are assigned to the OS hazard ratio, the DFS hazard ratio, and the 5-year postoperative recurrence and metastasis probability, and the basic weights are corrected by the risk co-correction coefficient to obtain a dynamic weight set. The gastric cancer-specific comprehensive prognostic index is obtained by weighting and summing the OS hazard ratio, DFS hazard ratio, and 5-year postoperative recurrence and metastasis probability according to the dynamic weight set.

8. The gastric cancer prognostic detection method based on Transformer deep learning according to claim 6, characterized in that, Prognostic risk stratification is performed based on the gastric cancer-specific comprehensive prognostic index, and a prognostic attribution report is generated, including: The optimal risk cutoff value is determined using the X-tile algorithm. Samples with a gastric cancer-specific comprehensive prognostic index greater than the optimal risk cutoff value are classified as high-prognostic-risk groups, while samples with a gastric cancer-specific comprehensive prognostic index less than the optimal risk cutoff value are classified as low-prognostic-risk groups. Attribution analysis of the pathological features, transcriptomic features, and clinical features was performed using attention weighted backtracking and SHAP values ​​to identify key influencing factors. The prognostic attribution report is generated based on the prognostic risk grouping results, key influencing factors, and the gastric cancer-specific comprehensive prognostic index.

9. The gastric cancer prognostic detection method based on Transformer deep learning according to claim 8, characterized in that, Attribution analysis of the pathological features, transcriptomic features, and clinical features using attention weighted backtracking and SHAP values ​​revealed key influencing factors, including: Attribution analysis was performed on the pathological features to obtain the top-N pathological ROI regions that contributed the most to the prognostic results; Attribution analysis was performed on the transcriptome features to identify the top-N prognostic-related core genes that contributed the most to the prognostic results. Attribution analysis was performed on the clinical features to obtain the top-N clinical features that contributed the most to the prognostic outcome.

10. A gastric cancer prognostic detection system based on Transformer deep learning, characterized in that, include: The data preprocessing module is used to construct a multimodal heterogeneous dataset of gastric cancer and to perform prior-guided collaborative preprocessing on the whole-slice pathological images, transcriptome sequencing data and clinical follow-up data in the multimodal heterogeneous dataset of gastric cancer to obtain a pathological ROI dataset, a prognosis-related gene expression matrix and a structured clinical time series feature set. The Transformer feature extraction module is used to construct a three-branch Transformer feature extraction network. The pathological ROI dataset, the prognosis-related gene expression matrix, and the structured clinical time-series feature set are respectively input into different branches of the three-branch Transformer feature extraction network for deep feature extraction, so as to obtain pathological morphological deep features, transcriptomic molecular deep features, and clinical time-series deep features. The multimodal fusion module is used to perform modal alignment projection, adaptive bidirectional cross-attention fusion, and dynamic adjustment of modal weights on the pathological morphological deep features, the transcriptomic molecular deep features, and the clinical temporal deep features to obtain multimodal fused deep features. The prognostic index calculation module is used to predict and analyze the multimodal fusion deep features through a multi-task joint prognostic prediction head, and calculate the gastric cancer-specific comprehensive prognostic index based on the results of overall survival (OS) risk prediction, disease-free survival (DFS) risk prediction, and postoperative recurrence and metastasis binary classification prediction. The prognostic grading and result visualization module is used to perform prognostic risk grading based on the gastric cancer-specific comprehensive prognostic index and generate a prognostic attribution report.

Images