Gastric cancer prognosis system based on pathological omics integrated model and gastric cancer prognosis system based on multi-modal deep learning

Abstract

The invention discloses a gastric cancer prognosis system based on a pathomics integrated model and a gastric cancer prognosis system based on multi-modal deep learning, and relates to the technical field of gastric cancer prognosis. Compared with a single machine learning method, the gastric cancer prognosis system based on the pathomics integrated model provided by the invention shows good prediction accuracy on postoperative early recurrence in a training set and a verification set, and the system has remarkable sensitivity and specificity. The invention further provides a gastric cancer prognosis system based on multi-mode deep learning. Compared with a clinical feature model, the multi-modal prediction model fusing clinical features and pathological omics features shows a very strong prediction capability on early postoperative recurrence of LAGC patients, and the accuracy, sensitivity and specificity of prediction of early postoperative recurrence are greatly improved. In addition, the multi-modal prediction model provided by the invention can increase additional prognosis value for the current TNM staging system.

Description

Technical Field

[0001] The present invention relates to the technical field of gastric cancer prognosis, and in particular to a gastric cancer prognosis system based on a pathological omics integrated model and a gastric cancer prognosis system based on multimodal deep learning. Background Art

[0002] Traditional prognostic evaluation mainly uses histopathological evaluation, serum markers and imaging technology. With the development of technology, the prognostic evaluation of gastric cancer has evolved from a single morphological analysis to a multi-dimensional system integrating molecular characteristics, imaging omics, artificial intelligence and big data. Among them, modern molecular biology technology mainly uses: molecular typing, key molecular markers (such as HER2 detection) and liquid biopsy technology. The integration of artificial intelligence and big data includes: AI analysis of pathological images, imaging omics analysis and multi-omics prognostic models. However, these current solutions mainly focus on single modality data and fail to fully utilize the correlation and complementarity between multimodal data. Therefore, there is still room for improvement in the prognosis prediction of gastric cancer.

[0003] In view of this, the present invention is proposed. Summary of the Invention

[0004] The purpose of the present invention is to provide a gastric cancer prognosis system based on a pathological omics integrated model and a gastric cancer prognosis system based on multimodal deep learning to solve the above technical problems.

[0005] The present invention is achieved in that:

[0006] In a first aspect, the present invention provides a gastric cancer prognosis system based on a pathology omics integrated model, comprising:

[0007] Input unit: used to obtain the patient's gastric cancer pathological image and segment the WSI image to obtain a segmented image;

[0008] Pathology feature extraction unit: used to extract pathology features from patients' gastric cancer pathology images based on the open source CellProfiler software;

[0009] The deep learning-based pathology feature extractor construction unit selects a portion of the segmented image and divides it into a training set and a validation set; a residual neural network architecture is used to train and obtain a feature extractor;

[0010] Deep learning pathomic feature extraction unit: used to flatten all WSI images into segmented images of the same size as the segmented images used during training, put them into the feature extractor, use the output values ​​of the feature extractor as deep learning features, and sum and average the feature information of the segmented images belonging to the same WSI image to generate deep learning pathomic features;

[0011] Pathological omics feature selection unit: used to select recurrence-related pathological omics features based on the pathological omics features extracted by the pathological feature extraction unit and the deep learning-based pathological omics feature extraction unit;

[0012] The pathomic ensemble model construction and validation unit is used to construct a pathomic model for gastric cancer prognosis using multiple algorithms. The voting regressor averages the single WSI image predictions of each basic regressor to generate the pathomic ensemble model. The basic regressor refers to the predicted probability value of each algorithm.

[0013] Gastric cancer prognosis unit: used to input the patient's gastric cancer pathology images into the pathology omics integrated model to obtain the predicted patient prognosis results.

[0014] In a second aspect, the present invention further provides a gastric cancer prognosis system based on multimodal deep learning, comprising:

[0015] Input unit: used to input independent clinical features related to gastric cancer recurrence and the gastric cancer prognosis results output by the above-mentioned pathological omics integrated model;

[0016] Multimodal prediction model construction unit: used to integrate independent clinical features related to gastric cancer recurrence and gastric cancer prognosis results obtained based on the pathological omics integrated model to construct a multimodal prediction model;

[0017] Gastric cancer prognostic unit: used to obtain the final gastric cancer prognosis results by integrating independent clinical features and pathological omics integrated models through a multimodal prediction model.

[0018] In a third aspect, the present invention provides an application of a multimodal prediction model in constructing a TNM staging system.

[0019] In a fourth aspect, the present invention provides an electronic device comprising: a processor and a memory; the processor and the memory are connected, wherein:

[0020] The memory is used to store the computer program, and the processor is used to call the computer program to execute the above-mentioned method for gastric cancer prognosis or TNM staging based on multimodal deep learning.

[0021] In a fifth aspect, the present invention provides a computer-readable storage medium comprising instructions, which, when executed on a computer, enables the computer to execute the above-mentioned method for gastric cancer prognosis or TNM staging based on multimodal deep learning.

[0022] The present invention has the following beneficial effects:

[0023] The present invention provides a gastric cancer prognosis system based on a pathology omics integrated model. Compared with a single machine learning method, the system shows good prediction accuracy for early postoperative recurrence in the training cohort, and the system has significant sensitivity and specificity. In the prediction of internal validation cohorts and external validation cohorts, a high prediction accuracy was shown. Compared with a single machine learning method, the system maintained high sensitivity and specificity. The pathology omics integrated model showed that the negative predictive value (NPV) of all cohorts was greater than 0.87, and the positive predictive value (PPV) exceeded 0.89. Therefore, the gastric cancer prognosis system provided by the present invention has high reliability in detecting negative and positive results.

[0024] The present invention also provides a gastric cancer prognosis system based on multimodal deep learning. Compared with the clinical feature model, the multimodal prediction model that integrates clinical features and pathological genomic features shows a strong predictive ability for early recurrence after surgery in LAGC patients. Compared with a single feature model, the multimodal prediction model provided by the present invention (i.e., the RSA model) greatly improves the accuracy of early postoperative recurrence prediction. Compared with a single clinical model, the multimodal prediction model can identify more low-risk recurrence patients. In addition, the decision curve further confirms that the RSA model has the greatest clinical benefit, and the 5-year OS of high-risk patients is significantly lower than that of low-risk patients. Through multivariate Cox regression analysis, clinical pathological variables including TNM stage and histological subtype were adjusted, and it was found that the RSA model is still an independent prognostic factor for the 5-year OS of imaging LAGC patients. According to the expression of different peripheral blood tumor markers and the molecular marker HER2 in tissue biopsy, a stratified analysis was performed. The multimodal prediction model provided by the present invention is superior to the single feature model in predicting early recurrence. The AUC box plot generated after ten-fold cross validation showed that the multimodal prediction model had better discrimination ability, sensitivity and specificity compared with the single clinical feature and pathological group feature model.

[0025] The multimodal prediction model provided by the present invention showed good discrimination ability, sensitivity and specificity in internal validation, multicenter external validation, prospective clinical studies and TCGA datasets.

[0026] In addition, the multimodal prediction model provided by the present invention can add additional prognostic value to the current TNM staging system and can provide better survival prediction within the same clinical stage. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for use in the embodiments. It should be understood that the following drawings only illustrate certain embodiments of the present invention and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other relevant drawings can be obtained based on these drawings without paying any creative work.

[0028] Figure 1 Flowchart for the study design: (A) Acquisition and preprocessing of histopathological images: Whole-slice images (WSIs) were obtained from hematoxylin and eosin (H&E)-stained tissue sections using a high-resolution scanner; each WSI was segmented into multiple non-overlapping patches to capture fine-grained morphological features under high-power magnification; (B) Multimodal feature extraction of histopathological images: Top: Deep learning-based pathological feature extraction using ResNet18 for feature representation and classifier training of WSIs; Bottom: Traditional manual features were extracted using CellProfiler; representative regions of interest (ROIs) were processed to quantify predefined morphological descriptors and generate an interpretable feature matrix; (C) Pathological features and integrated model construction: In After dimensionality reduction, recursive-related features were selected for model development. Predictive models were constructed using machine learning algorithms such as support vector machines, decision trees, random forests, and logistic regression. These individual models were then integrated into an ensemble framework to improve overall predictive performance. Receiver operating characteristic (ROC) curves and radar plots were used to evaluate the robustness and accuracy of the models. (D) Model validation: The final ensemble model was validated in multiple independent cohorts, including two internal validation cohorts (n = 410), two external validation cohorts (n = 498), a prospective cohort (n = 456), and a TCGA cohort (n = 251). A recurrence risk 40-score model (RSA) was developed based on selected pathological and clinical features to stratify patients according to their recurrence risk.

[0029] Figure 2Training process of the RSA model for predicting early recurrence of LAGC; (A) Training cohort screening process; (B) Nomogram combining pathological and clinical features for early recurrence prediction; (C) ROC curves of various prediction models in the training cohort; (D) Radar chart illustrating the performance indicators of different models in the training cohort; (E) Confusion matrix of the prediction models in the training cohort; (F) Calibration curve of the RSA model in the training cohort; (G) Kaplan-Meier survival curves stratified by risk group (high vs low) based on the Youden index derived from the nomogram; (H) Two-layer homology curves Heart-shaped circle diagram illustrating the clinical application of different prediction models in the training cohort; (I) Decision curve analysis curves of different models in the training cohort; (J) Comparison of the AUC, sensitivity, specificity, and accuracy of different models in the stratified analysis based on serum tumor markers and HER2 expression in biopsy specimens; (K) Performance indicators of various models in subgroups defined by peripheral blood tumor markers and HER2 status; (L) Box plots showing the AUC, sensitivity, specificity, and accuracy of different models in the training cohort after ten-fold cross-validation; (M) ROC curves of various prediction models in the training cohort after 1000 bootstrap resampling validation;

[0030] Figure 3 Figure 3 Internal validation of the RSA model for predicting early recurrence in LAGC patients; (A) The internal validation cohort was divided into cohort I (2012-2014) and cohort II (2017-2019) according to the year of admission; (B) ROC curves of different prediction models in the internal validation cohort I; (C) PR curves of different models in the internal validation cohort I; (D) Calibration curve of the RSA model in the internal validation cohort I; (E) Confusion matrix of the prediction model in the internal validation cohort I; (F) Radar chart showing the performance indicators of various models in the internal validation cohort I; (G) DCA curves of different models in the internal validation cohort I; (H) Double-layer concentric circle chart illustrating the internal validation of the RSA model. Clinical utility of the model in the internal validation cohort I; (I) Kaplan-Meier survival curves of the internal validation cohort I according to risk groups; (J) Kaplan-Meier survival curves of the internal validation cohort II according to risk groups based on the Youden index; (K) ROC curves of the prediction model in the internal validation cohort II; (L) PR curves of different models in the internal validation cohort II; (M) Double concentric circles showing the clinical application value of the model in the internal validation cohort II; (N) Confusion matrix of different models in the internal validation cohort II; (O) Radar chart showing the performance indicators in cohort II; (P) DCA curve of the RSA model in the internal validation cohort II;

[0031] Figure 4Figure 3 External validation of the RSA model for predicting early recurrence in LAGC patients; (A) Patients in the external validation cohort were grouped according to geographical regions: external validation cohort I (Northern China) and external validation cohort II (Southern China); (B) ROC curves of different prediction models in external validation cohort I; (C) PR curves of prediction models in the external validation cohort; (D) Calibration curve of the RSA model in the external validation cohort I; (E) Radar chart showing the performance indicators of various models in the external validation cohort; (F) Confusion matrix of prediction models in external validation cohort I; (G) DCA curves of different models in external validation cohort I; (H) Kaplan-Meier survival curves of patients in external validation cohort I, with patients classified according to the Youden index in the nomogram. Divided into high-risk group and low-risk group; (I) Double concentric circles show the clinical utility of the prediction model in the external validation cohort I; (J) Double concentric circles show the clinical utility of the prediction model in the external validation cohort II; (K) ROC curves of different models in the external validation cohort II; (L) PR curves of the prediction model in the external validation cohort II; (M) Calibration curve of the RSA model in the external validation cohort II; (N) Radar chart showing performance indicators in the external validation cohort II; (O) Confusion matrix of different models in the external validation cohort II; (P) DCA curve of the prediction model in the external validation cohort II; (Q) Kaplan-Meier survival curves of patients in the external validation cohort II. Patients were divided into high-risk group and low-risk group according to the Youden index obtained from the nomogram;

[0032] Figure 5To prospectively validate the RSA model in predicting early recurrence in patients with LAGC; (A) Patient recruitment and subgroup classification in the prospective validation cohort (NCT 01516944); (B) ROC curve of the prediction model in the direct surgery subgroup; (C) PR curve of the prediction model in the direct surgery subgroup; (D) Calibration curve of the RSA model in the direct surgery subgroup; (E) Radar chart of performance indicators of the prediction model in the direct surgery subgroup in the prospective cohort; (F) Confusion matrix of different models in the direct surgery subgroup; (G) Double-layer concentric circles showing the clinical utility of the prediction model in the direct surgery subgroup (upper) and neoadjuvant chemotherapy subgroup (lower); (H) DCA curves of different models in the direct surgery subgroup; (I) Kaplan-Meier survival curves of patients in the direct surgery subgroup divided into high-risk and low-risk groups according to the Youden index; (J) Confusion matrix of different models in the neoadjuvant chemotherapy subgroup ROC curve; (K) PR curve of the prediction model in the neoadjuvant chemotherapy subgroup; (L) Calibration curve of the RSA model in the neoadjuvant chemotherapy subgroup; (M) Radar chart showing the performance indicators of the prediction model in the neoadjuvant chemotherapy subgroup; (N) Confusion matrix of different models in the neoadjuvant chemotherapy subgroup; (O) DCA curves of different models in the neoadjuvant chemotherapy subgroup; (P) Kaplan-Meier survival curves of the neoadjuvant chemotherapy subgroup stratified by Youden index; (Q) Comparative analysis of radiation response in neoadjuvant treatment regimens based on risk stratification defined by Youden index; (R) Comparative analysis of postoperative pathological grades of different patients. The chemotherapy regimen for the neoadjuvant treatment group was formulated based on the risk groups divided by the Youden index;

[0033] Figure 6Validation of the RSA model in the TCGA cohort and molecular subgroup analysis; (A) Flowchart showing the inclusion and exclusion criteria of the TCGA external dataset; (B) ROC curves of different prediction models in the TCGA validation cohort; (C) PR curves of various models in the TCGA cohort; (D) Radar chart summarizing the performance of different models in the TCGA validation cohort; (E) Confusion matrix of prediction models in the TCGA validation cohort; (F) Double-layer concentric circles showing the clinical utility of different models in the TCGA cohort; (G) DCA curves of prediction models in the TCGA validation cohort; (H) T-test based on the Youden index derived from the nomogram Kaplan-Meier survival curves of the CGA cohort divided into high-risk and low-risk groups; (I) Confusion matrix of the RSA model applied to the four TCGA molecular subtypes; (J) Flowchart for subgroup selection in the ACRG molecular classification; (K) Confusion matrix of the RSA model in the four ACRG molecular subtypes; (L) Radar chart summarizing the model performance in the ACRG molecular subtypes; (M) ROC curves of different prediction models in the ACRG molecular subtypes; (N) Kaplan-Meier survival analysis comparison of the high-risk and low-risk groups divided by the RSA model in each ACRG molecular subtype based on the Youden index;

[0034] Figure 7 Visual demonstration of the auxiliary role of the RSA model in TNM staging and the interpretability of pathological omics features; (A) Kaplan-Meier survival curves comparing RSA-defined risk groups (high-risk group vs. low-risk group) under different TNM stages in training, internal and external validation cohorts; (B) Propensity score-matched survival analysis (1:1 matching) comparing high-risk and low-risk RSA groups under different TNM stages; (CJ) SHAP comprehensive honeycomb plots showing the importance of top pathological omics features in different datasets; each row represents the ranking according to its contribution to the model features; the x-axis shows the SHAP value (impact on the model output), each dot represents a patient, and its color is determined by the feature value (e.g., yellow for high, purple for low); the density of the dots illustrates the variability of the feature impact; C: training cohort; D: internal validation I; E: internal validation II; F: external validation I; G: external validation II; H: prospective cohort study (direct surgery subgroup); I: prospective cohort study (neoadjuvant treatment subgroup); J: TCGA cohort; (K) Visual comparison of postoperative clinical outcomes in two patients with different RSA-defined risk categories;

[0035] Figure 8 Schematic diagram of the gastric cancer prognosis system based on the patho-omics integrated model;

[0036] Figure 9 Schematic diagram of the gastric cancer prognosis system based on multimodal deep learning;

[0037] Figure 10 Feature selection and model performance in the training set; (A) LASSO coefficient curve of candidate pathway omics features; (B)

[0038] Select the optimal regularization parameter λ based on 10-fold cross validation;

[0039] (C) ROC curve comparing the performance of various machine learning models in predicting postoperative recurrence; (D) PR curves of different machine learning models in the training cohort; (E) DCA curve comparing the clinical utility of different models; (F) Radar chart comparing the predictive performance of models in the training cohort; (G) Confusion matrix of machine learning models for predicting postoperative recurrence;

[0040] Figure 11 Predictive performance of the ensemble model in different validation groups; (A) Performance comparison of the machine learning model and the ensemble model in internal validation group I; (B) Performance comparison in internal validation group II; (C) Performance comparison in external validation group I; (D) Performance comparison in external validation group II;

[0041] Figure 12 Figure 5. Stratified analysis of RSA model performance based on HER2 and tumor marker status; (AD) ROC curves of different prediction models in stratified validation subgroups based on serum tumor marker and HER2 expression; (EH) PR curves of the corresponding subgroups; (IL) confusion matrix of each model in the stratified subgroups; (MP) calibration curves of the prediction models in each subgroup;

[0042] Figure 13 Survival analysis of molecular and serological subgroups risk-stratified by the RSA model; (A) Comparison of five-year overall survival in HER2-positive patients; (B) DCA curves of different models in the HER2-positive subgroup; (C) Five-year overall survival in HER2-negative patients; (D) DCA curves in the HER2-negative subgroup; (E) Five-year overall survival in patients with serum tumor markers; (F) DCA curves in the marker-positive subgroup; (G) Five-year overall survival in patients with markers-negative; (H) DCA curves in the marker-negative subgroup;

[0043] Figure 14 The radar chart shows the performance of the model in TCGA (Tumor Genomics Database) (divided by molecular subtype); (A) CIN subtype; (B) EBV subtype; (C) GS subtype; (D) MSI subtype;

[0044] Figure 15The importance of global-level pathway omics features in different datasets; the SHAP-based global feature importance map was generated by averaging the absolute SHAP values ​​of all samples; features with higher average SHAP values ​​had a greater impact on the model's prediction of postoperative recurrence; (A) training set; (B) internal validation set I; (C) internal validation set II; (D) external validation set I; (E) external validation set II; (F) prospective cohort study (direct surgery group); (G) prospective cohort study (neoadjuvant chemotherapy group); (H) TCGA cohort.

[0045] Figure numerals: 301-first input unit; 302-pathological feature extraction unit; 303-pathological omics feature extractor construction unit based on deep learning; 304-pathological omics feature extraction unit based on deep learning; 305-pathological omics feature selection unit; 306-pathological omics integrated model construction and verification unit; 307-first gastric cancer prognosis unit; 401-second input unit; 402-multimodal prediction model construction unit; 403-second gastric cancer prognosis unit. DETAILED DESCRIPTION

[0046] Reference will now be made in detail to embodiments of the present invention, one or more examples of which are described below. Each example is provided to illustrate, not to limit, the present invention. Indeed, it will be apparent to those skilled in the art that various modifications and variations may be made to the present invention without departing from the scope or spirit of the invention. For example, features illustrated or described as part of one embodiment may be used in another embodiment to produce further embodiments.

[0047] Explanation of terms:

[0048] NRI: Net Reclassification Improvement Index, focuses on the reclassification accuracy of risk categories.

[0049] IDI (Integrated Discrimination Improvement): Evaluates the global improvement in predicted probability (continuous probability), reflecting the improvement in model discrimination.

[0050] In a first aspect, the present invention provides a gastric cancer prognosis system based on a pathology omics integrated model, comprising:

[0051] Input unit: used to obtain the patient's gastric cancer pathological image; segment the WSI image to obtain a segmented image;

[0052] Pathology feature extraction unit: used to extract pathology features from patients' gastric cancer pathology images based on the open source CellProfiler software;

[0053] The deep learning-based pathology feature extractor construction unit selects a portion of the segmented image and divides it into a training set and a validation set; a residual neural network architecture is used to train and obtain a feature extractor;

[0054] Deep learning pathomic feature extraction unit: used to flatten all WSI images into segmented images of the same size as the segmented images used during training, put them into the feature extractor, use the output values ​​of the feature extractor as deep learning features, and sum and average the feature information of the segmented images belonging to the same WSI image to generate deep learning pathomic features;

[0055] Pathological omics feature selection unit: used to select recurrence-related pathological omics features based on the pathological omics features extracted by the pathological feature extraction unit and the deep learning-based pathological omics feature extraction unit;

[0056] The pathomic ensemble model construction and validation unit is used to construct a pathomic model for gastric cancer prognosis using multiple algorithms. The voting regressor averages the single WSI image predictions of each basic regressor to generate the pathomic ensemble model. The basic regressor refers to the predicted probability value of each algorithm.

[0057] Gastric cancer prognosis unit: used to input the patient's gastric cancer pathology images into the pathology omics integrated model to obtain the predicted patient prognosis results.

[0058] The present invention provides a gastric cancer prognosis system based on a pathology omics integrated model. Compared with a single machine learning method, the system shows good prediction accuracy for early postoperative recurrence in the training cohort, and the system has significant sensitivity and specificity. In the prediction of internal validation cohorts and external validation cohorts, a high prediction accuracy was shown. Compared with a single machine learning method, the system maintained high sensitivity and specificity. The pathology omics integrated model showed that the negative predictive value (NPV) of all cohorts was greater than 0.87, and the positive predictive value (PPV) exceeded 0.89. Therefore, the gastric cancer prognosis system provided by the present invention has high reliability in detecting negative and positive results.

[0059] The present invention improves the classification accuracy of the feature extractor by creating a new pathomic feature extractor, which helps improve the performance of late-stage gastric cancer prognosis prediction. The pathomic feature extractor can accurately and more deeply extract case-based features.

[0060] In a preferred embodiment of the present invention, in a deep learning-based pathology feature extractor construction unit, each segmented image is annotated to correspond to an annotation map, and each segmented image is labeled with the tissue type that occupies the largest proportion of the corresponding area on the matching annotation map. Eight types of tissues related to the tumor microenvironment (TME) are annotated in each WSI: adipose tissue (ADI), debris (DEB), mucus (MUC), muscle (MUS), lymphocyte aggregates (LYM), stroma (STR), normal mucosa (NOR) and tumor epithelium (TUM).

[0061] In a preferred embodiment of the present invention, the proportion of segmented images annotated with eight types of tissues in some selected segmented images is adjusted according to actual needs. For example, the proportion of segmented images of the eight types of tissues is: ADI (12.71%), DEB (13.15%), LYM (11.82%), MUC (8.73%), MUS (15.58%), NOR (8.99%), STR (10.57%), and TUM (18.45%). Under the above proportions, the dataset can be trained at a low cost.

[0062] In a preferred embodiment of the present invention, the WSI image is segmented into segmented images of 224*224 pixels;

[0063] In a preferred embodiment of the present invention, a cross entropy loss function and an Adam optimizer are used during training, with an initial learning rate set to 0.001. The learning rate is adjusted during training using a learning rate decay strategy to accelerate convergence and prevent overfitting.

[0064] In a preferred embodiment of the present invention, the residual neural network architecture used is selected from: ResNet18, ResNet34 or ResNet50. In other embodiments, the residual neural network architecture may also be: ResNet-152 or ResNet-101.

[0065] In a preferred embodiment of the present invention, the residual neural network architecture used is ResNet18; the final fully connected classification layer in the basic network is removed, and only its pre-order convolution module and pooling structure are retained to create a universal feature extractor.

[0066] To ensure the best prediction performance, the present invention compared several network architectures, including ResNet18, ResNet34, and ResNet50. Among these architectures, ResNet18 showed the best performance, with the trained classifier showing an accuracy of 98.44% on the HMU-GC-HE-30K dataset.

[0067] The residual neural network architecture used is ResNet18; the final fully connected classification layer in the base network is removed, and only the pre-order convolution module and pooling structure in the base network are retained to create a universal feature extractor.

[0068] The pathology feature extraction unit is used to extract the feature matrix in the patient's representative region of interest (ROI) as pathology features based on the open source CellProfiler software.

[0069] Representative regions of interest (ROIs) were processed to quantify predefined morphological descriptors, generating an interpretable feature matrix.

[0070] In a preferred embodiment of the present invention, before training, the image is pre-processed by standardization using methods such as style transfer, grayscale conversion, histogram equalization, and elastic deformation.

[0071] In a preferred embodiment of the present invention, the pathology feature selection unit is configured to perform the following steps:

[0072] (1) Use maximum correlation and minimum redundancy algorithms to remove redundant features;

[0073] (2) performing a Pearson correlation matrix to eliminate features with low recurrence correlation from highly correlated paired features;

[0074] (3) Use the least absolute shrinkage with ten-fold cross validation and selector as the logical algorithm to reduce the features.

[0075] In a preferred embodiment of the present invention, the pathology feature selection unit obtains eight features.

[0076] In a preferred embodiment of the present invention, the multiple algorithms in the construction and verification unit of the pathology omics integrated model are selected from at least one of the following:

[0077] Voting regressor, random forest (RF), support vector machine (SVM), k-nearest neighbor, partial least squares (PLS), least squares support vector machine (LSSVM), BP neural network (BPNN), convolutional neural network (CNN), extreme learning machine (ELM), particle swarm optimization BP neural network (PSO-BP), recurrent neural network, Bayesian classifier, K-nearest neighbor algorithm, K-means algorithm, linear regression, logistic regression;

[0078] In a preferred embodiment of the present invention, the gastric cancer prognosis unit outputs at least one of the following information about the patient: survival period, risk of early postoperative recurrence, risk of complications, expected functional recovery, and quality of life prediction.

[0079] In a preferred embodiment of the present invention, the input unit is the first input unit, and the gastric cancer prognosis unit is the first gastric cancer prognosis unit.

[0080] In a second aspect, the present invention further provides a gastric cancer prognosis system based on multimodal deep learning, comprising:

[0081] Input unit: used to input independent clinical features related to gastric cancer recurrence and the gastric cancer prognosis results output by the above-mentioned pathological omics integrated model;

[0082] Multimodal prediction model construction unit: used to integrate independent clinical features related to gastric cancer recurrence and gastric cancer prognosis results obtained based on the pathological omics integrated model to construct a multimodal prediction model;

[0083] Gastric cancer prognostic unit: used to obtain the final gastric cancer prognosis results by integrating independent clinical features and pathological omics integrated models through a multimodal prediction model.

[0084] In a preferred embodiment of the present invention, the expression of the multimodal prediction model is:

[0085]

[0086] Among them, x1 is the pathological feature score, x2 is the clinical feature score, and p represents the multimodal prediction probability.

[0087] The above expression can also be expressed as follows:

[0088] logit(p)=7.441·PathScore+4.222·ClinicScore-5.949

[0089]

[0090] In a preferred embodiment of the present invention, the input unit is the second input unit, and the gastric cancer prognosis unit is the second gastric cancer prognosis unit.

[0091] In a preferred embodiment of the present invention, the independent clinical feature is selected from at least one of pathological type, invasion depth (pT stage), and lymph node metastasis (pN stage);

[0092] In a preferred embodiment of the present invention, the gastric cancer prognosis unit outputs at least one of the following information about the patient: survival period, risk of early postoperative recurrence, risk of complications, expected functional recovery, and quality of life prediction.

[0093] Preferably, the pathological type includes the gastric cancer type assessed by Lauren classification, Borrmann classification or TNM staging system.

[0094] In other embodiments, the pathological type may also be the ACRG classification given based on transcriptomic, genomic, and clinical data. The ACRG classification divides gastric cancer into four pathological subtypes.

[0095] In a third aspect, the present invention provides an application of a multimodal prediction model in constructing a TNM staging system.

[0096] In a fourth aspect, the present invention provides an electronic device comprising: a processor and a memory; the processor and the memory are connected, wherein:

[0097] The memory is used to store the computer program, and the processor is used to call the computer program to execute the above-mentioned method for gastric cancer prognosis or TNM staging based on multimodal deep learning.

[0098] Specifically, the electronic device may include a memory, a processor, a bus, and a communication interface, wherein the memory, processor, and communication interface are electrically connected to each other directly or indirectly to enable data transmission or interaction. For example, these components may be electrically connected to each other via one or more buses or signal lines. The processor may process information and / or data related to target identification to perform one or more functions described in this application.

[0099] The memory may be, but is not limited to, random access memory (RAM), read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), etc.

[0100] A processor can be an integrated circuit chip with signal processing capabilities. The processor can be a general-purpose processor, including a central processing unit (CPU) or a network processor (NP). It can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0101] In a fifth aspect, the present invention provides a computer-readable storage medium comprising instructions, which, when executed on a computer, enables the computer to execute the above-mentioned method for gastric cancer prognosis or TNM staging based on multimodal deep learning.

[0102] To make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention are described clearly and completely below. Where specific conditions are not specified in the embodiments, conventional conditions or conditions recommended by the manufacturer are used. Where the manufacturer of the reagents or instruments is not specified, all are conventional products that can be purchased commercially.

[0103] The features and performance of the present invention are further described in detail below with reference to the embodiments.

[0104] Experimental methods and materials:

[0105] (1) This retrospective cohort study utilized data from the Hebei Gastric Cancer Collaborative Network Database (http: / / hbss.suvalue.com / ), which prospectively collects gastric cancer data from large medical centers in Hebei Province, China. This study retrospectively analyzed patients with locally advanced gastric cancer in the database who were diagnosed and treated at four medical centers (the Fourth Hospital of Hebei Medical University, Shijiazhuang People's Hospital, Baoding Central Hospital, and Hengshui People's Hospital) between January 2012 and December 2019. Patients who were treated at two medical centers in southern China (Wuhan University People's Hospital and Nanjing Jinling Hospital) were also analyzed. Figure 1According to strict inclusion and exclusion criteria (see sodium exclusion criteria below for more details), a total of 1580 LAGC patients were recruited from six medical centers and divided into three different cohorts: the training set consisted of 855 LAGC patients recruited from the Fourth Hospital of Hebei Medical University from January 2014 to December 2017, and the internal validation set consisted of 410 patients. Based on the different consultation time of these patients, we divided them into internal validation set I (consultation time: January 2012-December 2013, N=215) and internal validation set II (consultation time: January 2018-December 2019, N=195). Patients from the remaining five medical centers served as an external validation set. Based on the patient's geographic location, the validation set was divided into external validation set I (patients in northern China, including Shijiazhuang People's Hospital, Baoding Central Hospital, and Hengshui People's Hospital; N = 286) and external validation set II (patients in southern China, including Wuhan University Renmin Hospital and Nanjing Jinling Hospital; N = 212). To further validate the clinical applicability of the model, we retrospectively analyzed a multicenter prospective clinical study (NCT01516944) as an additional validation step. All patients met strict inclusion and exclusion criteria and enrolled a total of 456 patients with LAGC, including 106 patients undergoing primary surgery and 350 patients receiving neoadjuvant chemotherapy. To further elucidate the biological significance of the model, we prospectively collected fresh tissue specimens from 120 patients with LAGC for exploratory transcriptome sequencing. Functional enrichment analysis was then performed to investigate the biological roles of the identified features. Details regarding RNA processing, sequencing, and analysis are provided in the Supplementary Methods. All procedures described in this paper conformed to the Declaration of Helsinki and were approved by the institutional review boards of all participating institutions. Written informed consent was not required for the retrospective portion of this observational study, whereas informed consent was obtained from each patient for the cohort undergoing transcriptome sequencing.

[0106] Inclusion and exclusion criteria:

[0107] The inclusion criteria were as follows: (1) aged between 21 and 18 years; (2) underwent enhanced computed tomography (CT) within 14 days before surgery; (3) was diagnosed with gastric cancer by postoperative pathology; and (4) underwent radical surgical resection.

[0108] Exclusion criteria included: (1) receiving neoadjuvant chemotherapy, targeted therapy, or immunotherapy before surgery; (2) surgery-related death (within 90 days after surgery); (3) loss to follow-up; (4) artifacts or missing CT images; (5) no pathological tissue staining (hematoxylin-eosin) images provided; and (6) lack of clinically relevant information.

[0109] Clinical data collection:

[0110] Within 14 days before surgery, detailed demographic and clinical data, including age, sex, Eastern Cooperative Oncology Group performance status, neutrophil count, lymphocyte count, platelet count, serum albumin, carcinoembryonic antigen (CEA), carbohydrate antigen 19-9 (CA19-9), carbohydrate antigen 72-4 (CA72-4), and postoperative treatment information, were extracted from electronic medical records. Pathological data, including tumor differentiation, lymph node metastasis, perineural invasion, lymphovascular invasion, surgical margin status, tumor size, tumor location, and expression of molecular markers such as HER2 and PD-L1, were obtained from pathology reports.

[0111] The pathological TNM stage was reassessed according to the 8th edition of the American Joint Committee on Cancer (AJCC) staging system. The following indices were calculated: neutrophil-to-lymphocyte ratio (NLR = neutrophils / lymphocytes), platelet-to-lymphocyte ratio (PLR = platelets / lymphocytes), prognostic nutritional index (PNI = albumin [g / L] + 5 × lymphocyte count), and systemic immune-inflammatory index (SII = platelets × neutrophils / lymphocytes).

[0112] RNA sequencing:

[0113] Bulk RNA sequencing was performed on 149 gastric cancer tissue samples.

[0114] cDNA synthesis and library construction were performed according to the specific chain experimental protocol (Guangzhou Gene Denno Company). The library quality was tested by agarose gel electrophoresis, nanophotometer spectrophotometer, Qubit 2.0 fluorometer and Agilent 2100 bioanalyzer. After passing the test, the Illumina Novaseq X Plus was used for detection. Fastp was used to assess the quality of sequencing data. HISAT2 software was used for alignment analysis based on the reference genome. Based on the HISAT2 alignment results, transcripts were reconstructed using Stringtie, and the expression levels of all genes in each sample were calculated using RSEM.

[0115] Differential expression analysis:

[0116] Differentially expressed genes (DEGs) between the high-risk and low-risk groups were identified using the edgeR software package. An overdispersion Poisson framework was used to model gene expression variability, and overdispersion was adjusted for using the empirical Bayesian method. Counts per million (CPM) values ​​were calculated, and only genes with a CPM greater than 2 in at least two samples were retained for analysis. Generalized linear models combined with likelihood ratio tests were used to assess expression changes. Genes were considered significantly differentially expressed if the P value was less than 0.05 and the absolute base-2 logarithm expression change value exceeded 1.0.

[0117] Functional enrichment analysis:

[0118] Gene set enrichment analysis (GSEA) was performed to explore biological differences between gene signatures. Fifty signature gene sets were selected from the Molecular Signature Database (MSigDB). Standard GSEA analysis was performed on normalized expression data, with 1000 random permutations. Pathways with a P-value less than 0.05 were considered significantly enriched.

[0119] Immune activity assessment:

[0120] Tumor immune activity was assessed using the ESTIMATE algorithm, which calculates an immune score for each sample based on gene expression profiles to reflect the degree of immune cell infiltration. Differences in immune scores between high-risk and low-risk groups were compared using the R package limma, with statistical significance defined as P < 0.05.

[0121] Immune infiltration:

[0122] CIBERSORT is an unmixing algorithm that uses the expression profiles of 547 genes to estimate the proportion of immune cells in tissues. In this study, we applied CIBERSORT to quantify the relative proportions of 22 infiltrating immune cell types in gastric cancer samples based on gene expression data and performed 1000 random permutations. A P value of less than 0.05 was considered statistically significant.

[0123] To further assess the infiltration of immune and stromal cells, we used MCPcounter, a tool that estimates the abundance of eight immune-cell populations, including CD4-positive T cells, CD8-positive T cells, natural killer cells, B lymphocytes, monocytes, myeloid dendritic cells, neutrophils, and cytotoxic lymphocytes, as well as two stromal-cell types that constitute fibrocytes and endothelial cells.

[0124] In addition, we used EcoTyper, a computational framework designed to identify cell states and multicellular ecosystems (CEs) from large-scale transcriptomic data. Using the EcoTyper v1.0 pipeline (https: / / ecotyper.stanford.edu / ), we characterized the distribution of immune-related cell states and CEs in gastric cancer samples based on their co-occurrence patterns across multiple cancer types.

[0125] (2) Clinical results:

[0126] All patients in this invention refer to the guidelines of the Chinese Society of Clinical Oncology (CSCO) and the National Comprehensive Cancer Network (NCCN) after surgery. If the patient's physical condition is good, it is recommended that all patients receive adjuvant chemotherapy based on 5-fluorouracil, and they should also receive systematic postoperative follow-up. Follow-up content includes comprehensive medical history and physical examination, follow-up once every 3 to 6 months within the first 1 to 2 years after surgery, and then every 6 months within the third to fifth years after surgery, and once a year thereafter; at the same time, blood routine, serum biochemical tests, imaging examinations or endoscopic examinations are performed according to clinical conditions. In this study, early postoperative recurrence is defined as recurrence occurring within 24 months of follow-up after radical surgery of LAGC patients. All recurrences are determined by abdominal ultrasound, CT or PET / CT, clinical symptoms or reoperation. Overall survival is defined as the time of death due to any cause. In this study, the primary endpoint is early postoperative recurrence and the secondary endpoint is overall survival. Follow-up of all patients in this study was completed in June 2024, with a median follow-up time of 69 months (range, 58 to 165 months).

[0127] (3) Pathological feature selection:

[0128] Hematoxylin and eosin (H&E)-stained sections were prepared from formalin-fixed, paraffin-embedded specimens for all enrolled patients. The director of the Department of Pathology at the Fourth Hospital of Hebei Medical University (YPL), who has 25 years of experience in gastric cancer pathology, selected the sections that best represented the depth of tumor invasion for each case. All hematoxylin and eosin (HE)-stained sections were scanned using a PRECICE 600 fully automated digital slide scanner (Chongqing, China), and the WSI images were managed using the scanner's dedicated image analysis software, iViewer. Tumor area was determined in each section under quality control by the director of the Department of Pathology at the Fourth Hospital of Hebei Medical University. For each case, a single pathologist selected 10 non-overlapping representative blocks, each containing the most tumor cells, with a field of view of 1000 × 1000 pixels (one pixel equals 0.504 μm). These blocks were then confirmed by two other pathologists (FL and HYD), who have 15 and 10 years of experience in gastric cancer pathology, respectively, to reduce computational time. Tissue folds were excluded, and the selected blocks were saved as .tif files. If the two pathologists disagreed, they consulted a third pathologist (YPL) to make a decision.

[0129] (4) Biological characteristics and immune infiltration:

[0130] A total of 120 tissue samples were prospectively collected and subjected to RNA sequencing for exploratory analyses. Functional enrichment analysis was performed to investigate the biological roles of identified features. Immune infiltration was assessed using CIBERSORTx to estimate the relative abundance of immune cell subsets based on RNA transcriptome data. In addition, the EcoTyper framework was used to infer cell types, cell states, and multicellular ecosystems. Detailed procedures for RNA processing, sequencing, and data analysis are provided in the Supplementary Methods.

[0131] (5) Statistical analysis

[0132] All statistical analyses were performed using SPSS version 28.0 (IBM) and R version 4.3.3 (http: / / www.r-project.org). Continuous variables were analyzed using unpaired two-tailed t-tests and Mann-Whitney U tests, while variables were analyzed using the X-test. 2 Categorical variables were evaluated using the statistician's exact test and Fisher's exact test. Univariate and multivariate Cox regression analyses were performed to assess the prognostic ability of variables for survival. A two-sided P value of less than 0.05 was considered statistically significant.

[0133] (6) Patient characteristics

[0134] A total of 1763 eligible patients with LAGC were enrolled in this study, including 855 patients (48.5%) in the training cohort, 410 patients (23.3%) in the internal validation cohort, and 498 patients (28.2%) in the external validation cohort. Detailed clinicopathological characteristics of patients in the different cohorts across the six medical centers are summarized in Table 1.

[0135] Table 1 Characteristics of patients in the training group, internal validation group, and external validation group [Percentage (%)]

[0136]

[0137] Note: ECOGPS = Eastern Cooperative Oncology Group Performance Status Score; SII = Systemic Immuno-Inflammatory Index; PNI = Prognostic Nutrition Index; NLR = Neutrophil-to-Lymphocyte Ratio; PLR = Platelet-to-Lymphocyte Ratio. * Median values ​​were used as the cutoff for stratification into high-risk and low-risk groups.

[0138] Most characteristics were similar and well-balanced across the different cohorts (all P > 0.05). Among all patients with LAGC enrolled in this study, 1154 (65.5%) were male and 544 (34.5%) were female, with a median age of 58.0 years (IQR 45.0–66.0). In the training cohort, 451 patients (52.7%) experienced early postoperative recurrence. Early recurrence occurred in 222 patients (54.1%) in the internal validation cohort and 281 patients (56.4%) in the external validation cohort. Thus, the prevalence of early postoperative recurrence was very similar across the three cohorts. Furthermore, a comparison of the clinicopathological characteristics of patients who experienced recurrence versus those who did not across the different cohorts is summarized in Table 2.

[0139] Table 2 Statistical results of clinical characteristics of patients according to peritoneal recurrence in internal and external validation cohorts [n (%)].

[0140]

[0141] Note: ECOGPS = Eastern Cooperative Oncology Group Performance Status Score; SII = Systemic Immuno-Inflammatory Index; PNI = Prognostic Nutrition Index; NLR = Neutrophil-to-Lymphocyte Ratio; PLR = Platelet-to-Lymphocyte Ratio. * Median values ​​were used as the cutoff for stratification into high-risk and low-risk groups.

[0142] Example 1

[0143] This embodiment provides a method for constructing a pathology omics integrated model and a gastric cancer prognosis system based on the pathology omics integrated model. The schematic diagram of the gastric cancer prognosis system is shown in FIG. Figure 8 As shown, it includes: a first input unit 301, a pathological feature extraction unit 302, a deep learning-based pathological genomics feature extractor construction unit 303, a deep learning-based pathological genomics feature extraction unit 304, a pathological genomics feature selection unit 305, a pathological genomics integrated model construction and verification unit 306 and a first gastric cancer prognosis unit 307.

[0144] The first input unit 301 is used to obtain a gastric cancer pathological image of a patient;

[0145] Pathological feature extraction unit 302: used for extracting pathological features from gastric cancer pathological images of patients based on CellProfiler tool;

[0146] A deep learning-based pathology feature extractor construction unit 303 is configured to segment the WSI image to obtain segmented images, each of which is annotated; select a portion of the segmented images and divide them into a training set and a validation set; and adopt a residual neural network architecture to train and obtain a feature extractor.

[0147] Deep learning-based pathonomics feature extraction unit 304: used to flatten all WSI images into segmented images of the same size as the segmented images used during training, put them into a feature extractor, use the result output value of the feature extractor as a deep learning feature, and sum and average the feature information of the segmented images belonging to the same WSI image to generate a deep learning pathonomics feature;

[0148] Pathological genomics feature selection unit 305: configured to select recurrence-related pathological genomics features based on the pathological genomics features extracted by the pathological feature extraction unit and the deep learning-based pathological genomics feature extraction unit;

[0149] Pathology omics integrated model construction and verification unit 306: used to construct a pathology omics model for gastric cancer prognosis using multiple algorithms, and average the single WSI image predictions of each basic regressor using a voting regressor to generate the pathology omics integrated model; the basic regressor refers to the predicted probability value of each algorithm;

[0150] The first gastric cancer prognosis unit 307 is used to input the patient's gastric cancer pathological image into the pathology omics integrated model to obtain the predicted patient prognosis result.

[0151] The construction method of the pathomic integrated model includes the following steps:

[0152] (1) Extract pathological features from images using CellProfiler.

[0153] We first extracted quantitative pathomic features of the selected blocks using CellProfiler (version 4.2.6), an open-source image analysis software developed by the Broad Institute (Cambridge, MA). The H&E-stained images were separated into grayscale images of hematoxylin and eosin staining using the “UnmixColors” module. The H&E-stained images were also converted into grayscale images using the “ColorToGray” module based on the “Combine” method for further analysis. First, features representing the image quality of grayscale H&E, hematoxylin, and eosin images were evaluated using the “MeasureImageQuality” and “MeasureImageIntensity” modules, which contain three types of features, including blur features, intensity features, and threshold features. Threshold features were extracted by automatically calculating the threshold of each image to identify tissue foreground from unstained background using the Otsu algorithm. Subsequently, the colocalization and correlation between the intensities of each hematoxylin-stained image and eosin-stained image in the entire image were calculated pixel by pixel using the “MeasureColocalization” module. In addition, the granularity characteristics of each image are evaluated using the “MeasureGranularity” module, which outputs a spectrum of texture size measurements in the image with a granularity spectrum range of 16.

[0154] (2) Deep learning tissue classifier training and verification.

[0155] We also leveraged deep learning techniques to further extract key pathomic features. We tiled 50 WSI images into 224x224 pixel patches, each annotated to correspond to an annotation map. Each patch was labeled with the tissue type that occupies the largest proportion of the corresponding area on the matching annotation map. Eight tissue types associated with the tumor microenvironment (TME) were annotated for each slide image: adipose tissue (ADI), debris (DEB), mucus (MUC), muscle (MUS), lymphocyte aggregates (LYM), stroma (STR), normal mucosa (NOR), and tumor epithelium (TUM). Because we generated a massive dataset of over 5,000,000 patches, significantly larger than the original ImageNet dataset, training on the entire dataset would be prohibitively expensive. Therefore, we decided to randomly select 400,000 patches (ADI (12.71%), DEB (13.15%), LYM (11.82%), MUC (8.73%), MUS (15.58%), NOR (8.99%), STR (10.57%), TUM (18.45%). Then divide them into training and validation sets in a ratio of 6:4, and use HMU-GC-HE-30K as the external test set. In this study, we did not use overlapping patches. Non-square patches at the boundaries were excluded because most of these patches are blank. The images were normalized and pre-processed by applying methods such as style transfer. Subsequently, during the training process, the NVIDIA RTX The training was run on a 4090 GPU using the cross-entropy loss function and the Adam optimizer with a learning rate of 0.001. Training was performed for 50 epochs. To ensure optimal prediction performance, we compared several network architectures, including ResNet18, ResNet34, and ResNet50. Among these architectures, ResNet18 demonstrated the best performance, with the trained classifier achieving 98.44% accuracy on the HMU-GC-HE-30K dataset (Table 3). This study employed the ResNet18 architecture, removing the final fully connected layer to create the feature extractor.

[0156] Table 3 Performance of three ResNet processing methods for gastric cancer prediction in holographic ultramicroscopic images (WSI) on the HMU-GC-HE-30K dataset

[0157]

[0158] Note:AUC=area under the curve:ACC=accuracy;SENS=sensitivity,SPEC=specificity:Cl=confidence interval.

[0159] (3) Deep learning pathogenomics feature extraction.

[0160] We flattened all WSIs into patches of the same size as those used during training, preserving their respective WSI associations. The processed patches were then fed into a feature extractor, whose output was used as deep learning features. The feature information of patches belonging to the same WSI image was summed and averaged to generate WSI-level feature data. A total of 512 deep learning pathomic features were extracted.

[0161] (4) Pathological feature selection

[0162] A total of 794 features were obtained through the two feature extraction methods of deep learning and CellProfiler. We designed a three-step scheme to select recurrence-related pathological omics features based on the training cohort. First, the maximum correlation and minimum redundancy (mRMR) algorithm was used to remove redundant features; second, the Pearson correlation matrix was performed to eliminate features with low recurrence correlation among highly correlated paired features. Finally, the least absolute shrinkage and selector operation (LASSO) logistic algorithm with ten-fold cross validation was used to further narrow the features ( Figure 10 Finally, eight features were obtained (Table 4).

[0163] Table 4 Statistical results of multivariate pathological characteristics of recurrence in patients with advanced gastric cancer

[0164]

[0165] (5) Development and validation of integrated models driven by pathogenomics

[0166] An ensemble strategy was employed to predict early postoperative recurrence of LAGC. Specifically, the ensemble approach utilized multiple algorithms, including logistic regression, decision trees, random forests, support vector machines, and voting regressors. The voting regressor is an ensemble meta-estimator that is fitted with base regressors using the entire dataset. The individual WSI image predictions from each base regressor were averaged using the voting regressor to generate the pathomic ensemble model. The base regressor refers to the predicted probability value of each algorithm.

[0167] Based on the above-mentioned patho-omics integrated model, early postoperative recurrence of LAGC patients was predicted and evaluated. The specific prediction and evaluation methods are as follows:

[0168] Univariate logistic regression analysis was used to assess the association between clinical parameters and early postoperative recurrence in patients with LAGC in the training cohort. Multivariate logistic regression was then used to establish a clinical model based on clinical parameters to predict early postoperative recurrence in patients with LAGC. By combining independent clinical parameters associated with recurrence with pathomic features, we developed a nomogram based on multimodal data fusion, defined as the RSA model. The model-predicted early postoperative recurrence risk score was dichotomized into low and high scores, and the optimal cutoff value was selected based on the Youden index in the training cohort, which maximized the sum of sensitivity and specificity. Evaluation metrics, including the area under the receiver operating characteristic (ROC) curve (AUC), accuracy, sensitivity, specificity, positive predictive value (PPV), negative predictive value (NPV), positive likelihood ratio (PLR), negative likelihood ratio (NLR), and F1 score, were calculated. To quantify the relative improvement in predictive accuracy, the net reclassification improvement (NRI) and integrated discrimination improvement index (IDI) were calculated, and the area under the curve (AUC) was compared between different models using the Delong test. In addition, calibration curves were drawn to assess the consistency between the recurrence rates predicted by the nomogram and the actual observed rates, and the overall performance of these models was evaluated using prediction error curves and the comprehensive Brier score.

[0169] Experimental results:

[0170] First, we developed a pathomic-driven integrated model to predict early postoperative recurrence in LAGC patients ( Figure 1 Compared with the prediction of a single machine learning method, the integrated model showed good prediction accuracy for early postoperative recurrence in the training cohort, with an area under the curve (AUC) of 0.853 (95% CI: 0.828-0.878) ( Figure 10 In addition, the ensemble model showed significant sensitivity of 0.816 and specificity of 0.740 in the training cohort, as shown in Table 5.

[0171] Table 5 Comparison of performance indicators of different models in predicting postoperative recurrence of LAGC patients (in the training set).

[0172]

[0173] Note: AUC - area under the curve; PPV - positive predictive value; NVP - negative predictive value; PLR - positive likelihood ratio; NLR - negative likelihood ratio; LR - logistic regression; RF - random forest; SVM - support vector machine; DT - decision tree.

[0174] After successful development in the training cohort, the integrated model showed high accuracy in predicting postoperative recurrence with an AUC of 0.843 (95% CI: 0.790-0.895) in the internal validation cohort I and an AUC of 0.847 (95% CI: 0.791-0.904) in the internal validation cohort II ( Figure 11 The integrated model maintained high sensitivity (95% CI: 0.857-0.917) and specificity (95% CI 0.78 to 0.938) compared with the single model in two other external validation cohorts ( Figure 11 The integrated model showed that the negative predictive value (NPV) of all cohorts was greater than 0.87, and the positive predictive value (PPV) exceeded 0.89.

[0175] Example 2

[0176] This embodiment provides a gastric cancer prognosis system based on multimodal deep learning, the schematic diagram of which is shown in FIG. Figure 9 As shown, it includes: a second input unit 401, a multimodal prediction model construction unit 402 and a second gastric cancer prognosis unit 403.

[0177] The second input unit 401 is used to input independent clinical features related to gastric cancer recurrence and the gastric cancer prognosis result output by the above-mentioned pathogenomics integrated model;

[0178] Multimodal prediction model construction unit 402: used to integrate independent clinical features related to gastric cancer recurrence and gastric cancer prognosis results obtained based on the pathological omics integrated model to construct a multimodal prediction model;

[0179] The second gastric cancer prognosis unit 403 is used to obtain the final gastric cancer prognosis result by fusing independent clinical features and the gastric cancer prognosis result obtained based on the pathological omics integrated model through a multimodal prediction model.

[0180] The construction method of the multimodal prediction model is as follows:

[0181] Based on the prognostic results of gastric cancer obtained from the patho-omics integrated model and independent clinical features associated with gastric cancer recurrence, an integrated algorithm called logistic regression was used to construct a model called RSA (i.e., a multimodal prediction model). The output of RSA is the predicted probability of recurrence in patients with locally advanced gastric cancer.

[0182] The expression of the multimodal prediction model is:

[0183]

[0184] Among them, x1 is the pathological feature score, x2 is the clinical feature score, and p represents the multimodal prediction probability.

[0185] When multivariate logistic regression analysis was performed on clinical characteristics and pathological features, pathological type, invasion depth (pT stage), lymph node metastasis (pN stage), and pathological features remained independent risk factors for predicting early recurrence in patients with LAGC after surgery (Table 6).

[0186] Table 6 Univariate and multivariate logistic regression analysis of factors influencing postoperative recurrence in patients with advanced gastric cancer (LAGC) in the training set

[0187]

[0188]

[0189] Note: ECOGPS = Eastern Cooperative Oncology Group Performance Status Score; SII = Systemic Immuno-Inflammatory Index; PNI = Prognostic Nutrition Index; NLR = Neutrophil-to-Lymphocyte Ratio; PLR = Platelet-to-Lymphocyte Ratio. *Divided into high-risk and low-risk groups using the median as the cutoff.

[0190] Therefore, using the above variables and their regression coefficients, this example constructed a comprehensive nomogram to evaluate the development of a risk stratification assessment (RSA) model that combines pathological omics features with clinical variables ( Figure 2 Middle A), the model showed strong predictive ability for early recurrence in patients with LAGC (AUC was 0.887, 95% CI: 0.864–0.909; Figure 2 Table 7 provides a detailed comparison of the predictive performance between the different models. We further quantified the improvement in recurrence prediction accuracy between the comprehensive nomogram and the single-feature model and found that the RSA model improved the NRI by 0.531 (95% CI: 0.397-0.668; P < 0.001) and the IDI by 0.273 (95% CI: 0.244-0.304; P < 0.001) compared with the clinical feature model (Table 8).

[0191] Table 7 Comparison of performance indicators of different models in predicting postoperative recurrence of LAGC patients (in the training set)

[0192]

[0193] Note: AUC - area under the curve; PPV - positive predictive value; NVP - negative predictive value; PLR - positive likelihood ratio; NLR - negative likelihood ratio.

[0194] Table 8 Proportion of different models in predicting postoperative recurrence in LAGC patients

[0195]

[0196]

[0197] In addition, the DeLong test showed that the AUC of the RSA model was significantly higher than that of the clinical model in the training set (0.887 vs. 0.746; P < 0.001, Table 8). The calibration curve of the model further highlighted its predictive accuracy ( Figure 2 Middle F).

[0198] According to the Youden index, patients were divided into low-risk and high-risk groups. The two-layer concentric circle diagram showed that the RSA model identified more low-risk recurrence patients than the clinical model (46.7% vs. 42.1%) ( Figure 2 Middle H).

[0199] In addition, the decision curve further confirmed that the RSA model had the greatest clinical benefit ( Figure 2 Follow-up data showed that the 5-year OS of high-risk patients was significantly lower than that of low-risk patients (42.8% vs. 59.4%; P < 0.001) ( Figure 2 Middle G).

[0200] Next, this example performed a multivariate Cox regression analysis, adjusting for clinicopathological variables including TNM stage and histological subtype, and found that the RSA model remained an independent prognostic factor for 5-year OS in patients with imaging-based LAGC (Table 9). Stratified analysis based on the expression of different peripheral blood tumor markers and the molecular marker HER2 in tissue biopsies showed that the RSA model was still superior to the single feature model in predicting early recurrence ( Figure 2 JK, Figure 12 , Table 10). At the same time, the 5-year OS of patients with high-risk RSA model was lower than that of patients with low-risk RSA model in different stratification analyses ( Figure 13 ).

[0201] Table 9 Univariate and multivariate Cox regression analysis of factors affecting the 5-year overall survival rate of patients with advanced gastric cancer who underwent surgical treatment in the training set

[0202]

[0203]

[0204] Note: ECOGPS = Eastern Cooperative Oncology Group Performance Status Score; SII = Systemic Immuno-Inflammatory Index; PNI = Prognostic Nutrition Index; NLR = Neutrophil-to-Lymphocyte Ratio; PLR = Platelet-to-Lymphocyte Ratio. *Divided into high-risk and low-risk groups using the median as the cutoff.

[0205] Table 10 Comparison of the performance of different models in predicting postoperative recurrence of LAGC patients in the training set based on different molecular markers HER2 expression patterns and peripheral blood tumor markers

[0206]

[0207] Note: AUC – area under the curve; PPV – positive predictive value; NVP – negative predictive value; PLR – positive likelihood ratio; NLR – negative likelihood ratio.

[0208] The AUC box plot generated by ten-fold cross validation showed that the RSA model had better discrimination ability, sensitivity and specificity ( Figure 2 In addition, 1000 bootstrap resampling methods further confirmed that the RSA model had the best diagnostic performance for predicting recurrence (Mean AUC = 0.8864, 95% CI: 0.8857-0.8871) compared with the single pathological feature prediction model (Mean AUC = 0.8531, 95% CI: 0.8523-0.8539) ( Figure 2 Middle M).

[0209] Example 3

[0210] This embodiment performs internal verification on the multimodal prediction model (ie, the RSA model) established in Example 2.

[0211] To further validate the accuracy of the RSA model, we retrospectively analyzed 410 LAGC patients who were admitted to the Fourth Hospital of Hebei Medical University as an internal validation set. Based on the different consultation times of these patients, we divided them into internal validation set I (consultation time: January 2012-January 2014, N=215) and internal validation set II (consultation time: January 2017-January 2019, N=195) ( Figure 3 The clinical characteristics of the two groups are summarized in Table 11. There were no significant differences between the two groups (all P>0.05). In the internal validation set I, the RSA model showed better predictive ability compared with the clinical feature model and the pathological feature model (AUC=0.881, 95% CI: 0.833–0.929; Figure 3 Middle BC, Figure 3The EF in the RSA model was significantly improved (P < 0.001, Table 12). The Delong test found that P < 0.001 (Table 8). In addition, compared with the clinical feature model, the RSA model improved the NRI by 0.531 (95% CI: 0.396-0.675; P < 0.001) and the IDI by 0.273 (95% CI: 0.244-0.303; P < 0.001, Table 8). Calibration curve analysis further confirmed the enhanced predictive accuracy (Brier Score: 0.136, Figure 3 Similar results were observed in the second internal validation set ( Figure 3 Central KL, Figure 3 No. in Table 12).

[0212] Table 11 Comparison of clinical characteristics of LAGC patients admitted to the hospital at different time periods in the internal validation set

[0213]

[0214]

[0215] Note: ECOGPS = Eastern Cooperative Oncology Group Performance Status Score; SII = Systemic Immuno-Inflammatory Index; PNI = Prognostic Nutrition Index; NLR = Neutrophil-to-Lymphocyte Ratio; PLR = Platelet-to-Lymphocyte Ratio. *Divided into high-risk and low-risk groups using the median as the cutoff.

[0216] Table 12 Comparison of the performance indicators of different models in predicting postoperative recurrence in LAGC patients in the internal validation set

[0217]

[0218] Note: AUC - area under the curve; PPV - positive predictive value; NVP - negative predictive value; PLR - positive likelihood ratio; NLR - negative likelihood ratio.

[0219] The double-layer concentric circle plots showed that in both internal validation sets, the RSA model identified more early recurrence patients in the high-risk group compared with the clinical feature model (internal validation set I: 41.9% vs. 37.7%, Figure 3 Middle H; Internal validation set II: 50.3% vs. 39.5%, Figure 3 In addition, the DCA curves show that the RSA model provides the best performance in both internal validation sets ( Figure 3 Middle G, Figure 3The follow-up survival data of all LAGC patients in the two internal validation cohorts showed that the 5-year OS rate of low-risk patients stratified by the RSA model was significantly higher than that of high-risk patients (validation set I: 57.3% vs. 34.4%, P < 0.001; validation set II: 62.8% vs. 42.0%, P = 0.001, Figure 3 Next, multivariate Cox regression analysis in two internal validation sets revealed that the RSA model remained an independent prognostic factor affecting the 5-year OS of LAGC patients (Tables 13-14).

[0220] Table 13 Univariate and multivariate Cox regression analysis of factors influencing 5-year overall survival of LAGC patients in internal validation set I (2012-2014)

[0221]

[0222] Note: ECOGPS = Eastern Cooperative Oncology Group Performance Status Score; SII = Systemic Immuno-Inflammatory Index; PNI = Prognostic Nutrition Index; NLR = Neutrophil-to-Lymphocyte Ratio; PLR = Platelet-to-Lymphocyte Ratio. * Median values ​​were used as the cutoff for stratification into high-risk and low-risk groups.

[0223] Table 14 Univariate and multivariate Cox regression analysis of factors affecting 5-year overall survival of LAGC patients in the internal validation set II (2017-2019)

[0224]

[0225] Note: ECOGPS = Eastern Cooperative Oncology Group Performance Status Score; SII = Systemic Immuno-Inflammatory Index; PNI = Prognostic Nutrition Index; NLR = Neutrophil-to-Lymphocyte Ratio; PLR = Platelet-to-Lymphocyte Ratio. * Median values ​​were used as the cutoff for stratification into high-risk and low-risk groups.

[0226] Example 4

[0227] Multicenter external validation of the RSA model.

[0228] To further validate the generalizability of the model, we first selected 286 LAGC patients from three medical centers in northern China (SJZPH, BDCH, and HSCPH) as the first external validation cohort (external validation set I). To account for regional differences, we also selected 212 patients from two medical centers in southern China (WHPH and NJJLH) as the second external validation cohort (external validation set II) ( Figure 4The clinicopathological characteristics of the two external validation cohorts are summarized in Table 15. In external validation set I, the RSA model had the best ability to predict early postoperative recurrence in LAGC patients compared with the single-feature model (AUC = 0.882, 95% CI: 0.841–0.923; Figure 4 Middle BC, Figure 4 In addition, compared with the clinical characteristics model, the RSA model improved the NRI by 0.643 (95% CI: 0.373-0.887; P < 0.001) and the IDI by 0.263 (95% CI: 0.208-0.315; P < 0.001, Table 8). Calibration curve analysis further confirmed its predictive accuracy (BrierScore = 0.123, Figure 4 The same results were observed in the second external validation set ( Figure 4 Central KL, Figure 4 NO, Table 16).

[0229] Table 15 Comparison of clinical characteristics of LAGC patients in the external validation group

[0230]

[0231] Table 16 Comparison of the performance indicators of different models in predicting postoperative recurrence of LAGC patients on the external validation set

[0232]

[0233] Note: AUC – area under the curve; PPV – positive predictive value; NVP – negative predictive value; PLR – positive likelihood ratio; NLR – negative likelihood ratio.

[0234] like Figure 4 Middle G and Figure 4 As shown in Figure 1, DCA curve analysis found that the RSA model provided the best clinical benefit in both external validation sets. In addition, the double-layer concentric circle diagram further found that in both validation sets, compared with the single clinical feature model, the RSA model was able to identify more early recurrence patients in the high-risk group (external validation set I: 49.7% vs. 40.9%; external validation set II: 46.4% vs. 39.6%) ( Figure 4 Follow-up showed that the 5-year OS rate of high-risk patients was significantly worse than that of low-risk patients (validation set I: 38.6% vs. 59.3%, P < 0.001; validation set II: 40.4% vs. 61.2%, P < 0.001, Figure 4 Middle H, Figure 4Multivariate Cox regression analysis found that the RSA model was an independent prognostic factor affecting the 5-year OS of patients in the two external validation sets (Tables 17-18).

[0235] Table 17 Univariate and multivariate Cox regression analysis of factors influencing 5-year overall survival in patients with LAGC who underwent surgical treatment in external validation I (Northern region)

[0236]

[0237]

[0238] Note: ECOGPS = Eastern Cooperative Oncology Group Performance Status Score; SII = Systemic Immuno-Inflammatory Index; PNI = Prognostic Nutrition Index; NLR = Neutrophil-to-Lymphocyte Ratio; PLR = Platelet-to-Lymphocyte Ratio. * Median values ​​were used as the cutoff for stratification into high-risk and low-risk groups.

[0239] Table 18 Univariate and multivariate Cox regression analysis of factors affecting 5-year overall survival in patients with LAGC (external validation II)

[0240]

[0241]

[0242] Note: ECOGPS = Eastern Cooperative Oncology Group Performance Status Score; SII = Systemic Immuno-Inflammatory Index; PNI = Prognostic Nutrition Index; NLR = Neutrophil-to-Lymphocyte Ratio; PLR = Platelet-to-Lymphocyte Ratio. *Divided into high-risk and low-risk groups using the median as the cutoff.

[0243] Example 5

[0244] Revalidation of the RSA model in a prospective study.

[0245] To verify the performance of the RSA model in prospective studies, we retrospectively analyzed a population of LAGC patients enrolled in a multicenter prospective clinical study (NCT 01516944). According to strict inclusion and exclusion criteria, 456 patients were finally included, including 106 patients who underwent primary surgery, 168 patients who received neoadjuvant chemotherapy with the XELOX regimen (oxaliplatin + capecitabine), and 182 patients who received the SOX regimen (oxaliplatin + 1-1). Figure 5 The clinical characteristics of all patients are summarized in Table 19. In addition, the clinical outcomes of the present invention are analyzed in detail in our previous literature.

[0246] Table 19 Comparison of clinical characteristics of patients with LAGC (locally advanced gastric cancer) in the prospective validation cohort (NCT 01516944)

[0247]

[0248] Note: ECOGPS = Eastern Cooperative Oncology Group Performance Status Score; SII = Systemic Immuno-Inflammatory Index; PNI = Prognostic Nutrition Index; NLR = Neutrophil-to-Lymphocyte Ratio; PLR = Platelet-to-Lymphocyte Ratio. * Median values ​​were used as the cutoff for stratification into high-risk and low-risk groups.

[0249] First, we analyzed the cohort of patients undergoing primary surgery and found that the RSA model still had the best ability to predict early postoperative recurrence compared with the single feature model (AUC = 0.873, 95% CI: 0.806–0.940; Figure 5 Compared with the clinical feature model, the RSA model improved NRI by 0.708 (95% CI: 0.360-1.075; P < 0.001) and IDI by 0.264 (95% CI: 0.179-0.352; P < 0.001). Similar results were found when the RSA model was compared with the pathological feature model (Table 8). The double-layer concentric circle diagram showed that the RSA model identified more early recurrence patients in the high-risk group compared with the clinical feature model (42.3% vs. 37.6%, Figure 5 In addition, the DCA curve shows that the RSA model provides the best clinical benefit performance compared with the single feature model ( Figure 5 According to the RSA model, the 5-year OS rate of low-risk patients was significantly higher than that of high-risk patients (63.5% vs. 38.9%, P < 0.001, Figure 5 Consistent with the above analysis method, we performed the same analysis on the patient cohort receiving neoadjuvant chemotherapy and found that the RSA model had poor performance in predicting early postoperative recurrence (AUC = 0.741, 95% CI: 0.689–0.793; Figure 5 JO, Table 8, Table 20), and no significant correlation with the efficacy of neoadjuvant therapy ( Figure 5 Further follow-up revealed that the 5-year OS rate of patients with high risk in the RSA model was significantly lower than that of patients with low risk (41.8% vs. 58.3%, P < 0.001, Figure 5 (in P).

[0250] Table 20 Comparison of the performance indicators of different models in predicting postoperative recurrence in LAGC patients in the prospective validation group

[0251]

[0252] Note: AUC - area under the curve; PPV - positive predictive value; NVP - negative predictive value; PLR - positive likelihood ratio; NLR - negative likelihood ratio.

[0253] Example 6

[0254] Validation of the RSA model in the TCGA dataset.

[0255] To verify the clinical application performance of the RSA model in different ethnic groups, we selected 442 GC patients included in the TCGA dataset. According to the inclusion and exclusion criteria, 251 GC patients were finally included for model validation ( Figure 6 In addition, among the 251 patients, except for 85 cases without TCGA molecular typing information, the molecular typing was CIN type in 83 cases (33.1%), EBV type in 17 cases (6.8%), GS type in 34 cases (13.5%), and MSI type in 32 cases (12.7%) ( Figure 6 Middle A).

[0256] Consistent with the previous validation results of the present invention, in the TCGA cohort, the accuracy of the RSA model in predicting postoperative recurrence (AUC = 0.782, 95% CI: 0.711-0.852) was significantly better than the single clinical feature model (AUC = 0.619, 95% CI: 0.548-0.689) ( Figure 6 medium BE, Table 21).

[0257] Table 21 Comparison of performance indicators of different models in predicting postoperative recurrence in LAGC patients in the prospective validation group

[0258]

[0259] The results of the double-layer concentric circles showed that the RSA model could identify more patients with low-risk recurrence (71.3% vs. 53.0%) than the clinical feature model, and could also identify more patients with recurrence in the high-risk population (23.5% vs. 19.9%) ( Figure 6 The DCA curve further confirmed that the RSA model can benefit more patients ( Figure 6 In addition, stratified analysis based on different molecular analyses of TCGA revealed that the RSA model still had a high diagnostic performance ( Figure 6 Middle I, Figure 14 , Table 22). Further analysis of the survival data revealed that the 5-year OS of patients in the high-risk group of the RSA model was significantly worse than that of patients in the low-risk group (38.9% vs. 63.5%, Figure 6 Middle H).

[0260] Table 22 Comparison of performance indicators of different models in predicting postoperative recurrence in LAGC patients in the prospective validation group

[0261]

[0262] Note: CIN - chromosomal instability; EBV - Ebola virus positive; GS - genomic stability; MSI - microsatellite instability.

[0263] In addition to the TCGA molecular classification, the common molecular classification standards for gastric cancer also often use the classification proposed by the Asian Cancer Research Group (ACRG) in 2015 to achieve accurate classification. We tried to randomly select 217 LAGC patients from the training set for immunohistochemistry, and the expression of E-cadherin, Vimentin, N-cadherin, MLH1, MSH2, MSH6, PMS2 and P53 proteins in LAGC patients, and then divided these LAGC patients into one of four subtypes based on the main expression of the eight proteins. After immunohistochemistry, the 217 patients were found to have 45 cases (20.7%) of MSI subtype; 38 cases (17.5%) of MSS / EMT subtype; 80 cases (36.9%) of MSS / TP53- subtype; 54 cases (24.8%) of MSS / TP53+ subtype ( Figure 6 Interestingly, the RSA model showed higher diagnostic performance compared with the single feature model in the four different ACRG classifications ( Figure 6 KM, Table 23). It is worth noting that in different ACRG classifications, the survival analysis of high-risk and low-risk groups based on the RSA model showed that the 5-year OS of patients in the high-risk group was significantly worse than that of patients in the low-risk group (log-rank P = 0.004 for MSI subtype; P = 0.017 for MSS / EMT subtype; P < 0.001 for MSS / TP53- subtype; P = 0.007 for MSS / TP53+ subtype) ( Figure 6 (in N).

[0264] Table 23 Comparison of performance indicators of different models in predicting postoperative recurrence in LAGC patients in the prospective validation group

[0265]

[0266] These findings indicate that the RSA model can not only accurately predict postoperative recurrence for different molecular subtypes, but further survival analysis also found that the RSA model can also provide more comprehensive prognostic information beyond the molecular subtype itself.

[0267] Example 7

[0268] The RSA model adds value to the current staging system.

[0269] To evaluate whether the RSA model can provide better survival prediction in the same clinical stage, we conducted stratified analysis on LAGC patients with different TNM stages. Our results indicate that the RSA model can potentially improve the existing prognosis based on TNM stage. Kaplan–Meier survival analysis showed that in all data sets, we divided patients with different TNM stages into high-risk and low-risk groups based on the RSA model. There was a significant difference in OS between the high-risk group and the low-risk group, and this difference existed in patients with stage I to stage III (all log-rank P < 0.05, Figure 7 In order to eliminate the influence of clinical variables on survival prognosis, we further performed a 1:1 PSM analysis on patients of different stages in the entire data set. The clinical characteristics before and after matching are summarized in Table 24. The results of survival analysis found that the prognosis of the high-risk group in the RSA model before PSM was worse than that of the low-risk group (stage I: 47.5% vs. 81.4%, P < 0.001; stage II: 48.8% vs. 68.9%, P < 0.001; stage III: 34.5% vs. 51.7%, P < 0.001). Further, the prognosis of the high-risk group in the RSA model after PSM was still worse than that of the low-risk group (stage I: 44.8% vs. 86.2%, P < 0.001; stage II: 49.6% vs. 66.4%, P = 0.003; stage III: 38.0% vs. 53.0%, P < 0.001) ( Figure 7 Furthermore, multivariate Cox regression analysis revealed that the RSA model was an independent prognostic factor for 5-year OS in patients with different TNM stages (Tables 25-27). These results suggest that the RSA model can add additional prognostic value to the current TNM staging system.

[0270] Table 24 Comparison of baseline characteristics of patients with locally advanced gastric cancer at different TNM stages before and after propensity score matching

[0271]

[0272]

[0273]

[0274]

[0275] Note: ECOGPS = Eastern Cooperative Oncology Group Performance Status Score; SII = Systemic Immuno-Inflammatory Index; PNI = Prognostic Nutrition Index; NLR = Neutrophil-to-Lymphocyte Ratio; PLR = Platelet-to-Lymphocyte Ratio. * Median values ​​were used as the cutoff for stratification into high-risk and low-risk groups.

[0276] Table 25 Univariate and multivariate Cox regression analysis of factors affecting 5-year overall survival in patients with LAGC (pancreatic ductal adenocarcinoma) after surgery in TNM stage I

[0277]

[0278]

[0279] Note: ECOGPS = Eastern Cooperative Oncology Group Performance Status Score; SII = Systemic Immuno-Inflammatory Index; PNI = Prognostic Nutrition Index; NLR = Neutrophil-to-Lymphocyte Ratio; PLR = Platelet-to-Lymphocyte Ratio. *Divided into high-risk and low-risk groups using the median as the cutoff.

[0280] Table 26 Univariate and multivariate Cox regression analysis of factors affecting the 5-year overall survival of patients with LAGC (pancreatic ductal adenocarcinoma) after surgery in TNM stage II

[0281]

[0282]

[0283] Note: ECOGPS = Eastern Cooperative Oncology Group Performance Status Score; SII = Systemic Immuno-Inflammatory Index; PNI = Prognostic Nutrition Index; NLR = Neutrophil-to-Lymphocyte Ratio; PLR = Platelet-to-Lymphocyte Ratio. * Median values ​​were used as the cutoff for stratification into high-risk and low-risk groups.

[0284] Table 27 Univariate and multivariate Cox regression analysis of factors affecting the 5-year overall survival rate of patients with LAGC in TNM stage III after surgery

[0285]

[0286]

[0287] Note: ECOGPS = Eastern Cooperative Oncology Group Performance Status Score; SII = Systemic Immuno-Inflammatory Index; PNI = Prognostic Nutrition Index; NLR = Neutrophil-to-Lymphocyte Ratio; PLR = Platelet-to-Lymphocyte Ratio. * Median values ​​were used as the cutoff for stratification into high-risk and low-risk groups.

[0288] Additionally, complex models like ensemble models are not easy to understand. We cannot rely on the original model itself to interpret it. Instead, a simplified interpretation framework must be adopted, called the approximation of the primary model interpretation. In the interpretation of feature attribution, SHAP acts as an additive method to interpret the predicted value of the model as the cumulative sum of the attribution values ​​assigned to each input feature. Significant SHAP values ​​indicate a significant effect of the predictor on predicting the effectiveness of immune checkpoint inhibition therapy. First, we used SHAP to calculate feature importance and impact. We sorted the feature importance in descending order and identified 8 pathological omics features that had a strong impact on recurrence prediction. Across different datasets, the results were summarized using a beeswax summary plot ( Figure 7 CJ) and feature importance plots ( Figure 15 ) is presented. Figure 7 Zhong K shows in detail two patients with the same clinical characteristics and treatment, but with differences only in the patients' pathological genomic characteristics, and the final follow-up revealed different clinical outcomes.

[0289] In summary, the multimodal prediction model provided by this invention demonstrated excellent discriminatory power, sensitivity, and specificity across internal validation, multicenter external validation, prospective clinical studies, and TCGA datasets. It can add additional prognostic value to the current TNM staging system and provide improved survival prediction within the same clinical stage. It holds great promise for application in gastric cancer prognosis.

[0290] The foregoing description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Those skilled in the art will readily appreciate that various modifications and variations of the present invention are possible. Any modifications, equivalent substitutions, or improvements made within the spirit and principles of the present invention are intended to be within the scope of protection of the present invention.

Claims

1. A gastric cancer prognosis system based on a pathomic integrated model, characterized in that: It includes: Input unit: used to obtain gastric cancer pathological images of patients; Segment the WSI image to obtain a segmented image; Pathology feature extraction unit: used to extract pathology features from patients' gastric cancer pathology images based on the open source CellProfiler software; The deep learning-based pathology feature extractor construction unit selects a portion of the segmented image and divides it into a training set and a validation set; a residual neural network architecture is used to train and obtain a feature extractor; Deep learning pathomic feature extraction unit: used to flatten all WSI images into segmented images of the same size as the segmented images used during training, put them into the feature extractor, use the output values ​​of the feature extractor as deep learning features, and sum and average the feature information of the segmented images belonging to the same WSI image to generate deep learning pathomic features; Pathological genomics feature selection unit: configured to select recurrence-related pathological genomics features based on the pathological genomics features extracted by the pathological feature extraction unit and the deep learning-based pathological genomics feature extraction unit; The pathomic ensemble model construction and validation unit is used to construct a pathomic ensemble model for gastric cancer prognosis using multiple algorithms. The voting regressor averages the individual WSI image predictions of each basic regressor to generate the pathomic ensemble model. The basic regressor refers to the predicted probability value of each algorithm. Gastric cancer prognosis unit: used to input the patient's gastric cancer pathological image into the pathology omics integrated model to obtain the predicted patient prognosis result.

2. The gastric cancer prognosis system based on the pathogenomics integrated model according to claim 1, characterized in that: Each segmented image is annotated to correspond to the annotation map. Each segmented image is labeled with the tissue type that occupies the largest proportion of the corresponding area on the matching annotation map. Eight types of tissues related to the tumor microenvironment (TME) are annotated in each WSI: adipose tissue (ADI), debris (DEB), mucus (MUC), muscle (MUS), lymphocyte aggregates (LYM), stroma (STR), normal mucosa (NOR) and tumor epithelium (TUM); Preferably, the WSI image is segmented into segmented images of 224*224 pixels; Preferably, during training, the cross entropy loss function and Adam optimizer are used, and the initial learning rate is set to 0.001; Preferably, the residual neural network architecture used is selected from: ResNet18, ResNet34 or ResNet50; Preferably, the residual neural network architecture used is ResNet18; the final fully connected classification layer in the base network is removed, and only the pre-order convolution module and pooling structure in the base network are retained to create a universal feature extractor; Preferably, the pathological feature extraction unit is used to: extract a feature matrix in a representative region of interest (ROI) of the patient as a pathological feature based on the open source CellProfiler software; Preferably, said representative region of interest (ROI) is processed to quantify predefined morphological descriptors to generate said interpretable feature matrix.

3. The gastric cancer prognosis system based on the pathogenomics integrated model according to claim 1, characterized in that: The pathology feature selection unit is used to perform the following steps: (1) Use maximum correlation and minimum redundancy algorithms to remove redundant features; (2) performing a Pearson correlation matrix to eliminate features with low recurrence correlation from highly correlated paired features; (3) Using the least absolute shrinkage and selector logic algorithm with ten-fold cross validation to reduce features; Preferably, the pathology feature selection unit obtains eight features.

4. The gastric cancer prognosis system based on the pathogenomics integrated model according to claim 1, characterized in that: The multiple algorithms in the construction and verification unit of the pathology omics integrated model are selected from at least one of the following: Voting regressor, random forest (RF), support vector machine (SVM), k-nearest neighbor, partial least squares (PLS), least squares support vector machine (LSSVM), BP neural network (BPNN), convolutional neural network (CNN), extreme learning machine (ELM), particle swarm optimization BP neural network (PSO-BP), recurrent neural network, Bayesian classifier, K-nearest neighbor algorithm, K-means algorithm, linear regression, logistic regression; Preferably, the gastric cancer prognosis unit outputs at least one of the following content of the patient: survival period, risk of early postoperative recurrence, risk of complications, expected functional recovery and quality of life prediction.

5. A gastric cancer prognosis system based on multimodal deep learning, characterized by: It includes: Input unit: used for inputting independent clinical features related to gastric cancer recurrence and the gastric cancer prognosis result outputted by the pathogenomics integrated model according to any one of claims 1 to 4; A multimodal prediction model construction unit: used to integrate independent clinical features related to gastric cancer recurrence and the gastric cancer prognosis results obtained based on the pathogenomics integrated model to construct a multimodal prediction model; Gastric cancer prognosis unit: used to obtain the final gastric cancer prognosis result by fusing the independent clinical features and the gastric cancer prognosis result obtained by the pathological omics integrated model through the multimodal prediction model.

6. The gastric cancer prognosis system based on multimodal deep learning according to claim 5, characterized in that: The expression of the multimodal prediction model is: Among them, x1 is the pathological feature score, x2 is the clinical feature score, and p represents the multimodal prediction probability.

7. The gastric cancer prognosis system based on multimodal deep learning according to claim 5, characterized in that: The independent clinical feature is selected from at least one of pathological type, invasion depth (pT stage), and lymph node metastasis (pN stage); Preferably, the gastric cancer prognosis unit outputs at least one of the following information about the patient: survival period, risk of early postoperative recurrence, risk of complications, expected functional recovery, and quality of life prediction; Preferably, the pathological type includes the gastric cancer type assessed by Lauren classification, Borrmann classification or TNM staging system.

8. Use of the multimodal prediction model according to any one of claims 5 to 7 in constructing a TNM staging system.

9. An electronic device, characterized in that: It includes: processor and memory; The processor is connected to the memory, wherein: The memory is used to store a computer program, and the processor is used to call the computer program to execute the method for gastric cancer prognosis or TNM staging based on multimodal deep learning as described in any one of claims 5 to 7.

10. A computer-readable storage medium, characterized in that The method comprises instructions which, when executed on a computer, cause the computer to execute the method for gastric cancer prognosis or TNM staging based on multimodal deep learning as described in any one of claims 5 to 7.

Images