An esophageal cancer immunotherapy prediction method and system based on an artificial intelligence spatial pathology framework

Abstract

The application discloses an esophageal cancer immunotherapy response prediction method and system based on an artificial intelligence spatial pathology framework. In the model training stage, the NicheAlign cross-scale niche migration model is used to migrate the multi-cell reference niche defined in the surgical section to the pre-treatment endoscopic biopsy sample for training, to generate a CAF-Epi niche supervision mask registered with the H&E image of the same section; then, a SHINE model is constructed to identify and reconstruct the CAF-Epi niche area related to immunotherapy resistance from the conventional H&E image through double-granularity spatial modeling and spatial consistency learning constraints; finally, in the clinical reasoning stage, only based on the conventional biopsy H&E section of the patient before treatment, a structured auxiliary decision report is output through the SHINE-Predict patient-level dynamic set prediction module. The application can be used for immunotherapy response prediction and prognosis risk stratification analysis, and provides a deployable auxiliary tool for individualized immunotherapy decision of esophageal squamous cell carcinoma and other solid tumors.

Description

Technical Field

[0001] This invention belongs to the fields of computer-aided diagnosis, biomedical informatics and artificial intelligence deep learning technology, specifically relating to a method and system for predicting the response to neoadjuvant immunotherapy for esophageal cancer based on an artificial intelligence spatial pathology framework. Background Technology

[0002] Neoadjuvant immunotherapy based on PD-1 / PD-L1 blockade has become an important perioperative treatment for solid tumors such as esophageal squamous cell carcinoma (ESCC). However, patients' responses to immunotherapy exhibit significant heterogeneity. To avoid unnecessary treatment toxicity and delays in optimal surgical timing for non-responders, accurate assessment and identification of potential non-responders before treatment is a critical issue that urgently needs to be addressed in the interdisciplinary field of clinical oncology and computational pathology.

[0003] In routine clinical practice, preoperative endoscopic biopsy is the standard procedure for obtaining tumor tissue. Artificial intelligence prediction based on these routine hematoxylin and eosin (H&E) stained sections has high potential for clinical translation. However, existing clinical detection methods and computational pathology techniques suffer from the following significant technical bottlenecks and limitations in practical applications: Traditional single-feature biomarkers have inherent limitations in predictive dimensions: existing immunotherapy prediction strategies (such as PD-L1 expression, tumor mutational burden, and immune cell infiltration scores) mainly rely on single-dimensional molecular or cellular features. Since the tumor microenvironment (TME) is a highly complex spatial ecosystem, anti-tumor immunity is not determined by a single feature but is driven by multi-cellular population interactions across spatial tissues (such as cancer-associated fibroblast-epithelial cell interactions). Traditional methods fail to capture the architectural information of this multi-cellular spatial niche, resulting in limited predictive efficacy and inconsistent results across different cohorts.

[0004] Spatial transcriptomics technology cannot meet the needs of large-scale routine clinical applications: In recent years, spatial transcriptomics (such as platforms like Xenium) has been able to resolve cellular states and spatial organizational forms within tissues at single-cell resolution, successfully identifying specific spatial niches associated with immune resistance (such as the CAF-Epi niche). However, this type of technology relies on expensive specialized instruments and equipment, has extremely low throughput, and is highly dependent on prior expert knowledge and manual annotation. This technological barrier keeps it confined to the stage of basic scientific discovery, preventing its direct integration into routine large-sample pathological diagnostic workflows in clinical practice.

[0005] Existing AI pathology models face serious problems of "spatial scale mismatch" and "niche structure loss": Current conventional computational pathology models face significant technical challenges when processing endoscopic biopsy samples. First, the fragmentation and spatial discontinuity of biopsy samples. High-quality spatial transcriptome maps typically rely on large-field-of-view, structurally complete surgical slices, while endoscopic biopsy samples are tiny (usually only one-tenth the size of surgical slices) and extremely fragmented in tissue structure. When existing AI models directly transfer spatial relationships learned from large samples to small biopsy samples, they often encounter "scale mismatch," leading to feature extraction failure. Second, there is a lack of molecular-level ground truth guidance and global topological alignment with large field-of-view (WSI / large patch) data. Traditional deep learning multiple instance learning (MIL) algorithms mostly extract visual texture at the isolated image patch level, lacking explicit biological spatial niche priors and failing to effectively utilize the global ecological context information contained in the large patch sample size at the whole-slice level. This makes the model prone to producing false activations (false positives) that deviate from common biological knowledge, resulting in poor generalization ability and robustness in multi-center, real-world data.

[0006] In summary, existing technologies lack an effective means to bridge large-scale spatial molecular maps with tiny, fragmented biopsy samples. There is an urgent need in this field for a novel artificial intelligence-based spatial pathology framework that can automatically infer and reconstruct specific immunotherapeutic resistance niches corresponding to spatial transcriptomic reference tags directly from routine whole-slice H&E images during the clinical reasoning stage. This would allow for highly accurate and interpretable prediction of immunotherapy responses without increasing clinical testing costs. Summary of the Invention

[0007] The purpose of this invention is to provide a method and system for predicting immunotherapy response based on an artificial intelligence spatial pathology framework. This invention aims to overcome the limitations of small size and fragmented structure of preoperative endoscopic biopsy samples. Through cross-scale spatial migration and large-field-of-view deep learning technology, it achieves the identification and reconstruction of the spatial niche of immune resistance (CAF-Epi) on conventional whole-slice H&E images. This provides a personalized efficacy prediction and prognostic risk stratification tool for neoadjuvant immunotherapy of solid tumors (especially esophageal squamous cell carcinoma) without the need for additional expensive spatial molecular omics testing during the clinical inference stage.

[0008] The technical solution of the present invention to solve the above-mentioned technical problems is as follows: In a first aspect, the present invention provides a method for predicting the response to neoadjuvant immunotherapy for esophageal cancer based on an artificial intelligence spatial pathology framework, characterized by comprising the following steps: 1) Data acquisition and preprocessing: Obtain whole-slice images (WSI) of endoscopic biopsy tissue obtained before treatment of patients with routine hematoxylin and eosin (H&E) staining, and perform quality control, staining normalization, resolution unification and image block segmentation on the WSI; 2) Niche feature detection: The preprocessed image patch is input into the trained SHINE dual-granularity Transformer model, which outputs the CAF-Epi niche probability, spatial niche embedding features and local CAF-Epi probability map of each image patch, and then spatially stitches them to form a full-slice CAF-Epi niche spatial heatmap. 3) Patient-level response prediction: The spatial niche embedding features, CAF-Epi probability and spatial location information within the whole slice are input into the SHINE-Predict dynamic ensemble prediction network to capture the spatial distribution of niches and cross-regional contextual relationships within the whole slice, and output the patient's non-response probability to immunotherapy, binary response classification and auxiliary interpretive heatmap.

[0009] Furthermore, the training supervision labels for the SHINE dual-granularity Transformer model are prepared using a cross-scale spatial niche migration method; this preparation process includes, in the order of data processing: 1) Define multicellular reference niches on spatial transcriptomics surgical slices with complete structures and obtain reference niche labels for each cell or cell neighborhood. 2) Using a single cell or cell neighborhood as the analysis unit, construct a multidimensional input feature vector that includes local cell composition, neighborhood abundance distribution, local cell density, absolute spatial coordinates, and neighborhood heterogeneity descriptor; 3) Using the aforementioned multidimensional input feature vectors as model inputs and reference niche labels as model output labels, train the NicheAlign classifier; apply the trained classifier to the pre-treatment endoscopic biopsy spatial transcriptomics data used for label preparation during the training phase, output the posterior probability and predicted label of each cell or cell neighborhood for the reference niche, and map it to the H&E image through same slice image registration to generate a CAF-Epi niche pixel-level binary supervision mask, which serves as the supervision label for subsequent SHINE model training.

[0010] Furthermore, the dual-granularity Transformer deep learning model introduces a spatial consistency learning loss during training; this consistency loss achieves topological alignment and false positive suppression through the following mechanism: 1) When the organization-level prior label is a positive sample, if the highest local region-level prediction probability in the image patch does not reach the preset local activation threshold, the prediction deviation is penalized to prompt the model to focus on the local morphological changes corresponding to the CAF-Epi niche. 2) When the tissue-level prior label is a negative sample, the activation probability of all cells in that region is strictly suppressed, thereby eliminating pathological false positive predictions caused by tissue heterogeneity at the source.

[0011] Furthermore, the prediction network employs a deep patient-to-niche attention hierarchy, specifically implemented as follows: 1) Extract the class label features ([CLS]), global average pooling features, and global max pooling features from the dual-granularity Transformer output, and concatenate them to construct a multi-order context embedding vector; 2) Unlike the simple pooling operation of traditional multiple instance learning (MIL), it uses global patient state labels as active query vectors to calculate the ecological affinity between them and the niche embedding features (Key) of each image patch; 3) Adaptive normalization of the affinity is performed on unfixed and unfilled variable-length whole-slice sequences. This process allows attention weights to be influenced by the number, spatial distribution, and relative abundance of niches across the whole slice, thereby enhancing the model's ability to perceive sparse but clinically relevant mesenchymal interaction regions (such as CAF-Epi clusters) and supporting immunotherapy response prediction and prognostic risk stratification analysis.

[0012] Secondly, the present invention provides an immunotherapy response prediction system based on an artificial intelligence spatial pathology framework, comprising: 1) Image acquisition module: used to acquire routine H&E stained whole-section images (WSI) of endoscopic biopsy tissues taken before treatment of patients; 2) Niche Feature Detection Module: It has a built-in dual-granularity Transformer model constrained by the spatial consistency learning mechanism, which is used to divide WSI into blocks and extract high-dimensional niche embedding features. 3) Patient-level risk prediction module: It has a built-in dynamic ensemble prediction network to perform global ecological association attention calculation on the spatial niche embedding features, CAF-Epi probability and spatial location information output by SHINE, and outputs the patient's non-response probability, binary response classification and visualization of CAF-Epi niche spatial heatmap for a specific neoadjuvant immunotherapy regimen.

[0013] In practical applications, after obtaining the patient's pre-treatment endoscopic biopsy H&E slides, the slides are digitized and input into a terminal device or cloud server deployed with the system of this invention. The system then sequentially performs image quality control and preprocessing, SHINE niche feature detection, and SHINE-Predict patient-level risk prediction steps in the background. Within a preset time, it outputs a structured auxiliary decision-making report to the clinician's workstation, including a non-response probability score, binary response labels, risk stratification suggestions, and a CAF-Epi niche spatial heatmap. This application method eliminates the need for additional spatial molecular sequencing of patients during the clinical reasoning stage and can be integrated into existing routine pathological diagnostic workflows.

[0014] Compared with the prior art, the present invention has the following significant advantages: 1) Overcoming the bottleneck of biopsy "scale mismatch" and achieving cross-scale migration of spatial microenvironments: This invention constructs the NicheAlign cross-scale alignment algorithm as the data foundation for model training, migrating the molecular niche architecture defined by large-scale spatial transcriptomics to fragmented biopsy slices used for training, and generating CAF-Epi niche supervision labels that can be used for H&E model training. This process enables the niche features defined by spatial molecular omics to be computationally reconstructed on conventional pathological slices.

[0015] 2) The algorithm structure deeply integrates biological priors, significantly improving generalization ability: The SHINE model of this invention does not simply apply general image recognition algorithms, but innovatively introduces "dual-granularity spatial modeling" and "spatial consistency learning loss". This mechanism strictly binds the macroscopic context of clinical tissues with the microscopic cellular state, effectively eliminating the interference caused by tissue heterogeneity, and enabling the model to maintain high accuracy and robustness in independent multi-center validation sets.

[0016] 3) Preserving global spatial topological dependencies and enhancing immunotherapy response prediction capabilities: The SHINE-Predict prediction module of this invention differs from the traditional MIL framework in that simple pooling leads to the loss of spatial distribution information. It employs a "patient-niche attention" architecture specifically designed for non-fixed-length WSI. This mechanism captures the spatial distribution of niches within the tumor and cross-regional contextual relationships. Clinical validation results show that this system has high predictive efficacy in multicenter retrospective cohorts and prospective trials, and its discriminative ability in prospective cohorts is superior to PD-L1 immunohistochemical detection. It also supports risk stratification analysis of patient event-free survival (EFS) and overall survival (OS).

[0017] 4) High clinical translational value and economic benefits: During the clinical inference phase, this system requires only routine H&E-stained biopsy slides as a single input, without relying on high-cost single-cell spatial molecular platforms or artificial niche annotation. Its high degree of automation and interpretability lowers the threshold for clinical deployment, enabling it to serve as an auxiliary decision-making tool for clinicians in developing precision treatment plans for solid tumors such as esophageal cancer, helping to reduce ineffective treatments and their related toxic side effects.

[0018] The technical solution of the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. Attached Figure Description

[0019] Figure 1 A schematic diagram of the overall process and architecture of a method and system for predicting the response to neoadjuvant immunotherapy for esophageal cancer based on an artificial intelligence spatial pathology framework, provided in an embodiment of the present invention; Figure 2 This is a reference multicellular niche map for esophageal squamous cell carcinoma defined by Xenium spatial transcriptomics. Different colors in the map represent different baseline multicellular niches, including CAF-Epi, CAF-enriched, invasive epithelial cell-enriched, non-invasive and invasive epithelial cell-enriched, non-invasive epithelial cell-enriched, NF-enriched, NF-endothelial, non-invasive epithelial cell–NF, and TLS niches. This map serves as a reference spatial label for subsequent NicheAlign cross-scale niche transfer and SHINE model training. Figure 3 This is a schematic diagram illustrating the algorithm principle and processing flow of the cross-scale spatial niche migration alignment module (NicheAlign) provided in an embodiment of the present invention.

[0020] Figure 4 This is a comparison of the performance of conventional spatial clustering algorithms and NicheAlign algorithms in identifying spatial niches in biopsy samples in this embodiment of the invention, as well as a detailed schematic diagram of biopsy niche spatial mapping and local cell composition; wherein, Figure A is the identification result of conventional algorithm, Figure B is the identification result of NicheAlign, and Figure C is the global biopsy niche map and local single cell type mapping view inferred by NicheAlign. Figure 5 A schematic diagram of the deep learning network architecture and processing flow of the niche spatial histology identification model (SHINE) based on conventional pathological images provided in an embodiment of the present invention; Figure 6 This is a schematic diagram illustrating the classification performance evaluation and spatial mapping visualization of the niche spatial histology identification model (SHINE) based on conventional H&E images provided in this embodiment of the invention; wherein, Figure A and Figure B respectively show the classification confusion matrix and receiver operating characteristic (ROC) curve of the model, and Figure C shows the spatial mapping comparison between the niche inference results and the real molecular map. Figure 7 A schematic diagram of the network architecture and prediction process of the patient-level immunotherapy response prediction module (SHINE-Predict) based on niche-aware aggregation provided in an embodiment of the present invention; Figure 8 This invention relates to the sample size of the patient cohort and the distribution of responders and non-responders; Figure 9 This section presents the prediction of immunotherapy response by SHINE-Predict based on H&E images on the training set (n = 153). Figure A shows the clinical characteristics, PD-L1 status, and the predictive performance of SHINE-Predict on the training set (n = 153); Figure B shows the Kaplan-Meier survival curves of overall survival (OS) in the training set, stratified by SHINE-Predict into predicted responders and non-responders. Figure 10 This section presents the prediction performance of SHINE-Predict based on H&E images for immunotherapy response on a test set (n = 39). Figure A shows the clinical characteristics and the predictive performance of SHINE-Predict for immunotherapy response on the test set (n = 39); Figure B shows the Kaplan-Meier survival curves for overall survival (OS) on the test set, stratified by SHINE-Predict into predicted responders and non-responders. Figure 11 This section presents the prediction performance of SHINE-Predict based on H&E images for immunotherapy response in a combined retrospective multicenter cohort (n = 168). Figure A shows the predictive performance of clinical characteristics and SHINE-Predict for immunotherapy response in the combined retrospective multicenter cohort (n = 168); Figure B shows the Kaplan-Meier survival curves of OS in the combined retrospective multicenter cohort, stratified by SHINE-Predict into predicted responders and non-responders. Figure 12 This section validates SHINE-Predict in a prospective cohort. Figure A shows the predictive performance of clinical characteristics and SHINE-Predict on immunotherapy response in a prospective cohort (n = 95); Figure B shows the confusion matrix of SHINE-Predict's classification performance. Detailed Implementation

[0022] In the description of this invention, it should also be noted that, unless otherwise expressly specified and limited, these embodiments are for illustrative purposes only and are not intended to limit the scope of the invention. Furthermore, it should be understood that after reading the teachings of this invention, those skilled in the art can make various alterations or modifications to the invention, and these equivalent forms also fall within the scope defined by the appended claims.

[0023] I. A method for predicting esophageal cancer immunotherapy response based on an artificial intelligence spatial pathology framework like Figure 1 As shown, this method is a clinical decision support technology workflow for predicting the response before neoadjuvant immunotherapy in esophageal squamous cell carcinoma (ESCC). The workflow includes four core steps: routine H&E image acquisition and preprocessing, cross-scale niche label preparation, SHINE niche feature detection, and SHINE-Predict patient-level response prediction. Cross-scale niche label preparation is used in the model training phase, while the clinical inference phase mainly performs H&E image acquisition and preprocessing, SHINE niche feature detection, and SHINE-Predict patient-level response prediction. The specific steps are as follows: Step 1: Data Acquisition and Preprocessing Whole-slide images (WSIs) of endoscopic biopsy tissue obtained before treatment were acquired using routine hematoxylin and eosin (H&E) staining. First, pathological quality control was performed on the WSIs to exclude non-tissue areas, severely necrotic areas, areas with obvious folds or tears, areas with poor staining quality, and areas with blurred scans. Then, images from different scanning platforms or at different resolutions were normalized, and H&E staining was normalized based on a fixed reference image. Finally, the evaluable tissue area was segmented into multiple local tissue image blocks, which served as input to the SHINE model. The output of this step was a sequence of quality-controlled and normalized image blocks and their corresponding spatial location information.

[0024] Step 2: Preparation of cross-scale spatial niche migration tags (NicheAlign) During the model training phase, to provide molecularly anchored spatial supervision labels for conventional H&E images, this embodiment employs NicheAlign for cross-scale spatial niche migration and label generation. The inputs to this step include: structurally complete Xenium spatial transcriptomics surgical slices, pre-treatment endoscopic biopsy Xenium spatial transcriptomics data used for label preparation during the training phase, H&E images from the same slices or after registration, and major cell population information obtained based on cell marker annotations. The outputs of this step include: predicted niche labels for each cell or cell neighborhood in the training biopsy samples, a spatial distribution map of the niche, niche abundance information, and a CAF-Epi pixel-level binary supervision mask registered with the H&E images.

[0025] 1) Reference Niche Definition: Using large-field-of-view, structurally intact Xenium spatial transcriptomics surgical slides, the cell type composition, spatial proximity, and local microenvironment structure of esophageal squamous cell carcinoma tissue were analyzed. Specifically, cell neighborhoods were constructed based on single-cell spatial coordinates, and unsupervised clustering was performed according to the relative composition of major cell populations such as epithelial cells, cancer-associated fibroblasts, normal fibroblasts, endothelial cells, and immune cells in each cell neighborhood, thereby defining nine baseline multicellular niches. For example... Figure 2 As shown, the nine benchmark multicellular niches include: the cancer-associated fibroblast-epithelial cell niche (CAF-Epi niche), the cancer-associated fibroblast-enriched niche (CAF-enriched niche), the invasive epithelial cell-enriched niche, the non-invasive and invasive epithelial cell-enriched niche, the non-invasive epithelial cell-enriched niche, the normal fibroblast-enriched niche (NF-enriched niche), the normal fibroblast-endothelial cell niche (NF-endothelial niche), the non-invasive epithelial cell-normal fibroblast niche (non-invasive epithelial cell-NF niche), and the tertiary lymphoid structure niche (TLS niche). These nine benchmark multicellular niches collectively constitute the reference spatial niche map of esophageal squamous cell carcinoma and serve as the training label source for the NicheAlign model.

[0026] 2) Multidimensional Feature Construction: After obtaining the above nine reference niches and their labels, multidimensional features reflecting the local microenvironment state are extracted using a single cell or cell neighborhood as the basic analysis unit. These features include local cell type composition, neighborhood abundance distribution, local cell density, absolute spatial coordinates, and neighborhood heterogeneity descriptors. The multidimensional feature vectors serve as input to the NicheAlign model, and the above nine reference niche labels serve as output labels for model training, enabling the model to learn the correspondence between different spatial composition features and reference niche categories. In this step, the niche plays a role in standardizing spatial microenvironment categories and supervising the learning of labels.

[0027] 3) Model training: such as Figure 3 As shown, the multidimensional feature vectors obtained from the surgical slices are used as input data, and the corresponding nine reference niche labels are used as output labels to train a multi-class gradient boosting classifier, preferably the NicheAlign classifier based on LightGBM. Specifically, each cell or cell neighborhood is first represented as a multidimensional feature vector, and then the reference niche category to which the cell or cell neighborhood belongs is used as the supervision label to construct the training dataset; subsequently, the dataset is hierarchically divided according to the niche category to form a training set and a validation set for model parameter learning and performance monitoring; the classifier uses multi-class log loss as the optimization objective to learn the mapping relationship between multidimensional spatial microenvironment features and the nine reference niche categories. In this embodiment, the training set and validation set are hierarchically divided at 80% and 20% respectively; key hyperparameters include: maximum boosting rounds of 1200, early stopping rounds of 60, learning rate of 0.05, number of leaf nodes per tree of 96, feature sampling rate of 0.85, sample sampling rate of 0.85, and L2 regularization coefficient of 1.0.

[0028] 4) Niche Inference: After training, the NicheAlign classifier is fixed and used for niche inference and label preparation of pre-treatment endoscopic biopsy samples during the training phase. For pre-treatment endoscopic biopsy samples used for label preparation during the training phase, following the same data processing workflow as the training phase, multidimensional feature vectors of each cell or cell neighborhood are extracted from their Xenium spatial transcriptomics data and input into the trained NicheAlign classifier. The classifier outputs the posterior probability distribution of the cell or cell neighborhood belonging to 9 reference niches, and uses the niche category with the highest probability as the predicted label. This yields the niche category, spatial distribution map, and abundance information of each niche for each cell or cell neighborhood in the training biopsy samples; among which, the CAF-Epi niche ratio or abundance can be used for subsequent treatment response association analysis.

[0029] 5) Same-slice registration and label mapping: The niche spatial labels inferred by NicheAlign are mapped to the H&E image coordinate system of the same tissue slice. Specifically, based on Xenium images, DAPI images, and subsequent H&E staining images, same-slice image registration is performed using common morphological markers such as overall tissue contour, cell nucleus / hematoxylin signal, epithelial-mesenchymal boundary, tumor contour, and stromal septum. For regions identified as CAF-Epi niches with clear spatial correspondence, pixel-level CAF-Epi niche binary masks registered with H&E images are generated; regions with uncertain registration, significant local deformation, tissue loss, or unclear histological boundaries are not included in mask generation.

[0030] 6) The final CAF-Epi niche binary mask output in this step is paired one-to-one with the corresponding H&E image patches, serving as strong supervision labels for subsequent SHINE deep learning model training. Simultaneously, the niche spatial distribution map and abundance information output by NicheAlign can also be used to verify the association between CAF-Epi niches and non-response to immunotherapy. Therefore, NicheAlign plays a role in cross-scale migration of reference niches and generation of H&E supervision labels at the front end, and in the clinical inference stage, it demonstrates the ability to directly identify H&E in CAF-Epi niches through the trained SHINE model.

[0031] Step 3: Niche Feature Detection Based on Dual-Granularity Transformer (SHINE) During the training phase, the input to this step is the preprocessed H&E image patch and the CAF-Epi niche binary supervision mask generated in step 2. During the inference phase, the input to this step is the preprocessed H&E image patch and its spatial location information. This step outputs the tissue-level CAF-Epi probability, the local region-level CAF-Epi probability map, the niche embedding vector, and the full-slice CAF-Epi niche spatial heatmap for each image patch. The SHINE model is used to learn the mapping relationship between conventional H&E morphology and Xenium-anchored CAF-Epi niche labels.

[0032] 1) Backbone Network and Dual Granularity Prediction: A data-efficient image Transformer (DeiT) is used as the backbone network. After inputting a single H&E image patch, it outputs class-labeled features ([CLS]) to characterize the macroscopic tissue context and image patch labeling features to characterize local cellular structures. The network further branches into tissue-level prediction branches and local region-level prediction branches, which output the tissue-level probability of whether the image patch contains the CAF-Epi niche and the CAF-Epi probability map of the local region within the image patch, respectively.

[0033] 2) Spatial Consistency Learning Constraint: During the training phase, the CAF-Epi binary supervision mask obtained in step 2 is used to simultaneously construct tissue-level labels and local region-level labels, and a spatial consistency learning loss is introduced. This loss requires that there be at least some local region-level CAF-Epi activation evidence with sufficient confidence in the tissue-level positive image patches, while requiring that local region-level activation in the tissue-level negative image patches be suppressed, thereby reducing false positive predictions caused by tissue heterogeneity and enhancing the spatial topological consistency of the CAF-Epi niche heatmap.

[0034] 3) WSI-level inference output: During the inference phase, SHINE processes the entire H&E whole-slice image using a sliding window or image patch sequence method, generating CAF-Epi probabilities, niche embedding vectors, and spatial coordinates for all image patches. Subsequently, the local prediction results are stitched together or interpolated according to their spatial positions within the whole slice, forming a CAF-Epi niche spatial heatmap covering the entire slice. The aforementioned niche embedding vectors, CAF-Epi probabilities, and spatial coordinates serve as inputs to the SHINE-Predict patient-level prediction module in step 4.

[0035] Step 4: Patient-Niece Dynamic Ensemble Prediction (SHINE-Predict) 1) The inputs to this step are the whole-slice image patch niche embedding vector, CAF-Epi probability, and image patch spatial coordinates output from step 3; the outputs are the patient-level immunotherapy non-response probability, binary response labels, and the CAF-Epi niche spatial heatmap used to interpret the model's judgment.

[0036] 2) Multi-level context embedding and variable-length sequence construction: For each image patch, the class label features, global average pooling features and global max pooling features output by SHINE are spliced ​​to form a multi-level context embedding vector; the embedding vectors, CAF-Epi probabilities and spatial coordinates of all image patches in the whole slice of the same patient are combined to form a whole slice niche sequence of variable length, so as to maintain the difference in tissue area and number of image patches among patients.

[0037] 3) Patient-Necology Attention Aggregation and Predictive Output: A learnable global patient state marker is added before the variable-length full-slice niche sequence. This global state marker is used as a query vector, and its ecological affinity with the niche embedding features of each image patch is calculated. Attention weights are then normalized across the entire slice sequence. A patient-level global representation is obtained through attention-weighted aggregation, and then a multilayer perceptron and classification head are used to output the probability of no response. A binary response label is further obtained based on a preset decision threshold. This output, together with the CAF-Epi niche spatial heatmap generated in step 3, constitutes a structured auxiliary decision report, used to indicate whether the patient may benefit from neoadjuvant immunotherapy.

[0038] II. Clinical Validation and Evaluation of Technical Efficacy like Figure 4-12 As shown, to demonstrate the actual technical effectiveness of the method described in Embodiment 1 of the present invention, the inventors conducted a large-scale, multi-center retrospective and prospective verification experiment: Validation cohort: This cohort integrated seven independent cohorts, totaling 455 ESCC patients who received neoadjuvant PD-1 blockade combined with chemotherapy. It included a multicenter retrospective cohort and a prospective investigator-initiated trial (IIT) cohort.

[0039] Predictive performance: In identifying non-responders to immunotherapy, the area under the receiver operating characteristic (AUC) of this invention in the training set, internal test set, multicenter retrospective validation set, and prospective IIT cohort were 0.934, 0.853, 0.807, and 0.812, respectively; in the prospective cohort, its discriminative ability was superior to that of PD-L1 immunohistochemical detection.

[0040] Survival risk stratification: This invention enables risk stratification analysis of event-free survival (EFS) and overall survival (OS) in PD-L1 negative and PD-L1 positive subgroups, suggesting that the non-response probability and binary classification results output by the model have value in predicting treatment response and prognostic risk stratification.

[0041] Clinical translational advantages: The clinical reasoning stage relies solely on a single routine biopsy H&E slide as input material, eliminating the need for additional invasive re-examinations or expensive spatial molecular testing, thus possessing both high technical feasibility and health economic value.

[0042] III. A Prediction System Based on an AI Spatial Pathology Framework An immunotherapy response prediction system based on the above method is deployed on a local server or in the cloud within a hospital pathology department. The system includes: Image acquisition module: used to connect to the digital pathology scanner and acquire routine H&E stained whole slide images of endoscopic biopsy tissues before patient treatment, and to perform automated quality control and preprocessing.

[0043] Niche feature detection module: It incorporates the SHINE dual-granularity image Transformer model constrained by the spatial consistency learning mechanism described in Example 1, processes full-slice image blocks in parallel, and outputs CAF-Epi niche probabilities, spatial niche embedding features, and high-resolution CAF-Epi niche spatial heatmaps.

[0044] Patient-level risk prediction module: It has a built-in SHINE-Predict dynamic ensemble prediction network, which aggregates niche embedding features, CAF-Epi probabilities and spatial coordinates across the entire slice based on the patient-niche global association attention mechanism. It outputs the patient's immunotherapy non-response probability, binary response label, CAF-Epi niche heatmap and structured auxiliary decision report, and sends them to the clinician workstation terminal.

[0045] The above description provides examples of the preferred embodiments of the present invention, and any parts not described in detail are common knowledge to those skilled in the art. The scope of protection of the present invention is determined by the claims, and any equivalent modifications made based on the technical teachings of the present invention are also within the scope of protection of the present invention.

Claims

1. A method for predicting esophageal cancer immunotherapy response based on an artificial intelligence spatial pathology framework, characterized in that, Includes the following steps: Step 1: Obtain whole-slice images of endoscopic biopsy with routine H&E staining before the patient's treatment; perform quality control, staining normalization, resolution unification, and image block segmentation on the whole-slice images, dividing the evaluable tissue area into multiple local tissue image blocks; Step 2: In the model training phase, a multidimensional input feature vector containing local cell composition, neighborhood abundance distribution, local cell density, absolute spatial coordinates, and neighborhood heterogeneity descriptors is used as input, and a reference niche label is used as the model output label to train the NicheAlign classifier. The trained classifier is then applied to the pre-treatment endoscopic biopsy spatial transcriptomics data used for label preparation during the training phase to output the posterior probability and predicted label of each cell or cell neighborhood. This is then mapped to the H&E image through registration with the same slice image to generate a CAF-Epi niche pixel-level binary supervision mask. Step 3: In the model training phase, the obtained CAF-Epi niche pixel-level binary supervision mask is used to construct tissue-level labels and local region-level labels. Spatial consistency learning loss is introduced to train the niche feature detection module based on dual-granularity Transformer with Transformer-DeiT as the backbone network, and the trained niche feature detection module based on dual-granularity Transformer is obtained. The preprocessed H&E image patch obtained in step 1 is input into the trained niche feature detection module based on dual granularity Transformer. The module outputs the organization-level probability of whether each image patch contains a CAF-Epi niche, the CAF-Epi probability map of the local region inside the image patch, the niche embedding vector and spatial coordinates, and generates a full-slice CAF-Epi niche spatial heatmap based on the spatial position of each image patch in the full slice. Step 4: The patient-niche dynamic set prediction module performs global ecological association attention calculation on the niche embedding vector, CAF-Epi probability and spatial coordinate information output in the previous step, and outputs the patient's non-response probability, binary response classification and visualized CAF-Epi niche spatial heatmap for a specific neoadjuvant immunotherapy regimen.

2. The method for predicting esophageal cancer immunotherapy based on an artificial intelligence spatial pathology framework according to claim 1, characterized in that, Step 2 specifically includes the following steps: Step 2.1: Using large-field-of-view, structurally intact Xenium spatial transcriptomics surgical slides, the cell type composition, spatial proximity, and local microenvironment structure of esophageal squamous cell carcinoma tissue were analyzed. Cell neighborhoods were constructed based on single-cell spatial coordinates, and unsupervised clustering was performed according to the relative composition of major cell populations such as epithelial cells, cancer-associated fibroblasts, normal fibroblasts, endothelial cells, and immune cells in each cell neighborhood to define nine baseline multicellular niches. These nine baseline multicellular niches include: cancer-associated fibroblasts... Nine baseline multicellular niches were identified, including cell-epithelial cell niche, cancer-associated fibroblast-enriched niche, invasive epithelial cell-enriched niche, co-enriched niche of non-invasive and invasive epithelial cells, non-invasive epithelial cell-enriched niche, normal fibroblast-enriched niche, normal fibroblast-endothelial cell niche, non-invasive epithelial cell-normal fibroblast niche, and tertiary lymphoid structure niche. These nine niches collectively constituted a reference spatial niche map for esophageal squamous cell carcinoma and served as the training label source for the NicheAlign classifier. Step 2.2, Multidimensional feature construction: Using a single cell or cell neighborhood as the basic analysis unit, extract multidimensional feature vectors that can reflect the local microenvironment state, including local cell type composition, neighborhood abundance distribution, local cell density, absolute spatial coordinates, and neighborhood heterogeneity descriptor; Step 2.3, Model Training: Using the multidimensional feature vectors obtained from surgical slices as input data and the corresponding 9 reference niche labels as output labels, a multi-class gradient boosting classifier is trained, preferably the NicheAlign classifier based on LightGBM. First, each cell or cell neighborhood is represented as a multidimensional feature vector, and then the reference niche category to which the cell or cell neighborhood belongs is used as the supervision label to construct the training dataset. Subsequently, the dataset is hierarchically divided according to the niche category to form training and validation sets for model parameter learning and performance monitoring. The classifier uses multi-class log loss as the optimization objective to learn the mapping relationship between multidimensional spatial microenvironment features and the 9 reference niche categories. Step 2.4, Niche Inference: After training, the NicheAlign classifier is fixed and used for niche inference and label preparation of pre-treatment endoscopic biopsy samples during the training phase. Multidimensional feature vectors of each cell or cell neighborhood are extracted from the Xenium spatial transcriptomics data of the training endoscopic biopsy samples and input into the trained NicheAlign classifier. The classifier outputs the posterior probability distribution of the cell or cell neighborhood belonging to nine reference niches, and the niche category with the highest probability is used as the predicted label. This yields the niche category, spatial distribution map, and abundance information of each niche in the training biopsy samples. Step 2.5, Same-slice registration and label mapping: The niche spatial labels inferred by the NicheAlign classifier are mapped to the H&E image coordinate system of the same tissue slice. Based on the Xenium image, DAPI image, and subsequent H&E staining image, same-slice image registration is performed using common morphological markers such as overall tissue contour, cell nucleus / hematoxylin signal, epithelial-mesenchymal boundary, tumor contour, and stromal septum. For regions identified as CAF-Epi niches with clear spatial correspondence, pixel-level CAF-Epi niche binary masks registered with the H&E images are generated. Regions with uncertain registration, significant local deformation, tissue loss, or unclear histological boundaries are not included in mask generation. Step 2.6: The final output CAF-Epi niche binary mask is paired one-to-one with the corresponding H&E image patch, serving as a strong supervision label for training the niche feature detection module of the subsequent dual-granularity Transformer.

3. The method for predicting esophageal cancer immunotherapy based on an artificial intelligence spatial pathology framework according to claim 1, characterized in that, Step 3 specifically includes the following steps: Step 3.1: The Data-Efficient Image Transformer-DeiT is used as the backbone network of the niche feature detection module based on dual-granularity Transformer. After inputting a single H&E image patch, it outputs class label features for representing the macroscopic tissue context and image patch label features for representing the local cell structure. The network further branches into tissue-level prediction branches and local region-level prediction branches, which output the tissue-level probability of whether the image patch contains the CAF-Epi niche and the CAF-Epi probability map of the local region inside the image patch, respectively. The niche feature detection module based on dual-granularity Transformer is trained. During the training phase, the CAF-Epi binary supervision mask obtained in Step 2 is used to construct tissue-level labels and local region-level labels simultaneously, and spatial consistency learning loss is introduced. After training, the trained niche feature detection module based on dual-granularity Transformer is obtained. Step 3.2: Using the trained niche feature detection module based on dual-granularity Transformer, process the entire H&E whole slice image in a sliding window or image patch sequence manner. Generate the tissue-level probability of whether it contains the CAF-Epi niche, the CAF-Epi probability map of the local region inside the image patch, the niche embedding vector, and the spatial coordinates for all image patches. Then, stitch or interpolate the CAF-Epi probability map of the local region inside the image patch according to its spatial position in the whole slice to form a CAF-Epi niche spatial heatmap covering the entire slice. Finally, the niche embedding vector, CAF-Epi probability, CAF-Epi niche spatial heatmap, and spatial coordinates are used as the input for step 4.

4. The method for predicting esophageal cancer immunotherapy based on an artificial intelligence spatial pathology framework according to claim 3, characterized in that, The consistency loss achieves topology alignment and false alarm suppression through the following mechanism: 1) When the organization-level prior label is a positive sample, if the highest local region-level prediction probability in the image patch does not reach the preset local activation threshold, the prediction deviation is penalized to prompt the model to focus on the local morphological changes corresponding to the CAF-Epi niche. 2) When the tissue-level prior label is a negative sample, the activation probability of all cells in that region is strictly suppressed.

5. The method for predicting esophageal cancer immunotherapy based on an artificial intelligence spatial pathology framework according to claim 1, characterized in that, Step 4 specifically includes the following steps: Step 4.1, Multi-level context embedding and variable-length sequence construction: For each image patch, the extracted class label features, global average pooling features and global max pooling features are spliced ​​together to form a multi-level context embedding vector; the embedding vectors, CAF-Epi probabilities and spatial coordinates of all image patches in the same patient's whole slice are combined to form a variable-length whole slice niche sequence with no fixed length, so as to maintain the differences in tissue area and number of image patches among patients. Step 4.2, Patient-Necology Attention Aggregation and Predictive Output: A learnable global patient state marker is added before the variable-length full-slice niche sequence. This global state marker is used as a query vector, and its ecological affinity with the niche embedding features of each image patch is calculated. The ecological affinity is normalized within the full-slice sequence to obtain attention weights. Patient-level global representations are obtained through attention-weighted aggregation, and then the probability of no response is output through a multilayer perceptron and a classification head. Based on a preset decision threshold, binary response labels with and without immune response are further obtained. These binary response labels, together with the CAF-Epi niche spatial heatmap generated in Step 3, constitute a structured auxiliary decision report, which is used to indicate whether the patient may benefit from neoadjuvant immunotherapy.

6. An esophageal cancer immunotherapy prediction system based on an artificial intelligence spatial pathology framework, characterized in that, include: Image acquisition module: used to acquire whole-slice images of endoscopic biopsy with routine H&E staining before patient treatment; to perform quality control, staining normalization, resolution unification and image block segmentation on the whole-slice images, and to segment the evaluable tissue area into multiple local tissue image blocks; Offline cross-scale spatial niche migration label preparation module: used in the model training stage, the multi-dimensional feature vector obtained from surgical slices, including local cell type composition, neighborhood abundance distribution, local cell density, absolute spatial coordinates and neighborhood heterogeneity descriptors, is used as input data, and the corresponding 9 reference niche labels are used as output labels to train the NicheAlign classifier. The NicheAlign classifier is used to classify the pre-treatment endoscopic biopsy spatial transcriptomics data during the training phase, outputting the reference niche posterior probability and predicted label for each cell or cell neighborhood, and mapping it to the H&E image through same slice image registration to generate a CAF-Epi niche pixel-level binary supervision mask. The mask is used to train the niche feature detection module based on dual-granularity Transformer. The niche feature detection module based on dual-granularity Transformer is obtained by constructing tissue-level and local region-level labels using the obtained CAF-Epi niche pixel-level binary supervised mask and introducing spatial consistency learning loss. This module is trained on a dual-granularity Transformer-based niche feature detection module with Transformer-DeiT as the backbone network. This tool is used to perform dual-granularity niche feature detection at the tissue level and local region level on preprocessed H&E image patches. It outputs the tissue level probability of whether each image patch contains a CAF-Epi niche, the CAF-Epi probability map of the local region inside the image patch, the niche embedding vector and spatial coordinates, and generates a full-slice CAF-Epi niche spatial heatmap. The patient-niche dynamic set prediction module is used to perform multi-level context embedding and construct a variable-length full-slice niche sequence. A learnable global patient state label is added before the variable-length full-slice niche sequence. This global state label is used as a query vector to calculate the ecological affinity between it and the niche embedding features of each image patch. The ecological affinity is normalized within the full-slice sequence to obtain attention weights. Patient-level global representation is obtained through attention-weighted aggregation, and then output as a no-response probability by a multilayer perceptron and a classification head. The no-response probability is compared with a preset decision threshold to obtain a binary response label. This binary response label and the CAF-Epi niche spatial heatmap generated in step 3 together constitute a structured auxiliary decision report to indicate whether the patient may benefit from neoadjuvant immunotherapy.

Images