Multi-modal prediction algorithm for esophageal cancer PD-L1 biomarker and immunotherapy response

Abstract

The invention provides a brand new programmed death ligand 1 (PD-L1) level and immunotherapy response prediction algorithm based on the application of a deep learning technology in medical images in combination with methods of multi-modal feature fusion, a self-attention mechanism, adversarial learning and the like. The algorithm comprises the following steps of data preprocessing, encoder feature extraction, radiomics feature extraction, multi-modal feature fusion, PD-L1 level prediction and immunotherapy response prediction. According to the invention, the pathological image (Hamp; according to the method, a CT image (e.g., E staining section), a CT image and clinical data are integrated into a deep learning model for predicting the PD-L1 level of an esophageal squamous cell carcinoma (ESCC) patient. In addition, by extracting PD-L1 features of an immunotherapy patient, a model for predicting immunotherapy response is constructed. By fusing multi-modal data, auc of the model in a PD-L1 prediction task is equal to 0.835, auc of the model in an immunotherapy reaction task is equal to 0.806, a new thought and method are provided for related research in the field of tumor treatment, and beneficial reference is provided for clinical treatment.

Description

Technical Field

[0001] The present invention relates to the field of deep learning technology, which is an application of deep learning technology in medical image classification, and in particular to a multimodal prediction algorithm for esophageal cancer PD-L1 biomarker and immunotherapy response Background Art

[0002] Esophageal squamous cell carcinoma (ESCC) is the main histological subtype of esophageal cancer with unique epidemiological and clinical characteristics. It is prone to lymphatic spread and has a poor prognosis. Immune checkpoint inhibitors (ICIs) have shown significant antitumor effects in ESCC patients, but the proportion of ESCC patients who benefit from ICIs is still limited. In current clinical practice, the selection of immunotherapy is mainly based on the PD-L1 biomarker level in tumor tissue. Elevated PD-L1 levels usually indicate a favorable response to immunotherapy. However, the method of evaluating PD-L1 biomarkers by immunohistochemistry (IHC) is usually quantified as a composite positive score (CPS) or tumor proportion score (TPS), which is a resource-intensive and time-consuming process. Hematoxylin-eosin (H&E) staining is a routine examination of clinical specimens to facilitate visual inspection of malignant cells. Pathologists rely on H&E staining for tumor diagnosis, including determining tumor subtype and grade. In recent years, deep learning methods can capture information from H&E images that exceeds human visual capabilities, thus providing new potential for pathological slides. For example, predicting the expression level of molecular biomarkers from H&E whole-slide images of breast cancer.

[0003] In the existing technology, pathological images or CT image data are used alone to predict PD-L1 biomarkers, and multiple modality data cannot be used to predict PD-L1 biomarkers and immunotherapy results, resulting in limited performance and weak generalization of the model. Summary of the invention

[0004] The purpose of this invention is to solve the shortcomings of the prior art. By combining multimodal feature fusion, self-attention mechanism and adversarial learning, a new PD-L1 level and immunotherapy response prediction algorithm is proposed. Excellent results have been achieved in the prediction of PD-L1 level and immunotherapy response, which provides new ideas and methods for related research in the field of tumor treatment and provides a useful reference for clinical treatment.

[0005] The idea of ​​the present invention is to construct a deep learning model through multimodal fusion technology to achieve the prediction of PD-L1 level and immunotherapy response. 1) For each patient, H&E full-slice images, CT images before and after treatment, and clinical variables were collected. 2) Labels were generated based on patient data. 3) Multimodal data were preprocessed. 4) A comparison model was trained to generate a feature extractor and extract features of H&E images. 5) Multimodal features were fused through a self-attention mechanism, and PD-L1 expression level prediction was performed to obtain an encoder for predicting PD-L1 expression. 6) The encoder for predicting PD-L1 expression was used to obtain immunotherapy patient characteristics and predict immunotherapy response.

[0006] The specific technical solution of the present invention is: a multimodal prediction algorithm for esophageal cancer PD-L1 biomarker and immunotherapy response, comprising the following steps:

[0007] The data required for the model are H&E whole-slice images, CT images before treatment, clinical variables, PD-L1 expression levels, and immunotherapy response signatures. Patient data are all from the Third Affiliated Hospital of Soochow University.

[0008] A binary (0 or 1) label was generated based on the PD-L1 expression data, with 0 indicating low expression and 1 indicating high expression. A binary (0 or 1) label was generated based on the immunotherapy response data, with 0 indicating non-responders and 1 indicating responders. The feature extractor trained with contrastive learning was used to extract H&E image features, and the LASSO algorithm was used to screen out CT features and clinical variables that were effective for prediction. Five-fold cross validation was used during the training process.

[0009] Multimodal data is input into the model, and the self-attention encoder fuses the features of the multimodal data. In order to avoid overfitting, Dropout is used to improve the fusion effect.

[0010] Considering that there is a certain relationship between PD-L1 expression level and immunotherapy effect, a multimodal fusion module is used to predict PD-L1 expression level and obtain PD-L1 feature encoder. Then, the PD-L1 feature encoder is used to obtain the fusion features of immunotherapy patients, and the fusion features are used to predict immunotherapy response.

[0011] The model network includes a feature extractor that can extract features from H&E images, a self-attention encoder, a predictor for predicting PD-L1 expression levels based on multimodal features, and a predictor for predicting immunotherapy response based on PD-L1 features and multimodal features.

[0012] The present invention has the following beneficial effects:

[0013] 1. This multimodal deep learning prediction algorithm for esophageal cancer PD-L1 biomarker and immunotherapy outcomes applies multimodal fusion technology to esophageal cancer PD-L1 biomarker for the first time. By learning the expression level of PD-L1 biomarker from data of multiple modalities, the prediction performance is greatly improved. By integrating pathological images, CT images and clinical data, the algorithm can more accurately assess PD-L1 levels and predict patients' response to immunotherapy, significantly improving the accuracy of prediction and clinical practicality.

[0014] 2. This multimodal deep learning prediction algorithm for esophageal cancer PD-L1 biomarker and immunotherapy outcome combines PD-L1 biomarker prediction with immunotherapy response prediction for the first time. Given the potential clinical connection between PD-L1 expression level and immunotherapy response, we use the features learned by the model from the PD-L1 prediction task to enhance the effect of immunotherapy response prediction. Experimental results show that the algorithm exhibits excellent predictive performance after transfer learning, not only performing well in predicting PD-L1 expression levels, but also achieving significant success in predicting immunotherapy response. This method effectively improves the accuracy and reliability of predictions, provides strong support and guidance for clinical practice, and demonstrates the great potential of multimodal deep learning in the field of tumor treatment. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 This is the model algorithm framework diagram;

[0016] Figure 2 The model uses multimodal data ablation experiments and ROC curves of the baseline model. DETAILED DESCRIPTION

[0017] The technical solutions in the embodiments of the present invention will be described clearly and completely below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments.

[0018] Reference Figure 1 , a multimodal prediction algorithm for esophageal cancer PD-L1 biomarker and immunotherapy response, comprising the following steps:

[0019] S1: H&E whole-slide images and pre-treatment CT images were collected from the Third Affiliated Hospital of Soochow University. PD-L1 expression labeling was independently evaluated by two pathologists for ESCC sections to obtain PD-L1 expression in tumor cells and immune cells (including lymphocytes and macrophages), and immunotherapy response was independently evaluated by two experienced radiologists and one oncologist according to RECIST version 1.1;

[0020] S2: Generate a binary (0 or 1) label based on PD-L1 expression value, and then classify it into low expression and high expression according to 50 as the threshold. The method for generating a binary (0 or 1) label for immunotherapy data is: patients with partial remission (PR) and complete remission (CR) are classified as responders 1, and patients with stable disease (SD) are classified as non-responders 0.

[0021] S3: For H&E stained pathology images, they were first read into memory using , and then the images were converted from RGB color space to HSV color space. To identify tissue regions (foreground), a binary mask was generated by thresholding the saturation channel in HSV space. Subsequently, the edges were smoothed, and morphological closing was applied to fill small gaps and holes, effectively segmenting the slices into tissue and non-tissue regions. After segmentation, each WSI was split into small blocks of 256 × 256 pixels at 20x magnification.

[0022] For CT data, we extracted radiomic features from ROIs manually annotated by doctors. For each CT image, 118 features were extracted, including first-order statistical features, shape features, and texture features.

[0023] For radiomic features and clinical variables, the LASSO algorithm was used to screen out features that were useful for the prediction task.

[0024] S4: On a large number of unlabeled patches, we use contrastive learning to train an encoder to extract the intrinsic features of each patch. The main idea of ​​contrastive learning is to set a pre-training task so that positive samples are close to each other in the latent space, while negative samples are far away. In our practice, we generate two views of a patch as positive sample pairs through image augmentation, while the enhanced views of other patches in a mini-batch are regarded as negative samples. We adopt multiple image augmentation methods, including random scale cropping and scaling, random color jittering, random Gaussian blurring, and random flipping. We use ResNet50 convolutional neural network (CNN) as the encoder f backbone to produce a 1024-dimensional latent representation h of the input patch. i , that is, h i =f(x i ). Next, the projection head g transforms the latent representation into an embedding z i Formally, we have z i =g(h i ), input small block x i The contrast loss is defined as:

[0025]

[0026] where z i and z′ iRepresents the embedding of the positive sample pair, with respect to the input patch x i , sim() is the cosine similarity obtained by L2 normalization of the dot product of the two embeddings. N is the size of the negative sample queue, z i refers to the embedding of negative samples, and τ is the temperature parameter. We adopt an adversarial contrastive learning method to pre-train the encoder. It treats negative samples as learnable weights and updates the adversarial samples alternately to generate the most challenging negative samples.

[0027] S5: The features of the above pathological images and the filtered features are concatenated to form a set of comprehensive fused features. These fused features are then input into the self-attention module to adjust their weights according to the PD-L1 prediction task. Finally, average pooling is applied to obtain patient-level features for downstream prediction tasks. Cross entropy is used as the loss function for PD-L1 binary classification:

[0028]

[0029] Among them, y i is the true label of PD-L1, p i is the predicted probability. In this task, we excluded clinical variables in the feature fusion stage because clinical information contributes less to the prediction of PD-L1 levels.

[0030] S6: Since the expression level of PD-L1 is closely related to the response to immunotherapy, and high PD-L1 levels usually indicate a favorable response to immunotherapy, we incorporated the features derived from the PD-L1 level prediction model into the immunotherapy response prediction task. Specifically, the 512-dimensional features extracted from the last layer of the trained PD-L1 prediction model were concatenated with the 8-dimensional radiomic features and 8-dimensional clinical variables selected by the LASSO algorithm. The combined features were then input into a single fully connected layer to predict the immunotherapy response. The same cross entropy was used as the loss function.

[0031] In the network model:

[0032] Feature Extractor: Obtain image features from H&E stained images.

[0033] Self-attention encoder: Fuse multimodal features to obtain the patient's fused features.

[0034] PD-L1 expression level predictor: Predicts PD-L1 expression level based on the input patient’s fusion signature.

[0035] Immunotherapy response predictor: predicts patient immunotherapy response based on input patient fusion signatures.

[0036] In this example, patients from the Third Affiliated Hospital of Soochow University were used. The dataset contains 220 ESCC patients who underwent esophagectomy and 75 patients who received immune checkpoint inhibitors alone or in combination with chemotherapy. For each patient, we manually collected H&E full-slice images, CT images before and at the beginning of treatment, and clinical variables. With the joint efforts of expert pathologists and computerized tools for rapid annotation, we successfully annotated the regions of interest (ROIs) in H&E slices and CT images.

[0037] The average AUROC performance index of each method was obtained by randomly training 1000 times. In the prediction task, the AUROC of the multimodal model was improved by about 8% to 10% compared with the baseline model, and the performance was improved by 3% to 5% compared with the model using only H&E staining images. The ROC curve of one training is as follows Figure 2 shown.

[0038] The above description is only a preferred specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can make equivalent replacements or changes according to the technical scheme and inventive concept of the present invention within the technical scope disclosed by the present invention, which should be covered by the protection scope of the present invention.

Claims

1. A multimodal prediction algorithm for esophageal cancer PD-L1 biomarker and immunotherapy response, characterized by: Includes steps: 1) Data screening: The data required for the model are the patient's pathological images, CT images and clinical data, PD-L1 protein expression levels and immunotherapy response labels. 2) Generate a binary (0 or 1) label based on the PD-L1 expression data, where 0 represents low expression and 1 represents high expression. Generate a binary label based on the immunotherapy response data, where 1 represents responders and 0 represents non-responders. 3) Data preprocessing: For each pathological image, we segmented it into small blocks. For CT images, we extracted radiomic features from the tumor region. For clinical variables, we standardized and coded them. 4) An unsupervised domain adaptation method based on contrastive learning is used to train a feature extractor on a large number of unlabeled patches in pathological images, thereby extracting patch-level features for downstream tasks. 5) The weights of the multimodal features are adjusted through the self-attention mechanism, and average pooling is applied to obtain patient-level features for downstream prediction tasks. Based on the fused patient-level features, we use the predictor to predict PD-L1 levels. 6) Considering that the expression level of PD-L1 is closely related to immunotherapy response and high PD-L1 level usually indicates a favorable response to immunotherapy, we used the features derived from the PD-L1 level prediction model to predict immunotherapy response using the predictor. In step 1), we retrospectively sorted out 220 ESCC patients who underwent esophagectomy as the PD-L1 cohort and 75 immunotherapy cohorts who received immune checkpoint inhibitors alone or in combination with chemotherapy. The data of this study were approved by the Ethics Committee of the Third Affiliated Hospital of Soochow University (approval number: 2024-KD139) and conducted in accordance with the Declaration of Helsinki.

2. The multimodal prediction algorithm of esophageal cancer PD-L1 biomarker and immunotherapy response according to claim 1, characterized in that: In step 4), unlabeled data are randomly sampled from each pathological image, and the adversarial contrastive learning method is used to train the feature extractor.

3. The multimodal prediction algorithm of esophageal cancer PD-L1 biomarker and immunotherapy response according to claim 1, characterized in that: In step 5), the self-attention mechanism includes a fully connected layer and a Transformer encoder. The predictor consists of two fully connected layers, and Dropout is set to improve the multimodal data fusion effect. The output value is a continuous value, indicating the probability of high expression of PD-L1.

4. The multimodal prediction algorithm of esophageal cancer PD-L1 biomarker and immunotherapy response according to claim 1, characterized in that: In step 6), the predictor consists of two fully connected layers, the input features are extracted from immunotherapy patients, and the output value is a continuous value indicating the effect of immunotherapy.

Images