A method for prognostic prediction of breast cancer based on pathological images

Abstract

This invention discloses a method for prognostic prediction based on breast cancer pathological images. Using deep learning, this method can directly predict the ER, PR, HER-2, PAM50, gene expression status, and prognostic subtype of breast cancer patients from hematoxylin and eosin (H&E) stained histopathological images, providing a reference for assessing the prognostic survival risk of breast cancer patients. This invention, which predicts the prognosis of breast cancer tumors through pathological images, has the potential to reduce medical costs and is of great significance in clinical practice.

Description

Technical Field

[0001] This invention relates to the field of image processing, and in particular to a method for prognostic prediction based on breast cancer pathology images. Background Technology

[0002] Breast cancer is one of the most common malignant tumors in the world, with 2.26 million new cases diagnosed in 2020, surpassing lung cancer (2.2 million cases) for the first time to become the most prevalent cancer globally. In 2022, the United States reported approximately 290,560 new diagnoses and 43,780 estimated deaths. Therefore, breast cancer poses a significant threat to improving the average lifespan of the global population. Treatment options for breast cancer include surgery, chemotherapy, radiation therapy, targeted therapy, and immunotherapy, with doctors developing treatment plans based on the patient's subtype. However, due to the high heterogeneity of breast cancer, patients have different subtype systems and biomarkers, but the evaluation of these biomarkers requires immunohistochemistry (IHC), in situ hybridization (ISH), and RNA sequencing. High costs, insufficient tumor tissue, and difficulties in standardization currently make the evaluation of these biomarkers quite challenging.

[0003] In recent years, deep learning methods have attracted widespread attention in the field of pathology. Deep learning is a machine learning approach that can automatically extract features from organisms, such as images and text. Computational pathology, incorporating deep learning techniques, has begun to show potential in various pathological areas, including mitotic cell recognition, lymph node metastasis detection, tissue segmentation, and tumor subtype classification. However, most previous successes of deep learning techniques have focused on detecting image-based features, requiring pathologists to spend considerable time on visual examination and annotation. The application of multiple instance learning (MIL) in computational pathology significantly reduces this difficulty. The goal of MIL is to learn from a training set consisting of labeled bags of unlabeled instances, assuming that positive labeled bags contain at least one positive instance, while negative labeled bags contain only negative instances. The trained MIL algorithm is able to predict the positive or negative label of unseen bags. MIL has recently been used for AI-driven histopathological diagnosis and prognosis. Meanwhile, some previous studies have attempted to use MIL algorithms to directly determine the predicted tumor biomarker status from H&E-stained whole-slice images (WSI). However, a common problem with MIL in computational pathology is that the model's generalization performance is limited by the training sample size, especially when predicting tumor biomarkers. Previous MIL algorithms typically treated each WSI as a single bag. In this case, even for large tumor datasets, the sample size is insufficient, leading to overfitting of the model on the training dataset. Summary of the Invention

[0004] To address the challenges of evaluating biomarkers in breast cancer and the poor model performance caused by limited sample size during deep learning training, this invention provides a method for breast cancer prognosis verification based on a pathological image biomarker prediction model. This method can accurately and efficiently annotate tumors in breast cancer and obtain high-performance prediction models based on relatively little original training data.

[0005] To achieve the above-mentioned technical objectives, the technical solution of the present invention is as follows:

[0006] A method for prognostic prediction based on breast cancer pathology images includes the following steps:

[0007] Step 1: Acquire labeled pathological WSI images and perform preprocessing.

[0008] Step 2: The preprocessed pathological WSI image is segmented to obtain segmented blocks. Then, based on the classifier, it is determined whether each segmented block is a tumor image. After the determination is completed, the pathological WSI images with fewer than a predetermined threshold of segmented blocks that are determined to be tumor images are removed, and the remaining pathological WSI images are used as training images.

[0009] Step 3: Randomly select a preset number of segmentation blocks in a training image to form a pseudo-packet for training, thereby generating multiple pseudo-packets from a single training image, and iterating through all training images to generate even more pseudo-packets. Each pseudo-packet inherits the label of the corresponding training image.

[0010] Step 4: Train a deep convolutional neural network model based on pseudo-packets so that the model outputs the predicted probability of tumors. Four-fold cross-validation is used during training until the model's performance no longer improves, thereby obtaining a model for prognostic prediction.

[0011] Step 5: Input the image to be predicted into the model used for prognostic validation, and then output the prognostic prediction result.

[0012] The method, wherein step 1, includes preprocessing:

[0013] All acquired pathological WSI images were saved in a uniform format. Then, the magnification of the images was uniformly adjusted to 10x, and image damage was checked to remove any pathological WSI images with image damage.

[0014] The method described above, in step 1, further includes a step of increasing the number of pathological WSI images after preprocessing:

[0015] The Color Jitter algorithm was used to simulate the color distribution of pathological WSI images under different staining conditions, thereby obtaining more pathological WSI images. In the first stage, the Adam optimizer was used with a batch size of 16 and a learning rate of 0.0001. In the second stage, the Ranger optimizer was used with a batch size of 4 and a learning rate of 0.0002. Based on an early stopping strategy, the maximum training time for both stages was set to 30.

[0016] The method described above, in step 2, before determining whether each segmented block is a tumor image, further includes a step of checking the background area in the segmented block:

[0017] The background area, i.e. the blank, unorganized area, of each segment is checked. If the background area is greater than a preset threshold, the segment is discarded.

[0018] In the method described above, step 2, which determines whether each segmentation block is a tumor image based on a classifier, firstly, a classifier for determining whether an image is a tumor image is trained based on a known tumor image dataset. After training, the classifier is used to determine the probability that a segmentation block is a tumor image, and segmentation blocks with a probability greater than a probability threshold are identified as tumor images.

[0019] In step 3 of the method, after obtaining the pseudo packet, the method further includes horizontally flipping, vertically flipping, or changing the perspective of the segmented blocks in the pseudo packet to obtain new pseudo packets and expand the number of pseudo packets.

[0020] In the method described above, step 4, where the model outputs the predicted probability of a tumor, is:

[0021] The pseudo-packet is input into the Transformer encoder of the deep convolutional neural network. The self-attention mechanism is used to construct long-range dependencies between different image patches, thereby updating the features of each segmentation block. Then, the features are aggregated through the attention-based method to assign label coefficients to each segmentation block and calculate the overall label probability of the pseudo-packet. Finally, the aggregated features are input into the fully connected layer to obtain the predicted probability of the tumor.

[0022] The technical advantage of this invention lies in that it expands the breast cancer pathological images used as training images in various ways and establishes pseudo-packets based on the segmentation blocks formed by segmenting the images, thereby training the model in a deep learning manner. This invention can directly predict the prognostic tumor status of breast cancer patients through histopathological images stained with hematoxylin and eosin (H&E), and can provide a reference for judging the prognostic survival risk of breast cancer patients. Attached Figure Description

[0023] Figure 1This is a schematic diagram of the system architecture in this invention, where a represents the process of training a tumor classifier to automatically identify breast cancer tumor regions as ROIs, and b represents the process of training a pseudo-packet-based multi-instance learning algorithm to predict multiple biomarkers in breast cancer patients.

[0024] Figure 2 This diagram illustrates the verification of the present invention, where 'a' represents training a segmentation classifier to distinguish between cancerous and benign tissue. The first column shows two unannotated WSIs. The second row shows manually labeled cancer regions, and the last row shows deep learning-based annotations for cancer regions. 'b' is a diagram illustrating the accuracy of breast cancer tumor segmentation using the AUC of the ROC curve. 'c', 'd', 'e', ​​and 'f' represent the performance of the present invention in predicting ER, PR, HER-2, and PAM50 in TCGA. 'g', 'h', 'i', and 'j' represent the performance of the present invention in predicting ER, PR, HER-2, and PAM50 in TCGA. 'k' represents the average AUC of the present invention and five comparative models across these 11 tasks. Detailed Implementation

[0025] The method for prognostic prediction based on breast cancer pathology images provided in this embodiment includes the following steps:

[0026] Step 1: Acquire labeled pathological WSI images and perform preprocessing: Save all acquired pathological WSI images in a uniform format, then uniformly adjust the image magnification to 10x, and perform image damage checks, discarding pathological WSI images with image damage. Next, simulate the color distribution of pathological WSI images under different staining conditions using the Color Jitter algorithm to obtain more pathological WSI images. In the first stage, the Adam optimizer is used with a batch size of 16 and a learning rate of 0.0001. In the second stage, the Ranger optimizer is used with a batch size of 4 and a learning rate of 0.0002. Based on an early stopping strategy, the maximum training time for both stages is set to 30.

[0027] Step 2 involves segmenting the preprocessed pathological WSI images into segments. The background area (blank, tissue-free area) of each segment is then checked; if the background area exceeds a preset threshold, the segment is discarded. A classifier is then used to determine whether each segment is a tumor image. After this determination, pathological WSI images with fewer than a predetermined threshold of segments identified as tumor images are removed, and the remaining pathological WSI images are used as training images. The classifier is trained based on a known tumor image dataset.

[0028] Step 3: Randomly select a preset number of segmentation blocks from a training image to form a pseudo-packet for training. This generates multiple pseudo-packets from a single training image. The process is repeated across all training images to generate even more pseudo-packets, each inheriting the label from the corresponding training image. Then, the segmentation blocks within the pseudo-packets are horizontally flipped, vertically flipped, or subjected to viewpoint transformations to obtain new pseudo-packets, thus expanding the number of pseudo-packets.

[0029] Step 4: Train a deep convolutional neural network model based on pseudo-packets to enable the model to output the predicted probability of tumors. Four-fold cross-validation is used during training until the model's performance no longer improves, thus obtaining a model for prognostic prediction. For example, thousands of segmented blocks from each pathological WSI image can be used as a set. Then, bootstrapping is used to sample K patches in each WSI to build a pseudo-packet. In this embodiment, K = 200, but it can be adjusted according to actual needs. Regarding pseudo-packets... Each instance (i.e., a segment) x k A 512-dimensional image-level feature h is extracted using a ResNet-18 network. k Then, the Transformer encoder network is used to construct long-range correlations between different instances through a multi-head self-attention mechanism. This is based on a set of instance features. Let l represent the input of the (l+1)th layer of the Transformer encoder, where l = 0, 1, ..., L. In this embodiment, L = 4 is set, and the input of the Transformer encoder network is... Then, an attention-based method was used to aggregate the pseudo-packet features output by the Transformer network. Thus, pseudo-packet level features are obtained. Attention-based aggregation is defined as:

[0030]

[0031]

[0032] Where w k V is the weight variable, a k This represents the component of the feature of the k-th instance in the pseudo-packet. Finally, The input is fed into a fully connected (FC) layer to obtain the final biomarker prediction probability.

[0033] Step 5: Input the image to be predicted into the model used for prognostic validation, and then output the prognostic prediction result.

Claims

1. A method for prognostic prediction based on breast cancer pathological images, characterized in that, Includes the following steps: Step 1: Acquire labeled pathological WSI images and perform preprocessing; Step 2: The preprocessed pathological WSI image is segmented to obtain segmentation blocks. Then, based on the classifier, it is determined whether each segmentation block is a tumor image. After the determination is completed, the pathological WSI images with fewer than a predetermined threshold of segmentation blocks that are determined to be tumor images are removed, and the remaining pathological WSI images are used as training images. Step 3: Randomly select a preset number of segmentation blocks in a training image to form a pseudo-packet for training, thereby generating multiple pseudo-packets from a single training image, and iterate through all training images to generate even more pseudo-packets. Each pseudo-packet inherits the label of the corresponding training image. Step 4: Train a deep convolutional neural network model based on pseudo-packets so that the model outputs the predicted probability of tumors. Four-fold cross-validation is used during training until the model's performance no longer improves, thereby obtaining a model for prognosis prediction. Step 5: Input the image to be predicted into the model used for prognostic verification, and then output the prognostic prediction result.

2. The method according to claim 1, characterized in that, In step 1, the preprocessing includes: All acquired pathological WSI images were saved in a uniform format. Then, the magnification of the images was uniformly adjusted to 10x, and image damage was checked to remove any pathological WSI images with image damage.

3. The method according to claim 1, characterized in that, Step 1, after preprocessing, also includes a step to increase the number of pathological WSI images: The Color Jitter algorithm was used to simulate the color distribution of pathological WSI images under different staining conditions, thereby obtaining more pathological WSI images. In the first stage, the Adam optimizer was used with a batch size of 16 and a learning rate of 0.0001. In the second stage, the Ranger optimizer was used with a batch size of 4 and a learning rate of 0.0002. Based on the early stopping strategy, the maximum training time for both stages was set to 30.

4. The method according to claim 1, characterized in that, In step 2, before determining whether each segmented block is a tumor image, the step of checking the background area in the segmented block is also included: The background area of ​​each segment is checked, i.e. the blank, unorganized area. If the background area is greater than a preset threshold, the segment is discarded.

5. The method according to claim 1, characterized in that, In step 2, determining whether each segmentation block is a tumor image based on a classifier involves first training a classifier to determine whether an image is a tumor image based on a known tumor image dataset. After training, the classifier is used to determine the probability that a segmentation block is a tumor image, and segmentation blocks with a probability greater than a probability threshold are identified as tumor images.

6. The method according to claim 1, characterized in that, In step 3, after obtaining the pseudo packet, the method further includes horizontally flipping, vertically flipping, or changing the perspective of the segmented blocks in the pseudo packet to obtain new pseudo packets and expand the number of pseudo packets.

7. The method according to claim 1, characterized in that, In step 4, the model outputs the predicted probability of a tumor, which is: The pseudo-packet is input into the Transformer encoder of the deep convolutional neural network. The self-attention mechanism is used to construct long-range dependencies between different image patches, thereby updating the features of each segmentation block. Then, the features are aggregated through the attention-based method to assign label coefficients to each segmentation block and calculate the overall label probability of the pseudo-packet. Finally, the aggregated features are input into the fully connected layer to obtain the predicted probability of the tumor.

Images