A breast cancer whole slide image classification method combining self-supervision and weakly supervised learning
By combining self-supervised contrast learning and weakly supervised multi-example learning, Swin Transformer is used to classify full-sliced pathological images of breast cancer, which solves the problem of high-resolution image input and category imbalance, and achieves highly accurate pathological image classification.
Patent Information
- Application Number
- CN202311269042.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2023-09-27
- Publication Date
- 2025-08-19
- Estimated Expiration
- 2043-09-27
AI Technical Summary
The prior art has the problem that high-resolution images cannot be directly input into deep learning models and fine-grained manual labeling in breast cancer in full-slice pathological image classification, and the self-supervised method has category imbalance, resulting in limited model performance.
A method combining a self-supervised contrast learning framework with a weakly supervised multi-example learning framework is adopted, and feature extraction is used to use Swin Transformer to automatically learn internal structure information of pathological images through the self-supervised contrast learning framework. Key examples are selected in combination with a weakly supervised multi-example learning method, and the Transformer encoder is used to enhance the correlation between examples and perform feature aggregation.
It improves the accuracy of pathological image classification of breast cancer, solves the problem of category imbalance, and improves the stability and performance of the model.
Smart Images
Figure CN117237733B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of breast cancer whole-slice pathological image classification, and in particular to a breast cancer whole-slice image classification method combining self-supervision and weakly supervised learning. Background Art
[0002] Histopathological image analysis is of great significance in clinical breast cancer diagnosis, prognosis analysis, and treatment response prediction. In recent years, the introduction of whole-slice pathology images has promoted the application of deep learning technology in breast cancer histopathology image analysis.
[0003] Currently, deep learning-based whole-slide image classification methods face two major challenges. First, WSIs have extremely high resolution (over 100,000 × 100,000 pixels), making them infeasible for direct input into deep learning models. Second, fine-grained manual annotation is costly, making traditional supervised learning methods incapable of tackling this challenge. Weakly supervised learning methods based on multi-instance learning and self-supervised learning have emerged as effective solutions to these problems. Existing multi-instance learning methods for whole-slide pathology image analysis can be broadly categorized as embedding-based and example-based. However, weakly supervised methods only provide limited supervisory information, limiting their performance. In embedding-based methods, the model is primarily trained using a packet-level loss. Therefore, after packet-level classification, it is difficult for the model to identify key examples that trigger the classifier. In key example-based methods, pseudo-label selection can be erroneous, leading to unstable classifier performance. Furthermore, self-supervised methods for whole-slide breast cancer pathology images suffer from severe class imbalance: the number of negative samples in the segmented slides far exceeds the number of positive samples. This results in existing self-supervised models being biased towards negative samples and failing to effectively learn features from different classes. Summary of the Invention
[0004] In view of the problems existing in the prior art, the present invention discloses a multi-example breast cancer full-slice pathology image classification method based on a combination of self-supervision and weak supervision, which specifically includes the following steps:
[0005] Obtain a dataset of breast cancer full-slice pathology images and their corresponding full-slice label information, and divide the dataset into a training set, a validation set, and a test set according to a certain ratio;
[0006] When preprocessing full-slice pathology images, a binary mask of the foreground tissue region of the image is generated through an inverse thresholding binarization operation. The detected background region is filtered using a regional threshold, retaining only the approximate contour, and a series of non-overlapping slices are generated from this region for model training and validation.
[0007] Classification of breast cancer whole-slide pathology images is achieved by integrating a self-supervised contrastive learning framework with a weakly supervised multi-instance learning framework. In the first stage, the Swin Transformer is used as the backbone model, and a contrastive learning strategy is employed for feature extraction. The parameters from the online encoder are updated using gradient updates, while those from the target encoder are updated using momentum updates. The update strategy is as follows:
[0008] θ k ←mθ k +(1-m)θ q
[0009] where θ k represents the target encoder, θ q represents the online encoder, m is the momentum value, and the default initial value is 0.99. During training, the dot product operation is used to measure the similarity of different samples. This is done by calculating the query vector q generated by the online encoder and the encoding vector queue K = [K1, K2, ..., K N ]. Samples with higher correlation are used as positive samples and samples with lower correlation are used as negative samples. With contrast loss as the goal, the model can automatically learn valuable feature representations by minimizing the distance between positive samples and maximizing the distance between negative samples. The contrast loss calculation formula is as follows:
[0010]
[0011] L q is the contrast loss value, γ is the temperature coefficient, K is the size of the keyword queue, k o represents the target feature from the positive sample, k i is the target feature in the keyword queue;
[0012] In the second stage, the second-stage network is initialized using the optimal weights from the first stage. A weakly supervised multi-instance learning method is then used to select key example features from each full-slice image. Considering the spatial information of the features and the correlation between slices, the Transformer encoder is used to adaptively aggregate features. In the second stage, a method based on weakly supervised multi-instance learning is used to optimize the model. The input image is divided into 4×4 local windows through image block partitioning, and converted into fixed-dimensional image block vectors through convolution operations or linear transformations. A positional encoding is added to each image block vector to preserve spatial information. The image block vectors are input into successive Swin Transformer blocks to transfer information between windows. Low-resolution windows share parameters and interact with high-resolution windows. The calculation of successive Swin Transformer blocks is:
[0013]
[0014]
[0015]
[0016]
[0017] After passing through the multi-layer Transformer encoder, the obtained feature representation is compressed through a global pooling layer or a global average pooling layer to generate a fixed-length feature vector. Finally, the feature vector is mapped to the probability distribution of the predicted category through the Softmax function. According to the predicted probability, the example with the highest probability is selected as the representative example to train the SwinTransformer model. The weights trained using the weakly supervised multi-instance learning method are used to predict the probability of positive examples in each full-slice breast cancer pathology image. The top n examples with the highest predicted probability are selected as the input of the Transformer encoder. The dimension of the Transformer encoder matches the number of N. In order to obtain the corresponding query vector (Query, Q), key vector (Key, K) and value vector (Value, V), the input feature vector is linearly projected and dot-producted with three different weight matrices. The calculation formula is as follows:
[0018] Q=W q X, K = W k X, V = W v X
[0019] Use the attention weight to multiply the corresponding value vector to calculate the self-attention score of the input vector. The specific calculation formula is as follows:
[0020]
[0021] The self-attention module captures the important correlations between different regions in the image. Multi-head self-attention is projected into multiple subspaces through multiple different linear transformations. The correlation weights are calculated in each subspace, and the obtained weighted representations are concatenated to form the final representation:
[0022] MultiHead(Q,K,V)=Concat(SA1,...,SA h )W o
[0023] Where W o is the projection matrix, and h represents the number of feature spaces. The Transformer encoder described above completes feature fusion and achieves packet-level classification.
[0024] The first-stage model is trained and validated by inputting slice data from the training set, with network parameters updated at each iteration. The validation set results are used to determine the optimal weights for the first-stage encoder. These optimal weights are then used to initialize the second-stage model, which is further optimized using packet-level labels. The optimal weights for the second-stage model are saved based on the best validation set accuracy.
[0025] The performance of the two-stage model is evaluated on the test set based on the optimal weights in the validation set.
[0026] By employing the aforementioned technical solutions and combining self-supervision with weak supervision, this study proposes a multi-instance breast cancer whole-slide pathology image classification method. This method leverages a self-supervised contrastive learning framework to fully exploit the internal structural information of unlabeled pathology image slices. To address the class imbalance issue, a weakly supervised approach based on multi-instance learning is further employed to optimize the model, resulting in an instance-level feature extractor with robust representational capabilities. Finally, a Transformer encoder is used to enhance the correlation between selected examples and adaptively aggregate key instance features, further improving the accuracy of breast cancer whole-slide pathology image classification. BRIEF DESCRIPTION OF THE DRAWINGS
[0027] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments recorded in this application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.
[0028] Figure 1 Flow chart of the method of the present invention
[0029] Figure 2 This is the flow chart of the full-slice pathology image preprocessing in the present invention
[0030] Figure 3 This is the overall structure diagram of the model in the present invention
[0031] Figure 4 This is the structure diagram of the self-supervised training model in the present invention
[0032] Figure 5 This is the feature fusion structure diagram of the present invention DETAILED DESCRIPTION
[0033] To make the technical solutions and advantages of the present invention more clear, the technical solutions in the embodiments of the present invention are clearly and completely described below in conjunction with the accompanying drawings in the embodiments of the present invention:
[0034] like Figure 1 The figure shows a multi-instance breast cancer full-slide pathology image classification method based on a combination of self-supervision and weak supervision, which specifically includes the following steps:
[0035] S1: Get the Camelyon-16 breast cancer whole-slice pathology image dataset and its corresponding package-level labels. This dataset is used to detect whether breast cancer lymph nodes are metastatic after hematoxylin and eosin (H&E) staining. There are 399 whole-slice pathology images in total. Dataset = [W1, W2, ..., W 399 In this method, the dataset is divided into training set, validation set and test set in a ratio of 2:1:1. The training set is used for model training, the validation set is used to select the best weight of the model, and the test set is used to test the performance of the model.
[0036] S12: Screen each slice through the preprocessing method to remove a large amount of background areas and generate a series of 512×512 slices from the tissue area;
[0037] S13: Input all slices in the training set into the self-supervised model for training without using any labels;
[0038] S14: Further fine-tune the self-supervised model using a weakly supervised approach based on multi-instance learning, using only package-level labels;
[0039] S15: Select n examples with the highest probability from each full-slice image and input them into the Transformer encoder for example feature fusion;
[0040] S16: Obtain a packet-level representation of the full-slice image, and continuously update the encoder parameters based on the result.
[0041] S2: Preprocessing process such as Figure 2 As shown in the figure, a full-slice pathology image is read and a color space conversion function is applied to the image to convert it into a grayscale image. A binary mask of the foreground tissue area is generated for each full-slice breast cancer image through an inverse threshold binarization operation. The detected background area is filtered using a regional threshold, retaining only the approximate outline, and the area is gridded to generate a series of non-overlapping slices. To further reduce the amount of computation, this study only retains slices where the tissue area accounts for more than 40% of the image area. i =[P i1 ,P i2 ,......,P in ], where n is the number of slices in each full-size breast cancer pathology image.
[0042] S3: The overall architecture of the model is as follows Figure 3As shown in the figure, by adopting a strategy that combines self-supervised contrastive learning with a weakly supervised method based on multi-instance learning, an example-level feature extractor with strong representational capabilities is obtained. The correlation between selected examples is strengthened through the Transformer encoder, and key example features are adaptively aggregated to complete the classification of full-slice pathology images of breast cancer.
[0043] S31: All slices in the training set are input into the MoBY-based self-supervised framework for training without using any labels. Figure 4 As shown in Figure 2, the framework adopts a contrastive learning strategy for feature extraction. The parameters from the online encoder are updated using gradient updates, while the parameters from the target encoder are updated using momentum updates. The update strategy is:
[0044] θ k ←mθ k +(1-m)θ q
[0045] where θ k represents the target encoder, θ q represents the online encoder, m is the momentum value, and the default initial value is 0.99. During the training process, it acts on the original image and forms two enhanced new images [x1, x2] through data augmentation, which are positive samples of each other. During training, other images in the same batch are stored in a queue, and the samples in the queue are all negative samples of x1 or x2. During the training process, dot product calculation is used to measure the similarity between positive and negative samples. This involves the query vector q generated by the online encoder and the encoding vector queue K=[K1,K2,…,K N The specific formula is as follows:
[0046]
[0047] L q is the contrast loss value, γ is the temperature coefficient, K is the size of the keyword queue, k o represents the target feature from the positive sample, k i It is the target feature in the keyword queue; the image is mapped to a certain projection space through the contrast learning strategy, and the distance between positive examples is shortened and the distance between negative examples is extended in this space, so as to learn the internal structural information of the pathological image.
[0048] S32: The encoder of the second-stage network is initialized using the self-supervised weights of the first stage, and is further optimized using packet-level labels using a weakly supervised model based on multi-instance learning. Specifically, if the full slice image is labeled as positive, then at least one slice in it is considered to be positive; conversely, if the full slice image is labeled as negative, then all slices in it are considered to be negative. In this model, some key examples are first selected and assigned packet-level pseudo labels, and an example-level classifier is trained in a supervised manner. The feature extractor is then continuously trained by iteratively updating the pseudo labels of key examples and classifiers, and finally a feature vector of the slice is generated by global average pooling.
[0049] S33: By using the Softmax function to calculate the probability of all examples being positive and sorting them, the top n key examples with the largest probability are selected Z=[Z1,Z2,......,Z n ] represents the full slice image;
[0050] S34: Figure 5 As shown in the figure, the selected key examples are used as the input of the Transformer encoder to strengthen the spatial information of the features and the correlation between slices, and adaptively aggregate the features to form a packet-level feature vector. The input dimension d of the Transformer encoder is n. To obtain the corresponding query vector (Query, Q), key vector (Key, K), and value vector (Value, V), the input feature vector is linearly projected and then dot-producted with three different weight matrices. The calculation formula is as follows:
[0051] Q=W q X, K = W k X, V = W v X
[0052] Use the attention weight to multiply the corresponding value vector to calculate the self-attention score of the input vector. The specific calculation formula is as follows:
[0053]
[0054] The self-attention module captures the important correlations between different regions in the image. Multi-head self-attention is projected into multiple subspaces through multiple different linear transformations. The correlation weights are calculated in each subspace, and the obtained weighted representations are concatenated to form the final representation:
[0055] MultiHead(Q,K,V)=Concat(SA1,...,SA h )W o
[0056] Where Wo is the projection matrix, h represents the number of feature spaces, and two Transformer encoder modules are used. In each block, the hidden layer dimension is set to 2048 and h is set to 32. After the above Transformer encoder, feature fusion is completed to achieve packet-level classification.
[0057] S4: This network is iteratively trained using whole-slide pathology images and packet-level labels from the Camelyon-16 training set. The training process involves forward propagation of network features and backpropagation of errors. The first-stage network is first trained and validated using the training set, with network parameters updated at each iteration. Based on the validation set results, the optimal weights for the first-stage encoder are determined. These optimal weights are then used to initialize the second-stage network, which is further tuned and optimized using packet-level labels. Finally, the optimal weights for the second-stage network are saved based on the best validation set accuracy.
[0058] S5: Using the optimal weights to initialize the network model, the performance of the two-stage breast cancer whole-slide pathology image classification network was evaluated on the test set. Accuracy (ACC) and area under the curve (AUC) were used to comprehensively measure the performance of the model. Experimental results showed that the model achieved highly competitive results on the Camelyon-16 dataset, reaching ACC and AUC values of 97.67% and 96.94% respectively, compared to state-of-the-art multi-instance learning methods.
[0059] The above description is only a preferred specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any technician familiar with the technical field, within the technical scope disclosed by the present invention, who makes equivalent replacements or changes based on the technical solution and inventive concept of the present invention, should be covered by the scope of protection of the present invention.
Claims
1. A breast cancer full-slice image classification method combining self-supervision and weakly supervised learning, characterized by include: Obtain a dataset of breast cancer full-slice pathology images and their corresponding full-slice label information, and divide the dataset into a training set, a validation set, and a test set according to a certain ratio; The whole-slice pathology image dataset was preprocessed. The inverse threshold binarization method was used to generate a binary mask of the image foreground tissue area. The detected background area was filtered using regional thresholding to retain only the approximate contour. A series of non-overlapping slices were generated from this area for model training and validation. A two-stage breast cancer whole-slide pathology image classification network was constructed by combining a self-supervised contrastive learning framework with a weakly supervised multi-instance learning framework. In the first stage, the feature information of unlabeled slices was fully learned using a MoBY-based contrastive learning framework. The second stage uses a weakly supervised multi-instance learning method to select key example features from each full-slice image. Taking into account the spatial information of the features and the correlation between slices, a Transformer encoder is used to adaptively aggregate features to complete the classification of full-slice pathology images. The slice data in the training set is input into the first-stage model for training and validation, and the network parameters are updated in each iteration. Based on the results of the validation set, the optimal weights in the first-stage encoder are determined, and the second-stage model is initialized with the optimal weights. The second-stage model is optimized using packet-level labels and the optimal weights of the second-stage model are saved based on the optimal validation set accuracy. The performance of the two-stage breast cancer whole-slide pathology image classification network was evaluated on the test set based on the optimal weights in the validation set.
2. The method according to claim 1, wherein: In the first stage, Swin Transformer is used as the backbone model, and a contrastive learning strategy is adopted for feature extraction. The encoder parameters of the online branch are updated by gradient update, and the target branch encoder is updated by momentum update. The update strategy is as follows: i k ←mθ k +(1-m)θ q where θ k represents the target encoder, θ q represents the online encoder, m is the momentum value, the default initial value is 0.99, in training, the dot product operation is used to measure the similarity of different samples, which is calculated by calculating the query vector q generated by the online encoder and the encoding vector queue K = [K1, K2, ..., K N ], marking samples with higher correlation as positive samples and those with lower correlation as negative samples, and then using contrast loss to optimize the model. This process aims to narrow the distance between positive samples and expand the distance between negative samples to help the model automatically learn valuable feature representations. The specific formula of contrast loss is as follows: L q is the contrast loss value, γ is the temperature coefficient, K is the size of the keyword queue, k o represents the target feature from the positive sample, k i is the target feature in the keyword queue.
3. The method according to claim 2, wherein: In the second stage, a two-stage breast cancer full-slice pathology image classification network is optimized using a weakly supervised multi-instance learning method. The input image is divided into local windows through image partitioning, and converted into fixed-dimensional image block vectors through convolution operations or linear transformations. A position code is added to each image block vector to preserve spatial information. The image block vectors are input into consecutive Swin Transformer blocks to transfer information between windows. The low-resolution windows share parameters and interact with the high-resolution windows. The calculation of the consecutive Swin Transformer blocks is: After passing through the multi-layer Transformer encoder, the obtained feature representation is compressed through the global pooling layer or the global average pooling layer to generate a feature vector of fixed length. The feature vector is mapped to the probability distribution of the predicted category through the Softmax function. The example with the highest probability is selected as the representative example according to the predicted probability, and the Swin Transformer model is trained.
4. The method according to claim 3, wherein: The weights trained using a weakly supervised multi-instance learning method are used to predict the probability of a positive example in each full-slice breast cancer pathology image. The top n examples with the highest predicted probability are selected as the input to the Transformer encoder. The dimensions of the Transformer encoder match the input dimensions. To obtain the corresponding query vector (Query, Q), key vector (Key, K), and value vector (Value, V), the input feature vector is linearly projected and then dot-producted with three different weight matrices. The calculation formula is as follows: Q=W q X,K=W k X,V=W v X The self-attention score of the input vector is calculated by multiplying the attention weight with the corresponding value vector. The specific calculation formula is as follows: To capture the important correlations between different regions in the image, multi-head self-attention is projected into multiple subspaces through multiple different linear transformations. The correlation weights are calculated in each subspace separately, and the obtained weighted representations are concatenated to form the final representation: MultiHead(Q,K,V)=Concat(SA1,...,SA h )W o Where W o is the projection matrix, h represents the number of feature spaces, and the above-mentioned Transformer encoder completes feature fusion to achieve packet-level classification.