A negative sample guided self-distillation method for pathological image analysis
By employing a negative sample-guided self-distillation method, two classification networks were constructed for self-distillation learning and pseudo-label optimization. This solved the problems of instance classification and positive region identification in large-size pathological images, achieving high-precision automatic diagnosis of pathological images and localization of positive regions, thus improving diagnostic efficiency and accuracy.
Patent Information
- Application Number
- CN202211192738.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-09-28
- Publication Date
- 2026-01-06
- Estimated Expiration
- 2042-09-28
AI Technical Summary
Existing histopathological image analysis methods struggle to achieve effective instance classification and generalization when processing large-sized pathological images, especially in the accurate identification of positive regions. Furthermore, existing methods fail to fully utilize negative sample information, resulting in poor classification results.
We employ a negative sample-guided self-distillation method, constructing two classification networks with identical structures but different parameters. We then utilize self-distillation learning and the Sinkhorn-Knopp algorithm to optimize pseudo-labels, achieving end-to-end instance-level classification. Combined with information entropy filtering and data augmentation techniques, we train a high-precision positive region localization model.
It enables automatic classification of the entire pathological image and high-precision localization of positive areas, improving the accuracy and interpretability of the model and reducing the cost and time of pathological diagnosis.
Smart Images
Figure CN115482232B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of image processing and application technology, and in particular to a negative sample-guided self-distillation pathological image analysis method. Background Technology
[0002] Histopathology is the gold standard for cancer diagnosis, and digital histopathological images obtained through whole-slice imaging (WSI) scanners play an increasingly important role in clinical decision-making. Developing computer-aided diagnostic methods for automated analysis of histopathological images is highly valuable. On the one hand, histopathological images are very large, making analysis laborious for pathologists and potentially introducing subjective biases. On the other hand, some classification tasks can only be performed by computers, as pathologists cannot make decisions based on visual examination, such as metastasis prediction and survival prediction. In recent years, deep learning has been widely applied to histopathological image analysis tasks, such as metastasis detection, cancer subtype classification, and survival prediction. Therefore, developing a deep multi-instance method capable of displaying the distribution of modeled data for automated diagnosis of whole-slice pathological images is a pressing problem that needs to be solved.
[0003] A major challenge in computer-aided analysis of histopathological images is the sheer size of the entire pathological image. For example, a typical breast or colon cancer WSI is 40,000 × 40,000 pixels, much larger than a normal natural image, making it impossible to feed the entire image into a deep learning model. Therefore, WSIs are typically cropped into many image patches that current computers can process. However, image patches cannot simply inherit the labels of their corresponding WSIs because only a subset of image patches are positive in a positive WSI. Furthermore, due to the sheer number of image patches, it is often impossible to label each one individually. Therefore, WSI classification is often viewed as a weakly supervised multiple instance learning (MIL) problem, where the entire pathological image is treated as a bag, and image patches as instances. Unlike traditional MIL problems that focus solely on bag classification, instance classification, also known as positive region localization, is also important in WSI analysis.
[0004] Most mainstream MIL methods use bag classification. They first extract instance features and aggregate the features of instances in a bag to obtain bag features, and then train a bag classifier using the bag labels. This approach has two problems. First, its instance classification ability is limited. Bag classification methods focus on training the bag classifier, and the classification of instances usually depends on the attention scores learned by the network. Since the loss function is built on the bag classification results, and the loss function is minimized after each bag is correctly classified, even if the network can correctly classify all bags, it cannot guarantee that all instances can be distinguished using attention scores. Second, the generalization ability of bag classification methods is limited. Due to the gap between the obtained bag classifier and the optimal classifier, bags falling into the gap may be misclassified. Due to the above shortcomings of bag classification methods, end-to-end training of instance-level classifiers may be a better solution. However, existing instance-level methods are not as good as bag classification methods because it is difficult to find enough true positive instances from weak bag labels for model training. Summary of the Invention
[0005] The purpose of this invention is to provide a negative sample-guided self-distillation pathological image analysis method to achieve automatic diagnosis and positive region localization of the entire pathological image, thereby reducing the cost of pathological diagnosis.
[0006] To achieve the above objectives, the present invention provides the following solution:
[0007] A negative sample-guided self-distillation pathological image analysis method includes:
[0008] The entire pathological image is preprocessed to obtain a first classification view and a second classification view;
[0009] Construct a first classification network and a second classification network, and perform self-distillation learning training on the first classification network based on the first classification view to obtain a first prediction result;
[0010] The second classification network is trained using the first prediction result and the second classification view to obtain the first loss function and the second loss function, respectively.
[0011] The parameters of the second classification network are optimized based on the first loss function and the second loss function, and the parameters of the first classification network are updated at the same time to obtain the trained first classification network and the second classification network.
[0012] The trained second classification network is used to perform pathological image analysis to obtain the analysis results.
[0013] Preferably, the entire pathological image is preprocessed, including:
[0014] The entire pathological image is divided into several image blocks of equal size without overlap. The information entropy of all the image blocks is calculated, and background image blocks with information entropy less than a preset threshold are screened out. Then, all image blocks from positive pathological images and all image blocks from negative pathological images are mixed respectively.
[0015] Preferably, the preprocessing further includes:
[0016] The sampled image patch is subjected to data augmentation. The data augmentation method includes flipping the image patch with different probabilities in the top, bottom, left, and right directions, performing grayscale transformation on the image patch, and independently performing several random data augmentations on the image patch to obtain the first classification view and the second classification view, respectively.
[0017] Preferably, the first classification network and the second classification network each include an encoder and a classification head network, and the parameters of the first classification network are different from the parameters of the second classification network.
[0018] Preferably, training the first classification network based on the first classification view includes:
[0019] The local slice in the first category view The input is fed into the first classification network for self-distillation training, and the first prediction result is output. The first prediction result As the local slice pseudo-tags:
[0020]
[0021] in, Represents data augmentation operations, This represents the first category network.
[0022] Preferably, the first classification network further includes a Prediction Bank unit. The Prediction Bank unit is used to prevent self-distillation learning after negative sample guidance from falling into degradation. After obtaining the first prediction result, the first prediction result is input into the Prediction Bank unit, and the pseudo-label is updated based on the Sinkhorn-Knopp method.
[0023] Preferably, the solution based on the Sinkhorn-Knopp method includes:
[0024]
[0025] Among them, China|X p | Represents set Xp The number of positive local image patches, K represents the estimated proportion of positive local image patches in the whole positive image, and CE(·) represents the cross-entropy loss function.
[0026] Preferably, obtaining the first loss function includes:
[0027] The first prediction result is input into the second classification network for training to obtain the first output result. The cross-entropy between the first output result and the pseudo-label is calculated to obtain the first loss function L. self-distillation :
[0028]
[0029] in, This represents the second category network.
[0030] Preferably, obtaining the second loss function includes:
[0031] The second classification view is input into the second classification network for training to obtain the second output result. The cross-entropy between the second output result and the second classification view is calculated to obtain the second loss function L. guidance :
[0032]
[0033] in, A local section representing a negative result.
[0034] Preferably, optimizing the parameters of the second classification network based on the first loss function and the second loss function includes: optimizing using the Adam optimization algorithm, as shown in the following formula:
[0035]
[0036] Where, θ s These are the parameters of the second classification network;
[0037] The parameters of the first classification network are updated using the momentum update method, including:
[0038] θ t ←λθ t +(1-λ)θ s
[0039] Where λ is the updated weight, θ t These are the parameters of the first classification network.
[0040] The beneficial effects of this invention are as follows:
[0041] The negative sample-guided self-distillation pathological image analysis method provided in this application can automatically classify the entire pathological image and predict the positive areas in the entire pathological image. Existing international methods cannot make full use of the negative sample information in the pathological image, resulting in poor classification results, especially in the inaccuracy of distinguishing positive areas. Compared with existing international methods, the method proposed in this application has better accuracy and model interpretability, and has the potential value of improving the efficiency and accuracy of clinical diagnosis. Attached Figure Description
[0042] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0043] Figure 1 This is a flowchart of the negative sample-guided self-distillation pathological image analysis method in an embodiment of the present invention;
[0044] Figure 2 This is a flowchart of the Prediction Bank in an embodiment of the present invention. Detailed Implementation
[0045] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0046] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.
[0047] This embodiment provides a negative sample-guided self-distillation pathological image analysis method. According to this application, automatic diagnosis and positive region localization of the entire pathological image can be achieved. The method provided in this application is simple to operate, possesses the highest accuracy and robustness currently available internationally, and automated detection saves time and costs, significantly reducing the cost of pathological diagnosis.
[0048] like Figure 1-2 The diagram shown is a flowchart of the method provided in this application, including:
[0049] S1. Preprocess the entire pathological image to obtain the first classification view and the second classification view;
[0050] First, the entire pathological image is divided into non-overlapping image blocks of size 512x512. Then, the information entropy of all image blocks is calculated, and background image blocks with an information entropy less than 15 are filtered out. Next, image blocks from positive pathological images and image blocks from negative pathological images are mixed separately, and 64 image blocks are randomly sampled from them each time for subsequent training.
[0051] For the sampled image patches, random data augmentation methods are applied, including random left-right flipping, random up-down flipping, and random color perturbation with a probability of 0.5, image grayscale transformation with a probability of 0.2, and image cropping at a scale of 80% to 100% of the original image size. After performing random data augmentation twice on the same image patch, two different views are obtained: a first classification view and a second classification view.
[0052] S2. Construct a first classification network and a second classification network. Based on the first classification view, perform self-distillation learning training on the first classification network to obtain a first prediction result.
[0053] The first classification network is the teacher network, and the second classification network is the student network. The teacher network uses ResNet18 as the encoder and a linear fully connected layer as the prediction head network. The student network uses the exact same network structure as the teacher network, but the network parameters are different.
[0054] During training, local slices of the entire pathological image are used. After data augmentation, the data is input into the teacher network, and the first prediction result is used as the pseudo-label for that slice.
[0055]
[0056] in, Represents data augmentation operations, This represents the first category network.
[0057] After obtaining the output of the teacher network, it is input into the Prediction Bank. Once the Prediction Bank stores the new teacher network predictions, the Sinkhorn-Knopp algorithm is used to solve the optimal transmission problem, thereby updating all pseudo-labels.
[0058]
[0059] Among them, China|X p | Represents set X p The number of positive local image patches, K represents the estimated proportion of positive local image patches in the whole positive image, and CE(·) represents the cross-entropy loss function.
[0060] S3. Train the second classification network using the first prediction result and the second classification view to obtain the first loss function and the second loss function, respectively.
[0061] The student network is trained using the second classification view, and the pseudo-labels predicted by the first classification network and the real labels of the negative samples are also used to train the student network.
[0062] Image patches from a positive whole image Its true label is unknown, so a pseudo label needs to be generated using the first network; while for image patches from the entire negative image... Since its true label is known and is negative, there is no need to generate a pseudo label.
[0063] In the student network, for image patches from positive pathological images, calculate the cross-entropy L between the output and the pseudo-labels in the Prediction Bank. self-distillation That is, the first loss function; while for image patches from negative images, the cross-entropy L between the output and the negative category is calculated. guidance That is, the second loss function:
[0064]
[0065] in, Represents the second classification network; Image patches representing images derived from a positive full-body pathology image;
[0066]
[0067] in, Image blocks representing images from a negative full-length pathology image.
[0068] S4. Optimize the parameters of the second classification network based on the first loss function and the second loss function, and update the parameters of the first classification network at the same time to obtain the trained first classification network and second classification network.
[0069] The Adam optimization algorithm is used to optimize the parameters of the student network. Simultaneously, based on the parameters of the thrombosis network, momentum updates are used to optimize the parameters of the teacher network.
[0070]
[0071] Where, θ s These are the parameters of the second classification network;
[0072] θ t ←λθ t +(1-λ)θ s
[0073] Where λ is the updated weight, θ t represents the parameters of the first classification network.
[0074] S5. Pathological image analysis is performed using the trained second classification network to obtain analysis results.
[0075] After training, the entire pathological image to be predicted is sliced and input into the trained student network to obtain the prediction results for all image patches. The image patches with high probabilities are the positive regions predicted by the network. The prediction results of all image patches are aggregated through a pooling operation to obtain the prediction result of whether the entire pathological image is positive.
[0076] This invention enables automatic classification of entire pathological images and localization of positive regions within those images. The method constructs two identical classification networks: a student network and a teacher network. These networks are trained using a self-distillation method. Prior knowledge that all image patches in a negative pathological image are negative guides this self-distillation process, thereby training an instance-level classifier end-to-end. The resulting instance-level classifier can classify local image patches within the entire pathological image. Pooling the classification results of all image patches yields the classification of the entire pathological image. This invention has achieved leading accuracy on multiple international public datasets and has the potential for clinical application, improving diagnostic efficiency and accuracy.
[0077] The embodiments described above are merely preferred embodiments of the present invention and are not intended to limit the scope of the present invention. Various modifications and improvements made by those skilled in the art to the technical solutions of the present invention without departing from the spirit of the present invention should fall within the protection scope defined by the claims of the present invention.
Claims
1. A negative sample guided self-distillation method for pathological image analysis, characterized in that, The method comprises the following steps: preprocessing the whole pathological image to obtain a first classification view and a second classification view; constructing a first classification network and a second classification network, performing self-distillation learning training on the first classification network based on the first classification view to obtain a first prediction result; training the second classification network based on the first prediction result and the second classification view to obtain a first loss function and a second loss function respectively; optimizing the parameters of the second classification network based on the first loss function and the second loss function, and updating the parameters of the first classification network at the same time to obtain trained first classification network and second classification network; performing pathological image analysis through the trained second classification network to obtain an analysis result; the first classification network and the second classification network each comprise an encoder and a classification head network, and the parameters of the first classification network are different from the parameters of the second classification network; training the first classification network based on the first classification view comprises: slices a local slice in the first classification view input into the first classification network for self-distillation training, output the first prediction result , the first prediction result as pseudo-labels of the local slice wherein, representing a data augmentation operation, representing a first classification network; obtaining the first loss function comprises: inputting the first prediction result into the second classification network for training, obtaining a first output result, and calculating cross-entropy of the first output result and the pseudo label to obtain a first loss function : wherein, represents a second classification network; obtaining the second loss function comprises: inputting the second classification view into the second classification network for training, obtaining a second output result, calculating cross entropy of the second output result and the second classification view to obtain a second loss function : wherein Representative negative sections.
2. The negatively sampled guided self-distilling pathological image analysis method of claim 1, wherein, preprocessing the whole pathological image comprises: cutting the whole pathological image into several image blocks of equal size without overlapping, calculating the information entropy of all the image blocks, and screening out background image blocks with information entropy less than a preset threshold; then mixing all the image blocks from positive pathological images and all the image blocks from negative pathological images respectively.
3. The negatively sampled guided self-distilling pathological image analysis method of claim 2, wherein, The preprocessing further comprises: performing data enhancement on the sampled image blocks, the data enhancement method comprising flipping the image blocks in different probabilities up, down, left and right, performing grayscale transformation on the image blocks, and independently performing random data enhancement on the image blocks several times to obtain the first classification view and the second classification view respectively.
4. The negatively sampled guided self-distilling pathological image analysis method of claim 1, wherein, The first classification network further comprises a Prediction Bank unit, which is used to prevent the self-distillation learning guided by negative samples from falling into degradation, and after obtaining the first prediction result, the first prediction result is input into the Prediction Bank unit, and the Sinkhorn-Knopp method is used to solve and update the pseudo-label.
5. The negatively sampled guided self-distilling pathological image analysis method according to claim 4, characterized in that, Solving based on the Sinkhorn-Knopp method comprises: wherein the number of positive local image patches in the positive whole image is represented by representative set the number of positive local image patches in the positive whole image, representing the estimated proportion of positive local image patches in the positive whole image, (·) represents the cross-entropy loss function.
6. The negatively sampled guided self-distilling pathological image analysis method of claim 1, wherein, optimizing the parameters of the second classification network based on the first loss function and the second loss function comprises: using Adam optimization algorithm for optimization, as follows: wherein, are parameters of the second classification network; updating the parameters of the first classification network by momentum update method comprises: wherein, is the updated weight, is a parameter of the first classification network.