Prototype-guided self-optimizing weakly supervised pathological tissue image segmentation model

Abstract

The application discloses a kind of self-optimization weakly supervised pathological tissue image segmentation model based on prototype guide, including classification stage and segmentation stage;In classification stage, a classification model is trained, and pseudo-labeling is generated for pseudo-supervised segmentation of segmentation stage;The segmentation performance of test set is evaluated using CAMs on BCSS-WSSS and LUAD-HistoSeg dataset respectively;Then the best checkpoint is selected as the best value on the test set, and cam, similar graph, fused cam and refined pseudo-mask are generated;In the segmentation stage, a segmentation model is trained through the pseudo-mask;The performance of the model is evaluated on the test set and the validation set.The application compensates for the information gap caused by weak supervision in a more effective way, thereby solving the underactivation problem, and introduces a self-optimization method based on multi-layer feature affinity to denoise, while using a reliable region learning strategy to reduce noise influence.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image segmentation, and in particular to a self-optimizing weakly supervised pathological tissue image segmentation model based on prototype guidance. BACKGROUND

[0002] Weakly supervised pathological tissue image segmentation is a task of segmenting pathological tissue images through weakly supervised semantic segmentation technology. Weakly supervised semantic segmentation completes the segmentation task through weakly supervised signals (such as class labels, point annotations, and frame annotations appearing in images), and compared with fully supervised semantic segmentation, it can greatly reduce the workload of labeling because it does not require pixel-level labeling, and is a technical means suitable for large-size and high-resolution whole-slice digital pathological tissue images.

[0003] The current weakly supervised pathological tissue image segmentation technology mainly includes the following steps: first, training a classification network to obtain seed regions based on original class activation maps (CAMs) to form pseudo-labels; then training a segmentation network through the pseudo-labels; and finally obtaining a segmentation result.

[0004] The traditional method has the problem that because the supervision signal is only image-level labeling, the classification network will gradually only focus on the most prominent class region, so that the target whole cannot be fully activated, which greatly negatively affects the quality of the pseudo-labels. In addition, the noise in the pseudo-labels also has a certain impact on the segmentation performance.

[0005] The current mainstream existing technology relies on various means to solve the under-activation problem, and the most representative one is MLPS. MLPS introduces a PDA mechanism to gradually erase regions with high activation levels, forcing the model to focus on other regions with insufficient activation intensity, so as to expand the activation region to achieve the purpose of solving the under-activation problem. As shown in FIG. 1, it uses a CNN classification network to generate CAMs in the classification stage, and according to the activation intensity of different classes in the CAMs, it uses PDA to expand the activation region to make up for the impact of information difference. In the segmentation stage, the pseudo-label fusion result of multiple layers of features is used to improve the quality of the pseudo-labels, thereby improving the segmentation effect. Figure 1

[0006] Although the MLPS method can alleviate the under-activation problem to some extent, it ignores the fact that the class-related information of the entire data set can make up for the information difference from another angle, so it has limitations. In addition, it lacks means to deal with noise problems, so the noise problem in the segmentation stage still has a great negative impact on the segmentation performance. SUMMARY

[0007] ​The present application aims to overcome the shortcomings and deficiencies of the prior art, and provides a prototype-guided self-optimization weakly supervised pathological tissue image segmentation model.

[0008] The present application is implemented by the following technical solutions:

[0009] A prototype-guided self-optimization weakly supervised pathological tissue image segmentation model includes a classification stage and a segmentation stage.

[0010] In the classification stage, a classification model is trained to generate pseudo-labels for the pseudo-supervised segmentation in the second stage. We evaluated the segmentation performance of the test set using CAMs on the BCSS-WSSS and LUAD-HistoSeg datasets. Then, the checkpoint with the best performance on the test set was selected as the best weight value to generate cam, similarity map, fused cam and refined pseudo mask.

[0011] In the segmentation stage, a segmentation model is trained using the pseudo mask. We evaluated the performance of the model on the test set and the validation set. The results in Tables 1, 2, 3 and 4 are derived from the evaluation of the test set. These specific results are selected based on their best performance on the validation set.

[0012] In the classification stage, ResNet-38 is used as the backbone of the classification network, the learning rate is 0.01, the optimizer is Polyoptimizer, the weight decay is 5e-4, thr=0.8, δ=0.8; for BCSS-WSSS, the training length is 25 epochs, the batch size is 20, the sampling radius is r=3, α=0.25, β=2, and the random walk step is 3; for LUAD-HistoSeg, the training length is 20 epochs, the batch size is 20, the sampling radius is r=2, α=0.5, β=4, and the random walk step is 4.

[0013] In the segmentation stage, Unet++ with efficient-b3 as the backbone is used as the segmentation network, the learning rate is 5e-4, the optimizer is PolyAdamW, the weight decay is 0.05, and thrur =0.5; the training length is 20 epochs, and the batch size is 20.

[0014] The above experiments are based on a single NVIDIA GeForce RTX 3090 GPU.

[0015] Compared with the prior art, the present application has the following advantages and effects:

[0016] 1. Prototype-based guidance. More fine-grained information related to categories is mined from the perspective of the whole dataset to make up for the information gap.

[0017] 2. Self-optimization based on multi-layer feature affinity. The affinity relationship of deep supervision corresponding feature extraction has local information and classification information, and self-optimization is performed through random walk to improve the pseudo-labeling quality.

[0018] 3. Strategy based on reliable region learning to reduce the impact of noise on the segmentation stage.

[0019] In summary, the present application makes up for the information gap brought by weak supervision in a more effective way, thereby solving the under-activation problem, and introduces a self-optimization method based on multi-layer feature affinity to denoise, while using a reliable region learning strategy to reduce the impact of noise. BRIEF DESCRIPTION OF DRAWINGS

[0020] Figure 1 is a flowchart of the prior art MLPS.

[0021] Figure 2 is a schematic diagram of the overall process of the prototype-based self-optimization weakly supervised pathological tissue image segmentation.

[0022] Figure 3 is a detailed schematic diagram of the internal structure of PGM.

[0023] Figure 4 is a detailed schematic diagram of the internal structure of AMP. DETAILED DESCRIPTION

[0024] The present application will be further specifically and in detail described below in conjunction with specific embodiments.

[0025] The present application discloses a prototype-guided self-optimization weakly supervised pathological tissue image segmentation model, which includes a classification stage and a segmentation stage.

[0026] In the classification stage, a classification model is trained to generate pseudo-labels for the pseudo-supervised segmentation of the segmentation stage.

[0027] The segmentation performance of the test set was evaluated using CAMs on the BCSS-WSSS and LUAD-HistoSeg datasets; then the best checkpoint was selected as the best value on the test set, and cam, similarity map, fused cam and refined pseudo mask were generated.

[0028] In the segmentation stage, a segmentation model is trained through the pseudo mask; the performance of the model is evaluated on the test set and the validation set.

[0029] In the classification stage, ResNet-38 is used as the backbone of the classification network, the learning rate is 0.01, the optimizer is Polyoptimizer, the weight decay is 5e-4, thr = 0.8, and δ = 0.8; for BCSS-WSSS, the training length is 25 epochs, the batch size is 20, the sampling radius is r = 3, α = 0.25, β = 2, and the random walk step is 3; for LUAD-HistoSeg, the training length is 20 epochs, the batch size is 20, the sampling radius is r = 2, α = 0.5, β = 4, and the random walk step is 4.

[0030] In the segmentation stage, Unet++ with efficient-b3 as the backbone is used as the segmentation network, the learning rate is 5e-4, the optimizer is PolyAdamW, the weight decay is 0.05, and thrur = 0.5.

[0031] The training length is 20 epochs, and the batch size is 20. The above experiments are based on a single NVIDIA GeForce RTX3090 GPU.

[0032] The classification stage and the segmentation stage are further described as follows:

[0033] In the classification stage, deep supervision is used to add the weighted sum of classification scores of multiple layers of features to perform classification supervision.

[0034] The PGM module selects the optimal feature as the class prototype according to the activation of the CAMs, and generates similarity maps using the prototype and the image features. The similarity maps provide complementary information for the CAMs, thereby optimizing the CAMs.

[0035] After the classification training is completed, the seed region Fused CAMs after the fusion of the CAMs and the similarity maps is obtained. To further optimize the pseudo-label quality, an ARM module based on the affinity relationship of multiple layers of features is introduced.

[0036] The ARM module extracts the affinity relationship from the multiple layers of features corresponding to the deep supervision ( Figure 3 ), converts the sampled affinity matrix into a transition matrix, and optimizes the original seed region through the random walk algorithm ( Figure 1 );

[0037] After obtaining the optimized pseudo-label, a reliable region learning strategy RRL is used in the segmentation training in the segmentation stage; the strategy RRL generates a loss map Ml by calculating the cross-entropy loss corresponding to each pixel position, for measuring whether the classification result of the position is reliable; the positions with cross-entropy loss higher than a threshold thrur are determined as unreliable regions Rur, so as to prevent the gradient of these regions from being back-propagated when back-propagating, thereby reducing the influence of noise on the segmentation training.

[0038] The present application compares the performance on the public pathological tissue image dataset BCSS-WSSS and LUAD-HistoSeg through experiments and prior art, and the evaluation indexes are Intersection over Union (IoU), mean Intersection over Union (mIoU) and Frequencyweighted Intersection over Union (FwIoU).

[0039] Table 1. Evaluation results on the BCSS-WSSS dataset, the optimal result is in bold

[0040]

[0041] Table 2. Evaluation results on the LUAD-HistoSeg dataset, the optimal result is in bold

[0042]

[0043] Table 3. Ablation results on the BCSS-WSSS dataset, the optimal result is in bold

[0044]

[0045] Table 4. Ablation results on the LUAD-HistoSeg dataset, the optimal result is in bold

[0046]

[0047] The above table 1 and table 2 show the comparison results of the present application and the prior art method, which proves that the performance of the present method is better than that of the prior art method on two datasets. Table 3 and table 4 show the ablation results, each module improves the performance, which proves the effectiveness of our module.

[0048] As described above, the present application can be better realized.

[0049] The embodiments of the present application are not limited to the above examples, and any changes, modifications, substitutions, combinations, simplifications made without departing from the spirit and principles of the present application shall be equivalent replacements and shall be included in the protection scope of the present application.

Claims

1. A prototype-guided based self-optimizing weakly supervised pathological tissue image segmentation model, characterized in that, The classification stage and the segmentation stage are included. The classification stage adopts a deep supervision manner, and scores obtained by classifying multiple features are weighted and added to perform classification supervision. The PGM module screens optimal features as class prototypes according to activation conditions of the CAMs, and generates similarity maps by using the class prototypes and image features, so as to provide complementary information of the CAMs and optimize the CAMs. After the classification training is completed, a seed region fused with the CAMs and the similarity maps is obtained, and an ARM module based on affinity relationships of multiple features is introduced to further optimize the pseudo-labeling quality. The ARM module extracts the affinity relationships from multiple features corresponding to deep supervision, converts a sampled affinity matrix into a transition matrix, and optimizes the original seed region by using a random walk algorithm. After the optimized pseudo-labeling is obtained, a reliable region learning strategy RRL is used in segmentation training in the segmentation stage.

2. The self-optimizing weakly supervised histopathological image segmentation model based on prototype guidance according to claim 1, wherein, In the classification stage, a classification model is trained to generate pseudo-labeling for pseudo-supervised segmentation in the segmentation stage. The segmentation performance of the CAMs on the test set is evaluated on the BCSS-WSSS and LUAD-HistoSeg data sets, respectively. In the segmentation stage, a segmentation model is trained by using the pseudo-mask, and the performance of the model is evaluated on the test set and the validation set.

3. The self-optimizing weakly supervised histopathological image segmentation model based on prototype guidance according to claim 2, wherein, In the classification stage, ResNet-38 is used as a classification network backbone.

4. The self-optimizing weakly supervised histopathological image segmentation model based on prototype guidance according to claim 3, wherein, In the process of using ResNet-38 as the classification network backbone, the learning rate is 0.01, the optimizer is Polyoptimizer, the weight decay is 5e-4, thr is 0.8, and δ is 0.

8. For the BCSS-WSSS, the training length is 25 epochs, the batch size is 20, the sampling radius is r=3, α is 0.25, β is 2, and the random walk step number is 3. For the LUAD-HistoSeg, the training length is 20 epochs, the batch size is 20, the sampling radius is r=2, α is 0.5, β is 4, and the random walk step number is 4.

5. The self-optimizing weakly supervised histopathological image segmentation model based on prototype guidance according to claim 2, wherein, In the segmentation stage, Unet++ with efficient-b3 as the backbone is used as the segmentation network.

6. The self-optimizing weakly supervised histopathological image segmentation model based on prototype guidance according to claim 5, wherein, In the process of using Unet++ with efficient-b3 as the backbone as the segmentation network, the learning rate is 5e-4, the optimizer is PolyAdamW, the weight decay is 0.05, and thrur is 0.

5. The training length is 20 epochs, and the batch size is 20.

Images