Self-optimization weak supervision pathological tissue image segmentation model based on prototype guidance

Abstract

The invention discloses a self-optimization weak supervision pathological tissue image segmentation model based on prototype guidance. The model comprises a classification stage and a segmentation stage. In the classification stage, a classification model is trained, and a pseudo label is generated for pseudo-supervised segmentation in the segmentation stage; the segmentation performance of the test set is evaluated on the BCSS-WSSS data set and the LUAD-HistoSeg data set by using the CAMs respectively; then selecting a check point with the best performance on the test set as a final weight, generating a cam, a similar graph, fusing the cam and refining a pseudo mask; in the segmentation stage, a segmentation model is trained through the pseudo mask; and evaluating the performance of the model on the test set and the verification set. According to the method, the information difference caused by weak supervision is made up in a more effective manner, so that the problem of under-activation is solved, a self-optimization method based on the multi-layer feature affinity relationship is introduced for denoising, and meanwhile, the noise influence is reduced by adopting a reliable region learning strategy.

Description

Technical Field

[0001] The present invention relates to the technical field of image segmentation, and in particular to a prototype-guided self-optimizing weakly supervised pathological tissue image segmentation model. Background Art

[0002] Weakly supervised pathological tissue image segmentation is a task that uses weakly supervised semantic segmentation technology to segment pathological tissue images. Weakly supervised semantic segmentation uses weakly supervised signals to complete the segmentation task (such as category labels, point annotations, box annotations, etc. that appear in the image). Compared with fully supervised semantic segmentation, it does not require pixel-level annotations, which can greatly reduce the workload of annotation. It is a technical means suitable for large-size, high-resolution full-slice digital pathological tissue images.

[0003] The main process of the current weakly supervised pathological tissue image segmentation technology is as follows: first, a classification network is trained to obtain the seed region based on the original class activation map (Class Activation Maps) to form a pseudo label. Then a segmentation network is trained with the pseudo label to obtain the segmentation result.

[0004] In traditional methods, since the supervisory signal is only annotated at the image level, the classification network will gradually focus only on the most prominent category area, thus failing to fully activate the target as a whole, which will have a great negative impact on the quality of pseudo-annotation. In addition, the noise in the pseudo-annotation will also have a certain impact on the segmentation performance.

[0005] The current mainstream existing technologies rely on various means to solve the under-activation problem, and the most representative one is MLPS. MLPS introduces the PDA mechanism to gradually erase areas with high activation levels, forcing the model to focus on other areas with insufficient activation strength, thereby expanding the activation area to achieve the purpose of solving the under-activation problem. Figure 1 As shown in the figure, in the classification stage, CNN classification network is used to generate CAMs. According to the activation strength of different categories in CAMs, PDA is used to expand the activation area and compensate for the impact of information difference. In the segmentation stage, the pseudo-annotation fusion results of multi-layer features are used to improve the pseudo-annotation quality, thereby improving the segmentation effect.

[0006] Although the MLPS method can alleviate the under-activation problem to a certain extent, it ignores the fact that the information related to the category of the entire data set can make up for the information gap from another perspective, so it has limitations. In addition, it lacks a means to deal with the noise problem, resulting in the noise problem in the segmentation stage still having a significant negative impact on the segmentation performance. Summary of the invention

[0007] The purpose of the present invention is to overcome the shortcomings and deficiencies of the above-mentioned prior art and provide a prototype-guided self-optimizing weakly supervised pathological tissue image segmentation model. The present invention compensates for the information gap caused by weak supervision in a more effective way, thereby solving the under-activation problem, and introduces a self-optimization method based on multi-layer feature affinity to denoise, while adopting a reliable region learning strategy to reduce the impact of noise.

[0008] The present invention is achieved through the following technical solutions: A prototype-guided self-optimizing weakly supervised pathological tissue image segmentation model, including a classification stage and a segmentation stage; In the classification stage, a classification model is trained to generate pseudo annotations for the pseudo-supervised segmentation in the second stage. We evaluate the segmentation performance of the test set using CAMs on the BCSS-WSSS and LUAD-HistoSeg datasets, respectively. Then the checkpoints with the best performance on the test set are selected as the final weights to generate CAMs, similarity maps, fused CAMs, and refined pseudo masks.

[0009] In the segmentation phase, a segmentation model is trained by pseudo-masking. We evaluate the performance of the model on the test set and the validation set. The results in Tables 1, 2, 3, and 4 are derived from the evaluation on the test set. These specific results are selected based on their best performance on the validation set.

[0010] In the classification stage, ResNet-38 is used as the classification network backbone, the learning rate is 0.01, the optimizer is Polyoptimizer, the weight decay is 5e-4, thr=0.8, δ=0.8; for BCSS-WSSS, the training length is 25 epochs, the batch size is 20, the sampling radius is r=3, α=0.25, β=2, and the number of random walk steps is 3; for LUAD-HistoSeg, the training length is 20 epochs, the batch size is 20, the sampling radius is r=2, α=0.5, β=4, and the number of random walk steps is 4.

[0011] In the segmentation stage, Unet++ with efficient-b3 as the backbone is used as the segmentation network, the learning rate is 5e-4, the optimizer is PolyAdamW, the weight decay is 0.05, and thrur = 0.5; the training length is 20 epochs, and the batch size is 20.

[0012] The above experiments are based on a single NVIDIA GeForce RTX 3090 GPU.

[0013] Compared with the prior art, the present invention has the following advantages and effects: 1. Prototype-based guidance: It mines more fine-grained information related to categories from the perspective of the entire dataset to make up for the information gap.

[0014] 2. Self-optimization based on multi-layer feature affinity. The affinity extracted by deep supervision corresponding features has local information and classification information, and self-optimization is performed through random walks to improve the quality of pseudo-labeling.

[0015] 3. A reliable region learning-based strategy mitigates the impact of noise on the segmentation stage.

[0016] In summary, the present invention compensates for the information gap caused by weak supervision in a more effective way, thereby solving the under-activation problem, and introduces a self-optimization method based on multi-layer feature affinity to denoise, while adopting a reliable region learning strategy to reduce the impact of noise. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 It is a flow chart of the prior art MLPS.

[0018] Figure 2 This is a schematic diagram of the overall process of prototype-based self-optimization weakly supervised pathological tissue image segmentation.

[0019] Figure 3 It is a detailed schematic diagram of the internal structure of PGM.

[0020] Figure 4 It is a detailed schematic diagram of the internal structure of AMP. DETAILED DESCRIPTION

[0021] The present invention is further described in detail below in conjunction with specific embodiments.

[0022] The present invention discloses a prototype-guided self-optimizing weakly supervised pathological tissue image segmentation model, which includes a classification stage and a segmentation stage; In the classification stage, a classification model is trained to generate pseudo-annotations for the pseudo-supervised segmentation in the segmentation stage; The segmentation performance of the test set was evaluated using CAMs on the BCSS-WSSS and LUAD-HistoSeg datasets, respectively. Then, the checkpoints with the best performance on the test set were selected as the final weights to generate CAMs, similarity maps, fused CAMs, and refined pseudo masks. In the segmentation stage, a segmentation model is trained by pseudo-masking; the performance of the model is evaluated on the test set and the validation set.

[0023] In the classification stage, ResNet-38 is used as the classification network backbone, the learning rate is 0.01, the optimizer is Polyoptimizer, the weight decay is 5e-4, thr=0.8, δ=0.8; for BCSS-WSSS, the training length is 25 epochs, the batch size is 20, the sampling radius is r=3, α=0.25, β=2, and the number of random walk steps is 3; for LUAD-HistoSeg, the training length is 20 epochs, the batch size is 20, the sampling radius is r=2, α=0.5, β=4, and the number of random walk steps is 4.

[0024] In the segmentation stage, Unet++ with efficient-b3 as the backbone is used as the segmentation network, the learning rate is 5e-4, the optimizer is PolyAdamW, the weight decay is 0.05, and thrur = 0.5; The training length is 20 epochs and the batch size is 20. The above experiments are based on a single NVIDIA GeForce RTX3090 GPU.

[0025] The classification stage and segmentation stage are further explained as follows: The classification stage uses deep supervision to perform weighted addition of classification scores of multiple layers of features to perform classification supervision; The PGM module selects the best features as category prototypes according to the activation of CAMs, and uses them and image features to generate similarity maps. Similarity maps provide complementary information of CAMs to optimize them. After the classification training is completed, the seed region Fused CAMs is obtained after the fusion of CAMs and Similarity maps. In order to further optimize the quality of pseudo-labeling, a module ARM based on multi-layer feature affinity is introduced. This module ARM extracts affinity relationships from the multi-layer features corresponding to deep supervision ( Figure 3 ), convert the sampled affinity matrix into a transfer matrix, and optimize the original seed region using a random walk algorithm ( Figure 1 ); After obtaining the optimized pseudo-annotations, the reliable region learning strategy RRL is adopted in the segmentation training in the segmentation stage; this strategy RRL generates a loss map Ml by calculating the cross entropy loss corresponding to each pixel position, which is used to measure whether the classification result of the position is reliable; the positions where the cross entropy loss is higher than a certain threshold thrur are determined as unreliable regions Rur, thereby preventing the gradients of these regions from being back-propagated during back propagation, thereby reducing the impact of noise on segmentation training.

[0026] Through experiments, the present invention compares the performance with the prior art on the public pathological tissue image datasets BCSS-WSSS and LUAD-HistoSeg. The evaluation indicators are Intersection over Union (IoU), meanIntersection over Union (mIoU) and Frequencyweighted Intersection over Union (FwIoU).

[0027] Table 1. Evaluation results on the BCSS-WSSS dataset. The best results are in bold. Table 2. Evaluation results on the LUAD-HistoSeg dataset. The best results are in bold. Table 3. Ablation results on the BCSS-WSSS dataset. The best results are in bold. Table 4. Ablation results on the LUAD-HistoSeg dataset. The best results are in bold. Tables 1 and 2 above show the comparison results between the present invention and the existing advanced methods, proving that the performance of the present method is better than the existing methods on both datasets. Tables 3 and 4 show the ablation results, and each module improves the performance, proving the effectiveness of our module.

[0028] As described above, the present invention can be implemented well.

[0029] The implementation methods of the present invention are not limited to the above-mentioned embodiments, and any other changes, modifications, substitutions, combinations, and simplifications made without departing from the spirit and principles of the present invention shall be equivalent replacement methods and shall be included in the protection scope of the present invention.

Claims

1. A prototype-guided self-optimizing weakly supervised pathological tissue image segmentation model, characterized in that: It includes classification stage and segmentation stage; In the classification stage, a classification model is trained to generate pseudo-annotations for pseudo-supervised segmentation in the segmentation stage; The segmentation performance of the test set was evaluated using CAMs on the BCSS-WSSS and LUAD-HistoSeg datasets, respectively. Then, the checkpoints with the best performance on the test set were selected as the final weights to generate CAMs, similarity maps, fused CAMs, and refined pseudo masks. In the segmentation stage, a segmentation model is trained by pseudo-masking; the performance of the model is evaluated on the test set and the validation set.

2. According to the prototype-guided self-optimizing weakly supervised pathological tissue image segmentation model of claim 1, it is characterized in that: In the classification stage, ResNet-38 is used as the classification network backbone.

3. According to the prototype-guided self-optimizing weakly supervised pathological tissue image segmentation model of claim 2, it is characterized in that: In the process of using ResNet-38 as the classification network backbone, the learning rate is 0.01, the optimizer is Polyoptimizer, the weight decay is 5e-4, thr=0.8, δ=0.8; For BCSS-WSSS, the training length is 25 epochs, the batch size is 20, the sampling radius is r=3, α=0.25, β=2, and the number of random walk steps is 3; For LUAD-HistoSeg, the training length is 20 epochs, the batch size is 20, the sampling radius is r=2, α=0.5, β=4, and the number of random walk steps is 4.

4. The prototype-guided self-optimizing weakly supervised pathological tissue image segmentation model according to claim 1, characterized in that: In the segmentation stage, Unet++ with efficient-b3 as the backbone is used as the segmentation network.

5. The prototype-guided self-optimizing weakly supervised pathological tissue image segmentation model according to claim 4, characterized in that: In the process of using Unet++ with efficient-b3 as the main backbone as the segmentation network, the learning rate is 5e-4, the optimizer is PolyAdamW, the weight decay is 0.05, and thrur = 0.5; The training length is 20 epochs and the batch size is 20.

6. A prototype-guided self-optimizing weakly supervised pathological tissue image segmentation model, characterized in that: It includes classification stage and segmentation stage; The classification stage uses deep supervision to perform weighted addition of classification scores of multiple layers of features to perform classification supervision; The PGM module selects the best features as category prototypes according to the activation of CAMs, and uses them and image features to generate similarity maps. Similarity maps provide complementary information of CAMs to optimize them. After the classification training is completed, the seed region Fused CAMs is obtained after the fusion of CAMs and Similarity maps. In order to further optimize the quality of pseudo-labeling, an ARM module based on multi-layer feature affinity is introduced. The ARM module extracts affinity relationships from the multi-layer features corresponding to deep supervision, converts the sampled affinity matrix into a transfer matrix, and optimizes the original seed region through a random walk algorithm; After obtaining the optimized pseudo-annotations, the reliable region learning strategy RRL is adopted in the segmentation training in the segmentation stage; this strategy RRL generates a loss map Ml by calculating the cross entropy loss corresponding to each pixel position, which is used to measure whether the classification result of the position is reliable; the position where the cross entropy loss is higher than a certain threshold thrur is determined as an unreliable region Rur, thereby preventing the gradient backpropagation of these areas during backpropagation, thereby reducing the impact of noise on segmentation training.

Images