Cell nucleus segmentation method based on weak supervision and semi-supervision

Abstract

The invention discloses a cell nucleus segmentation method based on weak supervision and semi-supervision, and the method comprises the following steps: S1, selecting any pathological image in a breast cancer pathological image library and part of point labels thereof as to-be-detected images, and carrying out the initial training of mark points through employing an extended Gaussian filtering algorithm; then, a background propagation method is adopted to carry out self-training, a relatively accurate cell nucleus detection result is obtained through continuous iterative updating, a course learning strategy is utilized, a model is trained by using a simple sample, and then the difficulty of the sample is gradually increased along with the improvement of the model so as to enhance the learning ability of the model; and S2, generating three weak tags by using the complete point tag obtained in the S1, wherein the three weak tags comprise: 1) generating a Voronoi tag; according to the method, the requirement on the model is reduced, limited weak label data can be utilized, relatively high segmentation accuracy can be kept, meanwhile, the separation efficiency is also improved, and processing of edge and shape information of the cell nucleus is facilitated.

Description

Technical Field

[0001] The present invention relates to the field of medical image segmentation, and in particular to a cell nucleus segmentation method based on weak supervision and semi-supervision. Background Art

[0002] Pathological image analysis is an important part of medical diagnosis, among which cell nucleus segmentation is one of the key steps. However, due to the complexity of pathological images, cell nucleus segmentation still faces many challenges.

[0003] Traditional image processing techniques such as OTSU threshold and watershed segmentation algorithms have poor effects on cell crowding or adhesion, and can easily lead to under-segmentation or over-segmentation. With the development of deep learning, researchers have explored the field of cell nucleus segmentation in pathological images more deeply, and have achieved higher accuracy in completing cell nucleus segmentation tasks through deep learning methods.

[0004] However, fully supervised learning methods require a large amount of manual pixel-by-pixel annotation, which is a very time-consuming and labor-intensive process and requires pathologists to have rich prior knowledge, which greatly limits the practical application of fully supervised methods.

[0005] In recent years, weakly supervised learning and semi-supervised learning have received widespread attention in the field of image segmentation. Weakly supervised learning uses less labeled data for training, which greatly reduces the labeling cost, but has higher requirements on the model. Semi-supervised learning combines labeled data and unlabeled data to improve the performance of the model by utilizing the information in the unlabeled data. Although some studies have made certain progress in the field of weakly supervised and semi-supervised cell nucleus segmentation, the existing methods still need to be improved in segmentation accuracy and efficiency, especially when processing the edge and shape information of the cell nucleus, the performance of existing methods is not ideal. Summary of the invention

[0006] The purpose of the present invention is to solve the shortcomings of the prior art and propose a cell nucleus segmentation method based on weak supervision and semi-supervision.

[0007] In order to achieve the above object, the present invention adopts the following technical solutions:

[0008] A cell nucleus segmentation method based on weak supervision and semi-supervision includes the following steps:

[0009] S1. Select any pathological image and some of its point labels in the breast cancer pathological image library as the image to be detected, and use the extended Gaussian filtering algorithm to perform initial training on the marked points; then use the background propagation method for self-training, and obtain more accurate cell nucleus detection results through continuous iterative updates, and use the course learning strategy to first train the model with simple samples, and then gradually increase the difficulty of the samples as the model improves to enhance the learning ability of the model;

[0010] S2. Use the complete point labels obtained in S1 to generate three weak labels, including: 1) generate Voronoi labels; 2) use the K-means clustering algorithm to generate cluster labels; 3) use superpixel technology to generate superpixel labels; by combining the Voronoi diagram, K-means cluster labels and superpixel labels, the boundary and shape information of the cell nucleus are obtained, and the three generated weak labels are used to train the improved U-Net architecture model to enhance the feature extraction capability. The model finally outputs the probability map of the cell nucleus and the background, and the cross entropy loss function is used to calculate the loss. The information of the three different weak labels is combined to guide the model optimization process;

[0011] S3. Use confidence learning method to denoise the K-means clustering labels and superpixel labels to improve the accuracy of the labels; then propose an edge optimization strategy based on pixel affinity to generate edge optimized labels to improve the accuracy of cell nucleus edge segmentation; combine the edge optimized labels with the denoised labels to provide refined pseudo labels, further optimize model training, and improve the accuracy of the final segmentation results.

[0012] Preferably, the extended Gaussian filtering algorithm is as shown in formula (1):

[0013]

[0014] Where D is the distance from pixel i to the nearest marker in the image, r1 is the average radius of the nucleus, which is calculated from the training set data, and r2 is a parameter used to control the range of the background area of ​​the cell nucleus, which is set to r2=2r1. In M, 0 represents background pixels and -1 represents pixels that were ignored during the initial training process. It is reasonable to assume that r1<D i The pixels in the area <r2 belong to the background because most of the cell nuclei are surrounded by background pixels.

[0015] Preferably, the initial training stage uses a mean square error loss function (MSE) to train the model, and the backbone network of the model is to replace the encoder part of U-net with the convolution layer of ResNet-34.

[0016] Preferably, in the course learning strategy, the sample training difficulty is defined as formula (3):

[0017]

[0018] A is the average distance from the connected component of the i-th pixel to the k closest existing samples. i is the area of ​​the connected component of the ith pixel, S i is the average score predicted by the detection model of all pixels in the connected component where the i-th pixel is located.

[0019] Preferably, the self-training method includes performing refinement processing through background propagation, and the self-training process specifically includes the following steps:

[0020] In each round of self-training, we first obtain a probability map from the training model of the previous round; then we combine the probability map with the original labeled points to generate a new training mask; during the background map generation process, if the predicted probability of a pixel is p i <0.1 or p i >0.7, then it is selected as a background pixel; where p i is the probability of being a cell nucleus; finally, the updated mask is generated by adding the new background information to the original expanded mask As formula (2)

[0021]

[0022] Preferably, the sorting and selection of the training samples specifically includes the following steps:

[0023] First, the probability map is binarized to generate a point label map with a fixed threshold (set to 0.65), then the connected components of the binary map are calculated and these values ​​are mapped to the normalization function of [0,1]. The training difficulty of the model after 30 epochs is calculated, and all samples with non-overlapping labels are sorted from low to high according to the training difficulty. The top samples will be added to the true label and supervised for subsequent training. Specifically, the number of added samples is controlled by formula (4):

[0024]

[0025] Among them, n det and n gt is the number of detected valid samples and the number of pseudo-label samples.

[0026] Preferably, the backbone network of the improved U-Net network model is to replace the encoder part of U-net with the convolutional layer of ResNet-34.

[0027] Preferably, the pixel affinity-based boundary optimization strategy steps are as follows:

[0028] 1) Binarize the probability map of the previous stage, set the foreground threshold and background threshold, and the pixels with a predicted score higher than the threshold are classified as foreground, the pixels with a predicted score lower than the threshold are classified as background, and the pixels between the two are considered uncertain and will be ignored when generating pixel affinity;

[0029] 2) Based on the rough annotation, a pixel affinity map is constructed. For each pixel in the determined area (foreground and background), the pixels around it are sampled and the affinity between them is determined according to the distance threshold. If two pixels belong to the same instance, the affinity is set to 1; otherwise, it is set to 0. If one of the pixels is uncertain, the affinity is set to -1 and will be ignored during training. Formally, the pixel affinity map is defined as formula (5):

[0030]

[0031] Among them C i and C j Represent the categories of the i-th and j-th pixels respectively. In addition, the affinity set A of all pixel pairs is divided into four subsets, namely They represent pixels in the same cell nucleus, pixels in different cell nuclei, pixels in the background, and pixel pairs where one pixel is in the foreground and the other is in the background.

[0032] Preferably, the method for constructing the pixel affinity map is as follows: when two pixels have a positive affinity, it means that the connection path between them will not cross the boundary; otherwise, the path will cross the boundary; therefore, the affinity map can be obtained from the boundary prediction map, as shown in formula (6):

[0033]

[0034] In the above equation, is the boundary prediction map, and k∈Π ij is the set of pixels on the path between the i-th and j-th pixels. According to the above assumption, if the path between the i-th and j-th pixels crosses the boundary, then the path k∈Π ij The maximum value of the boundary prediction map on should be close to 1, while the affinity of the pixel pair will be close to 0. Using the cross entropy L B To train the boundary prediction network, L B The definition is as follows:

[0035]

[0036] Among them, the loss is normalized by the affinity set to deal with the problem of class imbalance;

[0037] By calculating the cross entropy loss between four different weak labels (after optimization) and the model prediction results, the final fine segmentation loss is obtained, as shown in formula (8):

[0038] L total =L c ' luster +L s ' uper +L voronoi +βL B (8)

[0039] Among them, β is the weight value of the boundary prediction loss.

[0040] Compared with the prior art, the beneficial effects of the present invention are: reducing the requirements for the model, being able to utilize limited weakly labeled data, and maintaining high segmentation accuracy, while also improving the efficiency of segmentation, which is beneficial to the processing of the edge and shape information of the cell nucleus. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 1 The overall architecture of a cell nucleus segmentation model based on a weakly supervised and semi-supervised cell nucleus segmentation method according to an embodiment of the present invention;

[0042] Figure 2 This is a structural diagram of a training backbone network model of a cell nucleus segmentation method based on weak supervision and semi-supervision according to an embodiment of the present invention;

[0043] Figure 3 A weak label schematic diagram of a cell nucleus segmentation method based on weak supervision and semi-supervision;

[0044] Figure 4 A weak label optimization effect diagram of a cell nucleus segmentation method based on weak supervision and semi-supervision after confidence learning;

[0045] Figure 5 Process diagram of edge optimization based on weakly supervised and semi-supervised methods for cell nucleus segmentation. DETAILED DESCRIPTION

[0046] The technical solutions in the embodiments of the present invention will be described clearly and completely below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments.

[0047] Reference Figure 1-Figure 5 , a cell nucleus segmentation method based on weak supervision and semi-supervision, comprising the following steps:

[0048] S1. Select any pathological image and some of its point labels in the breast cancer pathological image library as the image to be detected, and use the extended Gaussian filtering algorithm to perform initial training on the marked points; then use the background propagation method for self-training, and obtain more accurate cell nucleus detection results through continuous iterative updates, and use the course learning strategy to first train the model with simple samples, and then gradually increase the difficulty of the samples as the model improves to enhance the learning ability of the model;

[0049] S2. Use the complete point labels obtained in S1 to generate three weak labels, including: 1) generate Voronoi labels; 2) use the K-means clustering algorithm to generate cluster labels; 3) use superpixel technology to generate superpixel labels; by combining the Voronoi diagram, K-means cluster labels and superpixel labels, the boundary and shape information of the cell nucleus are obtained, and the three generated weak labels are used to train the improved U-Net architecture model to enhance the feature extraction capability. The model finally outputs the probability map of the cell nucleus and the background, and the cross entropy loss function is used to calculate the loss. The information of the three different weak labels is combined to guide the model optimization process;

[0050] S3. Use confidence learning method to denoise K-means clustering labels and superpixel labels to improve the accuracy of labels. Then propose edge optimization strategy based on pixel affinity to generate edge optimized labels to improve the accuracy of cell nucleus edge segmentation. Combine edge optimized labels with denoised labels to provide refined pseudo labels, further optimize model training, and improve the accuracy of final segmentation results.

[0051] The extended Gaussian filtering algorithm is shown in formula (1):

[0052]

[0053] Where D is the distance from pixel i to the nearest marker in the image, r1 is the average radius of the nucleus, which is calculated from the training set data, and r2 is a parameter used to control the range of the background area of ​​the cell nucleus, which is set to r2=2r1. In M, 0 represents background pixels and -1 represents pixels that were ignored during the initial training process. It is reasonable to assume that r1<D i The pixels in the area <r2 belong to the background, because most of the cell nuclei are surrounded by background pixels;

[0054] In the initial training phase, the mean square error loss function (MSE) was used to train the model. The backbone network of the model was to replace the encoder part of U-net with the convolutional layer of ResNet-34.

[0055] In the course learning strategy, the sample training difficulty is defined as follows:

[0056]

[0057] A is the average distance from the connected component of the i-th pixel to the k closest existing samples. i is the area of ​​the connected component of the ith pixel, S i is the average score predicted by the detection models for pixels in the connected components of the ith pixel.

[0058] 5. A cell nucleus segmentation method based on weak supervision and semi-supervision according to claim 1, characterized in that the self-training method includes refinement processing through background propagation, and the self-training process specifically includes the following steps:

[0059] In each round of self-training, we first obtain a probability map from the training model of the previous round; then we combine the probability map with the original labeled points to generate a new training mask; during the background map generation process, if the predicted probability of a pixel is p i <0.1 or p i >0.7, then it is selected as a background pixel; where p i is the probability of being a cell nucleus; finally, the updated mask is generated by adding the new background information to the original expanded mask As formula (2)

[0060]

[0061] The sorting and selection of training samples specifically includes the following steps:

[0062] First, the probability map is binarized to generate a point label map with a fixed threshold (set to 0.65), then the connected components of the binary map are calculated and these values ​​are mapped to the normalization function of [0,1]. The training difficulty of the model after 30 epochs is calculated, and the samples with non-overlapping labels are sorted from low to high according to the training difficulty. The top samples will be added to the true label and supervised for subsequent training. Specifically, the number of added samples is controlled by equation (4):

[0063]

[0064] Among them, n det and n gt is the number of detected valid samples and pseudo-label samples;

[0065] The backbone network of the improved U-Net network model replaces the encoder part of U-net with the convolutional layer of ResNet-34;

[0066] The steps of the boundary optimization strategy based on pixel affinity are as follows:

[0067] 1) Binarize the probability map of the previous stage, set the foreground threshold and background threshold, and the pixels with a predicted score higher than the threshold are classified as foreground, the pixels with a predicted score lower than the threshold are classified as background, and the pixels between the two are considered uncertain and will be ignored when generating pixel affinity;

[0068] 2) Based on the rough annotation, a pixel affinity map is constructed. For each pixel in the determined area (foreground and background), the pixels around it are sampled and the affinity between them is determined according to the distance threshold. If two pixels belong to the same instance, the affinity is set to 1; otherwise, it is set to 0. If one of the pixels is uncertain, the affinity is set to -1 and will be ignored during training. Formally, the pixel affinity map is defined as formula (5):

[0069]

[0070] Among them C i and C j Represent the categories of the i-th and j-th pixels respectively. In addition, the affinity set A of pixel pairs is divided into four subsets, namely They represent pixels in the same cell nucleus, pixels in different cell nuclei, pixels in the background, and pixel pairs where one pixel is in the foreground and the other is in the background;

[0071] The method to construct the pixel affinity graph is as follows:

[0072] When two pixels have a positive affinity, it means that the connection path between them will not cross the boundary; otherwise, the path will cross the boundary; therefore, the affinity map can be obtained from the boundary prediction map, as shown in formula (6)

[0073]

[0074] In the above equation, is the boundary prediction map, and k∈Π ij is the set of pixels on the path between the i-th and j-th pixels. According to the previous assumption, if the path between the i-th and j-th pixels crosses the boundary, then the path k∈Π ij The maximum value of the boundary prediction map on should be close to 1, while the affinity of the pixel pair will be close to 0. Using the cross entropy L B To train the boundary prediction network, L B The definition is as follows:

[0075]

[0076] Among them, the loss is normalized by the affinity set to deal with the problem of class imbalance;

[0077] By calculating the cross entropy loss between four different weak labels (after optimization) and the model prediction results, the final fine segmentation loss is obtained, as shown in formula (8):

[0078] L total =L c ' luster +L s ' uper +L voronoi +βL B (8)

[0079] Among them, β is the weight value of the boundary prediction loss.

[0080] In the present invention, firstly, pathological images and some point labels in any breast cancer pathological image library are selected as images to be detected, and the extended Gaussian filtering algorithm is used to perform initial training on the marked points; then, the background propagation method is used for self-training, and more accurate cell nucleus detection results are obtained through continuous iterative updates, and the course learning strategy is used to first train the model with simple samples, and then the difficulty of the samples is gradually increased as the model is improved to enhance the learning ability of the model;

[0081] The obtained complete point labels are then used to generate three weak labels: first, Voronoi labels are generated; second, cluster labels are generated using the K-means clustering algorithm; third, superpixel labels are generated using superpixel technology;

[0082] Then, by combining the Voronoi diagram, K-means clustering labels and superpixel labels, the boundary and shape information of the cell nucleus is obtained. The three generated weak labels are used to train the improved U-Net architecture model to enhance the feature extraction capability. The model finally outputs the probability map of the cell nucleus and the background. The cross entropy loss function is used to calculate the loss. The information of the three different weak labels is combined to guide the model optimization process.

[0083] Next, the confidence learning method is used to denoise the K-means clustering labels and superpixel labels to improve the accuracy of the labels. Then, an edge optimization strategy based on pixel affinity is proposed to generate edge optimized labels to improve the accuracy of cell nucleus edge segmentation. The edge optimized labels are combined with the denoised labels to provide refined pseudo labels, further optimize model training, and improve the accuracy of the final segmentation results.

[0084] The above description is only a preferred specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can make equivalent replacements or changes according to the technical scheme and inventive concept of the present invention within the technical scope disclosed by the present invention, which should be covered by the protection scope of the present invention.

Claims

1. A cell nucleus segmentation method based on weak supervision and semi-supervision, characterized in that: The following steps are involved: S1. Select any pathological image and some of its point labels in the breast cancer pathological image library as the image to be detected, and use the extended Gaussian filtering algorithm to perform initial training on the marked points; Then, the background propagation method is used for self-training. Through continuous iterative updates, more accurate cell nucleus detection results are obtained. The course learning strategy is used to first train the model with simple samples, and then the difficulty of the samples is gradually increased as the model improves to enhance the model's learning ability. S2. Use the complete point labels obtained in S1 to generate three weak labels, including: 1) generate Voronoi labels; 2) use the K-means clustering algorithm to generate cluster labels; 3) use superpixel technology to generate superpixel labels; by combining the Voronoi diagram, K-means cluster labels and superpixel labels, the boundary and shape information of the cell nucleus are obtained, and the three generated weak labels are used to train the improved U-Net architecture model to enhance the feature extraction capability. The model finally outputs the probability map of the cell nucleus and the background, and the cross entropy loss function is used to calculate the loss. The information of the three different weak labels is combined to guide the model optimization process; S3. Use confidence learning method to denoise the K-means clustering labels and superpixel labels to improve the accuracy of the labels; then propose an edge optimization strategy based on pixel affinity to generate edge optimized labels to improve the accuracy of cell nucleus edge segmentation; combine the edge optimized labels with the denoised labels to provide refined pseudo labels, further optimize model training, and improve the accuracy of the final segmentation results.

2. A cell nucleus segmentation method based on weak supervision and semi-supervision according to claim 1, characterized in that: The extended Gaussian filtering algorithm is shown in formula (1): Where D is the distance from pixel i to the nearest marker in the image, r1 is the average radius of the nucleus, which is calculated from the training set data, and r2 is a parameter used to control the range of the background area of ​​the cell nucleus, which is set to In M, 0 represents background pixels and -1 represents pixels that were ignored during the initial training process. It is reasonable to assume that r1 < D i The pixels in the area <r2 belong to the background because most of the cell nuclei are surrounded by background pixels.

3. The method for cell nucleus segmentation based on weak supervision and semi-supervision according to claim 1, characterized in that: The initial training phase uses the mean square error loss function (MSE) to train the model, and the backbone network of the model is to replace the encoder part of U-net with the convolution layer of ResNet-34.

4. The method for cell nucleus segmentation based on weak supervision and semi-supervision according to claim 1, characterized in that: In the course learning strategy, the sample training difficulty is defined as formula (3): A represents the average distance from the connected component of the i-th pixel to the k closest existing samples. i is the area of ​​the connected component of the ith pixel, S i is the average score predicted by the detection model of all pixels in the connected component where the i-th pixel is located.

5. The method for cell nucleus segmentation based on weak supervision and semi-supervision according to claim 1, characterized in that: The self-training method includes performing refinement processing through background propagation, and the self-training process specifically includes the following steps: In each round of self-training, we first obtain a probability map from the training model of the previous round; then we combine the probability map with the original labeled points to generate a new training mask; during the background map generation process, if the predicted probability of a pixel is p i <0.1 or p i >0.7, then it is selected as a background pixel; where p i is the probability of being a cell nucleus; finally, the updated mask is generated by adding the new background information to the original expanded mask As formula (2) 6. The method for cell nucleus segmentation based on weak supervision and semi-supervision according to claim 1, characterized in that: The sorting and selection of the training samples specifically include the following steps: First, the probability map is binarized to generate a point label map with a fixed threshold (set to 0.65), then the connected components of the binary map are calculated and these values ​​are mapped to the normalization function of [0,1]. The training difficulty of the model after 30 epochs is calculated, and all samples with non-overlapping labels are sorted from low to high according to the training difficulty. The top samples will be added to the true label and supervised for subsequent training. Specifically, the number of added samples is controlled by formula (4): Among them, n det and n gt is the number of detected valid samples and the number of pseudo-label samples.

7. The method for cell nucleus segmentation based on weak supervision and semi-supervision according to claim 1, characterized in that: The backbone network of the improved U-Net network model is to replace the encoder part of U-net with the convolutional layer of ResNet-34.

8. The method for cell nucleus segmentation based on weak supervision and semi-supervision according to claim 1, characterized in that: The steps of the boundary optimization strategy based on pixel affinity are as follows: 1) Binarize the probability map of the previous stage, set the foreground threshold and background threshold, and the pixels with a predicted score higher than the threshold are classified as foreground, the pixels with a predicted score lower than the threshold are classified as background, and the pixels between the two are considered uncertain and will be ignored when generating pixel affinity; 2) Based on the rough annotation, a pixel affinity map is constructed. For each pixel in the determined area (foreground and background), the pixels around it are sampled and the affinity between them is determined according to the distance threshold. If two pixels belong to the same instance, the affinity is set to 1; otherwise, it is set to 0. If one of the pixels is uncertain, the affinity is set to -1 and will be ignored during training. Formally, the pixel affinity map is defined as formula (5): Among them C i and C j Represent the categories of the i-th and j-th pixels respectively. In addition, the affinity set A of all pixel pairs is divided into four subsets, namely They represent pixels in the same cell nucleus, pixels in different cell nuclei, pixels in the background, and pixel pairs where one pixel is in the foreground and the other is in the background.

9. The method for cell nucleus segmentation based on weak supervision and semi-supervision according to claim 1, characterized in that: The method for constructing a pixel affinity graph is as follows: When two pixels have a positive affinity, it means that the connection path between them will not cross the boundary; otherwise, the path will cross the boundary; therefore, the affinity map can be obtained from the boundary prediction map, as shown in formula (6) In the above equation, is the boundary prediction map, and k∈Π ij is the set of pixels on the path between the i-th and j-th pixels. According to the above assumption, if the path between the i-th and j-th pixels crosses the boundary, then the path k∈Π ij The maximum value of the boundary prediction map on should be close to 1, while the affinity of the pixel pair will be close to 0. Using the cross entropy L B To train the boundary prediction network, L B The definition is as follows: Among them, the loss is normalized by the affinity set to deal with the problem of class imbalance; By calculating the cross entropy loss between four different weak labels (after optimization) and the model prediction results, the final fine segmentation loss is obtained, as shown in formula (8): L total =L′ cluster +L′ super +L voronoi +βL B (8) Among them, β is the weight value of the boundary prediction loss.

Images