Low-cost cell segmentation method, system and device based on active learning

Abstract

The invention discloses a low-cost cell segmentation method, system and device based on active learning. The method comprises the following steps: selecting an initial cell image for preprocessing and bounding box labeling; forming a training set and constructing a target detection model for training so as to predict a bounding box of pictures of the verification set; taking the bounding box as a prompt for segmenting a cut model (SAM), and outputting a corresponding cell mask; adopting a Monte Carlo discarding method to predict the uncertainty of the unlabeled pictures, and selecting the pictures with higher uncertainty to be labeled by experts; adding the marked image data into the training set of the last round for updating, and repeating the training and sample selection of the model; and outputting a model weight and an index record after the required performance is achieved. According to the method, the data annotation cost of a cell segmentation algorithm is greatly reduced on the premise of ensuring the model performance.

Description

Technical Field

[0001] The present invention belongs to the field of Internet, and in particular relates to a low-cost cell segmentation method, system and device based on active learning. Background Art

[0002] Cell instance segmentation is a key challenge in medical image analysis tasks. It can achieve quantitative and qualitative analysis of cells and assist in diagnosis. Cell segmentation algorithms based on deep learning can automatically extract the best features from cell images and achieve high accuracy. However, many deep learning methods currently used for cell segmentation are fully supervised learning methods based on mask annotation, and the accuracy of the model depends on the quality and quantity of annotation data, which places high demands on data. However, the annotation of cell images is limited by the complex cell morphology, unclear boundaries and noise interference in microscopic images, and requires a lot of professional knowledge. Therefore, the cost of annotating cells in microscopic images is very high. It is imperative to explore methods to reduce the cost of annotation.

[0003] The purpose of cell segmentation is to obtain the position and shape of cells, but the annotation process is very time-consuming whether it is based on masks or polygons. Studies have shown that annotating the bounding boxes of objects in COCO only takes 8.8% of the time (i.e., 7 seconds and 79.2 seconds) compared to annotating the masks of objects based on polygons. Therefore, deep learning methods that use coarse annotation as a supervision method can effectively reduce the cost of annotation. Active learning, as an interactive learning method in which people are in the loop, can make effective use of samples. It greatly reduces the redundancy of annotation work by selecting samples with the most information and handing them over to professionals for annotation. Summary of the invention

[0004] The purpose of the present invention is to effectively reduce the annotation cost of cell segmentation, and proposes a low-cost cell segmentation method, system and device based on active learning.

[0005] The object of the present invention is achieved through the following technical solution: A low-cost cell segmentation method based on active learning, comprising the following steps:

[0006] Step 1: Obtain cell images for preprocessing and divide them into a training set and a validation set;

[0007] Step 2: The validation set images and a fixed number of cell images randomly selected from the training set are handed over to experts to annotate the cell bounding boxes;

[0008] Step 3, using the labeled cell image data in the training set as the training set, training with the target detection network model to generate model weights;

[0009] Step 4: Use the model in step 3 to predict the bounding box of the validation set cell image, and use it as a hint for the segmentation model SAM to extract global features of the cell image and output the predicted mask of the cell instance.

[0010] Step 5: Calculate and record segmentation metrics based on the predicted mask of the validation set;

[0011] Step 6: The Monte Carlo dropout method is used to predict the uncertainty of the unlabeled images in the training set, and several cell images with the highest uncertainty are selected and handed over to experts for bounding box labeling again;

[0012] Step 7: Add the newly labeled cell image data in step 6 to the training set in step 3 as a new training data set, and repeat steps 3 to 6 until the segmentation index of the validation set meets the requirements.

[0013] Preferably, in step 1, the image preprocessing operation includes adjusting the image size, denoising and enhancement, and Gaussian filtering.

[0014] Preferably, in step 2, the expert marks the boundary of each cell in the image with a tight box, and outputs the starting point and the width and height B (x, y, w, h) of the box.

[0015] Preferably, in step three, the target detection network YOLOv8 with excellent performance and speed is used to predict the bounding box.

[0016] Preferably, in step 5, the segmentation index used is Dice Coefficient (DSC) to objectively evaluate the segmentation performance.

[0017] Preferably, in step six, the Monte Carlo drop method requires different outputs, so a DropBlock module is added to the target detection network to obtain multiple different outputs.

[0018] Preferably, in step six, the analysis of the uncertainty c includes calculations of the bounding box and the instance mask, as shown in formulas (1)-(4):

[0019]

[0020] c=c i ·c b ·c m (4)

[0021] Where m is the number of categories, t is the number of instance sets, and P(k i ) is the confidence score for each category, and are the average bounding box and average mask of all instance sets, N(s i ) and M(sj ) are the predicted bounding box and mask of each instance, c i ,c b ,c m Represent the uncertainty scores of the category, bounding box, and cell instance mask respectively.

[0022] Furthermore, the present invention also provides a low-cost cell segmentation system based on active learning, including an image annotation module, a mask prediction module, a sample selection module, and an indicator evaluation module;

[0023] The image annotation module provides bounding box data annotated by experts;

[0024] The mask prediction module is responsible for outputting the corresponding mask instance of the bounding box annotation;

[0025] The sample selection module selects the most difficult-to-predict images and sends them to experts for annotation and adds them to the training set;

[0026] The index evaluation module provides the segmentation performance of the training model;

[0027] In the low-cost cell segmentation system based on active learning, the input picture is annotated with a bounding box by the image annotation module, and the cell mask indicated by the output box is handed over to the mask prediction module. The indicator evaluation module tests the performance of the trained model and gives an evaluation. If the performance does not meet the requirements, the sample selection module selects the most difficult samples to be annotated and continues to be annotated by experts. If the performance meets the requirements, the model weight is output.

[0028] The beneficial effects of the present invention are:

[0029] 1. Adopting the framework of active learning and using human-computer interaction to complete the segmentation task, the efficiency of sample labeling is improved;

[0030] 2. The participation of active learning methods greatly reduces the amount of labeled samples required;

[0031] 3. The box2mask box-supervised deep learning algorithm formed by combining CNN and Transformer can reduce the number of labeled samples required in the form of masks;

[0032] 4. Integrate the box-supervised deep learning algorithm into the active learning framework, and only need a small number of box-annotated samples to train a model with performance comparable to the masked full-supervision algorithm;

[0033] 5. On the one hand, active learning reduces the amount of labeled samples used. On the other hand, the box supervision algorithm only needs to label the pictures with bounding boxes, which will greatly reduce the labeling time of each picture. The combination of these two aspects greatly reduces the overall labeling cost.

[0034] 6. In the cell segmentation task, it will greatly simplify the process and improve work efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Figure 1 is a schematic diagram of a low-cost cell segmentation method based on active learning;

[0036] Figure 2 It is a module structure diagram combining the YOLOv8 detection model and the SAM segmentation model;

[0037] Figure 3 Flowchart of a low-cost cell segmentation system based on active learning.

[0038] Figure 4 It is a structural diagram of a low-cost cell segmentation device based on active learning provided by the present invention. DETAILED DESCRIPTION

[0039] The present invention is described in detail below with reference to the accompanying drawings and specific embodiments.

[0040] The present invention provides a low-cost cell segmentation method based on active learning, such as Figure 1 As shown in the figure, active learning methods are used to label cell images at low cost. On the other hand, Y-SAM is a box2mask algorithm that combines YOLOv8 and SAM, which further reduces the labeling time.

[0041] Example 1: Cell segmentation using the low-cost cell segmentation method based on active learning of the present invention

[0042] The frame-supervised instance segmentation model Y-SAM in this embodiment consists of a YOLOv8 detection network and SAM (image encoder, hint encoder, mask decoder), as shown in Figure 2 As shown, the DropBlock module is added to the SPPF module and the first upsampling module of the YOLOv8 network to prevent overfitting of the neural network and calculation of the Monte Carlo dropout method. The experimental steps are as follows:

[0043] (1) Perform preprocessing operations such as denoising and enhancement on the cell image data, including adjusting the image size, denoising and enhancement, and Gaussian filtering; randomly select N images from the divided training set and ask experts to perform tight bounding box annotation for each cell, and at the same time perform bounding box annotation on the validation set images, and output the starting point and the width and height B (x, y, w, h) of the box;

[0044] (2) The labeled images and data are used as training sets, and the YOLOv8 network is used for 150 epochs training. Then, the bounding boxes of the images in the validation set are predicted.

[0045] (3) The bounding box predicted in step (2) is used as a hint for SAM, and the validation set image is input for feature extraction and encoding and decoding, and the cell mask is output;

[0046] (4) The predicted cell mask is evaluated using the DSC indicator to obtain the segmentation performance of this round of the model;

[0047] (5) The Monte Carlo dropout method is used to make T predictions on the unlabeled images in the data pool and take the average value as the uncertainty, from which N images with the highest uncertainty are selected; the Monte Carlo dropout method requires different outputs, so the DropBlock module is added to the target detection network to obtain multiple different outputs.

[0048] The analysis of uncertainty c includes the calculation of bounding box and instance mask, as shown in formulas (1)-(4):

[0049]

[0050] c=c i ·c b ·c m (4)

[0051] Where m is the number of categories, t is the number of instance sets, and P(k i ) is the confidence score for each category, and are the average bounding box and average mask of all instance sets, B(s i ) and M(s j ) are the predicted bounding box and mask of each instance, c i ,c b ,c m Represent the uncertainty scores of the category, bounding box, and cell instance mask respectively.

[0052] (6) The N images selected in step (5) are handed over to experts for bounding box annotation and added to the training set of the previous round as a new training set. Steps (2) to (5) are repeated until the DSC index meets the requirements.

[0053] The pseudo code of the entire algorithm is as follows:

[0054] Input: data pool P, sampling volume sz, discard probability dp, Monte Carlo method iteration number MC_i

[0055] Output: indicator list DSC-list

[0056] 1: Initial data ← random sampling (P, sz)

[0057] 2: Labeled initial data L0←labeled(initial data)

[0058] 3: Model weight F←train YOLOv8(L0)

[0059] 4: Bounding box ← F.predict (validation set)

[0060] 5: Mask ← SAM (bounding box)

[0061] 6:dsc←calculate DSC(mask)

[0062] 7:DSC-list.insert(dsc)

[0063] 8: P←P-initial data

[0064] 9:for i←1to loop do

[0065] 10:if sampling strategy = 'random sampling' then

[0066] 11: Sampled image ← random sampling method (P, sz)

[0067] 12:else if sampling strategy = 'uncertain sampling' then

[0068] 13: Sampled image ← Uncertain sampling method (P, sz, dp, MC_i)

[0069] 14:L i ← Annotation (sample image)

[0070] 15:F←Train YOLOv8(L0+L i ,F)

[0071] 16: Bounding box ← F.predict (validation set)

[0072] 17: Mask ← SAM (bounding box)

[0073] 18:dsc←calculate DSC(mask)

[0074] 19:DSC-list.insert(dsc)

[0075] 20: P←P-sampled image

[0076] 21:end

[0077] 22:return DSC-list

[0078] Example 2: Analysis of labeling cost using low-cost cell segmentation method based on active learning

[0079] This embodiment adopts the same bounding box supervised instance segmentation algorithm as the first embodiment. The flowchart of the whole method is as follows: Figure 3 As shown in Figure 5, the required value of the segmentation index DSC is set to 99% of the performance of the mask supervision algorithm Mask RCNN, and the calculation formula of the annotation cost is as described in Formula (5).

[0080]

[0081] Among them, N b is the number of images annotated with bounding boxes, N m is the number of images annotated with masks, and 8.8% is the average time ratio of annotating the bounding box of a natural object to annotating the polygon mask.

[0082] The results of preliminary tests on three public medical datasets Pannuke, 2018DSB, and MoNuSeg are shown in Table 1. This shows that the method of the present invention will greatly reduce the annotation cost required for the cell segmentation task.

[0083] Table 1. The annotation cost of the proposed method tested on three public medical datasets.

[0084]

[0085] Corresponding to the aforementioned embodiment of a low-cost cell segmentation method based on active learning, the present invention also provides a low-cost cell segmentation system based on active learning, including an image annotation module, a mask prediction module, a sample selection module, and an indicator evaluation module; for the specific process of implementing each module, please refer to the specific implementation steps of the aforementioned embodiment of a low-cost cell segmentation method based on active learning.

[0086] The image annotation module is used to obtain cell images to build a data pool, randomly select a fixed number of cell images for preprocessing, and then hand over the cell images to experts for cell boundary box annotation;

[0087] The mask prediction module is used to use the labeled cell images as training data sets and use the target detection network model for training; the trained model is used to predict the bounding box of the validation set cell images, and the global feature extraction of the cell images is performed as a prompt for the segmentation model SAM, and the predicted mask of the cell instance is output;

[0088] The indicator evaluation module is used to calculate and record the segmentation indicator according to the prediction mask of the validation set, record the model performance, and output the model weight if the performance meets the requirements;

[0089] The sample selection module is used to predict the uncertainty of unlabeled images in the training set using the Monte Carlo discard method, and select several cell images with the highest uncertainty, which are handed over to experts for bounding box annotation; the newly labeled cell image data and the training data set in the mask prediction module are used together as a new training data set.

[0090] Corresponding to the aforementioned embodiment of a low-cost cell segmentation method based on active learning, the present invention also provides an embodiment of a low-cost cell segmentation device based on active learning.

[0091] See also Figure 4 A low-cost cell segmentation device based on active learning provided in an embodiment of the present invention includes a memory and one or more processors, wherein the memory stores executable code, and when the processor executes the executable code, it is used to implement a low-cost cell segmentation method based on active learning in the above embodiment.

[0092] An embodiment of a low-cost cell segmentation device based on active learning provided by the present invention can be applied to any device with data processing capabilities, and the device with data processing capabilities can be a device or apparatus such as a computer. The device embodiment can be implemented through software, or through hardware or a combination of software and hardware. Taking software implementation as an example, as a device in a logical sense, it is formed by the processor of any device with data processing capabilities in which it is located reading the corresponding computer program instructions in the non-volatile memory into the memory for execution. From the hardware level, if Figure 4 As shown in FIG. 1 , a hardware structure diagram of a low-cost cell segmentation device based on active learning provided by the present invention is provided in any device with data processing capability, except Figure 4 In addition to the processor, memory, network interface, and non-volatile memory shown, any device with data processing capabilities in which the apparatus in the embodiments is located may also include other hardware, generally based on the actual functions of the device with data processing capabilities, which will not be described in detail.

[0093] The implementation process of the functions and effects of each unit in the above-mentioned device is specifically described in the implementation process of the corresponding steps in the above-mentioned method, and will not be repeated here.

[0094] For the device embodiment, since it basically corresponds to the method embodiment, the relevant parts can refer to the partial description of the method embodiment. The device embodiment described above is only schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of the present invention. Ordinary technicians in this field can understand and implement it without paying creative work.

[0095] An embodiment of the present invention further provides a computer-readable storage medium having a program stored thereon. When the program is executed by a processor, a low-cost cell segmentation method based on active learning in the above embodiment is implemented.

[0096] The computer-readable storage medium may be an internal storage unit of any device with data processing capability described in any of the aforementioned embodiments, such as a hard disk or a memory. The computer-readable storage medium may also be an external storage device of any device with data processing capability, such as a plug-in hard disk, a smart media card (SMC), an SD card, a flash card, etc. equipped on the device. Furthermore, the computer-readable storage medium may also include both an internal storage unit and an external storage device of any device with data processing capability. The computer-readable storage medium is used to store the computer program and other programs and data required by any device with data processing capability, and may also be used to temporarily store data that has been output or is to be output.

[0097] The present invention also provides a computer program product, comprising a computer program, and when the computer program is executed by a processor, the low-cost cell segmentation method based on active learning is implemented.

[0098] The above embodiments are used to illustrate the present invention rather than to limit the present invention. Any modification and change made to the present invention within the spirit of the present invention and the protection scope of the claims shall fall within the protection scope of the present invention.

Claims

1. A low-cost cell segmentation method based on active learning, characterized in that: The method comprises the following steps: Step 1: Obtain cell images for preprocessing and divide them into a training set and a validation set; Step 2: The validation set images and a fixed number of cell images randomly selected from the training set are handed over to experts to annotate the cell bounding boxes; Step 3, using the labeled cell image data in the training set as the training set, training with the target detection network model to generate model weights; Step 4: Use the model in step 3 to predict the bounding box of the validation set cell image, and use it as a hint for the segmentation model SAM to extract global features of the cell image and output the predicted mask of the cell instance. Step 5: Calculate and record segmentation metrics based on the predicted mask of the validation set; Step 6: The Monte Carlo dropout method is used to predict the uncertainty of the unlabeled images in the training set, and several cell images with the highest uncertainty are selected and handed over to experts for bounding box labeling again; Step 7: Add the newly labeled cell image data in step 6 to the training set in step 3 as a new training data set, and repeat steps 3 to 6 until the segmentation index of the validation set meets the requirements.

2. The low-cost cell segmentation method based on active learning according to claim 1, characterized in that: In step 1, the preprocessing operations of the cell image include image resizing, denoising enhancement and Gaussian filtering.

3. The low-cost cell segmentation method based on active learning according to claim 1, characterized in that: In step 2, the expert annotates the boundaries of each cell in the cell image with a tight box and outputs the starting point and the width and height B(x, y, w, h) of the box.

4. The low-cost cell segmentation method based on active learning according to claim 1, characterized in that: In step three, the target detection network used is the YOLOv8 detector.

5. The low-cost cell segmentation method based on active learning according to claim 1, characterized in that: In step 5, the segmentation index used is Dice Coefficient (DSC).

6. The low-cost cell segmentation method based on active learning according to claim 1, characterized in that: In step six, the Monte Carlo dropout method requires different outputs, so a DropBlock module is added to the target detection network to obtain multiple different outputs.

7. The low-cost cell segmentation method based on active learning according to claim 1, characterized in that: In step 6, the uncertainty analysis includes the calculation of bounding box and instance mask. The calculation formula of uncertainty c is as shown in equations (1)-(4): c=c i ·c b ·c m (4) Where m is the number of categories, t is the number of instance sets, and P(k i ) is the confidence score for each category, and are the average bounding box and average mask of all instance sets, B(s i ) and M(s j ) are the predicted bounding box and mask of each instance, c i ,c b ,c m Represent the uncertainty scores of the category, bounding box, and cell instance mask respectively.

8. A low-cost cell segmentation system based on active learning, characterized in that: It includes image annotation module, mask prediction module, sample selection module and indicator evaluation module; The image annotation module is used to obtain cell images to build a data pool, randomly select a fixed number of cell images for preprocessing, and then hand over the cell images to experts for cell boundary box annotation; The mask prediction module is used to use the labeled cell images as training data sets and use the target detection network model for training; the trained model is used to predict the bounding box of the validation set cell images, and the global feature extraction of the cell images is performed as a prompt for the segmentation model SAM, and the predicted mask of the cell instance is output; The indicator evaluation module is used to calculate and record the segmentation indicator according to the prediction mask of the validation set, record the model performance, and output the model weight if the performance meets the requirements; The sample selection module is used to predict the uncertainty of unlabeled images in the training set using the Monte Carlo discard method, and select several cell images with the highest uncertainty, which are handed over to experts for bounding box annotation; the newly labeled cell image data and the training data set in the mask prediction module are used together as a new training data set.

9. A low-cost cell segmentation device based on active learning, comprising a memory and one or more processors, wherein the memory stores executable code, characterized in that: When the processor executes the executable code, a low-cost cell segmentation method based on active learning as described in any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium having a program stored thereon, characterized in that: When the program is executed by a processor, a low-cost cell segmentation method based on active learning as described in any one of claims 1 to 7 is implemented.

Images