A self-learning-based cell image semantic segmentation annotation method

Abstract

The application relates to a self-learning-based cell image semantic segmentation labeling method, which comprises the following steps: I. manual sampling step: (1) performing preliminary manual labeling on an original image, sampling 2-5 white blood cells, 5-10 red blood cells and 1-2 background areas; (2) when manually labeling, manually framing a rectangular area of a single cell, applying a Grabcut algorithm to segment a sampling image of the single cell, obtaining an energy formula for foreground and background segmentation of an image X in the rectangular area, and then applying a minimum cut algorithm to segment the foreground and the background; (3) after sampling a sufficient number of cells, framing a background area in the original image, analyzing the color of the background area, fitting a Gaussian distribution function to a histogram of R, G and B channels, and obtaining a background color range of the original image. The application can capture the differences in cell morphology, color, texture and size, realizes fast and accurate automatic segmentation, is simpler, more effective and has a wider application range than existing tools.

Description

TECHNICAL FIELD

[0001] The application belongs to the field of cell image processing, and particularly relates to a cell image semantic segmentation labeling method based on self-learning. BACKGROUND

[0002] Semantic segmentation is one of the representative tasks in the field of computer vision image processing, and the goal is to perform pixel-level segmentation on different objects appearing in the picture. In 2015, Ronneberge et al. first proposed a medical image semantic segmentation model UNet based on feature fusion in the paper U-Net: Convolutional Networks for Biomedical Image Segmentation, and the performance of the model was quickly proved in the following years and was widely used in the industry. However, training a mature semantic segmentation model certainly requires a large number of manually annotated samples, and manually annotating tens of thousands of medical images certainly requires a lot of manpower and material resources, and the annotation quality cannot be guaranteed. Therefore, reducing human participation and improving annotation quality is a problem to be solved. Existing semantic segmentation labeling tools usually train a very generalized semantic segmentation model on a public data set of the order of hundreds of thousands or even millions, and then label specific samples for the downstream task, and then manually adjust. However, such a generalized model cannot be used for medical image annotation work due to the difference between images. At the same time, public medical image data sets are very scarce and almost completely insufficient to train a generalized model for medical image annotation.

[0003] Currently, for the semantic segmentation annotation of cell images, Bamford proposed a solution based on viterbi algorithm, watershed algorithm and threshold algorithm in the article Empirical comparison of cell segmentation algorithms using an annotated dataset when sufficient samples cannot be collected for deep learning. Meanwhile, Jianhua et al. adopted an iterative Otsu algorithm based on saturation in the article A novel color image segmentation method and its application to white blood cell image analysis, and some other solutions are based on edge detection or use GrabCut for full image segmentation. However, on the one hand, the edges of cell images are not obvious, and sometimes there is not enough morphological difference between white blood cells and red blood cells, on the other hand, cell images are often disturbed by impurities, dyes and lighting conditions. This makes the segmentation algorithm based on color difference or edge detection have very unsatisfactory annotation effect. SUMMARY

[0004] The purpose of the present application is to overcome the above-mentioned deficiencies in the prior art, and to provide a cell image semantic segmentation annotation method based on self-learning, which can realize fast and accurate automatic segmentation, greatly reduce manual participation and improve annotation quality.

[0005] The technical scheme adopted by the present application to solve the above problems is to provide a cell image semantic segmentation annotation method based on self-learning, comprising the following steps:

[0006] I. Manual sampling step:

[0007] (1) Preliminary manual annotation is performed on the original image, and 2-5 white blood cells, 5-10 red blood cells and 1-2 background areas are sampled;

[0008] (2) When manually annotating, manually frame the rectangular area of a single cell, and apply the Grabcut algorithm to segment the sampling image of a single cell, obtain the energy formula of the foreground and background segmentation of the image X in the rectangular area, and then apply the minimum cut algorithm to segment the foreground and background;

[0009] (3) After sampling a sufficient number of cells, frame the background area in the original image, analyze the color of the background area, fit the Gaussian distribution function to the histogram of the R, G and B channels, and obtain the background color range of the original image, the formula is as follows:

[0010]

[0011] where a is the peak of the histogram, μ is the pixel value corresponding to the peak of the curve, and σ is the standard deviation obtained by statistics.

[0012] II. Background generation:

[0013] (4) Obtain the color distribution information N of the R, G, and B channels R (x), N G (x), N B After (x), when generating an image, take the average color of the three channels within one standard deviation to generate a background.

[0014] (5) On the generated image, add salt and pepper noise to the background, that is, take the pixel position and generate noise according to the Gaussian distribution of μ = 15, σ = 30, and then fuse it into the image.

[0015] (6) On the generated image, use a Gaussian function to generate a dot on the background to simulate the noise situation of the original image. Select the position μ and the impurity size s, and then generate a two-dimensional Gaussian kernel function:

[0016] Then overlay the simulated impurity image on the background of the generated image to generate the background.

[0017] III. Cell generation:

[0018] (7) Select a red blood cell sampled in step (2) and perform color transformation on the saturation, brightness, and contrast.

[0019] (8) Then perform affine transformation on the cell to stretch and rotate it.

[0020] (9) Use the method of step (8) to generate 300-500 red blood cells and 20-40 white blood cells on the generated image. When generating cells, also generate an image of the target semantic segmentation. When adding new cells, perform an inflation operation on the existing semantic segmentation image to select the generation position from the background pixels and avoid excessive overlap of cell positions.

[0021] IV. Lightweight neural network training:

[0022] (10) Repeat steps (4) to (9) to generate 10-30 images.

[0023] (11) Train the lightweight U-Net convolutional neural network with the images generated in step (10) to reduce the original number of channels (64-128-256-512-1024) to (4-8-16-32-64) and perform 20 rounds of iterative training. Take the segmentation result of each round as a candidate result.

[0024] (12) Take the segmentation results of 20 rounds, select the round with better results, and make manual fine adjustments to output the labeled segmentation map.

[0025] Preferably, the energy formula for segmenting the foreground and background of the image X within the rectangular region in step (2) is as follows:

[0026] E(X)=∑ v∈V g v (X v )+∑ (u,v)∈E h uv (X u ,X v )

[0027] Where g(·) calculates the probability that a pixel belongs to the foreground or background, and h(·) calculates the similarity between a pixel and its surrounding pixels.

[0028] Preferably, in step (8), the affine transformation of the cells is performed using cubic matrix multiplication, specifically as follows:

[0029]

[0030] Where θ is the rotation angle, λ x For the lateral stretching dimension, λ y The longitudinal stretching dimension.

[0031] Compared with existing technologies, this invention has the following advantages and effects: Addressing the differences between red blood cells, white blood cells, background, impurities, and dyes in captured single cell images, this invention innovatively uses a neural network-based self-learning method to simultaneously capture differences in morphology, color, texture, and size, achieving rapid and accurate automatic segmentation. Cell images can be accurately and quickly labeled without prior sample collection. Compared to existing tools, this is simpler, more effective, and has a wider range of applications. Attached Figure Description

[0032] Figure 1 This is a flowchart of a self-learning-based semantic segmentation and annotation method for cell images according to the present invention;

[0033] Figure 2 These are original images of embodiments of the present invention;

[0034] Figure 3 This is a sampled image of a single cell segmented by Grabcut in an embodiment of the present invention;

[0035] Figure 4 These are the cell and background sampling images that need to be collected in this embodiment of the invention;

[0036] Figure 5 This is an image generated by mimicking the original image in an embodiment of the present invention;

[0037] Figure 6 is a segmentation map of the image generated by the original image simulation of the embodiment of the present application;

[0038] Figure 7 is a segmentation annotation map of the original image by the self-training model of the embodiment of the present application;

[0039] Figure 8 is the final output segmentation annotation map of the embodiment of the present application. DETAILED DESCRIPTION

[0040] The present application will be further described in detail below with reference to the accompanying drawings and by embodiments, and the following embodiments are an explanation of the present application and the present application is not limited to the following embodiments.

[0041] EMBODIMENT

[0042] Reference Figure 1 , the embodiment of the present application provides a cell image semantic segmentation annotation method based on self-learning, comprising the following steps:

[0043] I. Manual sampling step:

[0044] (1) The original picture is preliminarily manually annotated, and 5 white blood cells, 8 red blood cells and 1 background area are sampled, as shown in Figure 2

[0045] (2) When sampling a single cell, a rectangular area is framed around the single cell, and a Grabcut algorithm is applied to segment the sampling map of the single cell, as shown in Figure 3 , the energy formula of the foreground and background segmentation of the image X in the rectangular area is obtained, and the minimum cut algorithm is applied to segment the foreground and background to make the segmentation energy minimum;

[0046] (3) As shown in Figure 4 , 5 white blood cells, 8 red blood cells and 1 background area are sampled, and then the background area is framed in the original image. The color analysis of the background area is obtained by fitting the Gaussian distribution function to the histogram of the R, G and B channels, and the background color range of the original image is obtained, and the formula is as follows:

[0047]

[0048] Where ɑ is the peak value of the histogram, μ is the pixel value corresponding to the curve peak, and σ is the standard deviation obtained by statistics.

[0049] II. Background generation:

[0050] (4) The color distribution information N R (x), N G (x), N​B (x) After step (x), the background color of the three channels is counted, and when generating an image, the average color of the three channels is taken within one standard deviation to generate the background;

[0051] (5) On the generated image, add salt and pepper noise to the background, that is, take the pixel position, generate noise according to the Gaussian distribution of μ = 15, σ = 30, and then fuse it into the image;

[0052] (6) On the generated image, use a Gaussian function to generate a dot on the background to simulate the noise condition of the original image, select the position μ and the impurity size s, and then generate a two-dimensional Gaussian kernel function:

[0053] Then, the simulated impurity image is overlaid on the background of the generated image to generate the background.

[0054] III. Cell generation:

[0055] (7) Select a red blood cell sampled in step (2) and perform color transformation on saturation, brightness and contrast;

[0056] (8) Then, perform stretching and rotation on the cell through affine transformation, and perform affine transformation on the cell sampling image through three matrix multiplications: Where the rotation angle is θ, the horizontal stretching scale is λ x , and the vertical stretching scale is λ y ;

[0057] (9) Use the method of step (8) to generate 400 red blood cells and 30 white blood cells on the generated image. When generating cells, the image of the target semantic segmentation is also generated. When new cells are added, an inflation operation is performed on the existing semantic segmentation image to select the generation position from the background pixels to avoid excessive overlap of cell positions. The generated image and segmentation annotation image are shown in Figure 5 、 Figure 6 .

[0058] IV. Lightweight neural network training:

[0059] (10) Repeat steps (4) to (9) to generate 25 images;

[0060] (11) Train the lightweight U-Net convolutional neural network using the images generated in step (10). Since the generated image data has small variation, the original number of channels (64-128-256-512-1024) is reduced to (4-8-16-32-64), and 20 rounds of iteration training are performed. The segmentation result of each round is taken as a candidate result;

[0061] (12) In this embodiment, the 15th round segmentation result is selected, as shown inFigure 7 manual fine-tuning is performed;

[0062] (13) The final annotation segmentation map of the original picture is derived, as shown in Figure 8 .

[0063] The energy formula for obtaining the foreground and background segmentation of the image X in the rectangular region in the above step (2) is as follows:

[0064] E(X) = ∑ v∈V ɡ v (Xv) + ∑ (u,v)∈E h uv (X u ,X v )

[0065] Wherein, ɡ(·) calculates the probability of a pixel belonging to the foreground or the background, and h(·) calculates the similarity of a pixel and its surrounding pixels.

[0066] It is found in the implementation of the embodiment that the segmentation and annotation process of a single original picture takes 3-5 minutes, while the traditional manual annotation takes about 1 hour. The method of the present application is significantly better than the traditional manual annotation method, realizes fast and accurate automatic segmentation, greatly reduces the participation of artificial, and improves the annotation quality. In addition, the annotated original picture can be further trained, and a new model can be trained according to the style differences such as cell color and morphology between different original pictures.

[0067] In addition, it should be noted that the above described in the specification is only an example of the present application. Any equivalent changes or simple changes according to the features and principles described in the patent concept of the present application are included in the protection scope of the present application. Those skilled in the art can make various modifications or supplements to the described specific embodiments or replace them with similar ways, as long as they do not deviate from the technical solutions of the present application or exceed the scope defined by the present claims, which shall belong to the protection scope of the present application.

Claims

1. A self-learning-based semantic segmentation and annotation method for cell images, characterized in that, Includes the following steps: I. Manual sampling steps: (1) Perform preliminary manual annotation on the original image, and sample 2-5 white blood cells, 5-10 red blood cells and 1-2 background areas; (2) When manually annotating, manually select the rectangular area of ​​a single cell, and use the Grabcut algorithm to segment the sampled image of a single cell. Calculate the energy formula for the segmentation of the foreground and background of the image X within the rectangular area, and then use the minimum cut algorithm to segment the foreground and background. (3) After completing the sampling in step (1), select the background area in the original image, analyze the color of the background area, and obtain the background color range of the original image by fitting a Gaussian distribution function to the histogram of the R, G, and B channels. The formula is as follows: , Where α is the peak value of the histogram, μ is the pixel value corresponding to the peak value of the curve, and σ is the standard deviation obtained statistically. II. Background Generation: (4) Obtain the color distribution information N of the three channels R, G, and B. R (x), N G (x), N B After (x), when generating the image, the average color of the three channels is taken within one standard deviation to generate the background; (5) Add salt and pepper noise to the background of the generated image, that is, take the pixel position and generate noise according to the Gaussian distribution of μ = 15 and σ = 30, and then fuse it into the image; (6) In the generated image, Gaussian function is used to generate dots on the background to simulate the noise in the original image. The position μ and the size s of the noise are selected, and then a two-dimensional Gaussian kernel function is generated: Then, the simulated impurity image is overlaid on the background of the generated image to generate the background; III. Cell generation: (7) Select a red blood cell sampled in step (2) and perform color transformation on saturation, brightness and contrast; (8) Next, perform an affine transformation on the cell to stretch and rotate it; (9) Using the method in step (8), generate 300-500 red blood cells and 20-40 white blood cells on the generated image. At the same time as generating cells, generate the target semantic segmentation image. When adding new cells, perform a dilation operation on the existing semantic segmentation image and select the generation position from the background pixels to avoid excessive overlap of cell positions. IV. Lightweight Neural Network Training: (10) Repeat steps (4) to (9) to generate 10-30 images; (11) Train a lightweight U-Net convolutional neural network on the image generated in step (10), reduce its original number of channels (64-128-256-512-1024) to (4-8-16-32-64), and perform 20 rounds of iterative training, taking the segmentation result of each round as the candidate result; (12) Take the segmentation results of 20 rounds, select the round with the best effect, and manually fine-tune it to output the labeled segmentation map.

2. The cell image semantic segmentation and annotation method based on self-learning according to claim 1, characterized in that, The energy formula for segmenting the foreground and background of the image X within the rectangular region obtained in step (2) is as follows: E(X)=∑ v∈V ɡ v (X v )+∑ (u , v)∈E h uv (X u , X v ) Where g(‧) calculates the probability that a pixel belongs to the foreground or background, and h(‧) calculates the similarity between a pixel and its surrounding pixels.

3. The cell image semantic segmentation and annotation method based on self-learning according to claim 1, characterized in that, In step (8), the affine transformation of the cells is performed using cubic matrix multiplication, specifically as follows: Where θ is the rotation angle, λ x For the lateral stretching dimension, λ y The longitudinal stretching dimension.

Images