A bacterial microscopic image segmentation method based on deep learning network

Abstract

A bacterial microscopic image segmentation method based on a deep learning network comprises the following steps: 1) culturing bacteria, and taking a group of bacterial growth pictures at fixed time intervals under a microscope, performing image preprocessing, constructing a training set, a validation set and a test set that have no intersection with each other, and the training set includes an original image and a corresponding label image, and the validation set and the test set only contain the original image respectively; 2) constructing a U‑Net++ model, which has an encoder module and a decoder module, the encoder module performs feature extraction, the decoder module performs feature restoration decoding to the size of the original image, the training set is input into the U‑Net++ model for training, and then the validation set is input into the trained U‑Net++ model for verification, and a trained U‑Net++ model is obtained; 3) the test set is input into the trained U‑Net++ model, and a binary segmented image is output. The method of the present invention can automatically segment bacterial microscopic images quickly and accurately, eliminating too many complex image preprocessing links in the early stage and saving time.

Description

Technical Field

[0001] The present invention relates to the field of bacterial microscope image segmentation, and in particular to a bacterial microscope image segmentation method based on a deep learning network. Background Art

[0002] In the study and processing of images, the information contained in the image is often not what we are interested in. Our brain will automatically judge which image information we need based on our needs, and often this specific image part containing the information we want corresponds to the special properties (edges, shapes, colors, etc.) in the image, which we often call foreground in biological image processing, and the corresponding other image parts are background. In the biological field, cultivated bacteria are imaged under a microscope. These images are often affected by factors such as changes in the focus of the microscope, impurities in the culture medium, and environmental brightness. These noisy images often make it difficult for researchers to clearly separate the bacteria from the background, which also affects some subsequent judgments.

[0003] At present, the image segmentation methods in the field of biological image processing are mainly divided into: 1. Traditional methods for image segmentation using digital image processing, topology, statistics, mathematics, etc. 2. With the increase in computing power and the explosive development of deep learning in recent years, methods based on deep learning have increasingly occupied an irreplaceable position in the field of image segmentation.

[0004] Traditional methods mainly include:

[0005] 1. Threshold-based segmentation method: Its main idea is to calculate a single or multiple grayscale thresholds based on the grayscale features of the image, and then traverse all pixels in the image, compare their grayscale values ​​with the set threshold, and set pixels greater than the threshold as the foreground, and pixels less than the threshold as the background. Its advantages are simple calculation and high efficiency, but its disadvantages are also obvious: it does not consider spatial features at all, is extremely sensitive to noise in the image, and has poor robustness.

[0006] 2. Segmentation algorithm based on edge detection: It solves the segmentation problem by focusing on detecting edge pixels in different areas. The more representative ones are gradient algorithm, Roberts algorithm, Sobel algorithm, Laplacian algorithm, etc. Its advantages are fast speed and accurate edge detection, but its disadvantage is that it can only detect edge pixels and cannot guarantee the continuity and integrity of the edge.

[0007] 3. Image segmentation method based on wavelet transform: It is a relatively important part of digital image processing technology: first, the image histogram is decomposed into wavelet coefficients of different levels by binary wavelet transform, then the threshold is selected according to the given segmentation criteria and wavelet coefficients, and finally the image segmentation area is marked by the threshold. Its advantages are local transformation of spatial and frequency domains, multi-scale analysis, and since it operates in the frequency domain, it is not sensitive to noise. The disadvantage is that it is difficult to select a suitable filter.

[0008] Bacteria images have different shapes and grow in the form of colonies. They are numerous and overlapped, which makes segmentation more difficult. In the field of bacteria segmentation, there are methods that use superpixels to perform image segmentation (such as CN103914841A). This method is still based on traditional image segmentation methods (such as calculating superpixels, and calculating the color, shape, and size characteristics of each superpixel area, using prior knowledge to perform preliminary filtering on each superpixel area, and segmenting to determine candidate bacteria areas). The process is relatively cumbersome and the effect is not good.

[0009] Deep learning method: completely different from traditional image segmentation methods, it is based on computing power support and trains a large number of labeled data sets to allow the machine to autonomously learn the inherent laws and characteristics of the image. It mainly includes:

[0010] 1. CNN-based image segmentation: The main representative is Mask R-CNN proposed by He Kaiming et al. [He Kaiming, GkioxariGeorgia, DollarPiotr, Girshick Ross.Mask R-CNN.[J].IEEE transactions on pattern analysis and machine intelligence, 2020, 42(2).] in 2017, which mainly completes the pixel-level segmentation of the image.

[0011] 2. RNN-based image segmentation: The main representative is the ReSeg model proposed by Visin et al. [Visin F, Kastner K, Courville A, et al. ReSeg: A Recurrent Neural Network for Object Segmentation [J]. Computer ence, 2015.] in 2015. Its core is to combine multiple (Recurrent neural networks) RNNs together to fully consider the information relationship between the two image contexts.

[0012] 3. Image segmentation method based on upsampling: The main representative is the FCN model proposed by Jonathan Long et al. in 2015 [Long J, Shelhamer E, Darrell T. Fully Convolutional Networks for Semantic Segmentation [J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2015, 39 (4): 640-651.], which has become an industry benchmark in the field of image segmentation. Many of its segmentation ideas are groundbreaking. And some famous networks such as SegNet, Unet, Deeplab, etc. have more or less borrowed its ideas.

[0013] Currently, many deep learning-based methods are widely used in the processing of natural images, medical images, and biological images. In particular, there are still many segmentation problems that need to be solved in the recognition of bacterial images in biological images.

[0014] A method for image segmentation of cells based on a U-Net network is currently disclosed, which mainly uses the U-Net network to segment the image of cervical cell nuclei. It mainly includes three steps:

[0015] 1. Perform image preprocessing on the image, including using the Canny operator and Otsu algorithm combined with mathematical morphological operations to enhance the contrast of the cell nucleus.

[0016] 2. Use the U-Net network to segment the image, and add hollow convolution and residual network to the single U-Net network to improve the network's perception domain.

[0017] 3. Use clustering detection methods to detect the edges of cells.

[0018] It does a lot of work in image preprocessing, which is rather cumbersome. If these contrast enhancements are not performed, the segmentation effect will be poor. Summary of the invention

[0019] The main purpose of the present invention is to overcome the cumbersome and complicated early image preprocessing when the deep learning network used in the prior art is used for segmentation, and proposes a bacterial microscopic image segmentation method based on a deep learning network, which can quickly and accurately perform automatic segmentation of bacterial microscopic images, eliminating the excessive and complicated image preprocessing links in the early stage and saving time.

[0020] The present invention adopts the following technical solution:

[0021] A bacterial microscopic image segmentation method based on a deep learning network, characterized in that it comprises the following steps:

[0022] 1) Cultivate bacteria and take a set of bacterial growth pictures under a microscope at fixed time intervals, perform image preprocessing, and construct a training set, a validation set, and a test set that have no intersection with each other. The training set includes the original image and the corresponding labeled image, and the validation set and the test set only contain the original image;

[0023] 2) Construct a U-Net++ model, which has an encoder module and a decoder module. The encoder module performs feature extraction, and the decoder module performs feature restoration and decoding to the size of the original image. The training set is input into the U-Net++ model for training, and then the verification set is input into the trained U-Net++ model for verification to obtain a trained U-Net++ model;

[0024] 3) Input the test set into the trained U-Net++ model and output the binary segmented image.

[0025] Preferably, the image preprocessing in step 1) includes first adjusting the brightness and contrast of the image using ImageJ image processing software, and then performing a batch cropping operation on the image.

[0026] Preferably, in step 1), the open source software Labelme is used to manually label the bacteria in the bacterial growth image to obtain the labeled image, and ensure that the original image and the labeled image correspond one to one.

[0027] Preferably, step 1) further includes inverting, distorting, mirroring, and cropping the images in the training set, the validation set, and the test set to achieve data set expansion.

[0028] Preferably, the encoder module includes a plurality of encoder sub-modules of the same structure, and the encoder sub-module includes two convolution operations plus one pooling operation.

[0029] Preferably, the convolution operation adopts a 3*3 convolution kernel, the activation function is leaky Relu, the filling method is samepadding, and the method of initializing weights is he_normal; the pooling operation adopts a maximum pooling scheme, and the size of the pooling convolution kernel used is 2*2.

[0030] Preferably, the decoder module includes a plurality of decoder submodules of the same structure, and each decoder submodule includes a deconvolution operation, a skip connection parameter fusion operation and two convolution operations.

[0031] Preferably, the deconvolution operation uses a 2*2 convolution kernel, the activation function is leaky Relu, the step size is 2, and the padding method is samepadding; the skip connection parameter fusion operation is to fuse image parameters with the same features in the same layer, and only the depth of the parameters is increased after fusion; the convolution operation uses a 3*3 convolution kernel, the activation function is leaky Relu, the padding method is samepadding, and the method of initializing weights is he_normal.

[0032] Preferably, the U-Net++ model uses Adam as the optimizer with a learning rate of 1*10 -4 , the loss function selects the binary classifier binary_crossentropy, and uses accuracy as the measurement indicator.

[0033] Preferably, the method further comprises step 4) calculating the proportion of bacteria in the binary segmented image obtained by the U-Net++ model, so as to obtain a growth graph of bacteria with time as the horizontal axis.

[0034] It can be seen from the above description of the present invention that, compared with the prior art, the present invention has the following beneficial effects:

[0035] 1. The method of the present invention adopts the U-Ne++ network structure, which can autonomously learn and select the most appropriate number of network layers, and combines the different features obtained by the shallow network and the deep network, and integrates them, which is more robust, greatly reduces the time-consuming tedious image preprocessing operations, and achieves better image segmentation effects. Once the network learning is completed, the subsequent image processing speed will also be rapidly improved.

[0036] 2. The method of the present invention simply adjusts the brightness and contrast of the image before deep learning, and then performs batch cropping operations on the image, eliminating the need for excessive and complex image preprocessing steps in the early stage and saving time.

[0037] 3. The method of the present invention, in the encoder module, each time the encoder submodule is performed, certain information contained in the image, such as boundaries and colors, will be extracted. The more convolution times are, the more abstract features of the image can be captured, which can increase the robustness to some small disturbances of the input image, such as image translation, rotation, etc., reduce the risk of overfitting, reduce the amount of calculation, and increase the size of the receptive field.

[0038] 4. The method of the present invention, whose decoder module includes multiple decoder sub-modules, restores and decodes the abstract features to the size of the original image, and finally obtains the segmentation result. The skip connection parameter fusion operation involved is to fuse the image parameters with the same features in the same layer. After fusion, only the depth of the parameters is increased. It connects a lot of information of the input image and helps to restore the information loss caused by downsampling.

[0039] 5. The method of the present invention can automatically segment bacterial microscopic images quickly and accurately, and can quantitatively represent the growth of bacteria based on a group of segmented images without interfering with the growth state of bacteria. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1 is a flow chart of the method of the present invention;

[0041] Figure 2a is the original image in the training set;

[0042] Figure 2b is the label image;

[0043] Figure 3 It is the overall framework diagram of the U-Net++ model of the present invention;

[0044] Figure 4a is the original image in the test set;

[0045] Figure 4b is the output binary segmentation image;

[0046] Figure 5 This is the growth curve of Escherichia coli.

[0047] The present invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments. DETAILED DESCRIPTION

[0048] The present invention is further described below through specific implementation modes.

[0049] See also Figure 1 , a bacterial microscopic image segmentation method based on deep learning network, comprising the following steps:

[0050] 1) Cultivate bacteria and take a set of bacterial growth pictures under a microscope at fixed time intervals, perform image preprocessing, and construct training sets, validation sets, and test sets that have no intersection with each other. The training set includes the original image and the corresponding label image, and the validation set and test set only contain the original image.

[0051] Since the pixel values ​​of bacteria images taken under a microscope are generally large and the image quality varies, it is necessary to perform simple image cropping, brightness and contrast enhancement preprocessing before deep learning. The specific steps are as follows:

[0052] 1.1) Cultivate bacteria and take a set of bacterial growth pictures at fixed time intervals under a microscope. When culturing bacteria, the introduction of impurities should be reduced as much as possible. For example, when culturing E. coli, the solid culture medium needs to be inverted to reduce the water vapor produced by bacterial respiration from introducing impurities into the culture dish, thereby affecting the quality of subsequent images.

[0053] At the same time, it is necessary to select the appropriate microscope observation magnification, which can make the image clearer. It is also necessary to select the appropriate shooting interval time.

[0054] Take E. coli as an example: E. coli is imaged once every 5 minutes using a 40x microscope under an optical microscope to generate a set of 100 2048*2048 growth images. When culturing E. coli, the solid culture medium needs to be inverted to reduce the water vapor produced by bacterial respiration from introducing impurities into the culture dish, thereby affecting the quality of subsequent images.

[0055] 1.2) Open the ImageJ image processing software and adjust the brightness and contrast of the image with appropriate parameters. ImageJ software is an open source image processing software and is currently the preferred software for image processing operations. For the images taken in step A1 above, a group of images can be directly opened by ImageJ, and then through the Image drop-down box of the software, select the Adjust operation, and open the Brightness / Contrast option therein. In the opened operation box, the Auto button can be directly selected, and the software will automatically select the appropriate brightness and contrast. Of course, if the effect is not ideal, you can also manually adjust the brightness Brightness and Contrast values, and finally click the Apply option to complete the batch preprocessing of the image. Using ImageJ software to perform simple basic operations on images the fastest, the batch operation of the software also saves the operator a lot of time.

[0056] For example: Open the ImageJ image processing software, convert the image into a single-channel 8-bit image, and select appropriate parameters to adjust the brightness and contrast of the image.

[0057] 1.3) Crop the high-pixel image processed in the previous step. If proper cropping is not performed, the amount of data input into the deep learning network for single processing will be very large, and the computing power required for the computer will also increase linearly. The size of the cropping is determined by the input size of the deep learning network. Generally, the size of each image after cropping is no larger than 512*512 pixels. If the training computing power is very high, the pixel value can be appropriately increased. The cropping process is performed in a sliding window manner. After processing, the 2048*2048 pixel image becomes 16 512*512 pixel images. A group of 100 2048*2048 images are finally cropped into 1600 512*512 pixel images.

[0058] For constructing the data set, ensure that the training set includes the original image and the corresponding labeled image for input into the neural network for training, and the validation set and test set only contain the original image. For example: 1600 512*512 pixel images correspond to 16 non-intersecting positions of each image, and take out the images of 10 positions, and get 1000 512*512 images with a 5-minute interval as the training set images, and the remaining 600 512*512 images with a 5-minute interval correspond to the remaining two positions of the image, and take out 200 images of the two positions as the validation set, and 400 images of the four positions as the test set. Training set: validation set: test set = 5:1:2, and there is no intersection between them.

[0059] In the training set, the open source software Labelme can be used to manually label the bacteria in the bacterial images to ensure that the original images and the labeled images correspond one to one. Figure 2a , Figure 2b ; Then, the images in the training set, validation set, and test set are inverted, distorted, mirrored, cropped, and other operations are performed to expand the data set.

[0060] Furthermore, when manually labeling images, polygonal annotation is used, and the bacteria need to be fitted as closely as possible. The more polygonal sides there are, the more accurate the labeled image will be. The labeled image will generate a json file. After converting json to png image format, the corresponding label file can be finally generated. The closed polygon contains the labeled bacteria, the internal pixels are represented by pixel value 0 (i.e. white), and the rest are background filled with pixel value 255 (i.e. black).

[0061] Among them, the augmented data set can be quickly generated using the ImageDataGenerator function that has been written in the Tensorflow deep learning framework, including rotation|reflection transformation, flip transformation, scaling transformation, translation transformation, scale transformation, etc. After inheriting the method of the parent class ImageDataGenerator, rewrite the function to package the images in the training set and the corresponding label images, and generate augmented data at the same time, ensuring that the original image and the corresponding label image undergo the same operation conversion.

[0062] 2) Construct a U-Net++ model, which has an encoder module and a decoder module. The encoder module (downsampling) performs feature extraction, and the decoder module (upsampling) performs feature restoration and decoding to the size of the original image. The training set is input into the U-Net++ model for training, and then the verification set is input into the trained U-Net++ model for verification to obtain a trained U-Net++ model.

[0063] The U-Net++ model of the present invention has a corresponding decoder to restore after each encoder operation. The advantage is that it can capture features at different levels and integrate the learned shallow features and deep features. The network can learn the importance of different deep features by itself and has good robustness.

[0064] The encoder module is also called the feature extraction stage, which contains multiple encoder submodules with the same structure. Each encoder submodule includes two convolution operations plus one pooling operation. Each time the encoder submodule is performed, certain information contained in the image, such as boundaries and colors, is extracted. The more convolutions are performed, the more abstract features of the image can be captured. It can increase the robustness to some small disturbances of the input image, such as image translation and rotation, reduce the risk of overfitting, reduce the amount of calculation, and increase the size of the receptive field.

[0065] In this module, the parameters of the convolution operation are: a 3*3 convolution kernel, leaky Relu activation function, "samepadding" padding, and "he_normal" weight initialization method; the pooling operation adopts the maximum pooling scheme, and the size of the pooling convolution kernel used is 2*2.

[0066] In practical applications, the downsampling part can be replaced by the classic BottleNeckblock, Residualblock or Denseblock.

[0067] The decoder module is also called the feature restoration module. Its function is to restore the abstract features to the size of the original image and finally obtain the segmentation result. It includes multiple decoder submodules with the same structure. Each decoder submodule includes a deconvolution operation, a skip connection parameter fusion operation and two convolution operations.

[0068] In this module, the parameters of the deconvolution operation are: a 2*2 convolution kernel, leaky Relu as the activation function, a step size of 2, and a padding method of "samepadding"; the skip connection parameter fusion operation is to fuse the image parameters with the same features in the same layer. After fusion, only the depth of the parameters is increased. It connects a lot of information of the input image and helps to restore the information loss caused by downsampling; the parameters of the convolution operation are: a 3*3 convolution kernel, leaky Relu as the activation function, the padding method is "samepadding", and the method of initializing weights is "he_normal".

[0069] The model structure of the present invention is as follows Figure 3 As shown in the figure, after four downsamplings, long and short jump connections are performed in each layer, and ten upsampling operations, the specific U-Net++ model network structure parameters are shown in the following table:

[0070]

[0071]

[0072] Preferably, the number of network layers in the present invention is 4. The number of network layers here can also be replaced by any value. Tasks with high segmentation difficulty often use deeper network layers, and vice versa. After building the U-Net++ model, Adam (Adaptive Moment Estimation) is used as the optimizer, and the learning rate is 1*10 -4 , the loss function selects the binary classifier binary_crossentropy, and uses accuracy as the measurement indicator.

[0073] In the present invention, the U-Net++ network parameters such as convolution kernel size, number, learning rate, optimizer, etc. can be set to appropriate parameters according to specific circumstances.

[0074] 3) Input the test set into the trained U-Net++ model and output the binary segmented image. That is, the trained U-Net++ model can be used for bacterial microscopic image segmentation.

[0075] For example, a server with a GeForce GTX 1080Ti high-performance graphics card was used as the hardware platform, and the Tensorflow deep learning framework was selected. The dataset used was the 1,600 processed images above plus the images after data augmentation. After 20 cycles of training for 5 hours, the accuracy of the U-Net++ network on the validation set reached 91.2%. After the training was completed, the 400 images in the test set were tested and output.

[0076] In addition, the traditional method of measuring bacterial growth curve is: inoculate bacteria into a culture solution in a conical flask with a side wall test tube, take out the conical flask at a suitable culture temperature and good ventilation, measure the bacterial concentration (optical density value, OD value) with a spectrophotometer, and plot the results against the corresponding culture time to obtain the growth curve of the bacteria. This method requires taking out the culture solution containing bacteria at a regular interval, which is time-consuming and labor-intensive.

[0077] The present invention may also include step 4) calculating the proportion of bacteria in the binary segmented image obtained by the U-Net++ model, thereby obtaining a growth diagram of bacteria with time as the horizontal axis. The bacterial images taken at fixed intervals are directly segmented using a deep learning method, and then the proportion of bacteria is calculated in the segmented binary image to obtain a growth diagram of bacteria with time as the horizontal axis. This method does not interfere with the growth state of bacteria at all, and the use of deep learning for image segmentation can also separate the prospects of bacteria faster and better, so that the growth of bacteria can be quantitatively represented by the segmented image later.

[0078] For example, the output image during the test is a binary black-and-white image with E. coli as the foreground. The background pixel is 0, which means black, and the foreground bacteria pixel is 255, which means white. Figure 4a This is an E. coli image from the test set. Figure 4b The corresponding predicted segmentation output image is obtained. A group of 100 images with a 5-minute interval are selected from the 400 images output by the test set to quantitatively represent the growth of bacteria. The pixel values ​​of each image are read in sequence, and the percentage of the E. coli area in the entire image is calculated. This is used as the vertical axis, and the 5-minute time interval is used as the horizontal axis to draw the E. coli growth curve, as shown in the figure. Figure 5 shown.

[0079] The present invention adopts the U-Ne++ network structure, which can autonomously learn and select the most appropriate number of network layers, and combines the different features obtained from the shallow network and the deep network, and fuses them to make it more robust, greatly reducing the time-consuming tedious image preprocessing operations and achieving better image segmentation effects. Once the network learning is completed, the subsequent image processing speed will also be rapidly improved.

[0080] The above is only a specific implementation of the present invention, but the design concept of the present invention is not limited thereto. Any non-substantial changes to the present invention using this concept shall be deemed as an infringement of the protection scope of the present invention.

Claims

1. A bacterial microscopic image segmentation method based on deep learning network, characterized in that: The steps include: 1) Cultivate bacteria and take a set of bacterial growth pictures at fixed time intervals under a microscope for image preprocessing. Image preprocessing includes first using ImageJ image processing software to adjust the brightness and contrast of the image, and then batch cropping the image; using the open source software Labelme to manually label the bacteria in the bacterial growth picture to obtain a labeled image. When manually labeling the image, polygon annotation is used to make the annotation fit the bacteria and ensure that the original image and the labeled image correspond one to one; construct a training set, a validation set, and a test set that have no intersection with each other, and the training set includes the original image and the corresponding labeled image, and the validation set and the test set only contain the original image; also include inverting, distorting, mirroring, and cropping the images in the training set, validation set, and test set to achieve data set expansion; 2) Construct a U-Net++ model, which has an encoder module and a decoder module. The encoder module contains multiple encoder submodules with the same structure. The encoder submodule includes two convolution operations plus one pooling operation. The convolution operation uses a 3*3 convolution kernel, the activation function is leaky Relu, the padding method is samepadding, and the weight initialization method is he_normal. The pooling operation uses the maximum pooling scheme, and the size of the pooling convolution kernel used is 2*2. The decoder module includes multiple decoder submodules with the same structure. Each decoder submodule includes a deconvolution operation, a skip connection parameter fusion operation and two convolution operations. The deconvolution operation uses a 2*2 convolution kernel, the activation function is leaky Relu, the step size is 2, and the padding method is samepadding. The skip connection parameter fusion operation is to fuse the image parameters with the same features in the same layer, and only the depth of the parameters is increased after fusion. The convolution operation uses a 3*3 convolution kernel, the activation function is leaky Relu, the step size is 2, and the padding method is samepadding. Relu, the filling method is samepadding, and the method of initializing weights is he_normal; the encoder module extracts features, and the decoder module restores and decodes features to the size of the original image. The training set is input into the U-Net++ model for training, and then the verification set is input into the trained U-Net++ model for verification to obtain the trained U-Net++ model; the U-Net++ model uses Adam as the optimizer, and the learning rate is 1*10 -4 , the loss function selects the binary classifier binary_crossentropy, and uses accuracy as the measurement indicator; 3) Input the test set into the trained U-Net++ model and output the binary segmented image; 4) The binary segmented image obtained by the U-Net++ model is used to calculate the proportion of bacteria, thereby obtaining a bacterial growth graph with time as the horizontal axis.

Images