Automatic cell segmentation method and device for high-content fluorescent microscopic image

Abstract

The invention provides an automatic cell segmentation method and device for a high-content fluorescent microscopic image, and belongs to the technical field of image processing, and the method comprises the steps: inputting the high-content fluorescent microscopic image into a semantic segmentation model, and obtaining a mask image outputted by the semantic segmentation model; and inputting the mask image into an image segmentation model to obtain an instance segmentation mask image output by the image segmentation model. According to the automatic cell segmentation method and device for the high-content fluorescent microscopic image provided by the invention, a two-stage segmentation strategy combining semantic segmentation and image segmentation is adopted, so that deep decoupling can be performed on a color channel and a space channel of the high-content fluorescent microscopic image; and instance segmentation of the high-content fluorescent microscopic image is realized with low data annotation cost, and an automatic, rapid and universal high-content image cell segmentation algorithm with small parameter quantity and small calculation quantity is realized.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to an automatic cell segmentation method and device for high-content fluorescence microscopic images. Background Art

[0002] Microscopes have become an indispensable and important tool for observing life activities at cellular resolution, such as cell division and growth, metabolism, signal transduction, tumor detection, and drug screening. Simultaneous fluorescent staining of multiple colors can display complete cells, different cell types, and different subcellular structures. Different objects can then be segmented and features such as morphology (area, size, and shape), fluorescence intensity, and surface texture can be calculated. The acquisition of these morphological maps can fully and accurately describe cell phenotypes, which helps to identify possible targets in cell populations and / or cell substructures in the context of compound treatment. It can be seen that segmenting cells in high-content fluorescence microscopy images is a prerequisite for a comprehensive and accurate description of cell phenotypes and for achieving high-content screening.

[0003] At present, the main method for segmenting cells in high-content fluorescence microscopy images is to directly input high-content fluorescence microscopy images into image segmentation models such as UNet, Cellpose, and TissueNet. These image segmentation models generally use convolutional layers to build the entire network, and use deconvolution layers to upsample the feature map obtained by the last convolutional layer, restore it to the same size as the input image, and classify the upsampled image pixel by pixel, and finally output the image after cell segmentation to complete cell segmentation.

[0004] However, with the emergence of many complex cell analysis tasks and the development of high-throughput imaging technology, cell image data has shown explosive growth. On the one hand, the practice of directly using image segmentation models to process high-content fluorescence microscopy images has caused increasingly serious problems such as decreased cell segmentation accuracy due to failure to pay attention to the characteristics of high-content fluorescence microscopy images themselves, such as channel independence and single color. On the other hand, the large amount of information contained in high-throughput high-content fluorescence microscopy images has exceeded the ability of manual processing and analysis. The time cost and labor cost of data annotation for high-throughput sample images when training image segmentation models are also increasing.

[0005] Therefore, it is necessary to provide a more accurate and lower-training-cost automatic cell segmentation method for high-content fluorescence microscopy images. Summary of the invention

[0006] The present invention provides an automatic cell segmentation method and device for high-content fluorescence microscopy images, which are used to solve the defects in the prior art that directly inputting high-content fluorescence microscopy images into an image segmentation model leads to decreased cell accuracy and increased training costs, and realize a more accurate and lower-cost automatic cell segmentation method for high-content fluorescence microscopy images.

[0007] The present invention provides an automatic cell segmentation method for high-content fluorescence microscopy images, comprising: Inputting the high-content fluorescence microscopy image into a semantic segmentation model to obtain a mask image output by the semantic segmentation model; The mask image is input into an image segmentation model to obtain an instance segmentation mask image output by the image segmentation model.

[0008] According to an automatic cell segmentation method for high-content fluorescence microscopy images provided by the present invention, before inputting the mask image into an image segmentation model to obtain an instance segmentation mask image output by the image segmentation model, the mask image is subjected to a two-stage erosion process, comprising: When the area of ​​the first region to be segmented in the mask image is greater than the first threshold, iteratively eroding the first region to be segmented using a coarse erosion kernel until the area of ​​the first region to be segmented is less than or equal to the first threshold, thereby obtaining a mask image after first-stage erosion; When the area of ​​the second region to be segmented in the mask image after the first stage of erosion is greater than the second threshold, the second region to be segmented is iteratively eroded using a fine erosion kernel until the area of ​​the second region to be segmented is less than or equal to the second threshold, thereby completing the two-stage erosion processing of the mask image.

[0009] According to an automatic cell segmentation method for high-content fluorescence microscopy images provided by the present invention, the semantic segmentation model is constructed based on the NUSeg neural network.

[0010] According to an automatic cell segmentation method for high-content fluorescence microscopy images provided by the present invention, the encoder of the semantic segmentation model includes a first branch and a second branch; The first branch includes a first convolutional layer; The second branch includes a maximum pooling layer and two separable convolutional layers connected in sequence; The separable convolution layer includes a batch normalization layer, a depth-separable convolution layer and a ReLU activation function layer connected in sequence; The depth-wise separable convolutional layer includes a parallel convolutional layer, a splicing layer, and a second convolutional layer connected in sequence; The parallel convolutional layer includes three third convolutional layers connected in parallel.

[0011] According to an automatic cell segmentation method for high-content fluorescence microscopy images provided by the present invention, the decoder of the semantic segmentation model adopts a spatial channel compression and excitation attention mechanism to compress and excite the feature map input to the decoder in both spatial and channel dimensions to obtain a feature map output to the decoder.

[0012] According to an automatic cell segmentation method for high-content fluorescence microscopy images provided by the present invention, the loss function of the semantic segmentation model is determined based on the Dice loss function and the focal loss function.

[0013] The present invention also provides an automatic cell segmentation device for high-content fluorescence microscopic images, comprising: A semantic segmentation module, used for inputting the high-content fluorescence microscopy image into the semantic segmentation model to obtain a mask image output by the semantic segmentation model; The image segmentation module is used to input the mask image into the image segmentation model to obtain the instance segmentation mask image output by the image segmentation model.

[0014] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and running on the processor, wherein when the processor executes the computer program, the automatic cell segmentation method for high-content fluorescence microscopy images as described in any one of the above-mentioned methods is implemented.

[0015] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the automatic cell segmentation method for high-content fluorescence microscopy images as described in any one of the above is implemented.

[0016] The present invention also provides a computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the computer program implements any of the above-described automatic cell segmentation methods for high-content fluorescence microscopy images.

[0017] The automatic cell segmentation method and device for high-content fluorescence microscopic images provided by the present invention adopt a two-stage segmentation strategy combining semantic segmentation and image segmentation based on the characteristics of high-content fluorescence microscopic images, such as independent channels, relatively single colors, fine segmentation granularity, and sensitivity to changes in tiny boundaries. The method can deeply decouple the color channel and the spatial channel of the high-content fluorescence microscopic images, thereby improving the versatility of the cell segmentation method. Specifically, the high-content fluorescence microscopic images are first processed into mask images that only mark the foreground and the background using a semantic segmentation model, and then the mask images are processed using an image segmentation model to obtain an instance segmentation mask image that finally marks each cell on the foreground and the background, which means that the training of the semantic segmentation model and the image segmentation model can be realized at a lower data annotation cost, so as to realize the instance segmentation of the high-content fluorescence microscopic images at a lower data annotation cost. In general, an automated, fast, and universal high-content image cell segmentation algorithm can be realized with a smaller number of parameters and less computational effort, so as to help cell biologists promote large-scale analysis of high-content screening. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0019] Figure 1 This is one of the flow charts of the automatic cell segmentation method for high-content fluorescence microscopy images provided by the present invention.

[0020] Figure 2 It is a schematic diagram of the structure of the encoder provided by the present invention.

[0021] Figure 3 This is the second flow chart of the automatic cell segmentation method for high-content fluorescence microscopy images provided by the present invention.

[0022] Figure 4 It is an example diagram of the model training image provided by the present invention.

[0023] Figure 5 This is one of the structural schematic diagrams of the automatic cell segmentation device for high-content fluorescence microscopic images provided by the present invention.

[0024] Figure 6 The second structural schematic diagram of the automatic cell segmentation device for high-content fluorescence microscopic images provided by the present invention.

[0025] Figure 7 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0026] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0027] It should be noted that, in the description of the present invention, the term "include" or any other variant thereof is intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. For those of ordinary skill in the art, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0028] The terms "first", "second", etc. in the present invention are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the terms used in this way can be interchangeable under appropriate circumstances, so that the embodiments of the present invention can be implemented in an order other than those illustrated or described herein, and the objects distinguished by "first", "second", etc. are generally of the same type, and the number of objects is not limited. For example, the first object can be one or more.

[0029] Combine the following Figure 1-Figure 7 The invention describes an automatic cell segmentation method and device for high-content fluorescence microscopy images.

[0030] Figure 1 is one of the flow charts of the automatic cell segmentation method for high-content fluorescence microscopy images provided by the present invention, such as Figure 1 As shown, the automatic cell segmentation method for high-content fluorescence microscopy images includes but is not limited to step 101 and step 102.

[0031] It should be noted that the executor of the automatic cell segmentation method for high-content fluorescence microscopy images provided by the present invention is the corresponding automatic cell segmentation device for high-content fluorescence microscopy images, which can specifically be a server, computer equipment, such as a mobile phone, tablet computer, laptop computer, PDA, vehicle-mounted electronic equipment, wearable device, Ultra-Mobile Personal Computer (UMPC), netbook or Personal Digital Assistant (PDA), etc.

[0032] Step 101: inputting a high-content fluorescence microscopy image into a semantic segmentation model to obtain a mask image output by the semantic segmentation model.

[0033] The semantic segmentation model is a neural network model used to classify each pixel in an image into a specific label or category. It can identify the overall content of the image and also needs to assign a specific category to each pixel. It is usually used to process different objects or areas in the image.

[0034] In the mask image output by the semantic segmentation model, the background of the mask image is marked as 0, and all cells in the foreground are marked as 1.

[0035] Among them, the semantic segmentation model can be built based on any semantic segmentation network such as FCN (Fully Convolutional Network), DeepLab network, SegNet network, etc.

[0036] Specifically, a high-content fluorescence microscopy image captured by a high-content microscope or a confocal microscope is input into a pre-trained semantic segmentation model to obtain a mask image output by the semantic segmentation model.

[0037] It can be understood that the semantic segmentation model is trained using multiple high-content fluorescence microscopy image samples and their corresponding mask image labels.

[0038] Step 102: Input the mask image into an image segmentation model to obtain an instance segmentation mask image output by the image segmentation model.

[0039] The image segmentation model is a neural network model used to segment an image into several regions or objects.

[0040] In the instance segmentation mask image output by the image segmentation model, the background of the instance segmentation mask image is marked as 0, and each cell in the foreground is marked from 1 to n in sequence.

[0041] Specifically, the mask image obtained after the semantic segmentation model processes the high-content fluorescence microscopy image is input into the pre-trained image segmentation model to obtain the instance segmentation mask image output by the image segmentation model.

[0042] Optionally, the image segmentation model is built based on a marker-based watershed algorithm.

[0043] Optionally, the image segmentation model can also be built based on any image segmentation algorithm such as a watershed algorithm, a threshold-based segmentation method, etc.

[0044] The watershed algorithm is an image segmentation method based on mathematical morphology, which aims to divide an image into regions with similar characteristics, usually by simulating the expansion process of "water flow" in a terrain map. In this process, the pixel grayscale value of the image represents the height of the terrain, and the goal of the algorithm is to achieve image segmentation by calculating the "watershed" (i.e., boundary area) in the image. However, the watershed algorithm may cause over-segmentation due to the small height difference in some areas of the gradient map.

[0045] Compared with the watershed algorithm, the marker-based watershed algorithm can reduce the phenomenon of over-segmentation and improve the accuracy of segmentation by utilizing pre-defined marker information. Specifically, the marker-based watershed algorithm extracts the foreground area in the image and assigns a unique marker. In addition, it marks the extracted background area and guides the segmentation process by using the marker, which is similar to starting from the "seed" of the marked area. The water flow gradually expands and fills the low-gradient area until it encounters other markers or region boundaries. In this process, the watershed line on the boundary will divide different areas to ensure that over-segmentation does not occur. In other words, the marker-based watershed algorithm significantly reduces the over-segmentation problem that may occur in the traditional watershed algorithm when the boundary is unclear or the image complexity is high by introducing prior information (markers) in the segmentation process.

[0046] It can be understood that the image segmentation model is trained using a plurality of mask image samples and their corresponding instance segmentation mask image labels.

[0047] High-content screening for high-content fluorescence microscopy image processing generally requires fluorescent labeling of cell structures. This makes high-content fluorescence microscopy image processing different from other general image processing scenarios. It has the characteristics of channel independence, relatively single color, fine segmentation granularity, and sensitivity to changes in tiny boundaries.

[0048] Cell segmentation in the high-content screening task for high-content fluorescence microscopy image processing is an instance segmentation task, which requires the correct detection and accurate segmentation of each cell in the image, and is a fusion of semantic segmentation and target detection. This also means that when training a neural network model for processing high-throughput high-content fluorescence microscopy images, it takes a lot of time and manpower to annotate the data of the label images corresponding to the high-content fluorescence microscopy image samples.

[0049] The automatic cell segmentation method for high-content fluorescence microscopic images provided by the present invention adopts a two-stage segmentation strategy combining semantic segmentation and image segmentation based on the characteristics of high-content fluorescence microscopic images, such as independent channels, relatively single colors, fine segmentation granularity, and sensitivity to changes in tiny boundaries. The color channel and spatial channel of the high-content fluorescence microscopic images can be deeply decoupled, thereby improving the versatility of the cell segmentation method. Specifically, the high-content fluorescence microscopic images are first processed into mask images that only mark the foreground and background using a semantic segmentation model, and then the mask images are processed using an image segmentation model to obtain an instance segmentation mask image that finally marks each cell on the foreground and the background, which means that the training of the semantic segmentation model and the image segmentation model can be realized at a lower data annotation cost, so as to realize the instance segmentation of the high-content fluorescence microscopic images at a lower data annotation cost. In general, an automated and rapid high-content image cell segmentation algorithm can be realized with a smaller number of parameters and less computational effort, so as to help cell biologists promote large-scale analysis of high-content screening.

[0050] Based on the above embodiment, as an optional embodiment, before inputting the mask image into the image segmentation model to obtain the instance segmentation mask image output by the image segmentation model, a two-stage erosion process is performed on the mask image, including: When the area of ​​the first region to be segmented in the mask image is greater than the first threshold, iteratively eroding the first region to be segmented using a coarse erosion kernel until the area of ​​the first region to be segmented is less than or equal to the first threshold, thereby obtaining a mask image after first-stage erosion; When the area of ​​the second region to be segmented in the mask image after the first stage of erosion is greater than the second threshold, the second region to be segmented is iteratively eroded using a fine erosion kernel until the area of ​​the second region to be segmented is less than or equal to the second threshold, thereby completing the two-stage erosion processing of the mask image.

[0051] The region to be segmented is a cell image region located in the foreground part of the mask image when the image segmentation model processes the mask image.

[0052] The first area to be segmented is an area to be segmented on the mask image that is processed and output by the semantic segmentation model and has not been eroded.

[0053] The second region to be segmented is the region to be segmented on the mask image that has undergone the first stage erosion process in the two-stage erosion process.

[0054] The specific setting thresholds of the first threshold and the second threshold can be determined based on specific high-content screening needs, such as cell size, imaging device performance, semantic segmentation model or image segmentation model performance and other factors.

[0055] The size of the coarse erosion core is larger than that of the fine erosion core, and the shapes of the coarse erosion core and the fine erosion core are different.

[0056] Specifically, after the high-content fluorescence microscopy image is input into the semantic segmentation model to obtain a mask image and before the mask image is input into the image segmentation model, the mask image is subjected to a two-stage erosion process based on conditional erosion, that is, erosion is performed only when the size of the area to be segmented exceeds a predefined threshold.

[0057] First, the area of ​​the first region to be segmented in the mask image is determined. When the area of ​​the first region to be segmented is greater than the first threshold (T1), the coarse erosion kernel is used to erode the region in the first region to be segmented whose area is greater than T1 until the area of ​​the first region to be segmented is less than or equal to T1, thereby obtaining the mask image after the first stage of erosion.

[0058] Furthermore, the area of ​​the second region to be segmented in the mask image after the first stage of erosion is determined. When the area of ​​the second region to be segmented is greater than the second threshold (T2), the region with an area greater than T2 in the second region to be segmented is eroded using a fine erosion kernel until the area of ​​the second region to be segmented is less than or equal to T2, thereby completing the two-stage erosion processing of the mask image and obtaining the mask image input into the image segmentation model, so that the mask image is processed by the image segmentation model.

[0059] For example, when the image segmentation model is constructed using a marker-based watershed algorithm, the area to be segmented in the mask image that has undergone two-stage erosion processing can be used as a "seed" and processed by the image segmentation model.

[0060] The automatic cell segmentation method for high-content fluorescence microscopy images provided by the present invention can segment overlapping and adherent cells in a mask image by performing two-stage conditional erosion using coarse erosion kernels and fine erosion kernels of different sizes and shapes; by using the two-stage conditional erosion and image segmentation model as the post-processing link of the mask image, the over-segmentation problem caused by noise and local irregularity of gradient when the image segmentation model is built using algorithms such as the traditional watershed algorithm is specifically solved.

[0061] Based on the above embodiment, the semantic segmentation model is constructed based on the NUSeg (Multi-scale U-Net for Segmentation) neural network.

[0062] The NUSeg neural network is based on the U-Net architecture. The core structure includes an encoder (downsampling path) and a decoder (upsampling path). The encoder and decoder are connected through jump connections, so that information can be transmitted directly from the encoder to the decoder to maintain detail information and enhance segmentation accuracy. It mainly introduces key designs such as multi-scale feature extraction, multi-scale contextual information integration, and fusion of information at different scales. It has the advantages of strong adaptability, strong detail retention ability, and high accuracy.

[0063] Furthermore, the semantic segmentation model is constructed using the Unet++ architecture and the NUSeg neural network. Additional modules can be added to the decoder of the NUSeg neural network, and Unet++ can be effectively used to promote the fusion of feature maps at different levels and enhance the ability of the semantic segmentation model to extract semantic information from images.

[0064] The automatic cell segmentation method for high-content fluorescence microscopy images provided by the present invention can enhance the ability of the semantic segmentation model to extract semantic information from high-content fluorescence microscopy images by constructing a semantic segmentation model based on the NUSeg neural network using the Unet++ architecture, and effectively transfer low-level detail information to high-level semantic segmentation results, i.e., mask images, thereby improving the accuracy of image segmentation and retaining details.

[0065] Based on the above embodiment, as an optional embodiment, the encoder of the semantic segmentation model includes a first branch and a second branch; The first branch includes a first convolutional layer; The second branch includes a maximum pooling layer and two separable convolutional layers connected in sequence; The separable convolution layer includes a batch normalization layer, a depth-separable convolution layer and a ReLU activation function layer connected in sequence; The depth-wise separable convolutional layer includes a parallel convolutional layer, a splicing layer, and a second convolutional layer connected in sequence; The parallel convolutional layer includes three third convolutional layers connected in parallel.

[0066] Specifically, when building a semantic segmentation model, an encoder of the semantic segmentation model is built based on the Xception module. Figure 2 It is a schematic diagram of the structure of the encoder provided by the present invention, such as Figure 2 As shown, the encoder includes a first branch and a second branch; the first branch includes a first convolutional layer; the second branch includes a maximum pooling layer and two separable convolutional layers connected in sequence, each separable convolutional layer includes a batch normalization layer, a depth-wise separable convolutional layer and a ReLU activation function layer connected in sequence, and each depth-wise separable convolutional layer includes a parallel convolutional layer, a splicing layer and a second convolutional layer connected in sequence.

[0067] Optionally, the convolution kernel size of the first convolution layer is 1×1.

[0068] Optionally, the convolution kernel size of the second convolution layer is 1×1.

[0069] Optionally, the convolution kernel size of the third convolution layer is 3×3.

[0070] When a feature map is input to an encoder, the feature Figure 1 On the one hand, the processing result of the first branch is obtained after being processed by the first convolution layer in the first branch; on the other hand, the processing result of the second branch is obtained after being processed by the maximum pooling layer and two separable convolution layers in turn, and in each separable convolution layer, the batch normalization layer, the depth-wise separable convolution layer and the ReLU activation function layer are processed in turn, and in each depth-wise separable convolution layer, the tensor of the input depth-wise separable convolution layer is processed by three third convolution layers at the same time, and the processing results of the three third convolution layers are spliced ​​together through the splicing layer to obtain the output depth-wise separable convolution layer result, so as to finally obtain the processing result of the second branch. The processing result of the first branch is fused with the processing result of the second branch to obtain the feature map of the output encoder.

[0071] The automatic cell segmentation method for high-content fluorescence microscopy images provided by the present invention takes into account that high-content fluorescence microscopy images are labeled in monochrome (for example, cell nuclei are labeled with blue DAPI). By constructing an encoder of a semantic segmentation model based on the Xception module, it is possible to effectively decouple channel and spatial information in the feature map, thereby improving the extraction of low-level feature information and enhancing the details of fluorescence color, thereby improving the performance of the semantic segmentation model, and ultimately improving the accuracy and speed of cell segmentation.

[0072] Based on the above embodiment, as an optional embodiment, the decoder of the semantic segmentation model adopts a spatial and channel Squeeze and Excitation (scSE) mechanism to compress and excite the feature map input to the decoder in both spatial and channel dimensions to obtain a feature map output to the decoder.

[0073] The scSE attention mechanism combines channel attention and spatial attention to enable the network to recalibrate features in different dimensions of space and channels, effectively improving the capabilities of convolutional neural networks in feature extraction and representation learning, thereby improving the performance of the network.

[0074] The automatic cell segmentation method for high-content fluorescence microscopy images provided by the present invention introduces the scSE attention mechanism in each layer of the decoder to achieve spatial "squeezing and excitation" and channel "squeezing and excitation" on the feature map, so that the feature map output by the decoder is more accurately calibrated.

[0075] Based on the above embodiment, as an optional embodiment, the loss function of the semantic segmentation model is determined based on the Dice loss function and the focal loss function.

[0076] The Dice (Dice Similarity Coefficient) loss function, also known as the Dice similarity coefficient loss function, is derived from the Dice similarity coefficient and is used to measure the similarity between the predicted result and the true label. It is an evaluation indicator for image semantic segmentation tasks. The Dice loss function can alleviate the impact of the imbalance between the foreground and background areas in the image, allowing the model to pay more attention to the foreground during training. However, in the case of small targets such as segmented cells, the Dice loss function may show fluctuations and instability.

[0077] In order to solve this problem, the Focal loss function is used to enable the model to focus on the advantages of difficult-to-learn objects by reducing the weights of simple objects. The Dice loss function is combined with the Focal loss function by adding and weighted summation to obtain the loss function of the semantic segmentation model.

[0078] In one embodiment, the loss function of the semantic segmentation model is as follows: ; in, is the loss function of the semantic segmentation model, is the Dice loss function, is the Focal loss function.

[0079] The automatic cell segmentation method for high-content fluorescence microscopy images provided by the present invention can improve the performance of the semantic segmentation model by determining the loss function of the semantic segmentation model based on the Dice loss function and the focal loss function, so as to ultimately improve the accuracy and speed of automatic cell segmentation.

[0080] Figure 3 This is the second flow chart of the automatic cell segmentation method for high-content fluorescence microscopy images provided by the present invention, such as Figure 3As shown, taking the use of NUSeg to construct a semantic segmentation model and the use of a marker-based watershed algorithm to construct an image segmentation model as an example, overall, the automatic cell segmentation method for high-content fluorescence microscopy images provided in this embodiment includes a semantic segmentation model NUSeg, a two-stage conditional erosion, and a marker-based watershed algorithm. The semantic segmentation model NUSeg uses the Unet++ structure as the network backbone, uses Xception as the encoder, uses the scSE attention mechanism in each decoding layer to optimize spatial and channel details, uses a combination of the Dice loss function and the Focal loss function as the loss function, and uses the two-stage conditional erosion and the marker-based watershed algorithm as post-processing operations to ultimately achieve automatic cell instance segmentation of high-content fluorescence microscopy images.

[0081] Since the present embodiment is an algorithm designed and developed for the characteristics of high-content fluorescence microscopy images, it has achieved extremely high segmentation accuracy on multiple data sets, and has shown robustness and versatility for cells of different species (humans, mice, rats, etc.), different states (embryos, division, differentiation, toxicity, etc.), and different qualities (magnification, lighting quality, etc.) in imaging scenarios; moreover, the present embodiment has extremely high consistency in the segmentation between different staining channels of the same batch of cells, and is not affected by the different staining shape differences produced by various markers; finally, the semantic segmentation of the present embodiment is combined with post-processing to achieve instance segmentation of cells. When annotating data, it is only necessary to distinguish between the background and the foreground (i.e., the cells), without having to distinguish each cell, which greatly reduces the difficulty and cost of data annotation and improves the efficiency of training models.

[0082] In order to better illustrate the technical effect of low-cost data annotation achieved by the automatic cell segmentation method for high-content fluorescence microscopy images provided by the present invention during model training, an embodiment is provided below to illustrate the model training process involved in the automatic cell segmentation method for high-content fluorescence microscopy images provided by the present invention.

[0083] Figure 4 is an example of a model training image provided by the present invention, such as Figure 4 As shown in the figure, the high-content fluorescence microscopy image training data used in model training comes from the BBBC039 dataset, which contains 197 images such as Figure 4 (a) Figure 4 (b) Figure 4 (c) and Figure 4 (d) shows a high-content fluorescence microscopy image sample, and the corresponding manually annotated Figure 4 (e) Figure 4 (f) Figure 4 (g) and Figure 4(h) shows the mask image label. The size of each high-content fluorescence microscopy image sample is 696×520 pixels, and the format of all images is png. In the image preprocessing stage, two folders are created to store the original image and the mask image, named Images and Masks, respectively. At the same time, the size of the high-content fluorescence microscopy image sample and the corresponding mask image label is adjusted to 512×512.

[0084] The semantic segmentation model NUSeg is built with Pytorch and uses two NVIDIA V100 graphics cards to train the neural network. Before training, the weights pre-trained on the ImageNet dataset are used for initialization. The training process uses the Adam optimizer, with a learning rate of 1e-3, a batch size of 8, and the activation function of the last layer is Sigmoid, and the other layers use the ReLU activation function.

[0085] In order to avoid overfitting and enhance the robustness of the model, data augmentation techniques are used, including random rotation, flipping, and changing the hue, saturation, brightness, and contrast of the image. It should be noted that when performing data augmentation during training, the same operation needs to be performed on the high-content fluorescence microscopy image samples and the corresponding mask image labels to ensure consistency between the two.

[0086] The semantic segmentation model NUSeg outputs the segmented mask image. The model is trained for 100 rounds, and the loss function value and model weight of each round are recorded. After the training is completed, the model weight with the smallest loss function is saved as the final model. The output of the semantic segmentation model NUSeg is the semantic segmentation mask image. In the mask image label, the background label is 0 and the cell label is 1.

[0087] Further post-processing is performed to identify individual cells. The mask image output by the semantic segmentation model is subjected to a two-stage erosion operation. The threshold T1 of the coarse erosion stage is set to 200, and the threshold T2 of the fine erosion stage is set to 80. After the two-stage erosion operation, the area retained in the image is used as a seed to run the image segmentation model built using the marker-based watershed algorithm. Finally, each cell is identified and the cell's Figure 4 (i) Figure 4 (j) Figure 4 (k) and Figure 4 (l) shows an instance segmentation mask image, in which the background label is 0 and each cell is labeled from 1 to n in sequence.

[0088] Figure 5 is one of the structural schematic diagrams of the automatic cell segmentation device for high-content fluorescence microscopy images provided by the present invention, such as Figure 5As shown, the automatic cell segmentation device for high-content fluorescence microscopy images includes but is not limited to a semantic segmentation module 501 and an image segmentation module 502 .

[0089] The semantic segmentation module 501 is used to input the high-content fluorescence microscopy image into the semantic segmentation model to obtain a mask image output by the semantic segmentation model.

[0090] The image segmentation module 502 is used to input the mask image into an image segmentation model to obtain an instance segmentation mask image output by the image segmentation model.

[0091] It should be noted that the automatic cell segmentation device for high-content fluorescence microscopy images provided by the present invention can execute the automatic cell segmentation method for high-content fluorescence microscopy images described in any of the above embodiments during specific operation, which will not be described in detail in this embodiment.

[0092] The automatic cell segmentation device for high-content fluorescence microscopic images provided by the present invention adopts a two-stage segmentation strategy combining semantic segmentation and image segmentation based on the characteristics of high-content fluorescence microscopic images, such as independent channels, relatively single colors, fine segmentation granularity, and sensitivity to changes in tiny boundaries. The color channel and spatial channel of the high-content fluorescence microscopic images can be deeply decoupled, thereby improving the versatility of the cell segmentation method. Specifically, the high-content fluorescence microscopic images are first processed into mask images that only mark the foreground and background using a semantic segmentation model, and then the mask images are processed using an image segmentation model to obtain an instance segmentation mask image that finally marks each cell on the foreground and the background, which means that the training of the semantic segmentation model and the image segmentation model can be realized at a lower data annotation cost, so as to realize the instance segmentation of the high-content fluorescence microscopic images at a lower data annotation cost. In general, an automated and rapid high-content image cell segmentation algorithm can be realized with a smaller number of parameters and less computational effort, so as to help cell biologists promote large-scale analysis of high-content screening.

[0093] Figure 6 The second structural schematic diagram of the automatic cell segmentation device for high-content fluorescence microscopy images provided by the present invention is as follows: Figure 6 As shown, the automatic cell segmentation device for high-content fluorescence microscopy images also includes an image erosion module 601.

[0094] Among them, the image erosion module 601 is used to perform two-stage erosion processing on the mask image, including: when the area of ​​the first region to be segmented in the mask image is greater than the first threshold, iteratively eroding the first region to be segmented using a coarse erosion kernel until the area of ​​the first region to be segmented is less than or equal to the first threshold, thereby obtaining the mask image after the first stage of erosion; when the area of ​​the second region to be segmented in the mask image after the first stage of erosion is greater than the second threshold, iteratively eroding the second region to be segmented using a fine erosion kernel until the area of ​​the second region to be segmented is less than or equal to the second threshold, thereby completing the two-stage erosion processing of the mask image.

[0095] Figure 7 is a schematic diagram of the structure of the electronic device provided by the present invention, such as Figure 7 As shown, the electronic device may include: a processor (Processor) 710, a communication interface (Communications Interface) 720, a memory (Memory) 730 and a communication bus 740, wherein the processor 710, the communication interface 720, and the memory 730 communicate with each other through the communication bus 740. The processor 710 may call the logic instructions in the memory 730 to execute the automatic cell segmentation method for high-content fluorescence microscopy images provided in any of the above embodiments, and the automatic cell segmentation method for high-content fluorescence microscopy images includes but is not limited to the following steps: inputting the high-content fluorescence microscopy image into a semantic segmentation model to obtain a mask image output by the semantic segmentation model; inputting the mask image into an image segmentation model to obtain an instance segmentation mask image output by the image segmentation model.

[0096] In addition, the logic instructions in the above-mentioned memory 730 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on this understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art or the part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, ROM), random access memory (Random Access Memory, RAM), disk or optical disk and other media that can store program codes.

[0097] On the other hand, the present invention also provides a computer program product, which includes a computer program. The computer program can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the automatic cell segmentation method for high-content fluorescence microscopy images provided in any of the above embodiments. The automatic cell segmentation method for high-content fluorescence microscopy images includes but is not limited to the following steps: inputting the high-content fluorescence microscopy image into a semantic segmentation model to obtain a mask image output by the semantic segmentation model; inputting the mask image into an image segmentation model to obtain an instance segmentation mask image output by the image segmentation model.

[0098] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the automatic cell segmentation method for high-content fluorescence microscopy images provided in any of the above embodiments is implemented. The automatic cell segmentation method for high-content fluorescence microscopy images includes but is not limited to the following steps: inputting the high-content fluorescence microscopy image into a semantic segmentation model to obtain a mask image output by the semantic segmentation model; inputting the mask image into an image segmentation model to obtain an instance segmentation mask image output by the image segmentation model.

[0099] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0100] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0101] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. An automatic cell segmentation method for high-content fluorescence microscopy images, characterized in that: include: Inputting the high-content fluorescence microscopy image into a semantic segmentation model to obtain a mask image output by the semantic segmentation model; The mask image is input into an image segmentation model to obtain an instance segmentation mask image output by the image segmentation model.

2. The automatic cell segmentation method for high-content fluorescence microscopy images according to claim 1, characterized in that: Before inputting the mask image into the image segmentation model to obtain the instance segmentation mask image output by the image segmentation model, performing a two-stage erosion process on the mask image, including: When the area of ​​the first region to be segmented in the mask image is greater than the first threshold, iteratively eroding the first region to be segmented using a coarse erosion kernel until the area of ​​the first region to be segmented is less than or equal to the first threshold, thereby obtaining a mask image after first-stage erosion; When the area of ​​the second region to be segmented in the mask image after the first stage of erosion is greater than the second threshold, the second region to be segmented is iteratively eroded using a fine erosion kernel until the area of ​​the second region to be segmented is less than or equal to the second threshold, thereby completing the two-stage erosion processing of the mask image.

3. The automatic cell segmentation method for high-content fluorescence microscopy images according to claim 1, characterized in that: The semantic segmentation model is built based on the NUSeg neural network using the Unet++ architecture.

4. The automatic cell segmentation method for high-content fluorescence microscopy images according to claim 3, characterized in that: The encoder of the semantic segmentation model includes a first branch and a second branch; The first branch includes a first convolutional layer; The second branch includes a maximum pooling layer and two separable convolutional layers connected in sequence; The separable convolution layer includes a batch normalization layer, a depth-separable convolution layer and a ReLU activation function layer connected in sequence; The depth-wise separable convolutional layer includes a parallel convolutional layer, a splicing layer, and a second convolutional layer connected in sequence; The parallel convolutional layer includes three third convolutional layers connected in parallel.

5. The automatic cell segmentation method for high-content fluorescence microscopy images according to claim 4, characterized in that: The decoder of the semantic segmentation model adopts a spatial channel compression and excitation attention mechanism to compress and excite the feature map input to the decoder in both spatial and channel dimensions, thereby obtaining a feature map output to the decoder.

6. The automatic cell segmentation method for high-content fluorescence microscopy images according to claim 1, characterized in that: The loss function of the semantic segmentation model is determined based on the Dice loss function and the focal loss function.

7. An automatic cell segmentation device for high-content fluorescence microscopy images, characterized in that: include: A semantic segmentation module, used for inputting the high-content fluorescence microscopy image into the semantic segmentation model to obtain a mask image output by the semantic segmentation model; The image segmentation module is used to input the mask image into the image segmentation model to obtain the instance segmentation mask image output by the image segmentation model.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that: When the processor executes the computer program, the automatic cell segmentation method for high-content fluorescence microscopy images as described in any one of claims 1 to 6 is implemented.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the automatic cell segmentation method for high-content fluorescence microscopy images as claimed in any one of claims 1 to 6 is implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the automatic cell segmentation method for high-content fluorescence microscopy images as claimed in any one of claims 1 to 6 is implemented.

Images