A cell image segmentation method, device, electronic device and storage medium

Abstract

The present application provides a cell image segmentation method, device, electronic device and storage medium, the method comprising: obtaining an image to be segmented; inputting the image to be segmented into a preset region of interest recognition network to determine the region of interest, the region of interest only contains one complete cell; performing image cropping according to the region of interest to determine the image of interest; inputting the image of interest into a preset adversarial generation network to determine the cell boundary; performing contour search and extraction according to the cell boundary to determine the segmented image of each cell in the image to be segmented. The solution can solve the problem of segmenting highly overlapping cells in a complex background, and can obtain the image of each cell in the overlapping cells, with strong generalization and anti-interference capabilities.

Description

Technical Field

[0001] The present invention belongs to the technical field of image segmentation, and in particular relates to a cell image segmentation method, device, electronic equipment and storage medium. Background Art

[0002] Image segmentation of cervical cancer has always been a major difficulty in the field of cell images and medicine, and it is also a segmentation task with great research value. The difficulty in segmenting cervical cancer cell images lies in the complexity of cell images. Inflammatory cells, linear particles, impurities, etc. may appear in cell images, which greatly interferes with image segmentation and recognition. At the same time, the difficulty also lies in the fact that cervical cancer cell images contain a large number of overlapping cells. The low contrast of the segmentation boundaries of overlapping cells, coupled with the poor imaging effect caused by the film production process, make cervical cancer image segmentation extremely challenging. Image segmentation is involved in various fields, especially in the field of medical images, because the quality of image segmentation in the field of medical images directly determines the quality of the diagnosis results. Therefore, researchers around the world have developed many segmentation techniques, and the number of segmentation methods is almost the same as the number of segmentation problems. The difficulty of segmentation of different images is different, which mainly depends on the type of image to be segmented. The ideal situation is that the background is clear, the cells have no overlap, and the contrast between the cells and the background is obvious. For such images, a simple image threshold can be used to achieve accurate segmentation of cells and background. However, in practice, the contrast between cells and background is low, and there will be various impurities and irrelevant cells in the image. The most difficult thing to deal with is the high overlap between cells, which makes the segmentation task extremely difficult. Generative adversarial networks (GANs) are very useful in applications such as unsupervised representation learning or image-to-image translation. The segmentation task can be considered as an image conversion problem. For image generation, generative adversarial networks (GANs) provide a different approach. GANs are divided into two networks, namely the generator and the discriminator. The two structures compete with each other for learning. The goal of the discriminator is to classify the input image as a real image or a synthetic image, while the goal of the generator is to generate an image close to the real image so that the discriminator is fooled by it, that is, the discriminator has difficulty distinguishing the difference between the real image and the generated image.

[0003] Existing cell image segmentation algorithms are mainly divided into two categories: cell segmentation algorithms based on image processing and cell segmentation algorithms based on deep learning. Segmentation algorithms based on traditional methods have poor robustness, and the segmentation effect of cell images depends greatly on the quality of collected cell images; they have poor generalization ability, and it is difficult to find a set of suitable parameters that enable these segmentation methods to simultaneously segment cell images containing a variety of complex situations. In addition, image processing-based methods have difficulty in segmenting overlapping cells. Cell segmentation methods based on deep learning have better generalization than traditional segmentation methods. This is because the convolution structure can effectively capture image detail features and can perform cell segmentation in complex backgrounds. However, the current mainstream deep learning algorithms can only be used for the segmentation of single cells, and it is difficult to capture the boundary features of overlapping cells and cannot effectively segment overlapping cells. Summary of the invention

[0004] The purpose of the embodiments of this specification is to provide a cell image segmentation method, device, electronic device and storage medium.

[0005] To solve the above technical problems, the embodiments of the present application are implemented in the following ways:

[0006] In a first aspect, the present application provides a cell image segmentation method, the method comprising:

[0007] Obtain the image to be segmented;

[0008] The image to be segmented is input into a preset region of interest recognition network to determine the region of interest, which contains only one complete cell;

[0009] According to the region of interest, image cropping is performed to determine the image of interest;

[0010] The image of interest is input into the preset adversarial generative network to determine the cell boundaries;

[0011] According to the cell boundaries, contour search and extraction are performed to determine the segmentation image of each cell in the image to be segmented.

[0012] In one embodiment, the image to be segmented is input into a preset region of interest recognition network to determine the region of interest, including:

[0013] The image to be segmented is input into the backbone extraction network, and the backbone extraction network obtains the first effective feature layer and the second effective feature layer;

[0014] The second effective feature layer is input into the convolution layer for convolution to obtain the convolution feature;

[0015] The convolutional features are fused with the first effective feature layer from bottom to top to obtain fused features;

[0016] The fused features are input into a first feature detection head, and a first prediction box set is output, where the first prediction box set includes at least one first prediction box;

[0017] The convolution feature is input into a second feature detection head, and a second prediction box set is output, where the second prediction box set includes at least one second prediction box;

[0018] A region of interest is selected from the first prediction box set and the second prediction box set.

[0019] In one embodiment, the image to be segmented is input into a backbone extraction network, and the backbone extraction network obtains a first effective feature layer and a second effective feature layer, including:

[0020] The image to be segmented is input into the backbone extraction network after the first full pooling;

[0021] The backbone extraction network undergoes the second full pooling and the third full pooling before obtaining the first effective feature layer and the second effective feature layer, respectively.

[0022] In one embodiment, selecting a region of interest from the first prediction frame set and the second prediction frame set includes:

[0023] An improved non-maximum suppression algorithm is used to select a region of interest from the first prediction box set and the second prediction box set;

[0024] The improved non-maximum suppression algorithm includes:

[0025]

[0026] in, is the distance between the center points of the two prediction boxes, b gt is the box with the highest score, b is the currently selected box; ρ 2 is the distance; c is the diagonal length of the smallest box containing two predicted boxes; s i is the corresponding detection score; IoU is the intersection over union ratio; is the prediction box data; B i is the initial detection box list; ε is the threshold.

[0027] In one embodiment, the loss function of the region of interest recognition network is preset as:

[0028]

[0029] in, is the position loss function; loss obj is the confidence loss function; is the category loss function;

[0030] Among them, the category loss function for:

[0031]

[0032] Among them, α is the balance coefficient; y′ is the output of the activation function; γ is the factor.

[0033] In one of the embodiments, the preset adversarial generative network includes a discriminator and a generator;

[0034] The discriminator uses a full convolutional layer, which includes several convolutional layers and the final convolutional layer;

[0035] After passing through several convolutional layers, the image of interest is input into the final convolutional layer and an N*N matrix is ​​output. Each point in the N*N matrix represents the evaluation value of an area in the image of interest.

[0036] In one embodiment, the objective function of the preset adversarial generative network is:

[0037]

[0038] Among them, P data(x) is the real sample distribution, D(x) is the output value of the discriminator for the real sample; p z is the distribution of synthetic samples, D(G(z)) represents the output value of the discriminator for the synthetic samples; λ is the equilibrium constant; when λ>0, λL(G) is the L1 loss function:

[0039]

[0040] In a second aspect, the present application provides a cell image segmentation device, the device comprising:

[0041] An acquisition module, used for acquiring an image to be segmented;

[0042] The first determination module is used for inputting the image to be segmented into a preset region of interest recognition network to determine the region of interest, where the region of interest only contains one complete cell;

[0043] The second determination module is used to perform image cropping according to the region of interest to determine the image of interest;

[0044] The third determination module is used to input the image of interest into a preset adversarial generation network to determine the cell boundary;

[0045] The fourth determination module is used to search and extract the outline according to the cell boundary to determine the segmented image of each cell in the image to be segmented.

[0046] In a third aspect, the present application provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the cell image segmentation method of the first aspect is implemented.

[0047] In a fourth aspect, the present application provides a readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the cell image segmentation method of the first aspect.

[0048] It can be seen from the technical solution provided in the above embodiments of this specification that:

[0049] A preset region of interest recognition network is used to determine the location of cells and the region of interest for cell segmentation to guide subsequent segmentation tasks. Then a preset generative adversarial network is used to segment overlapping cells. This method can solve the problem of segmenting highly overlapping cells in complex backgrounds and obtain images of each cell in overlapping cells. It has strong generalization and anti-interference capabilities.

[0050] The preset region of interest recognition network is adopted, the YOLOv4 network structure is simplified, and a new loss function is added, which can greatly improve the detection rate while ensuring the detection accuracy.

[0051] The preset adversarial generative network used improves the generator and discriminator structures of the original network, thereby enhancing the network's generation capability and making the training process smoother, thereby making cell segmentation more accurate. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] In order to more clearly illustrate the embodiments of this specification or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this specification. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0053] Figure 1 A schematic diagram of the process of the cell image segmentation method provided in this application;

[0054] Figure 2 A schematic diagram of the structure of the optimized YOLOv4 network provided for this application;

[0055] Figure 3 A training flow chart of a preset region of interest recognition network provided for this application;

[0056] Figure 4 A training flow chart of the preset adversarial generative network provided for this application;

[0057] Figure 5 A schematic diagram of the structure of the cell image segmentation device provided in this application;

[0058] Figure 6 A schematic diagram of the structure of the electronic device provided in this application. DETAILED DESCRIPTION

[0059] In order to enable those skilled in the art to better understand the technical solutions in this specification, the technical solutions in the embodiments of this specification will be clearly and completely described below in conjunction with the drawings in the embodiments of this specification. Obviously, the described embodiments are only part of the embodiments of this specification, not all of the embodiments. Based on the embodiments in this specification, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of this specification.

[0060] In the following description, specific details such as specific system structures, technologies, etc. are provided for the purpose of illustration rather than limitation, so as to provide a thorough understanding of the embodiments of the present application. However, it should be clear to those skilled in the art that the present application may also be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to prevent unnecessary details from obstructing the description of the present application.

[0061] It will be apparent to those skilled in the art that various modifications and variations may be made to the specific embodiments of the present application description without departing from the scope or spirit of the present application. Other embodiments derived from the present application description will be apparent to those skilled in the art. The present application description and examples are merely exemplary.

[0062] The words “include,” “including,” “have,” “contain,” etc. used in this document are open-ended terms, meaning including but not limited to.

[0063] Unless otherwise specified, "parts" in this application are calculated by mass.

[0064] The present invention is further described in detail below with reference to the accompanying drawings and embodiments.

[0065] Reference Figure 1 , which shows a flow chart of the cell image segmentation method applicable to the embodiment of the present application.

[0066] like Figure 1 As shown, the cell image segmentation method may include:

[0067] S110: Obtain an image to be segmented.

[0068] Specifically, the image to be segmented may be various cell images that need to be segmented, or may be other images that need to be segmented. In the present application, the image to be segmented is described by taking a cervical cell image as an example.

[0069] If the generative idea is used directly to segment cells, although a clearer cell segmentation boundary can be formed in the entire cell image, the generated image is limited to the better segmentation effect from a human perspective, and it is difficult for the computer to match the segmentation boundary with the cell one-to-one. In other words, the direct use of the generative segmentation method does not complete the single cell extraction in the true sense. Therefore, before using generative segmentation, the present application uses a region of interest recognition network to determine the region of interest, that is, target recognition, and extracts each cell separately. Using the generative idea in the single cell image, the boundary and the cell can be matched one-to-one, and the single cell image can be directly extracted, which is convenient for the subsequent medical analysis of the single cell image.

[0070] S120, the image to be segmented is input into a preset region of interest recognition network to determine the region of interest, where the region of interest only contains one complete cell.

[0071] Specifically, the preset region of interest network is a pre-trained neural network. When the preset region of interest recognition network inputs the image to be segmented, the region of interest can be output.

[0072] The preset region of interest network uses the optimized YOLOv4 network. The structural diagram of the optimized YOLOv4 network is as follows Figure 2 shown.

[0073] The FPN structure used in the optimized YOLOv4 network extracts two effective feature layers obtained in the backbone network (respectively Figure 2 Resnet (38, 38, 256) and Conv output) perform bottom-up feature fusion ( Figure 2 The Concat layer performs feature fusion), and then stacks and convolves with the effective feature layer of the previous scale ( Figure 2 Concat layers are stacked and convolved).

[0074] In the prediction part, the existing YOLOv4 network is equipped with three detection heads, the purpose of which is to predict targets of different scales. Large objects are predicted using a small-scale feature layer, and small objects with fewer features are predicted using a large feature layer. A large number of cervical cancer cell data sets have been observed, and the size of cells is not very different, so this application is designed to reduce the output of one detection head. This application optimizes the YOLOv4 network to only set two feature detection heads (the first feature detection head and the second feature detection head, corresponding to Figure 2There are two Yolo Heads in Cell_yolo, which are responsible for detecting targets of different scales respectively. Similar to the design of the existing YOLOv4 network, three different prior frames are preset in the feature map of each detection head to predict the target object and generate a prediction frame. Since three detection heads are used in YOLOv4, there are a total of 9 prior frames. In Cell_yolo, there are two detection heads with a total of 6 prior frames. The preset prior frames can screen out representative preset frames in advance, making it easier to converge during training.

[0075] In addition, in order to ensure that the simplification of the network structure does not lead to a decrease in detection accuracy, this application uses a full pooling operation for the improvement of the pooling operation in the network. Although the original maximum pooling operation can reduce the size of the feature map to extract features, this also means that local information will be lost. The boundary features of overlapping cells are extremely important. Excessive use of pooling operations will have an adverse effect on the final result. Therefore, the optimization of the YOLOv4 network adds a full pooling operation. The use of full pooling will hardly add a burden to the algorithm's operating efficiency, and can greatly enhance the algorithm's feature extraction capabilities. The operation process of full pooling is: for the input image, take a value for every other pixel value, so that the original image becomes four images, and the four images contain all the information of the original image. Using the concat splicing method, the obtained image is four times larger than the number of channels of the original image, and then the obtained image is convolved, and finally a double down-sampling feature map with no information loss can be obtained. The full pooling operation can not only achieve the purpose of pooling to reduce the size of the feature map, but also greatly preserve the features of the image. This application optimizes the YOLOv4 network structure and designs full pooling operations in three places to replace maximum pooling. First, full pooling is performed when the image to be segmented enters the backbone extraction network, and then full pooling is added before the two effective feature layers prepared for feature fusion.

[0076] In one embodiment, the image to be segmented is input into a preset region of interest recognition network to determine the region of interest, including:

[0077] The image to be segmented is input into the backbone extraction network, and the backbone extraction network obtains the first effective feature layer and the second effective feature layer;

[0078] The second effective feature layer is input into the convolution layer for convolution to obtain the convolution feature;

[0079] The convolutional features are fused with the first effective feature layer from bottom to top to obtain fused features;

[0080] The fused features are input into a first feature detection head, and a first prediction box set is output, where the first prediction box set includes at least one first prediction box;

[0081] The convolution feature is input into a second feature detection head, and a second prediction box set is output, where the second prediction box set includes at least one second prediction box;

[0082] A region of interest is selected from the first prediction box set and the second prediction box set.

[0083] The image to be segmented is input into the backbone extraction network, and the backbone extraction network obtains the first effective feature layer and the second effective feature layer, including:

[0084] The image to be segmented is input into the backbone extraction network after the first full pooling;

[0085] The backbone extraction network undergoes the second full pooling and the third full pooling before obtaining the first effective feature layer and the second effective feature layer, respectively.

[0086] After obtaining the prediction frame, the non-maximum suppression (NMS) method is used to remove duplicate frames, that is, to remove redundant prediction frames and reduce false detection. In the NMS method, if the IoU between the current frame and the frame with the highest score is greater than the threshold, it will be deleted directly. However, in actual situations, the two objects may be very close, resulting in the IoU of the detection frame being too large, and one target frame will be mistakenly deleted. This situation is particularly prominent in images with many overlapping cells. In other words, the NMS method only considers the intersection and union information IoU to remove duplicate frames, and it is very easy to mistakenly delete the detection frames of overlapping cells, that is, the NMS method is not suitable for this application.

[0087] When the image to be segmented is a cervical cell image, and there are many overlapping cells in the cervical cell image, in order to prevent the problem of accidental deletion of overlapping cell detection frames, when selecting the region of interest from the first prediction frame set and the second prediction frame set, improved non-maximum suppression (NMS) can be used.

[0088] In one embodiment, improving the non-maximum suppression algorithm includes:

[0089]

[0090] in, is the distance between the center points of the two prediction boxes, b gt is the box with the highest score, b is the currently selected box; ρ 2 is the distance; c is the diagonal length of the smallest box containing two predicted boxes; s i is the corresponding detection score; IoU is the intersection over union ratio; is the prediction box data; B i is the initial detection box list; ε is the threshold.

[0091] In order to solve the problem of category imbalance, the category loss function is solved by using Focal loss. The original cross entropy function is modified. The formula of the binary cross entropy function is as follows:

[0092]

[0093] The output of the activation function is a y′ between 0 and 1. The goal of the neural network is to minimize the loss function. In the above formula, if you want to make the loss function smaller, then for positive samples, the probability value of the activation function output is larger, indicating that the network model is more certain that the sample is a positive sample; similarly, for negative samples, the smaller the probability value of the activation function output is, the higher the probability that the model believes that the sample is a negative sample. The formula of Focal loss is as follows:

[0094]

[0095] The improvement of Focal loss may seem small, but it is very helpful for training. Focal loss adds a γ factor to the original one. When γ>0, it will reduce the loss of easy-to-classify samples and make the model pay more attention to difficult samples. Let γ=2, and assume that the output probability of the activation function of the positive sample is 0.9. At this time, (1-y′) γ The value of will be very small, and the loss function value will become very small. On the contrary, if the probability value of the positive sample output by the activation function is 0.2, it means that although it is a positive sample, the model believes that it is likely to be a negative sample. This sample is a difficult sample and should be trained more. At this time (1-y′) γ The value of is very large, resulting in an excessively large value of the loss function. The model will increase the learning intensity and pay more attention to difficult samples. Similarly, for negative samples, the result of an output probability value of 0.1 is much smaller than the loss value of 0.8. This slight change in the loss function reduces the impact of simple samples, thus solving the problem of simple and difficult samples. In addition, in order to solve the problem of balancing positive and negative samples, the balance coefficient α is added. The formula of the final category loss function is as follows:

[0096]

[0097] The loss function for optimizing the YOLOv4 network consists of three parts: the confidence loss function loss obj , the confidence loss function still continues the cross entropy loss function in the YOLOv4 network; the second is the position loss function The position loss function uses the DIoU loss function; the third is the category loss function, which uses an improved loss function based on Focal loss. The loss function for optimizing the YOLOv4 network is shown in the following formula.

[0098]

[0099] Using the above improved algorithm to train the neural network, the training process converges faster and the location of the region of interest is accurate.

[0100] like Figure 3 The figure shows the training process of the preset region of interest recognition network: first, the cell image data is manually annotated. Considering the characteristics of medical data that are difficult to obtain, data enhancement is performed to expand the data volume. Data enhancement uses simple operations such as image translation, rotation, and color transformation. After that, the data is input into the network model and the initial parameters are selected for training. Whether the training is completely completed depends on the recognition effect of the region of interest. The training will evaluate the training effect. If the requirements are not met, the initial parameters need to be changed and re-trained until the training effect meets the requirements.

[0101] It should be noted that when manually annotating cell image data as mentioned above, it is not necessary to determine the type of cells, but only to determine the difference between the cells and the background. Use the rectangular box in the labelimg software for annotation. Each rectangular box must contain the complete cell image and only one complete cell can be annotated in each box. Other cells can exist, but two complete cells are not allowed to exist in the annotated box. Even overlapping cells must be annotated separately. This is the region of interest defined in this application. There is one and only one complete cell image in the region of interest.

[0102] S130: Perform image cropping according to the region of interest to determine an image of interest.

[0103] Specifically, the region of interest obtained in step S120 is calibrated to form coordinate values ​​of four points having a calibrated rectangular frame, and the image is cropped according to the coordinate position of each calibrated frame to form a cell image containing only a complete single cell image, which is the image of interest.

[0104] S140, the image of interest is input into a preset adversarial generative network to determine the cell boundary.

[0105] Specifically, a preset generative adversarial network is used to segment the image of interest and generate cell boundaries.

[0106] The preset adversarial generative network is an improvement on the existing adversarial generative network. Among them, the design of the discriminator is different from that of the original adversarial generative network. The design of the discriminator of the original adversarial generative network is to output only one evaluation value, which is an evaluation of the entire image generated by the generator. In the preset adversarial generative network, the discriminator is designed in the form of full convolution. After the image passes through various convolution layers, it will not be input into the fully connected layer or activation function, but the input is mapped into an N*N matrix using convolution. The matrix is ​​equivalent to the final evaluation value in the original adversarial generative network to evaluate the generated image of the generator. Each point in the N*N matrix represents the evaluation value of a small area in the original image, which is the application of the "receptive field". Originally, one value was used to measure the entire image, and now an N*N matrix is ​​used to evaluate the entire image. Similarly, the label also needs to be set to an N*N format, so that the loss calculation can be performed. Obviously, the preset adversarial generative network in this application can focus on more areas. The design of the original discriminator is similar to "one-man show", and a value generated by the discriminator is used to evaluate the generation of the image. The improved discriminator is more like a "divide and conquer method", generating evaluation values ​​of multiple area blocks to evaluate the image, thereby guiding the subsequent image generation to be more efficient and accurate.

[0107] In order to achieve more accurate segmentation, the objective function of the original adversarial generative network is improved to achieve more accurate generation of cell boundary images. The adversarial generative network can be seen as a teacher-student relationship, with the generator G being the student and the discriminator D being the teacher. The discriminator can identify the difference between the "student image" and the real image, and the generator will continuously update parameters under the guidance of the "teacher" to surpass the "teacher's" judgment.

[0108] The objective function of the preset adversarial generation network is:

[0109]

[0110] Among them, P data(x) is the real sample distribution, D(x) is the output value of the discriminator for the real sample; p z is the synthetic sample distribution, and D(G(z)) represents the output value of the discriminator for the synthetic sample.

[0111] In order to ensure that the generated image distribution can converge to the real image distribution faster and ensure that there is not much difference between the real image and the image synthesized by the generator G, a part similar to the regularization term is added to the loss function. Where λ is the equilibrium constant. When λ>0, λL(G) can be considered as the L1 loss function, that is:

[0112] It directly reflects the gap between real samples and generated samples.

[0113] The generator attempts to generate images that are indistinguishable from real images, and the discriminator attempts to more accurately distinguish the difference between the generated images and the real images. Consistent with the original GAN ​​training, the entire training process is completed by alternately optimizing the generator and discriminator objective functions, and the ideal goal of the entire training is to achieve Nash equilibrium. In this function, the addition of the regularization term ensures that the generator can converge to the ideal area faster during training and capture the distribution of real images faster, thereby speeding up the entire training process. Using the above improved algorithm to train the neural network, the training process converges faster and the boundary generation is more accurate.

[0114] In one embodiment, the preset adversarial generation network includes a discriminator and a generator;

[0115] The discriminator uses a full convolutional layer, which includes several convolutional layers and the final convolutional layer;

[0116] After passing through several convolutional layers, the image of interest is input into the final convolutional layer and an N*N matrix is ​​output. Each point in the N*N matrix represents the evaluation value of an area in the image of interest.

[0117] In one embodiment, the objective function of the preset adversarial generative network is:

[0118]

[0119] Among them, P data(x) is the real sample distribution, D(x) is the output value of the discriminator for the real sample; p z is the distribution of synthetic samples, D(G(z)) represents the output value of the discriminator for the synthetic samples; λ is the equilibrium constant; when λ>0, λL(G) is the L1 loss function:

[0120]

[0121] like Figure 4 The figure shows the training process of the preset adversarial generative network: first, the data is manually annotated. Considering the difficulty in obtaining medical data, data enhancement is performed to expand the data volume. Data enhancement uses simple operations such as image translation, rotation, and color transformation. Since the cell segmentation network is designed based on the adversarial generative network, it is necessary to refer to the gap between the original image and the labeled image during its training process to complete the training. Therefore, both the original data and the annotated data need to be input into the network model. After the initial parameters are selected in the network model, training is carried out. Whether the training is completely completed depends on the effect of cell segmentation. The training will evaluate the training effect. If the requirements are not met, the initial parameters need to be changed and re-trained until the training effect meets the requirements.

[0122] It should be noted that when manually annotating data, the PS tool can be used to annotate the data set. It is necessary to manually draw the cell boundary along the image boundary line. Since the image of interest contains only one complete cell, it is sufficient to use PS to draw the complete cell boundary.

[0123] S150 , searching and extracting contours according to cell boundaries to determine a segmented image for each cell in the image to be segmented.

[0124] The cell image segmentation method provided in the embodiment of the present application uses a preset region of interest recognition network to determine the position of cells, determine the region of interest for cell segmentation, guide subsequent segmentation tasks, and then uses a preset adversarial generative network to segment overlapping cells. This method can solve the problem of segmenting highly overlapping cells in complex backgrounds, and can obtain images of each cell in the overlapping cells, and has strong generalization and anti-interference capabilities.

[0125] The preset region of interest recognition network adopted in this application simplifies the YOLOv4 network structure and adds a new loss function, which can greatly improve the detection rate while ensuring the detection accuracy.

[0126] The preset adversarial generative network adopted in this application improves the generator and discriminator structures of the original network, thereby improving the generation ability of the network and making the training process smoother, thereby making cell segmentation more accurate.

[0127] Reference Figure 5 , which shows a schematic structural diagram of a cell image segmentation device described according to an embodiment of the present application.

[0128] like Figure 5 As shown, the cell image segmentation device 500 may include:

[0129] An acquisition module 510 is used to acquire an image to be segmented;

[0130] The first determination module 520 is used for inputting the image to be segmented into a preset region of interest recognition network to determine the region of interest, where the region of interest only contains one complete cell;

[0131] A second determination module 530 is used to perform image cropping according to the region of interest to determine an image of interest;

[0132] The third determination module 540 is used to input the image of interest into a preset adversarial generation network to determine the cell boundary;

[0133] The fourth determination module 550 is used to search and extract the outline according to the cell boundary, and determine the segmented image of each cell in the image to be segmented.

[0134] Optionally, the first determining module 520 is further configured to:

[0135] The image to be segmented is input into the backbone extraction network, and the backbone extraction network obtains the first effective feature layer and the second effective feature layer;

[0136] The second effective feature layer is input into the convolution layer for convolution to obtain the convolution feature;

[0137] The convolutional features are fused with the first effective feature layer from bottom to top to obtain fused features;

[0138] The fused features are input into a first feature detection head, and a first prediction box set is output, where the first prediction box set includes at least one first prediction box;

[0139] The convolution feature is input into a second feature detection head, and a second prediction box set is output, where the second prediction box set includes at least one second prediction box;

[0140] A region of interest is selected from the first prediction box set and the second prediction box set.

[0141] Optionally, the first determining module 520 is further configured to:

[0142] The image to be segmented is input into the backbone extraction network after the first full pooling;

[0143] The backbone extraction network undergoes the second full pooling and the third full pooling before obtaining the first effective feature layer and the second effective feature layer, respectively.

[0144] Optionally, the first determining module 520 is further configured to:

[0145] An improved non-maximum suppression algorithm is used to select a region of interest from the first prediction box set and the second prediction box set;

[0146] The improved non-maximum suppression algorithm includes:

[0147]

[0148] in, is the distance between the center points of the two prediction boxes, b gt is the box with the highest score, b is the currently selected box; ρ 2 is the distance; c is the diagonal length of the smallest box containing two predicted boxes; s i is the corresponding detection score; IoU is the intersection over union ratio; is the prediction box data; B i is the initial detection box list; ε is the threshold.

[0149] Optionally, the loss function of the region of interest recognition network is preset as:

[0150]

[0151] in, is the position loss function; loss obj is the confidence loss function; is the category loss function;

[0152] Among them, the category loss function for:

[0153]

[0154] Among them, α is the balance coefficient; y′ is the output of the activation function; γ is the factor.

[0155] Optionally, the preset adversarial generation network includes a discriminator and a generator;

[0156] The discriminator uses a full convolutional layer, which includes several convolutional layers and the final convolutional layer;

[0157] After passing through several convolutional layers, the image of interest is input into the final convolutional layer and an N*N matrix is ​​output. Each point in the N*N matrix represents the evaluation value of an area in the image of interest.

[0158] Optionally, the objective function of the preset adversarial generation network is:

[0159]

[0160] Among them, P data(x) is the real sample distribution, D(x) is the output value of the discriminator for the real sample; p z is the distribution of synthetic samples, D(G(z)) represents the output value of the discriminator for the synthetic samples; λ is the equilibrium constant; when λ>0, λL(G) is the L1 loss function:

[0161]

[0162] The cell image segmentation device provided in this embodiment can execute the embodiment of the above method. Its implementation principle and technical effect are similar and will not be described in detail here.

[0163] Figure 6 FIG. 1 is a schematic diagram of the structure of an electronic device provided by an embodiment of the present invention. Figure 6 As shown, a structural schematic diagram of an electronic device 300 suitable for implementing an embodiment of the present application is shown.

[0164] like Figure 6As shown, the electronic device 300 includes a central processing unit (CPU) 301, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 302 or a program loaded from a storage part 308 into a random access memory (RAM) 303. In the RAM 303, various programs and data required for the operation of the device 300 are also stored. The CPU 301, the ROM 302, and the RAM 303 are connected to each other via a bus 304. An input / output (I / O) interface 305 is also connected to the bus 304.

[0165] The following components are connected to the I / O interface 305: an input section 306 including a keyboard, a mouse, etc.; an output section 307 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 308 including a hard disk, etc.; and a communication section 309 including a network interface card such as a LAN card, a modem, etc. The communication section 309 performs communication processing via a network such as the Internet. A drive 310 is also connected to the I / O interface 305 as needed. A removable medium 311, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 310 as needed, so that a computer program read therefrom is installed into the storage section 308 as needed.

[0166] In particular, according to the embodiments of the present disclosure, the above reference Figure 1 The described process can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program tangibly contained on a machine-readable medium, and the computer program includes program code for performing the above-mentioned cell image segmentation method. In such an embodiment, the computer program can be downloaded and installed from a network through the communication part 309, and / or installed from a removable medium 311.

[0167] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present invention. In this regard, each box in the flow chart or block diagram can represent a module, a program segment, or a part of the code, and the aforementioned module, program segment, or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flow chart, and the combination of the boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0168] The units or modules involved in the embodiments described in the present application may be implemented by software or hardware. The units or modules described may also be arranged in a processor. The names of these units or modules do not constitute limitations on the units or modules themselves in certain circumstances.

[0169] The systems, devices, modules or units described in the above embodiments may be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer. Specifically, the computer may be, for example, a personal computer, a laptop, a mobile phone, a smart phone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or a combination of any of these devices.

[0170] As another aspect, the present application further provides a storage medium, which may be the storage medium included in the aforementioned device in the above embodiment; or may be a storage medium that exists independently and is not assembled into the device. The storage medium stores one or more programs, and the aforementioned programs are used by one or more processors to execute the cell image segmentation method described in the present application.

[0171] Storage media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer-readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer-readable media does not include temporary computer-readable media (transitory media), such as modulated data signals and carrier waves.

[0172] It should be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of further restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0173] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

Claims

1. A cell image segmentation method, characterized in that: The method comprises: Obtain the image to be segmented; The image to be segmented is input into a preset region of interest recognition network to determine the region of interest, including: The image to be segmented is input into the backbone extraction network after first full pooling; The backbone extraction network undergoes a second full pooling and a third full pooling respectively before obtaining the first effective feature layer and the second effective feature layer; The second effective feature layer is input into the convolution layer for convolution to obtain convolution features; The convolutional features are fused with the first effective feature layer from bottom to top to obtain fused features; The fused feature is input into a first feature detection head, and a first prediction box set is output, where the first prediction box set includes at least one first prediction box; The convolution feature is input into a second feature detection head, and a second prediction box set is output, where the second prediction box set includes at least one second prediction box; Using an improved non-maximum suppression algorithm, selecting the region of interest from the first prediction box set and the second prediction box set, wherein the region of interest only includes one complete cell; The improved non-maximum suppression algorithm comprises: in, is the distance between the center points of the two prediction boxes, b gt is the box with the highest score, b is the currently selected box; ρ 2 is the distance; c is the diagonal length of the smallest box containing two predicted boxes; s i is the corresponding detection score; IoU is the intersection over union ratio; is the prediction box data; B i is the initial detection box list; ε is the threshold; The loss function of the preset region of interest recognition network is: in, is the position loss function; loss obj is the confidence loss function; is the category loss function; Among them, the category loss function for: Among them, α is the balance coefficient; y′ is the output of the activation function; γ is the factor; Perform image cropping according to the region of interest to determine an image of interest; The image of interest is input into a preset adversarial generation network to determine the cell boundary; The preset adversarial generation network includes a discriminator and a generator; The discriminator adopts a full convolution layer, and the full convolution layer includes several convolution layers and a final convolution layer; The image of interest is input into the last convolution layer after passing through the several convolution layers, and an N*N matrix is ​​output, wherein each point in the N*N matrix represents an evaluation value of a region in the image of interest; The objective function of the preset adversarial generation network is: Among them, P data(x) is the real sample distribution, D(x) is the output value of the discriminator for the real sample; p z is the distribution of synthetic samples, D(G(z)) represents the output value of the discriminator for the synthetic samples; λ is the equilibrium constant; when λ>0, λL(G) is the L1 loss function: According to the cell boundary, contour search and extraction are performed to determine the segmented image of each cell in the image to be segmented.

2. A cell image segmentation device, characterized in that: The device comprises: An acquisition module, used for acquiring an image to be segmented; The first determination module is used for the image to be segmented to input a preset region of interest recognition network, including: the image to be segmented is input into a backbone extraction network after the first full pooling; the backbone extraction network is respectively subjected to the second full pooling and the third full pooling before obtaining the first effective feature layer and the second effective feature layer; the second effective feature layer is input into the convolution layer for convolution to obtain convolution features; the convolution features are fused with the first effective feature layer from bottom to top to obtain fusion features; the fusion features are input into the first feature detection head to output a first prediction box set, the first prediction box set includes at least one first prediction box; the convolution features are input into the second feature detection head to output a second prediction box set, the second prediction box set includes at least one second prediction box; the region of interest is selected from the first prediction box set and the second prediction box set by using an improved non-maximum suppression algorithm, and the region of interest only contains one complete cell; the improved non-maximum suppression algorithm includes: in, is the distance between the center points of the two prediction boxes, b gt is the box with the highest score, b is the currently selected box; ρ 2 is the distance; c is the diagonal length of the smallest box containing two predicted boxes; s i is the corresponding detection score; IoU is the intersection over union ratio; is the prediction box data; B i is the initial detection box list; ε is the threshold; The loss function of the preset region of interest recognition network is: in, is the position loss function; loss obj is the confidence loss function; is the category loss function; Among them, the category loss function for: Among them, α is the balance coefficient; y′ is the output of the activation function; γ is the factor; A second determination module is used to perform image cropping according to the region of interest to determine an image of interest; The third determination module is used for inputting the image of interest into a preset adversarial generation network to determine the cell boundary; the preset adversarial generation network includes a discriminator and a generator; the discriminator uses a full convolution layer, and the full convolution layer includes several convolution layers and a final convolution layer; the image of interest passes through the several convolution layers and then inputs the final convolution layer, outputting an N*N matrix, where each point in the N*N matrix represents an evaluation value of a region in the image of interest; the objective function of the preset adversarial generation network is: Among them, P data(x) is the real sample distribution, D(x) is the output value of the discriminator for the real sample; p z is the distribution of synthetic samples, D(G(z)) represents the output value of the discriminator for the synthetic samples; λ is the equilibrium constant; when λ>0, λL(G) is the L1 loss function: The fourth determination module is used to perform contour search and extraction according to the cell boundary to determine the segmented image of each cell in the image to be segmented.

3. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the cell image segmentation method as claimed in any one of claims 1 is implemented.

4. A readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the cell image segmentation method as claimed in any one of claims 1 is implemented.

Images