An automated method for segmenting leukocytes in bone marrow cells

Abstract

The application provides a method for automatically segmenting white blood cells in bone marrow cells, comprising the following steps: S1, extracting obvious cells CellGray from an input bone marrow cell rgb image through a convolutional neural network; S2, binarizing the bone marrow cell image CellGray to obtain a white blood cell nucleus binary image; S3, performing an opening and closing operation on the white blood cell nucleus binary image in the aspect of image morphology to obtain WhiteCellMorph, and performing a watershed algorithm operation on WhiteCellMorph to obtain a white blood cell nucleus segmentation image Leukocytenucleus; S4, shielding corresponding numbers in the CellGray data obtained in step S1 by using the Leukocytenucleus obtained in step S3, and obtaining a white blood cell membrane Leukocytemembrane by using an Otsu method and an opening and closing operation in the aspect of image morphology; and S5, combining the data obtained in steps S3 and S4 to obtain a segmented complete white blood cell image, extracting an accurate position of the white blood cell, and generating a standard data set. The application can effectively improve the accuracy of overall positioning and segmentation of white blood cell images, and generate a data set, thereby facilitating the development of artificial intelligence for subsequent white blood cell type recognition.

Description

Technical Field

[0001] This invention relates to the field of medical pathology analysis technology, specifically to an automated method for segmenting leukocytes in bone marrow cells. Background Technology

[0002] Leukemia is a malignant blood disease caused by epigenetic and genetic abnormalities in white blood cells due to various pathological factors both in vivo and in vitro, leading to hematopoietic transformation. Bone marrow aspiration is an indispensable examination for leukemia. A typical bone marrow smear image consists of white blood cells, red blood cells, platelets, and a background. Pathologists observe the types and quantities of white blood cells in the bone marrow smear under a microscope to provide diagnostic evidence for various types of leukemia. This is a very complex, tedious, and time-consuming task, and is easily affected by subjective factors.

[0003] Today, with the rapid development of computer-aided methods and the upgrading of related computing hardware, it has become possible for computers to automatically analyze and assist in diagnosis. Computer-aided diagnosis can not only simulate the diagnostic process of pathologists by first extracting and locating white blood cells in complex scenes, but also identify the located white blood cells.

[0004] Currently, cell segmentation typically employs instance segmentation networks, such as the Mask R-CNN segmentation model. However, these networks lack sufficient accuracy in cell edge segmentation, making them ill-suited for cell segmentation tasks in densely populated environments. To address this inaccurate edge segmentation, some studies have added an edge segmentation loss to the existing loss function. This loss extracts the boundaries between the predicted and labeled results, calculating the difference between the predicted and labeled boundaries, thus improving edge segmentation accuracy by reducing this loss. However, this approach is affected by the performance of the boundary extraction algorithm; the worse the boundary extraction, the greater the computational error in the edge segmentation loss. Especially in densely populated cell scenarios, cells are compressed and deformed, resulting in irregular cell boundaries and deteriorating boundary extraction performance, which negatively impacts the training of the segmentation model. In addition, some works extract the boundaries of the prediction results when calculating the loss and strengthen the segmentation loss of each pixel on the boundary. On the one hand, the above methods are also affected by the performance of the boundary extraction algorithm and are difficult to adapt well to dense cell segmentation scenarios. On the other hand, the approach of only strengthening the segmentation loss of pixels on the boundary is difficult to solve the classification errors of some signal points outside and inside the boundary. The classification errors of signal points will lead to a decrease in cell segmentation effect and have an adverse impact on downstream tasks.

[0005] In the prior art, Chinese patent application CN110060229A discloses an automatic cell localization and segmentation method for bone marrow leukocytes, including extracting the bone marrow leukocyte image WhiteCellGray from the input bone marrow RGB image, and binarizing the WhiteCellGray image using the Otsu's method to obtain a binary image WhiteCellBW. This method extracts the bone marrow leukocyte channel image from the specimen image through color deconvolution; then, it achieves segmentation and localization of bone marrow leukocytes through operations such as binarization, hole filling, morphological smoothing, and watershed segmentation. Color deconvolution can fundamentally eliminate the adverse effects of mature red blood cells on the subsequent segmentation and recognition of bone marrow leukocytes, thereby improving the accuracy of bone marrow leukocyte segmentation and localization; while the watershed segmentation can effectively separate adherent cells, reducing the missed detection rate of cells. However, this method is computationally complex, and using only one binarization process cannot effectively improve the accuracy of localization and segmentation, resulting in low accuracy and low operational efficiency in the cell segmentation process. Moreover, it can only be used for one type of white blood cell (white blood cells without cell membranes), so improvements are needed. Summary of the Invention

[0006] To address the shortcomings of existing technologies, this invention proposes an automatic segmentation method for white blood cells in bone marrow cells. It utilizes a convolutional neural network to extract features of bone marrow cells from images, enabling the processing of bone marrow cell data acquired through various operations. By employing multiple binarization processes to independently process the white blood cell nuclear and membrane images, and then superimposing them to obtain the overall white blood cell image, the accuracy of overall white blood cell image localization and segmentation can be effectively improved. Furthermore, it can generate datasets, facilitating the subsequent development of artificial intelligence for white blood cell type identification.

[0007] This invention provides an automated method for segmenting leukocytes in bone marrow cells, comprising the following steps:

[0008] S1. Extract obvious cell colors from the input bone marrow cell RGB image using a convolutional neural network;

[0009] S2. Binarize the bone marrow cell image CellGray to obtain a binary image of the leukocyte nucleus;

[0010] S3. Perform image morphology opening and closing operations on the binary image of the white blood cell nucleus to obtain WhiteCellMorph, and perform watershed algorithm operation on WhiteCellMorph to obtain the complete segmentation map of the white blood cell nucleus, Leukocytenucleus.

[0011] S4. In the CellGray data obtained in step S1, the corresponding number of Leukocytenucleus obtained in step S3 is masked out, and then the Leukocyte membrane is obtained by using the Otsu's method and the opening and closing operation in image morphology.

[0012] S5. Combine the data obtained in steps S3 and S4 to obtain a segmented complete white blood cell image, extract the accurate location of white blood cells, and generate a standard dataset.

[0013] Preferably, in step S1, the obvious cell colors (CellGray) are extracted from the bone marrow cell RGB image in the following way: A deep neural network VGG16 model is used to fine-tune the model and analyze the output, suppressing various interfering cells and extracting the obvious cell colors (CellGray), where: CellGray = VGG16['block1_conv2'], and VGG16 is composed of multiple layers of neurons processed by z... l p(i,j) A network structure model composed of functions.

[0014]

[0015] n l The number of convolutional kernels in the l-th layer.

[0016] k l p，q The convolution kernels corresponding to the p-channel of the l-th layer and the q-channel of the (l-1)-th layer are...

[0017] b l p For the convolution kernel corresponding to node p in layer l,

[0018] a l q This is the output of the q-channel of the l-th layer after the activation function.

[0019] z l p(i,j) This is the output of the p-channel (i,j) of the l-th layer after passing through the activation function.

[0020] Preferably, step S2 uses the maximum inter-class variance method. First, based on the grayscale characteristics of the bone marrow cell image CellGray, the image is divided into two parts: background and foreground. Then, the image is binarized and segmented according to the threshold obtained by Otsu's method. Finally, the maximum inter-class variance between the foreground and background images is obtained.

[0021] Preferably, in step S3, WhiteCellMorph is obtained using the following formula:

[0022]

[0023] In the above formula, b represents the structuring element, and ⊙ represents the morphological erosion operation. While discussing morphological dilation operations.

[0024] Preferably, the watershed algorithm operation on WhiteCellMorph in step S3 specifically includes the following steps:

[0025] S31. Classify all pixels in the gradient image according to their gray values ​​and set a geodesic distance threshold.

[0026] S32. Find the pixel with the smallest gray value, and let the threshold increase from the minimum value. These points are the starting points.

[0027] S33. As the horizontal plane grows, it will encounter surrounding neighboring pixels. The geodesic distance from these pixels to the starting point is measured. If it is less than a set threshold, these pixels are submerged; otherwise, dams are set on these pixels. These neighboring pixels are classified.

[0028] S34. As the horizontal plane rises higher, more and higher dams will be set up until the maximum gray value is reached. All areas meet on the watershed line, and these dams partition the entire image pixels.

[0029] Preferably, in step S4, the leukocyte membrane In the formula, b represents the structuring element, and ⊙ represents the morphological erosion operation. While discussing morphological dilation operations.

[0030] Preferably, in step S5, the leukocyte nucleus obtained in step S3 and the cell membrane obtained in step S4 are added together to obtain a complete leukocyte. Then, morphological operations are performed on the leukocyte, and the contour of the leukocyte is extracted to obtain its position. A dataset is generated according to the extracted coordinates in the PASCAL VOC dataset format.

[0031] The beneficial effects of the automatic segmentation method for leukocytes in bone marrow cells provided by this invention are as follows: This invention uses a convolutional neural network to extract features of bone marrow cells from images, enabling the processing of bone marrow cell data acquired through different operations on a large scale; then, through multiple combined operations such as binarization, hole filling, morphological smoothing, and watershed segmentation, bone marrow leukocytes are segmented; the convolutional neural network can fundamentally eliminate the adverse effects of differences in bone marrow cell images caused by factors such as staining agents, freezing, and manual operation on target cells, thereby improving the segmentation accuracy of bone marrow leukocytes; the watershed segmentation used can effectively separate adherent cells, reducing the false negative rate; after independently processing the leukocyte nucleus and leukocyte membrane through multiple binarizations, the overall localization and segmentation accuracy of leukocyte images can be effectively improved by superimposing the data, and a dataset can be generated, facilitating the subsequent development of artificial intelligence for leukocyte type identification. Attached Figure Description

[0032] Figure 1 This is a flowchart of the present invention.

[0033] Figure 2 This is a raw image of bone marrow cells.

[0034] Figure 3 It is an image after feature extraction using a convolutional neural network.

[0035] Figure 4 This is a binary image of leukocyte nuclei processed using the Otsu's method.

[0036] Figure 5 This is a white blood cell nucleus image processed using the watershed algorithm.

[0037] Figure 6 It is a cell membrane image after morphological calculations.

[0038] Figure 7 It is a segmented image of white blood cells obtained by combining data.

[0039] Figure 8 It is a dataset generated based on the extracted coordinates in the PASCAL VOC dataset format. Detailed Implementation

[0040] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. All other embodiments obtained by those skilled in the art without creative effort are within the protection scope of the present invention.

[0041] Example: An automated method for segmenting leukocytes in bone marrow cells.

[0042] Reference Figures 1 to 8 As shown, an automated method for segmenting leukocytes in bone marrow cells includes the following steps:

[0043] (I) Bone Marrow Cell Image Neural Network Processing: This method extracts prominent cell colors (CellGrays) from the input bone marrow RGB image, including leukocyte nuclei, leukocyte membranes, and erythrocytes. It employs the highly generalizable deep neural network VGG. Through fine-tuning the model and analyzing the output with minimal data, the shallow layers of the VGG deep neural network can suppress various interfering cells, extracting prominent cell colors (CellGrays). Here, CellGray = VGG16['block1_conv2'], where VGG16 is composed of multiple layers of neurons processed by z... l p(i,j) A network structure model composed of functions.

[0044]

[0045] n l The number of convolutional kernels in the l-th layer.

[0046] k l p，q The convolution kernels corresponding to the p-channel of the l-th layer and the q-channel of the (l-1)-th layer are...

[0047] b l p For the convolution kernel corresponding to node p in layer l,

[0048] a l q This is the output of the q-channel of the l-th layer after the activation function.

[0049] z l p(i,j) This is the output of the p-channel (i,j) of the l-th layer after passing through the activation function.

[0050] contrast Figure 1 and Figure 2 As shown, neural network processing of bone marrow cell images can fundamentally eliminate the adverse effects of differences in bone marrow cell images caused by factors such as staining agents, freezing, and manual manipulation on target cells, highlighting bone marrow cell images and thus improving the segmentation accuracy of bone marrow leukocytes.

[0051] (ii) The bone marrow cell image CellGray was binarized using the Otsu method to obtain a binary image of the leukocyte nucleus.

[0052] Otsu's algorithm is used to determine the threshold for image binarization segmentation. It's also known as the maximum inter-class variance (MOV) method because the threshold obtained using this method results in the largest inter-class variance between the foreground and background images. Otsu's method divides the image into foreground and background based on its grayscale characteristics. Since variance is a measure of the uniformity of grayscale distribution, a larger inter-class variance between the background and foreground indicates a greater difference between the two parts of the image. Misclassifying part of the foreground as background or vice versa will reduce the difference between the two parts. Segmentation with the largest inter-class variance means the probability of misclassification is minimized.

[0053] Otsu's key features include sensitivity to image noise, single-target segmentation, and the ability to reduce misclassification probability when the target and background sizes are significantly different or the inter-class variance function exhibits bimodal or multimodal characteristics. (See reference...) Figure 4 As shown, the binary image of leukocyte nuclei processed by the Otsu's method is clear and can significantly reduce the probability of misclassification.

[0054] (III) Holes exist in the binary image data of white blood cell nuclei, requiring image morphology opening and closing operations to obtain the WhiteCellMorph. The WhiteCellMorph is obtained using the following formula:

[0055]

[0056] In the above formula, b represents the structuring element, and ⊙ represents the morphological erosion operation. While discussing morphological dilation operations.

[0057] Then, perform a watershed operation on WhiteCellMorph as follows to obtain the segmentation map of leukocyte nucleus:

[0058] a. Classify all pixels in the gradient image according to their grayscale values ​​and set a geodesic distance threshold;

[0059] b. Find the pixel with the smallest gray value, and start increasing the threshold from the minimum value. These points are the starting points.

[0060] c. As the horizontal plane grows, it will encounter surrounding neighboring pixels. The geodesic distance from these pixels to the starting point is measured. If it is less than a set threshold, these pixels are submerged; otherwise, dams are set on these pixels. These neighboring pixels are classified.

[0061] d. As the horizontal plane rises higher, more and higher dams will be set up until the maximum gray value is reached. All areas meet on the watershed line, and these dams partition the entire image pixels.

[0062] Reference Figure 5 As shown, the watershed used in the segmentation can effectively separate adherent cells, reducing the false negative rate of cells.

[0063] (IV) In the CellGray data obtained in step (I), the corresponding numbers are masked using the Leukocytenucleus data obtained in step (III), and then the Leukocyte membrane is obtained by using the Otsu method for image morphology opening and closing operations. In the formula, b represents the structuring element, and ⊙ represents the morphological erosion operation. Let's discuss morphological dilation operations. (Refer to...) Figure 6 As shown, through the operation of step (four), a clear cell membrane image can be obtained, which enhances the cell membrane image and improves the accuracy of cell membrane localization and segmentation.

[0064] (V) Combine the data obtained in steps (III) and (IV) to obtain segmented white blood cell images, extract the accurate locations of white blood cells, and generate a standard dataset. Step (III) yields images of the white blood cell nucleus, and step (IV) yields images of the cell membrane; adding the two together creates a complete white blood cell image (see [reference]). Figure 7 As shown, adding the leukocyte nuclear image and the cell membrane image yields a clear and complete leukocyte image, requiring only simple superposition, making the calculation simple and fast. Then, morphological operations are performed on the leukocyte image to extract its contour and location, effectively improving the accuracy of localization and segmentation. Based on the extracted coordinates, a dataset is generated according to the PASCAL VOC dataset format (refer to...). Figure 8 (as shown), to facilitate subsequent system calls.

[0065] This invention utilizes a convolutional neural network to extract features of bone marrow cells from images, enabling the processing of bone marrow cell data acquired through various operations. Multiple combined operations, including binarization, hole filling, morphological smoothing, and watershed segmentation, are then applied to segment bone marrow leukocytes. The convolutional neural network fundamentally eliminates the adverse effects of staining agents, freezing, and manual manipulation on target cells caused by differences in generated bone marrow cell images, thereby improving the segmentation accuracy of bone marrow leukocytes. The watershed segmentation effectively separates adherent cells, reducing the false negative rate. Multiple binarization operations independently process leukocyte nuclear and membrane images, then overlay them to obtain the overall leukocyte image, effectively improving the accuracy of overall leukocyte image localization and segmentation. This also generates a dataset, facilitating the subsequent development of artificial intelligence for leukocyte species identification.

[0066] The above description is only a preferred embodiment of the present invention, but the present invention should not be limited to the content disclosed in the embodiments and drawings. Therefore, any equivalent or modified embodiments made without departing from the spirit of the present invention shall fall within the protection scope of the present invention.

Claims

1. A method of automatically segmenting leukocytes in bone marrow cells, characterized by Comprise the following steps: S1, extract obvious cells CellGray from the input bone marrow cell rgb image through a convolutional neural network; S2, binarize the bone marrow cell image CellGray to obtain a white blood cell nucleus binary image; S3, perform an open-close operation on the white blood cell nucleus binary image to obtain WhiteCellMorph, and perform a watershed algorithm operation on WhiteCellMorph to obtain a complete white blood cell nucleus segmentation image Leukocytenucleus; WhiteCellMorph is obtained by the following calculation formula: ; In the above formula, b represents a structure element, and represents a morphological erosion operation. Edge said morphological dilation operation; The watershed algorithm operation on WhiteCellMorph specifically comprises the following steps: S31, classify all pixels in the gradient image according to the gray value, and set a geodesic distance threshold; S32, find the pixel point with the smallest gray value, and let the threshold start growing from the minimum value, and these points are starting points; S33, in the growing process of the horizontal plane, the surrounding neighborhood pixels will be encountered, and the geodesic distance of these pixels to the starting point is measured, if it is less than the set threshold, then these pixels are flooded, otherwise a dam is set on these pixels, and these neighborhood pixels are classified; S34, as the horizontal plane becomes higher and higher, more and higher dams will be set until the maximum value of the gray value, and all regions meet on the watershed line, and these dams divide the entire image pixels into regions; S4, using the Leukocytenucleus obtained in step S3 to shield the corresponding number in the CellGray data obtained in step S1, and then using the maximum inter-class variance method and the open-close operation on the image morphology to obtain the white blood cell membrane Leukocytemembrane; Wherein, white blood cell membrane In the formula, b represents a structure element, and represents a morphological erosion operation. Edge morphological dilation operation; S5, combine the data obtained in steps S3 and S4 to obtain a segmented complete white blood cell image, extract the accurate position of the white blood cell, and generate a standard data set.

2. The method of claim 1, wherein the step of automatically segmenting the white blood cells in the bone marrow cells is performed by using a neural network. The obvious cell CellGray is extracted from the bone marrow cell rgb image in the step S1 in the following manner: the deep neural network VGG16 model is used to pass data fine-tuning model and analyze output, suppress various interference outputs obvious cells, and extract obvious cells CellGray, wherein: is a network structure model composed of multiple layers of neurons through z l p (i,j) functions. ； n l is the number of convolution kernels of the first layer, k l p，q is the (i-1)th layer q channel corresponding to the i th layer p channel convolution kernel b l p is the corresponding convolution kernel for the first layer p node, a l q is the output of the lth layer q-channel after the activation function z l p (i,j) is the output of the first layer p-channel (i, j) after the activation function.

3. The method of claim 1, wherein the step of automatically segmenting the white blood cells in the bone marrow cells is performed by using a neural network. In step S2, the maximum inter-class variance method is used, first, according to the gray characteristics of the bone marrow cell image CellGray, the image is divided into background and foreground two parts, then the threshold value obtained by the Otsu method is used for image binarization segmentation, finally the maximum inter-class variance of the foreground and background image is obtained.

4. The method of claim 1, wherein the step of automatically segmenting the leukocytes in the bone marrow cells is characterized by: In step S5, the white blood cell nucleus obtained in step S3 and the cell membrane obtained in step S4 are added to obtain a complete white blood cell, then morphological operation is performed on the white blood cell, then the contour of the operated white blood cell is extracted to obtain the position, and the data set is generated according to the extracted coordinates according to the data set format of PASCAL VOC.

Images