A high-precision automatic segmentation method for pathological slice cells based on convolutional neural network and DBScan

Abstract

The present invention belongs to the field of image processing and artificial intelligence, and discloses a high-precision automatic segmentation method for pathological slice cells based on convolutional neural networks and Dbscan. The method uses pathological slices and manually annotated cell masks to train a U-Net convolutional neural network model improved for pathological slice cell features, and obtains a preliminary semantic segmentation result with low accuracy; then, through the GrabCut post-processing module, the cell under-segmentation condition is compensated, the segmentation edge is optimized, and a high-precision semantic segmentation mask is obtained; then, according to the slice cell mask pixel value density in the semantic segmentation result, the Dbscan algorithm clustering radius and clustering density are adjusted, and cluster instance division is used to obtain a high-precision instance segmentation result. The present invention can perform high-precision and high-efficiency semantic segmentation and instance segmentation on multiple types of pathological slice cells.

Description

Technical Field

[0001] The invention belongs to the field of image processing and artificial intelligence, and in particular relates to a high-precision automatic segmentation method for pathological slice cells based on convolutional neural network and Dbscan. Background Art

[0002] Digital pathology sections refer to pathological tissues collected through surgery or biopsy, which are fixed, embedded, sliced, stained, and sealed to form tissue slides, which are then scanned and imaged using digital microscopes and other instruments, and finally stitched together using image processing software to form high-pixel full-view images. The identification and segmentation of pathology section cells can help pathologists complete tasks such as visualizing section morphology, locating areas of interest, and classifying sections.

[0003] Traditional pathological section cell segmentation methods based on image feature processing generally rely on the professional knowledge of pathologists, cannot achieve automatic image segmentation, and have low image segmentation efficiency. Although the pathological section cell segmentation method based on convolutional neural network can achieve automatic segmentation of pathological images, since the pathological section cell segmentation results will be used for the classification of section results, high requirements are placed on the interpretability, segmentation accuracy and robustness of the model. In addition, pathological sections are difficult to obtain and annotate, and the model has few training samples, which makes it difficult for the existing convolutional neural network segmentation method to complete the task of section result classification. Summary of the invention

[0004] The purpose of the present invention is to provide a high-precision automatic segmentation method for pathological section cells based on convolutional neural network and Dbscan to solve the above-mentioned technical problems.

[0005] In order to solve the above technical problems, the specific technical solution of the high-precision automatic segmentation method of pathological slice cells based on convolutional neural network and Dbscan of the present invention is as follows:

[0006] A high-precision automatic segmentation method for pathological slice cells based on convolutional neural network and Dbscan, comprising the following steps:

[0007] Step 1: Construction of pathological slice image dataset: Pathological slice dataset is constructed using pathological tissue slices collected from hospital biopsies and manually annotated cell masks, and random elastic deformation is used for data enhancement;

[0008] Step 2: Pathological section cell semantic segmentation network training: Use the pathological section dataset to train the convolutional neural network model improved for pathological section cell features, and obtain a preliminary semantic segmentation result with low accuracy;

[0009] Step 3: Post-processing of pathological slice cell semantic segmentation: GrabCut post-processing is used to compensate for under-segmented images, optimize segmentation edges, and obtain high-precision semantic segmentation masks. The slice type and area size are output based on the shape characteristics and pixel count of the semantic segmentation mask of the pathological tissue slice.

[0010] Step 4: Pathological section cell instance segmentation: According to the pixel value density of the pathological tissue section cell mask in the semantic segmentation result, the clustering radius and clustering density of the Dbscan algorithm are adjusted, and clustering instance division is used to obtain high-precision instance segmentation results, and the pathological cell instance category and number are output according to the instance segmentation mask.

[0011] Furthermore, the pathological slice image dataset used for semantic segmentation model training in step 1 is a VOC format dataset, and the pathological slice image dataset used for instance segmentation model training is a COCO format dataset. The random elastic deformation first performs affine transformations such as displacement, rotation, and scaling on the pathological slice image, and then creates a random displacement field on the image after the affine transformation. At each pixel point, two random numbers uniformly distributed between (-1, 1) are randomly generated to represent the displacement in the x and y directions. The random displacement field is then convolved with a Gaussian function and multiplied by a proportional factor that controls the deformation intensity to obtain an enhanced image.

[0012] Furthermore, in step 2, the U-Net network is used as the basic architecture of the convolutional neural network, the VGG16 network is used as the backbone feature extraction network of the U-Net model, and its pre-trained weights on the ImageNet dataset are loaded. The loss function type is a hybrid loss function of the Focal Loss and the cross entropy loss function. Focal Loss starts from the perspective of sample difficulty classification and adjusts the factor P t , reducing the proportion of slide background in model training, the calculation formula is:

[0013] Focal Loss = -(1-P t ) γ log(P t ).

[0014] Furthermore, the model training was completed on a computer with one Intel(R) Xeon(R) CPU E5-2680 v3@2.50GHz CPU and 128G memory and two RTX 2080 Ti GPUs, and the model pre-training weights used the ImageNet dataset VGG16 or ResNet model parameters.

[0015] Furthermore, in step 3, the GrabCut algorithm is used as an image post-processing method, and the semantic segmentation result output by the convolutional neural network is used as the post-processing input, replacing the foreground and background method of manually interactively specifying the image in the original GrabCut algorithm. The weighted graph of the pathological slice image is directly constructed in combination with the original image, and the global optimal segmentation is achieved by minimizing the energy function, thereby compensating for the under-segmentation of slice cells and optimizing the segmentation edge.

[0016] Furthermore, in step 3, the semantic segmentation of the original image and the improved U-Net output Figure 1 It starts with inputting GrabCut post-processing, improving the foreground and background attributes of the specified part of the pixels in the semantic segmentation map output by U-Net, performing set classification, realizing the construction of the weighted graph, and constructing the global energy function according to the distribution law of the pixels. By learning the Gaussian mixture model parameters of the original image, the global energy function is minimized and converged. This process will be repeated three times to achieve the purpose of minimizing the global energy function. Finally, the boundary filling algorithm is used to smooth the edges of the image to obtain a more ideal segmentation effect, and the output post-processing optimized image is obtained.

[0017] Furthermore, in step 4, the clustering density and clustering radius of the Dbscan algorithm are automatically generated using the pixel value density of the pathological section cell semantic segmentation results. For images with a section density higher than 15%, a large clustering density and a small clustering radius are selected; for images with a section density lower than 10%, a small clustering density and a large clustering radius are selected; for images with a section density between 10% and 15%, the image is divided into four equal parts, and the section density of each quarter of the image is repeatedly determined to obtain the clustering density and clustering radius values.

[0018] Furthermore, in step 4, after obtaining the cluster radius and cluster density parameters, the Dbscan algorithm scans the slice cell mask area in the semantic segmentation result to obtain the point P that has not been detected, and detects that the number of objects in its neighborhood points in the cluster radius is not less than the cluster density, then a new cluster C is established. This point is considered to be the core point, and all object points within its density are found to form a cluster. Then, other object points outside the cluster are scanned continuously to continuously form new clusters, thereby realizing instance clustering division of pathological slice cells.

[0019] The high-precision automatic segmentation method of pathological slice cells based on convolutional neural network and Dbscan of the present invention has the following advantages: the present invention improves the U-Net model according to the characteristics of pathological slice cells, combines the GrabCut post-processing method, realizes high-precision automatic semantic segmentation of slice cells, evaluates the mask pixel density of pathological slice cells of the semantic segmentation results, uses the Dbscan algorithm to perform clustering instance division, and realizes high-precision automatic instance segmentation of slice cells. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 This is a flow chart of a high-precision automatic segmentation method for pathological slice cells based on convolutional neural network and Dbscan of the present invention;

[0021] Figure 2 This is a data enhancement effect diagram of random elastic deformation of pathological sections of the present invention;

[0022] Figure 3 This is a structural diagram of the pathological section cell segmentation model of the present invention;

[0023] Figure 4 Schematic diagram of the automatic adjustment method of Dbscan clustering parameters for pathological section cell instance segmentation of the present invention;

[0024] Figure 5a This is the effect diagram of semantic segmentation of cells in TMA pathological sections;

[0025] Figure 5b This is the effect diagram of semantic segmentation of DNA pathological slice cells;

[0026] Figure 5c This is the effect diagram of semantic segmentation of cells in TCT pathological sections;

[0027] Figure 5d This is the semantic segmentation effect of GENERAL pathological slice cells;

[0028] Figure 6a This is the segmentation effect diagram of the medium-density cell instance of the STOM class;

[0029] Figure 6b This is the segmentation effect diagram of STOM-type high-density cell instance;

[0030] Figure 6c This is the segmentation effect diagram of STOM-type low-density cell instance;

[0031] Figure 6d This is the segmentation effect diagram of GLOM-type cell instance. DETAILED DESCRIPTION

[0032] In order to better understand the purpose, structure and function of the present invention, the following is a further detailed description of a high-precision automatic segmentation method for pathological section cells based on convolutional neural network and Dbscan in conjunction with the accompanying drawings.

[0033] The present invention provides a high-precision automatic segmentation method for pathological slice cells based on convolutional neural network and Dbscan, comprising the following steps:

[0034] Step 1: Create a pathology slice dataset using pathology tissue slices collected from hospital biopsies and manually annotated cell masks, and use random elastic deformation for data enhancement;

[0035] The pathological slice image dataset used for semantic segmentation model training is a VOC format dataset, and the pathological slice image dataset used for instance segmentation model training is a COCO format dataset. Random elastic deformation first performs affine transformations such as displacement, rotation, and scaling on the pathological slice image, and then creates a random displacement field on the image after affine transformation. At each pixel point, two random numbers uniformly distributed between (-1, 1) are randomly generated to represent the displacement in the x and y directions. Then, the random displacement field is convolved with a Gaussian function and multiplied by the proportional factor that controls the deformation intensity to obtain the enhanced image, such as Figure 2 shown.

[0036] Step 2: Use the pathology section dataset to train a convolutional neural network model that is improved based on the cell features of pathology sections, and obtain a preliminary semantic segmentation result with low accuracy;

[0037] The U-Net network is used as the basic architecture of the convolutional neural network, and the VGG16 network is used as the backbone feature extraction network of the U-Net model. Its pre-trained weights on the ImageNet dataset are loaded. The loss function type is a hybrid loss function of FocalLoss and cross entropy loss function. Focal Loss starts from the perspective of sample difficulty classification and adjusts the factor P t , reducing the proportion of slide background in model training, the calculation formula is:

[0038] Focal Loss = -(1-P t ) γ log(P t )

[0039] Step 3: Through GrabCut post-processing, compensate for the under-segmented image, optimize the segmentation edge, obtain a high-precision semantic segmentation mask, and output the slice type and area size according to the shape characteristics and pixel number of the semantic segmentation mask of the pathological tissue slice;

[0040] The GrabCut algorithm is used as the image post-processing method, and the semantic segmentation results output by the convolutional neural network are used as the post-processing input. Instead of the original GrabCut algorithm's method of manually interactively specifying the foreground and background of the image, the weighted graph of the pathological slice image is directly constructed in combination with the original image. The global optimal segmentation is achieved by minimizing the energy function, compensating for the under-segmentation of slice cells, and optimizing the segmentation edge.

[0041] Step 4: According to the pixel value density of the pathological tissue section cell mask in the semantic segmentation result, the clustering radius and clustering density of the Dbscan algorithm are adjusted, and clustering instance division is used to obtain high-precision instance segmentation results, and the pathological cell instance category and number are output according to the instance segmentation mask.

[0042] The clustering density and clustering radius of the Dbscan algorithm are automatically generated using the pixel value density of the pathological section cell semantic segmentation results. For images with a section density higher than 15%, a large clustering density and a small clustering radius are selected; for images with a section density lower than 10%, a small clustering density and a large clustering radius are selected; for images with a section density between 10% and 15%, because cells in pathological tissue sections are usually attached to the slide in a symmetrical distribution pattern, the image is divided into four equal parts, and the section density of each quarter of the image is repeatedly judged to obtain the clustering density and clustering radius values.

[0043] The embodiments of pathological section cell segmentation of the present invention are as follows:

[0044] The digital pathology slice images collected and produced by hospital biopsy have been annotated by professionals. This embodiment provides a high-precision automatic segmentation method for pathology slice cells based on convolutional neural network and Dbscan. Figure 1 , the method comprises the following steps:

[0045] 1) Construction of pathological slice image dataset

[0046] This method collects a total of liquid-based thin-layer cell detection images (TCT) for cervical cancer examination, DNA ploidy analysis slice images (DNA) for HPV virus infection and cervical precancerous lesions, serum thyroid microsomal antibody images (TMA) for thyroid disease examination, gastric tissue slice images (STOM) for gastric cancer examination, kidney puncture tissue slice images (GLOM) for kidney disease examination, and unclassified slice images (GENERAL) containing multiple types of pathological slices. Among them, STOM and GLOM pathological slice cells have instance segmentation requirements, and the other four categories only perform semantic segmentation.

[0047] From the collected digital pathological tissue slice images, according to the data volume of this type of slice images, we select images that cover most of the possible slice cell morphologies of this type to create a pathological slice image dataset. The semantic segmentation dataset uses the VOC dataset format, and the instance segmentation dataset uses the COCO dataset format. The training set and validation set are randomly divided in a ratio of 9:1. The selection of various pathological slices and the number of training and validation sets are shown in Table 1:

[0048] Table 1 Pathological section semantic segmentation dataset

[0049]

[0050]

[0051] 2) Pathological section cell semantic segmentation network training

[0052] This method analyzes and improves the U-Net model results based on the characteristics of the six categories of digital pathology sections, such as the small number of training samples, imbalance of positive and negative samples, similarity of target backgrounds, and noise influence. The VGG16 network is used as the backbone feature extraction network, transfer learning is used, pre-trained weights are loaded, and a hybrid loss function based on Focal Loss is used instead of the original cross entropy loss function.

[0053] The training set of digital pathological sections was subjected to random elastic deformation for data enhancement to simulate the morphological and grayscale changes that may occur in cells in pathological tissue sections, and then input into the improved U-Net model for training. The model training was completed on a computer with 1 Intel(R)Xeon(R)CPU E5-2680 v3@2.50GHz CPU and 128G memory and 2 RTX 2080 Ti GPUs. The model pre-training weights used ImageNet dataset VGG16 or ResNet model parameters, the training batch was unified to 36 times, the frozen training batch was 5 times, the frozen training learning rate was 10-4, the non-frozen training learning rate was 10-5, the optimizer used Adam optimizer, and the learning rate adjustment class was StepLR. The semantic segmentation accuracy mIoU of the convolutional neural network training of six types of pathological section cells reached 86%, among which the STOM, GLOM and GENERAL classes reached more than 91%.

[0054] 3) Post-processing of pathological section cell semantic segmentation

[0055] The semantic segmentation results of pathological sections cells after training with the improved U-Net model are prone to under-segmentation because the color of some pathological section cells is too similar to the slide background and there are too few training samples. This method uses the GrabCut algorithm to post-process them to compensate for the under-segmentation and optimize the segmentation edges.

[0056] Semantic segmentation of the original image and the improved U-Net output Figure 1It starts with inputting GrabCut for post-processing, improving the foreground and background attributes of the specified part of the pixels in the semantic segmentation map output by U-Net, performing set classification, realizing the construction of the weighted graph, and constructing the global energy function according to the distribution law of the pixels. By learning the Gaussian mixture model parameters of the original image, the global energy function is minimized and converged. This process will be repeated three times to achieve the purpose of minimizing the global energy function. Finally, the boundary filling algorithm is used to smooth the edges of the image to obtain a more ideal segmentation effect, and the post-processing optimized image is output. There are many types of GENERAL classes with a small number of unclassified pathological sections, and DNA and TCT classes with too similar target backgrounds. The segmentation accuracy after post-processing has increased by more than 3%, and the segmentation accuracy of other types of pathological section cells has also improved. The specific semantic segmentation effect diagram of the four types of pathological section cells after post-processing can be seen. Figure 5a-5d .

[0057] 4) Pathological section cell instance segmentation

[0058] There are multiple cells in STOM and GLOM pathological sections, and a single cell is composed of multiple small tissues. It is necessary to further analyze and evaluate the pathological tissue section cells through instance segmentation.

[0059] The pixel values ​​of pathological tissue sections in the pathological section cell optimization semantic segmentation results output by GrabCut post-processing are read, and their density is calculated based on the pixel values ​​of the entire image. For images with a section density higher than 15%, a large cluster density and a small cluster radius are selected; for images with a section density lower than 10%, a small cluster density and a large cluster radius are selected; for images with a section density between 10% and 15%, because pathological tissue sections are usually attached to the slide in a symmetrical distribution pattern, the image is divided into four equal parts, and then the section density of each quarter of the image is repeatedly determined to obtain the cluster density and cluster radius values.

[0060] After obtaining the cluster radius and cluster density parameters, the Dbscan algorithm scans the slice cell mask area in the semantic segmentation result to obtain the point P that has not been detected, and detects that the number of objects in its neighborhood points in the cluster radius is not less than the cluster density, then a new cluster C is established. This point is considered to be the core point, and all object points that can be reached by its density are found to form a cluster. Then, other object points outside the cluster are scanned to continuously form new clusters to achieve instance clustering of pathological slice cells. The instance segmentation accuracy mAP of pathological slice cells obtained by clustering with the Dbscan algorithm is 50 All reached more than 94%, and the instance segmentation effect diagram can be seen Figure 6a-6d .

[0061] It can be seen that the present invention can automatically segment multiple types of pathological section cells with high precision by improving the convolutional neural network semantic segmentation model, GrabCut post-processing, and Dbscan clustering instance segmentation.

[0062] It is to be understood that the present invention is described by some embodiments, and it is known to those skilled in the art that various changes or equivalent substitutions may be made to these features and embodiments without departing from the spirit and scope of the present invention. In addition, under the teachings of the present invention, these features and embodiments may be modified to adapt to specific circumstances and materials without departing from the spirit and scope of the present invention. Therefore, the present invention is not limited by the specific embodiments disclosed herein, and all embodiments falling within the scope of the claims of this application are within the scope of protection of the present invention.

Claims

1. A high-precision automatic segmentation method for pathological slice cells based on convolutional neural network and Dbscan, characterized in that: The following steps are included: Step 1: Construction of pathological slice image dataset: Pathological slice dataset is constructed using pathological tissue slices collected from hospital biopsies and manually annotated cell masks, and random elastic deformation is used for data enhancement; Step 2: Pathological section cell semantic segmentation network training: Use the pathological section dataset to train the convolutional neural network model improved for pathological section cell features, and obtain a preliminary semantic segmentation result with low accuracy; Step 3: Post-processing of pathological slice cell semantic segmentation: GrabCut post-processing is used to compensate for under-segmented images, optimize segmentation edges, and obtain high-precision semantic segmentation masks. The slice type and area size are output based on the shape characteristics and pixel count of the semantic segmentation mask of the pathological tissue slice. In step 3, the GrabCut algorithm is used as an image post-processing method, and the semantic segmentation result output by the convolutional neural network is used as the post-processing input, replacing the foreground and background method of manually interactively specifying the image in the original GrabCut algorithm, directly combining the original image to construct a weighted graph of the pathological slice image, and realizing global optimal segmentation by minimizing the energy function, compensating for the under-segmentation of the slice cells, and optimizing the segmentation edge; In the step 3, the original image and the semantic segmentation map output by the improved U-Net are input together for GrabCut post-processing, the semantic segmentation map output by the improved U-Net specifies the foreground and background attributes of some pixels, performs set classification, realizes the construction of the weighted graph, and constructs the global energy function according to the distribution law of the pixels. By learning the Gaussian mixture model parameters of the original image, the global energy function is minimized and converged. This process will be repeated three times to achieve the purpose of minimizing the global energy function. Finally, the edge of the image is smoothed by using the boundary filling algorithm to obtain a more ideal segmentation effect, and the post-processing optimized image is output; Step 4: Pathological section cell instance segmentation: According to the pixel value density of the pathological tissue section cell mask in the semantic segmentation result, the clustering radius and clustering density of the Dbscan algorithm are adjusted, and clustering instance division is used to obtain high-precision instance segmentation results, and the pathological cell instance category and number are output according to the instance segmentation mask.

2. According to claim 1, a high-precision automatic segmentation method for pathological sections based on convolutional neural network and Dbscan is characterized in that: The pathological slice image dataset used for semantic segmentation model training in step 1 is a VOC format dataset, and the pathological slice image dataset used for instance segmentation model training is a COCO format dataset. The random elastic deformation first performs affine transformations such as displacement, rotation, and scaling on the pathological slice image, and then creates a random displacement field on the image after the affine transformation. At each pixel point, two random numbers uniformly distributed between (-1, 1) are randomly generated to represent the displacement in the x and y directions. The random displacement field is then convolved with a Gaussian function and multiplied by a scaling factor that controls the deformation intensity to obtain an enhanced image.

3. The method for high-precision automatic segmentation of pathological slice cells based on convolutional neural network and Dbscan according to claim 1 is characterized in that: In step 2, the U-Net network is used as the basic architecture of the convolutional neural network, the VGG16 network is used as the backbone feature extraction network of the U-Net model, and its pre-trained weights on the ImageNet dataset are loaded. The loss function type is a hybrid loss function of the Focal Loss and the cross entropy loss function. The Focal Loss starts from the perspective of sample difficulty classification and adjusts the factor P t , reducing the proportion of slide background in model training, the calculation formula is: 。 4. The method for high-precision automatic segmentation of pathological slice cells based on convolutional neural network and Dbscan according to claim 3 is characterized in that: The model pre-trained weights use the ImageNet dataset VGG16 or ResNet model parameters.

5. The method for high-precision automatic segmentation of pathological slice cells based on convolutional neural network and Dbscan according to claim 1, characterized in that: In step 4, the clustering density and clustering radius of the Dbscan algorithm are automatically generated using the pixel value density of the pathological section cell semantic segmentation results. For images with a section density higher than 15%, a large clustering density and a small clustering radius are selected; for images with a section density lower than 10%, a small clustering density and a large clustering radius are selected; for images with a section density between 10% and 15%, the image is divided into four equal parts, and the section density of each quarter of the image is repeatedly determined to obtain the clustering density and clustering radius values.

6. The method for high-precision automatic segmentation of pathological slice cells based on convolutional neural network and Dbscan according to claim 5, characterized in that: In step 4, after obtaining the cluster radius and cluster density parameters, the Dbscan algorithm scans the slice cell mask area in the semantic segmentation result to obtain the point P that has not been detected, and detects that the number of objects in its neighborhood points in the cluster radius is not less than the cluster density, then a new cluster C is established. This point is considered to be a core point, and all object points within its density are found to form a cluster. Then, other object points outside the cluster are scanned continuously to continuously form new clusters, thereby realizing instance clustering division of pathological slice cells.

Images