A cell nucleus segmentation method based on attention learning

Abstract

A cell nucleus segmentation method based on attention learning. The present invention relates to the problem of cell nucleus segmentation in intelligent pathological diagnosis. Intelligent pathological diagnosis technology uses deep learning technology to segment and identify abnormal cells in cell images. However, there are few models for cell nucleus segmentation in cell images. The following problems exist: (1) The problem of cell nucleus overlap and unclear boundaries is not considered, resulting in low segmentation accuracy. (2) The contextual information of the edge of the cell nucleus is not considered, resulting in under-segmentation or over-segmentation, affecting the results of subsequent classification. To this end, we propose a cell nucleus segmentation method based on attention learning. Experiments show that our model can effectively solve the problem of overlapping cell segmentation, as well as the problem of under-segmentation or over-segmentation of unclear boundaries. The present invention is applied to cell nucleus segmentation in intelligent pathological diagnosis.

Description

Technical Field

[0001] The present invention relates to intelligent pathological diagnosis, in particular to cell nucleus segmentation Background Art

[0002] Cervical cancer is the second largest killer threatening women's health. Every two minutes, a woman dies of cervical cancer worldwide. Early cervical cancer can be completely cured, so early diagnosis and early treatment are effective means to deal with cancer outbreaks. Liquid-based thin-layer cell detection is currently the most commonly used cervical cancer cell pathology examination technology in the world, which can detect some precancerous lesions and microbial infections. However, traditional pathological diagnosis relies entirely on manual operation and manual reading of films by doctors with naked eyes. However, the workload is large, the diagnosis rate and accuracy are low, and large-scale screening cannot be promoted. With the development of computer image processing and artificial intelligence technology, automatic pathological diagnosis technology has emerged. In this technology, cell segmentation is the basis of analysis. The purpose of cell segmentation is to accurately locate cells and draw cell outlines to provide input for subsequent processing. However, due to the wide variety of human cell nuclei, large color differences, uneven staining and their own pathological changes, some cell nuclei are similar in color to the cytoplasm or background, and the boundary distinction is not obvious. In addition, a large number of adhesions or overlapping cells also make segmentation difficult and accurate, and are prone to under-segmentation or over-segmentation problems.

[0003] There are two types of segmentation methods at present. One is the traditional segmentation method, including threshold method, region-based segmentation, undirected graph-based segmentation, active contour segmentation and fuzzy clustering. The threshold method divides the pixels of the image into several categories by setting different feature thresholds. It has the advantages of simple implementation, small calculation amount and relatively stable performance. The threshold method does not use the spatial information of the pixel, cannot segment the target with large internal variance, and the segmentation effect is easily disturbed by noise. The region-based segmentation uses the spatial information of the image to classify each pixel into each region. The typical one is the watershed segmentation method. In order to improve the performance of watershed segmentation, researchers have proposed methods such as custom markers, improved algorithm implementation principles, and re-segmentation or merging of segmentation results. In addition, the region segmentation method also has superpixel segmentation methods such as simple linear iterative clustering and mean shift. Graphcut is the representative of undirected graph-based segmentation. Its basic idea is to build a graph with image pixels or superpixels as image vertices. Its optimization goal is to find a cut that makes each subgraph unconnected. The iterative version of Graphcut has achieved better performance, but the disadvantage of this type of method is high time complexity and space complexity. The basic idea of ​​the active contour model is to use a continuous curve to express the target contour, and transform the segmentation process into a process of solving the minimum value of the energy functional. Typical examples include parametric active contour models represented by the snake model and geometric active contour models represented by the level set method. However, most medical images are fuzzy, and the fuzzy clustering method introduces the concept of "membership" in fuzzy theory, which divides the pixels into areas with high membership, thereby improving the accuracy of segmentation.

[0004] The other is the deep learning method. UNet first applied deep learning technology to medical image segmentation and made progress in the segmentation of cell images and liver CT images. The jump connection in UNet directly combines shallow features with deep features, which is easy to produce a semantic gap. UNet++ borrows the dense connection idea of ​​DenseNet. Dense convolution blocks are added to the jump connection to reduce the semantic gap. Brügger et al. improved the training method of UNet and proposed a UNet model with partially reversible sequences. The reversible sequence can realize back propagation without saving the intermediate results of the previous propagation, which reduces the memory consumption during training. CE-Net replaces the encoding module of UNet with a pre-trained model and proposes a context encoding network to capture more high-level information and retain spatial information for 2D medical image segmentation. It mainly includes three modules: feature encoding module, context extraction module and feature decoding module. This method performs better than UNet on different 2D medical image segmentation tasks. 3D U2-Net changes the way of convolution in UNet. The main contribution is to replace the standard convolution in the UNet structure with a domain adapter based on depthwise separable convolution, reducing the number of network parameters. Oktay et al. proposed a new Attention Gate (AG) model for medical images. The model automatically focuses on target structures of different shapes and sizes. The AG-trained model suppresses irrelevant areas in the input image through implicit learning while highlighting salient features that are useful for specific tasks. AG can be easily integrated into the standard convolutional neural network architecture to reduce the time of model prediction.

[0005] Most of the above models are aimed at image segmentation in imaging, and there are few models for cell nucleus segmentation in TCT images. The main problems are as follows: (1) The problem of cell nucleus overlap and unclear boundaries is not considered, resulting in low segmentation accuracy. (2) The contextual information of the cell nucleus edge is not considered, resulting in under-segmentation or over-segmentation. To this end, we propose a cell nucleus segmentation method based on attention learning. The context encoding layer contains four sets of parallel dilated convolutions. The dilated convolutions with different expansion rates can extract features of receptive fields of different sizes. The fusion of different receptive fields can connect the contextual information in the image. It can effectively solve the problem of under-segmentation and over-segmentation caused by unclear boundaries. The attention learning mechanism consists of a scaling module, a codec module, and an attention loss. The scaling module can obtain more image detail information. The codec module learns the attention weight matrix to solve the problem that the traditional attention mechanism is limited to specific tasks. The attention loss can customize the area worth noting. In this paper, the edge of the cell nucleus is taken as the area to be learned to solve the problem of overlapping cell nuclei segmentation. Summary of the invention

[0006] The purpose of the present invention is to solve the problem of cell nucleus segmentation in intelligent pathological diagnosis, and to propose a cell nucleus segmentation method based on attention learning mechanism.

[0007] The above invention aims to be achieved mainly through the following technical solutions: the model structure and training method are as follows: Figure 1 As shown:

[0008] S1. Preparing a training data set includes four steps:

[0009] S11. Image selection: The cell nucleus images are from 100 cervical smears, and the exfoliated cells are collected from people of different ages and different conditions, including all lesion levels of the TBS diagnostic criteria.

[0010] S12. Label the cell nucleus: manually circle the outline of the cell nucleus and save the position information of the cell nucleus outline in each image into the annotation file.

[0011] S13 generates a labeling image: first, a completely black single-channel image (grayscale value is 0) is created, then the position information in the annotation file is read, and the grayscale value of each cell nucleus area is set to 255.

[0012] S14, crop image: crop the cell nucleus image and the marker image at the same time, and the crop size is 515*512. Ensure that the cell nucleus image and the marker Figure 1 To.

[0013] S2. Prepare an attention learning loss labeling map, including the following steps:

[0014] S21. Obtain a loss label map through calculation or image processing operations: use the Sobel operator to detect the edge maps of the label map in the X and Y directions respectively, and obtain a loss label map after fusing the two images.

[0015] S22, after corresponding, load into memory: cell nucleus image, label map, loss label Figure 1 After the correspondence, the overall order is disrupted and then loaded into the memory in batches.

[0016] S3. The attention mechanism is widely used in the field of semantic segmentation. It can be used to emphasize or select important information of the target processing object and suppress some irrelevant details. In the general attention mechanism, the attention weight is obtained through a series of calculations. This attention mechanism can only complete specific tasks, and the attention is uncontrollable. Therefore, the present invention proposes an attention learning mechanism, such as Figure 2The figure shows a scaling module, a codec module, and an attention loss. First, the input feature map is enlarged to twice the original image, and then a single-channel feature matrix is ​​obtained using the codec module. The feature matrix is ​​restored to its original resolution after the Sigmod function is calculated to obtain the attention matrix. The output O(x) of the attention learning structure is calculated as follows:

[0017]

[0018] Where F(x) represents the input feature map and A(x) represents the attention weight matrix.

[0019] The scaling module in the attention learning mechanism magnifies the edges of overlapping cell nuclei, allowing the network to obtain more detailed features. The encoder then encodes the global features of the image, and the decoder learns the attention weights. The attention loss makes the attention mechanism not limited to specific tasks, which further highlights its advantages.

[0020] S4. In image segmentation, feature extraction includes convolution and pooling operations. Pooling can increase the receptive field of the convolution kernel and improve computational efficiency. Image segmentation is pixel-level classification, so upsampling is required to restore the resolution of the image. However, there is always some information loss in the process of downsampling and then upsampling. Dilated convolution solves this problem to a certain extent. Dilated convolution is the process of adding some spaces or zeros between the elements in the convolution kernel to expand the convolution kernel. Dilated convolution has a relatively important parameter, the expansion rate Rate. For example Figure 3 As shown in the figure, when Rate = 1, there is no space between each element of the convolution kernel. Convolution kernels of different rates have different receptive fields. The fusion of dilated convolutions with different dilation rates can extract the context information of the image. Using this feature and drawing on CE-Net, we designed a context encoding structure. Dilated convolutions with different dilation rates are operated in series to form four parallel branches. The dilation rate of each branch increases continuously, and the receptive fields are 3, 7, 9, and 19 respectively. The features extracted by the branches are fused in an additive manner. Since the importance of features extracted by different branches is different, the SE channel attention mechanism is added on this basis. This structure extracts context information of different resolutions and avoids information loss.

[0021] Build the network structure. The network adopts the U-Net structure with attention mechanism, such as Figure 1 As shown in the figure. The encoder uses the pre-trained model of resnet34 in the ImageNet dataset and inserts a context encoding layer before its four-layer encoding structure. The decoder uses upsampling plus convolution operations. Each layer of the encoder is connected to an attention learning structure, and the output feature map is fused with the upsampled feature map of the same resolution for convolution. The output of the network includes the upsampled output and the output r of the attention learning structure. The calculation formula of r is as follows:

[0022]

[0023] Among them, C(x) represents 1*1 convolution, ups(x,i) means upsampling x i times, O i (x) represents the output of the i-th attention learning structure.

[0024] S5, train the model and segment the cell nucleus. First, use the data set created in S1 to train the model designed in S4. During the training, we use the fixed step size method to decay the learning rate. Then calculate the loss, the formula is as follows:

[0025] L=L bce (r,g)+L bce (c,m) (3)

[0026] In the formula, L represents the loss of the model, L bce represents the binary cross entropy loss function, r is as shown in Formula 2, g is the loss label map in S2, c represents the output of the model, and m represents the label map in S2. After 200 rounds of training, the loss tends to be stable, and the training is stopped to obtain the parameters of the cell nucleus segmentation model. Finally, the model is used to segment the cell nucleus.

[0027] Effects of the Invention

[0028] The present invention proposes a cervical cell nucleus segmentation network with attention learning. The attention learning mechanism focuses on the edge position of the cell nucleus and effectively segments the overlapping cell nuclei. The context encoding layer further extracts the context features of the cell nucleus to avoid under-segmentation and over-segmentation problems. Figure 5 As shown in Table 1, we have verified the advantages of our proposed model in the dataset of the present invention in experiments. And further verified that the present invention has the same advantages in other public datasets. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 Model structure and loss calculation method;

[0030] Figure 2 Attention learning mechanism structure diagram;

[0031] Figure 3 Dilated convolutions with different dilation rates;

[0032] Figure 4 Multi-scale context encoding layer;

[0033] Figure 5 Model segmentation effect diagram;

[0034] Table 1 Test results of different models on the dataset of the present invention. Specific implementation methods

[0035] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0036] Embodiment 1:

[0037] The present invention provides a cell nucleus segmentation method based on an attention learning mechanism, comprising the following steps:

[0038] S1. Prepare training data set;

[0039] S2. Prepare the attention learning loss labeling map and load the data;

[0040] S3, build attention learning structure;

[0041] S4, build network structure;

[0042] S5. Train the model and segment the cell nucleus.

[0043] The embodiments of the present invention are described in detail below:

[0044] The embodiment of the present invention is specifically implemented as follows.

[0045] S1. Preparing a training data set includes four steps:

[0046] (1) Image selection: The cell nuclear images were obtained from 100 cervical smears. The exfoliated cells were collected from people of different ages and conditions, including all lesion levels of the TBS diagnostic criteria.

[0047] (2) Marking the cell nucleus: Manually circle the outline of the cell nucleus and save the position information of the cell nucleus outline in each image into the annotation file.

[0048] (3) Generate a labeled image: First, create a completely black single-channel image (grayscale value is 0), then read the position information in the annotation file and set the grayscale value of each cell nucleus area to 255.

[0049] (4) Crop the image: Crop the nucleus image and the marker image at the same time, and the crop size is 515*512. Figure 1 To.

[0050] S2. Prepare the attention learning loss label map, which includes the following steps: first, use the Sobel operator to detect the edge map in the X and Y directions of the label map, and then fuse the two images to obtain a loss label map. Then, the cell nucleus image, label map, and loss label map are combined. Figure 1 After the correspondence, the overall order is disrupted and then loaded into the memory in batches.

[0051] S3. Build an attention learning structure such as Figure 2 As shown in the figure, it contains a scaling module, a codec module, and an attention loss. First, the input feature map is enlarged to twice the original image, and the result is input into the codec module to obtain a single-channel feature matrix. Then, the feature matrix is ​​subjected to a Sigmod operation, and the result is restored to the resolution of the original input to obtain the attention weight matrix. The output O(x) of the attention learning structure is calculated as follows:

[0052]

[0053] Where F(x) represents the input feature map and A(x) represents the attention weight matrix.

[0054] S4. Build a network structure such as Figure 1 As shown in , the network adopts a U-Net structure with an attention mechanism. The encoder uses the pre-trained model of resnet34 in the ImageNet dataset and inserts a context encoding layer before its four-layer encoding structure, as shown in Figure 4 As shown in the figure, the decoder uses upsampling plus convolution operations. Each layer of the encoder is connected to an attention learning structure, and the output feature map is fused with the upsampled feature map of the same resolution for convolution. The output of the network includes the upsampled output and the output r of the attention learning structure. The calculation formula for r is as follows:

[0055]

[0056] Among them, C(x) represents 1*1 convolution, ups(x,i) means upsampling x i times, O i (x) represents the output of the i-th attention learning structure. In the context coding layer, the dilated convolutions with different sampling rates form four branches, and the expansion rate of each branch increases continuously. The receptive fields are 3, 7, 9, and 19 respectively. The features extracted by the branches are fused in an additive way to form context coding. Then the channel attention mechanism is introduced to make the structural coding ability stronger.

[0057] S5, train the model and segment the cell nucleus. First, use the data set created in S1 to train the model designed in S4. During the training, we use the fixed step size method to decay the learning rate. Then calculate the loss, the formula is as follows:

[0058] L=L bce(r,g)+L bce (c,m) (3)

[0059] In the formula, L represents the loss of the model, L bce represents the binary cross entropy loss function, r is as shown in formula (2), g is the loss label map in S2, c represents the output of the model, and m represents the label map in S2.

[0060] Training settings: The main coding structure in the model is based on ResNet pre-trained on ImageNet. The source code is implemented based on the PyTorch platform. The training and testing platform is the Ubuntu Server 18.04.2LTS system, equipped with 4 RTX2080Ti graphics cards and dual CPUs. The present invention uses 3 graphics cards, batch=4, the learning rate is 0.00005, and the number of training times is 200. Stop training to obtain the parameters of the cell nucleus segmentation model. Finally, use the model to segment the cell nucleus. The final implementation effect is as follows Figure 5 As shown in the figure, it can be seen that the overlapping cell nuclei are well segmented, and the cell nuclei with unclear boundaries are not over-segmented.

[0061] The present invention may also have other various embodiments. Without departing from the spirit and essence of the present invention, those skilled in the art may make various corresponding changes and modifications based on the present invention, but these corresponding changes and modifications should all fall within the scope of the present invention.

Claims

1. A cell nucleus segmentation method based on attention learning, characterized in that: The following steps are included: S1. Prepare training data set; (1) Image selection: The cell images were obtained from 100 cell smears. The exfoliated cells were collected from people of different ages and conditions, including all lesion levels of the TBS diagnostic criteria; (2) Labeling the cell nucleus: manually circle the outline of the cell nucleus and save the position information of the cell nucleus outline in each image into the annotation file; (3) Generate a labeling image: First, create a completely black single-channel image with a grayscale value of 0, then read the position information in the annotation file and set the grayscale value of each cell nucleus area to 255; (4) Image cropping: Crop the cell nucleus image and the marker image at the same time, with the cropping size of 515*512; ensure that the cell nucleus image and the marker image are cropped consistently; S2. Prepare the attention learning loss labeling map and load the data; First, the edge detection algorithm is used to detect the edge images of the label image in the X and Y directions respectively, and the two images are fused to obtain a loss label image; then the cell nucleus image, the label image, and the loss label image are matched one by one, and the order is disrupted as a whole, and then loaded into the memory in batches; S3. Build an attention learning structure, which includes a scaling module, a codec module, and an attention loss. First, enlarge the input feature map to twice the original image, and input the result into the codec module to obtain a single-channel feature matrix. Then Perform a Sigmod operation on the feature matrix, and the result restores the resolution of the original input to obtain the attention weight matrix; the output O(x) of the attention learning structure is calculated as follows: In the formula, F(x) represents the input feature map, and A(x) represents the attention weight matrix; S4, build network structure; The network adopts the U-Net structure with attention mechanism; the encoder is a pre-trained model in the ImageNet dataset, and a context encoding layer is inserted before each layer of its encoding structure; the decoder adopts upsampling plus convolution operation; each layer of the encoder is connected to an attention learning structure, and the output feature map is fused with the upsampled feature map of the same resolution for convolution; the output of the network includes the upsampled output and the output r of the attention learning structure; the calculation formula of r is as follows: Among them, C(x) represents 1*1 convolution, ups(x,i) means upsampling x i times, O i (x) represents the output of the i-th attention learning structure; in the context encoding layer, the dilated convolutions with different sampling rates form four branches, and the expansion rate of each branch increases continuously. The receptive fields are 3, 7, 9, and 19 respectively. The features extracted by the branches are fused in an additive way to form context encoding; then the channel attention mechanism is introduced to make the structural encoding ability stronger; S5, train the model and segment the cell nucleus; First, the model designed in S4 is trained using the dataset created in S1. The fixed step size method is used to decay the learning rate during training. Then the loss is calculated using the following formula: L=L bce (r,g)+L bce (c,m) (3) In the formula, L represents the loss of the model, L bce represents the binary cross entropy loss function, r is as shown in formula (2), g is the loss labeling map in S2, c represents the output of the model, and m represents the labeling map in S2; after 200 rounds of training, the loss tends to be stable, and the training is stopped to obtain the parameters of the cell nucleus segmentation model; finally, the model is used to segment the cell nucleus.

Images