A method and system for cell medical image segmentation

Abstract

The present invention discloses a method for cell medical image segmentation, which belongs to the field of medical image analysis technology, and specifically comprises the following steps: using the original cell image as the input of a pre-trained cell medical image segmentation neural network model, the cell medical image segmentation neural network model performs feature enhancement and feature reuse on the cell nucleus of the image to obtain a significant multi-semantic feature map; performing pooling pyramid down-sampling on the feature map to obtain a multi-semantic feature map, up-sampling to restore the original image size, splicing each up-sampled feature map to obtain a segmentation result map, and superimposing the segmentation result map with the original cell image as a model output. The present invention can segment cell images and the features of the segmented area are prominent.

Description

Technical Field

[0001] The invention relates to a method and system for segmenting a cell medical image, and belongs to the technical field of medical image analysis. Background Art

[0002] Nucleus segmentation is an important and challenging step in computer-aided diagnosis of various pathological cells. This is because: 1) due to the irregular shape of cells in smears and the uneven distribution of chromatin, it is difficult to accurately segment the nucleus; 2) there are difficult samples with very small nuclei, and it is difficult to accurately segment small nuclei; 3) the pixel-by-pixel classification nature of semantic segmentation easily leads to the existence of pixels classified as cytoplasm and background within the nucleus in the segmentation results. Classic neural networks with similar encoder-decoder structures include: Fully Convolutional Network (FCN), U-Net, UNet++, DeepLab series, etc. In recent years, there are many methods for nucleus segmentation, including traditional methods such as level set and watershed, and machine learning methods such as clustering, unsupervised classification, and shape modeling. Most of these algorithms only use the spatial domain information of cell images for segmentation, and the segmentation accuracy is poor in some unclear transition areas between the nucleus and cytoplasm; a few algorithms use simple prior knowledge such as the shape of the nucleus, but the segmentation effect is still not accurate enough. Deep convolutional neural networks have made many achievements in the field of biomedical image segmentation, such as segmenting organs or lesions in magnetic resonance imaging (MR) images, or cells or tumors in pathological images. Due to its powerful feature extraction capabilities, deep learning methods have great advantages in medical image processing tasks and have become the mainstream method. Current deep learning methods often combine high- and low-level feature information. Low-level features contain more location information and less semantic information, while high-level features contain more semantic information and less location information. However, current methods do not focus on the contextual dependency of global information, and do not aggregate contextual information based on different regions to mine global contextual information. As a result, there is a possibility of a large number of pixel classification errors, it is difficult to distinguish easily confused categories, it is difficult to mine small cell nucleus sample information, the features of the segmented area are difficult to highlight well, and the noise of the segmented part is difficult to suppress. Summary of the invention

[0003] The purpose of the present invention is to overcome the deficiencies in the prior art and to provide a method and system for cell medical image segmentation, which can segment cell images and highlight the features of the segmented areas.

[0004] In order to achieve the above object, the technical solution adopted by the present invention is:

[0005] In a first aspect, an embodiment of the present invention provides a method for cellular medical image segmentation, comprising the following steps: using the original cell image as the input of a pre-trained cellular medical image segmentation neural network model, the cellular medical image segmentation neural network model sequentially performs feature enhancement and feature enhancement feature reuse on the background, cytoplasm, large cell nuclei and small cell nuclei of the image to obtain a feature map containing multiple semantics; the cellular medical image segmentation neural network model uses the importance of neurons to perform feature enhancement processing on the original cell image; after upsampling and downsampling the feature map containing multiple semantics, the upsampled feature maps are segmented to obtain a segmentation result map, and the segmentation result map is superimposed on the original cell image as a model output; the training sample of the pre-trained cellular medical image segmentation neural network model is a single cell image with annotated background, cytoplasm and cell nucleus.

[0006] Furthermore, the importance of the neuron is calculated by formula (1):

[0007]

[0008] Where E is the importance of the neuron, is the minimum energy function, are the mean and variance of all neurons in a channel except the target neuron, t is the target neuron of each channel, λ is the only hyperparameter, M is the number of energy functions of each channel, x i For each channel, other neurons except the target neuron are counted.

[0009] Furthermore, formula (2) is used for feature enhancement processing:

[0010]

[0011] in, is the feature map of feature enhancement, sigmoid() is the activation function, and X is the input feature map.

[0012] Furthermore, the feature map after feature enhancement is segmented using formula (3):

[0013]

[0014] in, is the feature map of the lth layer, H l () indicates network transformation, [] indicates the concatenation of feature maps, Represents the feature map of the l-1th layer.

[0015] Furthermore, the training of the cellular medical image segmentation neural network model includes the following steps: using a single cell image in the Herlev Pasteur cell data set as a training set; calculating the channel-by-channel mean of the entire training set image, and performing a mean subtraction operation on a single image in the training set; inputting the training set, and using a weighted cross entropy loss function of four categories: background, cytoplasm, large cell nucleus, and small cell nucleus for training.

[0016] Furthermore, the weighted cross entropy loss function is calculated by formula (4):

[0017]

[0018] in, Represent the main loss function and auxiliary loss function respectively, β is the loss weighted balance parameter, y is the annotation result, For the final prediction result, is the prediction result of the auxiliary branch, c=4, α j is the weighting coefficient of the category.

[0019] In a second aspect, an embodiment of the present invention provides a system for cellular medical image segmentation, comprising a backbone network, a pyramid pooling module and a loss layer; the backbone network is used for 3D enhancement feature enhancement extraction and reuse of sample feature maps, providing rich semantic feature information for the pyramid pooling module; the pyramid pooling module is used to fuse the contextual semantic feature information of each feature map, and each feature map is encoded and decoded at each scale and then superimposed with the original map to obtain a final cell segmentation map; the loss layer is used for learning optimization of the backbone network and the pyramid pooling module; the backbone network is composed of densely connected feature extraction sub-network blocks and alternating transition layers, and the feature extraction sub-network blocks include an attention mechanism layer.

[0020] Preferably, the loss layer includes an auxiliary loss layer and a main loss layer, both of which use weighted cross entropy loss functions of four categories: background, cytoplasm, large cell nuclei, and small cell nuclei as loss functions.

[0021] Preferably, the backbone network is composed of four densely connected network blocks, and the transition layers composed of convolutional layers between the densely connected network blocks control the number of channels; each of the densely connected network blocks is composed of six feature extraction sub-network blocks, and the input of each feature extraction sub-network block is the output of all feature extraction sub-network blocks preceding the feature extraction sub-network block; the pyramid pooling module contains three dimensions.

[0022] Preferably, the first transition layer of the backbone network uses a 2x2 convolutional layer to aggregate features without directly downsampling, and allows the network to learn appropriate convolution kernel parameters through back propagation.

[0023] The present invention enhances the input feature map by introducing the importance of neurons, thereby improving the significance of the target area. At the same time, the present invention enhances the image cell nucleus features, performs up and down sampling after feature weighting, and effectively avoids the problem of the inside of the cell nucleus being classified as cytoplasm and background pixels, thereby obtaining a clear and significant cell segmentation map of the target area.

[0024] The backbone network of the present invention is composed of closely connected feature extraction subnetwork blocks, each of which includes an attention mechanism layer, so that the features of the image cell nucleus are more prominent. At the same time, the present invention adopts a three-layer pyramid pooling module to integrate context information, which reduces parameters without affecting the effect, accelerates convergence and running speed, and effectively avoids the problem of being classified as cytoplasm and background pixels within the cell nucleus. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Figure 1 A flowchart of a method for cell medical image segmentation provided in Example 1 of the present invention;

[0026] Figure 2 A schematic diagram of the structure of a system for cellular medical image segmentation provided in Example 2 of the present invention;

[0027] Figure 3 This is the IoU trend chart of the loss during network training and testing of the present invention;

[0028] Figure 4 It is a schematic diagram of the segmentation result of the present invention;

[0029] Figure 5 This is the saliency heat map of the attention mechanism in the present invention. DETAILED DESCRIPTION

[0030] In order to better understand the essence of the present invention, the present invention is further described below in conjunction with specific embodiments and drawings.

[0031] Example 1

[0032] The present invention provides a method for segmenting a cellular medical image, comprising the following steps:

[0033] S1. Sample preparation.

[0034] The present invention uses the public cervical Papanicolaou cell dataset Herlev, which contains 917 single cell images and is labeled in four categories, namely background (red), cytoplasm edge (gray), cytoplasm (dark blue), and nucleus (light blue). In this case, the cytoplasm edge is classified into the background category, and only three categories are finally segmented. The sample size set in this case is l=256×256×3, and the samples are unified to a size of 256×256×3 after nearest neighbor sampling. In order to eliminate the differences between samples and improve the convergence speed of the model, the channel-by-channel mean of the entire training set image is first calculated, and the mean subtraction operation is performed on the image during pre-processing. Since the distribution in the training set is consistent, non-relevant areas can be excluded, and the segmentation-related areas can be preliminarily highlighted, thereby accelerating the convergence of the model and feature extraction. Therefore, the present invention performs a pre-processing operation of subtracting the channel-by-channel mean from the image element by element.

[0035] The ratio of training set to test set is 8:2. 733 images are randomly selected as training set and 184 images are randomly selected as test set.

[0036] S2. Construct a network model for cellular medical image segmentation.

[0037] The present invention uses the pytorch learning library to build a CNN network.

[0038] S21. Obtain neuron parameters after adding the attention mechanism.

[0039] Solve an energy function with a closed-form solution and bring it back to the original equation to get the weight of the attention mechanism module. Multiply the neuron parameter by the weight to get the neuron parameter after adding the attention mechanism. The minimum energy function obtained by solving for:

[0040]

[0041] in, are the mean and variance of all neurons in a channel except the target neuron, t is the target neuron of each channel, λ is the only hyperparameter, M is the number of energy functions of each channel, x i For each channel, other neurons except the target neuron are counted.

[0042] According to formula (1), the importance of the neuron E can be obtained:

[0043]

[0044] S22. Perform feature enhancement processing on the input feature map to extract channel, spatial and pixel-by-pixel 3D enhancement features:

[0045]

[0046] in, is the feature map after feature enhancement, sigmoid() is the activation function to prevent the E value from being too large, and X is the input feature map.

[0047] The feature map is a feature map obtained after feature extraction of the pre-processed sample image.

[0048] S23. Feature map after feature enhancement in S22 Reuse and extract features to obtain feature maps with rich semantics:

[0049]

[0050] in, is the feature map of the lth layer, H l () indicates network transformation, [] indicates the concatenation of feature maps, Represents the feature map of the l-1th layer.

[0051] S3. After downsampling the semantically rich feature map obtained in S23 at different scales, upsampling is performed to obtain a multi-scale feature map. After segmenting each upsampled feature map, the Canny operator is used to superimpose each segmentation result map and the original cell map as the output.

[0052] S4. Model training

[0053] The training set is used for training. The present invention adopts mini-batch training mode for training. The batch size is set to 16, λ = 1e -4 , load the constructed cellular medical image segmentation neural network model and initialize the network parameters. The optimizer uses Adam, and the initial learning rate is set to le=13e -3 , and then an exponential learning rate decay strategy is used during training, with the decay rate set to 0.95.

[0054] In order to solve the problem that it is difficult to accurately segment small cell nuclei, the method of the present invention is: 1) using a weighted cross entropy loss function to strengthen the network training of small cell nuclei; distinguishing the labels of small cell nuclei from the labels of large cell nuclei for training, that is, segmenting small cell nuclei and large cell nuclei as different categories, that is, segmenting four categories: setting the weights of the four categories of background, cytoplasm, large cell nuclei, and small cell nuclei to α1, α2, α3, and α4 respectively. 2) within a mini-batch, during training, the number of iterations of small cell nuclei is doubled compared to the number of iterations of other classes, thereby increasing the optimization contribution of small cell nuclei to the network and increasing the mining of difficult samples.

[0055] Main loss function From formula (4), we can get:

[0056]

[0057] Among them, y is the labeling result, To predict the results, is the final prediction result, c = 4, α j is the weighting coefficient of the category.

[0058] Auxiliary loss function From formula (5), we can get:

[0059]

[0060] in, is the prediction result of the auxiliary branch.

[0061] Then the weighted cross entropy loss function L total From formula (6), we can get:

[0062]

[0063] The weights of the four categories of background, cytoplasm, large nucleus, and small nucleus are α1=0.1, α2=0.1, α3=0.5, and α4=1 is optimal.

[0064] In this embodiment, β is set to 0.4.

[0065] The mIoU value is used to evaluate the segmentation effect of the method of the present invention. The specific results are shown in Table 1. As shown in Table 1, when the control data set samples have the same pre-processing, the segmentation effect of the method proposed by the present invention is better than that of the baseline model U-Net, and the mIoU exceeds U-Net by nearly 4 percentage points. The loss and mIoU index trend chart of each round of training and testing of the model is shown in Figure 3 The segmentation evaluation effect of the model is good. Figure 4 As shown in Figure 1, three sample images were selected for testing. The Canny operator was used to extract the edges in the segmentation results. The annotated lines in the rightmost segmentation overlay image basically overlap with the prediction results of the network model. The saliency heat map before and after the introduction of the attention mechanism in the model is shown in Figure 1. Figure 5 , Figure 5 Grad-CAM is used to visualize the SimAM before and after it is added, where A is the original cell image, B is the heat map before the attention mechanism is added, and C is the heat map after the attention mechanism is added. It can be verified that the features extracted by SimAM can better focus on the main target, and the effectiveness of the attention mechanism in strengthening the segmentation area and suppressing noise can be verified. The segmentation results are shown in Table 1:

[0066] Table 1 Segmentation results

[0067] Method QUR Cytoplasmic IoU Nuclear IoU U-Net control experiment 75.207％ 70.367％ 79.701％ Segmentation results 79.262％ 74.021％ 87.290％

[0068] Example 2

[0069] The network model for cellular medical image segmentation of the present invention is applicable to the segmentation of various types of cells or pathological cell nuclei. In order to prove the generalization of the present invention on various types of pathological cell data sets, a breast cancer pathological cell data set TNBC is introduced to retrain the established network model for cellular medical image segmentation. The method and training method for establishing the network model for cellular medical image segmentation are the same as those in Example 1.

[0070] The segmentation results of Example 2 were verified by sliding window preprocessing of 50 WSIs pathology images with a resolution of 512x512 from 11 patients to sample images of 256x256, which were used as inputs of the trained network model for cell medical image segmentation. The sampled single image contained multiple cells. The segmentation results are shown in Table 2:

[0071] Table 2

[0072] Method Cell IoU Segmentation results 80.975％

[0073] It can be seen from Table 2 that the network model of the present invention is also applicable to the segmentation of breast cancer pathological cells.

[0074] Example 3

[0075] The present invention also provides a system for segmenting cellular medical images. Figure 2 As shown, it includes a backbone network and a pyramid pooling module.

[0076] The backbone network is used for feature enhancement of feature maps, and extracts and reuses features of feature maps after feature enhancement, providing rich feature information for the pyramid pooling module. The backbone network consists of four densely connected network blocks (DenseCSSBlock), each of which is composed of six densely connected feature extraction sub-network blocks (CSS Block). Dense connection means that the input of each feature extraction sub-network block is the output of all feature extraction sub-network blocks before the feature extraction sub-network block. The four densely connected network blocks are reduced in dimension through 1x1 convolution. In particular, unlike the average pooling used in the PSPNet model, we use a 2x2 convolution layer to aggregate features between the first two densely connected network blocks. Instead of directly downsampling, the network optimizes and learns the appropriate convolution kernel parameters through back propagation. This improvement has improved the final segmentation effect to a certain extent.

[0077] The feature extraction subnetwork block is used for feature mining and extraction of salient regions of cell nuclei in cellular medical images. It consists of two convolutional layers and an attention mechanism layer. Each convolutional layer is connected to a Batch Norm layer and a ReLU layer by default, which introduces channel-by-channel, space-by-space, and pixel-by-pixel 3D context information without introducing additional network parameters, strengthening the segmentation part and suppressing noise. The convolution kernels of the two convolutional layers of the feature extraction subnetwork block are 1x1 and 3x3 respectively.

[0078] The system for cellular medical image segmentation of the present invention also includes an auxiliary loss layer and a main loss layer, respectively using an auxiliary loss function and a main loss function. The main loss is the main optimization direction, and the auxiliary loss helps to optimize the learning process. The model calculates the loss function and then back propagates, thereby optimizing the model parameters. The optimal model parameters are calculated by forward propagation to obtain the best segmentation result.

[0079] The pyramid pooling module is used to integrate the contextual semantic feature information of each feature map, and the feature map is obtained by encoding, decoding and backbone network at each scale, and the feature map is spliced ​​and segmented to obtain the final cell segmentation map. For the task of cell segmentation, different from the pyramid pooling module in the prior art that uses four dimensions of pooling pyramid: 1, 2, 3, 4, the present invention only uses three dimensions: 2, 3, 4, downsamples the feature map to 2x2x1, 3x3x1, 6x6x1 three scales, and then splices the channel with the feature map after the encoder, and finally obtains a large receptive field, contextual semantic information and global semantic information.

[0080] The pyramid pooling module adopted by the present invention reduces network parameters and speeds up calculation without affecting the model effect. Acquiring rich contextual relationships reduces the possibility of cytoplasm or background appearing in the cell nucleus of the segmentation result. This fusion of multi-scale semantic information extracts the features of rich high-level semantic information and low-level position information, and contains rich contextual semantic information, thereby reducing the possibility of pixel classification errors, distinguishing easily confused categories, mining small cell nucleus sample information, and improving segmentation accuracy.

[0081] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1A device that provides the functions specified in a block or multiple blocks.

[0082] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0083] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0084] The above are merely embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention are included in the scope of the claims of the present invention to be approved.

Claims

1. A method for segmenting a cellular medical image, characterized in that: The following steps are involved: The original cell image is used as the input of a pre-trained cellular medical image segmentation neural network model, and the cellular medical image segmentation neural network model performs feature enhancement and feature reuse on the cell nucleus of the image to obtain a feature image containing multiple semantics; the cellular medical image segmentation neural network model performs feature enhancement processing on the original cell image based on the importance of neurons; the backbone network used in the cellular medical image segmentation neural network model is composed of four densely connected network blocks, and the transition layers composed of convolutional layers between the densely connected network blocks control the number of channels; each of the densely connected network blocks is composed of six feature extraction sub-network blocks, and the input of each feature extraction sub-network block is the output of all feature extraction sub-network blocks before the feature extraction sub-network block, and the feature extraction sub-network block is composed of two convolutional layers and an attention mechanism layer, and each convolutional layer is connected to a Batch Norm layer and a ReLU layer by default; After upsampling and downsampling the feature maps containing multiple semantics, segmenting each upsampled feature map to obtain a segmentation result map, and superimposing the segmentation result map with the original cell map as a model output; The training samples of the pre-trained cellular medical image segmentation neural network model are single cell images with labeled background, cytoplasm, and cell nucleus.

2. The method for cellular medical image segmentation according to claim 1, characterized in that: The importance of the neuron is calculated by formula (1): Where E is the importance of the neuron, is the minimum energy function, are the mean and variance of all neurons in a channel except the target neuron, t is the target neuron of each channel, λ is the only hyperparameter, M is the number of energy functions of each channel, x i For each channel, other neurons except the target neuron.

3. The method for cellular medical image segmentation according to claim 2, characterized in that: Formula (2) is used for feature enhancement: in, is the feature map of feature enhancement, sigmoid() is the activation function, and X is the input feature map.

4. The method for cellular medical image segmentation according to claim 1, characterized in that: Formula (3) is used for feature reuse: in, is the feature map of the lth layer, H l () indicates network transformation, [] indicates the concatenation of feature maps, Represents the feature map of the l-1th layer.

5. The method for cellular medical image segmentation according to claim 1, characterized in that: The training of the cellular medical image segmentation neural network model comprises the following steps: The single cell images in the Herlev Pasteur cell dataset are used as training sets; Calculate the channel-by-channel mean of the entire training set of images and perform mean subtraction on individual images in the training set; The training set is input and trained using the weighted cross entropy loss function of four categories: background, cytoplasm, large nucleus, and small nucleus.

6. The method for cellular medical image segmentation according to claim 5, characterized in that: The weighted cross entropy loss function is calculated by formula (4): in, Represent the main loss function and auxiliary loss function respectively, β is the loss weighted balance parameter, y is the annotation result, For the final prediction result, is the prediction result of the auxiliary branch, c=4, α j is the weighting coefficient of the category.

7. A system for cellular medical image segmentation, characterized in that: It includes a backbone network and a pyramid pooling module; the backbone network is used for 3D enhancement feature enhancement extraction and reuse of sample feature maps, and provides rich semantic feature information for the pyramid pooling module; the backbone network is composed of four densely connected network blocks, and the transition layer composed of convolutional layers between the densely connected network blocks controls the number of channels; each of the densely connected network blocks is composed of six feature extraction sub-network blocks, and the input of each feature extraction sub-network block is the output of all feature extraction sub-network blocks before the feature extraction sub-network block. The feature extraction sub-network block is composed of two convolutional layers and an attention mechanism layer, and each convolutional layer is connected to a Batch Norm layer and a ReLU layer by default; The pyramid pooling module is used to fuse the contextual semantic feature information of each feature map, and each feature map is encoded and decoded at each scale and then superimposed with the original map to obtain the final cell segmentation map; The loss layer is used for learning optimization of the backbone network and pyramid pooling module; The backbone network is composed of densely connected feature extraction sub-network blocks and alternating transition layers, and the feature extraction sub-network blocks include attention mechanism layers.

8. The system for cellular medical image segmentation according to claim 7, characterized in that: The system also includes a loss layer, which includes an auxiliary loss layer and a main loss layer, both of which use weighted cross entropy loss functions of four categories: background, cytoplasm, large cell nucleus, and small cell nucleus as loss functions.

9. The system for cellular medical image segmentation according to claim 7, characterized in that: The pyramid pooling module includes three dimensions.

10. The system for cellular medical image segmentation according to claim 7, characterized in that: The first transition layer of the backbone network uses a 2x2 convolutional layer to aggregate features without directly downsampling, and allows the network to learn appropriate convolution kernel parameters through back propagation.

Images