Adenoid cell image segmentation method based on selective multi-branch hole convolution

Abstract

The application relates to a kind of gland cell image segmentation methods based on selective multi-branch hollow convolution, input gland cell image dataset, with improved UNet as main network, construct gland cell segmentation network based on selective multi-branch hollow convolution, the network is captured more advanced features with context encoding network, with channel attention module adaptive recalibration channel feature response and highlight the most relevant feature channel, with multi-scale selection module amplifies effective information and suppresses redundant information, selects the appropriate scale of hollow convolution branch;With the input gland cell image dataset, the segmentation network is trained to obtain a stable segmentation network;Input to be segmented gland cell image, with stable segmentation network to carry out gland cell segmentation.The application has better generalization ability, and is especially suitable for the size, shape change difference of gland in colorectal polyp.

Description

Technical Field

[0001] This invention belongs to the field of medical image processing technology, specifically relating to a method for segmenting glandular cell images based on selective multi-branch dilated convolution. Background Technology

[0002] Colorectal cancer is a common malignant tumor of the gastrointestinal tract, with approximately 95% of colorectal cancers being adenocarcinomas. Among malignant tumors of the digestive system, its incidence and mortality rates are second only to stomach cancer, esophageal cancer, and primary liver cancer.

[0003] Pathologists use glandular morphology to assess the grade or differentiation of colorectal cancer. Segmenting glandular instances in histological images is particularly challenging, requiring not only the detection of glands from a complex background but also precise boundary detection to isolate each instance. In clinical practice, pathologists visually analyze biopsy tissue sections under a microscope. Histopathological examination, considered the "gold standard" for diagnosing glandular lesions, is crucial for early diagnosis.

[0004] However, this work is not easy. It must address the variability of glands in shape, size, location, and texture. Furthermore, with the development of whole-slide imaging technology, the number and types of histological images are rapidly increasing, requiring the analysis of ever-growing histopathological data. It is impractical for qualified radiologists to extract crucial morphological features from large-scale histological images. The development of digital pathology has brought new methods, primarily used in computer-aided diagnostic systems. In this field, tumor type prediction and grading, tissue segmentation, and cancer cell identification have become several hot topics. Therefore, qualified radiologists urgently need to utilize computer-aided technology for histopathological image analysis, placing high demands on segmentation efficiency and reliability. This is also of great significance for reducing mortality, alleviating the burden on physicians, and improving patients' quality of life.

[0005] To accurately segment glands, various methods exist in current technology, broadly categorized into two main approaches: the first is based on traditional image processing methods, such as threshold-based and region-based segmentation; the second uses convolutional neural networks (CNNs) for segmentation. Currently, with the rapid development of deep learning, CNNs have successfully overcome the limitations of traditional hand-crafted features. These CNN-based methods have significantly improved the benchmarks of almost all widely used methods and, due to their efficiency and high performance, have gradually replaced traditional image segmentation methods. However, most existing U-Net-based methods still suffer from inaccurate gland segmentation and blurred boundaries. For example, the Deep Contour Aware Network (DCAN) architecture proposed by Chen et al. utilizes multi-level contextual features and auxiliary supervision mechanisms to handle large appearance variations and gradient vanishing problems of glands. Although effective, it is ineffective in severely degenerated glands in some malignant cases. Raza et al. proposed a Multiple Input Multiple Output Network (MIMO-Net), which visualizes input and output images at multiple resolutions. By integrating context, the network parameters can be trained well, but it lacks accuracy for glands on plaque boundaries. Graham et al. proposed a minimum information loss extended network by re-enlarging the original input image at multiple points within the network and using a shrunken spatial pyramid pooling and random transformations, but its performance in segmenting malignant glands is still poor. Summary of the Invention

[0006] To selectively learn glandular cell features from different scales, maximize image feature extraction, and improve boundary localization accuracy in segmentation tasks, this invention proposes a glandular cell image segmentation method based on selective multi-branch dilated convolution. It utilizes a context encoding network to capture more high-level features, employs a channel attention module (CA) to adaptively recalibrate channel feature responses and highlight the most relevant feature channels, and proposes a multiscale selection module (MS) to amplify effective information and suppress redundant information, thereby selecting dilated convolution branches of appropriate scales. This enables more efficient and accurate segmentation of glands at different scales.

[0007] The segmentation network proposed in this invention solves the problem of poor gland segmentation accuracy caused by irregular or severely degenerated malignant gland structures in the prior art. It not only improves segmentation accuracy but also enhances the network's generalization ability.

[0008] This invention employs the following technical solution: a method for segmenting glandular cell images based on selective multi-branch dilated convolution, the method comprising the following steps:

[0009] Step 1: Input the glandular cell image dataset;

[0010] Step 2: Using the improved UNet as the backbone network, construct a gland cell segmentation network based on selective multi-branch dilated convolution. The network captures more high-level features with a context encoding network, adaptively recalibrates the channel feature response with a channel attention module and highlights the most relevant feature channels with a channel attention module, amplifies effective information and suppresses redundant information with a multi-scale selection module, and selects dilated convolution branches of appropriate scale.

[0011] Step 3: Train the constructed segmentation network using the input glandular cell image dataset to obtain a stable segmentation network;

[0012] Step 4: Input the image of the glandular cells to be segmented, and use a stable segmentation network to segment the glandular cells.

[0013] Preferably, the network uses ResNet34 pre-trained on ImageNet as the feature encoder module, and downsamples the first four feature extractors of the feature encoder module before inputting them into the decoder module; the decoder module includes four combined decoders; the four feature extractors are configured in conjunction with the four combined decoders.

[0014] Preferably, any of the feature extractors includes a residual module, and two adjacent feature extractors are connected by a max pooling layer.

[0015] Preferably, the first feature extractor is skipped to the input layer of the fourth combined decoder;

[0016] The second feature extractor skips a connection to the input layer of the third combined decoder;

[0017] The third feature extractor is skipped to the input layer of the second combined decoder;

[0018] The fourth feature extractor is skipped to the input layer of the first combined decoder.

[0019] Preferably, the combined decoder includes a channel attention module, a multi-scale selection module, and a dense dilated convolution module connected in sequence, with convolutional layers disposed before the channel attention module and after the dense dilated convolution module.

[0020] Preferably, the channel attention module includes a global average pooling layer and a max pooling layer arranged in parallel; features are respectively input to the global average pooling layer and the max pooling layer, and the outputs of the global average pooling layer and the max pooling layer are respectively connected to a two-layer neural network. The two features output by the two two-layer neural networks are added together, and the weight coefficients are obtained by activation function; the weight coefficients are multiplied by the features input to the channel attention module and then added together to the features input to the channel attention module to output the new features processed by the channel attention module.

[0021] Preferably, the features output by the channel attention module are input into the multi-scale selection module. The input feature map is multiplied by the factor learned during training to obtain the value of each pixel in the selected feature map, and then added to the input feature map to obtain the selected features at different scales.

[0022] Preferably, the learning factor includes a convolutional layer, a batch normalization (BN) layer, and an activation function arranged sequentially; when the factor is less than a preset value, the pixel value is suppressed; when the factor is greater than the preset value, the pixel value is amplified.

[0023] Preferably, the dense dilated convolution module includes four cascaded branches, each cascaded branch including one or more convolutional layers; the first branch has a dilated convolution rate of 1, the second branch has a dilated convolution rate of 3, the third branch has a dilated convolution rate ranging from 1 to 3, and the fourth branch has a dilated convolution rate ranging from 1 to 3 and then to 5, with the receptive field sizes of each branch being 3, 7, 9, and 19 respectively; in each branch, a 1×1 convolutional correction linear activation is applied.

[0024] Preferably, in the combined decoder, transposed convolution is used to recover high-resolution features in the decoder, including consecutive 1×1 convolutions, 3×3 transposed convolutions, and 1×1 convolutions; the feature maps generated during the upsampling process are correspondingly spliced ​​with the feature maps extracted and retained during the downsampling process.

[0025] This invention relates to a glandular cell image segmentation method based on selective multi-branch dilated convolution. The method involves inputting a glandular cell image dataset, using an improved UNet as the backbone network to construct a glandular cell segmentation network based on selective multi-branch dilated convolution. This network uses a context encoding network to capture more high-level features, a channel attention module to adaptively recalibrate channel feature responses and highlight the most relevant feature channels, and a multi-scale selection module to amplify effective information and suppress redundant information, selecting dilated convolution branches of appropriate scales. The constructed segmentation network is trained using the input glandular cell image dataset to obtain a stable segmentation network. Finally, the stable segmentation network is used to segment glandular cells from the input glandular cell image dataset.

[0026] The beneficial effects of this invention are as follows:

[0027] (1) It has better generalization ability;

[0028] (2) Use the Channel Attention Module (CA) to automatically highlight relevant feature channels while suppressing irrelevant channels;

[0029] (3) Design a multi-scale selection module (MS) to amplify effective information and suppress redundant information, and select different scales;

[0030] (4) Use a context encoding network to extract features. Design a dense dilated convolution (DAC) block as a context feature extractor, which can extract features of objects of different sizes.

[0031] (5) It is particularly suitable for cases where the size and shape of glands in colorectal polyps vary greatly. Attached Figure Description

[0032] Figure 1 This is a network structure diagram of the present invention;

[0033] Figure 2 This is a structural diagram of the CA module in this invention;

[0034] Figure 3 This is a structural diagram of the MS module in this invention;

[0035] Figure 4 This is a structural diagram of the DAC module in this invention;

[0036] Figure 5 This is a flowchart of the present invention. Detailed Implementation

[0037] To illustrate the purpose, technical solution, and advantages of this invention, the following detailed description is provided in conjunction with specific embodiments and accompanying drawings.

[0038] like Figure 5 As shown, this invention relates to a method for segmenting glandular cell images based on selective multi-branch dilated convolution, comprising the following steps:

[0039] Step 1: Input the glandular cell image dataset; Input dataset X = {x1, x2, ..., x n}, where X represents the input sample in the dataset, x N ∈R 512×512 N represents the number of samples.

[0040] Step 2: Using the improved UNet as the backbone network, construct a gland cell segmentation network based on selective multi-branch dilated convolution. The network captures more high-level features with a context encoding network, adaptively recalibrates the channel feature response with a channel attention module and highlights the most relevant feature channels with a channel attention module, amplifies effective information and suppresses redundant information with a multi-scale selection module, and selects dilated convolution branches of appropriate scale.

[0041] like Figure 1As shown, the network uses an improved UNet as the backbone network, and its backbone includes an encoder-decoder structure; it uses ResNet34 pre-trained on ImageNet as the feature encoder module, and downsamples the first 4 feature extractors of the feature encoder module and inputs them into the decoder module; the decoder module includes 4 combined decoders; the 4 feature extractors and 4 combined decoders are configured in combination.

[0042] In this invention, since continuous pooling and convolution operations can lead to the loss of some spatial information, ResNet34 pre-trained on ImageNet is used as the feature encoder module. This module retains the first four feature extractors and does not have average pooling layers or fully connected layers, thus avoiding gradient vanishing and accelerating network convergence. When using max pooling layers and residual modules for downsampling operations, the size of each layer is 256×256×64, 128×128×64, 64×64×128, 32×32×256, and 16×16×512, respectively.

[0043] Each of the aforementioned feature extractors includes a residual module, with adjacent feature extractors connected by a max-pooling layer. The residual module (residual block) includes two sequentially arranged convolutional layers, with the input being the sum of the outputs of the two convolutional layers to produce the output.

[0044] The first feature extractor is skipped to the input layer of the fourth combined decoder;

[0045] The second feature extractor skips a connection to the input layer of the third combined decoder;

[0046] The third feature extractor is skipped to the input layer of the second combined decoder;

[0047] The fourth feature extractor is skipped to the input layer of the first combined decoder.

[0048] The combined decoder includes a channel attention module, a multi-scale selection module, and a dense dilated convolution module connected in sequence, with convolutional layers arranged before the channel attention module and after the dense dilated convolution module.

[0049] like Figure 2 As shown, the channel attention module includes a global average pooling layer and a max pooling layer arranged in parallel. Features are input to the global average pooling layer and the max pooling layer respectively. The outputs of the global average pooling layer and the max pooling layer are each connected to a two-layer neural network. The two features output by the two two-layer neural networks are added together, and then the weight coefficients are obtained by applying an activation function. The weight coefficients are multiplied by the features input to the channel attention module and then added together to the features input to the channel attention module to output the new features processed by the channel attention module.

[0050] In this invention, in order to make better use of the most useful feature channels, channel attention (CA) is used to automatically highlight relevant feature channels while suppressing irrelevant channels. Low-level features from the encoder and high-level features from the decoder are combined so that more relevant channels are weighted by higher coefficients.

[0051] In this invention, average pooling is used to excite feature channels and max pooling is used to retain more information. Let x represent a concatenated input feature map with C channels. First, global average pooling P is used. avg and global max pooling P max To obtain global information for each channel, the output information is represented as P. avg (x)∈R C×1×1 and P max (x)∈R C×1×1 They are then fed into a shared two-layer neural network, which includes two fully connected layers with a ReLU activation function between them, using a multilayer perceptron M. r To obtain the channel attention coefficient β∈[0,1] C×1×1 M r It is implemented using two fully connected layers. The first layer has C / r output channels, and the second layer has C output channels. Considering the trade-off between performance and computational cost, r is set to 2. Here, M... r It was P avg and P max For shared features, the results are summed and then input into the sigmoid function to obtain β (weight coefficient). Finally, the weight coefficient is multiplied by the features of the input channel attention module and then added to the features of the input channel attention module to obtain the scaled new features.

[0052] In this invention, the output of the CA module is y CA =x·β+x.

[0053] like Figure 3 As shown, the feature x output by the channel attention module is input into the multi-scale selection module. The input feature map is multiplied by the factor learned during training to obtain the value of each pixel in the selected feature map, and then added to the input feature map to obtain the selected features at different scales.

[0054] The learning factor includes a sequentially set convolutional layer, a batch normalization (BN) layer, and an activation function; when the factor is less than a preset value, the pixel value is suppressed; when the factor is greater than the preset value, the pixel value is amplified.

[0055] In this invention, a multi-scale selection module (MS) is used to amplify effective information and suppress redundant information based on factors learned during training. The operating formula is as follows:

[0056]

[0057] cbt(x) = tanh(bn(conv(x))

[0058] The value of each pixel in the selected feature map s is obtained by multiplying the input feature map x by the factor learned during training. Since the output of the activation function tanh ranges from -1 to 1, the range of cbt(x)+1 is from 0 to 2. Therefore, the preset value here is 1. When the factor is less than 1 or greater than 1, the pixel value is suppressed or amplified. It is used to select features at different scales to focus on important information rather than all information.

[0059] like Figure 4 As shown, the dense dilated convolution module includes four cascaded branches, each of which includes one or more convolutional layers; the first branch has a dilated convolution rate of 1, the second branch has a dilated convolution rate of 3, the third branch has a dilated convolution rate ranging from 1 to 3, and the fourth branch has a dilated convolution rate ranging from 1 to 3 and then to 5, with the receptive field sizes of each branch being 3, 7, 9, and 19, respectively; in each branch, a 1×1 convolutional correction linear activation is applied.

[0060] In this invention, since the size and shape of glands in colorectal polyps vary greatly, a context coding network is used to extract features. Dense dilated convolution (DAC) blocks are used as context feature extractors to extract features of objects of different sizes. The process is as follows: the DAC module has four cascaded branches. The first branch has a dilation rate of 1, the second branch has a dilation rate of 3, the third branch has a dilation rate ranging from 1 to 3, and the fourth branch has a dilation rate ranging from 1 to 3 and then to 5. The receptive field sizes of each branch are 3, 7, 9, and 19, respectively.

[0061] In this invention, a 1×1 convolution is applied in each branch to correct linear activation.

[0062] In this invention, generally speaking, when the receptive field is smaller than a single gland cell, too much local information is obtained, leading to the loss of global information and affecting the recognition rate. Since glands vary in size, four receptive fields of different sizes are used in parallel to extract feature information of different sizes, taking into account gland cells of different sizes. Furthermore, dilated convolution is used instead of ordinary convolution to obtain a larger receptive field, while reducing the number of parameters and improving the generalization ability of the extracted network.

[0063] In the combined decoder, transposed convolution (deconvolution) is used to recover high-resolution features in the decoder. The transposed convolution includes consecutive 1×1 convolutions, 3×3 transposed convolutions, and 1×1 convolutions. The feature maps generated during the upsampling process are correspondingly concatenated with the feature maps extracted and retained during the downsampling process.

[0064] In this invention, transposed convolution is used to recover high-resolution features in the decoder, which mainly includes consecutive 1×1 convolution, 3×3 transposed convolution and 1×1 convolution; at the same time, the feature maps generated by the decoder during the upsampling process are correspondingly concatenated with the feature maps extracted and retained by the encoder during the downsampling process to achieve multi-level feature fusion and improve pixel prediction accuracy.

[0065] Step 3: Train the constructed segmentation network using the input glandular cell image dataset to obtain a stable segmentation network;

[0066] Step 4: Input the image of the glandular cells to be segmented, and use a stable segmentation network to segment the glandular cells.

[0067] This invention uses colonic tissue pathology images as input to train the designed model to obtain parameters, and then uses the trained parameters to input test set images to predict the final segmentation result.

[0068] 1) Simulation conditions

[0069] The experiment used a workstation configured with an Intel(R) Xeon(R) Gold 6161 CPU @ 2.20GHz (two processors), 64GB of RAM, a Windows operating system, and two Nvidia GTX 3080Ti graphics cards. The model was implemented based on the PyTorch deep learning framework, version 1.8.0, and Python version 3.7. The input image size was uniformly adjusted to 512×512, and a multi-scale training strategy was adopted. The ADAM algorithm was used to optimize the overall parameters, with a learning rate set to 1e-4. This invention uses the Warwick-QU dataset as the research dataset. It contains 165 colon tissue images obtained using a Zeiss MIRAX MIDI microscope, with each image having been realistically annotated by medical experts. These 165 images have histological grading from benign to malignant, including 85 training images (37 benign and 48 malignant) and 80 test images (37 benign and 43 malignant). Furthermore, the test images were divided into two test sets: Test A (60 images) and Test B (20 images). This invention compares the proposed method with seven medical image segmentation methods: FCN, UNet, DCAN, MIMIONet, SegNet, DeepLab, and MILD-Net.

[0070] 2) Simulation Results

[0071] The method of this invention was compared with seven medical image segmentation methods—FCN, UNet, DCAN, MIMIONet, SegNet, DeepLab, and MILD-Net—on the Glas dataset. The performance of the method was evaluated using the same evaluation criteria as the MICCAI Glas Challenge, including F1 score, Mean Dice, and target-level Hausdorff distance. The F1 score measures the detection accuracy of a single gland target; the Dice index measures the similarity between two sets of samples; and the Hausdorff distance measures the boundary-based segmentation accuracy.

[0072] Table 1. Performance test results of the method of the present invention compared with seven other medical image segmentation methods.

[0073]

[0074] As can be seen from Table 1, the method of the present invention has better results than other methods, greatly improving the segmentation performance, especially the generalization ability, and can segment glands of different grades well. It can be better applied to gland segmentation and has better practical engineering application value.

[0075] The content not described in detail in this document is existing technology known to those skilled in the art.

Claims

1. A gland cell image segmentation method based on selective multi-branch hollow convolution, characterized by: The method comprises the following steps: Step 1: input the gland cell image data set; Step 2: build a gland cell segmentation network based on selective multi-branch hollow convolution with an improved UNet as the backbone network, which captures more high-level features with a context encoding network, adaptively recalibrates channel feature responses and highlights the most relevant feature channels with a channel attention module, and amplifies effective information and suppresses redundant information with a multi-scale selection module, and selects different scale hollow convolution branches; The network uses ResNet34 pre-trained on ImageNet as a feature encoder module, uses the first four feature extractors of the feature encoder module for down-sampling, and inputs a decoder module; the decoder module comprises four combined decoders, each combined decoder comprises a channel attention module, a multi-scale selection module and a dense hollow convolution module connected in sequence, and a convolution layer is arranged before the channel attention module and after the dense hollow convolution module; the features output by the channel attention module are input into the multi-scale selection module, the input feature maps are multiplied by a learned factor during training, and the value of each pixel in the selected feature map is obtained and then added to the input feature map to obtain the selected features of different scales; the learned factor comprises a convolution layer, a BN layer and an activation function arranged in sequence; when the factor is less than a preset value, the pixel value is suppressed; when the factor is greater than the preset value, the pixel value is amplified; the four feature extractors are arranged in cooperation with the four combined decoders; Step 3: train the constructed segmentation network with the input gland cell image data set to obtain a stable segmentation network; Step 4: input the gland cell image to be segmented into the stable segmentation network for gland cell segmentation.

2. The gland cell image segmentation method based on selective multi-branch hollow convolution according to claim 1, characterized in that: Any of the feature extractors comprises a residual module, and two adjacent feature extractors are connected through a max-pooling layer.

3. The gland cell image segmentation method based on selective multi-branch hollow convolution according to claim 1, characterized in that: The first feature extractor is connected to the input layer of the fourth combined decoder in a skip connection manner; The second feature extractor is connected to the input layer of the third combined decoder in a skip connection manner; The third feature extractor is connected to the input layer of the second combined decoder in a skip connection manner; The fourth feature extractor is connected to the input layer of the first combined decoder in a skip connection manner.

4. The gland cell image segmentation method based on selective multi-branch hollow convolution according to claim 1, characterized in that: The channel attention module comprises a global average pooling layer and a max-pooling layer arranged side by side; the features are input into the global average pooling layer and the max-pooling layer respectively, the output ends of the global average pooling layer and the max-pooling layer are connected to a two-layer neural network respectively, two features output by the two two-layer neural networks are added together, and a weight coefficient is obtained through an activation function; the weight coefficient is multiplied by the features input into the channel attention module, and then added to the features input into the channel attention module to output new features processed by the channel attention module.

5. The gland cell image segmentation method based on selective multi-branch hollow convolution according to claim 1, characterized in that: The dense hollow convolution module comprises 4 cascaded branches, and each cascaded branch comprises one or more convolution layers; the hollow convolution rate of the first branch is 1, the hollow convolution rate of the second branch is 3, the hollow convolution rate of the third branch is from 1 to 3, and the hollow convolution rate of the fourth branch is from 1 to 3 to 5; the receptive field size of each branch is 3, 7, 9 and 19 respectively; in each branch, a 1*1 convolution is applied to correct linear activation.

6. The gland cell image segmentation method based on selective multi-branch hollow convolution according to claim 1, characterized in that: In the combined decoder, transpose convolution is used to restore high-resolution features in the decoder, comprising a continuous 1*1 convolution, a 3*3 transpose convolution and a 1*1 convolution; in the upsampling process, each feature map is correspondingly spliced in the channel with the feature map extracted and reserved in the downsampling process.

Images