DCNN and Transform integrated polyp segmentation method and system

Abstract

In order to solve the problem of difficulty in accurate segmentation of a polyp area in practical application, the invention discloses a DCNN and Transform integrated polyp segmentation method and system. According to the method, the local information in the polyp image is mined by using the DCNN, and the relationship between the pixels is captured by using the Transform, so that the context information in the image is better understood, and the accurate segmentation of the polyp region is realized. Besides, aiming at target objects with different shapes and sizes possibly existing in the medical image, a three-level feature fusion module is further designed to capture feature information with different scales in the image, and the features are effectively merged, so that the segmentation precision is improved. The method can achieve the precise segmentation of the polyp region, and is suitable for a scene where the polyp image is difficult to segment manually.

Description

Technical Field

[0001] The present invention relates to the technical field of deep learning medical image processing, and in particular to a polyp segmentation method and system integrating DCNN and Transformer. Background Art

[0002] Polyps are precursor lesions of some potential malignant diseases (such as colon cancer). Early detection and segmentation of polyps can prevent them from developing into cancer, thereby improving cure and survival rates. Some types of polyps may cause bleeding, obstruction or other pathological changes. Timely segmentation can prevent these complications and reduce the impact on patients' health. However, in clinical practice, manual segmentation by doctors is very time-consuming and costly, and there are also problems with doctor subjectivity and limited reproducibility. Therefore, there is an urgent need for a high-precision, fully automatic polyp image segmentation algorithm to improve clinical work efficiency and accuracy.

[0003] The development of automatic polyp image segmentation algorithm is of great significance for clinical practice. It can greatly reduce the workload of doctors, save time and resources, and provide doctors with reliable segmentation results.

[0004] Many methods have been proposed by scholars at home and abroad for the problem of polyp image segmentation. Traditional methods rely on manually defined features or algorithms based on mathematical models, but are limited by professional knowledge requirements and lack robustness to image changes. In contrast, methods based on deep convolutional neural networks (DCNNs) show better generalization, scalability, and accuracy in medical image segmentation. U-Net is a classic convolutional neural network architecture designed for medical image segmentation tasks. Its unique U-shaped structure includes an encoder and a decoder. Its variants such as unet++ and unet3+ also perform well in lesion segmentation tasks. Although DCNN has made significant progress in medical image segmentation, it still faces some challenges. First, the limited receptive field of the convolutional layer limits the model's ability to capture global contextual information. In medical image segmentation, multi-scale contextual information is crucial for accurate segmentation, but DCNN performs poorly in effectively integrating features of different scales. To solve this problem, researchers began to introduce transformers into the field of medical image segmentation. Transformers can capture the global relationship between pixels, not just local regions. The self-attention mechanism introduced significantly enhances the generalization and robustness of the model when processing medical images of different sizes and shapes. Some researchers use Swin Transformer as encoder and decoder and propose a new method called Swin-Unet. In addition, some researchers have proposed the introduction of BiFusion module in TransFuse, combining the advantages of DCNN and transformer to better integrate and utilize these two technologies. However, the fixed complexity of medical images will lead to unsatisfactory segmentation results.

[0005] Therefore, how to use a network architecture based on DCNN and Transformer to quickly and accurately segment the polyp area is a difficult problem in polyp image segmentation. To this end, the present invention discloses a polyp segmentation method and system integrating DCNN and Transformer to address this problem. Summary of the invention

[0006] In view of the above research status and existing problems, the present invention discloses a polyp segmentation method and system integrating DCNN and Transformer.

[0007] The present invention discloses a polyp segmentation method integrating DCNN and Transformer, which is characterized by: preprocessing the polyp image, constructing a three-level feature fusion module to capture feature information of different scales in the image and effectively merge these features, then inputting the images of the test data set into a network architecture based on DCNN and Transformer for training, and finally using the trained model for testing; specifically, the method includes four steps: data preprocessing, constructing a network architecture based on DCNN and Transformer, training and testing, which are as follows: Step 1, before the polyp image training sample is input into the network architecture based on DCNN and Transformer, the polyp image training sample is preprocessed in sequence: randomly rotating the image, randomly flipping the image horizontally, normalizing the image using a specified mean and standard deviation, and converting the training sample image and spleen region label into a tensor format; Step 2: Build a network architecture based on DCNN and Transformer, which includes EfficientNet encoding module, Swin Transformer encoding module, three-level feature fusion module, hybrid attention module, Swin Transformer decoding module and convolution module in the order of image processing; The EfficientNet encoding module is composed of MBConvBlocks, which include convolutional layers, batch normalization layers, Drop layers, and Swish layers. The Swin Transformer encoding module consists of a normalization layer, a multi-head self-attention module, and a multi-layer perceptron layer; The three-level feature fusion module consists of three branches and a feature splicing layer, each branch includes a hole convolution layer, a first fully connected layer, an average pooling layer, a second fully connected layer, and a third fully connected layer. The module is used to capture feature information of different scales in the image and effectively merge these features, wherein the hole convolution module includes three convolution layers of different scales, and the average pooling layer includes three adaptive average pooling layers of different sizes; The hybrid attention module consists of a channel attention module and a spatial attention module; The Swin Transformer decoding module is used to perform feature fusion and upsampling operations of the decoder part; The convolution module is used to perform feature fusion and upsampling operations in the decoder part; Step 3: Load the training data set and train according to the parameters, save the weights of the trained DCNN and Transformer-based network models, and complete the training; Step 4: Load the test data set, load the DCNN and Transformer-based network model, initialize the evaluation index of the network test model; traverse the test data set and calculate the evaluation index to complete the verification.

[0008] The present invention provides a polyp image segmentation system integrating a polyp segmentation method of DCNN and Transformer, characterized by comprising: A preprocessing module, before the polyp image training samples are input into the network architecture based on DCNN and Transformer, includes performing preprocessing operations on the polyp image training samples in sequence: randomly rotating the images, randomly flipping the images horizontally, normalizing the images using the specified mean and standard deviation, and converting the training sample images and spleen region labels into tensor formats; The network construction module builds a network architecture based on DCNN and Transformer, which includes EfficientNet encoding module, Swin Transformer encoding module, three-level feature fusion module, hybrid attention module, Swin Transformer decoding module and convolution module in the order of image processing; The EfficientNet encoding module is composed of MBConvBlocks, which include convolutional layers, batch normalization layers, Drop layers, and Swish layers. The Swin Transformer encoding module consists of a normalization layer, a multi-head self-attention module, and a multi-layer perceptron layer; The three-level feature fusion module consists of three branches and a feature splicing layer, each branch includes a hole convolution layer, a first fully connected layer, an average pooling layer, a second fully connected layer, and a third fully connected layer. The module is used to capture feature information of different scales in the image and effectively merge these features, wherein the hole convolution module includes three convolution layers of different scales, and the average pooling layer includes three adaptive average pooling layers of different sizes; The hybrid attention module consists of a channel attention module and a spatial attention module; The Swin Transformer decoding module is used to perform feature fusion and upsampling operations of the decoder part; The convolution module is used to perform feature fusion and upsampling operations in the decoder part; Image training module, loads the training data set and trains according to the parameters, saves the weights of the trained DCNN and Transformer-based network models, and completes the training; The image testing module loads the test data set, loads the network model based on DCNN and Transformer, initializes the evaluation index of the network test model; traverses the test data set and calculates the evaluation index to complete the verification.

[0009] Compared with the prior art, it has the following beneficial effects: (1) The present invention proposes a preprocessing strategy for polyp images, which includes randomly rotating the image, randomly flipping the image horizontally, normalizing the image, and converting the image into a tensor format.

[0010] (2) The present invention proposes a network architecture based on DCNN and Transformer, using DCNN to mine local information and Transformer to capture the relationship between pixels, so as to better understand the contextual information in the image.

[0011] (3) The present invention proposes a three-level feature fusion module to capture feature information of different scales within the image and effectively merge these features, thereby significantly improving the segmentation accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the prior art descriptions are briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention, and for ordinary technicians in this field, other drawings can be obtained based on the provided drawings without creative work.

[0013] Figure 1 This is a network structure diagram based on DCNN and Transformer provided in Example 1 of the present invention.

[0014] Figure 2 This is a schematic diagram of the three-level feature fusion module provided in Example 1 of the present invention. DETAILED DESCRIPTION

[0015] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0016] The first aspect of the embodiment of the present invention discloses a polyp segmentation method integrating DCNN and Transformer, including an image preprocessing stage, a network training stage and a network testing stage. The image preprocessing stage includes a data preprocessing stage including randomly rotating the image, randomly flipping the image horizontally, normalizing the image, and converting the image into a tensor format. The network training stage includes the design of a network architecture based on DCNN and Transformer and network training. The network testing stage includes inputting a polyp image into a trained convolutional neural network, outputting a predicted segmentation map, and performing a performance evaluation. Specifically, the following steps are included: This embodiment can use software to realize the automatic operation of the process. The Python platform is used to call the medical image reading function based on the Pytorch library, input the polyp image file name and the segmentation map file name corresponding to the image, and use the tiff.imread (.) function to read them respectively, and finally store the read image data in the imgs_train list, and store the read label data in the masks_train list.

[0017] In one embodiment, polyp image training samples with polyp area labels are input into a network based on DCNN and Transformer for training; before the polyp image training samples are input into the network based on DCNN and Transformer, a data preprocessing stage is also included; the data preprocessing stage includes randomly rotating the image, randomly flipping the image horizontally, normalizing the image, and converting the image into a tensor format.

[0018] In this embodiment, the preprocessing step includes randomly rotating the image.

[0019] When executing: We randomly rotate the images by 90, 180, or 270 degrees using the A.RandomRotate90(.) function from the Albumentations library to increase the diversity of the data and make the model more robust.

[0020] In this embodiment, the preprocessing step includes randomly flipping the image horizontally.

[0021] When executing: The image is randomly flipped horizontally or vertically by using the A.Flip(.) function in the Albumentations library, where the probability of performing the flip operation is set to 0.5.

[0022] In this embodiment, the preprocessing step includes normalizing the image.

[0023] When executing: By using the A.Normalize(.) function in the Albumentations library to normalize each channel of the image, that is, subtracting the mean and then dividing by the standard deviation, the image data is standardized to have zero mean and unit standard deviation, which helps to speed up the training of the model and improve convergence. Among them, Used to specify the mean of each channel, its value is [0.485, 0.456, 0.406]; Used to specify the standard deviation of each channel, its value is [0.229, 0.224, 0.225]; Used to set the maximum value of pixel value, which is 255.0; Used to set the probability of an operation, its value is 1.

[0024] In this embodiment, the preprocessing step includes converting the image into a tensor format.

[0025] When executing: The image is converted to the PyTorch tensor format using the A.ToTensorV2(.) function in the Albumentations library, and the channel order is changed from HW(C) to (C)HW, ensuring that the image data can be correctly processed by the PyTorch model.

[0026] A specific network architecture design structure is given below: The network architecture uses the U-Net type network as the basic skeleton, in which the three-level feature fusion module is designed, such as Figure 2 As shown in the figure, the designed network architecture based on DCNN and Transformer is shown in Figure 1 As shown in Figure 3, the network architecture consists of the following six parts: EfficientNet encoding module, Swin Transformer encoding module, three-level feature fusion module, hybrid attention module, Swin Transformer decoding module and convolution module.

[0027] The EfficientNet encoding module is a module based on neural architecture search (NAS); The Swin Transformer encoding module is the encoding part based on the Swin Transformer architecture; The three-level feature fusion module includes three branches and a feature splicing layer: the first branch includes a convolutional layer with a dilation rate of 1, the first fully connected layer, an average pooling layer with a parameter of 1, the second fully connected layer, the third fully connected layer and a feature fusion layer; the second branch includes a convolutional layer with a dilation rate of 3, the first fully connected layer, an average pooling layer with a parameter of 2, the second fully connected layer, the third fully connected layer and a feature fusion layer; the third branch includes a convolutional layer with a dilation rate of 5, the first fully connected layer, an average pooling layer with a parameter of 3, the second fully connected layer, the third fully connected layer and a feature fusion layer; the feature splicing layer splices features of the three scales together.

[0028] The hybrid attention module is based on the united attention (UA) block of HTC-Net; The Swin Transformer decoding module is the decoding part based on the Swin Transformer architecture; The convolution module includes a convolution layer with a kernel size of 3 and padding of 1.

[0029] In one embodiment, when training the network, the loss function used is the cross entropy loss, the optimizer function is AdamW(.), the resize is set to 224, the batch size is set to 8, and the learning rate is set to 1e-4.

[0030] In one embodiment, during the network testing phase, a predicted segmentation map is output through a network architecture based on DCNN and Transformer, and a performance evaluation is performed.

[0031] When executing: For the polyp image to be predicted, the image is input into the trained network to obtain the segmentation map of the predicted polyp image. Using the Dice coefficient as the evaluation indicator, all the final prediction results and the corresponding label values ​​are passed to the calculate_mdice(.) function, which uses the Dice calculation formula to average the Dice coefficients of all images in the entire batch and returns the final result.

[0032] The second aspect of the embodiment of the present invention discloses a polyp segmentation system integrating a DCNN and a Transformer polyp segmentation method according to the first aspect, comprising: A preprocessing module, before the polyp image training samples are input into the network architecture based on DCNN and Transformer, includes performing preprocessing operations on the polyp image training samples in sequence: randomly rotating the images, randomly flipping the images horizontally, normalizing the images using the specified mean and standard deviation, and converting the training sample images and spleen region labels into tensor formats; The network construction module builds a network architecture based on DCNN and Transformer. The architecture includes EfficientNet encoding module, Swin Transformer encoding module, three-level feature fusion module, hybrid attention module, Swin Transformer decoding module and convolution module in the order of image processing. The EfficientNet encoding module is composed of MBConvBlocks, which include convolutional layers, batch normalization layers, Drop layers, and Swish layers. The Swin Transformer encoding module consists of a normalization layer, a multi-head self-attention module, and a multi-layer perceptron layer; The three-level feature fusion module consists of three branches and a feature splicing layer, each branch includes a hole convolution layer, a first fully connected layer, an average pooling layer, a second fully connected layer, and a third fully connected layer. The module is used to capture feature information of different scales in the image and effectively merge these features, wherein the hole convolution module includes three convolution layers of different scales, and the average pooling layer includes three adaptive average pooling layers of different sizes; The hybrid attention module consists of a channel attention module and a spatial attention module; The Swin Transformer decoding module is used to perform feature fusion and upsampling operations of the decoder part; The convolution module is used to perform feature fusion and upsampling operations in the decoder part; Image training module, loads the training data set and trains according to the parameters, saves the weights of the trained DCNN and Transformer-based network models, and completes the training; The image testing module loads the test data set, loads the network model based on DCNN and Transformer, initializes the evaluation index of the network test model; traverses the test data set and calculates the evaluation index to complete the verification.

[0033] The second aspect of this embodiment can be used to execute all the method steps in the first aspect of the embodiment.

[0034] The beneficial effects of the present invention are verified by comparative experiments as follows.

[0035] The data set used in this experiment is the official data set CVC-ClinicDB of the training phase of the MICCAI 2015 Colonoscopy Video Automatic Polyp Detection Sub-Challenge, which contains 612 static images. We divided them into 490 training sets and 122 test sets at a ratio of about 8 to 2. This experiment used Swin-Unet (method 1), TransUnet (method 2), HTC-Net (method 3) and the method of the present invention for comparison. The experimental results were expressed by retaining four decimal places. The method of the present invention takes the method of the specific implementation as an example.

[0036] Evaluation index of polyp image segmentation method: Dice coefficient.

[0037] The Dice coefficient is a set similarity metric function used to measure the overlap between the predicted segmentation map and the true segmentation map. Its range is 0-1, where 0 means there is no overlap between the predicted segmentation map and the true segmentation map, and 1 means that the predicted segmentation map and the true segmentation map completely overlap.

[0038] Table 1 Comparative experimental results Dataset Method of the present invention Method 1 Method 2 Method 3 CVC-ClinicDB 0.8479 0.7258 0.7250 0.7380 As can be seen from Table 1, the method of the present invention can obtain a higher Dice coefficient on the experimental data set. Compared with the existing deep learning-based methods such as methods 1, 2 and 3, the Dice coefficient of the method of the present invention is higher, indicating that the method of the present invention has a better effect in processing polyp image segmentation.

[0039] It can be concluded that compared with most existing polyp image segmentation methods, the method of the present invention has a higher segmentation effect. The polyp image preprocessing step proposed in the present invention lays the foundation for various subsequent operations and improves the network performance and training stability. The polyp area of ​​the image is segmented by a network based on DCNN and Transformer, which significantly improves the segmentation effect of the polyp area.

[0040] The above is a detailed introduction to the polyp segmentation method and system integrating DCNN and Transformer provided by the present invention. Specific examples are used in this article to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea; at the same time, for general technicians in this field, according to the ideas of the present invention, there will be changes in the specific implementation methods and application scopes. In summary, the content of this specification should not be understood as limiting the present invention.

[0041] In this article, relational terms such as first and second, etc. are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "comprise", "include" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the statement "comprise a ..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.

Claims

1. The present invention discloses a polyp segmentation method integrating DCNN and Transformer, characterized in that: The polyp images were preprocessed, and a three-level feature fusion module was built to capture feature information of different scales in the image and effectively merge these features. Then the images of the test data set were input into the network architecture based on DCNN and Transformer for training, and finally the trained model was used for testing. The specific steps include data preprocessing, building a network architecture based on DCNN and Transformer, training, and testing, as follows: Step 1, before the polyp image training sample is input into the network architecture based on DCNN and Transformer, the polyp image training sample is preprocessed in sequence: randomly rotating the image, randomly flipping the image horizontally, normalizing the image using a specified mean and standard deviation, and converting the training sample image and spleen region label into a tensor format; Step 2: Build a network architecture based on DCNN and Transformer, which includes EfficientNet encoding module, Swin Transformer encoding module, three-level feature fusion module, hybrid attention module, Swin Transformer decoding module and convolution module in the order of image processing; The EfficientNet encoding module is composed of MBConvBlocks, which include convolutional layers, batch normalization layers, Drop layers, and Swish layers. The Swin Transformer encoding module consists of a normalization layer, a multi-head self-attention module, and a multi-layer perceptron layer; The three-level feature fusion module consists of three branches and a feature splicing layer, each branch includes a hole convolution layer, a first fully connected layer, an average pooling layer, a second fully connected layer, and a third fully connected layer. The module is used to capture feature information of different scales in the image and effectively merge these features, wherein the hole convolution module includes three convolution layers of different scales, and the average pooling layer includes three adaptive average pooling layers of different sizes; The hybrid attention module consists of a channel attention module and a spatial attention module; The Swin Transformer decoding module is used to perform feature fusion and upsampling operations of the decoder part; The convolution module is used to perform feature fusion and upsampling operations in the decoder part; Step 3: Load the training data set and train according to the parameters, save the weights of the trained DCNN and Transformer-based network models, and complete the training; Step 4: Load the test data set, load the DCNN and Transformer-based network model, initialize the evaluation index of the network test model; traverse the test data set and calculate the evaluation index to complete the verification.

2. A polyp image segmentation system according to the polyp segmentation method integrating DCNN and Transformer as described in claim 1, characterized in that: include: A preprocessing module, before the polyp image training samples are input into the network architecture based on DCNN and Transformer, includes performing preprocessing operations on the polyp image training samples in sequence: randomly rotating the images, randomly flipping the images horizontally, normalizing the images using the specified mean and standard deviation, and converting the training sample images and spleen region labels into tensor formats; The network construction module builds a network architecture based on DCNN and Transformer, which includes EfficientNet encoding module, Swin Transformer encoding module, three-level feature fusion module, hybrid attention module, Swin Transformer decoding module and convolution module in the order of image processing; The EfficientNet encoding module is composed of MBConvBlocks, which include convolutional layers, batch normalization layers, Drop layers, and Swish layers. The Swin Transformer encoding module consists of a normalization layer, a multi-head self-attention module, and a multi-layer perceptron layer; The three-level feature fusion module consists of three branches and a feature splicing layer, each branch includes a hole convolution layer, a first fully connected layer, an average pooling layer, a second fully connected layer, and a third fully connected layer. The module is used to capture feature information of different scales in the image and effectively merge these features, wherein the hole convolution module includes three convolution layers of different scales, and the average pooling layer includes three adaptive average pooling layers of different sizes; The hybrid attention module consists of a channel attention module and a spatial attention module; The Swin Transformer decoding module is used to perform feature fusion and upsampling operations of the decoder part; The convolution module is used to perform feature fusion and upsampling operations in the decoder part; Image training module, loads the training data set and trains according to the parameters, saves the weights of the trained DCNN and Transformer-based network models, and completes the training; The image testing module loads the test data set, loads the network model based on DCNN and Transformer, initializes the evaluation index of the network test model; traverses the test data set and calculates the evaluation index to complete the verification.

Images