A polyp segmentation method based on split attention network combined with TRW-S algorithm

The polyp segmentation method using a split attention multi-scale aggregation network and TRW-S algorithm addresses the limitations of existing algorithms by enhancing edge detection and generalization, achieving superior performance on colorectal polyp segmentation.

CN114359214BActive Publication Date: 2024-12-03MIANYANG LIDE ELECTRONICS CO LTD
View PDF 2 Cites 0 Cited by

Patent Information

Application Number
CN202111662906.6
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2021-12-30
Publication Date
2024-12-03
Estimated Expiration
2041-12-30

AI Technical Summary

Technical Problem

Existing colorectal polyp segmentation algorithms, such as UNet and its variants, struggle with weak expression power for polyp images, leading to poor generalization ability and difficulty in clearly segmenting the edges of lesions, especially tiny ones.

Method used

A polyp segmentation method combining a split attention multi-scale aggregation network with a TRW-S algorithm, utilizing ResNeSt-50 as the backbone, multi-layer branch receptive field modules, dense aggregation, and test-time data augmentation, followed by Markov random field-based edge smoothing.

Benefits of technology

The method achieves improved segmentation accuracy with mDice of 91.6% and mIoU of 86.3% on public datasets, demonstrating enhanced generalization ability and clear edge detection of both large and small polyps.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN114359214B_ABST
    Figure CN114359214B_ABST
Patent Text Reader

Abstract

A polyp segmentation method based on a split attention network combined with a TRW‑S algorithm comprises the following steps: step 1: collecting several data sets of polyp segmentation; step 2: performing data enhancement on the data set; step 3: dividing the data set; step 4: constructing a split attention multi-scale aggregation network; step 5: putting the data set divided in step 3 into the network established in step 4 for training; step 6: performing test-time data enhancement (TTA) on the test set and inputting the trained network in step 5 to obtain a prediction graph; step 7: inputting the prediction graph into a sequential tree weighted information transfer algorithm (TRW‑S) based on a Markov random field for edge smoothing; step 8: outputting a predicted image; step 9: constructing an evaluation system to evaluate the segmentation results.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention belongs to the technical field of image segmentation, and in particular relates to a polyp segmentation method combining a split attention multi-scale aggregation network with a sequential tree weighted information propagation (TRW-S) algorithm. Background Art

[0002] Colorectal cancer is the third most common cancer in the world, and colon polyps are the early pathological features of colorectal cancer. Colonoscopy is an important means and method for screening and preventing colorectal cancer. Through colonoscopy, the shape and location of colorectal polyps can be obtained, so that they can be removed before they develop into colorectal cancer. However, the segmentation and positioning of polyps are often difficult. First, the shape, color, and size of polyps are often different; second, the boundary between polyps and surrounding mucosa is often not clear enough, which affects the doctor's judgment of polyps. Therefore, the segmentation of colon polyps in clinical practice often needs to be positioned by experienced doctors, and there is a certain probability of misdiagnosis; the use of computer vision technology, machine learning, and deep learning for polyp segmentation has become an important auxiliary medical method.

[0003] The deep learning medical image segmentation algorithm based on the fully convolutional network has gradually replaced the traditional method of distinguishing polyps from surrounding normal tissues based on the combination of features such as color, texture, appearance, and shape, forming a new and effective scientific way to segment polyps. For the segmentation of medical images, the UNet fully convolutional deep learning method is often used. The network uses the encoding-decoding model to achieve full pixel classification, and uses jump connections to retain shallow detail information and deep spatial information. The UNet network has achieved good segmentation results in medical image segmentation because it integrates feature information at different depths. However, the boundary between the medical image feature information of polyps and the surrounding normal tissues is not obvious, and the structure is relatively complex. The UNet encoding structure cannot extract the feature information of polyps well. Therefore, the direct application of the UNet network to segment polyp images is not effective. The subsequent networks such as UNet++ and ResUNet++ have been used to improve the segmentation effect of polyps to a certain extent, but they have the same problem as UNet, that is, they cannot clearly segment the edge of the lesion, and some tiny lesions cannot be detected at all. Many researchers have subsequently proposed other colorectal polyp segmentation algorithms, including the reverse attention network PraNet proposed by Deng-Ping Fan et al., the DoubleU-Net network proposed by Debesh Jha et al. Summary of the invention

[0004] The purpose of the present invention is to solve the technical problem that the backbone network of the network adopted by the existing colorectal polyp segmentation algorithm has weak expression power for polyp images, and even ultimately leads to weak generalization ability of the model and poor prediction ability on unknown data sets other than training data sets.

[0005] A polyp segmentation method combining a split attention multi-scale aggregation network and a TRW-S algorithm comprises the following steps:

[0006] Step 1: Collect several polyp segmentation datasets;

[0007] Step 2: Perform data augmentation on the dataset;

[0008] Step 3: Divide the data set;

[0009] Step 4: Build a split attention multi-scale aggregation network;

[0010] Step 5: Put the data set divided in step 3 into the network established in step 4 for training;

[0011] Step 6: Perform test-time data augmentation (TTA) on the test set and input it into the network trained in step 5 to obtain the prediction graph;

[0012] Step 7: Input the predicted image into the Markov random field based sequential tree weighted information transfer algorithm (TRW-S) for edge smoothing;

[0013] Step 8: Output the predicted image.

[0014] Step 9: Build an evaluation system to evaluate the segmentation results.

[0015] In step 4, the colorectal polyps are segmented using an encoding-decoding method, which specifically includes the following steps:

[0016] 4-1) Use ResNeSt-50 coding structure as the backbone network for network coding;

[0017] 4-2) Use multi-layer branch receptive field module (RFB) and dense aggregation module (Dense Aggregation) to fuse context information;

[0018] 4-3) Adopt fast decoding structure with 8 times upsampling.

[0019] In step 4-2), the output A1 of stage 2, the output A2 of stage 3, and the output A3 of stage 4 in the ResNeSt-50 encoding structure are obtained respectively. A1 obtains the feature map B1 through the RFB module, A2 obtains the feature map B2 through the RFB module, and A3 obtains the feature map B3 through the RFB module. B3 obtains C3-2 through 2x upsampling, 3×3 convolution and BN layer, B3 obtains C3-1 through 4x upsampling, 3×3 convolution and BN layer, and B2 obtains C3-2 through 2x upsampling, 3×3 convolution and BN layer. ×3 convolution and BN layer obtain C2-1, B2 and C3-2 are multiplied pixel by pixel to obtain D2, B1 and C2-1 are multiplied pixel by pixel and then with C3-1 to obtain D1, D2 is then feature fused with C3-2 to obtain E2, E2 undergoes 3×3 convolution, BN layer, 2x upsampling, 3×3 convolution and BN layer to obtain E2-1, D1 and E2-1 are feature fused to obtain F1, and F1 finally passes through a 3x3 convolution layer and BN to obtain the output G1.

[0020] In step 5, during training, several training polyp images are input for multiple rounds of training, and multi-scale training is used as an image augmentation method, and convergence is achieved after multiple rounds of training.

[0021] In step 6, several images in the data set are input into the model to obtain verification results, and test-time image enhancement (TTA) is used during verification to improve the detection effect.

[0022] In step 6, when the test set is augmented during testing and input into the network trained in step 5 to obtain the prediction graph, the following steps are taken:

[0023] 6-1) Flip the test set horizontally into the model to get the predicted image and then flip it back

[0024] 6-2) Flip the test set vertically, input it into the model to get the predicted image, and then flip it back

[0025] 6-3) Directly input the test set into the model to obtain the predicted image

[0026] 6-4) Sum and average the prediction results of the above three test sets

[0027] In step 7, a Markov random field is constructed in the output result of the test-time data augmentation (TTA) to smooth the binary terms, thereby improving the edge continuity of the segmentation effect and the consistency within the segmentation region; specifically, the following steps are included:

[0028] 7-1) Construct a Markov random field for the grayscale prediction graph generated by deep learning, take each pixel of the image as a node of the Markov random field, and use the pixel grayscale value as the unary weight of the node;

[0029] 7-2) Construct a sequential tree weighted information transfer algorithm (TRW-S) based on Markov random fields to perform binary smoothing on the segmentation edge, where the unary weight is the pixel value of the segmentation result generated by the segmentation network

[0030] In step 9, an evaluation system is constructed to evaluate this method; the mean intersection over union (mIoU), mean confidence (mDice), structural measure (Smeasure) and mean absolute error (MAE) are used as objective evaluation indicators to quantitatively evaluate the segmentation effect.

[0031] Compared with the prior art, the present invention has the following technical effects:

[0032] 1) The model proposed in the present invention is used for the public data set Kvasir-SEG for colorectal polyps. Experiments show that this method can achieve mDice (91.6%) and mIoU (86.3%), which is ahead of the currently popular polyp segmentation algorithms. In addition, the model of the present invention achieves good segmentation effects when tested on the unknown data sets ETIS-LaribPolypDB and ColonDB, which proves that the model has good generalization ability. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] The present invention will be further described below in conjunction with the accompanying drawings and embodiments:

[0034] Figure 1 is a flow chart of the present invention;

[0035] Figure 2 It is a schematic diagram of the structure of the network in the present invention;

[0036] Figure 3 This is the segmentation result diagram tested on the Kvasir-SEG dataset;

[0037] Figure 4 This is the segmentation result diagram tested on the ETIS-LaribPolypDB dataset; DETAILED DESCRIPTION

[0038] like Figures 1 to 4 As shown, a polyp segmentation method combining a split attention multi-scale aggregation network and a TRW-S algorithm includes the following steps:

[0039] Step 1: Collect several polyp segmentation datasets;

[0040] Step 2: Perform data augmentation on the dataset;

[0041] Step 3: Divide the data set;

[0042] Step 4: Build a split attention multi-scale aggregation network;

[0043] Step 5: Put the data set divided in step 3 into the network established in step 4 for training;

[0044] Step 6: Perform test-time data augmentation (TTA) on the test set and input it into the network trained in step 5 to obtain the prediction graph;

[0045] Step 7: Input the predicted image into the Markov random field based sequential tree weighted information transfer algorithm (TRW-S) for edge smoothing;

[0046] Step 8: Output the predicted image.

[0047] Step 9: Build an evaluation system to evaluate the segmentation results.

[0048] The dataset was divided in step 2; the training set has 550 CVC-ClinicDB images, 900 Kvasir-SEG images, and the test set has 100 Kvasir-SEG, 62 CVC-ClinicDB images, 380 ColonDB images, 196 ETIS-LaribPolypDB images, and 60 CVC-300 images. The dataset sizes are shown in Tables 1 and 2.

[0049] Table 1 Five data sets used in the test

[0050]

[0051] Table 2 Distribution of training set and test set

[0052]

[0053] In step 4, the colorectal polyps are segmented using an encoding-decoding method, which specifically includes the following steps:

[0054] 4-1) Use ResNeSt-50 coding structure as the backbone network for network coding;

[0055] 4-2) Use multi-layer branch receptive field module (RFB) and dense aggregation module (Dense Aggregation) to fuse context information;

[0056] 4-3) Adopt fast decoding structure with 8 times upsampling.

[0057] In step 4-2), the output A1 of stage 2, the output A2 of stage 3, and the output A3 of stage 4 in the ResNeSt-50 encoding structure are obtained respectively. A1 obtains the feature map B1 through the RFB module, A2 obtains the feature map B2 through the RFB module, and A3 obtains the feature map B3 through the RFB module. B3 obtains C3-2 through 2x upsampling, 3×3 convolution and BN layer, B3 obtains C3-1 through 4x upsampling, 3×3 convolution and BN layer, and B2 obtains C3-2 through 2x upsampling, 3×3 convolution and BN layer. ×3 convolution and BN layer obtain C2-1, B2 and C3-2 are multiplied pixel by pixel to obtain D2, B1 and C2-1 are multiplied pixel by pixel and then with C3-1 to obtain D1, D2 is then feature fused with C3-2 to obtain E2, E2 undergoes 3×3 convolution, BN layer, 2x upsampling, 3×3 convolution and BN layer to obtain E2-1, D1 and E2-1 are feature fused to obtain F1, and F1 finally passes through a 3x3 convolution layer and BN to obtain the output G1.

[0058] In step 5, the model is trained; 1450 training polyp images are input for 30 rounds of training, and multi-scale training is used as the image augmentation method. That is, 352×352 images are input into the model at 0.75 times, 1 times, and 1.25 times their sizes for training, which can achieve good data augmentation effects and shorten training time. RAdam is used as the loss function, the learning rate is 1e-4, and convergence is achieved after 30 rounds of training.

[0059] In step 6, the test sets are 100 test images in the Kvasir-SEG dataset, 62 test images in the CVC-ClinicDB dataset, 380 test images in the ColonDB dataset, 196 test images in the ETIS-LaribPolypDB dataset, and 60 test images in the CVC-300 dataset. When the test sets are enhanced for testing and input into the network trained in step 5 to obtain the prediction graph, the following steps are taken:

[0060] 6-1) Flip the test set horizontally into the model to get the predicted image and then flip it back

[0061] 6-2) Flip the test set vertically, input it into the model to get the predicted image, and then flip it back

[0062] 6-3) Directly input the test set into the model to obtain the predicted image

[0063] 6-4) Sum and average the prediction results of the above three test sets

[0064] In step 7, a Markov random field is constructed for the output image of the test data enhancement (TTA) to perform smoothing on the binary terms, so as to improve the edge continuity of the segmentation effect and the consistency within the segmentation area; specifically, the following steps are included:

[0065] 7-1) Construct a Markov random field for the grayscale prediction graph generated by deep learning, take each pixel of the image as a node of the Markov random field, and use the pixel grayscale value as the unary weight of the node;

[0066] 7-2) Construct a sequential tree weighted information transfer algorithm (TRW-S) based on Markov random fields to perform binary smoothing on the segmentation edges, where the unary weight is the pixel value of the segmentation result generated by the segmentation network.

[0067] In step 9, an evaluation system is constructed to evaluate this method; the mean intersection over union (mIoU), mean confidence (mDice), structural measure (Smeasure) and mean absolute error (MAE) are used as objective evaluation indicators to quantitatively evaluate the segmentation effect.

[0068] In order to illustrate the segmentation effect of the model constructed by the present invention, the comparative experiment compares the model of the present invention with network models for polyp image segmentation such as U-Net, U-Net++, ResUnet++, DoubleUNet, SFA, PraNet, etc. qualitatively and quantitatively to verify the segmentation effect of the model of the present invention; and compares the segmentation effect of the model of the present invention with the model composed of the encoding-decoding module in the model of this paper and the model composed of the encoding-decoding module + TTA in this paper to verify the rationality of the construction of the model of the present invention. The experimental results show that the algorithms in this paper are stronger than the benchmark algorithms. Compared with the benchmark PraNet algorithm, mDice is improved by 1.8%, mIoU is improved by 2.3%, Smeasure is improved by 1.0%, and MAE is reduced by 0.7%. In addition, the algorithm in this paper has a good generalization ability on unknown data sets that are not training sets. As shown in Table 3.

[0069] Table 3 Results of training on CVC-ClinicDB and Kvasir-SEG datasets and testing on Kvasir-SEG datasets

[0070]

[0071] The segmentation effect of this model in the Kvasir-SEG dataset test is as follows Figure 3 As shown; Figure 3The segmentation results of seven methods, including UNet, UNet++, SFA, PraNet, the encoding-decoding model of this paper, the encoding-decoding model + TTA model, and the model of the present invention, are given; from the perspective of segmentation performance, the model of the present invention has a good segmentation effect not only for large-scale polyp lesions, but also for small-area polyp lesions. The segmentation effect of the algorithm in this paper has clear edges and good internal consistency in the segmented area.

[0072] Table 4 shows the results of training with 1450 training images, including 550 images from the CVC-ClinicDB dataset and 900 images from the Kvasir-SEG dataset, and then testing with 62 images from the CVC-ClinicDB dataset, 380 images from the ColonDB dataset, 196 images from the ETIS-LaribPolypDB dataset, and 60 images from the CVC-300 dataset. The quantitative evaluation indicators of the test results show that the segmentation effect of the proposed algorithm on these four datasets has improved, especially on the high-resolution dataset ETIS-LaribPolypDB, compared with the benchmark PraNet algorithm, mDice has increased by 16.4% and mIoU has increased by 14.5%. The segmentation effect on unknown datasets other than training sets shows that the proposed model has good generalization ability and has a good ability to segment small lesions.

[0073] Table 4 Comparative test results of training on CVC-ClinicDB and Kvasir-SEG datasets and on CVC-ClinicDB, ColonDB, ETIS-LaribPolypDB, and CVC-300 datasets

[0074]

[0075]

[0076] Figure 4 This is the comparison result tested on the high-resolution dataset ETIS-LaribPolypDB. It can be clearly seen that the algorithm in this paper has excellent detection ability for tiny lesions. This is attributed to the multi-scale receptive field module used in the model in this paper. Since tiny lesions are very similar to the mucosa of the surrounding normal tissue, a backbone network with a receptive field selection kernel can well detect tiny polyp lesions. Figure 4 It can be seen that the proposed model can accurately locate small lesions, while models such as PraNet can hardly find the location of small lesions and the segmentation effect is not ideal. In addition, the proposed model can also clearly segment the edges of large lesions, which is mainly due to the TRW-S algorithm constructed in the proposed model. Therefore, the performance improvement of the model on high-resolution data sets is the most obvious.

[0077] This paper proposes a fully convolutional network with an encoding-decoding structure using ResNeSt-50 as the backbone network. The network integrates split attention modules, multi-scale receptive fields, and dense aggregation modules for polyp image segmentation. During testing, test-time image enhancement (TTA) is used to improve the prediction effect and enhance the generalization ability of the network. Finally, a sequential tree weighted information transfer (TRW-S) algorithm based on MRF is constructed to post-process the grayscale image output by TTA. The encoding part of the network uses ResNeSt-50 as the backbone network. This structure can make attention span feature map groups and stack these scattered attentions in the style of ResNet; the decoding part of the network uses multi-layer receptive field modules (RFB) in the bottom three layers to obtain multi-scale information, and uses dense aggregation modules to integrate their outputs; finally, combined with the idea of ​​fast decoding, the decoded information output is realized. When the output information generates a predicted image, test-time image enhancement (TTA) is used to improve the prediction accuracy and model generalization ability. The sequential tree weighted information transfer algorithm constructed by the present invention is used to post-process the predicted image output by TTA, which improves the continuity of the segmentation edge and can make up for the errors of incoherent segmentation within the convolutional network segmentation result.

Claims

1. A polyp segmentation method based on a split attention network combined with a TRW-S algorithm, characterized in that: It includes the following steps: Step 1: Collect several polyp segmentation datasets; Step 2: Perform data augmentation on the dataset; Step 3: Divide the data set; Step 4: Construct a split attention multi-scale aggregation network, and then use the constructed network to segment the image of colorectal polyps; Step 5: Put the data set divided in step 3 into the network established in step 4 for training; Step 6: Perform data augmentation TTA on the test set and input it into the network trained in step 5 to obtain the prediction graph; Step 7: Input the predicted image into the Markov random field based sequential tree weighted information transfer algorithm TRW-S for edge smoothing; Step 8: Output the predicted image; Step 9: Build an evaluation system to evaluate the segmentation results; In step 4, the colorectal polyps are segmented using an encoding-decoding method, which specifically includes the following steps: 4-1) Use ResNeSt-50 coding structure as the backbone network for network coding; 4-2) Use multi-layer branch receptive field module RFB and dense aggregation module Dense Aggregation to fuse context information; 4-3) Adopt fast decoding structure with 8x upsampling; In step 7, a Markov random field is constructed in the data enhancement TTA output result during the test to smooth the binary terms, thereby improving the edge continuity of the segmentation effect and the consistency within the segmentation area; specifically, the following steps are included: 7-1) Construct a Markov random field for the grayscale prediction graph generated by deep learning, treat each pixel of the image as a node in the Markov random field, and use the pixel grayscale value as the unary weight of the node; 7-2) Construct a sequential tree weighted information transfer algorithm TRW-S based on Markov random fields to perform binary smoothing on the segmentation edges, where the unary weight is the pixel value of the segmentation result generated by the segmentation network.

2. The method according to claim 1, characterized in that In step 4-2), the output A1 of stage 2, the output A2 of stage 3, and the output A3 of stage 4 in the ResNeSt-50 encoding structure are obtained respectively. A1 obtains the feature map B1 through the RFB module, A2 obtains the feature map B2 through the RFB module, and A3 obtains the feature map B3 through the RFB module. B3 obtains C3-2 through 2x upsampling, 3×3 convolution, and BN layer. B3 obtains C3-1 through 4x upsampling, 3×3 convolution and BN layer, B2 obtains C2-1 through 2x upsampling, 3×3 convolution and BN layer, B2 and C3-2 are multiplied pixel by pixel to obtain D2, B1 and C2-1 are multiplied pixel by pixel and then multiplied pixel by pixel with C3-1 to obtain D1, D2 is then feature fused with C3-2 to obtain E2, E2 is subjected to 3×3 convolution, BN layer, 2x upsampling, 3×3 convolution and BN layer to obtain E2-1, D1 and E2-1 are feature fused to obtain F1, and F1 is finally passed through a 3x3 convolution layer and BN to obtain the output G1.

3. The method according to claim 1, characterized in that In step 5, during training, several training polyp images are input for multiple rounds of training, and multi-scale training is used as an image augmentation method, and convergence is achieved after multiple rounds of training.

4. The method according to claim 1, characterized in that: In step 6, several images in the data set are input into the model to obtain verification results, and test-time image enhancement TTA is used during verification to improve the detection effect.

5. The method according to claim 1, characterized in that In step 6, when the test set is augmented during testing and input into the network trained in step 5 to obtain the prediction graph, the following steps are taken: 6-1) Flip the test set horizontally, input it into the model to get the predicted image, and then flip it back; 6-2) Flip the test set vertically, input it into the model to get the predicted image, and then flip it back; 6-3) Directly input the test set into the model to obtain the predicted image; 6-4) Sum and average the prediction results of the above three test sets.

6. The method according to claim 1, characterized in that It also includes step 9: constructing an evaluation system to evaluate the segmentation results; constructing an evaluation system for evaluation in step 9; using the average intersection over union (mIoU), average confidence (mDice), structural measure (Smeasure) and mean absolute error (MAE) as objective evaluation indicators to quantitatively evaluate the segmentation effect.

Citation Information

Patent Citations

  • Image segmentation method based on lightweight recursive non-local self-attention

    CN112132844A

  • Melanoma segmentation method based on cavity convolution and multi-scale fusion

    CN112446890A