A Polyp Segmentation Method Based on Edge Guidance and Deep Supervision PVT

Abstract

This invention discloses a polyp segmentation method based on edge guidance and deep-supervised PVT. For colonoscopy images containing polyps, this method utilizes a constructed polyp segmentation network to output the location and shape of the polyps. The polyp segmentation network uses PVT-v2 as the backbone network for feature extraction and also includes a context fusion decoding branch, a boundary extraction branch, a boundary awareness module, and a cascaded feature fusion branch. The context fusion decoding branch acts as an image decoder, receiving the semantic feature map output by the PVT-v2 backbone network. The boundary extraction branch uses a thinning module to enrich the content of the detail feature map, obtaining edge prediction results. The boundary awareness module fuses the thinned features with the output feature map of the context fusion decoding to obtain a boundary-aware prediction result. The cascaded feature fusion branch fuses features at different scales layer by layer in a deep-to-shallow order, outputting the final polyp segmentation prediction result P5.

Description

Technical Field

[0001] This invention belongs to the field of image processing technology, specifically computer vision and medical image segmentation, and specifically relates to a polyp segmentation method based on edge guidance and deep supervised PVT. Background Technology

[0002] Automated polyp segmentation can quickly and accurately segment polyp regions from colonoscopy images containing polyps, providing auxiliary reference information for subsequent analysis by physicians and reducing their workload. Automated polyp segmentation has been extensively studied over the past few decades. Current automated polyp segmentation methods are mainly divided into traditional methods and deep learning-based methods. Traditional methods typically focus on the spatial domain information of polyps, such as shape, color, or texture, and then automatically search for matching sites in colonoscopy images based on pre-defined spatial domain information. Traditional polyp segmentation methods have made significant progress in automated polyp segmentation, but their reliance on pre-defined knowledge to find polyps makes them ineffective when faced with polyps of varying shapes and colors. Due to a lack of generalization ability, these methods still suffer from a high false negative rate.

[0003] With the development of convolutional neural networks, deep learning-based polyp segmentation methods have achieved significant breakthroughs in recent years, becoming a cutting-edge technology in automatic polyp segmentation. Deep learning-based polyp segmentation methods utilize deep convolutional neural networks to learn the semantic information of polyp images, enabling accurate segmentation of polyps of different shapes and sizes in colonoscopy images. Compared to traditional methods, deep learning-based polyp segmentation methods do not require assumptions about specific polyp features, thus effectively improving generalization ability and reducing the false negative rate.

[0004] However, convolutional neural networks also have inherent limitations. The method of gradually acquiring image information by moving the convolutional kernel makes it difficult for the model to capture the global contextual interactions of the image. Although some methods attempt to alleviate this problem by designing various attention modules, they are still insufficient to segment complete polyp regions. In addition, current polyp segmentation methods in the deep learning field generally tend to use lightweight semantic segmentation models and rely on polyp ground truth maps for fully supervised training. Although this approach shows advantages in improving model processing speed and generalization ability, it also exposes a significant problem—the model often struggles to focus deeply on the key feature of polyp boundaries, easily overlooking boundary details that are crucial for accurate segmentation. Summary of the Invention

[0005] To address the shortcomings of existing technologies, this invention proposes a polyp segmentation method based on edge guidance and deep supervised PVT. By combining the polyp ground truth map with the polyp boundary ground truth map, it solves the problems of polyp segmentation methods based on convolutional neural networks struggling to capture the global semantic interaction of the image, resulting in incomplete segmentation regions and inaccurate segmentation results due to the model's lack of attention to boundary regions.

[0006] A polyp segmentation method based on edge guidance and deep-supervised PVT, the specific steps of which are as follows:

[0007] Step 1: Collect colonoscopy images, manually label the locations of polyps in the images to obtain a colonoscopy polyp ground truth map, and use the Canny operator to perform edge detection to obtain a polyp boundary ground truth map.

[0008] Preferably, the collected colonoscopy images are subjected to data augmentation and then resized to a uniform size.

[0009] Step 2: Construct a polyp segmentation network based on the Transformer architecture. PVT-v2, version 2 of the Pyramid Vision Transformer, is used as the backbone network for feature extraction. It also includes a context fusion decoding branch, a boundary extraction branch, a boundary awareness module, and a cascaded feature fusion branch. Deep supervision techniques are used to refine the multi-level segmentation results.

[0010] The PVT-v2 backbone network outputs first-scale pyramid features X1, second-scale pyramid features X2, third-scale pyramid features X3, and fourth-scale pyramid features X4 from top to bottom; where X1 is a detail feature map; and X2, X3, and X4 are semantic feature maps.

[0011] The context fusion decoding branch is used to receive the semantic feature map output by the PVT-v2 backbone network. It restores the deep semantic feature map to the size of the shallow semantic feature map by bilinear interpolation upsampling. After multiplication, it performs a convolution operation to output fusion features F2, F3, and F4 at different scales.

[0012] Preferably, the context fusion decoding branch directly performs a 3×3 convolution operation on the fourth-scale pyramid feature X4 to obtain the output feature F4; after performing bilinear interpolation upsampling on the fourth-scale pyramid feature X4 by a factor of 2, it is multiplied by the third-scale pyramid feature X3, and then subjected to a 3×3 convolution operation to obtain the output feature F3; after performing bilinear interpolation upsampling on the fourth-scale pyramid feature X4 and the third-scale pyramid feature X3 by a factor of 4 and a factor of 2 respectively, they are multiplied by the second-scale pyramid feature X2, and then subjected to a 3×3 convolution operation to obtain the output feature F2.

[0013] The boundary extraction branch uses a refinement module to enrich the feature content of the first-scale pyramid feature X1, obtaining the decoded feature F1. After bilinear interpolation upsampling by 4 times and the Sigmoid function, the edge prediction result P1 is obtained.

[0014] The boundary perception module fuses the refined feature F1 from the boundary extraction branch with the fusion features F2, F3, and F4 at different scales output by the context fusion decoding branch. Then, after bilinear interpolation upsampling by 4 times and the Sigmoid function, the boundary perception prediction results P2, P3, and P4 are obtained.

[0015] The cascaded feature fusion branch fuses the fused features F2, F3, and F4 at different scales output by the context fusion decoding branch in a deep-to-shallow order. After bilinear interpolation upsampling by 8 times and the Sigmoid function, the final polyp segmentation prediction result P5 is output.

[0016] Step 3: Construct an edge-guided multi-task learning framework on the polyp segmentation network. Train the polyp segmentation network by jointly performing polyp semantic segmentation and polyp edge detection tasks to obtain the optimal polyp segmentation network. Define the following total loss function. :

[0017]

[0018] in, , and These represent segmentation loss, edge detection loss, and boundary awareness loss, respectively. , These are the weights for edge detection loss and boundary awareness loss, respectively. Segmentation loss. Edge detection loss and boundary perception loss They are described as follows:

[0019]

[0020]

[0021]

[0022] in, and These represent the weighted binary cross-entropy loss and the weighted intersection-union ratio loss, respectively, and the segmentation loss. It calculates the loss between the final predicted result P5 and the polyp ground truth map G, and includes edge detection loss. It calculates the loss between the edge prediction result P1 and the polyp boundary ground truth map B, which is the boundary-aware loss. It calculates the loss between the boundary-aware prediction results P2, P3, and P4 and the polyp truth map G.

[0023] Step 4: Perform polyp segmentation processing using the optimal polyp segmentation model to obtain segmented polyp images.

[0024] The present invention has the following beneficial effects:

[0025] 1. Given the complex shapes and boundaries of polyps in colonoscopy images, a PVT-v2 backbone network is used to extract image features. This captures long-range dependencies and extracts global information for accurate segmentation. A deep supervision strategy is employed to monitor the output of each layer of the decoder, enabling the network to judge the quality of hidden layer feature maps, thus achieving a regularization-like effect and improving the network's training performance and stability.

[0026] 2. An edge-guided multi-task learning framework was constructed on the polyp segmentation network, cleverly combining polyp region segmentation and polyp edge detection tasks. A boundary-aware module was used to pass boundary detail information captured by the lower-level feature maps to the higher-level semantic feature maps, while avoiding network performance degradation caused by mutual interference between semantic segmentation and boundary detection tasks. The outputs of the three boundary-aware modules were converted into prediction maps and subjected to deep supervision, enabling the higher-level feature maps to acquire polyp boundary detail information while performing only the semantic segmentation task. This overcomes interference caused by similar patterns between polyps and the background, thereby improving the model's ability to recognize polyp boundaries and accurately segment polyp regions from the background. Attached Figure Description

[0027] Figure 1 The flowchart shows a polyp segmentation method based on edge guidance and deep supervision PVT.

[0028] Figure 2 This is an example of training set data in the embodiment;

[0029] Figure 3 This is a schematic diagram of the polyp segmentation network structure constructed in the embodiment;

[0030] Figure 4 A schematic diagram of the receptive field block (RFB) structure;

[0031] Figure 5 This is a schematic diagram of the boundary sensing module structure in the embodiment;

[0032] Figure 6 This is a graph showing the prediction results of the polyp segmentation network in the embodiment. Detailed Implementation

[0033] The present invention will be further explained below with reference to the accompanying drawings;

[0034] like Figure 1 As shown, a polyp segmentation method based on edge guidance and deep-supervised PVT is presented, with the following specific steps:

[0035] Step 1: Collect colonoscopy polyp images from five public colonoscopy polyp datasets: Kvasir, CVC-ClinicDB, CVC-ColonDB, CVC-300, and ETIS. Then, perform random rotation, random vertical flipping, and random horizontal flipping to increase the data volume. Finally, resize all images to 352×352 pixels. The final data volume is shown in Table 1.

[0036] Table 1

[0037]

[0038] 900 images from Kvasir and 550 images from CVC-ClinicDB were randomly selected as the training set, for a total of 1450 samples. The test set consisted of five parts: the remaining 100 images from Kvasir, the remaining 62 images from CVC-ClinicDB, 380 images from the CVC-ColonDB dataset, 196 images from the ETIS dataset, and 60 images from the CVC-300 dataset.

[0039] Based on the annotation information in the dataset, the ground truth map G of the polyp boundary is obtained. Then, the Canny operator is used for edge detection to extract the ground truth map B of the polyp boundary, as shown below. Figure 2 As shown.

[0040] Step 2, construct as follows Figure 3 The polyp segmentation network based on the Transformer architecture shown includes a backbone network, a context fusion decoding branch, a boundary extraction branch, a boundary-aware module, and a cascaded feature fusion branch. This embodiment builds a deep learning model on the PyTorch platform and runs it on a single NVIDIA RTX 4090 GPU.

[0041] The PVT-v2 backbone network outputs, from top to bottom, a first-scale pyramid feature X1, a second-scale pyramid feature X2, a third-scale pyramid feature X3, and a fourth-scale pyramid feature X4, where X1 is a detail feature map; and X2, X3, and X4 are semantic feature maps. Specifically, the i-th scale pyramid feature... ,in, , Let H be the dimension of the feature map, H be the length of the feature map, and W be the width of the feature map. To refine features and reduce the computational cost of feature fusion, the feature map is... The dimension of the image is uniformly reduced to 64 dimensions through convolution operations. Compared with convolutional neural networks that focus more on local information, PVT-v2 can capture global contextual interactions in polyp images through a self-attention mechanism. This global modeling capability helps the model understand the overall structure and contextual information of the image.

[0042] The context fusion decoding branch, acting as an image decoder, receives the semantic feature map output by the PVT-v2 backbone network and outputs fusion features F2, F3, and F4 at different scales. Specifically, a 3×3 convolution operation is performed directly on the fourth-scale pyramid feature X4 to obtain the output feature F4; the fourth-scale pyramid feature X4 is upsampled by 2 times through bilinear interpolation, multiplied by the third-scale pyramid feature X3, and then convolved by 3×3 to obtain the output feature F3; the fourth-scale pyramid feature X4 and the third-scale pyramid feature X3 are upsampled by 4 times and 2 times through bilinear interpolation, respectively, multiplied by the second-scale pyramid feature X2, and then convolved by 3×3 to obtain the output feature F2.

[0043] The boundary extraction branch uses a receptive field block (RFB) as a refinement module to extract richer prior details, enrich the feature content of the first-scale pyramid feature X1, and obtain the decoded feature F1. After bilinear interpolation upsampling by 4 times and the Sigmoid function, the edge prediction result P1 is obtained.

[0044] The boundary perception module combines a deep supervision strategy to fuse the refined feature F1 from the boundary extraction branch with the fusion features F2, F3, and F4 at different scales output by the context fusion decoding branch. Then, after bilinear interpolation upsampling by 4 times and the Sigmoid function, the boundary perception prediction results P2, P3, and P4 are obtained.

[0045] The cascaded feature fusion branch fuses the fused features F2, F3, and F4 at different scales output by the context fusion decoding branch in a deep-to-shallow order. After bilinear interpolation upsampling by 8 times and the Sigmoid function, the final polyp segmentation prediction result P5 is output.

[0046] Step 3: Construct an edge-guided multi-task learning framework on the polyp segmentation network. Input the training set data from Step 1, and train the polyp segmentation network by jointly performing polyp region segmentation and polyp edge detection tasks. Use the polyp boundary ground truth map to supervise the edge detection prediction result P, thus completing the edge detection task. The loss function is denoted as... The boundary-aware prediction results P2, P3, and P4 are supervised using the polyp truth map, and the loss function is denoted as... The final prediction result P5 is supervised using the polyp ground truth graph to complete the semantic segmentation task. The loss function is denoted as... Total loss function The weighted sum of the three loss functions mentioned above:

[0047]

[0048] , These are the weights for the edge detection loss and the boundary awareness loss, respectively, in this embodiment. =2、 =1. The optimal segmentation model is obtained by training the model to minimize the total loss;

[0049] in , and These represent segmentation loss, edge detection loss, and boundary awareness loss, respectively. Edge detection loss and boundary perception loss They are described as follows:

[0050]

[0051]

[0052]

[0053] in, and and represent the weighted binary cross-entropy loss and the weighted cross-union ratio loss, respectively. G represents the polyp truth map, and B represents the polyp boundary truth map.

[0054] Step 4: Using the polyp segmentation model trained in Step 3, perform polyp segmentation on the test set images from Step 1 to obtain segmented polyp images, such as... Figure 6 As shown in the figure, this method can achieve accurate segmentation of polyps in colonoscopy images.

[0055] To demonstrate the advancements of this method, it was compared with commonly used polyp segmentation methods in the prior art, including PraNet, ResUnet++, SANet, ACENet, TGANet, FCBFormer, CFA-Net, PETNet, and APCNet. The two most commonly used performance metrics in image segmentation, mean Dice similarity coefficient (mDice) and mean intersection over union (mIoU), were used to quantitatively evaluate the segmentation performance of different models. mDice calculates the degree of overlap between the segmented region obtained by the model and the labeled ground truth segmented region, ranging from [0, 1]. A larger value indicates that the segmentation result is closer to the labeled result, and the better the segmentation effect.

[0056]

[0057] Where k represents the number of categories, TP stands for True Positives, FP stands for False Positives, and FN stands for False Negatives.

[0058] mIoU calculates the overlap ratio between the predicted bounding boxes and the ground truth bounding boxes, which is the ratio of the intersection to the union of the predicted and ground truth bounding boxes. The range is [0, 1]. A larger value indicates that the segmentation result is closer to the labeled result, and the better the segmentation effect.

[0059]

[0060] The meanings of the symbols are the same as above.

[0061] The comparison results of the evaluation index mDice are shown in Table 2, and the comparison results of the evaluation index mIoU are shown in Table 3.

[0062] Table 2

[0063]

[0064] Table 3

[0065]

[0066] The quantitative comparison results in Tables 2 and 3 show that our method achieves the best performance on the test sets CVC-ClinicDB, ETIS, and CVC-T. On the test sets Kvasir and CVC-ColonDB, our method performs essentially on par with the best method. Compared with other methods, our method achieves the best overall results, demonstrating its effectiveness and robustness.

[0067] The above description is merely a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the technical scope disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.

Claims

1. A polyp segmentation method based on edge-guided and deep-supervised PVT, comprising collecting colonoscopy images, manually annotating the polyp locations in the images to obtain a colonoscopy polyp ground truth map, and performing polyp segmentation on colonoscopy images containing polyps after network training, characterized in that: Specifically, the following steps are included: Step 1: Use the Canny operator to perform edge detection on the polyps in the colonoscopy image to obtain the ground truth map of the polyp boundary; Step 2: Construct a polyp segmentation network based on the Transformer architecture, including the backbone network PVT-v2, context fusion decoding branch, boundary extraction branch, boundary awareness module, and cascaded feature fusion branch; The PVT-v2 backbone network outputs first to fourth scale pyramid features X1, X2, X3, and X4 sequentially from top to bottom; where X1 is the detail feature map; and X2, X3, and X4 are semantic feature maps. The context fusion decoding branch receives the semantic feature map output by the PVT-v2 backbone network, and restores the deep semantic feature map to the size of the shallow semantic feature map by bilinear interpolation upsampling. After multiplication, it is then subjected to convolution operation to output fusion features F2, F3, and F4 at different scales. The boundary extraction branch uses the receptive field block RFB as the refinement module in the boundary extraction branch to enrich the feature content of the first-scale pyramid feature X1, obtain the decoded feature F1, and then after bilinear interpolation upsampling by 4 times and the Sigmoid function, the edge prediction result P1 is obtained. The boundary perception module fuses the refined feature F1 from the boundary extraction branch with the fusion features F2, F3, and F4 output from the context fusion decoding branch, and then performs bilinear interpolation upsampling by 4 times and the Sigmoid function to obtain the boundary perception prediction results P2, P3, and P4. The cascaded feature fusion branch fuses the fusion features F2, F3, and F4 at different scales output by the context fusion decoding branch in a deep-to-shallow order. After bilinear interpolation upsampling by 8 times and the Sigmoid function, the final polyp segmentation prediction result P5 is output. Step 3: Train the polyp segmentation network by combining the polyp semantic segmentation task and the polyp edge detection task to obtain the optimal polyp segmentation network; the total loss function L total for: L total =L seg +λ1L boun +λ2L ba Among them, L seg For semantic segmentation, L boun For edge detection loss, L ba For boundary-aware loss, λ1 and λ2 are the weights of edge detection loss and boundary-aware loss, respectively; Step 4: The optimal polyp segmentation model trained in Step 3 is used to process the colonoscopy image containing polyps to obtain the segmented polyp image.

2. The polyp segmentation method based on edge guidance and deep supervision PVT as described in claim 1, characterized in that: The collected colonoscopy images were subjected to data augmentation operations, including random rotation, random vertical flipping, and random horizontal flipping, and then resized to a uniform size.

3. The polyp segmentation method based on edge guidance and deep supervision PVT as described in claim 1, characterized in that: The context fusion decoding branch directly performs a 3×3 convolution operation on the fourth-scale pyramid feature X4 to obtain the output feature F4; after bilinear interpolation upsampling the fourth-scale pyramid feature X4 by 2 times, it is multiplied with the third-scale pyramid feature X3, and then subjected to a 3×3 convolution operation to obtain the output feature F3; after bilinear interpolation upsampling the fourth-scale pyramid feature X4 and the third-scale pyramid feature X3 by 4 times and 2 times respectively, they are multiplied with the second-scale pyramid feature X2, and then subjected to a 3×3 convolution operation to obtain the output feature F2.

4. The polyp segmentation method based on edge guidance and deep supervision PVT as described in claim 1, characterized in that: The edge detection prediction result P1 is supervised using the ground truth map of the polyp boundary to complete the edge detection task. The loss function is denoted as L. boun : The boundary-aware prediction results P2, P3, and P4 are supervised using the polyp truth map, and the loss function is denoted as L. ba : The final prediction result P5 is supervised using the polyp ground truth graph to complete the semantic segmentation task. The loss function is denoted as L. seg : in, and denoted as weighted binary cross-entropy loss and weighted crossover ratio loss, respectively. G represents the polyp truth map and B represents the polyp boundary truth map.

5. The polyp segmentation method based on edge guidance and deep supervision PVT as described in claim 1, characterized in that: Set λ1 = 2 and λ2 = 1.

6. A computer-readable storage medium having a computer program stored thereon, which, when executed in a computer, causes the computer to perform the method of any one of claims 1 to 5.

Images