Polyp segmentation method based on edge guidance and deep supervision PVT

Abstract

The invention discloses a polyp segmentation method based on edge guidance and deep supervision PVT. The method aims at a colonoscope image with a polyp, and outputs the position and the shape of the polyp by using a constructed polyp segmentation network. The polyp segmentation network uses PVT-v2 as a backbone network for feature extraction, and further comprises a context fusion decoding branch, a boundary extraction branch, a boundary sensing module and a cascade feature fusion branch. And the context fusion decoding branch serves as an image decoder and receives a semantic feature map output by the PVT-v2 backbone network. And the boundary extraction branch enriches the content of the detail feature map by using a refining module to obtain an edge prediction result. And the boundary sensing module fuses the refined features and the output feature map of the context fusion decoding to obtain a boundary sensing prediction result. And the cascaded feature fusion branch fuses fusion features under different scales layer by layer according to a sequence from deep to shallow, and outputs a final polyp segmentation prediction result P5.

Description

Technical Field

[0001] The present invention belongs to the field of image processing technology, relates to the fields of computer vision and medical image segmentation, and specifically relates to a polyp segmentation method based on edge guidance and deep supervision PVT. Background Art

[0002] The automatic polyp segmentation task can quickly and accurately segment the polyp area from the colonoscopic images with polyps, provide auxiliary reference information for the doctor's subsequent analysis, and reduce the workload. In the past few decades, automatic polyp segmentation has been widely studied. The current automatic polyp segmentation methods are mainly divided into traditional methods and deep learning-based methods. Traditional methods usually focus on the spatial domain information of polyps, such as shape, color or texture, and then automatically search for the corresponding parts in the colonoscopic image by presetting the spatial domain information of polyps. Traditional polyp segmentation methods have made great progress in the automatic polyp segmentation task, but its practice of finding polyps through preset prior knowledge makes it ineffective when facing polyps with variable shapes and colors. Due to the lack of generalization ability, such methods still have a high missed detection rate.

[0003] With the development of convolutional neural networks, polyp segmentation methods based on deep learning have made significant breakthroughs in recent years and have become cutting-edge technologies in automatic polyp segmentation tasks. Polyp segmentation methods based on deep learning use deep convolutional neural networks to learn the semantic information of polyp images to achieve accurate segmentation of polyps of different shapes or sizes in colonoscopy images. Compared with traditional methods, polyp segmentation methods based on deep learning do not need to assume a specific feature of polyps, so they effectively improve generalization ability and reduce missed detection rate.

[0004] However, convolutional neural networks also have inherent limitations. The way of gradually collecting image information by moving the convolution kernel makes it difficult for the model to capture the global contextual interaction of the image. Although some existing methods try to alleviate this problem by designing various attention modules, it is still not enough to segment the complete polyp area. In addition, the current polyp segmentation methods in the field of deep learning generally tend to adopt lightweight semantic segmentation models and rely on polyp truth maps for fully supervised training. Although this method shows advantages in improving model processing speed and generalization ability, it also exposes a significant problem - the model often finds it difficult to focus deeply on the key feature of the polyp boundary, and it is easy to ignore those boundary details that are crucial for accurate segmentation. Summary of the invention

[0005] In view of the shortcomings of the prior art, the present invention proposes a polyp segmentation method based on edge guidance and deep supervision PVT, which combines the polyp truth map with the polyp boundary truth map to solve the problem that the polyp segmentation method based on convolutional neural network is difficult to obtain the global semantic interaction of the image, resulting in incomplete segmentation area and the model does not pay attention to the boundary area, resulting in inaccurate boundaries of the segmentation results.

[0006] A polyp segmentation method based on edge guidance and deep supervision PVT, the specific steps are as follows: Step 1: Collect colonoscopic images, manually mark the locations of polyps in the images, obtain the true value map of colonoscopic polyps, use the Canny operator to perform edge detection operations, and obtain the true value map of polyp boundaries.

[0007] Preferably, data enhancement is performed on the collected colonoscopy images and then resized to a uniform size.

[0008] Step 2: Build a polyp segmentation network based on the Transformer architecture, using the v2 version of the Pyramid Vision Transformer PVT-v2 as the backbone network for feature extraction, including context fusion decoding branch, boundary extraction branch, boundary perception module and cascade feature fusion branch. Use deep supervision technology to refine the multi-level segmentation results.

[0009] The PVT-v2 backbone network outputs the first scale pyramid feature X1, the second scale pyramid feature X2, the third scale pyramid feature X3 and the fourth scale pyramid feature X4 from top to bottom; wherein X1 is a detail feature map; X2, X3 and X4 are semantic feature maps.

[0010] The context fusion decoding branch is used to receive the semantic feature map output by the PVT-v2 backbone network, restore the deep semantic feature map to the size of the shallow semantic feature map through bilinear interpolation upsampling, multiply and then perform convolution operation to output fusion features F2, F3, and F4 at different scales.

[0011] Preferably, the context fusion decoding branch directly performs a 3×3 convolution operation on the fourth scale pyramid feature X4 to obtain an output feature F4; after bilinear interpolation upsampling the fourth scale pyramid feature X4 by 2 times, multiplying it with the third scale pyramid feature X3, and then performing a 3×3 convolution operation to obtain the output feature F3; after bilinear interpolation upsampling the fourth scale pyramid feature X4 and the third scale pyramid feature X3 by 4 times and 2 times respectively, multiplying them with the second scale pyramid feature X2, and then performing a 3×3 convolution operation to obtain the output feature F2.

[0012] The boundary extraction branch uses a refinement module to enrich the feature content of the first scale pyramid feature X1 to obtain a decoded feature F1, and then obtains an edge prediction result P1 after bilinear interpolation upsampling by 4 times and a Sigmoid function.

[0013] The boundary perception module fuses the feature F1 refined by the boundary extraction branch with the fusion features F2, F3, and F4 at different scales output by the context fusion decoding branch, and then obtains the boundary perception prediction results P2, P3, and P4 through bilinear interpolation upsampling by 4 times and Sigmoid function.

[0014] The cascade feature fusion branch fuses the fusion features F2, F3, and F4 at different scales output by the context fusion decoding branch layer by layer in a descending order, and then outputs the final polyp segmentation prediction result P5 after bilinear interpolation upsampling by 8 times and Sigmoid function.

[0015] Step 3: Build an edge-guided multi-task learning framework on the polyp segmentation network, train the polyp segmentation network by combining the polyp semantic segmentation task and the polyp edge detection task, and obtain the optimal polyp segmentation network. Define the following total loss function :

[0016] in, , and Represent segmentation loss, edge detection loss and boundary perception loss respectively. , are the weights of edge detection loss and boundary perception loss respectively. Segmentation loss , edge detection loss and boundary-aware loss They are described as:

[0017] in, and Represent weighted binary cross entropy loss and weighted intersection-over-union loss, segmentation loss It is to calculate the loss between the final prediction result P5 and the true value map G of the polyp, the edge detection loss It is the loss between the edge prediction result P1 and the polyp boundary truth map B, the boundary perception loss It is to calculate the loss between the boundary-aware prediction results P2, P3 and P4 and the polyp ground truth map G.

[0018] Step 4: Perform polyp segmentation processing using the optimal polyp segmentation model to obtain a segmented polyp image.

[0019] The present invention has the following beneficial effects: 1. Considering the complex shapes and boundaries of polyps in colonoscopy images, the PVT-v2 backbone network is used to extract image features, which can capture long-distance dependencies in the image and extract global information for accurate segmentation. A deep supervision strategy is used to supervise the output of each layer of the decoder, so that the network can judge the quality of the hidden layer feature map, thus playing a similar regularization effect, which helps to improve the training effect and stability of the network.

[0020] 2. An edge-guided multi-task learning framework is constructed on the polyp segmentation network, which cleverly combines the polyp region segmentation task and the polyp edge detection task. The boundary perception module is used to pass the boundary detail information captured by the underlying feature map to the high-level semantic feature map, while avoiding the network performance degradation caused by the mutual interference between the semantic segmentation and boundary detection tasks. The outputs of the three boundary perception modules are converted into prediction maps and deep supervision is performed, so that the high-level feature map can obtain the boundary detail information of the polyp while only completing the semantic segmentation task, overcoming the interference caused by the similar patterns between the polyp and the background, thereby improving the model's ability to recognize the polyp boundary and accurately segmenting the polyp region from the background. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 Flowchart of the polyp segmentation method based on edge guidance and deep supervision PVT; Figure 2 This is an example of training set data in the embodiment; Figure 3 Schematic diagram of the polyp segmentation network structure constructed in the embodiment; Figure 4 Schematic diagram of the receptive field block (RFB) structure; Figure 5 This is a schematic diagram of the structure of the boundary perception module in the embodiment; Figure 6 Graph showing the prediction results of the polyp segmentation network in the embodiment. DETAILED DESCRIPTION

[0022] The present invention will be further explained below with reference to the accompanying drawings; like Figure 1 As shown in FIG. 1 , a polyp segmentation method based on edge guidance and deep supervision PVT is shown in FIG. 1 . The specific steps are as follows: Step 1: Collect colonoscopic polyp images from five public colonoscopic polyp datasets: Kvasir, CVC-ClinicDB, CVC-ColonDB, CVC-300, and ETIS. Then perform random rotation, random vertical flipping, and random horizontal flipping to increase the data volume. Finally, resize all images to 352×352. The final data volume is shown in Table 1: Table 1 900 images from Kvasir and 550 images from CVC-ClinicDB are randomly selected as training sets, with a total of 1450 samples. The test set includes five images, namely the remaining 100 images from Kvasir, the remaining 62 images from CVC-ClinicDB, 380 images from the CVC-ColonDB dataset, 196 images from the ETIS dataset, and 60 images from the CVC-300 dataset.

[0023] According to the annotation information in the data set, the polyp boundary truth map G is obtained, and then the Canny operator is used for edge detection to extract the polyp boundary truth map B, as shown in Figure 2 shown.

[0024] Step 2: Build Figure 3 The polyp segmentation network based on the Transformer architecture shown includes a backbone network, a context fusion decoding branch, a boundary extraction branch, a boundary perception module, and a cascade feature fusion branch. In this embodiment, a deep learning model is built on the PyTorch platform and runs on a single NVIDA RTX 4090 GPU.

[0025] The PVT-v2 backbone network outputs the first scale pyramid feature X1, the second scale pyramid feature X2, the third scale pyramid feature X3 and the fourth scale pyramid feature X4 from top to bottom, where X1 is a detail feature map; X2, X3 and X4 are semantic feature maps. Specifically, the i-th scale pyramid feature ,in, , is the dimension of the feature map, H is the length of the feature map, and W is the width of the feature map. In order to refine the features and reduce the amount of computation during feature fusion, the feature map The number of dimensions is uniformly reduced to 64 through convolution operations. Compared with convolutional neural networks that focus more on local information, PVT-v2 can capture global contextual interactions in polyp images through the self-attention mechanism. This global modeling capability helps the model understand the overall structure and contextual information of the image.

[0026] The context fusion decoding branch serves as an image decoder, and is used to receive the semantic feature map output by the PVT-v2 backbone network, and output fusion features F2, F3, and F4 at different scales. Specifically, the fourth scale pyramid feature X4 is directly subjected to a 3×3 convolution operation to obtain the output feature F4; the fourth scale pyramid feature X4 is subjected to bilinear interpolation upsampling by 2 times, multiplied with the third scale pyramid feature X3, and then subjected to a 3×3 convolution operation to obtain the output feature F3; the fourth scale pyramid feature X4 and the third scale pyramid feature X3 are subjected to bilinear interpolation upsampling by 4 times and 2 times, respectively, and then multiplied with the second scale pyramid feature X2, and then subjected to a 3×3 convolution operation to obtain the output feature F2.

[0027] The boundary extraction branch uses the Receptive Field Block (RFB) as a refinement module to extract richer detail prior content, enrich the feature content of the first scale pyramid feature X1, obtain the decoded feature F1, and then obtain the edge prediction result P1 after bilinear interpolation upsampling 4 times and Sigmoid function.

[0028] The boundary perception module combines the deep supervision strategy to fuse the feature F1 refined by the boundary extraction branch with the fusion features F2, F3, and F4 at different scales output by the context fusion decoding branch, and then obtains the boundary perception prediction results P2, P3, and P4 through bilinear interpolation upsampling by 4 times and Sigmoid function.

[0029] The cascade feature fusion branch fuses the fusion features F2, F3, and F4 at different scales output by the context fusion decoding branch layer by layer in a descending order, and then outputs the final polyp segmentation prediction result P5 after bilinear interpolation upsampling by 8 times and Sigmoid function.

[0030] Step 3: Build an edge-guided multi-task learning framework on the polyp segmentation network, input the training set data in step 1, train the polyp segmentation network by combining the polyp region segmentation task and the polyp edge detection task, and use the polyp boundary truth map to supervise the edge detection prediction result P to complete the edge detection task. The loss function is recorded as ; Use the polyp truth map to supervise the boundary perception prediction results P2, P3 and P4, and the loss function is recorded as ; Use the polyp truth map to supervise the final prediction result P5 to complete the semantic segmentation task, and the loss function is recorded as ; Total loss function It is the weighted sum of the above three loss functions: , are the weights of edge detection loss and boundary perception loss respectively. In this embodiment =2, = 1. The model is trained by minimizing the total loss to obtain the optimal segmentation model;

[0031] in , and Represent segmentation loss, edge detection loss and boundary perception loss respectively, segmentation loss , edge detection loss and boundary-aware loss They are described as: in, and Represent the weighted binary cross entropy loss and weighted intersection-over-union loss respectively. G represents the polyp truth map, and B represents the polyp boundary truth map.

[0032] Step 4: Use the polyp segmentation model trained in step 3 to perform polyp segmentation on the test set image in step 1 to obtain a segmented polyp image, such as Figure 6 As shown in the figure, it can be seen that the accurate segmentation of polyps in colonoscopy images can be achieved through this method.

[0033] In order to demonstrate the progress of this method, we compared it with the commonly used polyp segmentation methods in the prior art, namely PraNet, ResUnet++, SANet, ACENet, TGANet, FCBFormer, CFA-Net, PETNet and APCNet. We used the two most commonly used performance indicators in image segmentation, mean Dice coefficient (mDice) and mean Intersection over Union (mIoU), to quantitatively evaluate the segmentation performance of different models. mDice is the degree of overlap between the segmented area obtained by the calculation model and the true segmented area of ​​the label, ranging from [0, 1]. The larger the value, the closer the segmentation result is to the annotation result, and the better the segmentation effect is: Among them, k represents the number of categories, TP is the true positives, FP is the false positives, and FN is the false negatives.

[0034] mIoU calculates the overlap rate of the predicted segmented border and the real border, that is, the ratio of the intersection and union of the predicted segmented border and the real border. The range is [0, 1]. The larger the value, the closer the segmentation result is to the annotation result, and the better the segmentation effect is: The meanings of the symbols are the same as above.

[0035] The comparison results of the evaluation index mDice are shown in Table 2, and the comparison results of the evaluation index mIoU are shown in Table 3: Table 2 Table 3 From the quantitative comparison results in Table 2 and Table 3, it can be seen that on the test sets CVC-ClinicDB, ETIS and CVC-T, this method achieves the best performance, and on the test sets Kvasir and CVC-ColonDB, this method is basically on par with the best method. Compared with other methods, this method achieves the best results overall, proving the effectiveness and robustness of this method.

[0036] The above description is only a preferred specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can make equivalent replacements or changes according to the technical scheme and inventive concept of the present invention within the technical scope disclosed by the present invention, which should be covered by the protection scope of the present invention.

Claims

1. A polyp segmentation method based on edge guidance and deep supervision PVT, collecting colonoscopic images, manually marking the polyp positions in the images, obtaining a colonoscopic polyp truth map, and performing polyp segmentation on colonoscopic images with polyps after network training, characterized in that: The specific steps include: Step 1: Use the Canny operator to perform edge detection on the polyps in the colonoscopy image to obtain a true value map of the polyp boundary; Step 2: Build a polyp segmentation network based on the Transformer architecture, including the backbone network PVT-v2, context fusion decoding branch, boundary extraction branch, boundary perception module and cascade feature fusion branch; The PVT-v2 backbone network outputs the first to fourth scale pyramid features X1, X2, X3, and X4 from top to bottom in sequence; wherein X1 is a detail feature map; X2, X3, and X4 are semantic feature maps; The context fusion decoding branch receives the semantic feature map output by the PVT-v2 backbone network, restores the deep semantic feature map to the size of the shallow semantic feature map by bilinear interpolation upsampling, multiplies them and then performs convolution operation to output fusion features F2, F3, and F4 at different scales; The boundary extraction branch uses a refinement module to enrich the feature content of the first scale pyramid feature X1 to obtain a decoded feature F1, and then obtains an edge prediction result P1 after bilinear interpolation upsampling by 4 times and a Sigmoid function; The boundary perception module fuses the feature F1 after the boundary extraction branch is refined with the fusion features F2, F3, and F4 respectively, and then obtains the boundary perception prediction results P2, P3, and P4 through bilinear interpolation upsampling by 4 times and Sigmoid function; The cascade feature fusion branch fuses the fusion features F2, F3, and F4 at different scales output by the context fusion decoding branch layer by layer in a descending order, and then outputs the final polyp segmentation prediction result P5 after bilinear interpolation upsampling by 8 times and Sigmoid function. Step 3: Combine the polyp semantic segmentation task and the polyp edge detection task to train the polyp segmentation network and obtain the optimal polyp segmentation network; Step 4: Process the colonoscopy image with polyps using the optimal polyp segmentation model trained in step 3 to obtain a segmented polyp image.

2. The polyp segmentation method based on edge guidance and deep supervision PVT as claimed in claim 1, characterized in that: The collected colonoscopy images were subjected to data augmentation operations, including random rotation, random vertical flipping, and random horizontal flipping, and then resized to a uniform size.

3. The polyp segmentation method based on edge guidance and deep supervision PVT as claimed in claim 1, characterized in that: The context fusion decoding branch directly performs a 3×3 convolution operation on the fourth scale pyramid feature X4 to obtain an output feature F4; after bilinear interpolation upsampling the fourth scale pyramid feature X4 by 2 times, multiplying it with the third scale pyramid feature X3, and then performing a 3×3 convolution operation to obtain the output feature F3; after bilinear interpolation upsampling the fourth scale pyramid feature X4 and the third scale pyramid feature X3 by 4 times and 2 times respectively, multiplying them with the second scale pyramid feature X2, and then performing a 3×3 convolution operation to obtain the output feature F2.

4. The polyp segmentation method based on edge guidance and deep supervision PVT as claimed in claim 1, characterized in that: Use the receptive field block RFB as the refinement module in the boundary extraction branch.

5. The polyp segmentation method based on edge guidance and deep supervision PVT as claimed in claim 1, characterized in that: The edge detection prediction result P1 is supervised by the polyp boundary truth map to complete the edge detection task. The loss function is recorded as : The polyp truth map is used to supervise the boundary perception prediction results P2, P3 and P4, and the loss function is recorded as : The polyp truth map is used to supervise the final prediction result P5 to complete the semantic segmentation task. The loss function is recorded as : in, and They represent weighted binary cross entropy loss and weighted intersection-over-union loss respectively, G represents the true value map of polyps, and B represents the true value map of polyp boundaries.

6. The polyp segmentation method based on edge guidance and deep supervision PVT as claimed in claim 5, characterized in that: Define the total loss function of the polyp segmentation network for: in, , are the weights of edge detection loss and boundary perception loss respectively.

7. The polyp segmentation method based on edge guidance and deep supervision PVT as claimed in claim 6, characterized in that: set up =2, =1.

8. A computer-readable storage medium having a computer program stored thereon, which, when executed in a computer, causes the computer to execute the method according to any one of claims 1 to 7.

Images