Polyp cutting method and device

Abstract

The invention provides a polyp segmentation method and device. The method comprises the following steps: extracting global features of a polyp image by using a pyramid vision Transform model PVT, extracting local features of the polyp image by using a convolutional neural network U-Net, and fusing the global features and the local features by using a fusion module; a multi-scale enhancement module is used for processing a fusion result of the fusion module through a space attention mechanism so as to emphasize space information layer by layer, and a decoding module is used for decoding a processing result; and a boundary enhancement module is used to process a fusion result of the fusion module through cavity convolution and an attention mechanism to emphasize boundary information, and a processing result of the boundary enhancement module and a decoding result of the decoding module are fused and then decoded again to obtain a segmentation result of the polyp image. The accuracy and efficiency of polyp segmentation are improved.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and in particular to a polyp segmentation method and device. Background Art

[0002] Colorectal cancer poses a great threat to human life. Most colorectal cancers develop from colorectal polyps. As the disease worsens, the patient's survival rate will drop sharply, so early detection is of great significance for the prevention and treatment of the disease. Medical image segmentation has important application value in the field of medical imaging. By accurately segmenting different tissues or lesions in medical images, it assists doctors in diagnosing lesions based on the shape, size, volume, etc. of the lesions. The large differences in the size of polyps, blurred boundaries, low contrast, and similarity to the surrounding normal tissues are the main reasons for the low accuracy of polyp segmentation. Therefore, it is very necessary to design an automatic and reliable segmentation method, which can not only reduce the burden on doctors, but also reduce the missed detection rate.

[0003] Traditional segmentation methods mainly include threshold method, edge detection method, region growing method and clustering method. Although traditional methods can segment polyps, the segmentation accuracy is not high. In recent years, deep learning methods have shown their powerful effects in the field of semantic segmentation, and have outperformed traditional segmentation algorithms in segmentation effects, becoming a hot topic in recent research. The first proposed full convolutional neural network (FCN) was used as an early medical image segmentation model using convolutional neural networks. Then, after the fully connected layer was replaced by the convolutional layer, the end-to-end neural network Unet was proposed. Because it consists of a simple encoder and decoder and shows good segmentation effects in medical image segmentation, it has become a benchmark network in medical image segmentation. Subsequently, many researchers proposed many improved versions based on the UNet network, such as UNet++, which reduces the semantic difference between feature maps through multi-scale fusion in skip connections, ResUNet with residual networks, and Attention Unet with gated attention mechanisms to highlight important features while suppressing irrelevant features. All of these improved versions have achieved good performance in medical image segmentation.

[0004] There are also many models for polyp segmentation. DeepLabv3+ mainly captures semantic information through dilated convolution and spatial pyramid pooling, and has good results in polyp segmentation. MSNe mainly extracts features through five encoders, and then uses multi-scale subnets and multi-scale difference modules to better capture the differences between features, highlight useful difference information and eliminate redundant differences, thereby improving segmentation accuracy. PraNet network aggregates high-level features through the parallel partial decoder (PPD) module and indirectly extracts contour information with the reverse attention module (RA). CaraNet mines the feature information of small targets through axial reverse attention and channel feature pyramid to improve the segmentation accuracy of small targets. BSCA-Net uses ordinary convolution to extract features and uses multiple attention mechanisms to decode step by step.

[0005] Transformer was first used in the field of natural language processing. It has shown excellent results in the field of natural language processing because of its ability to capture the whole picture. Because convolutional neural networks are good at capturing local semantic information, but lack the ability to capture global semantic information. Subsequently, researchers introduced it into computer vision, and then proposed vision Transformer. TransUNet, Swin-Unet and XboundFormer came into being, showing that Transformer has great potential in medical image segmentation. Someone proposed Polyp-PVT, which uses pyramid Transformer for encoding to better extract deep semantic information. The cascade fusion module (CFM) obtains the semantic and location information of polyps. The aggregation module (SAM) mines local and global semantic clues from the polyp area and achieves good results in polyp segmentation. The proposed FCBFormer aims to combine the advantages of Transformer and fully convolutional network to improve the accuracy of segmentation. TransFuse combines CNN (Convolutional Neural Network) and Transformer in a parallel manner, and proposes a fusion module to efficiently fuse the features of the two, showing good segmentation capabilities. Based on Transformer, researchers proposed the Pyramid Vision Transformer (PVT), whose biggest feature is that it can output feature maps of different sizes and reduce the amount of calculation, becoming the backbone of most segmentation networks.

[0006] Although many models have emerged with the continuous development of deep learning, the accuracy of polyp segmentation still needs to be further improved. Mainly due to problems such as the medical image imaging environment and technology, the quality of imaging is worse than that of natural images, which is manifested in low resolution, low contrast and more noise. Polyp images are particularly prominent. The color of polyps is very similar to that of surrounding tissues, resulting in blurred and unclear boundaries. Especially for small and inconspicuous polyps in the early stage, even professional doctors sometimes find it difficult to detect. Summary of the invention

[0007] The present invention provides a polyp segmentation method and device to solve the defects of unclear segmentation boundary, low segmentation accuracy and efficiency in the prior art, and to provide a new polyp segmentation algorithm, aiming to improve the accuracy and efficiency of polyp segmentation.

[0008] The present invention provides a polyp segmentation method, comprising:

[0009] Extract global features of polyp images using a pyramid vision Transformer model PVT, extract local features of the polyp images using a convolutional neural network U-Net, and fuse the global features with the local features using a fusion module;

[0010] Using a multi-scale enhancement module to process the fusion result of the fusion module through a spatial attention mechanism to emphasize spatial information layer by layer, and using a decoding module to decode the processing result;

[0011] The boundary enhancement module is used to process the fusion result of the fusion module through the hole convolution and attention mechanism to emphasize the boundary information, and the processing result of the boundary enhancement module is fused with the decoding result of the decoding module and decoded again to obtain the segmentation result of the polyp image.

[0012] According to a polyp segmentation method provided by the present invention, the fusion module is used to fuse the global features and the local features, including:

[0013] When the current fusion module is a first-layer fusion module, convolving the global features and the local features of the same scale, and fusing the obtained features of the same size through Hadamard product;

[0014] When the current fusion module is not a first-layer fusion module, the final fusion result obtained by the previous-layer fusion module is fused with the features of the same size through Hadamard product;

[0015] The fusion result of the Hadamard product is convolved again, and then concatenated with the global feature and the local feature of the same scale;

[0016] The result of the splicing operation is passed through a residual network to obtain a final fusion result of the global feature and the local feature of the same scale.

[0017] According to a polyp segmentation method provided by the present invention, the multi-scale enhancement module is used to process the fusion result of the fusion module through a spatial attention mechanism to emphasize spatial information layer by layer, including:

[0018] When the current fusion module is not a first-layer fusion module, a final fusion result output by the current fusion module is weighted by a spatial attention mechanism;

[0019] The weight corresponding to the fusion module of the i-th layer is upsampled i-1 times, and the upsampling results from the 1st to the i-1th times are multiplied by the final fusion results output by the fusion modules of the i-1th to the 1st layers, where i is the layer number of the fusion module except the 1st layer;

[0020] The result obtained by multiplying the final fusion result output by the i-th layer fusion module and the final fusion result output by the i-th layer fusion module are spliced.

[0021] According to a polyp segmentation method provided by the present invention, the decoding of the processing result by using a decoding module comprises:

[0022] When the current decoding module is a first-layer decoding module, the fusion result of the last-layer fusion module is input into the PPM module for processing, and the processing result of the fusion result of the last-layer fusion module is spliced ​​as the input of the current decoding module;

[0023] When the current decoding module is not the first-layer decoding module, the decoding result of the previous-layer decoding module and the fusion result of the N-(j-1)th-layer fusion module are concatenated as the input of the decoding module, where N is the total number of layers of the fusion module and i is the layer number of the current decoding module;

[0024] Performing three dilated convolutions on the input of the current decoding module after convolution;

[0025] The results of the first and second dilated convolutions are concatenated and then convolved.

[0026] The result obtained after the second convolution is concatenated with the result of the third dilated convolution, and then convolution and deconvolution are performed to obtain the decoding result of the current decoding module.

[0027] According to a polyp segmentation method provided by the present invention, before extracting the global features of the polyp image using the pyramid visual Transformer model PVT and extracting the local features of the polyp image using the convolutional neural network U-Net, the method further includes:

[0028] Calculating the loss of the segmentation result of the polyp image obtained by each layer of decoding module and the segmentation result obtained after the output of the boundary enhancement module is decoded;

[0029] Backward propagation is performed according to the loss to update the weights of the PVT, U-Net, multi-scale enhancement module, decoding module and boundary enhancement module.

[0030] According to a polyp segmentation method provided by the present invention, the boundary enhancement module is used to process the fusion result of the fusion module through a dilated convolution and an attention mechanism to emphasize the boundary information, including:

[0031] The dilated convolution and attention mechanism are used to enhance the local details of the fusion result output by the first-layer fusion module.

[0032] According to a polyp segmentation method provided by the present invention, the boundary enhancement module is used to process the fusion result of the fusion module through the hole convolution and attention mechanism to emphasize the boundary information, and the processing result of the boundary enhancement module is fused with the decoding result of the decoding module and decoded again to obtain the segmentation result of the polyp image, including:

[0033] Convolving the fusion result output by the first-layer fusion module through multiple branches, wherein each branch except the last branch includes a normal convolution and a hole convolution, and the last branch includes a 1*1 convolution block;

[0034] After fusing the results obtained by the other branches, they are then fused with the result obtained by the last branch to form a residual connection, and processed by the Relu function;

[0035] The output of the Relu function is processed using spatial attention and channel attention in turn, and then added and multiplied with the decoding result of the last layer decoding module respectively, and the features obtained by the addition and multiplication are spliced ​​to obtain the segmentation result of the polyp image.

[0036] The present invention also provides a polyp segmentation device, comprising:

[0037] An extraction and fusion module is used to extract global features of the polyp image using a pyramid vision Transformer model PVT, extract local features of the polyp image using a convolutional neural network U-Net, and fuse the global features with the local features using a fusion module;

[0038] A spatial enhancement module, used for processing the fusion result of the fusion module by using a multi-scale enhancement module through a spatial attention mechanism to emphasize spatial information layer by layer, and decoding the processing result by using a decoding module;

[0039] The boundary enhancement module is used to process the fusion result of the fusion module through the hole convolution and attention mechanism to emphasize the boundary information, and fuse the processing result of the boundary enhancement module with the decoding result of the decoding module and decode it again to obtain the segmentation result of the polyp image.

[0040] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, any of the above-mentioned polyp segmentation methods is implemented.

[0041] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the polyp segmentation method described in any one of the above is implemented.

[0042] The present invention also provides a computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the polyp segmentation method described above is implemented.

[0043] The polyp segmentation method and device provided by the present invention propose a parallel dual-branch network (PDB-Net), capture local information through a convolutional neural network branch, capture global information through a PVT branch, and then fuse the obtained features through a fusion module to complete the feature extraction work; in order to better utilize deep features, a multi-scale enhancement module is proposed in the skip link part, and spatial information is emphasized layer by layer by using a spatial attention mechanism; in order to emphasize boundary information, a boundary enhancement module is proposed, and the ability to extract multi-scale information is improved through void convolution and multi-branch convolution, and then the boundary information is emphasized, thereby improving the accuracy and efficiency of polyp segmentation. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0045] Figure 1 It is a schematic flow chart of the polyp segmentation method provided by the present invention;

[0046] Figure 2 It is a structural schematic diagram of a parallel double-branch network model in the polyp segmentation method provided by the present invention;

[0047] Figure 3 is a schematic diagram of the structure of the fusion module in the polyp segmentation method provided by the present invention;

[0048] Figure 4 is a schematic structural diagram of a multi-scale enhancement module in the polyp segmentation method provided by the present invention;

[0049] Figure 5 It is a structural schematic diagram of a decoding module in the polyp segmentation method provided by the present invention;

[0050] Figure 6 It is a structural schematic diagram of the PPM module in the polyp segmentation method provided by the present invention;

[0051] Figure 7 is a schematic structural diagram of an edge enhancement module in the polyp segmentation method provided by the present invention;

[0052] Figure 8 This is one of the segmentation effect examples in the polyp segmentation method provided by the present invention;

[0053] Fig. 9 This is the second example of the segmentation effect in the polyp segmentation method provided by the present invention;

[0054] Fig.10 It is a schematic diagram comparing the segmentation results of different models in the polyp segmentation method provided by the present invention on the Kvasir-SEG dataset;

[0055] Fig.11 It is a schematic diagram comparing the segmentation results of different models in the polyp segmentation method provided by the present invention on the CVC-ClinicDB dataset;

[0056] Fig.12 It is a schematic diagram of the structure of the polyp segmentation device provided by the present invention;

[0057] Fig.13 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0058] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0059] Combine the following Figure 1 A polyp segmentation method of the present invention is described, comprising:

[0060] Step 101, using a pyramid vision Transformer model PVT to extract global features of a polyp image, using a convolutional neural network U-Net to extract local features of the polyp image, and using a fusion module to fuse the global features and local features;

[0061] In recent years, deep learning has been widely used in medical image segmentation. The U-shaped network based on the fully convolutional network has become the first choice for medical image segmentation models in recent years. The convolutional neural network gradually extracts the features of the image through convolution operations, which can capture local information well. However, due to the limited receptive field of the convolution operation, the convolutional neural network cannot establish long-range dependencies, which makes it difficult for the network to capture global information and the segmentation accuracy is difficult to further improve. Transformer can capture global information well because of its multi-head attention mechanism. Since it was introduced into medical image segmentation, it has shown good segmentation effects in specific segmentation tasks.

[0062] Among them, the visual Transformer model is a model based on the self-attention mechanism, which was originally used for natural language processing and later expanded to the field of computer vision. It divides the input image into small patches and then uses linear transformation to embed these patches into a vector space, using the self-attention mechanism to capture the global semantic information of the image.

[0063] U-Net is a classic convolutional neural network architecture for image segmentation tasks. It consists of a symmetrical encoder (downsampling path) and a decoder (upsampling path). The encoder extracts local features of the image by stacking convolutional layers and pooling layers, while the decoder restores spatial information and fine features through upsampling and skip connections.

[0064] like Figure 2 As shown in the figure, this model uses the pyramid vision Transformer model PVT to capture multi-scale global information and uses the convolutional neural network U-Net to capture local information. The advantages of the convolutional neural network and the transformer are combined, and the extracted features are passed through the fusion module so that the two can interact better to complete the encoding part. In order to better utilize the deep features, the multi-scale enhancement module (ME) is used in the jump connection part, and then decoding is performed. In order to perform multi-level supervision, each level of decoding has an output. Then, in order to better highlight the boundary information of the segmentation, the boundary enhancement module (DE) composed of the hole convolution and the attention mechanism is used.

[0065] In the transformer branch, the input image is first segmented into 4*4 patches through patch embedding, and then flattened as the input of PVT. It is divided into four stages in total. The feature size obtained in the first stage is one-fourth of the original image, and the feature size obtained in the following three stages is halved successively. In this way, four features of different scales are obtained, which captures multiple levels of information and is more conducive to segmenting polyp images of different sizes.

[0066] The other branch is composed of the encoder part of U-Net, which consists of 4 groups of convolution parts. The filter used in each convolution is a 2*2 maximum pooling layer, which halves the size of the feature map, and uses two convolutions with a kernel size of 3*3 and a step size of 1 for feature extraction. After each convolution, it is normalized and activated. After such convolution, four features of different scales are obtained to better capture local features. The size of the four scale features is the same as the feature size of the transformer branch.

[0067] PVT has a good ability to capture the global situation due to its attention mechanism, while the other part can capture local features well through convolution operation. Then the two parts of the features are fused through the fusion module. In this way, the features obtained after the fusion module can well represent the global and local semantic features.

[0068] Step 102, using a multi-scale enhancement module to process the fusion result of the fusion module through a spatial attention mechanism to emphasize spatial information layer by layer, and using a decoding module to decode the processing result;

[0069] In the skip link section, a multi-scale enhancement module is proposed to transfer the semantic information of high-level features to low-level features layer by layer by using the spatial attention mechanism.

[0070] Step 103, using the boundary enhancement module to process the fusion result of the fusion module through the hole convolution and attention mechanism to emphasize the boundary information, and fusing the processing result of the boundary enhancement module with the decoding result of the decoding module and decoding them again to obtain the segmentation result of the polyp image.

[0071] In order to better segment the edges of polyps, a boundary enhancement module is proposed, which uses hole convolution and multi-branch convolution to improve the ability to extract multi-scale information and make the boundaries clearer.

[0072] This embodiment proposes a parallel dual-branch network (PDB-Net), which captures local information through convolutional neural network branches and global information through PVT branches, and then fuses the obtained features through a fusion module to complete feature extraction. In order to better utilize deep features, a multi-scale enhancement module is proposed in the skip link part, which emphasizes spatial information layer by layer by using a spatial attention mechanism. In order to emphasize boundary information, a boundary enhancement module is proposed, which improves the ability to extract multi-scale information through void convolution and multi-branch convolution, and then emphasizes the boundary information, thereby improving the accuracy and efficiency of polyp segmentation.

[0073] Based on the above embodiment, the method of using a fusion module to fuse the global features and the local features in this embodiment includes:

[0074] When the current fusion module is a first-layer fusion module, convolving the global features and the local features of the same scale, and fusing the obtained features of the same size through Hadamard product;

[0075] When the current fusion module is not a first-layer fusion module, the final fusion result obtained by the previous-layer fusion module is fused with the features of the same size through Hadamard product;

[0076] The fusion result of the Hadamard product is convolved again, and then concatenated with the global feature and the local feature of the same scale;

[0077] The result of the splicing operation is passed through a residual network to obtain a final fusion result of the global feature and the local feature of the same scale.

[0078] The specific design of the fusion module is as follows: Figure 3 As shown. The first-layer fusion module first convolves the features from the two branches, and then obtains the features of the same size through the Hadamard product to fully fuse the features, and then concatenates them with the original features from the two branches after convolution, and finally passes through the residual network to ensure the robustness of the network. The calculation process is shown in formula (1) to formula (5). In the second to fourth layers, the fusion module will use the features obtained in the previous layer as input for fusion to emphasize the correlation of information under different modes.

[0079]

[0080] f i =Residual(CHannel_Concat[p i ,u i ,γ i ]) (5)

[0081] Among them, R represents all real numbers, M, N and P represent the number of channels of the feature map, and the relevant symbols represent the contents such as Figure 3 Shown in the lower right corner.

[0082] Based on the above embodiment, the multi-scale enhancement module in this embodiment processes the fusion result of the fusion module through the spatial attention mechanism to emphasize the spatial information layer by layer, including:

[0083] When the current fusion module is not a first-layer fusion module, a final fusion result output by the current fusion module is weighted by a spatial attention mechanism;

[0084] The weight corresponding to the fusion module of the i-th layer is upsampled i-1 times, and the upsampling results from the 1st to the i-1th times are multiplied by the final fusion results output by the fusion modules of the i-1th to the 1st layers, where i is the layer number of the fusion module except the 1st layer;

[0085] The result obtained by multiplying the final fusion result output by the i-th layer fusion module and the final fusion result output by the i-th layer fusion module are spliced.

[0086] In order to better transfer semantic information from high-level features upward, a multi-scale enhancement module based on spatial attention mechanism is proposed, such as Figure 4 As shown, unimportant background information is suppressed and important areas are highlighted.

[0087] The output of the fusion module is used as the input of the multi-scale enhancement module. First, the spatial attention mechanism is performed on the features of the 2nd to 4th layers to obtain weights. Because the resolution of high-level features is low, the obtained weights are upsampled by bilinear interpolation. For the weights obtained in the fourth layer, three upsamplings are performed and then multiplied with the features from the third to the first layer respectively. For the weights obtained in the third layer, two upsamplings are performed and then multiplied with the features of the second and first layers respectively. The weights obtained in the second layer only need to be upsampled once and multiplied with the features of the first layer. Finally, the output result of each layer is the concatenation of the features of this layer and the results obtained by multiplying the features of this layer with the weights from the deep layer. The relevant calculation process is shown in formulas (6) to (8). In this way, each underlying feature will obtain information from the deep features, which is more conducive to segmentation.

[0088]

[0089] Based on the above embodiment, the decoding module is used to decode the processing result in this embodiment, including:

[0090] When the current decoding module is a first-layer decoding module, the fusion result of the last-layer fusion module is input into the PPM module for processing, and the processing result of the fusion result of the last-layer fusion module is spliced ​​as the input of the current decoding module;

[0091] When the current decoding module is not the first-layer decoding module, the decoding result of the previous-layer decoding module and the fusion result of the N-(j-1)th-layer fusion module are concatenated as the input of the decoding module, where N is the total number of layers of the fusion module and i is the layer number of the current decoding module;

[0092] Performing three dilated convolutions on the input of the current decoding module after convolution;

[0093] The results of the first and second dilated convolutions are concatenated and then convolved.

[0094] The result obtained after the second convolution is concatenated with the result of the third dilated convolution, and then convolution and deconvolution are performed to obtain the decoding result of the current decoding module.

[0095] The structure of the decoding module is as follows Figure 5 As shown in Figure 2, the first decoding module has inputs from the output of the PPM module and Splice, then perform convolution, perform ordinary convolution and dilated convolution on the results respectively, splice the result of the first dilated convolution with the result of the second dilated convolution, and then perform convolution again. Splice the result with the result of the third dilated convolution. The purpose of this is to fully integrate the features and expand the receptive field to capture multi-scale contextual information. Then perform one more convolution and one more deconvolution, and then splice upward for the next step of decoding. The purpose of this is to fully integrate the features from the multi-scale enhancement module and the up-sampled features. Then the output of the first decoder is combined with Splicing, continue to input the decoding module, so that four segmentation result maps can be obtained through the decoding module, and the loss is calculated with the real label.

[0096] The calculation process is shown in formula (9) to formula (13), where Conv represents ordinary convolution, and we get Then DConv represents the Perform dilated convolution. The convolution kernel is 3*3, r i represents the size of the expansion rate, and DEConv stands for deconvolution.

[0097] The structure of the PPM module is as follows Figure 6 As shown in the figure, it mainly integrates multi-scale information through pyramid pooling, and has good segmentation ability for both small-sized and large-sized segmentation objects.

[0098] This model has a total of 5 decoding modules, and considering that the resolution of the input features of each decoding module is different, the expansion rates of the first and second decoding modules can be set to 1, 2, and 3 respectively, and the resolutions of the third to fifth modules can be set to 1, 3, and 5 respectively.

[0099]

[0100]

[0101] Based on the above embodiment, before using the pyramid vision Transformer model PVT to extract the global features of the polyp image and using the convolutional neural network U-Net to extract the local features of the polyp image, this embodiment also includes:

[0102] Calculating the loss of the segmentation result of the polyp image obtained by each layer of decoding module and the segmentation result obtained after the output of the boundary enhancement module is decoded;

[0103] Backward propagation is performed according to the loss to update the weights of the PVT, U-Net, multi-scale enhancement module, decoding module and boundary enhancement module.

[0104] like Figure 2 As shown, the entire model has five outputs in common, and the losses are calculated separately and added as the loss of the entire model, and then back propagation is performed to update the weight of the entire model. This embodiment does not limit the number of layers of the decoding module.

[0105] This embodiment proposes a deep supervised learning framework, which improves the ability to segment polyps by performing loss supervision on multi-level feature maps. Finally, good segmentation performance is achieved in comparative experiments and generalization experiments.

[0106] Based on the above embodiment, the boundary enhancement module in this embodiment processes the fusion result of the fusion module through the hole convolution and attention mechanism to emphasize the boundary information, including:

[0107] The dilated convolution and attention mechanism are used to enhance the local details of the fusion result output by the first-layer fusion module.

[0108] Combining the multi-scale enhancement module and the decoding module, relatively good segmentation results are obtained. However, the results obtained in this cascade structure are mainly affected by the low resolution, and higher-level features will lose local details, which leads to poor edge segmentation. To overcome this challenge, an edge enhancement module is further proposed, such as Figure 7 As shown in the figure, the dilated convolution and attention mechanism are used to enhance the local details of the low-level features, which are then fused with the features with high-level semantic information.

[0109] It is worth noting that because the original image is richer in details, through a series of convolutional layers, we try to directly extract these features. However, the original image also contains noise information. Therefore, more complex CNN modules accept direct extraction of intrinsic information. PVT1 is used as input, and the results are fused with features with high-level semantic information, and the result map is compared with the true label for loss.

[0110] This module obtains multi-scale information through different-sized dilated convolutions to expand the receptive field, and then uses spatial attention and channel attention to highlight important information, especially boundary information, for enhancement. The final result image is compared with the true label to calculate the loss.

[0111] On the basis of the above embodiment, the boundary enhancement module in this embodiment processes the fusion result of the fusion module through the hole convolution and attention mechanism to emphasize the boundary information, and fuses the processing result of the boundary enhancement module with the decoding result of the decoding module and decodes them again to obtain the segmentation result of the polyp image, including:

[0112] Convolving the fusion result output by the first-layer fusion module through multiple branches, wherein each branch except the last branch includes a normal convolution and a hole convolution, and the last branch includes a 1*1 convolution block;

[0113] After fusing the results obtained by the other branches, they are then fused with the result obtained by the last branch to form a residual connection, and processed by the Relu function;

[0114] The output of the Relu function is processed using spatial attention and channel attention in turn, and then added and multiplied with the decoding result of the last layer decoding module respectively, and the features obtained by the addition and multiplication are spliced ​​to obtain the segmentation result of the polyp image.

[0115] Depend on Figure 7 It can be seen that the multi-scale enhancement module consists of 5 branches. Each basic convolution module is composed of convolution, BatchNorm and ReLU. The calculation process is shown in formula (14).

[0116] BConv=Relu(BN(Conv(x))) (14)

[0117] Each branch is composed of one or more ordinary convolutions and dilated convolutions. The fifth branch consists of a 1*1 convolution block, which is used as a residual link to retain the original information. The calculation process of the five branches is shown in formulas (15) to (19).

[0118] x1=BConv 3*3 (BConv1*1 (x)) (15)

[0119]

[0120] x5=BConv 1*1 (x) (19)

[0122] The four branches on the left get the result y after the fusion of the results and the 3*3 convolution. i , and finally The residual connection is formed by fusion, and the Relu function is processed to obtain f i , as shown in formula (20) and formula (21), where Cat is a channel fusion operation.

[0123] y=Conv(Cat(x1+x2+x3+x4)) (20)

[0124] f=Relu(y+x5) (twenty one)

[0126] Finally, in order to highlight important semantic information, spatial and channel attention are performed in turn, and then the features from the decoded part are added and multiplied respectively, and finally the two features are concatenated. As shown in formula (22) and formula (23).

[0127]

[0128] For the model architecture, this embodiment provides a new deep convolutional neural network structure, which aims to effectively separate the polyp area from the complex background. The network combines the advantages of convolutional neural networks and Transformer, and enhances the edge details through the detail enhancement module, which can accurately capture the morphology and texture characteristics of polyps.

[0129] For training and optimization, we adjusted and optimized the parameters and loss function of the network through training on large-scale image data to maximize the generalization ability of the algorithm on the actual data set. The implementation effect evaluation was carried out on multiple public medical image data sets. By comparing with traditional methods and other advanced algorithms, it was proved that the algorithm of this embodiment has significant improvements in accuracy and processing speed. Specifically, the algorithm of this embodiment has a considerable improvement in segmentation accuracy compared with other networks.

[0130] After registration, log in to the system and click to import the polyp image that needs to be segmented. The specific implementation cases and results are as follows: Figure 8 and Fig. 9 shown.

[0131] In summary, this patent application describes an innovative polyp segmentation algorithm with excellent implementation cases and significant application effects. Our algorithm not only has breakthrough progress in theory, but also shows great potential and application prospects in actual medical practice. We believe that our invention has the technical features and innovation required for patent applications, can effectively solve the problems in the existing technology, and bring new solutions and progress to the field of polyp treatment.

[0132] The experimental results show that the Dice of this model on the Kvasir-SEG data is 0.9154, the IOU is 0.8623, the PRE is 0.9292, and the RE is 0.9212. Dice increased by 0.67 percentage points, IOU increased by 0.73 percentage points, and RE increased by 0.89 percentage points. On the CVC-ClinicDB dataset, Dice is 0.9530, IOU is 0.9119, PRE is 0.9458, and RE is 0.9625. Dice increased by 0.26 percentage points, IOU increased by 0.28 percentage points, and RE increased by 0.97 percentage points. The PRE index did not increase much or decreased, but overall the algorithm in this paper is better than other algorithms. The segmentation results of different segmentation networks are shown in Tables 1, 2, Fig.10 and Fig.11 As shown in the figure, the first row of each picture is the original picture and the segmentation results of different models, and the second row is the difference between the gold standard and the segmentation results of each model and the gold standard. The less yellow part means the better the segmentation effect. Through the result graph, we can intuitively find that the model proposed in this paper has a good segmentation effect.

[0133] Table 1 Comparison of results on the Kvasir-SEG dataset

[0134]

[0135]

[0136] Table 2 Comparison of results on the CVC-ClinicDB dataset

[0137]

[0138] The polyp segmentation device provided by the present invention is described below. The polyp segmentation device described below and the polyp segmentation method described above can be referred to each other.

[0139] like Fig.12 As shown, the device includes an extraction and fusion module 1201, a space enhancement module 1202 and a boundary enhancement module 1203, wherein:

[0140] The extraction and fusion module 1201 is used to extract the global features of the polyp image using the pyramid vision Transformer model PVT, extract the local features of the polyp image using the convolutional neural network U-Net, and fuse the global features and the local features using the fusion module;

[0141] The spatial enhancement module 1202 is used to use the multi-scale enhancement module to process the fusion result of the fusion module through the spatial attention mechanism to emphasize the spatial information layer by layer, and decode the processing result using the decoding module;

[0142] The boundary enhancement module 1203 is used to use the boundary enhancement module to process the fusion result of the fusion module through the hole convolution and attention mechanism to emphasize the boundary information, and fuse the processing result of the boundary enhancement module with the decoding result of the decoding module and decode it again to obtain the segmentation result of the polyp image.

[0143] In this embodiment, local information is captured by a convolutional neural network branch, global information is captured by a PVT branch, and the obtained features are then fused through a fusion module to complete feature extraction. In order to better utilize deep features, a multi-scale enhancement module is proposed in the skip link part, and spatial information is emphasized layer by layer by using a spatial attention mechanism. In order to emphasize boundary information, a boundary enhancement module is proposed, which uses dilated convolution and multi-branch convolution to improve the ability to extract multi-scale information, and then emphasizes boundary information, thereby improving the accuracy and efficiency of polyp segmentation.

[0144] Fig.13 An example of a physical structure diagram of an electronic device is shown in FIG. Fig.13As shown, the electronic device may include: a processor 1310, a communication interface 1320, a memory 1330 and a communication bus 1340, wherein the processor 1310, the communication interface 1320 and the memory 1330 communicate with each other through the communication bus 1340. The processor 1310 may call the logic instructions in the memory 1330 to execute the polyp segmentation method, which includes: extracting the global features of the polyp image using the pyramid visual Transformer model PVT, extracting the local features of the polyp image using the convolutional neural network U-Net, and fusing the global features and the local features using the fusion module; processing the fusion result of the fusion module using the multi-scale enhancement module through the spatial attention mechanism to emphasize the spatial information layer by layer, and decoding the processing result using the decoding module; processing the fusion result of the fusion module using the boundary enhancement module through the hole convolution and attention mechanism to emphasize the boundary information, and fusing the processing result of the boundary enhancement module with the decoding result of the decoding module and decoding again to obtain the segmentation result of the polyp image.

[0145] In addition, the logic instructions in the above-mentioned memory 1330 can be implemented in the form of software functional units and can be stored in a computer-readable storage medium when sold or used as an independent product. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes.

[0146] On the other hand, the present invention also provides a computer program product, which includes a computer program. The computer program can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the polyp segmentation method provided by the above methods, which includes: using a pyramid visual Transformer model PVT to extract global features of the polyp image, using a convolutional neural network U-Net to extract local features of the polyp image, and using a fusion module to fuse the global features and local features; using a multi-scale enhancement module to process the fusion result of the fusion module through a spatial attention mechanism to emphasize spatial information layer by layer, and using a decoding module to decode the processing result; using a boundary enhancement module to process the fusion result of the fusion module through a hole convolution and an attention mechanism to emphasize boundary information, and fusing the processing result of the boundary enhancement module with the decoding result of the decoding module and decoding them again to obtain the segmentation result of the polyp image.

[0147] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which is implemented when the processor executes the polyp segmentation method provided by the above-mentioned methods, the method comprising: using a pyramid visual Transformer model PVT to extract global features of a polyp image, using a convolutional neural network U-Net to extract local features of the polyp image, and using a fusion module to fuse the global features and the local features; using a multi-scale enhancement module to process the fusion result of the fusion module through a spatial attention mechanism to emphasize spatial information layer by layer, and using a decoding module to decode the processing result; using a boundary enhancement module to process the fusion result of the fusion module through a hole convolution and an attention mechanism to emphasize boundary information, and fusing the processing result of the boundary enhancement module with the decoding result of the decoding module and decoding them again to obtain the segmentation result of the polyp image.

[0148] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0149] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0150] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A polyp segmentation method, characterized in that: include: Extract global features of polyp images using a pyramid vision Transformer model PVT, extract local features of the polyp images using a convolutional neural network U-Net, and fuse the global features with the local features using a fusion module; Using a multi-scale enhancement module to process the fusion result of the fusion module through a spatial attention mechanism to emphasize spatial information layer by layer, and using a decoding module to decode the processing result; The boundary enhancement module is used to process the fusion result of the fusion module through the hole convolution and attention mechanism to emphasize the boundary information, and the processing result of the boundary enhancement module is fused with the decoding result of the decoding module and decoded again to obtain the segmentation result of the polyp image.

2. The polyp segmentation method according to claim 1, characterized in that: The using a fusion module to fuse the global features and the local features comprises: When the current fusion module is a first-layer fusion module, convolving the global features and the local features of the same scale, and fusing the obtained features of the same size through Hadamard product; When the current fusion module is not a first-layer fusion module, the final fusion result obtained by the previous-layer fusion module is fused with the features of the same size through Hadamard product; The fusion result of the Hadamard product is convolved again, and then concatenated with the global feature and the local feature of the same scale; The result of the splicing operation is passed through a residual network to obtain a final fusion result of the global feature and the local feature of the same scale.

3. The polyp segmentation method according to claim 1, characterized in that: The multi-scale enhancement module is used to process the fusion result of the fusion module through a spatial attention mechanism to emphasize spatial information layer by layer, including: When the current fusion module is not a first-layer fusion module, a final fusion result output by the current fusion module is weighted by a spatial attention mechanism; The weight corresponding to the fusion module of the i-th layer is upsampled i-1 times, and the upsampling results from the 1st to the i-1th times are multiplied by the final fusion results output by the fusion modules of the i-1th to the 1st layers, where i is the layer number of the fusion module except the 1st layer; The result obtained by multiplying the final fusion result output by the i-th layer fusion module and the final fusion result output by the i-th layer fusion module are spliced.

4. The polyp segmentation method according to claim 1, characterized in that: The method of decoding the processing result by using a decoding module includes: When the current decoding module is a first-layer decoding module, the fusion result of the last-layer fusion module is input into the PPM module for processing, and the processing result of the fusion result of the last-layer fusion module is spliced ​​as the input of the current decoding module; When the current decoding module is not the first-layer decoding module, the decoding result of the previous-layer decoding module and the fusion result of the N-(j-1)th-layer fusion module are concatenated as the input of the decoding module, where N is the total number of layers of the fusion module and i is the layer number of the current decoding module; Performing three dilated convolutions on the input of the current decoding module after convolution; The results of the first and second dilated convolutions are concatenated and then convolved. The result obtained after the second convolution is concatenated with the result of the third dilated convolution, and then convolution and deconvolution are performed to obtain the decoding result of the current decoding module.

5. The polyp segmentation method according to claim 4, characterized in that: Before extracting the global features of the polyp image using the pyramid vision Transformer model PVT and extracting the local features of the polyp image using the convolutional neural network U-Net, the method further includes: Calculating the loss of the segmentation result of the polyp image obtained by each layer of decoding module and the segmentation result obtained after the output of the boundary enhancement module is decoded; Backward propagation is performed according to the loss to update the weights of the PVT, U-Net, multi-scale enhancement module, decoding module and boundary enhancement module.

6. The polyp segmentation method according to claim 1, characterized in that: The using boundary enhancement module to process the fusion result of the fusion module through hole convolution and attention mechanism to emphasize boundary information includes: The dilated convolution and attention mechanism are used to enhance the local details of the fusion result output by the first-layer fusion module.

7. The polyp segmentation method according to claim 6, characterized in that: The use of the boundary enhancement module to process the fusion result of the fusion module through the hole convolution and attention mechanism to emphasize the boundary information, and fusing the processing result of the boundary enhancement module with the decoding result of the decoding module and decoding them again to obtain the segmentation result of the polyp image, including: Convolving the fusion result output by the first-layer fusion module through multiple branches, wherein each branch except the last branch includes a normal convolution and a hole convolution, and the last branch includes a 1*1 convolution block; After fusing the results obtained by the other branches, they are then fused with the result obtained by the last branch to form a residual connection, and processed by the Relu function; The output of the Relu function is processed using spatial attention and channel attention in turn, and then added and multiplied with the decoding result of the last layer decoding module respectively, and the features obtained by the addition and multiplication are spliced ​​to obtain the segmentation result of the polyp image.

8. A polyp segmentation device, characterized in that: include: An extraction and fusion module is used to extract global features of the polyp image using a pyramid vision Transformer model PVT, extract local features of the polyp image using a convolutional neural network U-Net, and fuse the global features with the local features using a fusion module; A spatial enhancement module, used for processing the fusion result of the fusion module by using a multi-scale enhancement module through a spatial attention mechanism to emphasize spatial information layer by layer, and decoding the processing result by using a decoding module; The boundary enhancement module is used to process the fusion result of the fusion module through the hole convolution and attention mechanism to emphasize the boundary information, and fuse the processing result of the boundary enhancement module with the decoding result of the decoding module and decode it again to obtain the segmentation result of the polyp image.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the polyp segmentation method according to any one of claims 1 to 7 is implemented.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the polyp segmentation method according to any one of claims 1 to 7 is implemented.

Images