Pixel-level polyp segmentation method based on parallel feature enhancement and attention mechanism

Abstract

The application provides a pixel-level polyp segmentation method based on parallel feature enhancement and attention mechanisms, wherein a network model adopts HarDNet68 as an encoder to extract multi-scale feature information; in a skip connection part, a parallel feature enhancement module is embedded to extract high-level representations of features; meanwhile, a parallel part decoder is used to aggregate high-level features to generate a global guide; finally, a reverse attention module is used to guide information to propagate from a global range to a local range, so as to enhance the fusion and refinement of boundary features.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer vision, and particularly relates to a pixel-level polyp segmentation method based on parallel feature enhancement and attention mechanism. BACKGROUND

[0002] Colorectal cancer originates from benign polyps, and can be prevented if it is found and removed before adenoma develops into cancer. Therefore, early screening and diagnosis play a key role in the prevention and treatment of colorectal cancer. Colonoscopy is a common method for screening colon lesions [1] , but due to the diversity of polyp appearance, long-time work visual fatigue of doctors and other factors, polyps may be missed or misdiagnosed. Therefore, it is very important to develop a computer-aided tool to help endoscopists make judgments.

[0003] In the field of computer vision, image segmentation is considered an important research direction. Using deep learning technology for polyp segmentation can learn image features end-to-end, adapt to different data sets, and help doctors quickly determine the location of polyps. Brandao et al. [2] used a fully convolutional network to identify and segment polyps in colonoscopy images, converted three existing networks into a fully convolutional structure, and fine-tuned the learned representations to complete the polyp image segmentation task, but this method lacks the fusion of shallow and high-level information. In 2019, Jha et al. [3] proposed ResUnet++ for polyp segmentation, which combines the advantages of deep residual learning [4] and U-Net [5] architecture. The residual block propagates information across layers, building a deeper neural network that addresses the degradation problem of each encoder. To address the problem of unclear boundaries between polyps and their surrounding mucosa, Fan et al. designed PraNet [6] , which uses reverse attention to establish relationships between regional and boundary cues and mine boundary cues. Considering that different levels of features contribute differently when transferring information between encoders and decoders in polyp segmentation, Dong et al. proposed Polyp-PVT [7] . It uses a pyramid vision transformer as an encoder to process low-level and high-level features separately, and then uses a similarity aggregation module (SAM) to expand polyp pixel features with high-level semantic location information to the entire polyp region, effectively fusing cross-layer features. MSNet [8] designs a subtraction unit to generate difference features between adjacent levels in the encoder, and pyramidically equips the subtraction unit with different receptive fields at different levels to obtain rich multi-scale difference information. Zhou et al. designed a cross-layer feature aggregation network (CFA-Net) for polyp segmentation [9]The architecture is based on a two-stream structure, which extracts hierarchical semantic information from cross-layer features. Finally, a boundary aggregation module is used to integrate boundary information into the segmentation network. Shao et al.

[10] A new boundary-sensitive polygon segmentation framework, Polyper, is designed. The boundary and polyp region are extracted from the initial segmentation map by morphological operators. Although the above method performs well in segmentation accuracy, there are still some problems, such as: the morphology and appearance of polyps will change greatly in different cases, the diversity of the data set, and the insufficient generalization ability of small sample data.

[0004] [1]Hong Z, Zhao X, Wang C. Colorectal cancer screening research status[J]. Journal of Dalian Medical University, 2022, 44(6): 534-538.

[0005] [2]Brandao P, Mazomenos E, Ciuti G, et al. Fully convolutional neural networks for polyp segmentation in colonoscopy[C] / / Medical Imaging 2017: Computer-Aided Diagnosis. Spie, 2017, 10134: 101-107.

[0006] [3]JHA D, Smedsrud P H, Riegler M A, et al. Resunet++: an advanced architecture for medical image segmentation: 2019 IEEE International Symposium on Multimedia (ISM)[C]: IEEE, 2019

[0007] [4]K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in Proceedings of IEEE conference on computer vision and pattern recognition (CVPR), 2016, pp. 770-778.

[0008] [5] O Ronneberger, P. Fischer, T. Brox. U-Net: Convolutional Networks for Biomedical Image Segmentation [C]. In: International Conference on Medical Image Computing and Computer-Assisted Intervention, Springer, Cham, 2015, pp. 234-241.

[0009] [6] D P Fan, G P Ji, T Zhou, et al. Pranet: Parallel Reverse Attention Network for Polyp Segmentation [C]. In: International Conference on Medical Image Computing and Computer-assisted Intervention: Springer, 2020, pp. 263-273

[0010] [7] Dong B, Wang W, Fan D P, et al. Polyp-pvt: Polyp segmentation with pyramid vision transformers [J]. arXiv preprint arXiv:2108.06932, 2021

[0011] [8] Zhao X, Zhang L, Lu H. Automatic polyp segmentation via multi-scale subtraction network [C]. International Conference on Medical Image Computing and Computer Assisted Intervention, Virtual Event, 2021, 120-130

[0012] [9] Zhou T, Zhou Y, He K, et al. Cross-level feature aggregation network for polyp segmentation [J]. Pattern Recognition, 2023, 140: 109555

[0013]

[10] Shao H, Zhang Y, Hou Q. Polyper: Boundary Sensitive Polyp Segmentation [C] / / Proceedings of the AAAI Conference on Artificial Intelligence. 2024, 38(5): 4731-4739. SUMMARY

[0014] In view of the defects and deficiencies of the prior art, the present application provides a pixel-level polyp segmentation method based on parallel feature enhancement and attention mechanism. The network model based on the present application uses HarDNet68 as an encoder to extract multi-scale feature information. In the jump connection part, a parallel feature enhancement module is embedded to extract high-level representations of the features. At the same time, a parallel part decoder is used to aggregate high-level features to generate a global guide. Finally, a reverse attention module is used to guide information to propagate from the global range to the local range, enhancing the fusion and refinement of boundary features.

[0015] The present application specifically adopts the following technical solutions:

[0016] A pixel-level polyp segmentation method based on parallel feature enhancement and attention mechanism:

[0017] For a given polyp image, multi-scale feature extraction is performed using different convolution layers of the HarDNet backbone network to obtain four features of different levels f i ; Then, three higher-level features f i are input into a parallel feature enhancement module composed of channel attention and spatial attention to enhance feature representation; the channel attention module performs channel weighting processing on the input features; the spatial attention module performs spatial weighting processing on the channel-weighted features; at the same time, a parallel part decoder is used to aggregate high-level features f i to obtain a global mapping S g as a global guide for the subsequent decoding part;

[0018] The global mapping S gand the highest level feature f4 after construction is sent into a reverse attention module for processing to obtain a prediction feature R4, and the prediction feature R4 and the global mapping Sg are added to obtain a prediction map result S4; the prediction feature map S4 and the high-level feature f3 after construction are input into a reverse attention module, and the prediction result S3 is obtained after processing; the prediction result S3 and the feature f2 after construction are input into a reverse attention module, and the prediction result S2 is obtained after processing; the loss loss is calculated by using the prediction map result and the real result to guide the network training, and finally a polyp segmentation model with the best performance is obtained to perform pixel-level polyp segmentation.

[0019] Further, the parallel feature enhancement module combines a plurality of dilated convolution layers with different dilation rates, and the output of each dilated convolution layer is connected with a channel attention mechanism module and a spatial attention mechanism module; after splicing between the output features of each dilated convolution layer, the original input feature is directly added to the enhanced feature to retain the information of the original feature and enhance the expression ability of the network; finally, the spliced feature map is processed using a ReLU activation function and is transmitted to a 1x1 convolution layer for dimension transformation to obtain a more detailed and accurate feature set.

[0020] Further, in the parallel partial decoder, each feature is first subjected to a receptive field module RFB to enhance the convolutional neural network feature representation; then each feature is aggregated by element-by-element multiplication; finally, a global mapping Sg=pd(f3,f4,f5) is obtained through a convolution and activation operation as the global guidance of the reverse attention module RA.

[0021] Further, the reverse attention module gradually excavates the polyp area by erasing the foreground object, wherein the operation of erasing the foreground is:

[0022] Rw=1-sigmoid(P(S i )) (1)

[0023] wherein P(·) represents an up-sampling operation; the reverse attention module is represented as:

[0024] R i =R w *f i (2).

[0025] In addition, a pixel-level polyp segmentation system based on parallel feature enhancement and attention mechanism is provided, which performs polyp image segmentation by using the polyp segmentation model obtained by the above method.

[0026] An electronic device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, the processor implementing the steps of the pixel-level polyp segmentation method based on parallel feature enhancement and attention mechanism as described above when executing the program.

[0027] A non-transitory computer-readable storage medium having stored thereon a computer program, the computer program implementing the steps of the pixel-level polyp segmentation method based on parallel feature enhancement and attention mechanism as described above when executed by a processor.

[0028] Compared with the prior art, the present application and its preferred schemes have better robustness and better segmentation effect, and can capture tiny and fine boundary details. BRIEF DESCRIPTION OF DRAWINGS

[0029] The present application will be further described in detail below in conjunction with the accompanying drawings and specific embodiments:

[0030] Figure 1 The polyp segmentation model of the embodiment of the present application.

[0031] Figure 2 The HarDNet network structure diagram.

[0032] Figure 3 The parallel feature enhancement module structure diagram of the embodiment of the present application.

[0033] Figure 4 The parallel partial decoder module structure diagram of the embodiment of the present application.

[0034] Figure 5 The reverse attention module structure diagram of the embodiment of the present application.

[0035] Figure 6 The polyp segmentation result visualization comparison diagram of each method. DETAILED DESCRIPTION

[0036] In order to make the features and advantages of the present patent more apparent, the following specific embodiments are described in detail below, and the accompanying drawings are used for detailed description as follows:

[0037] It should be noted that the following detailed description is exemplary and is intended to provide further explanation of the present application. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as generally understood by those skilled in the art to which the present application belongs.

[0038] It is to be understood that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of example embodiments in accordance with the present application. As used herein, the singular forms "a", "an" and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will be further understood that the terms "comprises" and / or "comprising," when used in this specification, specify the presence of stated features, steps, operations, elements, components, and / or groups thereof, but do not preclude the presence or addition of one or more other features, steps, operations, elements, components, and / or groups thereof.

[0039] The pixel-level polyp segmentation network structure based on parallel feature enhancement and attention mechanism proposed in the embodiments of the present application is shown in Figure 1 The model is composed of an encoder, a parallel feature enhancement module, a parallel partial decoder and a reverse attention module.

[0040] Among them, multi-scale feature extraction is performed on a given polyp image by means of different convolution layers of the HarDNet backbone network, and four features f i (i = 1, 2, 3, 4) of different levels are obtained. The three higher-level features f i (i = 2, 3, 4) are input into a parallel feature enhancement (PFE) module composed of channel attention and spatial attention to enhance the feature representation. Among them, the channel attention module performs channel weighting processing on the input features, emphasizing the features of important channels. The spatial attention module performs spatial weighting processing on the channel-weighted features, highlighting the relatively important regions in the image. The parallel partial decoder is used to aggregate the high-level features f i (i = 2, 3, 4) to obtain a global mapping S g as a global guide for the subsequent decoding part. The global mapping S g and the highest-level feature f4 constructed are input into a reverse attention module for processing to obtain a prediction feature R4, and then the prediction feature R4 and the global mapping Sg are added to obtain a prediction map result S4. By similar steps, the prediction feature map S4 and the constructed high-level feature f3 are input into the reverse attention module to obtain a prediction result S3. Then the prediction result S3 and the constructed feature f2 continue to be operated in a similar manner to obtain a prediction result S2. The loss loss is calculated using the prediction map result and the true result to guide the network training, and finally a polyp segmentation model with the best performance is obtained for pixel-level polyp segmentation.

[0041] The following further introduces each module of the model:

[0042] A. Encoding backbone

[0043] The present application uses HarDNet as a backbone network to extract multi-scale feature maps (f1, f2, f3 and f4). HarDNet is a low-memory and low-power network improved based on Densenet, which removes global dense connection and uses MaxPool for downsampling (as shown in the figure). Figure 2

[0044] B. Parallel feature enhancement module

[0045] The parallel feature enhancement module (PFE) module adopts the idea of residual connection, combines multiple dilated convolution layers with different expansion rates, and captures more extensive context information. The output of each dilated convolution layer contains a channel attention mechanism (CAM) and a spatial attention mechanism (SAM), which integrates image-specific channel and location features. After splicing between the output features of each dilated convolution layer, the idea of residual connection is introduced to directly add the original input features to the enhanced features to retain the information of the original features and enhance the expression ability of the network. Finally, the spliced feature map is processed using the ReLU activation function and passed to a 1x1 convolution layer for dimension transformation to obtain a more detailed and accurate feature set. Combining PFE into the skip connection of the network can remove noise and enhance the underlying representation information of polyps, improving the segmentation performance of the model. The structure diagram of the parallel feature enhancement module is as shown in the figure. Figure 3

[0046] C. Parallel partial decoder

[0047] The present application uses a special decoder (PPD) to aggregate high-level advanced features from high layers (as shown in the figure). Figure 4 Each feature first passes through a receptive field module (RFB, Receptive Field Block) to enhance the convolutional neural network feature representation. Then each feature performs aggregation through element-by-element multiplication. Finally, after a convolution and activation operation, a global mapping Sg=pd(f3,f4,f5) is obtained as the global guidance of the reverse attention module RA. The structure diagram of the parallel partial decoder is as shown in the figure. Figure 4

[0048] D. Reverse attention module

[0049] The aggregated advanced features and enhanced features are input into the reverse attention module, which (as shown in the figure) gradually excavates the polyp area by erasing the foreground objects, and adaptively learns the reverse attention in the three parallel advanced features, which can effectively improve the visibility of target edges and details. The operation of erasing the foreground is: Figure 5

[0050] Rw=1-sigmoid(P(S i ) (1) ​​​​

[0051] Where P(·) represents the upsampling operation. The inverse attention module is represented as:

[0052] R i =R w *f i (2).

[0053] Test Instance

[0054] 1. Experimental Dataset and Evaluation Metrics

[0055] The test experiments provided in this embodiment of the invention use the CVC-612 and Kvasir-SEG datasets, which are divided into training set and test set in a ratio of 9:1.

[0056] Six evaluation metrics are used, including Dice, IoU, mean absolute error (MAE), s-measure (Sα), precision, and weighted F-measure. They are calculated as follows, where tn represents true negatives, tp represents the number of true positives, fp represents the number of false positives, and fn represents false negatives.

[0057] (1) Dice calculates the degree of overlap between the predicted region and the real region, focusing on evaluating the spatial overlap of the segmentation results. This is represented as:

[0058]

[0059] (2) mIOU is the ratio of the intersection of the predicted segment and the true segment to their union, and it focuses on evaluating the spatial coverage of the segmentation results. It is expressed as:

[0060]

[0061] (3) MAE is the mean absolute difference between the predicted and actual values, focusing on evaluating the pixel-level error of the segmentation results. It is expressed as:

[0062]

[0063] Where w and h represent the width and height of the image, respectively, and (x,y) represents the coordinates of each pixel; P(x,y) and G(x,y) represent the predicted value and the true value, respectively.

[0064] (4) The S-measure combines global similarity and local consistency to measure the structural similarity between the predicted map and the ground truth mask. It focuses on evaluating the global and local quality of the segmentation results. It is expressed as:

[0065] S=α*S o +(1-α)*S r (6)

[0066] wherein, in the formula, a ∈ [0, 1], So represents the object perception structural similarity, and Sr represents the region perception structural similarity.

[0067] (5) The precision is a commonly used index in classification and segmentation tasks, which measures the proportion of correct predictions in the prediction results. In image segmentation, the precision represents the proportion of correctly predicted positive pixels in the predicted pixels. The precision focuses on evaluating the accuracy and false positive rate of the prediction results. It is represented as:

[0068]

[0069] (6) The weighted F-measure (F-measure) is a comprehensive index obtained by calculating the weighted average of precision and recall, and the weight can be adjusted according to specific circumstances. It focuses on evaluating the comprehensive performance of the prediction results. Precision w represents the weighted precision, Recall w represents the weighted recall, and β is a parameter. It is represented as

[0070]

[0071] 3 Experimental results

[0072] Tables 1 and 2 give the quantitative performance comparison of the present application and other existing methods on the CVC-612 and Kvasir-SEG datasets. It can be seen that the model of the present application has good robustness. In addition, Figure 6 shows some qualitative examples on the polyp dataset. It can be seen that the method of the present application is superior to other methods and can capture subtle and fine boundary details.

[0073] Table 1 Evaluation of segmentation effect of each network on the CVC-612 dataset

[0074]

[0075] Table 2 Evaluation of segmentation effect of each network on the Kvasir-SEG dataset

[0076]

[0077] Those skilled in the art will appreciate that embodiments of the application can be provided as methods, systems, or computer program products. Accordingly, the application can take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the application can take the form of a computer program product on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROMs, optical storage media, etc.) having computer-usable program code embodied in the medium.

[0078] The computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart block or blocks. Figure 1 one or more flow or blocks Figure 1 means for functionally implementing the steps in one or more flow or blocks.

[0079] These computer program instructions can also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instructions which implement the function specified in the flowchart block or blocks. Figure 1 one or more flow or blocks Figure 1 means for functionally implementing the steps in one or more flow or blocks.

[0080] The computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart block or blocks. Figure 1 one or more flow or blocks Figure 1 means for functionally implementing the steps in one or more flow or blocks. The patent is not limited to the above best mode, and anyone can derive other various forms of pixel-level polyp segmentation methods based on parallel feature enhancement and attention mechanism under the inspiration of the patent. Any equivalent changes and modifications made within the scope of the patent application should be covered by the patent.

Claims

1. A pixel-level polyp segmentation method based on parallel feature enhancement and attention mechanism, characterized in that: Multi-scale feature extraction is performed on a given polyp image by different convolution layers of the HarDNet backbone network to obtain four features at different levels Then the three higher-level features are input into a corresponding parallel feature enhancement module composed of channel attention and spatial attention to enhance the feature representation; the channel attention module performs channel weighting processing on the input feature; the spatial attention module performs spatial weighting processing on the channel-weighted feature; and a parallel partial decoder is used to aggregate the high-level features to obtain a global mapping S g as a global guide for the subsequent decoding part. The global mapping S g and the highest-level feature f4 after construction are sent into a reverse attention module for processing to obtain a predicted feature R4. Then, the predicted feature R4 and the global mapping S g are added to obtain a predicted map result S4. The predicted map result S4 and the high-level feature f3 after construction are input into a reverse attention module for processing to obtain a prediction result S3. The prediction result S3 and the feature f2 after construction are input into a reverse attention module for processing to obtain a prediction result S2. A loss loss is calculated by using the predicted map result and the real result to guide network training, and finally, an optimal performance polyp segmentation model is obtained to perform pixel-level polyp segmentation. The parallel feature enhancement module combines multiple dilated convolution layers with different dilation rates, and the output of each dilated convolution layer is connected with a channel attention mechanism module and a spatial attention mechanism module; after splicing between the output features of each dilated convolution layer, the original input features are directly added to the enhanced features to retain the information of the original features and enhance the expression ability of the network; finally, the spliced feature map is processed using the ReLU activation function and passed to a 1x1 convolution layer for dimension transformation to obtain a more detailed and accurate feature set; In the parallel partial decoder, each feature is first enhanced by a receptive field block RFB to enhance the convolutional neural network feature representation; then each feature is aggregated by element-by-element multiplication; finally, a convolution and activation operation is performed to obtain a global mapping S g = pd (f3, f4, f5), as the global guidance of the reverse attention module RA; The reverse attention module gradually excavates the polyp area by erasing the foreground object, wherein the operation of erasing the foreground is: （1） wherein denotes an up-sampling operation; the reverse attention module is denoted as: * （2）。 2. A pixel-level polyp segmentation system based on parallel feature enhancement and attention mechanism, characterized in that, A polyp segmentation model obtained by training the method of claim 1 is used for polyp image segmentation.

3. An electronic device comprising a memory, a processor, and a computer program stored on the memory and capable of running on the processor, characterized in that, The processor executes the program to implement the steps of the pixel-level polyp segmentation method based on parallel feature enhancement and attention mechanism of claim 1.

4. A non-transitory computer-readable storage medium having stored thereon a computer program, characterized in that, The computer program is executed by the processor to implement the steps of the pixel-level polyp segmentation method based on parallel feature enhancement and attention mechanism of claim 1.

Images