Pixel-level polyp segmentation method based on parallel feature enhancement and attention mechanism

Abstract

The invention provides a pixel-level polyp segmentation method based on parallel feature enhancement and an attention mechanism. A network model adopts HarDNet68 as an encoder to extract multi-scale feature information; in the jump connection part, a parallel feature enhancement module is embedded to extract high-level representation of features; meanwhile, the advanced features are aggregated by using a parallel partial decoder to generate a global guidance; and finally, a reverse attention module is adopted to guide information to be spread from a global range to a local range, and fusion and refinement of boundary features are enhanced.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and in particular to a pixel-level polyp segmentation method based on parallel feature enhancement and attention mechanism. Background Art

[0002] Colorectal cancer originates from benign polyps and can be prevented if the adenoma is found and removed before it develops into cancer. Therefore, early screening and diagnosis play a key role in colorectal cancer prevention and treatment. Colonoscopy is a common method for screening colon lesions. [1] However, due to the diversity of polyps in appearance, factors such as long-term work and visual fatigue of doctors may lead to misdetection or omission of polyps. Therefore, it is very important to research and develop computer-aided tools to help endoscopists make judgments.

[0003] In the field of computer vision, image segmentation is regarded as an important research direction. Using deep learning technology for polyp segmentation can learn image features end-to-end, adapt to different data sets, and help doctors quickly determine the location of polyps. Brandao et al. [2] A fully convolutional network is used to identify and segment polyps in colonoscopy images. Three existing networks are converted to fully convolutional structures and their learned representations are fine-tuned to complete the polyp image segmentation task. However, this method lacks the fusion of shallow and high-level information. In 2019, Jha et al. [3] proposed ResUnet++ for polyp segmentation, which combines deep residual learning with deep learning. [4] and U-Net [5] The residual block propagates information across layers, building a deeper neural network and solving the degradation problem of each encoder. In order to solve the problem of unclear boundaries between polyps and their surrounding mucosa, Fan et al. designed PraNet [6] , using reverse attention to establish the relationship between regions and boundary clues and mine boundary clues. Considering that different levels of features contribute differently to the segmentation task when transferring information between the encoder and decoder in polyp segmentation, Dong et al. proposed Polyp-PVT [7] It uses a pyramid visual transformer as an encoder to process low-level features and high-level features separately, and then uses a similarity aggregation module (SAM) to expand the polyp pixel features with high-level semantic position information to the entire polyp area, effectively fusing cross-layer features. MSNet [8] A subtraction unit is designed to generate difference features between adjacent levels in the encoder, and subtraction units with different receptive fields are equipped in a pyramidal manner at different levels to obtain rich multi-scale difference information. Zhou et al. designed a cross-layer feature aggregation network (CFA-Net) for polyp segmentation [9]The architecture is based on a two-stream structure to mine hierarchical semantic information from cross-layer features. Finally, a boundary aggregation module is used to integrate boundary information into the segmentation network. Shao et al.

[10] A new boundary-sensitive polygon segmentation framework Polyper is designed. The boundary and polyp region are extracted from the initial segmentation map through morphological operators. Although the above method performs well in segmentation accuracy, there are still some problems, such as: the morphology and appearance of polyps vary greatly in different cases, the diversity of data sets, and the lack of generalization ability of small sample data.

[0004] [1] Hong Zhijun, Zhao Xiaoyu, Wang Chengfang. Current status of colorectal cancer screening research[J]. Journal of Dalian Medical University, 2022, 44(6): 534-538.

[0005] [2]Brandao P, Mazomenos E, Ciuti G, et al.Fully convolutional neural networks for polyp segmentation in colonoscopy[C] / / Medical Imaging 2017: Computer-Aided Diagnosis.Spie, 2017,10134:101-107.

[0006] [3]JHA D, SMEDSRUD PH, RIEGLER MA, et al. Resunet++: an advanced architecture for medical image segmentation: 2019IEEE International Symposiumon Multimedia (ISM) [C]: IEEE, 2019

[0007] [4] K.He,

[0008] [5]O Ronneberger,P.Fischer,T.Brox.U-Net:Convolutional Networks forBiomedical Image Segmentation[C].In:International Conference on Medical ImageComputing and Computer-Assisted Intervention,Springer,Cham,2015,pp.234-241.

[0009] [6]D P Fan,G P Ji,T Zhou,et al.Pranet:Parallel Reverse AttentionNetwork for Polyp Segmentation[C].In:International Conference on MedicalImage Computing and Computer-assisted Intervention:Springer,2020,pp.263-273

[0010] [7]Dong B,Wang W,Fan D P,et al.Polyp-pvt:Polyp segmentationwithpyramid vision transformers[J].arXiv preprint arXiv:2108.06932,2021

[0011] [8]Zhao X,Zhang L,Lu H.Automatic polyp segmentation via multi-scalesubtraction network[C].International Conference on Medical Image Computingand ComputerAssisted Intervention,Virtual Event,2021,120-130

[0012] [9]Zhou T,Zhou Y,He K,et al.Cross-level feature aggregation networkfor polyp segmentation[J].Pattern Recognition,2023,140:109555

[0013]

[10] Shao H, Zhang Y, Hou Q. Polyper: Boundary Sensitive PolypSegmentation[C] / / Proceedings of the AAAI Conference on ArtificialIntelligence.2024,38(5):4731-4739. Summary of the invention

[0014] In view of the defects and shortcomings of the prior art, the present invention proposes a pixel-level polyp segmentation method based on parallel feature enhancement and attention mechanism. The network model based on it uses HarDNet68 as an encoder to extract multi-scale feature information. In the jump connection part, a parallel feature enhancement module is embedded to extract the high-level representation of the feature. At the same time, the parallel partial decoder is used to aggregate the high-level features to generate a global guide. Finally, the reverse attention module is used to guide the information to propagate from the global scope to the local scope, enhancing the fusion and refinement of boundary features.

[0015] The present invention specifically adopts the following technical solutions:

[0016] A pixel-wise polyp segmentation approach based on parallel feature enhancement and attention mechanism:

[0017] For a given polyp image, multi-scale feature extraction is performed with the help of different convolutional layers of the HarDNet backbone network to obtain four different levels of features f i ; Then the three higher-level features f i The features are respectively input into a corresponding parallel feature enhancement module consisting of channel attention and spatial attention to enhance feature representation; the channel attention module performs channel weighting processing on the input features; the spatial attention module performs spatial weighting processing on the channel-weighted features; and a parallel partial decoder is used to perform spatial weighting on the high-level features f i Aggregate and get a global mapping S g Serves as a global guide for the subsequent decoding part;

[0018] The global map S g And the highest level feature f after construction 4It is sent to a reverse attention module for processing to obtain the predicted feature R4, and then the predicted feature R4 and the global mapping Sg are added to obtain the predicted graph result S4; the predicted feature graph S4 and the constructed high-level feature f3 are input into a reverse attention module, and the predicted result S3 is obtained after processing; the predicted result S3 and the constructed feature f2 are input into a reverse attention module, and the predicted result S2 is obtained after processing; the loss loss is calculated using the predicted graph result and the actual result to guide the network training, and finally a polyp segmentation model with the best performance is obtained for pixel-level polyp segmentation.

[0019] Furthermore, the parallel feature enhancement module combines multiple dilated convolutional layers with different dilation rates, and the output of each dilated convolutional layer is connected to a channel attention mechanism module and a spatial attention mechanism module; after splicing the output features of each dilated convolutional layer, the original input features are directly added to the enhanced features to retain the information of the original features and enhance the expression ability of the network; finally, the spliced ​​feature map is processed using the ReLU activation function and passed to a 1x1 convolutional layer for dimensional transformation to obtain a more detailed and accurate feature set.

[0020] Furthermore, in the parallel partial decoder, each feature first passes through a receptive field module RFB to enhance the convolutional neural network feature representation; then each feature is aggregated by element-by-element multiplication; finally, after a convolution and activation operation, a global mapping Sg=pd(f3,f4,f5) is obtained as a global guide for the reverse attention module RA.

[0021] Furthermore, the reverse attention module gradually mines and distinguishes the polyp area by erasing the foreground object, wherein the operation of erasing the foreground is:

[0022] Rw=1-sigmoid(P(S i )) (1)

[0023] Where P(·) represents the upsampling operation; the reverse attention module is expressed as:

[0024] R i =R w *f i (2).

[0025] And, a pixel-level polyp segmentation system based on parallel feature enhancement and attention mechanism, which performs polyp image segmentation through the polyp segmentation model obtained by training with the method described above.

[0026] An electronic device comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the steps of the pixel-level polyp segmentation method based on parallel feature enhancement and attention mechanism as described above are implemented.

[0027] A non-transitory computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps of the pixel-level polyp segmentation method based on parallel feature enhancement and attention mechanism as described above.

[0028] Compared with the prior art, the present invention and its preferred solution have better robustness, better segmentation effect, and can capture tiny and fine boundary details. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] The present invention is further described in detail below with reference to the accompanying drawings and specific embodiments:

[0030] Figure 1 This is a diagram of a polyp segmentation model according to an embodiment of the present invention.

[0031] Figure 2 This is the HarDNet network structure diagram.

[0032] Figure 3 This is a structural diagram of a parallel feature enhancement module according to an embodiment of the present invention.

[0033] Figure 4 This is a structural diagram of the parallel partial decoder module according to an embodiment of the present invention.

[0034] Figure 5 This is a structural diagram of the reverse attention module according to an embodiment of the present invention.

[0035] Figure 6 The following is a visual comparison of the polyp segmentation results of each method. DETAILED DESCRIPTION

[0036] In order to make the features and advantages of this patent more obvious and easy to understand, the following embodiments are specifically described in detail with the accompanying drawings as follows:

[0037] It should be noted that the following detailed descriptions are exemplary and are intended to provide further explanation of the present application. Unless otherwise specified, all technical and scientific terms used herein have the same meanings as those commonly understood by those skilled in the art to which the present application belongs.

[0038] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit the exemplary embodiments according to the present application. As used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. In addition, it should be understood that when the terms "comprise" and / or "include" are used in this specification, it indicates the presence of features, steps, operations, devices, components and / or combinations thereof.

[0039] The pixel-level polyp segmentation network structure based on parallel feature enhancement and attention mechanism proposed in the embodiment of the present invention is as follows: Figure 1 As shown, the model consists of an encoder, a parallel feature enhancement module, a parallel partial decoder, and a reverse attention module.

[0040] Among them, the given polyp image is subjected to multi-scale feature extraction with the help of different convolutional layers of the HarDNet backbone network to obtain four different levels of features f i (i=1, 2, 3, 4). The three higher-level features f i (i=2,3,4) are input into a parallel feature enhancement (PFE) module consisting of channel attention and spatial attention to enhance feature representation. The channel attention module performs channel weighting on the input features to emphasize the features of important channels. The spatial attention module performs spatial weighting on the channel weighted features to highlight the relatively important areas in the image. A parallel partial decoder is used to decode the high-level features f i (i=2,3,4) aggregation, we get a global mapping S g As a global guide for the subsequent decoding part. g And the highest level feature f after construction 4 It is sent to a reverse attention module for processing to obtain the predicted feature R4, and then the predicted feature R4 is added to the global map Sg to obtain the predicted graph result S4. Through similar steps, the predicted feature graph S4 and the constructed high-level feature f3 are input into the reverse attention module to obtain the predicted result S3. Then the predicted result S3 and the constructed feature f2 are subjected to similar operations to obtain the predicted result S2. The loss loss is calculated using the predicted graph result and the true result to guide the network training, and finally a polyp segmentation model with the best performance is obtained for pixel-level polyp segmentation.

[0041] The following is a further introduction to each module of the model:

[0042] A. Coding Backbone

[0043] The present invention uses HarDNet as the backbone network to extract multi-scale feature maps (f1, f2, f3 and f4). HarDNet is a low-memory and low-power network improved based on Densenet. It removes global dense connections and uses MaxPool for downsampling ( Figure 2 shown).

[0044] B. Parallel Feature Enhancement Module

[0045] The parallel feature enhancement module (PFE) module adopts the idea of ​​residual connection and combines multiple dilated convolutional layers with different dilation rates to capture a wider range of contextual information. The output of each dilated convolutional layer contains a channel attention mechanism (CAM) and a spatial attention mechanism (SAM) to integrate image-specific channel and position features. After splicing between the output features of each dilated convolutional layer, the idea of ​​residual connection is introduced to directly add the original input features to the enhanced features to retain the information of the original features and enhance the expression ability of the network. Finally, the spliced ​​feature map is processed using the ReLU activation function and passed to a 1x1 convolutional layer for dimensional transformation to obtain a more detailed and accurate feature set. Incorporating PFE into the jump connection of the network can remove noise and enhance the underlying representation information of the polyp, thereby improving the segmentation performance of the model. The structure diagram of the parallel feature enhancement module is shown in the figure. Figure 3 shown.

[0046] C. Parallel Partial Decoder

[0047] The present invention uses a special decoder (PPD) to aggregate high-level features from high-level Figure 4 ). Each feature first passes through a receptive field block (RFB) to enhance the convolutional neural network feature representation. Then each feature is aggregated by element-by-element multiplication. Finally, after a convolution and activation operation, a global mapping Sg = pd (f3, f4, f5) is obtained as the global guide of the reverse attention module RA. The parallel partial decoder structure is shown in the figure Figure 4 shown.

[0048] D. Reverse Attention Module

[0049] The aggregated high-level features and enhanced features are input into the reverse attention module. The reverse attention module ( Figure 5 ) By erasing the foreground object, the polyp area is gradually mined and distinguished. Adaptively learning reverse attention in three parallel high-level features can effectively improve the visibility of target edges and details. The operation of erasing the foreground is:

[0050] Rw=1-sigmoid(P(S i ) (1)

[0051] Where P(·) represents the upsampling operation. The reverse attention module is expressed as:

[0052] R i =R w *f i (2).

[0053] Test Examples

[0054] 1 Experimental Dataset and Evaluation Indicators

[0055] The test experiment provided in the embodiment of the present invention uses the data sets CVC-612 and Kvasir-SEG, which are divided into training set and test set in a ratio of 9:1.

[0056] Six evaluation metrics are used, including Dice, IoU, mean absolute error (MAE), s-measure (Sα), precision (Precision) and weighted F-measure (F-measure). They are calculated as follows, where tn represents true negatives, tp represents the number of true positives, fp represents the number of false positives, and fn represents false negatives.

[0057] (1) Dice calculates the degree of overlap between the predicted area and the true area, and it focuses on evaluating the spatial overlap of the segmentation results. It can be expressed as:

[0058]

[0059] (2) mIOU is the ratio of the intersection of the predicted segmentation and the true segmentation to their union, which focuses on evaluating the spatial coverage of the segmentation result. It is expressed as:

[0060]

[0061] (3) MAE is the mean absolute difference between the predicted value and the true value, which focuses on evaluating the pixel-level error of the segmentation result. It is expressed as:

[0062]

[0063] Among them, w and h represent the width and height of the image respectively, (x, y) represents the coordinates of each pixel; P(x, y) and G(x, y) represent the predicted value and the true value respectively.

[0064] (4) S-measure combines global similarity and local consistency to measure the structural similarity between the predicted image and the true value mask. It focuses on evaluating the global and local quality of the segmentation results. It is expressed as:

[0065] S=α*S o +(1-α)*S r (6)

[0066] Among them, α∈[0,1], So is the object-aware structural similarity, and Sr represents the region-aware structural similarity.

[0067] (5) Precision is a commonly used indicator in classification and segmentation tasks, which measures the proportion of correct predictions in the prediction results. In image segmentation, precision refers to the proportion of predicted pixels that are correctly predicted as positive categories. Precision focuses on evaluating the accuracy and false alarm rate of the prediction results. It is expressed as:

[0068]

[0069] (6) Weighted F-measure is a comprehensive indicator obtained by calculating the weighted average of precision and recall. The weight can be adjusted according to the specific situation. It focuses on evaluating the comprehensive performance of the prediction results. w Represents weighted accuracy, Recall w Represents the weighted recall rate, and β is a parameter. It is expressed as

[0070]

[0071] 3 Experimental results

[0072] Tables 1 and 2 show the quantitative performance comparison of the present invention and other existing methods on the CVC-612 and Kvasir-SEG datasets, and it can be seen that the model of the present invention has good robustness. In addition, Figure 6 Some qualitative examples on the polyp dataset are shown, and it can be seen that the proposed method outperforms other methods and can capture tiny and fine boundary details.

[0073] Table 1 Evaluation of segmentation effects of various networks on the CVC-612 dataset

[0074]

[0075] Table 2 Evaluation of segmentation effects of each network on the Kvasir-SEG dataset

[0076]

[0077] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0078] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0079] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0080] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 This patent is not limited to the above-mentioned best implementation mode. Anyone can derive other forms of pixel-level polyp segmentation methods based on parallel feature enhancement and attention mechanism under the inspiration of this patent. All equal changes and modifications made according to the scope of the patent application of the present invention shall fall within the scope of this patent.

Claims

1. A pixel-level polyp segmentation method based on parallel feature enhancement and attention mechanism, Features: For a given polyp image, multi-scale feature extraction is performed with the help of different convolutional layers of the HarDNet backbone network to obtain four different levels of features f i ; Then the three higher-level features f i The features are respectively input into a corresponding parallel feature enhancement module consisting of channel attention and spatial attention to enhance feature representation; the channel attention module performs channel weighting processing on the input features; the spatial attention module performs spatial weighting processing on the channel-weighted features; and a parallel partial decoder is used to perform spatial weighting on the high-level features f i Aggregate and get a global mapping S g Serves as a global guide for the subsequent decoding part; The global map S g And the highest level feature f after construction 4 It is sent to a reverse attention module for processing to obtain the predicted feature R4, and then the predicted feature R4 and the global mapping Sg are added to obtain the predicted graph result S4; the predicted feature graph S4 and the constructed high-level feature f3 are input into a reverse attention module, and the predicted result S3 is obtained after processing; the predicted result S3 and the constructed feature f2 are input into a reverse attention module, and the predicted result S2 is obtained after processing; the loss loss is calculated using the predicted graph result and the actual result to guide the network training, and finally a polyp segmentation model with the best performance is obtained for pixel-level polyp segmentation.

2. The pixel-level polyp segmentation method based on parallel feature enhancement and attention mechanism according to claim 1, Features: The parallel feature enhancement module combines multiple dilated convolutional layers with different dilation rates, and the output of each dilated convolutional layer is connected to a channel attention mechanism module and a spatial attention mechanism module; after splicing the output features of each dilated convolutional layer, the original input features are directly added to the enhanced features to retain the information of the original features and enhance the expression ability of the network; finally, the spliced ​​feature map is processed using the ReLU activation function and passed to a 1x1 convolutional layer for dimensional transformation to obtain a more detailed and accurate feature set.

3. The pixel-level polyp segmentation method based on parallel feature enhancement and attention mechanism according to claim 1, Features: In the parallel partial decoder, each feature first passes through a receptive field module RFB to enhance the convolutional neural network feature representation; then each feature is aggregated by element-by-element multiplication; finally, after a convolution and activation operation, a global mapping Sg=pd(f3,f4,f5) is obtained as a global guide for the reverse attention module RA.

4. The pixel-level polyp segmentation method based on parallel feature enhancement and attention mechanism according to claim 1, Features: The reverse attention module gradually mines and distinguishes the polyp area by erasing the foreground object, wherein the operation of erasing the foreground is: Rw=1-sigmoid(P(S i )) (1) Where P(·) represents the upsampling operation; the reverse attention module is expressed as: R i =R w *f i (2)。 5. A pixel-level polyp segmentation system based on parallel feature enhancement and attention mechanism, It is characterized in that Polyp image segmentation is performed using a polyp segmentation model obtained by training according to any one of the methods described in claims 1-4.

6. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, It is characterized in that When the processor executes the program, the steps of the pixel-level polyp segmentation method based on parallel feature enhancement and attention mechanism as described in any one of claims 1 to 4 are implemented.

7. A non-transitory computer-readable storage medium having a computer program stored thereon, It is characterized in that When the computer program is executed by a processor, the steps of the pixel-level polyp segmentation method based on parallel feature enhancement and attention mechanism as described in any one of claims 1 to 4 are implemented.

Images