A polyp image segmentation method based on shallow supervision and feedback attention
Through the combination of edge extraction module and feedback attention module, multi-level supervision is used to perform multi-level supervision, which solves the problems of edge blur and complex morphology in polyp image segmentation, and achieves efficient and accurate automatic segmentation effect.
Patent Information
- Application Number
- CN202210920701.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-08-02
- Publication Date
- 2025-08-08
- Estimated Expiration
- 2042-08-02
AI Technical Summary
The existing polyp image segmentation methods are not effective in complex situations, especially small and complex polyp segmentation with low accuracy, and relying on the experience of doctors, it is difficult to achieve efficient automatic segmentation.
The edge extraction module is used to obtain shallow feature information, and combined with the feedback attention module, the ResNeSt backbone network is used to perform multi-level supervision, and an encoder-decoder architecture is built to realize the full utilization and iterative filtering of shallow and deep features.
It improves the accuracy and generalization ability of polyp image segmentation, can effectively detect complex and different sizes of polyps, simplifies the segmentation process, reduces the dependence on doctor experience, and improves segmentation efficiency.
Smart Images
Figure CN115393283B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of medical image segmentation, and in particular to a polyp image segmentation method based on shallow supervision and feedback attention. Background Art
[0002] In recent years, colorectal cancer has become the third most common type of cancer worldwide, and its mortality rate ranks second among all cancers. Most colorectal cancers begin as polyps on the lining of the colon or rectum, making early detection of polyps using optical colonoscopy crucial for prevention and timely treatment. Detecting polyps requires segmentation technology to identify them. Accurate segmentation results can help doctors quickly locate polyps and provide better diagnoses.
[0003] Currently, the commonly used polyp image segmentation methods include manual segmentation, semi-automatic segmentation, and fully automatic segmentation. The results of manual segmentation are the most accurate, but this method relies heavily on the doctor's experience, and the segmentation process is time-consuming and labor-intensive, with low efficiency and poor repeatability. Semi-automatic segmentation is a method of segmentation through human-computer interaction. The doctor can provide the computer with some effective information to help the computer perform segmentation processing. Semi-automatic segmentation technology also needs to rely on the doctor's experience and level. Compared with manual segmentation and semi-automatic segmentation, fully automatic segmentation can not only improve the efficiency of segmentation, but also achieve a high accuracy rate, and does not require the direct participation of the doctor during the segmentation process. Early fully automatic segmentation methods relied on manually crafted features, such as color, shape, and texture, to process images. However, in some complex situations, there is still a high missed detection rate.
[0004] In recent years, artificial intelligence (AI) methods have become a new trend in polyp segmentation research. Compared to traditional manual feature design methods, convolutional neural networks (CNNs) can automatically learn image features. Some early methods used fully convolutional neural networks combined with conditional random fields for polyp image segmentation, but these methods still have some limitations. Currently, mainstream deep learning-based polyp segmentation methods are essentially end-to-end segmentation networks.
[0005] In their paper "Selective Feature Aggregation Network with Area-Boundary Constraints for Polyp Segmentation," Yuqi Fang et al. proposed a selective feature aggregation network (SFAN) that uses a shared encoder branch and two mutually constrained decoders to predict the area and boundary of a polyp, respectively. This method performs poorly for segmenting smaller polyps and suffers from poor generalization performance. In their paper "PraNet: Parallel Reverse Attention Network for Polyp Segmentation" (Medical Image Computing and Computer Assisted Intervention 2020, pp. 263–273), Deng-Ping Fan et al. proposed a parallel reverse attention network (PRAN). This network utilizes both area and boundary cues through a reverse attention module to improve polyp image segmentation results. However, this method only utilizes deep features and ignores shallow features, resulting in poor segmentation results in some complex situations. Summary of the Invention
[0006] The purpose of the present invention is to propose a polyp image segmentation method based on shallow supervision and feedback attention in response to the shortcomings of the existing technology. The method adopts an edge extraction module to obtain edge information of shallow features to assist model learning, fully utilizes shallow features in convolutional neural networks, and utilizes feedback attention modules to iteratively filter high-level semantic information through cascaded feature aggregation attention gating modules and multi-scale modules, strengthens the important features of complex polyps, and uses multi-scale modules to capture polyps of different sizes to achieve automatic segmentation of colonoscopic polyp images. The multi-level deep supervision used in the training process can help the model to segment polyp targets of different sizes, and can effectively detect complex polyps and bizarre polyps, greatly improving the efficiency of segmentation. The method is simple and has good segmentation effect. It better solves the two major problems of polyp segmentation tasks, which are often blurred edges and complex morphology. Especially in some complex cases, it can achieve good segmentation effect and generalization performance, and has broad and good application prospects.
[0007] The specific technical solution for achieving the purpose of the present invention is: a polyp image segmentation method based on shallow supervision and feedback attention, which is characterized by using an edge extraction module to effectively detect complex polyps and bizarre polyps by fully utilizing shallow features in a convolutional neural network. The multi-level deep supervision used during training can help the model segment polyp targets of different sizes. The polyp image segmentation specifically includes the following steps:
[0008] Step S1: Preprocessing of the dataset
[0009] We selected an existing colonoscopy polyp segmentation dataset and performed image augmentation on the training data from the dataset. The augmentation included random cropping, random horizontal flipping, random vertical flipping, and random scaling. Furthermore, we used the label segmentation map to obtain the edge map of the polyps, which served as the edge label map.
[0010] Step S2: Construction of the overall network framework
[0011] The overall network design is an encoder-decoder architecture. The encoder selects ResNeSt as the backbone feature extraction network to extract features at five levels. ; The decoder will use bilinear interpolation to restore the feature map to an output map with the same resolution as the input map. and Will be used as shallow feature map, is a deep feature map.
[0012] Step S3: Edge feature extraction and supervision
[0013] For the shallow features extracted in step S2, an edge extraction module is used in the jump connection part of the encoder-decoder architecture to obtain its edge information, and supervision is constructed with the edge label information obtained in step S1. The loss function used in the supervision is binary cross entropy loss.
[0014] Step S4: Screening and filtering of high-level semantic features
[0015] The deep features extracted in step S2 are filtered using a feedback attention module in the skip connection part of the encoder-decoder architecture to enhance the part containing polyp information. This module is implemented by cascading feature aggregation attention gating modules and multi-scale modules.
[0016] Step S5: Training and use of network model
[0017] Use the training data set of step S1 to step S2 The network structure constructed in S4 is trained, and deep multi-level supervision is used to assist in the learning of early features. During the training process, the model with the best effect on the verification data set is saved, and the saved model is used to segment the polyps. The use of the saved model for polyp segmentation is to send the colonoscopy polyp image to be segmented into the saved model. Through the processing of the neural network, a polyp segmentation result map can be obtained, which effectively detects complex polyps and bizarre polyps.
[0018] Compared with the prior art, the present invention has the following beneficial technical effects and significant technical progress:
[0019] 1) The prior information of polyp images is fully utilized, and the shallow information extracted by the convolutional neural network is further utilized. The output of the supervised edge extraction module is used to assist in the early learning of polyp images, which to some extent alleviates the difficulty of segmenting polyp images with blurred edges.
[0020] 2) ResNeSt is used as the backbone feature extraction network, combining multi-branch design and feature attention. This structure can effectively extract powerful and robust feature representations from de-polyp images, further improving the performance and generalization ability of the model.
[0021] 3) The deep semantic information of polyps extracted by the convolutional neural network is fully filtered and integrated. The feedback attention module is used to further refine the recognition of polyps with complex morphologies and to help the model recognize polyps of different sizes through a multi-scale module. BRIEF DESCRIPTION OF THE DRAWINGS
[0022] Figure 1 A schematic diagram of the network model constructed for the present invention;
[0023] Figure 2 Schematic diagram of edge extraction module;
[0024] Figure 3 Schematic diagram of the feedback attention module;
[0025] Figure 4 A schematic diagram showing the comparison of segmentation effects. DETAILED DESCRIPTION
[0026] See Figure 1 The present invention performs colonoscopic polyp image segmentation according to the following steps:
[0027] Step S1: Preprocessing of the dataset
[0028] We selected an existing colonoscopy polyp segmentation dataset and performed image augmentation on the training data from the dataset. The augmentation included random cropping, random horizontal flipping, random vertical flipping, and random scaling. Furthermore, we used the label segmentation map to obtain the edge map of the polyps, which served as the edge label map.
[0029] Step S2: Construction of the overall network framework
[0030] The overall network design is an encoder-decoder architecture. The encoder selects ResNeSt as the backbone feature extraction network to extract features at five levels. .in and Will be used as shallow feature map, The decoder will use bilinear interpolation to restore the feature map to an output map with the same resolution as the input map.
[0031] Step S3: Edge feature extraction and supervision
[0032] For the shallow feature map extracted in step S2, the edge extraction module is used in the jump connection part of the encoder-decoder architecture to obtain its edge information, and supervision is constructed with the edge label information obtained in step S1. The loss function used for supervision is binary cross entropy loss.
[0033] Step S4: Screening and filtering of high-level semantic features
[0034] For the deep feature map extracted in step S2, the feedback attention module is used to filter it in the skip connection part of the encoder-decoder architecture to enhance the part containing polyp information. This module is implemented by cascading feature aggregation attention gating modules and multi-scale modules.
[0035] Step S5: Training and use of network model
[0036] Use the training data set from step S1 to perform step S2 The network structure constructed in S4 is trained, and deep multi-level supervision is used to assist in the learning of early features. During the training process, the model with the best effect on the validation dataset is saved, and the saved model is used to segment polyps.
[0037] The edge extraction module will fuse two shallow feature maps by splicing and fusion of shallow features. The two feature maps are first adjusted by a 3x3 convolution layer to adjust the number of channels. Then, the bilinear interpolation upsampling method is used to increase the resolution of the two feature maps to the same size as the input image. Finally, the number of channels of the fused feature map is reduced to 1 through a 1x1 convolution layer to obtain the final edge prediction map.
[0038] The feedback attention module is composed of a cascaded feature aggregation attention gating module and a multi-scale module. The input of the feedback attention module includes the feature map of the current level and the feature map of the previous level. Its output will be re-injected into the module body as input again, and finally propagated to the decoder of the network through a specified number of feedback cycles.
[0039] The cascaded feature aggregation attention gating module obtains two input feature maps and obtains two single-channel feature representation maps through 1x1 convolution. The two single-channel maps are then spliced and enter the channel-level full connection operation and Sigmoid activation function to obtain an attention feature map with a value range between 0 and 1. The attention map filters the input feature map to further highlight the polyp area.
[0040] The present invention will be further described below with reference to specific examples:
[0041] Example 1
[0042] Step S1: Preprocessing of the dataset
[0043] This example uses the Kvasir-SEG dataset for gastrointestinal polyp image segmentation, released in 2017, to construct training, validation, and test sample sets. The Kvasir-SEG dataset contains 1,000 colonoscopic polyp images with resolutions ranging from 332x487 to 1920x1072 pixels. These images are color images with three channels: red (R), green (G), and blue (B). The dataset also includes 1,000 annotated images, which are single-channel binary images with a one-to-one correspondence between the annotated images and the polyp images. 800 images were randomly selected as the training dataset, 100 images as the validation dataset, and the remaining 100 images as the test dataset.
[0044] Data preprocessing was performed on the original and annotated images in the training set. The corresponding two images were resized, randomly cropped, randomly flipped horizontally, randomly flipped vertically, and randomly scaled to enhance the diversity of the training sample dataset. For the enhanced annotated images, the Canny operator edge detection method was used to obtain edge label images.
[0045] Step S2: Construction of the overall network framework
[0046] See attached Figure 1 The overall network structure design is an encoder-decoder architecture. ResNeSt pre-trained on the ImageNet dataset is used as the encoder. The last fully connected layer and pooling layer in ResNeSt-50 are removed. The rest of the network is the same as the original network. The encoder extracts five levels of features. .in, and Will be used as shallow feature map, In the decoder, convolution operations are combined with bilinear interpolation to gradually restore the feature map to the same resolution as the input map.
[0047] Step S3: Edge feature extraction and supervision.
[0048] See Figure 1 Figure 2 The edge extraction module in the present invention acts on the shallow features extracted in step S1 and .
[0049] Shallow features , you need to first perform an upsampling operation to obtain the shallow feature map The two feature maps are resized using 3x3 convolutions and then fused. Upsampling is then performed to obtain a feature map with the same resolution as the original input image. Finally, a 1x1 convolution is performed to reduce the number of channels to 1, resulting in the final edge prediction map. This prediction map is used in conjunction with the edge label map to construct supervision. The loss function used here is binary cross entropy loss.
[0050] Step S4: Screening and filtering of high-level semantic features
[0051] See Figure 1 and Figure 3 The feedback attention module in the present invention acts on the deep features extracted in step S1 .
[0052] The network in this embodiment contains two feedback attention modules, one of which has an input from and , and the other input comes from and The feedback attention module is composed of a cascaded feature aggregation attention gating module and a multi-scale module.
[0053] The input of the feedback attention module includes the feature map of the current layer and the feature map of the previous layer. Its output will be injected into the module body again as input. After a specified number of feedback loops (the number of loops is set to 3 in this embodiment), the final output result after the loop will be propagated to the decoder of the network.
[0054] Specifically, the two inputs to the feature aggregation attention gating module first undergo a 1x1 convolution to obtain two single-channel feature representations, which serve as the feature representations for both. These two single-channel maps are then fused and expanded and aggregated using a channel-level fully connected operation to obtain a single-channel feature representation. Next, a sigmoid activation function is used to perform attention activation on this single-channel feature map, obtaining an attention feature map with values ranging from 0 to 1. Finally, the attention map is used to filter the input feature map, further highlighting the polyp area.
[0055] Step S5: Training and use of network model
[0056] The Adam optimizer is used when training the network, and the optimizer parameters are set to: , , the initial learning rate The learning rate decay uses an exponential decay strategy, with the decay base gamma set to 0.99. The batch size is set to 16 during training. The encoder's initial parameters are those pre-trained on ImageNet, and the initial parameters of the remaining components are initialized using Kaiming. This example uses an NVIDIA GeForce RTX 3090 graphics card for training, with 100 epochs and an early stopping strategy.
[0057] The loss function included in this embodiment includes two parts, namely edge supervision loss and multi-level segmentation loss. The edge supervision loss function The binary cross entropy loss function is expressed as follows (a):
[0058] (a).
[0059] in, is the tag value; To predict the results;
[0060] The multi-level segmentation loss function By binary cross entropy loss Sum intersection loss It is composed of and is expressed by the following formula (b):
[0061] (b).
[0062] in, is the weight of the binary cross entropy loss. In this embodiment The multi-level loss function is composed of three segmentation losses in the decoder process, so the final overall loss function is It can be expressed by the following formula (c):
[0063] (c).
[0064] in, is the segmentation label map; For the The segmentation result map of the layer.
[0065] This example uses four commonly used evaluation metrics, namely Dice coefficient, intersection over union (IoU), precision, and recall, to quantitatively analyze the performance of the polyp segmentation model.
[0066] The Dice coefficient (Dice) is calculated by the following formula (d):
[0067] (d);
[0068] The intersection over union (IoU) is calculated by the following formula (e):
[0069] (e);
[0070] The precision is calculated by the following formula (f):
[0071] (f);
[0072] The recall rate (Recall) is calculated by the following formula (g):
[0073] (g).
[0074] Among them, TP is the positive sample predicted by the model as the positive class; FP is the negative sample predicted by the model as the positive class; FN is the positive sample predicted by the model as the negative class.
[0075] The training is performed according to the above process, and the model with the best performance in the validation set is saved as the optimal model. The colonoscopy polyp image to be segmented is sent to the optimal network model obtained by the above training. After processing by the neural network, the polyp segmentation result image can be obtained.
[0076] See Figure 4 The neural network constructed by the present invention is qualitatively and quantitatively compared with other polyp segmentation networks UNet, ResUNet++, and HarDNet-MSEG. It can be found that the present invention has the best segmentation effect on small polyps and complex polyps. The quantitative polyp segmentation comparison is shown in Table 1 below:
[0077] Table 1 Comparison of polyp segmentation
[0078]
[0079] As can be seen from the above table, the network constructed by the present invention outperforms other models in four commonly used evaluation indicators: Dice coefficient (Dice), intersection over union (IoU), precision (Precision), and recall (Recall).
[0080] The above examples provide a detailed introduction to the polyp image segmentation method based on shallow supervision and feedback attention proposed in the present invention. Based on the main architecture of the present invention, researchers in this field may make changes in implementation strategies and application data, such as replacing different encoders for feature extraction. In summary, this specification is not intended to limit the present invention. Any modifications and improvements made within the spirit and principles of the present invention shall be included in the scope of protection of the present invention.
Claims
1. A colonoscopy polyp image segmentation method based on shallow supervision and feedback attention, characterized by The segmentation of colonoscopic polyp images specifically includes the following steps: Step S1: Preprocessing of the dataset Select an existing colonoscopy polyp segmentation dataset, perform image enhancement on the training data in the dataset, and obtain an edge map of the polyp through the label segmentation map as the edge label map. The image enhancement specifically includes: random cropping, random horizontal flipping, random vertical flipping, and random scaling; Step S2: Construction of the overall network framework The network adopts an encoder-decoder architecture, and the encoder selects ResNeSt as the backbone feature extraction network for extracting , the decoder uses a bilinear interpolation method to restore the feature map extracted by the encoder to an output map with the same resolution as the input map; ,in and Shallow features ; Deep features ; Step S3: Edge feature extraction and supervision In the jump connection part of the encoder-decoder architecture, an edge extraction module is used to obtain shallow feature maps. and The polyp edge information is obtained and combined with the edge label map to construct supervision using binary cross entropy loss; Step S4: Screening and filtering of high-level semantic features In the skip connection part of the encoder-decoder architecture, a feedback attention module is used to Filtering is performed to enhance areas containing polyps; Step S5: Training and use of network model The constructed network structure is trained using the training data in step S1, and deep multi-level supervision is used to assist in early feature learning. During the training process, the model with the best effect on the validation dataset is saved, and the saved model is used to segment polyps; The feedback attention module in step S4 is composed of a cascaded feature aggregation attention gating module and a multi-scale module. The input of the feedback attention module includes the feature map of the current level and the feature map of the previous level. Its output will be re-injected into the feedback attention module as input, and finally propagated to the decoder of the network through a specified number of feedback cycles; the cascaded feature aggregation attention gating module obtains the two input feature maps of the current level and the previous level, obtains two single-channel feature representation maps through 1x1 convolution, and splices them into the full connection operation and Sigmoid activation function at the channel level to obtain an attention feature map with a value range between 0 and 1. The attention feature map is used to filter the input feature map to further highlight the area of the polyp.
2. The colonoscopy polyp image segmentation method based on shallow supervision and feedback attention according to claim 1 is characterized in that The edge extraction module in step S3 extracts shallow features and Follow the steps below to perform splicing and fusion: 1) Pass the two feature maps through a 3x3 convolutional layer to adjust the number of channels; 2) Use bilinear interpolation upsampling to increase the resolution of the two feature maps to the same size as the input image; 3) The number of channels of the fused feature map is reduced to 1 through a 1x1 convolution layer, and the Sigmoid function is used to obtain the edge prediction map, which is then combined with the edge label map to construct supervision using binary cross entropy loss.
3. The colonoscopy polyp image segmentation method based on shallow supervision and feedback attention according to claim 1 is characterized in that The method of using the stored model to segment the polyps is to send the colonoscopy polyp image to be segmented into the stored model, and obtain a polyp segmentation result image through processing by the neural network.