A two-stage iterative context boundary feedback network semantic segmentation method

Abstract

The present invention belongs to the field of semantic segmentation in computer vision, and specifically relates to a semantic segmentation method of a two-stage iterative context boundary feedback network. By imitating the "primary to fine" working paradigm of doctors, ICBNet adopts an iterative feedback learning strategy, which refines encoder features through context and boundary perception details in preliminary segmentation and boundary prediction, and executes this strategy in an iterative manner to achieve gradual improvement. Features are enhanced under the guidance of segmentation and boundary prediction to achieve iterative learning. First, in the first stage, the initial polyp area segmentation result and the lesion boundary segmentation result are generated, and then the result is input into the IFU module for iterative optimization. After two-stage iterative optimization, the final segmentation result is obtained. The present invention uses two-stage iterative optimization to effectively solve the problems of appearance changes and fuzzy boundaries.

Description

Technical Field

[0001] The present invention belongs to the field of semantic segmentation of computer vision, and in particular relates to a semantic segmentation method of a two-stage iterative context boundary feedback network. Background Art

[0002] Semantic segmentation algorithms classify every pixel in an image, a pixel-level problem. Currently, widely used semantic segmentation networks are all improved upon FCNs. FCNs leverage the powerful learning capabilities of CNNs on images through a fully convolutional concept. These networks replace the fully connected layers of some commonly used deep classification network models with convolutional layers. This results in an image output rather than a one-dimensional vector, achieving end-to-end semantic segmentation. Furthermore, by removing the fully connected layers, the network can accept images of any size, ensuring that the input and output images are of equal size. Because convolutional layers are followed by pooling layers, also known as downsampling layers, they affect image resolution. To ensure that the input and output images are of equal size, FCNs use deconvolution to upsample the image to maintain resolution. A challenge with polyp segmentation is that their boundaries are unclear and easily confused with surrounding tissue. Early CNN-based methods only considered segmentation of the entire polyp region, ignoring boundary constraints.

[0003] To address this issue, PraNet aggregates encoder features at multiple levels and uses reverse boundary attention to mine boundary clues. MSNet explores different methods to obtain complementary information between different levels. Polyp-PVT, developed on the basis of Transformer, uses a self-attention mechanism to explore the similarity between pixels to improve segmentation accuracy. Some works, including HRENet and LOD-Net, extract boundary regions or candidate boundaries from segmentation predictions as explicit supervision; SANet, which focuses more on compensating for color differences between colonoscopy images, adopts a single-stage processing pipeline. Although feature representation can be improved, the details of refined features are still insufficient. Summary of the Invention

[0004] The purpose of the present invention is to provide a semantic segmentation method of a two-stage iterative context boundary feedback network to address the shortcomings of the existing technology, which can solve the problem of insufficient accuracy of the single-stage network model in polyp semantic segmentation technology.

[0005] In order to achieve the above object, the present invention adopts the following technical solutions:

[0006] A semantic segmentation method based on a two-stage iterative context boundary feedback network, comprising:

[0007] Step 1: Input the original image into the ICBNet encoder and pass it through each layer of the encoder to obtain the semantic features of each layer;

[0008] Step 2: After downsampling the boundary area features, they are input into the IFU module together with the semantic features to obtain enhanced semantic features, which are then input into the decoder to obtain the initial segmentation result.

[0009] Step 3: After downsampling the initial segmentation result, it is input into the IFU module together with the semantic features and the sampled boundary features to obtain the enhanced semantic features of the second iteration, and the optimized segmentation result is generated again through the decoder.

[0010] Preferably, the step 1 further includes:

[0011] The highest-level features and the lowest-level features of the encoder part of ICBNet are resampled to the same size and input into the convolutional layer encoder to generate boundary features. The boundary features are further reduced in dimension to obtain boundary segmentation predictions.

[0012] Preferably, in step 2, obtaining enhanced semantic features includes:

[0013] In the first iteration, the IFU module fuses boundary features. In the second iteration, the IFU module fuses boundary features and also enhances the contextual semantic information of the foreground and background regions to obtain new enhanced features.

[0014] Preferably, it also includes:

[0015] 1) Input the initial segmentation prediction results, boundary features and semantic features of the first iteration into the IFU module;

[0016] 2) The segmentation result of the first iteration is combined with the semantic feature mask of the layer to obtain the features of the foreground information, and the preliminary foreground enhancement features are obtained through the foreground enhancement module;

[0017] 3) After inverting the segmentation result of the first iteration, it is masked with the semantic features of the layer to obtain the background information features, and then the background context attention is obtained through the text attention module;

[0018] 4) Applying background context attention to the initially obtained foreground enhancement features and background features respectively, by taking the inverted mask / direct mask method, paying attention to the initially obtained foreground enhancement features and background features respectively, to obtain the final foreground enhancement features and background enhancement features;

[0019] 5) Add the foreground enhancement features, background enhancement features and boundary features to obtain the final enhancement features.

[0020] The beneficial effect of the present invention is that it can better segment the lesion area through an iterative context boundary feedback method. ICBNet has a two-stage iterative structure. The first stage performs coarse segmentation and the second stage completes refinement. When both stages are completed, the gradient is updated after a back propagation. The present invention implements the iterative optimization process through the IFU module. Compared with the existing technology, ICBNet uses an iterative method to feedback context and boundary-aware details from preliminary segmentation and boundary prediction, transforming from a single stage to a multi-stage training method, and applies iterative feedback learning to polyp segmentation. By designing a dual-branch iterative feedback unit, it can better enhance important features in the foreground / background and mix boundary features. Among them, ICBNet helps to improve the accuracy and robustness of the results on five widely used public benchmarks. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Features, advantages, and technical effects of exemplary embodiments of the present invention will be described below with reference to the accompanying drawings.

[0022] Figure 1 Flowchart of the iterative context boundary feedback network semantic segmentation method of the present invention.

[0023] Figure 2 This is the network structure diagram of the present invention.

[0024] Figure 3 This is a structural diagram of the iterative feedback unit of the present invention.

[0025] Figure 4 These are the test results of the present invention on the Kvasir-seg dataset and the CVC-ClinicDB dataset.

[0026] Figure 5 These are the test results of the present invention on the ETIS dataset, CVC-300 dataset, and CVC-CloneDB dataset.

[0027] Figure 6 This is the visual comparison result of the present invention and other advanced performance methods. DETAILED DESCRIPTION

[0028] For example, certain words are used in the specification and claims to refer to specific components. Those skilled in the art should understand that hardware manufacturers may use different terms to refer to the same component. This specification and claims do not use differences in names as a way to distinguish components, but use differences in the functions of the components as the criteria for distinction. For example, "including" mentioned throughout the specification and claims is an open term, so it should be interpreted as "including but not limited to". "Approximately" means that within an acceptable error range, those skilled in the art can solve technical problems within a certain error range and basically achieve technical effects.

[0029] Furthermore, the terms “first,” “second,” etc. are used for descriptive purposes only and are not to be understood as indicating or implying relative importance.

[0030] In the present invention, unless otherwise expressly specified or limited, terms such as "mounted," "connected," "connect," and "fixed" should be interpreted broadly. For example, they may refer to fixed, detachable, or integral connections; mechanical or electrical connections; direct or indirect connections through an intermediary; and internal communication between two components. Those skilled in the art will understand the specific meanings of the above terms in the present invention based on specific circumstances.

[0031] The following is combined with Figures 1 to 6 The present invention is described in further detail, but is not intended to limit the present invention.

[0032] A two-stage iterative context boundary feedback network semantic segmentation method, including:

[0033] Step 1: Input the original image into the ICBNet encoder and pass it through each layer of the encoder to obtain the semantic features of each layer;

[0034] Step 2: After downsampling the boundary area features, they are input into the IFU module together with the semantic features to obtain enhanced semantic features, which are then input into the decoder to obtain the initial segmentation result.

[0035] Step 3: After downsampling the initial segmentation result, it is input into the IFU module together with the semantic features and the sampled boundary features to obtain the enhanced semantic features of the second iteration, and the optimized segmentation result is generated again through the decoder.

[0036] In the semantic segmentation method of the two-stage iterative context boundary feedback network according to the present invention, step 1 further includes:

[0037] The highest-level features and the lowest-level features of the encoder part of ICBNet are resampled to the same size and input into the convolutional layer encoder to generate boundary features. The boundary features are further reduced in dimension to obtain boundary segmentation predictions.

[0038] In the semantic segmentation method of the two-stage iterative context boundary feedback network according to the present invention, in step 2, obtaining enhanced semantic features includes:

[0039] In the first iteration, the IFU module fuses boundary features. In the second iteration, the IFU module fuses boundary features and also enhances the contextual semantic information of the foreground and background regions to obtain new enhanced features.

[0040] In the semantic segmentation method according to the dual-stage iterative context boundary feedback network of the present invention, the method further includes:

[0041] 1) Input the initial segmentation prediction results, boundary features and semantic features of the first iteration into the IFU module;

[0042] 2) The segmentation result of the first iteration is combined with the semantic feature mask of the layer to obtain the features of the foreground information, and the preliminary foreground enhancement features are obtained through the foreground enhancement module;

[0043] 3) After inverting the segmentation result of the first iteration, it is masked with the semantic features of the layer to obtain the background information features, and then the background context attention is obtained through the text attention module;

[0044] 4) Applying background context attention to the initially obtained foreground enhancement features and background features respectively, by taking the inverted mask / direct mask method, paying attention to the initially obtained foreground enhancement features and background features respectively, to obtain the final foreground enhancement features and background enhancement features;

[0045] 5) Add the foreground enhancement features, background enhancement features and boundary features to obtain the final enhancement features.

[0046] The semantic segmentation method ICBNet of the iterative context boundary feedback network of the present invention adopts an iterative feedback training process, including an iterative feedback unit IFU, a decoder and a loss function. Specifically:

[0047] The flowchart of iterative feedback training is as follows Figure 1 As shown, the following steps are included:

[0048] First, input the polyp image into ICBNet to obtain the initial segmentation prediction result as the segmentation prediction result of the first iteration. The steps are as follows:

[0049] 1) The image is sampled to 352*352 and passed through the deep encoder PVT to obtain the semantic features of each layer.

[0050] 2) The highest-level semantic features are sampled to the size of the lowest-level semantic features and input into another encoder to obtain a feature map, and boundary segmentation prediction is obtained through dimensionality reduction. The feature map is the boundary feature.

[0051] 3) Each encoder layer is connected to an iterative feedback unit (IFU). The semantic features and an all-one matrix of the same size are input into the IFU. The IFU is then fused with the boundary features using matrix addition to obtain enhanced semantic features.

[0052] 4) The enhanced semantic features are passed through the decoder to obtain the initial segmentation result prediction of the first iteration.

[0053] Second, the polyp prediction segmentation results and boundary prediction segmentation results obtained in the first stage are compared with the actual segmentation results to calculate the loss function;

[0054] Third, the polyp segmentation prediction results and boundary features obtained in the first stage are downsampled and input into the iterative feedback unit (IFU) again to obtain enhanced features.

[0055] (1) The segmentation result of the first iteration and the semantic feature mask of this layer are used to obtain the features of the foreground information, and the foreground enhancement features are initially obtained through the foreground enhancement module;

[0056] (2) After the first iteration of the segmentation result is reversed, the background region segmentation result is obtained, and the result and the semantic features of the layer are masked to obtain the features of the background information. After the effective text attention module is used to obtain the attention map, it is applied to the initially obtained foreground enhancement features and background features respectively. The method is to apply the inverse mask / direct mask method to the initially obtained foreground enhancement features and background features respectively to obtain the final foreground enhancement features and background enhancement features;

[0057] (3) The foreground enhancement features, background enhancement features and boundary features are added together to obtain the final enhancement features, and the final segmentation prediction results are obtained through the decoder.

[0058] Fourth, the polyp prediction segmentation results and boundary prediction segmentation results obtained in the second stage are compared with the actual segmentation results, and the loss function loss2 is calculated;

[0059] Iterative Feedback Unit IFU

[0060] (1) Foreground Feature Enhancement (FFE). The extracted foreground features reflect the appearance distribution of the initially estimated polyp region and are used to enhance more global semantic information while maintaining the fit of local details. To this end, two 3×3 convolutional layers and a poolformer layer are applied in FFE. The poolformer layer is an effective MLP-based layer that replaces the complex and computationally intensive self-attention with a simple pooling layer in the Transformer structure without compromising performance.

[0061] (2) Effective Context Attention (ECA). The ECA block attempts to refine the segmentation result by enhancing background features. In the ECA block, the present invention applies three convolutions with different kernel sizes in parallel, and then sums the inputs into a 3*3 convolution, which captures context information from the background at different scales. The obtained features are subjected to a 1*1 convolution to produce a refined attention mask M i , i indicates that the IFU is connected to the i-th layer of the encoder.

[0062] (3) Feature Mixing. i Take the opposite, that is, 1-M i , including enhanced foreground regions and uncertain regions, can provide additional complementary information for the encoder features. Therefore, the improved features can be calculated as the following combination:

[0063]

[0064] Finally, the improved features Further fusion with the boundary feature Q to obtain the final enhanced feature

[0065]

[0066] t represents the number of iterations.

[0067] Decoder

[0068] The enhanced features output by each layer of IFU are reduced in dimension by a 1*1 convolution kernel, sampled to the original image size, and added layer by layer. The results of each layer are supervised by the true segmentation value.

[0069] Loss Function

[0070] The loss function uses the IoU_BCE loss function. The loss of each iteration is equal to the loss of the decoder generating prediction results of different scales and the sum of the boundary prediction losses. The total loss is composed of the sum of the first iteration loss and the second iteration loss:

[0071]

[0072] In the above formula, t is the number of iterations, i is the i-th layer of the encoder, and T is the total number of iterations. In the two-stage network ICBNet, T is equal to 2.

[0073] Based on the disclosure and teachings of the above description, those skilled in the art will be able to make changes and modifications to the above embodiments. Therefore, the present invention is not limited to the above specific embodiments. Any obvious improvements, substitutions, or modifications made by those skilled in the art based on the present invention fall within the scope of protection of the present invention. In addition, although certain specific terms are used in this description, these terms are only for convenience of description and do not constitute any limitation to the present invention.

Claims

1. A semantic segmentation method based on a two-stage iterative context boundary feedback network, characterized in that: include: Step 1: Input the original image into the ICBNet encoder and pass it through each layer of the encoder to obtain the semantic features of each layer; Step 2: After downsampling the boundary area features, they are input into the IFU module together with the semantic features to obtain enhanced semantic features, which are then input into the decoder to obtain the initial segmentation result; Step 3: After downsampling the initial segmentation result, it is input into the IFU module together with the semantic features and the sampled boundary features to obtain the enhanced semantic features of the second iteration, and the decoder is used to generate the optimized segmentation result again; The IFU module includes: (1) foreground feature enhancement, which is used to enhance more global semantic information while keeping the fit of local details; (2) effective context attention, which is used to apply three convolutions with different kernel sizes in parallel, and then sum them up and input them into a 3*3 convolution to capture contextual information from the background at different scales. The obtained features are then passed through a 1*1 convolution to produce a fine attention mask M. i , i indicates that the IFU is connected to the i-th layer of the encoder; (3) Feature mixing, including enhanced foreground regions and uncertain regions, is used to provide additional supplementary information for the encoder features.

2. The semantic segmentation method of the two-stage iterative context boundary feedback network according to claim 1, characterized in that: The step 1 further includes: The highest-level features and the lowest-level features of the encoder part of ICBNet are resampled to the same size and input into the convolutional layer encoder together to generate boundary features, which are further reduced in dimension to obtain boundary segmentation predictions.

3. The semantic segmentation method of the two-stage iterative context boundary feedback network according to claim 1, characterized in that: In the second step, obtaining enhanced semantic features includes: In the first iteration, the IFU module fuses boundary features. In the second iteration, the IFU module fuses boundary features and also enhances the contextual semantic information of the foreground and background regions to obtain new enhanced features.

4. The semantic segmentation method of the two-stage iterative context boundary feedback network according to claim 1, characterized in that: Also includes: 1) Input the initial segmentation prediction results, boundary features, and semantic features of the first iteration into the IFU module; 2) The segmentation result of the first iteration is masked with the semantic feature to obtain the features of the foreground information, and the preliminary foreground enhancement features are obtained through the foreground enhancement module; 3) The segmentation result of the first iteration is inverted and masked with the semantic features to obtain background information features, which are then passed through the text attention module to obtain background context attention; 4) Applying background context attention to the initially obtained foreground enhancement features and background features respectively, by taking the inverted mask / direct mask method, paying attention to the initially obtained foreground enhancement features and background features respectively, to obtain the final foreground enhancement features and background enhancement features; 5) Add the foreground enhancement features, background enhancement features and boundary features to obtain the final enhancement features.

Images