A polyp image segmentation method based on separation-squeeze-bottleneck-joint codec

Abstract

The present invention provides a polyp image segmentation technology based on separation-squeezing-bottleneck-joint codec, the following steps: step 1: for a given polyp image, extract five levels of features from a res2net()-based backbone network; divide the features into low-level features and high-level features; step 2: use a separation-squeezing-bottleneck-joint encoder to encode different levels; step 3: in the encoding stage, use the encoder to fuse features of different levels; step 4: in the refinement stage, use the information extracted by the separation-squeezing-bottleneck-joint block encoder to refine the features of the initial prediction map step by step; step 5: in the training of the entire network, use the cross entropy loss function and the intersection-over-union loss function to guide the learning of the network; the application of this technical solution can realize the mining of geometric context information from different angles, and at the same time calculate the channel correlation, enhance the representation ability of the features, so as to solve the problem of polyp segmentation.

Description

Technical Field

[0001] The invention relates to the technical field of computer vision, in particular to a polyp image segmentation method based on separation-squeezing-bottleneck-joint codec. Background Art

[0002] Image segmentation is an important research field in computer vision. Polyp segmentation is one of the application research of image segmentation methods in specific fields. Polyp segmentation is a pixel-level task that classifies each pixel according to the context in the image. The polyp segmentation task is applied to medical image analysis and provides valuable information for clinical diagnosis and pathological research.

[0003] Polyp segmentation research has certain value in medicine, but accurate segmentation of polyps is difficult. First, polyps are similar to surrounding tissues, and it is not easy for algorithms to identify polyps; second, polyps of the same type have different sizes, colors, and textures, and it is not easy for algorithms to learn the characteristics of polyps. Therefore, polyp segmentation methods usually focus on extracting features.

[0004] Early traditional methods mainly used artificially designed features to extract color, shape, texture and appearance features, such as the method of Mamonov (Alexander V. Mamonov, Isabel N. Figueiredo, Pedro N. Figueiredo and Yen-Hsi Richard Tsai. Automated polyp detection in colon capsule endoscopy. IEEE Transactions on Medical Imaging, vol. 33, no. 7, pp. 1488–1502, 2014) and the method of Tajbakhsh (Nima Tajbakhsh, Suryakanth R Gurudu, and Jianming Liang. Automated polyp detection in colonoscopy videos using shape and context information. IEEE Transactions on Medical Imaging, vol. 35, no. 2, pp. 630–644, 2015.). However, due to the limited representation ability of artificially designed features, these methods often have a high rate of missed detection. Therefore, polyp segmentation has gradually evolved from traditional methods to deep learning methods.

[0005] In recent years, deep neural network methods based on encoder-decoder structures, such as U-Net (Ronneberger, Olaf, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. International Conference on Medical image Computing and Computer Assisted Intervention. 2015) and its variants, have been widely used because of their ability to fuse semantic information and spatial details of features at different levels. For the U-Net structure, the encoder block extracts feature maps from the input image, and the decoder block optimizes the features of the encoder and designs the polyp segmentation task. U-Net directly uses simple skip connections to fuse feature maps. Simple skip connections in U-Net may reduce segmentation performance. U-Net and its variants generally use simple addition or concatenation operations in the decoder to gradually fuse features at different levels. In the process of feature fusion, these two operations often generate a large amount of redundant information, which will weaken the truly useful features and suppress the specific expression of features at different levels, resulting in inaccurate details of the feature map and blurred boundaries. Therefore, how to effectively extract and fuse features is very challenging. Summary of the invention

[0006] In view of this, the purpose of the present invention is to provide a polyp image segmentation method based on separation-squeezing-bottleneck-joint codec, which can mine geometric context information from different angles, calculate channel correlation at the same time, enhance the feature representation ability, and solve the polyp segmentation problem.

[0007] To achieve the above object, the present invention adopts the following technical solution: a polyp image segmentation method based on separation-squeezing-bottleneck-joint codec, comprising the following steps:

[0008] Step 1: For a given polyp image I, extract five levels of features {f i ,i=1,2,3,4,5}; the feature f i Divided into low-level features {f i ,i=1,2} and high-level features {f i ,i=3,4,5};

[0009] Step 2: Encode different levels using an encoder with a split-squeeze-bottleneck-join block; the split-squeeze-bottleneck-join block obtains geometric context information of features at each level in different directions; the split-squeeze-bottleneck-join block is specifically divided into four steps: split operation, squeeze operation, bottleneck operation and join operation;

[0010] Separation operation: given input feature F in ∈R H×W×C , divide the input features into four groups of features FG∈R H×W×C / 4 , where H and W represent the size of the input feature, C is the number of channels of the input feature, and FG S Indicates one of four groups of characteristics;

[0011] Squeeze operation: Using global average pooling to capture features {FG s ,s=1,2,3,4}, the definition of GC is as follows:

[0012] Bottleneck operation: We input the channel information GC into the bottleneck attention module BAM to obtain the bottleneck attention BA. The bottleneck attention BA effectively distinguishes the target from the background. The definition of BA is as follows: BA = BAM (GC);

[0013] Next, the number of channels of the bottleneck attention BA is expanded to C, and the bottleneck attention BA is divided into 4 groups {BA s ,s=1,2,3,4},BA S represents one of the four groups of bottleneck attention BA, and then the feature group {FG s ,s=1,2,3,4} and {BA s ,s=1,2,3,4} dot product, the final feature group FG' s , FG' is defined as follows:

[0014]

[0015] Among them, ⊙ is the dot product operation;

[0016] Joint operation: All features are aggregated by union; we axially connect the final feature group to obtain the output feature F out , F out The definition is as follows: F out =Cat{FG'1,FG'2,FG'3,FG'4};

[0017] Use the Split-Squeeze-Bottleneck-Union block as an encoder to enhance feature extraction. First, design the SSBU encoder (SSBU-e), which combines the Split-Squeeze-Bottleneck-Union block and the receptive field strategy to enhance the network's ability to extract features and extract global features through different receptive fields. Add a Split-Squeeze-Bottleneck-Union block on each receptive field path.

[0018] Step 3: In the encoding stage, use an encoder with a split-squeeze-bottleneck-joint block to fuse features at different levels; adjust the output features of the three split-squeeze-bottleneck-joint encoders to the same size, and use dot product operations to reduce the noise in low-level features; aggregate all features together through concatenation operations to obtain the initial prediction graph G m ;

[0019] Step 4: In the refinement stage, the information extracted by the separation-squeeze-bottleneck-joint block encoder is used to refine the features of the initial prediction map step by step, and finally an accurate polyp segmentation image is generated; the initial prediction map G is connected by an addition operation m And the information of the encoder that extracts the hierarchical feature f5 is used to generate the next prediction graph G5; then the prediction graph G5 is connected with the information of the encoder that extracts the hierarchical feature f4 by the addition operation to generate the next prediction graph G4, and finally the prediction graph G4 is connected with the information of the encoder that extracts the hierarchical feature f3 by the addition operation to generate the prediction graph G3, and then the sigmoid function, that is, F out =sigmoid(G3), get the final result;

[0020] Step 5: In the training of the entire network, the cross entropy loss function and the intersection-over-union loss function are used to guide the learning of the network; the cross entropy loss function L BCE As shown in the formula:

[0021]

[0022] Among them, y i It means label, y' i represents the predicted value;

[0023] Intersection-over-Union loss function L IoU As shown in the formula:

[0024]

[0025] Among them, y i It means label, y' i represents the predicted value;

[0026] The network loss function L is shown in the formula: L = L BCE +L IoU ,

[0027] The total network loss L total As shown in the formula:

[0028] Among them, G represents the real segmentation map, G m represents the initial prediction graph, G i Represents a prediction graph.

[0029] Compared with the prior art, the present invention has the following beneficial effects:

[0030] (1) We introduce the Split-Squeeze-Bottleneck-Unite Block (SSBU) to mine geometric context information from different perspectives and simultaneously calculate channel correlations to enhance the representation ability of features to solve the polyp segmentation problem.

[0031] (2) We establish a Split-Squeeze-Bottleneck-Union encoder (SSBU-e) and decoder (SSBU-d), which combines the advantages of the Split-Squeeze-Bottleneck-Union block (SSBU) in mining geometric context information and enhancing the representational power of features.

[0032] (3) Quantitative and qualitative comparisons of our proposed method and the state-of-the-art matching methods are performed on a public dataset (CVC300), and the results show that our technique significantly outperforms other algorithms. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1 A flowchart of a preferred embodiment of the present invention;

[0034] Figure 2 A neural network framework for a preferred embodiment of the present invention;

[0035] Figure 3 The encoder based on the separation-squeeze-bottleneck-join block of the preferred embodiment of the present invention;

[0036] Figure 4 A decoder of the separation-squeeze-bottleneck-join block of the preferred embodiment of the present invention;

[0037] Figure 5 This is a diagram showing the effect of performing network segmentation on the CVC300 data set according to a preferred embodiment of the present invention. DETAILED DESCRIPTION

[0038] The present invention will be further described below in conjunction with the accompanying drawings and embodiments.

[0039] It should be noted that the following detailed descriptions are illustrative and are intended to provide further explanation of the present application. Unless otherwise specified, all technical and scientific terms used herein have the same meanings as those commonly understood by those skilled in the art to which the present application belongs.

[0040] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit the exemplary embodiments according to the present application; as used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. In addition, it should be understood that when the terms "comprise" and / or "include" are used in this specification, they indicate the presence of features, steps, operations, devices, components and / or their combinations.

[0041] A polyp image segmentation method based on separation-squeezing-bottleneck-joint codec, reference Figures 1 to 5 , including the following steps:

[0042] Step 1: For a given polyp image I, extract five levels of features {f i ,i=1,2,3,4,5}; the feature f i Divided into low-level features {f i ,i=1,2} and high-level features {f i ,i=3,4,5};

[0043] Step 2: Encode different levels using an encoder with a split-squeeze-bottleneck-join block; the split-squeeze-bottleneck-join block obtains geometric context information of features at each level in different directions; the split-squeeze-bottleneck-join block is specifically divided into four steps: split operation, squeeze operation, bottleneck operation and join operation;

[0044] Separation operation: given input feature F in ∈R H×W×C , divide the input features into four groups of features FG∈R H×W×C / 4 , where H and W represent the size of the input feature, C is the number of channels of the input feature, and FG S Indicates one of four groups of characteristics;

[0045] Squeeze operation: Using global average pooling to capture features {FG s ,s=1,2,3,4}, the definition of GC is as follows:

[0046] Bottleneck operation: We input the channel information GC into the bottleneck attention module BAM to obtain the bottleneck attention BA. The bottleneck attention BA effectively distinguishes the target from the background. The definition of BA is as follows: BA = BAM (GC);

[0047] Next, the number of channels of the bottleneck attention BA is expanded to C, and the bottleneck attention BA is divided into 4 groups {BA s ,s=1,2,3,4},BA Srepresents one of the four groups of bottleneck attention BA, and then the feature group {FG s ,s=1,2,3,4} and {BA s ,s=1,2,3,4} dot product, the final feature group FG' s , FG' is defined as follows:

[0048]

[0049] Among them, ⊙ is the dot product operation;

[0050] Joint operation: All features are aggregated by union; we axially connect the final feature group to obtain the output feature F out , F out The definition is as follows: F out =Cat{FG'1,FG'2,FG'3,FG'4};

[0051] Use the Split-Squeeze-Bottleneck-Union block as an encoder to enhance feature extraction. First, design the SSBU encoder (SSBU-e), which combines the Split-Squeeze-Bottleneck-Union block and the receptive field strategy to enhance the network's ability to extract features and extract global features through different receptive fields. Add a Split-Squeeze-Bottleneck-Union block on each receptive field path.

[0052] Step 3: In the encoding stage, use an encoder with a split-squeeze-bottleneck-joint block to fuse features at different levels; adjust the output features of the three split-squeeze-bottleneck-joint encoders to the same size, and use dot product operations to reduce the noise in low-level features; aggregate all features together through concatenation operations to obtain the initial prediction graph G m ;

[0053] Step 4: In the refinement stage, the information extracted by the separation-squeeze-bottleneck-joint block encoder is used to refine the features of the initial prediction map step by step, and finally an accurate polyp segmentation image is generated; the initial prediction map G is connected by an addition operation m And the information of the encoder that extracts the hierarchical feature f5 is used to generate the next prediction graph G5; then the prediction graph G5 is connected with the information of the encoder that extracts the hierarchical feature f4 by the addition operation to generate the next prediction graph G4, and finally the prediction graph G4 is connected with the information of the encoder that extracts the hierarchical feature f3 by the addition operation to generate the prediction graph G3, and then the sigmoid function, that is, F out =sigmoid(G3), get the final result;

[0054] Step 5: In the training of the entire network, the cross entropy loss function and the intersection-over-union loss function are used to guide the learning of the network; the cross entropy loss function L BCE As shown in the formula:

[0055]

[0056] Among them, y i It means label, y' i represents the predicted value;

[0057] Intersection-over-Union loss function L IoU As shown in the formula:

[0058]

[0059] Among them, y i It means label, y' i represents the predicted value;

[0060] The network loss function L is shown in the formula: L = L BCE +L IoU ,

[0061] The total network loss L total As shown in the formula:

[0062] Among them, G represents the real segmentation map, G m represents the initial prediction graph, G i Represents a prediction graph.

Claims

1. A polyp image segmentation method based on separation-squeezing-bottleneck-joint codec, characterized in that The following steps are involved: Step 1: For a given polyp image I, extract five levels of features {f i ,i=1,2,3,4,5}; the feature f i Divided into low-level features {f i ,i=1,2} and high-level features {f i ,i=3,4,5}; Step 2: Encode different levels using an encoder with a split-squeeze-bottleneck-join block; the split-squeeze-bottleneck-join block obtains geometric context information of features at each level in different directions; the split-squeeze-bottleneck-join block is specifically divided into four steps: split operation, squeeze operation, bottleneck operation and join operation; Separation operation: given input feature F in ∈R H×W×C , divide the input features into four groups of features FG ∈ R H×W×C / 4 , where H and W represent the size of the input feature, C is the number of channels of the input feature, and FG S Indicates one of four groups of characteristics; Squeeze operation: Using global average pooling to capture features {FG s ,s=1,2,3,4}, the definition of GC is as follows: Bottleneck operation: We input the channel information GC into the bottleneck attention module BAM to obtain the bottleneck attention BA. The bottleneck attention BA effectively distinguishes the target from the background. The definition of BA is as follows: BA = BAM (GC); Next, the number of channels of the bottleneck attention BA is expanded to C, and the bottleneck attention BA is divided into 4 groups {BA s ,s=1,2,3,4},BA S represents one of the four groups of bottleneck attention BA, and then the feature group {FG s ,s=1,2,3,4} and {BA s ,s=1,2,3,4} dot product, the final feature group FG' s , FG' is defined as follows: Among them, ⊙ is the dot product operation; Joint operation: All features are aggregated by union; we axially connect the final feature group to obtain the output feature F out , F out The definition is as follows: F out =Cat{FG'1,FG'2,FG'3,FG'4}; Use the Split-Squeeze-Bottleneck-Union block as an encoder to enhance feature extraction. First, design the SSBU encoder (SSBU-e), which combines the Split-Squeeze-Bottleneck-Union block and the receptive field strategy to enhance the network's ability to extract features and extract global features through different receptive fields. Add a Split-Squeeze-Bottleneck-Union block on each receptive field path. Step 3: In the encoding stage, use an encoder with a split-squeeze-bottleneck-joint block to fuse features at different levels; adjust the output features of the three split-squeeze-bottleneck-joint encoders to the same size, and use dot product operations to reduce the noise in low-level features; aggregate all features together through concatenation operations to obtain the initial prediction graph G m ; Step 4: In the refinement stage, the information extracted by the separation-squeeze-bottleneck-joint block encoder is used to refine the features of the initial prediction map step by step, and finally an accurate polyp segmentation image is generated; the initial prediction map G is connected by an addition operation m And the information of the encoder that extracts the hierarchical feature f5 is used to generate the next prediction graph G5; then the prediction graph G5 is connected with the information of the encoder that extracts the hierarchical feature f4 by the addition operation to generate the next prediction graph G4, and finally the prediction graph G4 is connected with the information of the encoder that extracts the hierarchical feature f3 by the addition operation to generate the prediction graph G3, and then the sigmoid function, that is, F out =sigmoid(G3), get the final result; Step 5: In the training of the entire network, the cross entropy loss function and the intersection-over-union loss function are used to guide the learning of the network; the cross entropy loss function L BCE As shown in the formula: Among them, y i It means label, y' i represents the predicted value; Intersection-over-Union loss function L IoU As shown in the formula: Among them, y i It means label, y' i represents the predicted value; The network loss function L is shown in the formula: L = L BCE +L IoU , The total network loss L total As shown in the formula: Among them, G represents the real segmentation map, G m represents the initial prediction graph, G i Represents a prediction graph.

Images