A method for segmenting colorectal polyps based on P2T and reverse attention mechanism

Abstract

The application provides a colorectal polyp image segmentation method based on P2T and a reverse attention mechanism, which comprises the following steps: first, preprocessing the input image, adjusting the size of the image to 352*352, png format, and performing random flipping, random rotation and normalization processing on the image; second, using a pyramid pooling transformer (P2T) as a backbone network to complete the main task of encoding; third, constructing a convolutional neural network-based cascaded partial decoder to aggregate features in the high layer and generate a global feature map as an initial guide area of a reverse attention (RA) mechanism; fourth, using the reverse attention mechanism to mine boundary information in the low layer features and establishing a relationship between the colorectal polyp region and the boundary information to complete the colorectal polyp image segmentation. Experimental results show that the colorectal polyp image segmentation method based on P2T and the reverse attention mechanism can realize accurate segmentation.

Description

TECHNICAL FIELD

[0001] The application belongs to the field of image segmentation, and specifically, a medical image segmentation algorithm based on P2T and reverse attention mechanism is designed to realize accurate segmentation of colorectal polyp images. BACKGROUND

[0002] Colorectal cancer (CRC) has become a serious threat to human health, and its mortality rate ranks fourth in the world. Polyps on the intestinal mucosa are considered a precursor to CRC, and CRC is prone to malignant transformation. Therefore, early diagnosis and treatment of polyps are of great significance. With the development of computer vision and deep learning, many automatic polyp segmentation models have been developed, and significant progress has been made. Accurate automatic polyp segmentation models can help doctors improve efficiency in assisted diagnosis and improve existing medical systems.

[0003] Although the U-Net model based on convolutional neural networks can achieve good results in polyp segmentation, due to the limited receptive field, CNN is only good at capturing local features, but lacks the ability to capture long-distance dependencies, which makes it difficult to better complete global modeling. Therefore, in order to solve the above problems, researchers have introduced the Transformer in the field of NLP that uses self-attention mechanism to obtain global features into the field of computer vision to optimize image automatic segmentation technology.

[0004] However, due to the large computational cost of the Multi-Head Self-Attention (MHSA) mechanism in Transformer, researchers have made improvements. For example, PVT and MViT use a single pooling operation to downsample the feature map in the multi-head self-attention module. Swin Transformer proposes to only calculate the multi-head self-attention module within a small window when modeling the relationship, rather than calculating it on the entire window.

[0005] With the rapid development of society, accurate automatic polyp segmentation models can help doctors improve efficiency in assisted diagnosis and improve existing medical systems. SUMMARY

[0006] The purpose of the present application is to overcome the shortcomings of the prior art and provide a medical image segmentation method based on P2T and reverse attention mechanism to improve the accuracy of colorectal polyp image segmentation, and the segmentation result is better than other segmentation methods.

[0007] To achieve the above purpose, the technical solution adopted by the present application is:

[0008] The key steps of the colon polyp image segmentation method based on P2T and reverse attention mechanism include:

[0009] Step one, pre-process the input image, adjust the size of the input image to 352x352, in png format, and perform random flipping, random rotation and normalization processing on it;

[0010] Step two, use Pyramid Pooling Transformer (P2T) as the main network to complete the main task of encoding;

[0011] Step three, build a convolutional neural network-based cascaded partial decoder to aggregate features in high layers and generate a global feature map as the initial guide region for Reverse Attention (RA);

[0012] Step four, use reverse attention mechanism to mine boundary information in low-level features and establish the relationship between colon polyp region and boundary information to complete colon polyp image segmentation;

[0013] In step one, the input image is pre-processed, and the specific operation description is as follows:

[0014] Given a set of two-dimensional images, first rotate them at a random angle between (-90, 90), then perform horizontal and vertical flipping with a probability of 0.5, and finally calculate the mean and standard deviation to normalize the image;

[0015] In step two, P2T is used to build the main network of the encoder, and the construction method is as follows:

[0016] For an input image of size HxWxC, first split it into patches, each patch is flattened into 48 (4x4x3) elements, and these flattened image blocks are input into an image block encoding module composed of a linear projection layer and learnable position encoding. The image block encoding module expands the feature dimension of 48 to a specified dimension ;

[0017] Then, repeat the stacking of the Pyramid Pooling Transformer module. This module is composed of a multi-head self-attention module based on pyramid pooling, a Feed-Forward Network (FFN), and two residual connections. The Pyramid Pooling Transformer module can be expressed as:

[0018] ，

[0019] ,

[0020] wherein, X , and are the input, output and output of the Transformer block of the multi-head self-attention module respectively, P-MHSA is the multi-head self-attention module based on pyramid pooling, which reshapes the input X into a two-dimensional space and applies multiple average pooling layers with different scales to generate a pyramid feature map, which is shown as follows:

[0021] ,

[0022] wherein represents the generated pyramid feature map, n is the number of pooling layers, and then the pyramid feature map is sent to deep convolution for relative position encoding:

[0023] i=1,2， ，n，

[0024] wherein ( ) represents deep convolution with a kernel size of 3x3, is the relative position encoding of , and then we flatten and concatenate these pyramid feature maps:

[0025] ，

[0026] wherein, is the layer normalization, is the concatenation operation, and Q, K and V in the multi-head self-attention module are calculated as follows:

[0027] ，

[0028] wherein, represents the weight matrix of the linear transformation to generate the query (Q), key (K) and value (V), then Q, K and V are sent to the attention module to calculate the attention feature A, which can be expressed as follows:

[0029] ，

[0030] wherein, is the channel size of K, and the Softmax function is calculated along the row direction of the matrix;

[0031] In step three, the cascaded partial decoder based on convolutional neural network aggregates the features in high layers and generates a global feature map as the initial guiding region of the reverse attention mechanism, defined as It is composed of a convolutional layer with a kernel size of 3x3 and padding of 1, BatchNormlization and ReLU activation function, and the detailed steps are described as follows:

[0032] ，

[0033] ，

[0034] wherein, is the feature map output by each layer of the encoder in step two, represents the Hadamard product, is the output of the cascaded partial encoder, that is, the global feature map;

[0035] In step four, two reverse attention modules are used to mine the boundary information in the low layer features and establish the relationship between the polyp region and the boundary information to complete the polyp segmentation, and the method is described as follows:

[0036] ，

[0037] ，

[0038] wherein represents the down-sampling operation, is the Sigmoid function, is the inverse operation of subtracting the input from the matrix E, is the reverse attention weight in the first reverse attention module, is the output of the first reverse attention module, and then and the down-sampled are added element by element, and the output is The second reverse attention module can be described as follows:

[0039] ,

[0040] ，

[0041] Unlike the first reverse attention module, represents the up-sampling operation, and the up-sampled Element addition operation is performed, and the output is , and finally, the segmentation is completed through a Sigmoid activation function.

[0042] The main advantages of the method proposed by the application are: (1) P2T is used as an encoder, and pyramid pooling is introduced into the multi-head self-attention module, which not only reduces the length of the image word symbol sequence, but also extracts stronger semantic features; (2) a cascaded partial decoder is used to aggregate features from high layers, and a reverse attention module is used to mine boundary clues and establish the relationship between regions and boundaries, thereby improving the segmentation accuracy. BRIEF DESCRIPTION OF DRAWINGS

[0043] Figure 1 is a flowchart of the colon polyp image segmentation method (P2TNet) based on P2T and reverse attention mechanism of the application;

[0044] Figure 2 is the network model based on P2TNet proposed by the application;

[0045] Figure 3 is an example of image segmentation results obtained by the colon polyp image segmentation method based on P2T and reverse attention mechanism of the application and a comparative algorithm. DETAILED DESCRIPTION

[0046] The application will be described in detail below with reference to the accompanying drawings, so that those skilled in the art can better understand the application. It should be pointed out that those skilled in the art can make some improvements to the application without departing from the core idea of the application, and these improvements are within the protection scope of the application;

[0047] As shown in Figure 1 , the colon polyp segmentation method based on P2T and reverse attention mechanism includes the following steps:

[0048] Step one, pre-process the input image, the operation description is:

[0049] Given a set of two-dimensional images, first, randomly select an angle between (-90, 90) for rotation, then perform horizontal and vertical flipping with a probability of 0.5, and finally calculate the mean and standard deviation to normalize the image;

[0050] Step two, use P2T to construct the backbone network of the encoder, the construction method is:

[0051] For an input image with a size of HxWxC, first, split it into each patch is flattened into 48 (4x4x3) elements, and these flattened patches are input into a patch encoding module, which consists of a linear projection layer and a learnable positional encoding, the patch encoding module expands the feature dimension of 48 to a specified dimension ;

[0052] Then, the stacked pyramid pooling Transformer module is repeated. This module consists of a multi-head self-attention module based on pyramid pooling, a feed-forward network (FFN), and two residual connections, and the pyramid pooling Transformer module can be expressed as:

[0053] ,

[0054] ,

[0055] wherein, X , and are the input, output of the multi-head self-attention module and output of the Transformer block, respectively, P-MHSA is the multi-head self-attention module based on pyramid pooling, which reshapes the input X into a two-dimensional space and applies multiple average pooling layers with different scales to generate pyramid feature maps, as shown below:

[0056] ,

[0057] wherein represents the generated pyramid feature maps, n is the number of pooling layers, and then the pyramid feature maps are sent to deep convolution for relative position encoding:

[0058] i=1,2， ，n，

[0059] wherein ( ) represents the deep convolution, with a kernel size of 3x3, is the relative position encoding , and then we flatten and concatenate these pyramid feature maps:

[0060] ，

[0061] wherein, is the layer normalization, For the concatenation operation, Q, K and V in the multi-head self-attention module are calculated as follows:

[0062] ，

[0063] wherein, denotes the weight matrix of the linear transformation for generating the query (Q), the key (K) and the value (V), and then Q, K and V are sent to the attention module to calculate the attention feature A, which can be expressed as follows:

[0064] ，

[0065] wherein, is the channel size of K, and the Softmax function is calculated along the row direction of the matrix;

[0066] Step three, a convolutional neural network-based cascaded partial decoder is constructed to aggregate the features in the high layer and generate a global feature map as the initial guiding area of the reverse attention mechanism, and is defined as follows: which consists of a convolutional layer with a kernel size of 3x3 and a padding of 1, BatchNormlization and ReLU activation function, and the detailed steps are described as follows:

[0067] ，

[0068] ，

[0069] wherein, is the feature map output by each layer of the encoder in step two, denotes the Hadamard product, is the output of the cascaded partial encoder, i.e., the global feature map;

[0070] Step four, two reverse attention modules are used to mine the boundary information in the low layer features and establish the relationship between the polyp region and the boundary information, and the polyp segmentation is completed, and the method is described as follows:

[0071] ，

[0072] ，

[0073] wherein denotes the downsampling operation, is the Sigmoid function, is the inverse operation of subtracting the input from the matrix E, is the reverse attention weight in the first reverse attention module, is the output of the first reverse attention module, and then and the down-sampled are element-wise added to output , and the second reverse attention module can be described as follows:

[0074] ,

[0075] ，

[0076] Unlike the first reverse attention module, represents an up-sampling operation, and the up-sampled are element-wise added to output , and finally pass through the Sigmoid activation function to complete the segmentation;

[0077] In this embodiment, Figure 2 is a network model based on P2TNet proposed by the present application, which takes the original image as input and outputs the colorectal polyp segmentation result;

[0078] Figure 3 is an example of image segmentation result obtained by the colorectal polyp image segmentation method based on P2T and reverse attention mechanism of the present application and the comparative algorithm, Figure 3 contains 5 groups of segmentation results, wherein, Figure 3 (a) is 5 original images, Figure 3 (b) is the corresponding colorectal polyp standard segmentation result, Figure 3 (c) is the segmentation result obtained by PraNet, Figure 3 (d) is the segmentation result obtained by P2TNet of the present application, as can be seen from the figure, the polyp region boundary segmented by the present application is relatively clear, and its segmentation result is closer to the standard segmentation result

[0079] Table 1 is the evaluation results of two different segmentation algorithms in Figure 3 . Among them, the optimal result is marked in bold. From the table, it can be seen that the experimental results obtained by P2TNet of the present application are better than PraNet, the comparative algorithm, in each index, which objectively verifies the above subjective evaluation

[0080]

Claims

1. A P2T and reverse attention mechanism based method for colorectal polyp segmentation, characterized in that, The implementation is achieved by the following steps: Step one, pre-process the input image, adjust the size of the input picture to 352*352, png format, and perform random flipping, random rotation and normalization processing; Step two, adopt the pyramid pooling transformer module, namely P2T, as the main network to complete the main task of encoding, which is composed of a multi-head self-attention module based on pyramid pooling, a feedforward network and two residual connections, and adopts P2T to construct the main network of the encoder; Step three, construct a convolutional neural network-based cascaded partial decoder to aggregate features in the high layer and generate a global feature map as the initial guiding region of the reverse attention mechanism, we define F(·) as a convolutional layer with a kernel size of 3*3 and a padding of 1, BatchNormlization and ReLU activation function, which includes: f 34 = F3(Concat(F1(f4) Θ f3, F2(f4))), S g = F8(F7(Concat(F4(f4) Θ F5(f3) Θ f2, F6(f 34 )))), where f i is the feature map output by each layer of the encoder in step two, denotes the Hadamard product, S g is the output of the concatenated part-encoder, i.e., the global feature map; Step four, adopt two reverse attention modules to mine the boundary information in the low layer features and establish the relationship between the colorectal polyp region and the boundary information, and complete the polyp segmentation, which includes: R3 = f3 o A3, wherein p(·) represents a down-sampling operation, and σ(·) is a Sigmoid function, is the inverse operation of subtracting the input from the matrix E, A3 is the reverse attention weight in the first reverse attention module, R3 is the output of the first reverse attention module, and then R3 and the down-sampled S g is added, and the output is S3, and the second reverse attention module can be described as follows: R1 = f1 o A1, Different from the first reverse attention module, p(·) represents the up-sampling operation, and R1 and the up-sampled S3 are added element by element, and the output is S1, which is finally passed through the Sigmoid activation function to complete the segmentation.

2. The P2T and reverse attention mechanism-based colorectal polyp segmentation method according to claim 1, characterized in that, In step one, the picture preprocessing operation is described as: Given a set of two-dimensional images, first rotate them at a random angle between (-90, 90), then flip them horizontally and vertically with a probability of 0.5, and finally calculate the mean and standard deviation to normalize the image.

3. The method of claim 2, wherein the method is based on P2T and reverse attention mechanism. The construction method is: For an input image of size H x W x C, it is first split into patches, each of which is flattened into 48 (4 x 4 x 3) elements, and these flattened patches are input into a patch encoding module consisting of a linear projection layer and a learnable positional encoding, which expands the feature dimension of 48 to a specified dimension C1. For an input image of size H x W x C, it is first split into patches, each of which is flattened into 48 (4 x 4 x 3) elements, and these flattened patches are input into a patch encoding module consisting of a linear projection layer and a learnable positional encoding, which expands the feature dimension of 48 to a specified dimension C1. Then, repeat the stacking of the pyramid pooling transformer module, and delete the pyramid pooling transformer module, which can be expressed as: X att = LayerNorm(X + P - MHSA(X)), X out = LayerNorm(X att + FFN(X att )), wherein X, X att are the input and output of the multi-head self-attention module, respectively, X out is the output of the pyramid-pooling Transformer module, and P-MHSA is a pyramid-pooling based multi-head self-attention module that reshapes the input X into a two-dimensional space and applies multiple average pooling layers with different scales to generate a pyramid feature map, as shown below: P n = AvgPool n (X), where P n denotes the generated pyramid feature map, n is the number of pooling layers, and the pyramid feature map is fed into deep convolution for relative position encoding: P i enc = DWConv(P i )+ P i , i = 1, 2,..., n, where DWConv(·) denotes a depthwise convolution with kernel size 3 3, P i enc P i After that, we flatten and concatenate these pyramid feature maps: Where LayerNorm is the layer normalization, Concat is the concatenation operation, and Q, K and V in the multi-head self-attention module are calculated as follows: (Q, K, V) = (XW q , PW k , PW v ), where XW q ,PW k ,PW v denotes the weight matrix generating the linear transformation of the query (Q), key (K) and value (V); then, Q, K, V are fed into the attention module to compute the attention feature A, which can be formulated as follows: where d K is the channel size of K, and the Softmax function is computed along the row direction of the matrix.

Images