A colon polyp segmentation method based on multi-scale spatial reverse attention mechanism

Abstract

The application provides a colonoscopy polyp segmentation method based on a multi-scale space reverse attention mechanism, first, a public colon polyp segmentation dataset is collected on a network, and data enhancement processing is performed, and a training set, a test set and a verification set are divided; then, the colon polyp data is input into a designed pyramid Swin Transformer encoder to extract features at four different stages, and multi-scale feature information is extracted through a multi-scale channel attention module to obtain a preliminary aggregated pre-segmentation result; finally, the pre-segmentation result is input into a space reverse attention mechanism module to gradually supplement the edge structure and detail information of the polyp region, so that the purpose of automatically segmenting the colon polyp is achieved. The application is convenient to use, accurate and efficient, has good colonoscopy polyp segmentation effect, and has excellent generalization performance.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of big data deep learning, and particularly relates to a colon polyp segmentation method based on a multi-scale space reverse attention mechanism. BACKGROUND

[0002] Colorectal cancer is a common malignant tumor, which is divided into colon cancer and rectal cancer. In order to timely remove colon polyps and prevent them from becoming cancerous, early screening is very important. Medical image segmentation is an important task in screening and diagnosis, and the result of semantic segmentation helps to identify and evaluate the lesion area of colon polyps, and reduces the risk of missed diagnosis.

[0003] Manual segmentation of colonoscopy polyp images requires experienced experts to collaborate, which is very time-consuming, and different experts may have different opinions on whether the same lesion area is classified as cancerous, resulting in low efficiency of manual segmentation and evaluation of the lesion area, and cannot guarantee the quality. Computer-aided automatic medical image segmentation method provides a more efficient, accurate and high-quality medical image segmentation scheme, and reduces the resource and manpower consumption in the diagnosis process. SUMMARY

[0004] The present application provides a colon polyp segmentation method based on a multi-scale space reverse attention mechanism, which comprises the following steps:

[0005] Step 1, prepare colon polyp image dataset for training and testing;

[0006] Step 2, construct a colon polyp segmentation network based on a multi-scale space reverse attention mechanism;

[0007] Step 3, train the colon polyp segmentation network model based on the multi-scale space reverse attention mechanism using the training set images;

[0008] Step 4, use the network model trained in step 3 to segment the test set images to obtain the image semantic segmentation result.

[0009] Moreover, the colon polyp dataset collected in step 1 is subjected to data enhancement processing of random rotation, random horizontal flipping and random vertical flipping, and then the collected colon polyp data is divided into a training set, a validation set and a test set. The training set data is used to train the constructed colon polyp segmentation network, the validation set is used to test the training effect of the network model, the test set is used to verify the learning ability and generalization performance of the network model.

[0010] Moreover, the input image is first encoded by the Swin Transformer encoder in step 2 through four different stages to obtain four feature maps, then the four feature maps are respectively sent into the multi-scale channel attention mechanism to learn the features at different scales and aggregate to generate a pre-segmentation result, and finally the multi-level spatial reverse attention mechanism is used to gradually repair the boundary information of the polyps.

[0011] The results of each stage of the Swin Transformer encoder are output to form a pyramid Swin Transformer encoder. The sliding window and pixel block size of the Swin Transformer remain unchanged, and the Swin Transformer encoder of each stage models the weight-related information between different pixel blocks in the current input feature map and reduces the resolution of the input feature map. The four output feature maps have different scale global feature information and long-range correlation weights, enriching the correlation information between pixels in the spatial dimension, improving the model's ability to identify polyps of different sizes, types, and colors, and enhancing the model's generalization ability.

[0012] The multi-scale channel attention module first extracts multi-scale feature information from the multi-level feature maps extracted by the Swin Transformer through convolution with different kernel sizes, then uses trainable weight parameters to fuse the feature information to suppress the information gap caused by different convolution kernels and different receptive fields, then uses the channel attention mechanism to redistribute the weights of the feature maps obtained by different receptive fields, and obtains a feature module with multi-scale feature information and channel attention weight information, and finally performs feature fusion on the extracted multi-scale channel attention feature maps and performs residual connection with the input feature maps. The multi-scale channel attention module not only learns the features of the segmented object at different receptive fields, but also to some extent preserves the correlation between the pixel blocks extracted by the Swin Transformer encoder, and can effectively prevent the gradient degradation problem that occurs with the deepening of the network.

[0013] The spatial reverse attention mechanism is composed of a spatial attention mechanism module and a reverse attention mechanism module, and the generated pre-segmentation result is aggregated to obtain boundary information of the segmentation target through the reverse attention mechanism module, then the position information of the segmentation target is obtained through the spatial attention mechanism, and then the polyp feature matrix fused with the boundary information is obtained by multiplying the output features of the four stages of the Swin Transformer, and finally the polyp segmentation result repaired with the boundary information is output through the loop residual convolution; the global feature map obtained by multi-level feature aggregation lacks boundary details, and the reverse attention mechanism can eliminate the predicted segmentation area and learn the details of the segmentation target boundary; the spatial attention mechanism can extract the position information in the pre-segmentation feature map, and when the model locates the polyp boundary area in the feature map, the extreme binary processing of the polyp area and other areas is easy to lose the edge details, and through the spatial attention mechanism, the non-polyp area of the initial positioning is given greater weight in the spatial direction, and the weight in the polyp area direction is reduced, so that when the boundary area features are learned at all feature levels, there is more abundant spatial information.

[0014] Moreover, the size of the data set is set to N*N pixels in step 3, the optimizer is set to AdamW optimizer, the learning rate is set to 1* e -4 , the learning rate decay period is 40, the decay rate is set to 0.1, the batchsize size is set to 16, and the training is performed for 80 cycles.

[0015] Compared with the prior art, the present application has the following advantages:

[0016] 1) The Swin Transformer encoder can adjust the size and individual parameters of the decoder as needed to build models of different sizes, and further research and optimization can be carried out for different types of data sets to meet the application and demand of more medical segmentation tasks.

[0017] 2) A multi-scale channel attention module is proposed, which can not only learn the features of the segmentation object under different receptive fields, but also to a certain extent, preserve the correlation between the pixel blocks extracted by the Swin Transformer encoder, effectively prevent the gradient degradation problem that occurs with the deepening of the network, and has good multi-scale information processing capability and channel information extraction capability, and can be flexibly applied to other computer vision tasks.

[0018] 3) The spatial reverse attention mechanism is proposed, including the spatial attention mechanism and the reverse attention mechanism, and the reverse attention mechanism can eliminate the predicted segmentation area and learn the details of the segmentation target boundary; through the spatial attention mechanism, the non-polyp region of the initial positioning is given greater weight in the spatial direction, and the weight of the polyp region direction is reduced, so that when learning the boundary region features at all feature levels, there is more rich spatial information to assist the segmentation network model to repair the colon polyp boundary information.

[0019] 4) The loop residual convolution block is improved and stacked to strengthen the feature learning ability of the convolution network and effectively prevent the gradient vanishing problem. BRIEF DESCRIPTION OF DRAWINGS

[0020] Figure 1 The flowchart of the embodiment of the present application.

[0021] Figure 2 The colon polyp image data of the embodiment of the present application.

[0022] Figure 3 The overall structure diagram of the colon polyp segmentation network of the embodiment of the present application.

[0023] Figure 4 The structure diagram of the multi-scale channel attention mechanism of the embodiment of the present application.

[0024] Figure 5 The structure diagram of the spatial reverse attention mechanism of the embodiment of the present application.

[0025] Figure 6 The structure diagram of the loop residual convolution block of the embodiment of the present application. DETAILED DESCRIPTION

[0026] The present application provides a colon polyp segmentation method based on a multi-scale spatial reverse attention mechanism, and the technical solutions of the present application are further described below in conjunction with the drawings and embodiments.

[0027] As shown in the figure, the flow of the embodiment of the present application includes the following steps: Figure 1

[0028] Step 1, prepare the colon polyp image data set for training and testing.

[0029] ​The collected colon polyp dataset: Kvasir, CVC-ClinicDB, CVC-ColonDB, ETIS, Endoscene-CVC-T, and the collected colon polyp data are subjected to random rotation, random horizontal flipping and random vertical flipping data enhancement processing, and then the collected colon polyp data are divided into training set, validation set and test set according to 8:1:1. The training set data are used to train the constructed colon polyp segmentation network, the validation set is used to test the training effect of the network model, and the test set is used to verify the learning ability and generalization performance of the network model.

[0030] Step 2, constructing a colon polyp segmentation network based on a multi-scale spatial reverse attention mechanism.

[0031] The overall structure of the colon polyp segmentation network is shown in Figure 3 The input image is first passed through four different stage Swin Transformer encoders to obtain four feature maps, then the four feature maps are respectively sent into the multi-scale channel attention mechanism to learn the features at different scales and aggregate to generate a pre-segmentation result, and finally the pre-segmentation result and the output features of the four stages of Swin Transformer are input into the spatial reverse attention mechanism to gradually repair the feature information of the polyp boundary.

[0032] The results of each stage of the Swin Transformer encoder are output to form a pyramid Swin Transformer encoder, and the sliding window and pixel block size of the Swin Transformer remain unchanged. The Swin Transformer encoder at each stage models the weight-related information between different pixel blocks in the current input feature map and reduces the resolution of the input feature map. The four output feature maps have different scale global feature information and long-range correlation weights, enriching the related information between pixels in the spatial dimension, improving the model's ability to identify polyps of different sizes, types and colors, and enhancing the model's generalization ability.

[0033] The structure of the multi-scale channel attention module is shown in Figure 4As shown, first, multi-scale feature information is extracted from the multi-level feature maps extracted from the SwinTransformer by convolution with different kernel sizes, then the feature information is fused using trainable weight parameters to suppress the information gap caused by different convolution kernels and different receptive fields, then the channel attention mechanism is used to redistribute the weights of the feature maps obtained by different receptive fields, and a feature module with multi-scale feature information and channel attention weight information is obtained, finally the extracted multi-scale channel attention feature maps are fused and connected with the input feature maps. The multi-scale channel attention module can not only learn the features of the segmented object under different receptive fields, but also to some extent preserve the correlation between the pixel blocks extracted by the Swin Transformer encoder, and effectively prevent the gradient degradation problem that occurs with the deepening of the network.

[0034] As shown in Figure 5 The spatial reverse attention mechanism is composed of a spatial attention mechanism module and a reverse attention mechanism module. The aggregated generated pre-segmentation result first obtains the boundary information of the segmentation target through the reverse attention mechanism module, then obtains the position information of the segmentation target through the spatial attention mechanism, and then multiplies the polyp feature matrix fused with the boundary information with the output features of the Swin Transformer4 stage, and finally outputs the polyp segmentation result after repairing the boundary information through the loop residual convolution. The global feature map obtained by aggregating multi-level features lacks boundary details, and the use of reverse attention mechanism can eliminate the predicted segmentation area and learn the details of the segmentation target boundary. The spatial attention mechanism can extract the position information in the pre-segmentation feature map. When the model locates the polyp boundary area in the feature map, the extreme binary processing of the polyp area and other areas is easy to lose edge details. Through the spatial attention mechanism, the non-polyp area of the initial positioning is given greater weight in the spatial direction, and the weight in the polyp area direction is reduced, which can learn more rich spatial information when learning the boundary area features at all feature levels. The loop residual convolution structure is as shown in Figure 6 The improved loop residual convolution block enhances the feature learning ability of the convolutional network and can effectively prevent the gradient disappearance problem.

[0035] Step 3: Train the colon polyp segmentation network model based on the multi-scale spatial reverse attention mechanism using the training set images.

[0036] In this embodiment, the size of the data set is uniformly set to 352x352 pixels, the optimizer is set to AdamW optimizer, the learning rate is set to 1x e -4 , the learning rate decay period is 40, the decay rate is set to 0.1, the batchsize size is set to 16, and the training is performed for 80 cycles.

[0037] Step 4: using the trained network model in step 3 to segment the test set images to obtain image semantic segmentation results.

[0038] In specific implementation, the above process can be realized by computer software technology to automatically run the process.

[0039] The specific embodiments described herein are merely illustrative of the spirit of the present application. Those skilled in the art of the present application can make various modifications or supplements to the described specific embodiments or replace them with similar ways, but will not deviate from the spirit of the present application or exceed the scope defined by the appended claims.

Claims

1. A colon polyp segmentation method based on a multi-scale spatial reverse attention mechanism, characterized in that, Comprising the following steps: Step 1, prepare the colon polyp image dataset for training and testing; Step 2, build a colon polyp segmentation network based on a multi-scale spatial reverse attention mechanism; First, the input image is encoded by four different stages of Swin Transformer encoder to obtain four feature maps, then the four feature maps are sent into the multi-scale channel attention mechanism to learn the features at different scales and aggregate to generate the pre-segmentation result, and finally the multi-level spatial reverse attention mechanism is used to gradually repair the boundary information of the polyp; The results of each stage of the Swin Transformer encoder are output to form a pyramid Swin Transformer encoder; the size of the sliding window and the pixel block of the Swin Transformer remains unchanged, and each stage of the Swin Transformer encoder models the weight correlation information between different pixel blocks in the current input feature map and reduces the resolution of the input feature map, and the four output feature maps have different scale global feature information and long-range correlation weights; The multi-scale channel attention module first extracts multi-scale feature information from the multi-level feature maps extracted by the Swin Transformer through convolution with different kernel sizes, then uses trainable weight parameters to fuse the feature information to suppress the information gap caused by different convolution kernels and different receptive fields, then uses the channel attention mechanism to redistribute the weights of the feature maps obtained by different receptive fields, and obtains a feature module with multi-scale feature information and channel attention weight information, finally, the extracted multi-scale channel attention feature maps are fused and connected with the input feature maps; The spatial reverse attention mechanism is composed of a spatial attention mechanism module and a reverse attention mechanism module, and the aggregated pre-segmentation result is first obtained by the reverse attention mechanism module to obtain the boundary information of the segmentation target, then the position information of the segmentation target is obtained by the spatial attention mechanism, then the polyp feature matrix fused with the boundary information is obtained by multiplying the output features of the four stages of the Swin Transformer, and finally the polyp segmentation result with the boundary information repaired is output by the loop residual convolution; Step 3, train the colon polyp segmentation network based on the multi-scale spatial reverse attention mechanism using the training set images; Step 4, use the network model trained in step 3 to segment the test set images to obtain the image semantic segmentation result.

2. The colon polyp segmentation method based on multi-scale spatial reverse attention mechanism according to claim 1, characterized in that: Step 1 collects public colon polyp datasets and performs data enhancement processing such as random rotation, random horizontal flipping and random vertical flipping on the collected colon polyp data, then divides the collected colon polyp data into training set, validation set and test set, the training set data is used to train the constructed colon polyp segmentation network, the validation set is used to test the training effect of the network model, the test set is used to verify the learning ability and generalization performance of the network model.

3. The colon polyp segmentation method based on multi-scale spatial reverse attention mechanism according to claim 1, characterized in that: The size of the dataset is set to N in step 3 N pixels, the optimizer is set to AdamW optimizer, the learning rate is set to , the learning rate decay period is 40, the decay rate is set to 0.1, the batchsize size is set to 16, and the training is 80 cycles.

Images