Polyp segmentation method based on lightweight network model and reverse attention module

Abstract

The present invention provides a polyp segmentation method based on a lightweight network model and a reverse attention module, including removing the last classification network part of the lightweight network model MobileNetV3 and replacing the SENet module with a coordinated attention module to improve MobileNetV3, so as to obtain a lightweight network segmentation model CaNet as a backbone network; the high-level features output by the backbone network are transmitted in parallel to the parallel axial receptive field module, and the global dependency and local representation are extracted without losing detail information; the feature maps output in parallel by the parallel axial receptive field module are upsampled to a uniform size by different multiples, and a global mapping map is initially obtained by point-by-point multiplication and bilinear interpolation; the reverse attention module is used to erase the existing estimated polyp area from the high-level output features, so as to sequentially mine complementary areas and details, obtain a more refined polyp segmentation result, and improve the segmentation accuracy. The present invention is more lightweight and efficient for the segmentation of polyp image data sets.

Description

Technical Field

[0001] The present invention relates to the technical field of polyp segmentation, and in particular to a polyp segmentation method based on a lightweight network model and a reverse attention module. Background Art

[0002] Data from research reports show that colorectal cancer is one of the most common cancers and one of the most deadly. When the disease develops to the late stage, the 5-year survival rate of colon cancer is much lower than that of early detection, with a minimum of only 10%. In cases diagnosed early, if early detection and treatment are successful, the 5-year survival rate will increase to more than 90%. Therefore, it is crucial to prevent colorectal cancer by screening and removing precancerous lesions. In the actual polyp removal surgery, a lot of reliance is required on the physician's observation and analysis of the polyp site. How to quickly and accurately locate the location of the polyp, measure the size of different polyps, and other operations all depend on the accurate positioning and segmentation of the polyp. Therefore, accurate polyp segmentation is very important and urgently needed in clinical practice.

[0003] In recent years, in traditional polyp segmentation methods, early learning-based methods rely on manually extracted features, such as color, texture, shape, appearance, and a combination of these features. In 2015, Bernal et al. proposed a texture-based polyp detection method, which uses the window valley stacking median depth energy map to obtain continuous boundaries related to polyps, and then locate the polyp area. In 2018, Sasmal et al. used the principal component tracking method to perform low-rank sparse decomposition of the image based on the shape of the polyp, and then segmented the polyp image using the Chan-Vese algorithm. This type of method cannot effectively combine all the features contained in the polyp area at the same time, but only considers some features alone, resulting in certain limitations in the segmentation results.

[0004] Compared with traditional polyp segmentation methods, deep convolutional neural networks have more outstanding advantages in medical image segmentation. Most deep learning methods focus on segmenting the entire area of ​​the polyp, but ignore the regional boundary constraints, and accurate boundary information is crucial to improving segmentation performance. In response to the above problems, in 2019, Murugesan et al. used both regional and boundary information in polyp segmentation, but did not fully capture the relationship between regions and boundaries. In addition, Fang et al. proposed a three-step selective feature aggregation network with both regional and boundary constraints to segment polyps. This method explicitly considers the dependency between regions and boundaries and obtains good results through additional edge supervision, but its computational efficiency is low and there is a risk of overfitting. In 2020, Fan et al. proposed a parallel local high-level feature decoding scheme, which uses the reverse attention mechanism to better find the target area features and boundary details, and adds a receptive field module to obtain more visual information, but it abandons low-level features to improve efficiency and loses some semantic information. The model has a large number of parameters and high computational complexity.

[0005] In view of some problems existing in the above methods, it is particularly important to innovatively design an efficient and accurate polyp segmentation method. Summary of the invention

[0006] In view of the technical problems in existing polyp segmentation methods, such as large number of model parameters, high computational complexity and low polyp segmentation accuracy, the present invention provides a polyp segmentation method based on a lightweight network model and a reverse attention module.

[0007] In order to solve the above technical problems, the present invention adopts the following technical solutions:

[0008] A polyp segmentation method based on a lightweight network model and an inverse attention module comprises the following steps:

[0009] S1. Improve the lightweight network model MobileNetV3 using the coordinated attention module. Specifically, remove the last classification network part of the lightweight network model MobileNetV3 and replace the SENet module with the coordinated attention module to obtain the lightweight network segmentation model CaNet as the backbone network.

[0010] S2, the high-level features output by the backbone network are passed in parallel to the parallel axial receptive field module, and additional global refinement is performed at different scales and the outputs are connected to extract global dependencies and local representations without losing detail information;

[0011] S3, upsampling the feature maps output by the parallel axial receptive field modules to a uniform size by different multiples, aggregating them by point-by-point multiplication, and then performing bilinear interpolation to preliminarily obtain an unstructured global mapping map to provide rough positioning information of the polyp;

[0012] S4. The high-level side features output in parallel by the global mapping image and the parallel axial receptive field module are sent to multiple reverse attention modules constructed in a cascade manner. The reverse attention module erases the existing estimated polyp area from the high-level output features, thereby sequentially mining complementary areas and details to obtain a more refined polyp segmentation result. The output of the last reverse attention module is sent to the Sigmoid activation function to generate the final prediction map of the polyp segmentation area.

[0013] Furthermore, in the step S1, the coordinated attention module first uses pooling kernels of size (H, 1) and (1, W) to encode each channel input feature map along the horizontal coordinate direction and the vertical coordinate direction, and then the two transformations perform feature aggregation along the two spatial directions, and return a pair of direction-aware attention maps, each of which captures the long-range dependency of the input feature map along one spatial direction and preserves the precise position information of the other spatial direction. Then, regularization and nonlinear functions are performed, and then the feature map is convolved and sigmoid function is performed, and finally the attention weight is output.

[0014] Furthermore, the high-level features output by the backbone network in step S2 are transmitted in parallel to three parallel axial receptive field modules. Each parallel axial receptive field module first replaces the 3×3 convolution layer with a 1×3 and 3×1 convolution layer, replaces the 5×5 convolution layer with a 1×5 and 5×1 convolution layer, and replaces the 7×7 convolution layer with a 1×7 and 7×1 convolution layer, and then passes through the dilated convolution modules with dilated rates of 3, 5, and 7, respectively, and adds parallel axial attention modules in parallel to perform additional global refinement on each scale and connect the outputs, and finally aggregates and adds with the 1×1 convolution layer, and then passes through the 3×3 convolution layer and the 1×1 convolution layer to output the feature map.

[0015] Furthermore, the parallel axial attention module adopts an axial attention strategy and uses the input feature map to calculate the non-local operations of the horizontal and vertical axes in parallel. Specifically, in the horizontal and vertical axes, two 1×1 convolutions are first performed followed by matrix multiplication transformation, and then a 1×1 convolution is performed for matrix multiplication transformation. Finally, the outputs of the two axes are added point by point to output the feature map.

[0016] Furthermore, in step S4, three reverse attention modules are specifically constructed in a cascade manner, and the downsampled global mapping image and the high-level side features output by the first parallel axial receptive field module are sent to the first reverse attention module to obtain the first reverse attention feature. The output after adding the first reverse attention feature and the downsampled global mapping image is sent to the second reverse attention module with the high-level side features output by the second parallel axial receptive field module to obtain the second reverse attention feature, and on the other hand, it is added to the second reverse attention feature after upsampling. The added output is sent to the third reverse attention module with the high-level side features output by the third parallel axial receptive field module to obtain the third reverse attention feature, and on the other hand, it is added to the third reverse attention feature after upsampling. The added output is sent to the Sigmoid activation function to generate the final prediction map of the polyp segmentation area.

[0017] Furthermore, each reverse attention module is connected by the corresponding parallel axial receptive field module output high-level side features f i With the feature map S i The reverse attention weight A obtained after Sigmoid activation function and inversion i Multiply to obtain the output reverse attention feature R i , the specific description is as follows:

[0018] R i =f i ⊙A i

[0019] Among them, i = 3, 4, 5, ⊙ represents multiplication, reverse attention weight A i Described as:

[0020]

[0021] Where P(·) represents the upsampling operation, σ(·) is the Sigmoid activation function, is the reverse operator that subtracts the input from the all-ones matrix E.

[0022] Furthermore, the output of each reverse attention module and the global map adopt the deep supervision overall loss function as the optimization target, and the overall loss function is defined as follows:

[0023]

[0024] Among them, G is the true value map, S g is the global map, The global map is upsampled, S i is the output of the reverse attention module, The output of the reverse attention module is upsampled, and L is the combination of the weighted IoU loss function and the weighted binary cross entropy BCE loss function, which is described as follows:

[0025]

[0026] in, represents the weighted IoU loss based on global constraints, represents the weighted binary cross entropy BCE loss based on local constraints; and They are defined as follows:

[0027]

[0028]

[0029] Among them, i∈I refers to the pixel point in the predicted value and the true value, y represents the true value, represents the predicted value, and w is the weighted value.

[0030] Furthermore, the method also includes using a weighted Dice metric Structural index S α , mean absolute error MAE, enhanced alignment index Indicators to measure the final prediction graph S p The similarity and dissimilarity between the graph and the true value graph G, each index is specifically expressed as:

[0031]

[0032] Among them, ω is the weight assigned to each category based on the number of samples in each category; Precision refers to the accuracy, which means the probability of correctly predicting as a positive sample among the samples predicted as positive samples in the prediction results; Recall refers to the recall rate, which means the probability of being correctly predicted as a positive sample in the positive samples of the original samples;

[0033] S α =(1-α)*S o (S p ,G)+α*S r (S p ,G)

[0034] Among them, α is used to control the object level similarity S o and the region-level similarity S r The balance coefficient of

[0035]

[0036]

[0037] Among them, w and h represent the width and height of the truth map G, (x, y) represents the coordinates of each pixel in the truth map G, and the symbol is the augmented alignment matrix.

[0038] Compared with the prior art, the polyp segmentation method based on the lightweight network model and the reverse attention module provided by the present invention has the following advantages:

[0039] 1. The present invention uses the coordinated attention module to improve the lightweight network model MobileNetV3, and uses the improved lightweight network segmentation model CaNet as the backbone network, so that it can more accurately identify and locate the position of interest, improve computing efficiency and performance, and make the polyp segmentation network model more lightweight.

[0040] 2. The parallel axial receptive field module used in the present invention can better perform local representation and extract global dependencies without losing detail information. It can perform additional global refinement on different scales and connect the outputs. Therefore, it not only focuses on local representation, but also has a better extraction of global dependencies, thereby making the segmentation of regional information more comprehensive.

[0041] 3. The present invention uses a foreground erasing method to better mine the polyp target area and boundary information through the output features of the reverse attention module, and uses the true value image to deeply supervise the network to reduce the gradient disappearance and solve the problem of slow network convergence, thereby effectively improving the polyp segmentation accuracy and making it more efficient. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 It is a principle block diagram of the polyp segmentation method based on a lightweight network model and a reverse attention module provided by the present invention.

[0043] Figure 2 The invention provides a method for improving MobileNetV3 to obtain Figure 1 Schematic diagram of the coordinated attention module structure of the backbone network CaNet.

[0044] Figure 3 The present invention provides Figure 1 Schematic diagram of the parallel axial receptive field module structure.

[0045] Figure 4 The present invention provides Figure 3 Schematic diagram of the parallel axial attention module structure.

[0046] Figure 5 The present invention provides Figure 1 Schematic diagram of the reverse attention module structure.

[0047] Figure 6 This is a comparative analysis result diagram of the computational complexity (FLOPs) of the present invention and other typical model methods.

[0048] Figure 7 It is a comparative analysis result diagram of the model parameters (Params) of the present invention and other typical model methods.

[0049] Figure 8 This is a comparison chart of qualitative results between the present invention and other typical model methods on five different polyp data sets. DETAILED DESCRIPTION

[0050] In order to make the technical means, creative features, objectives and effects achieved by the present invention easy to understand, the present invention is further explained below with reference to specific diagrams.

[0051] Please refer to Figure 1 As shown, the present invention provides a polyp segmentation method based on a lightweight network model and a reverse attention module, comprising the following steps:

[0052] S1. Improve the lightweight network model MobileNetV3 using the coordinated attention module CA (Coordinated Attention), specifically by removing the last classification network part of the lightweight network model MobileNetV3 and replacing the SENet module with the coordinated attention module CA, to obtain the lightweight network segmentation model CaNet as the backbone network, so as to help the polyp segmentation network model MobileRaNet based on the lightweight network model and the reverse attention module provided by the present invention to more accurately locate and identify the target of interest, and improve the computational efficiency and performance;

[0053] S2, the high-level features output by the backbone network CaNet are passed in parallel to the parallel axial receptive field module PA_RFB (Parallel Axial Receptive Field Module), and additional global refinement is performed at different scales and the outputs are connected to extract global dependencies and local representations without losing detail information;

[0054] S3, the feature map output by the parallel axial receptive field module PA_RFB is upsampled to a uniform size by different multiples, aggregated by point-by-point multiplication, and then bilinearly interpolated to obtain a global map S with no structure. g , used to provide rough location information of polyps;

[0055] S4, because the global mapping graph S g From the deepest convolutional neural network, it can only capture the relative rough position of polyp tissue without structural details, so the global map Sg The high-level side features output in parallel by the parallel axial receptive field modules PA_RFB are sent to multiple reverse attention modules RA (Reverse Attention) constructed in a cascade manner. The reverse attention module RA erases the existing estimated polyp area from the high-level output features, thereby sequentially mining complementary areas and details to obtain a more refined polyp segmentation result, improve the segmentation accuracy, and make the segmentation of polyp image datasets more lightweight and efficient. The output of the last reverse attention module RA is sent to the Sigmoid activation function to generate the final prediction map S of the polyp segmentation area. p .

[0056] As a specific embodiment, the design background of the backbone network CaNet in the present invention is as follows: Considering the limited computing power of lightweight networks, the most popular attention mechanism is still the SE Attention used in SENet, which calculates channel attention through 2D global pooling, providing significant performance improvement at a relatively low computational cost. Unfortunately, the SE module only considers the encoding of inter-channel information and ignores the importance of position information, which is actually crucial for many visual tasks that need to capture the target structure. Therefore, later methods such as CBAM used position information by reducing the number of channels and then using large-size convolutions. However, convolution can only capture local correlations, and long-range dependencies that are very important for modeling visual tasks seem to be somewhat powerless. Based on this, the CA module embeds position information into channel attention, allowing the lightweight network to pay attention to a larger area while avoiding a large amount of computational overhead. In order to effectively integrate spatial coordinate information into the generated attention map to improve the segmentation accuracy of the network, similar to the usage of the SENet module in MobileNetV3, the SENet module after point-wise convolution is replaced with the CA module. The CA module can use two one-dimensional global pooling operations to aggregate the input features in the vertical and horizontal directions into two independent direction-aware feature maps, and then the two feature maps embedded with specific direction information are encoded into two attention maps, each of which captures the long-range dependency of the input feature map along a spatial direction. Therefore, the position information is stored in the generated attention map, and the two attention maps are then multiplied by the input feature map to enhance the representation capability of the feature map. Finally, by using the CA module in the backbone network CaNet, the global dependency representation is increased, and the accuracy of the network is improved with almost no additional computational overhead.

[0057] As a specific example, please refer to Figure 2As shown in the figure, global pooling is often used in channel attention to globally encode spatial information as channel descriptors, so it is difficult to save position information. In order to facilitate the attention module to capture spatial long-range dependencies with precise position information, global pooling is decomposed into a pair of one-dimensional feature encoding operations. Specifically, the coordinated attention module CA in step S1, for input X, first uses pooling kernels of size (H, 1) and (1, W) to encode each channel input feature map along the horizontal coordinate direction and the vertical coordinate direction, and then these two transformations perform feature aggregation along two spatial directions, and return a pair of direction-aware attention maps, each of which captures the long-range dependency of the input feature map along one spatial direction and saves the precise position information of another spatial direction, which helps the network to more accurately locate the target of interest, and then performs regularization and nonlinear functions, and then convolves and sigmoid functions on the feature map, and finally outputs the attention weight. Therefore, in order to alleviate the loss of position information caused by 2D global pooling, the coordinated attention module CA provided by the present invention decomposes the channel attention into two parallel 1D feature encoding processes, effectively integrating the spatial coordinate information into the generated attention map, and encodes the channel relationship and long-range dependency through precise position information, which can not only capture cross-channel information, but also capture direction perception and position perception information.

[0058] As a specific embodiment, the parallel axial receptive field module PA_RFB described in the present invention is composed of a parallel axial attention module PAA (Parallel Axia Attention) and a receptive field module RFB (Receptive Field Block), which is used to extract global dependencies and local representations. That is, in order to reduce the number of channels without losing any detail information, the present invention designs a parallel axial receptive field module PA_RFB with a receptive field module RFB strategy. As a specific implementation method, please refer to Figure 1 and Figure 3 As shown, the high-level features (left frame) output by the backbone network CaNet in step S2 are passed in parallel to three parallel axial receptive field modules PA_RFB. Each parallel axial receptive field module PA_RFB first replaces the 3×3 convolution layer with 1×3 and 3×1 convolution layers, replaces the 5×5 convolution layer with 1×5 and 5×1 convolution layers, and replaces the 7×7 convolution layer with 1×7 and 7×1 convolution layers to reduce the amount of calculation. Then, the dilated convolution modules with dilation rates of 3, 5, and 7 are respectively passed through, and the parallel axial attention module PAA is added in parallel to perform additional global refinement on each scale and connect the outputs. Finally, it is aggregated and added with the 1×1 convolution layer, and then passes through the 3×3 convolution layer and the 1×1 convolution layer to output the feature map.

[0059] As a specific example, please refer to Figure 4As shown, in the era of deep learning, usually fine-grained feature maps are difficult to extract and combine with each other. This problem is solved by using axial attention, which performs non-local operations on a single axis and connects each operation sequentially. The present invention designs parallel axial attention to extract global dependencies and local representations. By adopting the axial attention strategy, non-local operations of the horizontal and vertical axes are calculated, but they are placed in parallel. By positioning the vertical and horizontal attention, the contribution of these two methods to the final output is almost the same as the sequential method. In addition, it has been observed that when using parallel connections, pixel-level summation can aggregate feature maps more effectively than series connection without degrading performance. Because both the horizontal and vertical axes use the same input, and their contribution to the parallel connection output is almost equal. However, since attention based on a single axis can cause unexpected deformation, pixel-level summation can help compensate for this artifact. The present invention uses the input feature map to calculate two non-local operations, one for the horizontal axis and the other for the vertical axis, that is, the input feature map is used to calculate the non-local operations of the horizontal axis and the vertical axis in a parallel placement manner. Specifically, in the horizontal axis and the vertical axis, two 1×1 convolutions are first performed followed by matrix multiplication transformation, and then a 1×1 convolution is performed for matrix multiplication transformation, and finally the outputs of the two axes are added point by point to output a feature map. Therefore, the parallel axial attention module provided by the present invention calculates the non-local operations of the horizontal axis and the vertical axis by adopting an axial attention strategy, and adopts a parallel placement method to achieve additional global refinement of each scale and connect the outputs.

[0060] As a specific example, please refer to Figure 1 As shown, in step S4, three reverse attention modules RA are specifically constructed in a cascade manner, and the downsampled global map S g The high-level side feature f5 output by the first parallel axial receptive field module is sent to the first reverse attention module RA1 to obtain the first reverse attention feature R5. The output after adding the first reverse attention feature R5 and the downsampled global mapping map is sent to the second reverse attention module RA2 with the high-level side feature f4 output by the second parallel axial receptive field module to obtain the second reverse attention feature R4. On the other hand, it is added to the second reverse attention feature R4 after upsampling. The added output is sent to the third reverse attention module with the high-level side feature f3 output by the third parallel axial receptive field module to obtain the third reverse attention feature R3. On the other hand, it is added to the third reverse attention feature R3 after upsampling. The added output is sent to the Sigmoid activation function to generate the final prediction map S of the polyp segmentation area. pThe three reverse attention modules in this empirical example are constructed in a cascade manner, which essentially means that the output of the second reverse attention module RA2 depends on the output of the first reverse attention module RA1, and the output of the third reverse attention module RA3 depends on the output of the second reverse attention module RA2. Finally, the output of the last third reverse attention module RA3 is fed into the Sigmoid activation function to generate the final prediction map S of the polyp segmentation area. p .

[0061] As a specific embodiment, the design background of the reverse attention module RA of the present invention is as follows: In clinical practice, doctors first roughly locate the polyp area, and then carefully examine the local tissue to accurately mark the polyp. Referring to their approach, firstly, a global map S with an unstructured g , used to provide rough location information of polyps; secondly, a progressive framework is used as a fine annotator to gradually mine distinguishable polyp regions in a foreground erasing manner. Specifically, compared with the existing method of simply aggregating all feature layers, the present invention chooses to erase the estimated polyp regions from the high-level output features in a way to sequentially mine complementary regions and detail information, while the current prediction results are obtained by upsampling the information of deeper network layers.

[0062] As a specific example, please refer to Figure 5 As shown, each reverse attention module RA is connected by the corresponding parallel axial receptive field module output high-level side features f i With the feature map S i The reverse attention weight A obtained after Sigmoid activation function and inversion i Multiply to obtain the output reverse attention feature R i , the specific description is as follows:

[0063] R i =f i ⊙A i

[0064] Among them, i = 3, 4, 5, ⊙ represents multiplication, reverse attention weight A i It is a salient object detection task that has been widely used in the field of computer vision research. It is described as:

[0065]

[0066] Where P(·) represents the upsampling operation, σ(·) is the Sigmoid activation function, is the reverse operator that subtracts the input from the all-one matrix E. Where, Figure 5The details of the process are shown, and the erasing mechanism driven by reverse attention can eventually refine the imprecise and rough estimation into an accurate and complete edge prediction map.

[0067] As a specific embodiment, the loss function L of the present invention is designed as a combination of a weighted IoU loss function and a weighted binary cross entropy BCE loss function, which is specifically described as follows:

[0068]

[0069] in, represents the weighted IoU loss based on global constraints, represents the weighted binary cross entropy BCE loss based on local (pixel-level) constraints; and They are defined as follows:

[0070]

[0071]

[0072] Among them, i∈I refers to the pixel point in the predicted value and the true value, y represents the true value, represents the predicted value, and w is the weighted value. Unlike the standard IoU loss currently widely used in segmentation tasks, the weighted IoU loss highlights the importance of difficult sample pixels by increasing their weights. In addition, compared with the standard BCE loss function, More attention is paid to difficult sample pixels rather than allocating equal weights to all pixels. At the same time, the present invention uses the outputs of the three reverse attention modules (i.e., S3, S4, S5) and the global map S g Using a deep supervision strategy, that is, using a deep supervision overall loss function as the optimization target, the mappings are upsampled one by one (e.g. ) to the same size as the truth map G, so that the overall loss function of the proposed MobileRaNet is defined as follows:

[0073]

[0074] Among them, G is the true value map, S g is the global map, The global map is upsampled, S i is the output of the reverse attention module, The output of the reverse attention module is upsampled.

[0075] As a specific embodiment, the present invention will mainly use Kvasir's official website as the basis for comparison, namely meanDice and mean IoU. At the same time, in order to have a deeper understanding of the model performance and compare with other methods, the present invention will also use four other metrics widely used in the field of object detection. In the comparison, the present invention selects S3 with Sigmoid activation function as the final prediction graph S p Therefore, the polyp segmentation method of the present invention also includes using the weighted Dice metric Structural index S α , mean absolute error MAE, enhanced alignment index Indicators to measure the final prediction graph S p The similarity and dissimilarity between the graph and the true value graph G, each index is specifically expressed as:

[0076]

[0077] Among them, the The indicator is used to correct the problem of "equally important defects" in Dice. ω is the weight given to each category based on the number of samples in each category; Precision refers to the accuracy rate, which means the probability of correctly predicting as a positive sample in the prediction result; Recall refers to the recall rate, which means the probability of correctly predicting as a positive sample in the positive sample of the original sample;

[0078] S α =(1-α)*S o (S p ,G)+α*S r (S p ,G)

[0079] Among them, the S α The indicator is used to measure the structural similarity between the predicted image and the true image, and α is used to control the object level similarity S o and the region-level similarity S r The balance coefficient of

[0080]

[0081]

[0082] Among them, the MAE indicator is used to simultaneously measure the local and overall similarity between two binary images. The indicator is used to measure the final prediction graph S p The pixel-level error between the true value map G and the true value map G, w and h represent the width and height of the true value map G, (x, y) represents the coordinates of each pixel in the true value map G, and the symbol is an enhanced alignment matrix. The present invention predicts the image Sp According to the threshold value 0 to 255, a set of binary images are obtained, thereby obtaining a set of score.

[0083] Specifically, the experiment of this invention follows the same principle as the other methods, using randomly selected images from Kvasir and CVC-ClinicDB for training, but this paper uses the same training data for fair comparison, which has been extracted from Kvasir and CVC-ClinicDB, containing a total of 1450 images. For the benchmark datasets, this paper uses five different datasets, namely ETIS

[48] , CVC-ClinicDB / CVC-612, CVC-ColonDB, CVC-300 and Kvasir. The first four are standard evaluation datasets, and the last one is the largest challenging dataset recently released. The five polyp datasets will be introduced in detail below.

[0084] ETIS: This dataset contains 196 images collected from 34 colonoscopy videos. The image size is 1225×966, which is the largest among other datasets, making it more challenging.

[0085] CVC-ClinicDB: This dataset is also known as CVC-612 and contains 612 images from 25 colonoscopy videos. The image size is 384×288, of which 62 images are used for testing and the rest are used for training.

[0086] CVC-ColonDB: This dataset is collected from 15 different colonoscopy sequences, and 380 images are sampled from these sequences. The image size is 574×500, and all images are used as the test set data of the present invention.

[0087] CVC-300: This dataset is a test dataset from EndoScene, which contains 912 images from 44 colonoscopy sequences of 36 patients, with an image size of 574500. EndoScene includes Endosece-CVC300 and Endosece-CVC612. Because part of the CVC-612 dataset may be used for training, this experiment only uses the Endosece-CVC300 test set, which has a total of 60 samples.

[0088] Kvasir: This dataset is a challenging dataset recently released. It contains 1,000 images selected from a subclass (polyp) of the Kvasir dataset, consisting of polyp images and corresponding annotations. Unlike other datasets, the image sizes range from 332×487 to 1920×1072, and the sizes and shapes of the polyps appearing in the images are also different, including 700 large polyps larger than 160×160, 48 small polyps smaller than 64×64, and 323 medium-sized polyps within the size range. 900 images are used for training and 100 images are used for testing. The specific information of the above five polyp datasets is shown in Table 1 below.

[0089] Table 1 Polyp dataset information

[0090] Dataset Number of samples Sample size use ETIS 196 1255×966 Test set(196) CVC-ClinicDB 612 384×288 Training set (550) Test set (62) CVC-ColonDB 380 574×500 Test set (380) CVC-300 912 574×500 Test set (60) Kvasir 1000 332×487～1920×1072 Training set (900) Test set (100)

[0091] Compared with the prior art, the polyp segmentation method based on the lightweight network model and the reverse attention module provided by the present invention has the following advantages:

[0092] First, the present invention conducts some experiments on the Kvasir dataset to verify the performance of each key module of MobileRaNet (including CA, PA_RFB, RA), and the experimental results are shown in Table 2 below.

[0093] Table 2 Ablation experiments on the Kvasir dataset

[0094]

[0095] 1) Effectiveness of CA: In order to explore the effectiveness of the CA module, the present invention obtains two baseline models: No. 1 (backbone network only) & No. 2 (backbone network + CA) in Table 2. The results clearly show the effectiveness of the CA module in improving performance.

[0096] 2) Effectiveness of RA: The present invention studies the importance of the RA module. As can be seen from Table 2, No. 3 (backbone network + RA) improves the performance of the backbone network in terms of major indicators, which indicates that the introduction of the RA module can enable the model of the present invention to accurately distinguish the true target area.

[0097] 3) Effectiveness of PA_RFB: The present invention verifies the effectiveness of PA_RFB. It is observed from Table 2 that No. 4 improves performance in various aspects compared to the backbone network, which demonstrates the effectiveness of the PA_RFB module.

[0098] 4) Effectiveness of the combination of two modules: The present invention verifies the effectiveness of the combination of RA&CA, RA&PA_RFB, and CA&PA_RFB modules. As shown in Table 2, the performance of the module combination is better than that of a single module (i.e., No. 5 to No. 7) in important indicators. These improvements show that the combination of modules has excellent performance in the network model and verify the effectiveness of the combined module.

[0099] 5) Effectiveness of CA&RA&PA_RFB: Finally, the present invention studies the importance of the combination of three modules. From the results in the table (No.8 vs. No.1 to No.7), it can be clearly seen that the performance advantage of the combination of three modules over other single modules and the combination of two modules. This shows that the combination of three modules can effectively improve the segmentation effect of the model and achieve the best effect.

[0100] Second, in order to compare the performance of polyp area segmentation, four most typical methods, U-Net, U-Net+, SFA and PraNet, are used for comparison on five different polyp datasets and six evaluation indicators to verify the lightweight and high efficiency of the network. The quantitative results are shown in Table 3 below.

[0101] Table 3 Comparison of experimental results of different methods on multiple polyp datasets

[0102]

[0103] It can be seen in the Kvasir dataset that the MeanDice of MobileRaNet is 9.4% higher than that of U-Net and 1.4% higher than that of PraNet. Other indicators also have obvious advantages over other methods. In the ETIS, CVC-ClinicDB, CVC-ColonDB, and CVC-300 datasets, it can be clearly observed from the results in Table 3 that most of the indicators of MobileRaNet are significantly better than the other four model methods. In the complex dataset ETIS, the MeanDice of MobileRaNet is 26.7% higher than that of U-Net and 3.7% higher than that of PraNet. The present invention attributes the performance improvement to the effective combination of the three major modules in the model, which provide robust feature expressions and further improve the segmentation performance of the network model.

[0104] Third, the present invention verifies the lightweightness of the method of the present invention by calculating the FLOPs and Params of the network model. Figure 6 and Figure 7As shown in Figure 3, the number of parameters of U-Net++ is about 5.24 times that of MobileRaNet, and its computational complexity is about 122 times that of MobileRaNet. The number of parameters and computational complexity of U-Net are about 4.9 times and 60.8 times that of MobileRaNet, respectively, while the number of parameters and computational complexity of PraNet are about 4.3 times and 6 times that of MobileRaNet, respectively. In short, both the number of parameters and computational complexity of MobileRaNet are the smallest, and it can be seen from Table 3 that the performance of MobileRaNet is the best in most of the five polyp datasets compared with other typical methods.

[0105] Fourth, the present invention also visualizes the polyp segmentation results, such as Figure 8 As shown. Figure 8 It can be seen from the visualization results shown that the MobileRaNet of the present invention is significantly better than the segmentation results of other methods. Specifically, the segmentation results generated by the MobileRaNet of the present invention are close to the true value map, and there are fewer incorrectly segmented tissue areas. On the contrary, the results given by U-Net are not satisfactory because there are a large number of incorrectly segmented areas. Although the U-Net++ and PraNet models have improved the results, the performance is still not ideal. The success of MobileRaNet is attributed to the coarse-to-fine segmentation strategy proposed by the present invention, that is, the parallel partial decoders aggregate high-level features, and through the parallel axial receptive field modules, the global dependency representation is improved, the polyp segmentation area is roughly located, and then multiple reverse attention modules are used for refined segmentation. This strategy imitates how clinicians segment polyp segmentation areas from two-dimensional CT images, thereby achieving good performance, so the target segmentation extracted in the results obtained by the method of the present invention is more complete and accurate.

[0106] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solution of the present invention can be modified or replaced by equivalents without departing from the purpose and scope of the technical solution of the present invention, which should be included in the scope of the claims of the present invention.

Claims

1. A polyp segmentation method based on a lightweight network model and a reverse attention module, characterized in that: The following steps are involved: S1. Improve the lightweight network model MobileNetV3 using the coordinated attention module. Specifically, remove the last classification network part of the lightweight network model MobileNetV3 and replace the SENet module with the coordinated attention module to obtain the lightweight network segmentation model CaNet as the backbone network. S2, passing the high-level features output by the backbone network in parallel into the parallel axial receptive field module, which is composed of a parallel axial attention module and a receptive field module, and is used to perform additional global refinement at different scales and connect the outputs, extracting global dependencies and local representations without losing detail information; S3, upsampling the feature maps output by the parallel axial receptive field modules to a uniform size by different multiples, aggregating them by point-by-point multiplication, and then performing bilinear interpolation to preliminarily obtain an unstructured global mapping map to provide rough positioning information of the polyp; S4. The high-level side features output in parallel by the global mapping image and the parallel axial receptive field module are sent to multiple reverse attention modules constructed in a cascade manner. The reverse attention module erases the existing estimated polyp area from the high-level output features, thereby sequentially mining complementary areas and details to obtain a more refined polyp segmentation result. The output of the last reverse attention module is sent to the Sigmoid activation function to generate the final prediction map of the polyp segmentation area.

2. The polyp segmentation method based on a lightweight network model and a reverse attention module according to claim 1 is characterized in that: In step S1, the coordinated attention module first uses pooling kernels of size (H, 1) and (1, W) to encode each channel input feature map along the horizontal coordinate direction and the vertical coordinate direction. Then, the two transformations perform feature aggregation along the two spatial directions and return a pair of direction-aware attention maps. Each direction-aware attention map captures the long-range dependency of the input feature map along one spatial direction and preserves the precise position information of the other spatial direction. Then, regularization and nonlinear functions are performed, and then the feature map is convolved and sigmoid function is performed, and finally the attention weight is output.

3. The polyp segmentation method based on a lightweight network model and a reverse attention module according to claim 1 is characterized in that: The high-level features output by the backbone network in step S2 are transmitted in parallel to three parallel axial receptive field modules. Each parallel axial receptive field module first replaces the 3×3 convolution layer with 1×3 and 3×1 convolution layers, replaces the 5×5 convolution layer with 1×5 and 5×1 convolution layers, and replaces the 7×7 convolution layer with 1×7 and 7×1 convolution layers, and then passes through the dilated convolution modules with dilated rates of 3, 5, and 7 respectively, and adds parallel axial attention modules in parallel to perform additional global refinement on each scale and connect the outputs, and finally aggregates and adds with the 1×1 convolution layer, and then passes through the 3×3 convolution layer and the 1×1 convolution layer to output the feature map.

4. The polyp segmentation method based on a lightweight network model and a reverse attention module according to claim 3 is characterized in that: The parallel axial attention module adopts an axial attention strategy and uses the input feature map to calculate the non-local operations of the horizontal and vertical axes in parallel. Specifically, in the horizontal and vertical axes, two 1×1 convolutions are performed followed by matrix multiplication transformation, and then a 1×1 convolution is performed for matrix multiplication transformation. Finally, the outputs of the two axes are added point by point to output the feature map.

5. The polyp segmentation method based on a lightweight network model and a reverse attention module according to claim 3 is characterized in that: In the step S4, three reverse attention modules are specifically constructed in a cascade manner. The downsampled global mapping image and the high-level side features output by the first parallel axial receptive field module are sent to the first reverse attention module to obtain the first reverse attention feature. The output after adding the first reverse attention feature and the downsampled global mapping image is sent to the second reverse attention module with the high-level side features output by the second parallel axial receptive field module to obtain the second reverse attention feature. On the other hand, it is added to the second reverse attention feature after upsampling. The added output is sent to the third reverse attention module with the high-level side features output by the third parallel axial receptive field module to obtain the third reverse attention feature. On the other hand, it is added to the third reverse attention feature after upsampling. The added output is sent to the Sigmoid activation function to generate the final prediction map of the polyp segmentation area.

6. The polyp segmentation method based on a lightweight network model and a reverse attention module according to claim 5, characterized in that: Each reverse attention module is connected by the corresponding parallel axial receptive field module output high-level side features f i With the feature map S i The reverse attention weight A obtained after Sigmoid activation function and inversion i Multiply to obtain the output reverse attention feature R i , the specific description is as follows: R i =f i e A i Among them, i = 3, 4, 5, e represents multiplication, reverse attention weight A i Described as: Where P(·) represents the upsampling operation, σ(·) is the Sigmoid activation function, is the reverse operator that subtracts the input from the all-ones matrix E.

7. The polyp segmentation method based on a lightweight network model and a reverse attention module according to claim 1, characterized in that: The output of each reverse attention module and the global map use the deep supervision overall loss function as the optimization target. The overall loss function is defined as follows: Among them, G is the true value map, S g is the global map, The global map is upsampled, S i is the output of the reverse attention module, The output of the reverse attention module is upsampled, and L is the combination of the weighted IoU loss function and the weighted binary cross entropy BCE loss function, which is described as follows: in, represents the weighted IoU loss based on global constraints, represents the weighted binary cross entropy BCE loss based on local constraints; and They are defined as follows: Among them, i∈I refers to the pixel point in the predicted value and the true value, y represents the true value, represents the predicted value, and w is the weighted value.

8. The polyp segmentation method based on a lightweight network model and a reverse attention module according to claim 1, characterized in that: The method also includes using a weighted Dice metric Structural index S α , mean absolute error MAE, enhanced alignment index Indicators to measure the final prediction graph S p The similarity and dissimilarity between the graph and the true value graph G, each index is specifically expressed as: Among them, ω is the weight assigned to each category based on the number of samples in each category; Precision refers to the accuracy, which means the probability of correctly predicting as a positive sample among the samples predicted as positive samples in the prediction results; Recall refers to the recall rate, which means the probability of being correctly predicted as a positive sample in the positive samples of the original samples; S α =(1-α)*S o (S p ,G)+α*S r (S p ,G) Among them, α is used to control the object level similarity S o and the region-level similarity S r The balance coefficient of Among them, w and h represent the width and height of the truth map G, (x, y) represents the coordinates of each pixel in the truth map G, and the symbol is the augmented alignment matrix.

Images