Polyp segmentation method based on joint attention U-shaped network and multi-scale feature fusion

Abstract

The present invention provides a polyp segmentation method combining attention U-shaped network and multi-scale feature fusion, including selecting U-Net as the backbone network, adding attention gates at the end of the jump connection of the corresponding layer in the encoder and decoder stages of the U-shaped backbone network structure to suppress unimportant features while strengthening important information; then fusing different scale features to obtain a rich global semantic information feature map, and after decoding the feature map, a global mapping map is obtained as the initial guide area for subsequent steps; then parallel high-level features are passed into the receptive field module to enhance the network depth representation; then, under the guidance of the global mapping map, they are sent to multiple reverse attention modules constructed in a cascade manner to better mine target area features and boundary clues; finally, the polyp target area and boundary information are refined by refining the residual module to obtain a more high-performance polyp segmentation result. The present invention has more accurate and excellent segmentation performance for polyp image data sets.

Description

Technical Field

[0001] The present invention relates to the technical field of polyp segmentation, and in particular to a polyp segmentation method combining a joint attention U-shaped network and multi-scale feature fusion. Background Art

[0002] Colorectal cancer is the third most common cancer and the second most common cause of death worldwide, according to the 2020 Global Colorectal Cancer Data Report. Polyps are abnormal tissue growths on the surface of the body that can be found in the colon, rectum, stomach, and even the throat. In most cases, polyps are benign, meaning they do not represent a disease or are malignant, but over time, some polyps in the colon or rectum can turn into colorectal cancer.

[0003] Regarding colorectal cancer, early screening and resection are crucial. If colorectal cancer patients are diagnosed and treated early, their 5-year survival rate will increase to more than 90%. On the contrary, when the cancer develops to the late stage, the patient's survival rate will be much lower, even as low as 10%. Therefore, the best way to prevent colon cancer is to identify and remove polyps before they become cancer. Colonoscopy is currently the most commonly used examination method, but this process requires manual operation, is expensive, and has a high misdiagnosis rate. Therefore, accurate polyp segmentation technology is very important and urgently needed in clinical practice.

[0004] Traditional polyp segmentation methods mainly rely on manually extracted features such as color, texture, shape, appearance, and a combination of these features. Although these methods have made some progress in clinical practice, they are still limited by box-level prediction results and cannot accurately capture the shape and contour of polyps.

[0005] Compared with traditional polyp segmentation methods, deep convolutional neural networks have more outstanding advantages in the performance of medical image segmentation. In 2017, Brandao et al. used a fully convolutional neural network (FCN) based on a pre-trained model to identify and segment polyps. In 2018, Akbari et al. improved FCN to improve the accuracy of polyp segmentation. In addition, many encoder-decoder network architecture methods have also achieved remarkable results in polyp segmentation tasks. In 2015, Ronneberger et al. proposed U-Net, which uses an encoder-decoder architecture to segment biomedical images with excellent performance. Inspired by U-Net, variants such as U-Net++, ResUNet and ResUNet++ have been applied to polyp segmentation tasks and achieved good segmentation results. However, the above methods rely on low-resolution features to generate the final prediction, resulting in rough segmentation results and blurred boundaries.

[0006] In response to the problem of ignoring regional boundary constraints, in 2019, Murugesan et al. simultaneously utilized regional and boundary information in polyp segmentation to achieve better results. In 2019, Fang et al. proposed a three-step selective feature aggregation network with both regional and boundary constraints for polyp segmentation. This method explicitly considers the dependency between regions and boundaries and obtains good results through additional edge supervision, but its computational efficiency is low and there is a risk of overfitting. In 2020, Fan et al. proposed a parallel local high-level feature decoding scheme, which uses the reverse attention mechanism to better find target region features and boundary details. In 2021, Huang et al. proposed a HarDNet-MSEG convolutional neural network for polyp segmentation, which can quickly locate the target region and improve the accuracy of polyp segmentation. However, both methods discard some low-level semantic feature information and lose some detail information. Summary of the invention

[0007] In view of the technical problem that the existing polyp segmentation methods focus on segmenting the entire area of ​​the polyp while ignoring the regional boundary constraints, resulting in unsatisfactory polyp segmentation performance, the present invention provides a polyp segmentation method that combines a joint attention U-shaped network and multi-scale feature fusion.

[0008] In order to solve the above technical problems, the present invention adopts the following technical solutions:

[0009] The polyp segmentation method based on the joint attention U-shaped network and multi-scale feature fusion includes the following steps:

[0010] S1. Select U-Net as the backbone network and add an attention gate at the end of the jump connection between the encoder stage and the decoder stage of the U-shaped backbone network structure to eliminate the semantic ambiguity caused by the jump connection, suppress the feature response of irrelevant areas in the input image and automatically focus on important features;

[0011] S2, the backbone network upsamples the different scale features in the input image to a uniform size, performs multi-scale semantic feature information fusion, and obtains a rich global semantic information feature map. The feature map is decoded by the decoder to obtain a global map, which is used as the initial guide area for the subsequent step S4;

[0012] S3, passing the three parallel high-level feature information output by the backbone network into the corresponding three receptive field modules to enhance the network depth representation;

[0013] S4, the high-level features outputted in parallel by the global map and the receptive field module are sent to multiple reverse attention modules constructed in a cascade manner. The reverse attention module erases the existing estimated polyp area in the high-level output features in a foreground erasing manner, so as to better mine the complementary area and boundary detail information to obtain discriminative information. The output of the last reverse attention module is sent to the Sigmoid activation function to generate a coarse polyp segmentation result map.

[0014] S5. Input the polyp rough segmentation result map into the refined residual module, and refine the polyp target area and boundary information by learning the residual between the rough result map and the true value to obtain a more comprehensive and refined polyp refined segmentation result map.

[0015] Furthermore, in step S3, each receptive field module first replaces the 3×3 convolution layer with a 1×3 and 3×1 convolution layer, replaces the 5×5 convolution layer with a 1×5 and 5×1 convolution layer, and replaces the 7×7 convolution layer with a 1×7 and 7×1 convolution layer, and then passes through a dilated convolution module with dilated rates of 3, 5, and 7, respectively, and finally aggregates and adds with the 1×1 convolution layer, and then passes through a 3×3 convolution layer and a 1×1 convolution layer to output a feature map.

[0016] Furthermore, in step S4, three reverse attention modules are specifically constructed in a cascade manner, and the downsampled global mapping image and the high-level features output by the first receptive field module are sent to the first reverse attention module to obtain the first reverse attention feature. The output after adding the first reverse attention feature and the downsampled global mapping image is sent to the second reverse attention module with the high-level features output by the second receptive field module to obtain the second reverse attention feature, and on the other hand, it is added to the second reverse attention feature after upsampling. The added output is sent to the third reverse attention module with the high-level features output by the third receptive field module to obtain the third reverse attention feature, and on the other hand, it is added to the third reverse attention feature after upsampling. The added output is sent to the Sigmoid activation function to generate a coarse polyp segmentation result image.

[0017] Furthermore, each reverse attention module outputs the high-level features f i With the feature map S i The reverse attention weight A obtained after Sigmoid activation function and inversion i Multiply to obtain the output reverse attention feature R i , the specific description is as follows:

[0018] R i =f i ⊙A i

[0019] Among them, i = 3, 4, 5, ⊙ represents multiplication, reverse attention weight A i Described as:

[0020]

[0021] Where P(·) represents the upsampling operation, σ(·) is the Sigmoid activation function, is the reverse operator that subtracts the input from the all-ones matrix E.

[0022] Furthermore, the refined residual module includes a sequentially connected input layer, an encoder, a skip connection, a decoder and an output layer. Each stage of the encoder and the decoder has a convolution layer, each convolution layer is followed by a regularization layer, maximum pooling is used for downsampling in the encoder stage, and bilinear interpolation is used for upsampling in the decoder stage; at the same time, there are skip connections between each stage of the encoder and the decoder.

[0023] Furthermore, the output of each reverse attention module and the global map adopt the deep supervision overall loss function as the optimization target, and the overall loss function is defined as follows:

[0024]

[0025] Among them, G is the true value map, S g is the global map, The global map is upsampled, S i is the output of the reverse attention module, The output of the reverse attention module is upsampled, and L is the combination of the weighted IoU loss function and the weighted binary cross entropy BCE loss function, which is described as follows:

[0026]

[0027] in, represents the weighted IoU loss based on global constraints, represents the weighted binary cross entropy BCE loss based on local constraints; and They are defined as follows:

[0028]

[0029]

[0030] Among them, i∈I refers to the pixel point in the predicted value and the true value, y represents the true value, represents the predicted value, and w is the weighted value.

[0031] Furthermore, the method also includes using a weighted Dice metric Structural index S α , mean absolute error MAE, enhanced alignment index Indicators are used to measure the polyp fine segmentation results Figure S p The similarity and dissimilarity between the graph and the true value graph G, each index is specifically expressed as:

[0032]

[0033] Among them, ω is the weight assigned to each category based on the number of samples in each category; Precision refers to the accuracy, which means the probability of correctly predicting as a positive sample among the samples predicted as positive samples in the prediction results; Recall refers to the recall rate, which means the probability of being correctly predicted as a positive sample in the positive samples of the original samples;

[0034] S α =(1-α)*S o (S p ,G)+α*S r (S p ,G)

[0035] Among them, α is used to control the object level similarity So and the region-level similarity S r The balance coefficient of

[0036]

[0037]

[0038] Among them, w and h represent the width and height of the truth map G, (x, y) represents the coordinates of each pixel in the truth map G, and the symbol is the augmented alignment matrix.

[0039] Compared with the prior art, the polyp segmentation method of the joint attention U-shaped network and multi-scale feature fusion provided by the present invention has the following advantages:

[0040] 1. The present invention mainly obtains polyp segmentation results with better performance by combining attention mechanism and multi-scale feature fusion.

[0041] 2. In the present invention, an attention gate is added at the end of the jump connection of the U-shaped network structure, i.e., the U-Net backbone network, to eliminate the semantic ambiguity caused by the jump connection, automatically focus on significant features, and fuse information of different scales through the receptive field module to enhance the network feature representation.

[0042] 3. The present invention uses a foreground erasing method to better mine the polyp target area and boundary information through the reverse attention mechanism of the output features; at the same time, the network is deeply supervised by using the true value image to reduce the gradient disappearance and solve the problem of slow network convergence.

[0043] 4. The present invention designs a refined residual module, which refines the rough segmentation result prediction map by utilizing a U-shaped network structure, a jump connection and a residual structure, and by learning the residual between the rough result map and the true value, thereby obtaining a more comprehensive and refined result. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1 It is a flow principle block diagram of the polyp segmentation method of the joint attention U-shaped network and multi-scale feature fusion provided by the present invention.

[0045] Figure 2 The present invention provides Figure 1 Schematic diagram of the medium receptive field module structure.

[0046] Figure 3 The present invention provides Figure 1 Schematic diagram of the reverse attention module structure.

[0047] Figure 4 The present invention provides Figure 1 Schematic diagram of the structure of the refined residual module.

[0048] Figure 5 This is a comparison chart of qualitative results between the present invention and other typical model methods on five different polyp data sets. DETAILED DESCRIPTION

[0049] In order to make the technical means, creative features, objectives and effects achieved by the present invention easy to understand, the present invention is further explained below with reference to specific diagrams.

[0050] Please refer to Figure 1 As shown, the present invention provides a polyp segmentation method combining attention U-shaped network and multi-scale feature fusion, comprising the following steps:

[0051] S1. Select U-Net as the backbone network (Backbone), and add an attention gate at the end of the jump connection between the encoder stage and the decoder stage of the U-shaped backbone network structure to eliminate the semantic ambiguity caused by the jump connection, suppress the feature response of irrelevant areas in the input image, and automatically focus on important features. Specifically, the overall architecture of the network model of the U-Net backbone network is an encoder-decoder structure. By adding an attention gate at the end of the jump connection of the corresponding layer in the encoder stage and the decoder stage of the network model, it automatically learns to focus on important target areas. The attention gate can automatically focus on important features and suppress the feature response of irrelevant areas in the input image, and eliminate irrelevant semantics in the jump connection and the corresponding ambiguity; in addition, adding the attention gate to the convolutional neural network model can significantly enhance the model sensitivity and improve the detection accuracy with very little additional calculation.

[0052] S2. The backbone network upsamples the different scale features in the input image to a uniform size, performs multi-scale semantic feature information fusion, and obtains a rich global semantic information feature map. The feature map is decoded by the decoder to obtain the global mapping map S g , used to provide rough positioning information of polyps, the global map S g As the initial boot area of ​​the subsequent step S4.

[0053] S3: The three parallel high-level feature information output by the backbone network is passed into the corresponding three receptive field modules (RFB) to enhance the network depth representation.

[0054] S4. The high-level features outputted in parallel by the global map and the receptive field module are sent to multiple reverse attention modules (RA) constructed in a cascade manner. The reverse attention module erases the existing estimated polyp area in a foreground erasing manner in the high-level output features, thereby better mining the complementary area and boundary detail information to obtain discriminative information, making the polyp segmentation result more comprehensive. The output of the last reverse attention module is sent to the Sigmoid activation function to generate a rough polyp segmentation result map. It is worth noting that the existing estimates are sampled upward from a deeper layer, and are sent to multiple reverse attention modules under the guidance of the global map to establish a cyclic cooperation between the area and boundary information, and these reverse attention modules are constructed in a cascade manner.

[0055] S5. Input the rough polyp segmentation result map into the refined residual module (Refined Residual Module, RRM), refine the polyp target area and boundary information by learning the residual between the rough result map and the true value, and obtain a more comprehensive and refined polyp segmentation result map. Therefore, compared with the current typical segmentation network, the polyp segmentation method (MAR-UNet) provided by the present invention has more accurate and excellent segmentation performance for polyp image datasets.

[0056] As a specific embodiment, the background of the design of adding an attention gate to the backbone network is as follows: the U-Net backbone network uses an encoder-decoder structure. By adding a jump connection between the two stages of the encoder-decoder and using data enhancement methods, it still shows good competitiveness and excellent performance when there is less medical image data. The emergence of U-Net provides great help for medical image processing tasks. Therefore, the present invention selects U-Net as the backbone network, and automatically learns to focus on important target areas by adding an attention gate at the end of the jump connection of the corresponding layer in the encoder stage and the decoder stage of the network model. The attention gate can automatically focus on important features and suppress the feature response of irrelevant areas in the input image, and eliminate irrelevant semantics in the jump connection and the corresponding ambiguity. The attention gate is easily integrated into the model, and very little additional calculation can bring significant improvements in model sensitivity and accuracy.

[0057] As a specific embodiment, the background of the receptive field module design is as follows: Initially, the receptive field module design originated from the human visual system, which is composed of multiple parts with different receptive fields. In the successive processing stages of the visual pathway, the size of the receptive field increases. At each processing stage, it increases with the distance from the fixation point (eccentricity). The retinal ganglion cells located in the center of vision, the fovea, have the smallest receptive field, while those located in the visual periphery have the largest receptive field. Therefore, the size of the receptive field is directly proportional to the distance from the center of the retina. The large receptive field size of peripheral visual neurons explains the poor visual spatial resolution outside the fixation point. At different stages of the visual pathway, the receptive fields of neurons differ not only in size but also in structure. The complexity of the receptive field structure, like the receptive field size, increases in the successive stages of the visual pathway. Please refer to Figure 2 As shown in the figure, the feature map of the backbone network is forwarded to each receptive field path. In the step S3, each receptive field module first replaces the 3×3 convolution layer with a 1×3 and 3×1 convolution layer, replaces the 5×5 convolution layer with a 1×5 and 5×1 convolution layer, and replaces the 7×7 convolution layer with a 1×7 and 7×1 convolution layer to reduce the amount of calculation, and then passes through the dilated convolution modules with dilated rates of 3, 5, and 7 respectively, and finally aggregates and adds with the 1×1 convolution layer, and then passes through the 3×3 convolution layer and the 1×1 convolution layer to output the feature map.

[0058] As a specific example, please refer to Figure 1 As shown, in step S4, three reverse attention modules RA are specifically constructed in a cascade manner, and the downsampled global map S gThe high-level feature f5 output by the first receptive field module is sent to the first reverse attention module RA1 to obtain the first reverse attention feature R5. The output after adding the first reverse attention feature R5 and the down-sampled global mapping map is sent to the second reverse attention module RA2 with the high-level feature f4 output by the second receptive field module to obtain the second reverse attention feature R4. On the other hand, it is added to the second reverse attention feature R4 after upsampling. The added output is sent to the third reverse attention module RA3 with the high-level feature f3 output by the third receptive field module to obtain the third reverse attention feature R3. On the other hand, it is added to the third reverse attention feature R3 after upsampling. The added output is sent to the Sigmoid activation function to generate a coarse polyp segmentation result map. The three reverse attention modules described in this embodiment are constructed in a cascade manner, which essentially means that the output of the second reverse attention module RA2 depends on the output of the first reverse attention module RA1, and the output of the third reverse attention module RA3 depends on the output of the second reverse attention module RA2. Finally, the output of the last third reverse attention module RA3 is sent to the Sigmoid activation function to generate a rough polyp segmentation result map.

[0059] As a specific embodiment, the design background of the reverse attention module RA of the present invention is as follows: In clinical practice, doctors first roughly locate the polyp area, and then carefully examine the local tissue to accurately mark the polyp. Referring to their approach, a global map S with a relatively coarse structure is first generated by decoding features of different scales. g , used to provide rough location information of polyps; secondly, a progressive framework is used as a fine annotator to gradually mine distinguishable polyp regions in a foreground erasing manner. Specifically, compared with the existing method of simply aggregating all feature layers, the present invention chooses to erase the estimated polyp regions from high-level output features, thereby sequentially mining complementary regions and detail information, and the current prediction results are obtained by upsampling the information of deeper network layers.

[0060] As a specific example, please refer to Figure 3 As shown in the figure, each reverse attention module outputs the high-level features f i With the feature map S i The reverse attention weight A obtained after Sigmoid activation function and inversion i Multiply to obtain the output reverse attention feature R i , the specific description is as follows:

[0061] R i =f i ⊙A i

[0062] Among them, i = 3, 4, 5, ⊙ represents multiplication, reverse attention weight A i The salient object detection task has been widely used in the field of computer vision research and is described as:

[0063]

[0064] Where P(·) represents the upsampling operation, σ(·) is the Sigmoid activation function, is the reverse operator that subtracts the input from the all-ones matrix E. Figure 3 The details of the process are shown, and the erasing mechanism driven by reverse attention can eventually refine the imprecise and rough estimation into an accurate and complete edge prediction map.

[0065] As a specific example, please refer to Figure 4 As shown, the rough segmentation result of the polyp through the reverse attention module is input into the refined residual module, and the refined residual module includes a sequentially connected input layer, an encoder, a skip connection, a decoder and an output layer. Each stage of the encoder and the decoder has a convolution layer, and each convolution layer is followed by a regularization layer. The encoder stage uses maximum pooling for downsampling, and the decoder stage uses bilinear interpolation for upsampling; at the same time, there are skip connections between the various stages of the encoder and the decoder to solve the problem of gradient disappearance as the number of network layers gradually deepens, and at the same time, it helps the back propagation of the gradient and speeds up the training process. This refined residual module adopts an encoder-decoder architecture similar to U-Net, and refines the network's polyp rough segmentation results by using a U-shaped network structure, a skip connection and a residual structure. The rough segmentation result prediction map is refined by learning the residual between the rough result map and the true value, and a more comprehensive and refined polyp refined segmentation result map is obtained, that is, a more high-performance polyp segmentation result is obtained.

[0066] As a specific embodiment, the loss function L of the present invention is designed as a combination of a weighted IoU loss function and a weighted binary cross entropy BCE loss function, which is specifically described as follows:

[0067]

[0068] in, represents the weighted IoU loss based on global constraints, represents the weighted binary cross entropy BCE loss based on local (pixel-level) constraints; and They are defined as follows:

[0069]

[0070]

[0071] Among them, i∈I refers to the pixel point in the predicted value and the true value, y represents the true value, represents the predicted value, and w is the weighted value. Unlike the standard IoU loss currently widely used in segmentation tasks, the weighted IoU loss highlights the importance of difficult sample pixels by increasing their weights. In addition, compared with the standard BCE loss function, More attention is paid to difficult sample pixels rather than allocating equal weights to all pixels. At the same time, the present invention uses the outputs of the three reverse attention modules (i.e., S3, S4, S5) and the global map S g A deep supervision strategy is adopted, that is, the output of each reverse attention module and the global map use the deep supervision overall loss function as the optimization target, and the maps are upsampled one by one (for example ) to the same size as the true value graph G, so that the overall loss function of the proposed MAR-UNet is defined as follows:

[0072]

[0073] Among them, G is the true value map, S g is the global map, The global map is upsampled, S i is the output of the reverse attention module, The output of the reverse attention module is upsampled.

[0074] As a specific embodiment, the present invention will mainly use Kvasir's official website as the basis for comparison, namely meanDice and mean IoU. At the same time, in order to have a deeper understanding of the model performance and compare with other methods, the present invention will also use four other metrics widely used in the field of target detection. Therefore, the polyp segmentation method of the present invention also includes the use of weighted Dice metric Structural index S α , mean absolute error MAE, enhanced alignment index Indicators are used to measure the polyp fine segmentation results Figure S p The similarity and dissimilarity between the graph and the true value graph G, each index is specifically expressed as:

[0075]

[0076] Among them, the The indicator is used to correct the problem of "equally important defects" in Dice. ω is the weight given to each category based on the number of samples in each category; Precision refers to the accuracy rate, which means the probability of correctly predicting as a positive sample in the prediction result; Recall refers to the recall rate, which means the probability of correctly predicting as a positive sample in the positive sample of the original sample;

[0077] S α =(1-α)*S o (S p ,G)+α*S r (S p ,G)

[0078] Among them, the S α The indicator is used to measure the structural similarity between the predicted image and the true image, and α is used to control the object level similarity S o and the region-level similarity S r The balance coefficient of

[0079]

[0080]

[0081] Among them, the MAE indicator is used to simultaneously measure the local and overall similarity between two binary images. Indicators used to measure the polyp rough segmentation results prediction Figure S p The pixel-level error between the true value map G and the true value map G, w and h represent the width and height of the true value map G, (x, y) represents the coordinates of each pixel in the true value map G, and the symbol is an enhanced alignment matrix. The present invention predicts the polyp rough segmentation result S p According to the threshold value 0 to 255, a set of binary images are obtained, thereby obtaining a set of score.

[0082] Specifically, the experiment of this invention follows the same principle as the other methods, using randomly selected images from Kvasir and CVC-ClinicDB for training, but this paper uses the same training data for fair comparison, which has been extracted from Kvasir and CVC-ClinicDB, containing a total of 1450 images. For the benchmark dataset, this paper uses five different datasets, namely ETIS, CVC-ClinicDB / CVC-612, CVC-ColonDB, CVC-300 and Kvasir. The first four are standard evaluation datasets, and the last one is the largest challenging dataset recently released. The following will introduce the next five polyp datasets in detail.

[0083] ETIS: This dataset contains 196 images collected from 34 colonoscopy videos. The image size is 1225×966, which is the largest among other datasets, making it more challenging.

[0084] CVC-ClinicDB: This dataset is also known as CVC-612 and contains 612 images from 25 colonoscopy videos. The image size is 384×288, of which 62 images are used for testing and the rest are used for training.

[0085] CVC-ColonDB: This dataset is collected from 15 different colonoscopy sequences, and 380 images are sampled from these sequences. The image size is 574×500, and all images are used as the test set data of the present invention.

[0086] CVC-300: This dataset is a test dataset from EndoScene, which contains 912 images of 44 colonoscopy sequences from 36 patients, with an image size of 574 × 500. EndoScene includes Endosece-CVC300 and Endosece-CVC612. Because part of the CVC-612 dataset may be used for training, this experiment only uses the Endosece-CVC300 test set, which has a total of 60 samples.

[0087] Kvasir: This dataset is a challenging dataset recently released. It contains 1,000 images selected from a subclass (polyp) of the Kvasir dataset, consisting of polyp images and corresponding annotations. Unlike other datasets, the image sizes range from 332×487 to 1920×1072, and the sizes and shapes of the polyps appearing in the images are also different, including 700 large polyps larger than 160×160, 48 small polyps smaller than 64×64, and 323 medium-sized polyps within the size range. 900 images are used for training and 100 images are used for testing. The specific information of the above five polyp datasets is shown in Table 1 below.

[0088] Table 1 Polyp dataset information

[0089]

[0090] Compared with the prior art, the polyp segmentation method of the joint attention U-shaped network and multi-scale feature fusion provided by the present invention has the following advantages:

[0091] First, the present invention conducts some experiments on the Kvasir dataset to verify the performance of each key module (including RFB, RA, RRM) of MAR-UNet and the combination of modules. The experimental results are shown in Table 2 below.

[0092] Table 2 Ablation experiments on the Kvasir dataset

[0093]

[0094]

[0095] 1) Effectiveness of RFB: In order to explore the effectiveness of the RFB module, the present invention obtains two baseline models: No. 1 (backbone network only) & No. 2 (backbone network + RFB) in Table 2. The results show the effectiveness of the RFB module in improving performance.

[0096] 2) Effectiveness of RA: The present invention studies the importance of the RA module. It can be seen from Table 2 that No. 3 (backbone network + RA) improves the performance of the backbone network in terms of major indicators, which indicates that the introduction of the RA component can enable the model of the present invention to obtain more discriminative information and mine the real target area and boundary information.

[0097] 3) Effectiveness of RRM: The present invention verifies the effectiveness of RRM. It can be observed from Table 2 that No. 4 has improved performance in all aspects compared to the backbone network, which shows the effectiveness of the RRM module and can further refine the network segmentation results and improve the segmentation accuracy.

[0098] 4) Effectiveness of the combination of two modules: The present invention verifies the effectiveness of the combination of RA&RFB, RA&RRM, and RFB&RRM modules. As shown in Table 2, the performance of the module combination is better than that of a single module (i.e., No. 5 to No. 7) in important indicators. These improvements show that the combination of modules has excellent performance in the network model and verify the effectiveness of the combined module.

[0099] 5) Effectiveness of RFB&RA&RRM: Finally, the present invention experiments the effectiveness of the combination of three modules. From the results in Table 2 (No. 8 vs. No. 1 to No. 7), it can be clearly seen that the combination of three modules has advantages over other modules and the combination of two modules. This shows that the combination of three modules can effectively improve the segmentation effect of the model and achieve the best effect.

[0100] Second, in order to compare the performance of polyp region segmentation, four most typical methods, U-Net, U-Net+, SFA and PraNet, are compared on five different polyp datasets and six evaluation indicators to verify the high efficiency of the network. The quantitative results are shown in Table 3 below.

[0101] Table 3 Comparison of experimental results of different methods on multiple polyp datasets

[0102]

[0103] It can be seen in the Kvasir dataset that the MeanDice of MAR-UNet is 8.5% higher than that of U-Net and 0.5% higher than that of PraNet. Other indicators also have obvious advantages over other methods. In the ETIS, CVC-ClinicDB, CVC-ColonDB, and CVC-300 datasets, it can be clearly observed from the results in Table 3 that most of the indicators of MAR-UNet are significantly better than the other four model methods. In the complex dataset ETIS, the MeanDice of MAR-UNet is 26.3% higher than that of U-Net and 3.3% higher than that of PraNet. The present invention attributes the performance improvement to the effective combination of the three major modules in the model, which provide robust feature expressions and further improve the segmentation performance of the network model.

[0104] Third, the present invention also visualizes the polyp segmentation results. For details, please refer to Figure 5 As shown. From the visualization results of the figure, it can be seen that the MAR-UNet of the present invention is significantly better than the segmentation results of other methods. Specifically, the segmentation results they produce are close to the true value map, and there are fewer incorrectly segmented tissue areas. On the contrary, the results given by U-Net are not satisfactory because there are a large number of incorrectly segmented areas. Although the U-Net++ and PraNet models have improved the results, the performance is still not ideal. The success of MAR-UNet is attributed to the coarse-to-fine segmentation strategy of the present invention, that is, by roughly locating the polyp area and then further refining the polyp segmentation results through a refined residual module. This strategy draws on the segmentation of polyp areas by clinicians from two-dimensional medical images, thereby achieving good performance. Therefore, in comparison, the target segmentation extracted in the results obtained by the method proposed by the present invention is more complete and accurate.

[0105] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solution of the present invention can be modified or replaced by equivalents without departing from the purpose and scope of the technical solution of the present invention, which should be included in the scope of the claims of the present invention.

Claims

1. A polyp segmentation method based on a joint attention U-shaped network and multi-scale feature fusion, characterized in that: The following steps are involved: S1. Select U-Net as the backbone network and add an attention gate at the end of the jump connection between the encoder stage and the decoder stage of the U-shaped backbone network structure to eliminate the semantic ambiguity caused by the jump connection, suppress the feature response of irrelevant areas in the input image and automatically focus on important features; S2, the backbone network upsamples the different scale features in the input image to a uniform size, performs multi-scale semantic feature information fusion, and obtains a rich global semantic information feature map. The feature map is decoded by the decoder to obtain a global map, which is used as the initial guide area for the subsequent step S4; S3, passing the three parallel high-level feature information output by the backbone network into the corresponding three receptive field modules to enhance the network depth representation; S4, the high-level features outputted in parallel by the global map and the receptive field module are sent to multiple reverse attention modules constructed in a cascade manner. The reverse attention module erases the existing estimated polyp area in the high-level output features in a foreground erasing manner, so as to better mine the complementary area and boundary detail information to obtain discriminative information. The output of the last reverse attention module is sent to the Sigmoid activation function to generate a coarse polyp segmentation result map. S5. Input the polyp rough segmentation result map into the refined residual module, and refine the polyp target area and boundary information by learning the residual between the rough result map and the true value to obtain a more comprehensive and refined polyp refined segmentation result map.

2. The polyp segmentation method based on the joint attention U-shaped network and multi-scale feature fusion according to claim 1 is characterized in that: In the step S3, each receptive field module first replaces the 3×3 convolution layer with a 1×3 and 3×1 convolution layer, replaces the 5×5 convolution layer with a 1×5 and 5×1 convolution layer, and replaces the 7×7 convolution layer with a 1×7 and 7×1 convolution layer, and then passes through the dilated convolution modules with dilation rates of 3, 5, and 7, respectively, and finally aggregates and adds with the 1×1 convolution layer, and then passes through the 3×3 convolution layer and the 1×1 convolution layer to output the feature map.

3. The polyp segmentation method of joint attention U-shaped network and multi-scale feature fusion according to claim 1 is characterized in that: In the step S4, three reverse attention modules are specifically constructed in a cascade manner. The downsampled global mapping image and the high-level features output by the first receptive field module are sent to the first reverse attention module to obtain the first reverse attention feature. The output after adding the first reverse attention feature and the downsampled global mapping image is sent to the second reverse attention module with the high-level features output by the second receptive field module to obtain the second reverse attention feature. On the other hand, after upsampling, the output after addition is sent to the third reverse attention module with the high-level features output by the third receptive field module to obtain the third reverse attention feature. On the other hand, after upsampling, the output after addition is added to the third reverse attention feature. The output after addition is sent to the Sigmoid activation function to generate a coarse polyp segmentation result image.

4. The polyp segmentation method of joint attention U-shaped network and multi-scale feature fusion according to claim 3 is characterized in that: Each reverse attention module outputs the high-level features f i With the feature map S i The reverse attention weight A obtained after Sigmoid activation function and inversion i Multiply to obtain the output reverse attention feature R i , the specific description is as follows: R i =f i ⊙A i Among them, i = 3, 4, 5, ⊙ represents multiplication, reverse attention weight A i Described as: Where P(·) represents the upsampling operation, σ(·) is the Sigmoid activation function, is the reverse operator that subtracts the input from the all-ones matrix E.

5. The polyp segmentation method of joint attention U-shaped network and multi-scale feature fusion according to claim 1 is characterized in that: The refined residual module includes a sequentially connected input layer, an encoder, a skip connection, a decoder and an output layer. Each stage of the encoder and the decoder has a convolution layer, and each convolution layer is followed by a regularization layer. Maximum pooling is used for downsampling in the encoder stage, and bilinear interpolation is used for upsampling in the decoder stage. At the same time, there are skip connections between each stage of the encoder and the decoder.

6. The polyp segmentation method of joint attention U-shaped network and multi-scale feature fusion according to claim 1 is characterized in that: The output of each reverse attention module and the global map use the deep supervision overall loss function as the optimization target. The overall loss function is defined as follows: Among them, G is the true value map, S g is the global map, The global map is upsampled, S i is the output of the reverse attention module, The output of the reverse attention module is upsampled, and L is the combination of the weighted IoU loss function and the weighted binary cross entropy BCE loss function, which is described as follows: in, represents the weighted IoU loss based on global constraints, represents the weighted binary cross entropy BCE loss based on local constraints; and They are defined as follows: Among them, i∈I refers to the pixel point in the predicted value and the true value, y represents the true value, represents the predicted value, and w is the weighted value.

7. The polyp segmentation method of joint attention U-shaped network and multi-scale feature fusion according to claim 1 is characterized in that: The method also includes using a weighted Dice metric Structural index S α , mean absolute error MAE, enhanced alignment index Indicators are used to measure the polyp fine segmentation results Figure S p The similarity and dissimilarity between the graph and the true value graph G, each index is specifically expressed as: Among them, ω is the weight assigned to each category based on the number of samples in each category; Precision refers to the accuracy, which means the probability of correctly predicting as a positive sample among the samples predicted as positive samples in the prediction results; Recall refers to the recall rate, which means the probability of being correctly predicted as a positive sample in the positive samples of the original samples; S α =(1-α)*S o (S p ,G)+α*S r (S p ,G) Among them, α is used to control the object level similarity S o and the region-level similarity S r The balance coefficient of Among them, w and h represent the width and height of the truth map G, (x, y) represents the coordinates of each pixel in the truth map G, and the symbol is the augmented alignment matrix.

Images