A colon polyp segmentation method with universality

Abstract

The application belongs to the field of intelligent medical computer-aided diagnosis and relates to a colon polyp segmentation method with universality. The method is based on a coding and decoding network structure, the encoder adopts a Mix Transformer structure with multi-scale output and efficient self-attention mechanism, and the decoder is a novel structure designed according to the multi-scale output of the encoder. The simple and effective convolutional neural network is used for the low-layer output of the encoder, then the multi-scale feature fusion module is used for the lower-layer, middle-layer and high-layer outputs of the encoder, then the results of the convolutional neural network and the multi-scale fusion are respectively input into the CBAM attention mechanism and the SE Block, and finally the results are input into the similarity aggregation module together, so as to enhance the feature representation. Compared with the existing deep learning method, the performance of the application is better on multiple data sets, and the application has good universality and effectiveness.

Description

TECHNICAL FIELD

[0001] The application belongs to the field of intelligent medical computer-aided diagnosis, and particularly relates to a colon polyp segmentation method with universality. TECHNICAL BACKGROUND

[0002] The statistical results of cancer patients in the world in 2018 show that the mortality rate and the confirmed rate of colorectal cancer rank the third among all cancers, and show a rising trend year by year. The number of patients with colorectal cancer in developed countries is the most, because the living standards in developed countries are relatively high, and people maintain high-fat and low-fiber dietary habits for a long time, resulting in insufficient intake of intestinal fiber. Looking at all age groups, the number of male patients in the middle and old age is the most, because the overall quality of the middle and old age groups changes due to less exercise, and the intestinal performance is also weakened and prone to diseases. Most men have bad habits such as smoking, drinking, high-salt diet, etc., so the incidence is higher. The incidence and mortality of colorectal cancer in China also show a rising trend, therefore, preventing colorectal cancer is a problem that people all over the world need to pay attention to.

[0003] Clinical treatment shows that the generation of colorectal cancer has a very close relationship with colon polyps. Colon polyps are benign (non-cancerous) tissues inside the colon, but if not treated in time, they may cause tissue lesions, and it takes about five to ten years from tissue lesions to colorectal cancer, but most patients are found to be in the middle and late stages of cancer. Because the early clinical symptoms of colorectal cancer are not obvious, as the disease progresses, symptoms such as dizziness, headache, weight loss, abdominal distension and abdominal pain begin to appear, but at this time the disease has basically reached the middle and late stages. Colon polyps are a precursor of colorectal cancer, and the earlier colon polyps are screened, the higher the survival rate of patients with colorectal cancer.

[0004] Nowadays, the key means of screening colon polyps is colonoscopy detection. Through colonoscopy detection, the shape, range and size of polyp tissue can be observed, and then part of the tissue is taken out for examination to determine whether the polyp tissue has lesions, so as to further diagnose and treat. However, due to the different sizes and shapes of colon polyps, and the low contrast between the color of polyps and the background, etc., some polyps may be missed during colonoscopy detection. The missed polyps may affect the diagnosis of the disease, and ultimately lead to the deterioration of the disease and death. Therefore, it is very meaningful to use computer-aided technology to improve the detection rate of colon polyps for the diagnosis and treatment of colorectal cancer.

[0005] At present, the use of deep learning methods in colon polyp segmentation tasks has been relatively widespread. The existing deep learning-based colon polyp segmentation methods are mainly divided into three categories: based on convolutional neural network, based on Transformer, and based on the combination of convolutional neural network and Transformer. The main task of these deep learning-based colon polyp segmentation models is still to improve the segmentation accuracy. In particular, the method based on the combination of convolutional neural network and Transformer shows certain advantages. However, the Transformers used in the current fusion methods are mostly based on the ViT (Vision Transformer) structure, which has two relatively large limitations: 1) output limitation: the Transformer based on ViT only outputs a feature map of one scale and is low resolution, so it may lead to the lack of rich contextual semantic information in the extracted feature map. 2) calculation limitation: the calculation amount of the self-attention mechanism in ViT has a quadratic relationship with the number of input tokens, which will produce huge calculation cost for some large resolution images. In addition, the parallel fusion method when combining convolutional neural network and Transformer will also increase the calculation amount of the overall model. SUMMARY

[0006] According to the deficiencies of the prior art, the present application provides a colon polyp segmentation method with universality. According to the appearance of colon polyps and the characteristics of deep learning methods, the present application proposes a colon polyp segmentation method based on an encoding and decoding structure, aiming to improve the segmentation accuracy of network models on multiple data sets and achieve better universality.

[0007] A colon polyp segmentation method with universality, comprising the following steps:

[0008] Step 1: Uniform training dataset resolution: The colonoscopy images in the dataset used by the present application have different resolutions, and the encoder Mix Transformer requires a fixed size, so the present application uniformly processes the training dataset before model training.

[0009] Step 2: Input the training set into the encoder of MixFormNet, and the encoder of MixFormNet adopts MixTransformer encoder. The specific operation is as follows: first, divide the image input into MixTransformer into patches with a size of 4x4, which is helpful for semantic segmentation using fine-grained patches. Then input these patches into the Mix Transformer encoder as input, and obtain multi-level feature maps where i = {1, 2, 3, 4}.

[0010] Third step: using the decoder to convert the multi-scale features obtained by the encoder into the target, the decoder models the output F of the MixTransformer i are processed respectively. Specifically, the low-level output F of the Mix Transformer is processed by the PreNet has a large resolution and contains rich geometric spatial information, but the semantic information is weak, so it is input into the PreNet for further processing by the convolution operation to output F1', and then F1' is sent to the CBAM attention mechanism to model the dependence between space and channel; for the lower, middle and high layer outputs F i , where i={2, 3, 4}, at this time the semantic and geometric spatial information of the three scale feature maps are different. Considering that the low-level feature map contains rich detail and edge information, and the high-level feature map contains detailed semantic information, the outputs of the three scales of the Mix Transformer are subjected to multi-scale feature fusion, aiming to obtain rich feature information and enhance the information representation capability. The multi-scale feature fusion output F' is then input into the SE Block module to establish the correlation between the feature map channels. Finally, the outputs of CBAM and SE Block are input into the similarity aggregation module, aiming to model the information relationship between the low-level features of CBAM output and the high-level features of SE Block output.

[0011] The processing process of the PreNet network is as follows: first, the input F1 is initially processed by means of Conv, Batch Normalization, ReLU and MaxPooling, and then input into 3 blocks stacked by convolution layers for local detail information extraction.

[0012] Fourth step: comprehensive decision: the output of the third step contains a main output prediction map P main and an auxiliary output prediction map P ass . In the training process, the sum of the prediction loss between P main and the real label and the prediction loss between P ass and the real label is used as the total loss of the network, and the parameters are updated reversely to optimize the model. In testing, the sum of P main and P ass is taken as the final segmentation result, which can optimize the final segmentation result as much as possible. P ass can not only optimize the overall model together with P main , but also supervise the intermediate output of multi-scale feature fusion.

[0013] Further, in order to increase the robustness of the model to different resolution images, the application adds a multi-scale training strategy during training, the scale range is {0.75, 1, 1.25} times the original input image.

[0014] Further, the parameters in the MixFormNet model are updated using the AdamW optimizer, the learning rate is set to 1e-4, the epoch is set to 200, and a set of network weights is saved every 10 epochs.

[0015] Further, in the comprehensive decision-making of the fourth step, the weighted binary cross-entropy loss function and the weighted IoU loss function are combined as the overall loss function L.

[0016] Further, the resolution in step 1 is unified. The application tests the size of the unified resolution of colonoscopy images. Since the resolution of the images in the data set is not uniform, too small resolution will cause information loss, and too large resolution will bring a large amount of calculation, so adjusting a suitable resolution is also very important for the performance of MixFormNet. Finally, the experimental verification shows that the unified resolution of 384x384 is more appropriate.

[0017] The advantages of the application include the following points:

[0018] (1) Model structure design: existing deep learning methods for segmenting colon polyps include convolutional neural network-based methods, Transformer-based methods, and convolutional neural network and Transformer combined methods. The convolutional neural network and Transformer combined method has certain advantages in performance, but most of them perform parallel computation first and then fusion, which greatly increases the computational load of the model. The application uses MixTransformer for feature extraction, then uses a convolutional neural network to further process the low-level feature map output by MixTransformer, and uses a multi-scale feature fusion module to process lower, middle and high layers. MixTransformer and subsequent processing form a kind of encoding and decoding structure, thereby avoiding the large computational problem caused by parallel computation.

[0019] (2) Construction of the encoder, in order to avoid the influence of single size output on the segmentation result, the application uses Mix Transformer with multi-scale output and efficient self-attention mechanism as the encoder of MixFormNet.

[0020] (3) Decoder construction: For the multi-scale output of the encoder, the application designs a new decoder structure. Since the low-layer feature map output by the encoder has large resolution and contains rich spatial information, but the semantic information is relatively lacking, a simple and effective convolutional neural network is designed in the decoder to act on the low-layer feature map, aiming to extract detailed detail information. For the lower layer, middle layer and high layer feature maps, since their contributions to feature information are different, a multi-scale feature fusion module is used in the decoder to fuse them, aiming to enhance feature representation.

[0021] (4) A large number of experiments show that the segmentation accuracy of the MixFormNet model designed in the application is significantly improved on more data sets compared with the related methods, thereby confirming the effectiveness and good universality of MixFormNet. BRIEF DESCRIPTION OF DRAWINGS

[0022] Figure 1 is the overall framework diagram of the algorithm in embodiment 1;

[0023] Figure 2 is the colonoscopy image preprocessing framework diagram in embodiment 1;

[0024] Figure 3 is the Mix Transformer structure diagram;

[0025] Figure 4 is the Prenet structure diagram;

[0026] Figure 5 is the multi-scale feature fusion module;

[0027] Figure 6 is the similarity aggregation module;

[0028] Figure 7 is the flowchart in embodiment 1;

[0029] Figure 8 is the comparison diagram of the prediction map in embodiment 1 and the prediction map of other methods;

[0030] Figure 9 is the overall framework diagram without the PreNet network in embodiment 2;

[0031] Figure 10 is the network overall framework diagram of the decoder as the UNet method decoder in embodiment 3;

[0032] Figure 11 is the network overall framework diagram of the decoder as the CaraNet method decoder in embodiment 3. DETAILED DESCRIPTION

[0033] The present invention will now be described in detail with reference to the accompanying drawings and embodiments:

[0034] like Figures 1-8 As shown, this invention implements a universally applicable method for segmenting colonic polyps. The method comprises three parts: an encoder, a decoder, and a comprehensive decision-making mechanism. The encoder utilizes a Mix Transformer structure with multi-scale output and an efficient self-attention mechanism to reduce the output and computational limitations inherent in single-scale output and self-attention. The decoder is a novel structure designed based on the multi-scale output of the Mix Transformer. A simple yet effective convolutional neural network is used for the low-level output of the encoder. A multi-scale feature fusion module then performs multi-scale fusion on the low-, middle-, and high-level feature maps of the encoder to enhance feature representation. The outputs of the convolutional neural network and the multi-scale features are then input into the CBAM attention mechanism and the SE Block, respectively. Finally, the results are jointly output to the similarity aggregation module. The comprehensive decision-making mechanism directly adds the outputs of the similarity aggregation module and the SE Block to obtain the final prediction map.

[0035] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0036] Example 1:

[0037] A universally applicable method for segmenting colonic polyps, characterized by the following steps:

[0038] (1) Standardizing the resolution of the training dataset: The colonoscopy images in the dataset used in this invention have different resolutions, while the input of the encoder Mix Transformer needs to be of a fixed size. Furthermore, if the resolutions vary, it will have a certain impact on subsequent processing. Therefore, this invention standardizes the training dataset to a resolution of 384×384 before model training.

[0039] (2) Input the training set into the MixFormNet network.

[0040] (3) Encoder: Given an input image size of... Due to the multi-scale training strategy in the training process, the image size in the training process is {288x288x3, 384x384x3, 480x480x3}, and the images of the three scales are processed respectively. First, the training image is divided into patches with a size of 4x4, which is different from the 16x16 size patch of ViT. The fine-grained patch is helpful for semantic segmentation. Then these patches are input into the Mix Transformer encoder to obtain multi-level feature maps F1(96x96x64) / (72x72x64) / (120x120x64), F2(36x36x128) / (48x48x128) / (60x60x128), F3(18x18x320) / (24x24x320) / (30x30x320), and F4=(9x9x512) / (12x12x512) / (15x15x512).

[0041] (4) Decoder: In the decoder, the low-level output F1 of the encoder is further extracted to obtain local detailed feature information, and output F1' is obtained. At the same time, F2, F3 and F4 are fused by using the multi-scale feature fusion module in the decoder to obtain F2'. Then F1' is input into the CBAM attention mechanism to increase the attention in the channel and spatial direction. F2' is input into the SEBlock, which is used to establish the correlation between the channels of the fused feature map. Finally, the outputs of CBAM and SEBlock are input into the similarity aggregation module, which is used to model the information relationship between the low-level features of CBAM output and the high-level features of SEBlock output.

[0042] (5) Loss function: The weighted binary cross-entropy loss function and the weighted IoU loss function are selected as the loss function of the MixFormNet network, which is because the weighted binary cross-entropy loss function and the weighted IoU loss function can realize the focused learning of the target pixels by increasing the weight of the target pixels, instead of the equal learning of all pixels with uniform weight.

[0043]

[0044] Where G is the real label, and O is the prediction output of the model.

[0045] (7) Comprehensive decision: There are two output prediction maps in the network structure of the present application, which include a main output prediction map P main and an auxiliary output prediction map P ass . In the training process, the prediction loss between P main and the real label and the prediction loss between P assThe sum of the prediction loss between the real label is taken as the total loss of the network, and the parameters are updated in reverse to optimize the model. In testing, the final P main and P ass are added as the final segmentation result, in this way, the final segmentation result can be optimized as much as possible.

[0046] Out=P main +P ass

[0047] Wherein, Out represents the output result after comprehensive decision.

[0048] The present application first makes MixFormNet trained on Kvasir and CVC-ClinicDB dataset when segmenting colon polyps, so that the model has certain learning ability, and then uses the test set for testing to verify the learning ability of the model. The data set used in embodiment 1 is shown in the test set in table 1:

[0049] Table 1: Data set information used in embodiment 1:

[0050]

[0051] Table 2 summarizes a series of methods compared with the present application. The comparison methods include: 6 colon polyp segmentation methods based on CNN mechanism, 1 colon polyp segmentation method based on Transformer, 2 colon polyp segmentation methods based on CNN and Transformer hybrid, and the specific information is shown in table 2:

[0052] Table 2 comparison method introduction

[0053]

[0054]

[0055] Table 3 summarizes the segmentation results of the present application in the test data set of embodiment 1. The present application uses two commonly used evaluation indexes of semantic segmentation: average Dice coefficient (mDice), average intersection over union (mIoU); In order to analyze the model more deeply, four commonly used evaluation indexes in the field of target detection are introduced: weighted F-measure S-measure (S α ), max-E-measure and the mean absolute error (MAE). In addition, the model complexity is measured by the total number of parameters params and the number of floating-point operations (FLOPs), which are used to measure the complexity of the model. According to the values of the indicators, the algorithm achieves excellent segmentation results on the five public data sets.

[0056] Table 3: Performance measurement values of the application on each data set in Example 1

[0057]

[0058] Tables 4-8 summarize the research results on colon polyp segmentation in recent years. As can be seen from the results in the table, our application obtains better segmentation accuracy on the five public data sets. The results on the CVC-ClinicDB and Kvasir data sets show that the performance indicator values of MixFormNet are better than the optimal values of the other three methods, achieving the best performance. Specifically, the mDice value of MixFormNet on the CVC-ClinicDB data set is 4.3% higher than that of the FCBFormer network, 2.4% higher than that of the ColonFormer, and 0.1% higher than that of the suboptimal HardNet-DFUS network; the mDice value of MixFormNet on the Kvasir data set is 0.2% higher than that of the FCBFormer network and 1.7% higher than that of the Fu-TransHNet network. Although the results of MixFormNet on other several data sets are not the best, at least they are comparable. The effectiveness of the model is not measured by the segmentation results on a certain data set, but by the segmentation performance on multiple data sets. Therefore, in summary, the application is effective and has good universality.

[0059] Table 4: CVC-ClinicDB data set measurement results

[0060]

[0061] Table 5: CVC-ColonDB data set measurement results

[0062]

[0063]

[0064] Table 6: CVC-EndoScene data set measurement results

[0065]

[0066] Table 7. Measurement Results of the ETIS-LaribPolypDB Dataset

[0067]

[0068]

[0069] Table 8. Kvasir Dataset Measurement Results

[0070]

[0071] Example 2

[0072] To verify that PreNet is helpful in improving the segmentation accuracy of colon polyps, this example compares the results of using PreNet without other methods. Figure 9 ) and possess PreNet ( Figure 1 A comparison was made. Finally, the segmentation results for each method were calculated and plotted in Table 9. Table 9 shows that the methods including PreNet outperformed the methods without PreNet on all datasets, thus proving the effectiveness of the PreNet proposed in this invention.

[0073] Table 9 PreNet Effectiveness

[0074]

[0075] Example 3

[0076] To verify that the decoder in this invention is helpful in improving the segmentation accuracy of colon polyps, this example compares the decoders in UNet and CaraNet applied to the multi-scale output of the Mix Transformer with the decoder proposed in this paper. The three sets of experiments in this section maintain the same network parameters except for the decoder, and the experimental results are shown in Table 10-11. Table 10-11 shows that the MixFormNet network generated by applying the novel decoder designed in this invention to the Mix Transformer encoder outperforms the other two methods in terms of segmentation accuracy and model complexity. This confirms that the decoder not only helps improve segmentation accuracy but also requires fewer computer resources compared to other methods.

[0077] Table 10 Comparison results of different decoder methods

[0078]

[0079] Table 11 Complexity Analysis of Different Decoder Methods

[0080]

Claims

1. A colon polyp segmentation method having universality, characterized by, The steps are as follows: Step 1: Uniform training data set resolution: the colonoscopy images in the adopted data set have different resolutions, and the encoder MixTransformer requires a fixed size, so the training data set is uniformly processed before model training; Step 2: Input the training set into the encoder of MixFormNet, and the encoder of MixFormNet adopts MixTransformer encoder; The specific operation is as follows: first, the image delivered into the MixTransformer is divided into patches with a size of 4*4, and it is helpful to use fine-grained patches for semantic segmentation; then, the patches are delivered as input into the MixTransformer encoder to obtain multi-level feature maps where i = {1, 2, 3, 4}. Step 3: Use the decoder to convert the feature information obtained by the encoder into the target, and the decoder is used to convert the output F of the Mix Transformer i are processed respectively; In particular, the low-level output of Mix Transformer have high resolution and contain rich geometric spatial information, but the semantic information is weak, so 3.1 F1 is input into the PreNet for further processing to get output F1 ′ and then F1 ′ is fed into the CBAM attention mechanism to get Z2; The processing process of the PreNet network is as follows: first, the initial processing of F1 input is performed by means of Conv, Batch Normalization, ReLU and MaxPooling, and then it is input into three blocks stacked by convolutional layers for local detail information extraction; 3.2 The lower layer, middle layer, high layer output F of Mix Transformer is input into the multi-scale feature fusion module to obtain the output feature map F i , wherein i={2, 3, 4} ′ , and then F ′ is input into the SE Block module to establish the correlation between the feature map channels, and finally the convolution processing is performed on the output Z1 of the SE Block to obtain the auxiliary output prediction map P ass ; 3.3 Then, Z1 and Z2 are input together into a similar aggregation module to obtain the main prediction map P main The purpose is to model the information relationship between the low-level features of the CBAM output and the high-level features of the SE Block output; Fourth step: comprehensive decision: the output of the third step contains a main output prediction map P main and an auxiliary output prediction map P ass ; during the training process, the sum of the prediction loss between P main and the real label and the prediction loss between P ass and the real label is used as the total loss of the network, and the parameters are updated reversely to optimize the model; during the testing, the sum of P main and P ass is taken as the final segmentation result, and in this way, the final segmentation result can be optimized as much as possible; P ass can not only optimize the overall model together with P main , but also supervise the intermediate output of multi-scale feature fusion.

2. The colon polyp segmentation method with universality according to claim 1, characterized in that: In step one, in order to increase the robustness of the model to different resolution images, a multi-scale training strategy is added during training, and the scale range is {0.75, 1, 1.25} times the original input image. 3.The colon polyp segmentation method with universality according to claim 1 or 2, characterized in that: In step one, the resolution is unified to 384x384.

4. The colon polyp segmentation method with universality according to any one of claims 1 or 2, characterized in that: In step two, the parameters in the MixFormNet model are updated by using the AdamW optimizer, the learning rate is set to 1e-4, the epoch is set to 200, and a set of network weights is saved every 10 epochs.

5. The colon polyp segmentation method with universality according to claim 3, characterized in that: In step two, the parameters in the MixFormNet model are updated by using the AdamW optimizer, the learning rate is set to 1e-4, the epoch is set to 200, and a set of network weights is saved every 10 epochs.

6. The colon polyp segmentation method with universality according to any one of claims 1, 2 or 5, characterized in that: In the comprehensive decision-making in the fourth step, the combination of weighted binary cross-entropy loss function and weighted IoU loss function is used as the overall loss function L.

7. The colon polyp segmentation method with universality according to claim 3, characterized in that: In the comprehensive decision-making in the fourth step, the combination of weighted binary cross-entropy loss function and weighted IoU loss function is used as the overall loss function L.

8. The colon polyp segmentation method with universality of claim 4, wherein: In the comprehensive decision-making in the fourth step, the combination of weighted binary cross-entropy loss function and weighted IoU loss function is used as the overall loss function L.

Images