A polyp image segmentation method of a multi-branch aggregation network

Abstract

A polyp image segmentation method of a multi-branch aggregation network comprises the following steps: step 1: using a PVTv2 encoding layer feature extraction module as a network encoding part to realize feature extraction of a polyp image, and outputting 4 layers of feature maps; step 2: inputting the output 4 layers of feature maps into a linear enhancement layer LE to obtain 4 layers of feature maps with enhanced local information; step 3: using parallel decoding constraints PDC1 and PDC2 to fuse the feature maps through the linear enhancement layer LE; step 4: up-sampling the feature maps processed through the parallel decoding constraints PDC1 and PDC2 respectively; step 5: in the test stage, cutting the parallel decoding constraint PDC1, and only using a test time data augmentation TTA mode for the parallel decoding constraint PDC2; step 6: using a Markov random field based sequential tree weighted information propagation algorithm TRW-S to smooth the edges of the gray scale image segmented by the test time data augmentation TTA output result; and segmenting the target polyp image through the above steps.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of medical image segmentation of deep learning, in particular to a polyp image segmentation method. BACKGROUND

[0002] In the prior art, since the polyp lesions are usually different in size, color and shape, and the contrast between the polyp and the surrounding mucosa is low, it is almost impossible to distinguish the boundary of the polyp. The traditional polyp segmentation method mainly relies on low-level features such as texture, color, contour and other information, so the traditional segmentation method is not ideal.

[0003] The patent document with the application publication number CN114842029A discloses a convolutional neural network polyp segmentation method fusing channel and spatial attention, which uses a residual module as an encoder to extract the features of the image, and a decoding part fuses channel and spatial attention modules. The patent document with the application publication number CN115393283A discloses a polyp image segmentation method based on shallow supervision and feedback attention, which uses an edge extraction module to obtain edge information of shallow features of the polyp, and uses a feedback attention module to strengthen the area containing the polyp in the deep features. The above-mentioned methods solve the segmentation difficulty of the fuzzy edge of the polyp, but in the case of artifacts and multi-center data existing in the polyp image, the missegmentation situation is easy to occur, and the segmentation effect is not ideal.

[0004] With the development of Transformer in the field of medical segmentation, researchers have proposed numerous variants of Transformer, which are widely used in image classification, object detection and image segmentation. PVTv2, as a variant of Transformer, has achieved remarkable results in classification, detection and segmentation. Therefore, this paper uses PVTv2 as the encoding part to realize a polyp image segmentation method based on a multi-branch aggregation network. SUMMARY

[0005] The purpose of the present application is to solve the problems of low contrast between the polyp and the surrounding mucosa in polyp segmentation, loss of detailed information in the segmentation process, and missegmentation, inaccurate segmentation of multi-center data, and appearance of artifacts in the prior art, and to provide a polyp image segmentation method based on a multi-branch aggregation network.

[0006] To solve the above technical problems, the technical scheme adopted by the present application is:

[0007] A polyp image segmentation method based on a multi-branch aggregation network, comprising the following steps:

[0008] Step 1: Use the PVTv2 encoding layer feature extraction module as the network encoding part to realize feature extraction of the polyp image, and output 4 layers of feature maps;

[0009] Step 2: input the output 4-layer feature map into a linear enhancement layer LE to obtain a 4-layer feature map with enhanced local information;

[0010] Step 3: fuse the feature map processed by the linear enhancement layer LE using parallel decoding constraints PDC1 and PDC2;

[0011] Step 4: upsample the feature maps processed by the parallel decoding constraints PDC1 and PDC2, respectively;

[0012] Step 5: in the test phase, trim the parallel decoding constraint PDC1 and only use the test-time data augmentation TTA for the parallel decoding constraint PDC2;

[0013] Step 6: smooth the edges of the gray image by performing the test-time data augmentation TTA on the output result and using the tree-structured weighted message passing algorithm based on Markov random fields TRW-S;

[0014] Through the above steps, the target polyp image is segmented.

[0015] In step 1, the PVTv2 encoding layer feature extraction module includes a first stage of the encoding layer feature extraction module, a second stage of the encoding layer feature extraction module, a third stage of the encoding layer feature extraction module, and a fourth stage of the encoding layer feature extraction module.

[0016] The encoding layer feature extraction module specifically includes:

[0017] Feature map→Patch Embedding→Reshape→Positional Encoding→Transformer→Reshape;

[0018] The Patch Embedding includes a linear mapping Linear and a normalization processing Norm.

[0019] The Transformer includes a normalization processing Norm, a multi-head attention layer Multi-Head Attention with linear space reduction Linear SRA, a normalization processing Norm, a feed forward network Feed Forward, and a skip connection with the multi-head attention layer Multi-Head Attention with linear space reduction Linear SRA.

[0020] In step 2, the linear enhancement layer LE includes a first linear enhancement layer LE, a second linear enhancement layer LE, a third linear enhancement layer LE, and a fourth linear enhancement layer LE.

[0021] The output of the PVTv2 encoding layer feature extraction module is converted from a four-dimensional tensor to a two-dimensional tensor → the two-dimensional tensor is input into a fully connected layer for linear transformation → the two-dimensional tensor is converted to a four-dimensional tensor → a grouped convolution operation is performed on the four-dimensional tensor.

[0022] In steps 3 to 5, the parallel decoding constraint PDC1 is:

[0023] The parallel decoding constraint PDC1 processing steps are as follows:

[0024] 1) 1x2 up-sampling is performed on the feature map X3 → 1x1 convolution operation and batch normalization operation are performed once;

[0025] 2) 1x2 up-sampling is performed on the feature map X3 → 1x1 convolution operation and batch normalization operation are performed once → element-wise multiplication operation is performed with the feature map X2;

[0026] 3) Concat operation is performed on 1) and 2) → first 1x1 convolution operation and batch normalization operation are performed → second 1x1 convolution operation and batch normalization operation are performed;

[0027] 4) 2x2 up-sampling is performed on the feature map X3 → 1x1 convolution operation and batch normalization operation are performed once;

[0028] 5) 1x2 up-sampling is performed on the feature map X2 → 1x1 convolution operation and batch normalization operation are performed once;

[0029] 6) element-wise multiplication operation is performed on 4) and 5) and the feature map X1 → first 1x1 convolution operation and batch normalization operation are performed → second 1x1 convolution operation and batch normalization operation are performed → 3x3 convolution operation and batch normalization operation are performed once;

[0030] The parallel decoding constraint PDC2 is:

[0031] 1) 1x2 up-sampling is performed on the feature map X4 → 1x1 convolution operation and batch normalization operation are performed once;

[0032] 2) 1x2 up-sampling is performed on the feature map X4 → 1x1 convolution operation and batch normalization operation are performed once → element-wise multiplication operation is performed with the feature map X3;

[0033] 3) Concat operation is performed on 1) and 2) → first 1x1 convolution operation and batch normalization operation are performed → second 1x1 convolution operation and batch normalization operation are performed;

[0034] 4) 2x2 up-sampling is performed on the feature map X4 → 1x1 convolution operation and batch normalization operation are performed once;

[0035] 5) 1x2 up-sampling is performed on the feature map X3 → 1x1 convolution operation and batch normalization operation are performed once;

[0036] Element-wise multiplication operation on 4) and 5) and feature map X2→ first 1x1 convolution operation and batch normalization operation→ second 1x1 convolution operation and batch normalization operation→ 3x3 convolution operation and batch normalization operation.

[0037] In step 5, the parallel decoding constraint PDC1 branch is pruned out at test time, and the test time image augmentation TTA is used to improve the prediction accuracy and model generalization ability for the prediction image generated by the parallel decoding constraint PDC2 branch. The following steps are used when using the test time image augmentation TTA:

[0038] 5-1) input the test data set into the model to obtain the original prediction image S;

[0039] 5-2) input the prediction image S obtained by the model after vertical flipping of the test data set VF ;

[0040] 5-3) input the prediction image S obtained by the model after horizontal flipping of the test data set HF ;

[0041] 5-4) average the results of 5-1), 5-2), and 5-3) to obtain the final prediction image S out，

[0042] In step 6, the model post-processes the image output by the test time image augmentation TTA using the sequential tree weighted information propagation algorithm TRW-S(12) of Markov random field (MRF) to strengthen the segmentation edge; the following steps are used for the Markov random field-based sequential tree weighted information propagation algorithm TRW-S processing:

[0043] 6-1) the pixels of the image output by the TTA are used as nodes in the construction of the MRF undirected probabilistic graph model, which is converted into an energy minimization problem through maximum a posteriori inference, and the energy function E is composed of the data energy E d and the smoothing energy E s . E = E d + λE s

[0044] 6-2) the data energy is the sum of each node energy d p (l p ), which is generated by the model's previous deep learning network output, and the output of the gray value of each pixel of the prediction image is the node data energy.

[0045] 6-3) a four-neighborhood is used to propagate the smoothing energy, which is the sum of the horizontal nearest neighbor and vertical nearest neighbor smoothing energies, and V pq (lp q q represents the neighborhood nodes of node p, N is the neighborhood of p.

[0046] 6-4) Tree-weighted information propagation algorithm: M t p→q represents the information transmitted from pixel p to pixel q at the tth iteration, c pq is the information transmitted from pixel p to pixel q at the tth iteration, c pq / ρ p (i.e. in any optional tree in ρ, the possibility of containing edge (p, q) given that it contains node p), the update rule of information is:

[0047]

[0048] A polyp image segmentation model of a multi-branch aggregation network, the structure of the model is:

[0049] The first stage of the encoding layer feature extraction module→the second stage of the encoding layer feature extraction module→the third stage of the encoding layer feature extraction module→the fourth stage of the encoding layer feature extraction module→the first branch of the decoding layer image recovery→the second branch of the decoding layer image recovery;

[0050] The first stage of the encoding layer feature extraction module→the first linear enhancement layer LE;

[0051] The second stage of the encoding layer feature extraction module→the second linear enhancement layer LE;

[0052] The third stage of the encoding layer feature extraction module→the third linear enhancement layer LE;

[0053] The fourth stage of the encoding layer feature extraction module→the fourth linear enhancement layer LE;

[0054] The first linear enhancement layer LE, the second linear enhancement layer LE, and the third linear enhancement layer LE→parallel decoding constraint PDC1;

[0055] The second linear enhancement layer LE, the third linear enhancement layer LE, and the fourth linear enhancement layer LE→parallel decoding constraint PDC2→test time data augmentation TTA→sequential tree-weighted information propagation algorithm based on Markov random field TRW-S.

[0056] The PVTv2 encoding layer feature extraction module includes the first stage of the encoding layer feature extraction module, the second stage of the encoding layer feature extraction module, the third stage of the encoding layer feature extraction module, and the fourth stage of the encoding layer feature extraction module.

[0057] The encoding layer feature extraction module is specifically:

[0058] ​Feature map → Patch Embedding → Reshape → Positional Encoding → Transformer Encoding → Reshape;

[0059] In the Patch Embedding, a linear mapping Linear is performed, and then a normalization processing Norm is performed.

[0060] In the Transformer Encoding, a normalization processing Norm is performed, then a multi-head attention layer Multi-Head Attention with linear space reduction Linear SRA is performed, then a normalization processing Norm is performed, and then a feed forward network Feed Forward is performed, and then a multi-head attention layer Multi-Head Attention with linear space reduction Linear SRA is performed in a skip connection manner.

[0061] In step 2, the linear enhancement layer LE includes a first linear enhancement layer LE, a second linear enhancement layer LE, a third linear enhancement layer LE, and a fourth linear enhancement layer LE.

[0062] The output of the PVTv2 encoding layer feature extraction module is converted from a four-dimensional tensor to a two-dimensional tensor, the two-dimensional tensor is input into a fully connected layer for linear transformation, the two-dimensional tensor is converted into a four-dimensional tensor, and a grouped convolution operation is performed on the four-dimensional tensor.

[0063] Compared with the prior art, the present application has the following technical effects:

[0064] The present application realizes a polyp image segmentation method based on a multi-branch aggregation network, wherein PVTv2 is used as an encoding part, the PVTv2, a linear enhancement layer (LE), and two parallel decoding constraints (PDC) constitute a network model, finally, test time data augmentation (TTA) is combined to complete the prediction segmentation of the polyp image, and a Markov random field (MRF) order tree reweighted information propagation (TRW-S) algorithm is constructed in the prediction segmentation image to perform post-processing, thereby strengthening the segmentation edge and enhancing the coherence of each segmentation region. The present application model is applied to the segmentation of polyp images in a public data set Kvasir-SEG, and the experimental results show that the mDice value reaches 91.3%, the mIoU value reaches 86.0%, and the Smeasure reaches 92.0%, which exceeds the current popular polyp segmentation algorithm and exhibits excellent segmentation effect of the present application model. The present application model is applied to the test of unknown data sets ETIS-LaribPolypDB and ColonDB relative to the training set, and good segmentation effect is obtained, thereby verifying the good generalization ability of the present application model. BRIEF DESCRIPTION OF DRAWINGS

[0065] The application will be further described below with reference to the accompanying drawings and embodiments.

[0066] Figure 1 A structure diagram of the polyp image segmentation method of the multi-branch aggregation network in the application;

[0067] Figure 2 For Figure 1 A structure diagram of the overall and stage composition of the PVTv2 encoding layer feature extraction module;

[0068] Figure 3 A structure diagram of the linear enhancement layer LE of the application;

[0069] Figure 4 A structure diagram of the parallel decoding constraint PDC of the application;

[0070] Figure 5 A comparison chart of test effects in the Kvasir-SEG dataset in the embodiment;

[0071] Figure 6 A comparison chart of test effects in the ETIS-LaribPolypDB dataset in the embodiment; DETAILED DESCRIPTION

[0072] A polyp image segmentation method of a multi-branch aggregation network, comprising the following steps:

[0073] Step 1: using the PVTv2 encoding layer feature extraction module as the network coding part to realize feature extraction of the polyp image, and outputting 4 layers of feature maps;

[0074] Step 2: inputting the output 4 layers of feature maps into the linear enhancement layer LE to obtain 4 layers of feature maps with enhanced local information;

[0075] Step 3: using the parallel decoding constraints PDC1 and PDC2 to fuse the feature maps through the linear enhancement layer LE;

[0076] Step 4: up-sampling the feature maps processed by the parallel decoding constraints PDC1 and PDC2 respectively;

[0077] Step 5: in the test stage, cutting the parallel decoding constraint PDC1 and only using the test time data augmentation TTA for the parallel decoding constraint PDC2;

[0078] Step 6: outputting the results of the test time data augmentation TTA through the sequential tree weighted information propagation algorithm TRW-S based on the Markov random field to smooth the edges of the gray scale image segmentation;

[0079] Through the above steps, the target polyp image is segmented.

[0080] In step 1, as shown in Figure 2 The PVTv2 encoding layer feature extraction module includes a first stage 1 of the encoding layer feature extraction module, a second stage 2 of the encoding layer feature extraction module, a third stage 3 of the encoding layer feature extraction module, and a fourth stage 4 of the encoding layer feature extraction module.

[0081] The encoding layer feature extraction module specifically includes:

[0082] The feature map→Patch Embedding→Reshape→Positional Encoding→Transformer Encoder→Reshape.

[0083] The Patch Embedding includes a Linear→Norm.

[0084] The Transformer Encoder includes a Norm→Multi-Head Attention with Linear SRA→Norm→Feed Forward→Multi-Head Attention with Linear SRA.

[0085] In step 2, as shown in Figure 3 The linear enhancement layer LE includes a first linear enhancement layer LE5, a second linear enhancement layer LE6, a third linear enhancement layer LE7, and a fourth linear enhancement layer LE8.

[0086] The output of the PVTv2 encoding layer feature extraction module is converted from a four-dimensional tensor to a two-dimensional tensor→the two-dimensional tensor is input into a fully connected layer for linear transformation→the two-dimensional tensor is converted to a four-dimensional tensor→group convolution is performed on the four-dimensional tensor.

[0087] In steps 3 to 5, as shown in Figure 4 The parallel decoding constraint PDC1 is:

[0088] The processing steps of the parallel decoding constraint PDC1 are as follows:

[0089] 1) 1x2 up-sampling is performed on the feature map X3→1x1 convolution and batch normalization are performed once;

[0090] 2) 1x2 up-sampling is performed on the feature map X3→1x1 convolution and batch normalization are performed once→element-wise multiplication is performed with the feature map X2;

[0091] 3) Concat operation on 1) and 2) -> first 1x1 convolution operation and batch normalization operation -> second 1x1 convolution operation and batch normalization operation;

[0092] 4) 2 times 2x up-sampling on feature map X3 -> one 1x1 convolution operation and batch normalization operation;

[0093] 5) 1 time 2x up-sampling on feature map X2 -> one 1x1 convolution operation and batch normalization operation;

[0094] 6) element-wise multiplication operation on 4) and 5) and feature map X1 -> first 1x1 convolution operation and batch normalization operation -> second 1x1 convolution operation and batch normalization operation -> one 3x3 convolution operation and batch normalization operation;

[0095] Parallel decoding constraint PDC2 is:

[0096] 1) 1 time 2x up-sampling on feature map X4 -> one 1x1 convolution operation and batch normalization operation;

[0097] 2) 1 time 2x up-sampling on feature map X4 -> one 1x1 convolution operation and batch normalization operation -> element-wise multiplication operation with feature map X3;

[0098] 3) Concat operation on 1) and 2) -> first 1x1 convolution operation and batch normalization operation -> second 1x1 convolution operation and batch normalization operation;

[0099] 4) 2 times 2x up-sampling on feature map X4 -> one 1x1 convolution operation and batch normalization operation;

[0100] 5) 1 time 2x up-sampling on feature map X3 -> one 1x1 convolution operation and batch normalization operation;

[0101] element-wise multiplication operation on 4) and 5) and feature map X2 -> first 1x1 convolution operation and batch normalization operation -> second 1x1 convolution operation and batch normalization operation -> one 3x3 convolution operation and batch normalization operation.

[0102] In step 5, parallel decoding constraint PDC1 branch is pruned out at test time, and test time image augmentation TTA11 is used for the predicted image generated by parallel decoding constraint PDC2 branch, and the test time image augmentation TTA is used to improve the prediction accuracy and model generalization ability. The following steps are used when using the test time image augmentation TTA:

[0103] 5-1) input test dataset to model to get original predicted image S;

[0104] 5-2) input the test dataset after vertical flip to the model to get the predicted image SVF ;

[0105] 5-3) The prediction map S obtained by inputting the model after horizontal flipping of the test data set HF ;

[0106] 5-4) The final prediction map S is obtained by averaging the results of 5-1), 5-2), and 5-3) out，

[0107] In step 6, the model post-processes the image output by the image enhancement TTA to construct a Markov Random Field (MRF) sequential tree re-weighting information propagation algorithm TRW-S12 to strengthen the segmentation edges; the Markov Random Field-based sequential tree weighting information transmission algorithm TRW-S processing adopts the following steps:

[0108] 6-1) The pixels of the image output by the TTA are taken as nodes in the construction of the MRF undirected probabilistic graph model, and are converted into an energy minimization problem by maximum a posteriori inference, and the energy function E is composed of the data energy E d and the smoothing energy E s . E = E d + λE s

[0109] 6-2) The data energy is the sum of the energy d p (l p ) of each node, and each node energy is generated by the model pre-deep learning network output, and the output prediction map gray value of each pixel is the node data energy.

[0110] 6-3) A four-neighborhood is used to propagate the smoothing energy, and the smoothing energy is the sum of the horizontal nearest neighbor and vertical nearest neighbor smoothing energy, denoted by V pq (l p ,l q ). q represents the neighborhood nodes of node p, and N is the neighborhood of p.

[0111] 6-4) Tree weighting information transmission algorithm: M t p→q represents the information transmitted from pixel p to pixel q at the tth iteration, c pq is the probability of ρ pq / ρ p (i.e. any optional tree in ρ, the possibility of containing edge (p, q) given that node p is contained), and the update rule of the information is:

[0112]

[0113] The application also comprises a polyp image segmentation model of a multi-branch aggregation network, and the structure of the model is as follows:

[0114] First stage 1 of the encoding layer feature extraction module→second stage 2 of the encoding layer feature extraction module→third stage 3 of the encoding layer feature extraction module→fourth stage 4 of the encoding layer feature extraction module→first branch 13 of the decoding layer image recovery→second branch 14 of the decoding layer image recovery;

[0115] First stage 1 of the encoding layer feature extraction module→first linear enhancement layer LE5;

[0116] Second stage 2 of the encoding layer feature extraction module→second linear enhancement layer LE6;

[0117] Third stage 3 of the encoding layer feature extraction module→third linear enhancement layer LE7;

[0118] Fourth stage 4 of the encoding layer feature extraction module→fourth linear enhancement layer LE8;

[0119] First linear enhancement layer LE5, second linear enhancement layer LE6, third linear enhancement layer LE7→parallel decoding constraint PDC19;

[0120] Second linear enhancement layer LE6, third linear enhancement layer LE7, fourth linear enhancement layer LE8→parallel decoding constraint PDC210→test time data augmentation TTA11→sequential tree weighted information transmission algorithm TRW-S12 based on Markov random field.

[0121] The PVTv2 encoding layer feature extraction module comprises first stage 1 of the encoding layer feature extraction module, second stage 2 of the encoding layer feature extraction module, third stage 3 of the encoding layer feature extraction module and fourth stage 4 of the encoding layer feature extraction module.

[0122] The encoding layer feature extraction module specifically comprises:

[0123] Feature map→patch embedding→reshape→embedding position encoding→transformer encoder→reshape;

[0124] The patch embedding comprises linear mapping and normalization processing.

[0125] The Transformer coding is: normalization processing Norm, multi-head attention layer Multi-Head Attention through linear space reduction Linear SRA, normalization processing Norm, feed forward network Feed Forward, and multi-head attention layer Multi-Head Attention through linear space reduction Linear SRA are connected in a skip connection.

[0126] In step 2, as shown in Figure 3 The linear enhancement layer LE includes a first linear enhancement layer LE5, a second linear enhancement layer LE6, a third linear enhancement layer LE7, and a fourth linear enhancement layer LE8.

[0127] The output of the PVTv2 coding layer feature extraction module is converted from a four-dimensional tensor to a two-dimensional tensor, the two-dimensional tensor is input into a fully connected layer for linear transformation, the two-dimensional tensor is converted into a four-dimensional tensor, and a grouped convolution operation is performed on the four-dimensional tensor.

[0128] To illustrate the segmentation effect of the model constructed by the present application, comparative experiments qualitatively and quantitatively compare the model of the present application with the network models of U-Net, U-Net++, ResUnet++, DoubleUNet, SFA and PraNet for polyp image segmentation. The experimental results show that the algorithm in this paper is stronger than the benchmark algorithm, compared with the benchmark PraNet algorithm, mDice is improved by 1.5%, mIoU is improved by 2.0%, Smeasure is improved by 0.5%, and MAE is reduced by 0.4%, as shown in Table 3. Therefore, the model of the present application has certain advantages in segmenting polyps.

[0129] Table 3 shows the results of training under the CVC-ClinicDB and Kvasir-SEG dataset and testing under the Kvasir-SEG dataset.

[0130]

[0131] The segmentation effect of the model of the present application under the Kvasir-SEG dataset test is shown in Figure 5 . Figure 5 The segmentation results of the five methods of using UNet, UNet++, SFA, PraNet and the model of the present application are given. From the segmentation performance, the area consistency of the segmentation result of the model of the present application is very high, and there are few cases of missegmentation of normal tissue, which benefits from the excellent expressiveness of PVTv2 as the coding backbone. Therefore, the model in this paper performs well in segmenting the polyp lesion area, and the segmentation result is very accurate, which has important reference value for doctors to diagnose the lesion area.

[0132] The dataset ETIS-LaribPolypDB is a training set of the model of the present application, which is an unknown dataset relative to the training set, and the model of the present application performs well on the unknown dataset, with mDice improved by 16.9%, mIoU improved by 15.4%, Smeasure improved by 7.7%, and MAE decreased by 1.7%, as shown in Table 4. Each objective evaluation index is improved by a large margin, and therefore, the generalization ability of the model is obviously improved, which benefits from the adoption of PVTv2, which has stronger adaptability when facing different environmental datasets. Since the ETIS-LaribPolypDB dataset is the most difficult to segment, there is a significant improvement in ETIS-LaribPolypDB, which indicates that the model of the present application can adapt to different environments and has good generalization ability.

[0133] Table 4 shows the results of training on CVC-ClinicDB and Kvasir-SEG datasets and testing on ETIS-LaribPolypDB dataset.

[0134]

[0135] The segmentation effect of the model of the present application on the ETIS-LaribPolypDB dataset is shown in Table 4. Figure 6 Figure 6 The segmentation results of five methods using UNet, UNet++, SFA, PraNet, and the model of the present application are given. Whether it is small lesion tissue or large lesion tissue, the model of the present application can well locate and segment; in the lesion image, there is a large interference of polyp tissue, and a part of it is due to the influence of the internal environment of the intestinal tract, such as reflection and texture, which will interfere with the segmentation effect, and the model has good generalization ability in this case, which indicates that the model of the present application has good robustness and good effect on polyp segmentation.

[0136] To illustrate the effect of the linear enhancement layer LE, the present application performs an ablation experiment on three datasets of ColonDB, ETIS-LaribPolypDB and CVC-300, and the results are shown in Table 5. The segmentation indexes of the model are improved to different degrees, with mDice improved by 1.2% on the ColonDB dataset, mIoU improved by 0.9%, and Smeasure improved by 0.5%; on the ETIS-LaribPolypDB dataset, mDice is improved by 2.4%, mIoU is improved by 2.7%, Smeasure is improved by 0.5%, and MAE is decreased by 0.4%; on the CVC-300 dataset, mDice is improved by 2.2%, mIoU is improved by 2.9%, and Smeasure is improved by 1.4%. In summary, after adding the linear enhancement layer, the segmentation ability of the model is obviously improved, which verifies the effectiveness of the model of the present application with the linear enhancement layer.​

[0137] Table 5 ablation experiment of linear convolution layer

[0138]

[0139] The ablation experiment of the multi-branch was tested on the ColonDB, ETIS-LaribPolypDB, CVC-300 three data sets, and the effect of parallel decoding constraint (PDC) on polyp segmentation was verified. The objective evaluation indexes of the experiment are shown in Table 6. The structure of the two PDC branches has no obvious improvement on the ColonDB data set, but has a substantial increase on the ETIS-LaribPolypDB and CVC-300 data sets. On the ETIS-LaribPolypDB data set, mDice is increased by 1.3%, mIoU is increased by 2.5%, and the Smeasure index is increased by 1.0%. On the CVC-300 data set, mDice is increased by 0.9%, mIoU is increased by 0.9%, and the Smeasure index is increased by 0.4%. In summary, in combination with the performance of the generalization ability of the three models on unknown data sets relative to the training set, it can be seen that the structure of the two PDC branches is effective in improving the generalization ability of the model.

[0140] Table 6 ablation experiment of multi-branch structure

[0141]

[0142] The ablation experiment was performed on the Kvasir-SEG, ColonDB, ETIS-LaribPolypDB, CVC-300 four data sets to verify the influence of data augmentation TTA on the segmentation ability of the model during testing, as shown in Table 7. After using TTA, on the Kvasir-SEG data set, mDice is increased by 0.1%, mIoU is increased by 0.3%, and Smeasure is increased by 0.3%; on the ColonDB data set, mDice is increased by 1.1%, mIoU is increased by 1.5%, and Smeasure is increased by 1.0%; on the ETIS-LaribPolypDB data set, mDice is increased by 4.4%, mIoU is increased by 4.5%, and Smeasure is increased by 3.2%; on the CVC-300 data set, mDice is increased by 0.5%, mIoU is increased by 0.9%, and Smeasure is increased by 0.3%. It can be seen that the use of TTA improves the segmentation ability and generalization ability of the model.

[0143] Table 7 ablation experiment of TTA

[0144]

[0145] In summary, the present application realizes a polyp image segmentation method of a multi-branch aggregation network, in which PVTv2 is used as an encoding part, PVTv2 and a linear enhancement layer (LE) and two parallel decoding constraints (PDC) constitute a network model; finally, test time data augmentation (TTA) is combined to complete the predicted segmentation of the polyp image; in the predicted segmentation image, a Markov random field (MRF) order tree reweighted information propagation (TRW-S) algorithm is used for post-processing to strengthen the segmentation edge and enhance the coherence of each segmentation region. The application of the model of the present application to the segmentation of polyp images in the public data set Kvasir-SEG shows that the mDice value reaches 91.3%, the mIoU value reaches 86.0%, and the Smeasure reaches 92.0%, which surpasses the current popular polyp segmentation algorithm and exhibits the excellent segmentation effect of the model of the present application. The application of the model of the present application to the test on the unknown data set ETIS-LaribPolypDB, ColonDB relative to the training set obtains a good segmentation effect, thereby verifying the good generalization ability of the model of the present application.

Claims

1. A polyp image segmentation method of a multi-branch aggregation network, characterized by, The method comprises the following steps: Step 1: feature extraction of polyp images is realized by using a PVTv2 encoding layer feature extraction module as a network coding part, and 4 layers of feature maps are output; Step 2: the output 4 layers of feature maps are input into a linear enhancement layer LE to obtain 4 layers of feature maps with enhanced local information; Step 3: the feature maps passing through the linear enhancement layer LE are fused using parallel decoding constraints PDC1 and PDC2; Step 4: the feature maps processed by the parallel decoding constraints PDC1 and PDC2 are respectively up-sampled; Step 5: in the test stage, the parallel decoding constraint PDC1 is cropped, and only the parallel decoding constraint PDC2 adopts a test-time data augmentation TTA mode; Step 6: the output result of the test-time data augmentation TTA is smoothed by using a Markov random field-based sequential tree weighted information transmission algorithm TRW-S to perform edge smoothing on a gray-scale image. Through the above steps, the target polyp image is segmented.

2. The method of claim 1, wherein, In step 1, the PVTv2 encoding layer feature extraction module comprises a first stage (1) of the encoding layer feature extraction module, a second stage (2) of the encoding layer feature extraction module, a third stage (3) of the encoding layer feature extraction module and a fourth stage (4) of the encoding layer feature extraction module. The encoding layer feature extraction module specifically comprises: a feature map→patch embedding→reshape→embedding position coding→transformer encoding→reshape. The patch embedding comprises a linear mapping and a normalization processing. The transformer encoding comprises a normalization processing, a multi-head attention layer with linear space reduction, a normalization processing, a feed forward network and a multi-head attention layer with linear space reduction for skip connection.

3. The method of claim 1, wherein, In step 2, the linear enhancement layer LE comprises a first linear enhancement layer LE (5), a second linear enhancement layer LE (6), a third linear enhancement layer LE (7) and a fourth linear enhancement layer LE (8). The output of the PVTv2 encoding layer feature extraction module is converted from a four-dimensional tensor to a two-dimensional tensor, input into a fully connected layer for linear transformation, converted from a two-dimensional tensor to a four-dimensional tensor, and subjected to grouped convolution operation.

4. The method of claim 1, wherein, In steps 3 to 5, the parallel decoding constraint PDC1 is as follows: The processing steps of the parallel decoding constraint PDC1 are as follows: 1) 1-time 2-fold up-sampling of the feature map X3→1-time 1×1 convolution operation and batch normalization operation; 2) 1-time 2-fold up-sampling of the feature map X3→1-time 1×1 convolution operation and batch normalization operation→element-wise multiplication operation with the feature map X2; 3) Concat operation on 1) and 2)→1-time 1×1 convolution operation and batch normalization operation→2-time 1×1 convolution operation and batch normalization operation; 4) 2x upsampling on feature map X3→1x1 convolution and batch normalization; 5) 2x upsampling on feature map X2→1x1 convolution and batch normalization; 6) element-wise multiplication of 4) and 5) and feature map X1→1x1 convolution and batch normalization→1x1 convolution and batch normalization→3x3 convolution and batch normalization; Parallel decoding constraint PDC2 is: 1) 2x upsampling on feature map X4→1x1 convolution and batch normalization; 2) 2x upsampling on feature map X4→1x1 convolution and batch normalization→element-wise multiplication with feature map X3; 3) Concat of 1) and 2)→1x1 convolution and batch normalization→1x1 convolution and batch normalization; 4) 2x upsampling on feature map X4→1x1 convolution and batch normalization; 5) 2x upsampling on feature map X3→1x1 convolution and batch normalization; element-wise multiplication of 4) and 5) and feature map X2→1x1 convolution and batch normalization→1x1 convolution and batch normalization→3x3 convolution and batch normalization.

5. The method of claim 1, wherein, In step 5, the parallel decoding constraint PDC1 branch is pruned when testing, and a test time augmentation TTA (11) is used for the prediction image generated by the parallel decoding constraint PDC2 branch to improve the prediction accuracy and the model generalization ability, and the following steps are used when using the test time augmentation TTA: 5-1) input the test data set into the model to obtain an original prediction image S; 5-2) The predicted map S obtained after inputting the model to the test data set after vertical flipping VF ; 5-3) The prediction map S obtained after inputting the model to the test data set after horizontal flip HF ; 5-4) The results of 5-1), 5-2), 5-3) are averaged to obtain the final prediction map S out， .

6. The method of claim 1, wherein, In step 6, the model post-processes the image output by the test time augmentation TTA using a Markov random field MRF sequential tree weighted message passing algorithm TRW-S (12) to strengthen the segmentation edge; the Markov random field sequential tree weighted message passing algorithm TRW-S processing adopts the following steps: 6-1) The image pixels output by the TTA are used as nodes in constructing a MRF undirected probabilistic graph model, which is converted to an energy minimization problem by maximum a posteriori inference, the energy function consists of data energy from the nodes and smoothing energy ; 6-2) Data energy is the sum of each node energy , each node energy is generated by the model pre-deep learning network output, the output predicts the gray value of each pixel of the image is the node data energy, ; 6-3) The four-neighborhood is used to propagate the smoothing energy, which is the sum of the horizontal and vertical nearest-neighbor smoothing energy, denoted by , where q denotes the neighborhood nodes of node p, and N is the neighborhood of p, ; 6-4) Tree-weighted information transfer algorithm: Represents the pixel at iteration t. To pixel The information conveyed; for That is, in Choose any tree and, given a node... In the case of including edges The possibility; the information update rules are: 。 7. A polyp image segmentation model of a multi-branch aggregation network, characterized in that, The structure of the model is: The PVTv2 encoding layer feature extraction module is used as the encoding part; the PVTv2 encoding layer feature extraction module includes a first stage (1) of the encoding layer feature extraction module, a second stage (2) of the encoding layer feature extraction module, a third stage (3) of the encoding layer feature extraction module, and a fourth stage (4) of the encoding layer feature extraction module; The first stage (1) of the encoding layer feature extraction module→the second stage (2) of the encoding layer feature extraction module→the third stage (3) of the encoding layer feature extraction module→the fourth stage (4) of the encoding layer feature extraction module→the first branch (13) of the decoding layer image recovery→the second branch (14) of the decoding layer image recovery; The first stage (1) of the encoding layer feature extraction module→the first linear enhancement layer LE (5); The second stage (2) of the encoding layer feature extraction module→the second linear enhancement layer LE (6); Third stage (3) of the encoding layer feature extraction module -> third linear enhancement layer LE (7); Fourth stage (4) of the encoding layer feature extraction module -> fourth linear enhancement layer LE (8); First linear enhancement layer LE (5), second linear enhancement layer LE (6), third linear enhancement layer LE (7) -> parallel decoding constraint PDC1 (9); Second linear enhancement layer LE (6), third linear enhancement layer LE (7), fourth linear enhancement layer LE (8) -> parallel decoding constraint PDC2 (10) -> test time augmentation TTA (11) -> Markov random field based sequential tree weighted information propagation algorithm TRW-S (12).

8. The model of claim 7, wherein, The encoding layer feature extraction module specifically comprises: Feature map -> patch embedding -> reshape -> embedding position encoding -> transformer encoding -> reshape; Wherein, the patch embedding comprises: linear mapping -> normalization processing; Wherein, the transformer encoding comprises: normalization processing -> multi-head attention layer with linear space reduction Linear SRA -> normalization processing -> feed forward network -> skip connection with the multi-head attention layer with linear space reduction Linear SRA.

9. The model of claim 7, wherein, In step 2, the linear enhancement layer LE comprises a first linear enhancement layer LE (5), a second linear enhancement layer LE (6), a third linear enhancement layer LE (7), and a fourth linear enhancement layer LE (8); The output of the PVTv2 encoding layer feature extraction module is converted from a four-dimensional tensor to a two-dimensional tensor -> the two-dimensional tensor is input into a fully connected layer for linear transformation -> the two-dimensional tensor is converted into a four-dimensional tensor -> group convolution operation is performed on the four-dimensional tensor.

Images