Endoscopic Image Segmentation Algorithm with Multi-Scale Feature Fusion

Abstract

The present invention relates to the technical field of enteroscopes, in particular to an enteroscope image segmentation algorithm for multi-scale feature fusion, including a ResNet50 backbone extraction network, a pyramid feature extraction module PFEM, and a multi-scale fusion module MPFM. The ResNet50 backbone extraction network is used to extract multi-scale polyp features from enteroscope images; the pyramid feature extraction module PFEM is used to expand the receptive field and extract multi-scale detailed information; the multi-scale fusion module MPFM adjusts different feature weights through the number of inputs, and fuses the processed deep and shallow features in different proportions, which can strengthen the polyp area, improve the segmentation accuracy of the edge, and make the network perform better in complex environments. By adopting the ResNet50 backbone extraction network, the pyramid feature extraction module PFEM, and the multi-scale fusion module MPFM, the present invention can better distinguish polyps in the intestine.

Description

Technical Field

[0001] The present invention relates to the technical field of enteroscopes, and in particular to an enteroscope image segmentation algorithm for multi-scale feature fusion. Background Technique

[0002] At present, the methods for intestinal polyp segmentation mainly include: traditional colon polyp image segmentation methods and intestinal polyp segmentation methods based on deep learning. The traditional colon polyp image segmentation method is based on low-level features and only considers single features, resulting in low segmentation accuracy. The deep learning-based method can extract deeper features, but due to the different sizes of intestinal polyps and large background noise, the segmentation of polyp edges is inaccurate, and even small polyps cannot be segmented. Summary of the Invention

[0003] The purpose of the present invention is to solve the disadvantages existing in the prior art, and to propose an enteroscope image segmentation algorithm for multi-scale feature fusion.

[0004] To achieve the above purpose, the present invention adopts the following technical solutions: an enteroscope image segmentation algorithm for multi-scale feature fusion, including a ResNet50 backbone extraction network, a pyramid feature extraction module PFEM, and a multi-ratio fusion module MPFM, characterized in that: the ResNet50 backbone extraction network is used to extract multi-scale polyp features from enteroscope images; the pyramid feature extraction module PFEM is used to expand the receptive field and extract multi-scale detailed information; the multi-ratio fusion module MPFM adjusts the weights of different features through the number of inputs, and fuses the processed deep and shallow features in different ratios, which can strengthen the polyp area, improve the segmentation accuracy of the edges, and make the network perform better in complex environments.

[0005] As a further description of the above technical solution:

[0006] The ResNet50 backbone extraction network outputs feature maps X1, X2, X3, and X4 of four scales from low level to high level.

[0007] As a further description of the above technical solution:

[0008] The pyramid feature extraction module PFEM extracts texture and color detail information.

[0009] As a further description of the above technical solution:

[0010] The Pyramid Feature Extraction Module (PFEM) includes a first part and a second part. The first part is a convolutional block that extracts features using a 3×3 convolution, passes them through the SiLU activation function, and then adjusts the channels using a 1×1 convolution. The second part removes the branch of the dilated convolution with a dilation rate of r = 24 in the ASPP structure, improves the 4 branches to 3 branches as a module, and uses three parallel dilated convolutions with different dilation rates to complete feature extraction at multiple different scales. The dilated convolutions with different dilation rates can improve the problem of the relatively small receptive field of the shallow network, and at the same time ensure that more details can be captured without too much computational effort. Finally, the results of these two parts are concatenated and output.

[0011] As a further description of the above technical solution:

[0012] The Multi-Scale Fusion Module (MPFM) is used to fuse the processed deep and shallow features. For the first, second, third, and fourth processed feature maps A, B, C, and D, in the first part, D1, D2, D3, and D4 obtained by upsampling and convolving the feature map D are multiplied by C, then connected to D1, and the concatenated features are smoothed through a convolutional layer to obtain the feature map C3 of the first part. In the second part, C1 and C2 are obtained by upsampling and convolutional layers for the feature map C, and C1 is multiplied by D3 and B to obtain the feature map B2 of the second part. The features of B2 and C3 are concatenated, and then passed through a 3×3 convolution to obtain the result T2 of deep feature fusion. In the third part, B1 obtained by upsampling the feature map B and passing it through a convolutional layer is multiplied by C2, D4, and A to obtain the feature map A2 of the third part. A2 is concatenated with the result T2 of deep feature fusion, and then passed through a 3×3 convolution to adjust the smoothed concatenated features and the number of channels to obtain the final output T1 of deep and shallow feature fusion.

[0013] As a further description of the above technical solution:

[0014] The Multi-Scale Fusion Module (MPFM) adopts a multi-scale fusion strategy, increasing the proportion of deep features by inputting the fourth-layer features four times, the third-layer features three times, the second-layer features twice, and the first-layer features once, fusing shallow features while obtaining clear semantics, strengthening the polyp area, and obtaining an accurate segmentation result.

[0015] As a further description of the above technical solution:

[0016] It also includes the following algorithms:

[0017] The formula for recall is as follows: The formula for precision is as follows: The formula for the mean intersection over union (mIoU) is as follows: The Dice formula of the dice coefficient is as follows: The formula for accuracy is as follows:

[0018] Among them, TP means that the pixel on the real image is a polyp and the pixel on the predicted image is also a polyp. TN means that the pixel on the real image is not a polyp and the pixel on the predicted image is also not a polyp. FP means that the pixel on the real image is not a polyp while the pixel on the predicted image is a polyp. FN means that the pixel on the real image is a polyp while the pixel on the predicted image is not a polyp.

[0019] The present invention has the following beneficial effects:

[0020] Compared with the prior art, the endoscopic image segmentation algorithm with multi-scale feature fusion uses the ResNet50 backbone extraction network to extract multi-scale polyp features from endoscopic images. The innovative use of the pyramid feature extraction module PFEM expands the receptive field, extracts multi-scale detailed information, and obtains accurate target boundaries. And the innovative use of the multi-scale fusion module MPFM fuses the processed deep and shallow features in different proportions, which can strengthen the polyp area, improve the segmentation accuracy of the edge, and make the network perform better in complex environments. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 It is the network structure diagram of the endoscopic image segmentation algorithm with multi-scale feature fusion proposed by the present invention;

[0022] Figure 2 It is the structure diagram of the pyramid feature extraction module of the endoscopic image segmentation algorithm with multi-scale feature fusion proposed by the present invention;

[0023] Figure 3 It is the structure diagram of the multi-scale fusion module of the endoscopic image segmentation algorithm with multi-scale feature fusion proposed by the present invention; DETAILED DESCRIPTION OF THE INVENTION

[0024] Referring to Figures 1-3 , the endoscopic image segmentation algorithm with multi-scale feature fusion provided by the present invention includes a ResNet50 backbone extraction network, a pyramid feature extraction module PFEM, and a multi-scale fusion module MPFM. The ResNet50 backbone extraction network is used to extract multi-scale polyp features from endoscopic images; the pyramid feature extraction module PFEM is used to expand the receptive field and extract multi-scale detailed information; the multi-scale fusion module MPFM adjusts the weights of different features through the number of inputs, and fuses the processed deep and shallow features in different proportions, which can strengthen the polyp area, improve the segmentation accuracy of the edge, and make the network perform better in complex environments.

[0025] As a further implementation of the above technical solution:

[0026] The output of the ResNet50 backbone extraction network consists of four feature maps X1, X2, X3, and X4 from low level to high level. By using the ResNet50 backbone extraction network as the backbone network to extract image features, four feature maps X1, X2, X3, and X4 from low level to high level are output. Among them, X1 contains rich detail information such as texture, color, etc., while X2, X3, and X4 have rich semantic information.

[0027] As a further implementation of the above technical solution:

[0028] The Pyramid Feature Extraction Module PFEM extracts texture and color detail information.

[0029] As a further implementation of the above technical solution:

[0030] The Pyramid Feature Extraction Module PFEM includes a first part and a second part. The first part is a convolutional block. After extracting features with a 3×3 convolution, passing through the SiLU activation function, and then adjusting the channels with a 1×1 convolution. The second part removes the branch of the dilated convolution with a dilation rate of r = 24 in the ASPP structure, improves the 4 branches to 3 branches as a module, and uses three parallel dilated convolutions with different dilation rates to complete feature extraction at multiple different scales. The dilated convolutions with different dilation rates can improve the problem that the receptive field of the shallow network is relatively small, and at the same time ensure that more details can be captured without too much computational effort. Finally, the results of these two parts are concatenated and output.

[0031] As a further implementation of the above technical solution:

[0032] The Multi-scale Fusion Module MPFM is used to fuse the processed deep and shallow features, namely the first, second, third, and fourth processed feature maps A, B, C, and D. In the first part, D1, D2, D3, and D4 obtained by upsampling and convolving the feature map D are multiplied by C, then connected with D1, and the feature map is smoothed through a convolutional layer to obtain the feature map C3 of the first part. In the second part, C1 and C2 are obtained by upsampling and convolutional layers of the feature map C, and C1 is multiplied by D3 and B to obtain the feature map B2 of the second part. The features of B2 and C3 are concatenated, and then passed through a 3×3 convolution to obtain the result T2 of deep feature fusion. In the third part, B1 obtained by upsampling the feature map B and passing through a convolutional layer is multiplied by C2, D4, and A to obtain the feature map A2 of the third part. A2 is concatenated with the result T2 of deep feature fusion, and then passed through a 3×3 convolution to adjust the feature smoothing and the number of channels to obtain the final output result T1 of deep and shallow feature fusion.

[0033] As a further implementation of the above technical solution:

[0034] The multi-scale fusion module MPFM adopts a multi-scale fusion strategy. By inputting the features of the fourth layer four times, the features of the third layer three times, the features of the second layer twice, and the features of the first layer once, the proportion of deep features is increased. While obtaining clear semantics, shallow features are fused, the polyp region is strengthened, and an accurate segmentation result is obtained.

[0035] Subjective evaluation is carried out first:

[0036] 1. The algorithm of the present invention can accurately segment small polyps;

[0037] 2. The algorithm of the present invention can accurately predict the position and boundary of polyps;

[0038] 3. The algorithm of the present invention can segment all polyps and the edges are basically accurately segmented;

[0039] 4. The algorithm of the present invention has significant advantages in the position and contour of polyp segmentation compared with commonly used ones.

[0040] As a further implementation manner of the above technical solution:

[0041] It also includes the following algorithms:

[0042] The formula for recall rate Recall is as follows: The formula for precision rate Precision is as follows: The formula for mean intersection over union mIoU is as follows: The formula for dice coefficient Dice is as follows: The formula for accuracy rate Accuracy is as follows:

[0043] Among them, TP represents that the pixel on the true image is a polyp and the pixel on the predicted image is also a polyp, TN represents that the pixel on the true image is a non-polyp and the pixel on the predicted image is also a non-polyp, FP represents that the pixel on the true image is a non-polyp while the pixel on the predicted image is a polyp, and FN represents that the pixel on the true image is a polyp while the pixel on the predicted image is a non-polyp.

[0044] Then, an objective evaluation is carried out. The objective evaluation of the algorithm in this paper is realized by comparing five performance indicators, namely recall, precision, mean intersection over union (mIoU), Dice coefficient, and accuracy, with other algorithms. The values of the evaluation indicators are between [0, 1], and the closer the value is to 1, the better the segmentation effect. As shown in Table 1, when compared with existing excellent algorithms, all indicators of the algorithm of the present invention are much higher than those of the compared algorithms. For the mean intersection over union and Dice coefficient of the test set, ResUnet++ and PraNet in the compared algorithms have the lowest and highest values respectively. The mean intersection over union index of the algorithm of the present invention is increased by 18.49% and 3.05% compared with ResUnet++ and PraNet, and the Dice coefficient index is increased by 22.69% and 2.8% compared with ResUnet++ and PraNet.

[0045] Table 1 Index results of different algorithms on the test set

[0046]

[0047]

[0048] Finally, it should be noted that the above are only the preferred embodiments of the present invention and are not used to limit the present invention. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions recorded in the foregoing embodiments or perform equivalent replacements on some of the technical features. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. An endoscopic image segmentation algorithm with multi-scale feature fusion, including a ResNet50 backbone extraction network, a pyramid feature extraction module PFEM, and a multi-scale fusion module MPFM, characterized in that: The ResNet50 backbone extraction network is used to extract multi-scale polyp features from colonoscopy images; the Pyramid Feature Extraction Module (PFEM) is used to expand the receptive field and extract multi-scale detailed information; the Multi-Proportion Fusion Module (MPFM) adjusts the weights of different features according to the number of inputs, fuses the processed deep and shallow features in different proportions, can strengthen the polyp area, improve the segmentation accuracy of the edge, and make the network perform better in complex environments; The Pyramid Feature Extraction Module (PFEM) includes a first part and a second part. The first part is a convolutional block. After extracting features with a 3×3 convolution, passing through the SiLU activation function, and then adjusting the channels with a 1×1 convolution; the second part removes the branch of the dilated convolution with a dilation rate of r = 24 in the ASPP structure, improves the 4 branches to 3 branches as a module, and uses three parallel dilated convolutions with different dilation rates to complete feature extraction at multiple different scales; The Multi-Proportion Fusion Module (MPFM) is used to fuse the processed deep and shallow features. For the processed first, second, third, and fourth layer feature maps A, B, C, and D, in the first part, the feature map D is upsampled and convolved to obtain D1, D2, D3, and D4. Multiply the obtained D2 by C, then connect it with D1, and smooth the spliced features through a convolutional layer to obtain the feature map C3 of the first part; In the second part, the feature map C is upsampled and passed through a convolutional layer to obtain C1 and C2, and multiply C1 with D3 and B to obtain the feature map B2 of the second part. Splice the features of B2 and C3, and then pass through a 3×3 convolution to obtain the result T2 of deep feature fusion; in the third part, multiply the B1 obtained by upsampling the feature map B and passing through a convolutional layer with C2, D4, and A to obtain the feature map A2 of the third part. Splice A2 with the result T2 of deep feature fusion, and then pass through a 3×3 convolution to smooth the spliced features and adjust the number of channels to obtain the final output result T1 of deep and shallow feature fusion.

2. The endoscopic image segmentation algorithm with multi-scale feature fusion according to claim 1, wherein: The ResNet50 backbone extraction network outputs four-scale feature maps X1, X2, X3, and X4 from low level to high level.

3. The endoscopic image segmentation algorithm with multi-scale feature fusion according to claim 1, characterized in that: The Pyramid Feature Extraction Module (PFEM) contains texture and color detail information.

4. The endoscopic image segmentation algorithm with multi-scale feature fusion according to claim 1, characterized in that: The Multi-Proportion Fusion Module (MPFM) adopts a multi-proportion fusion strategy. By inputting the fourth layer features four times, the third layer features three times, the second layer features two times, and the first layer features once, the proportion of deep features is increased. While obtaining clear semantics, shallow features are fused to strengthen the polyp area and obtain accurate segmentation results.

5. The endoscopic image segmentation algorithm with multi-scale feature fusion according to claim 1, characterized in that: It also includes the following algorithms: The formula for recall is as follows: The Precision formula is as follows: The formula for the mean Intersection over Union (mIoU) is as follows: The Dice coefficient formula is as follows: The accuracy formula is as follows: Among them, TP represents that the pixel on the real image is a polyp and the pixel on the prediction image is also a polyp, TN represents that the pixel on the real image is not a polyp and the pixel on the prediction image is also not a polyp, FP represents that the pixel on the real image is not a polyp while the pixel on the prediction image is a polyp, and FN represents that the pixel on the real image is a polyp while the pixel on the prediction image is not a polyp.

Images