A polyp image semantic segmentation method based on a deep convolutional neural network
By using multi-scale semantic enhancement and cross-layer feature fusion modules, the problem of uncertainty in polyp image segmentation in existing technologies is solved, and the segmentation accuracy of colorectal polyp images is improved, especially when processing complex images.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2023-04-26
- Publication Date
- 2026-03-20
AI Technical Summary
Existing medical image segmentation networks struggle to effectively utilize the complementarity of superficial and deep features when segmenting colorectal polyps, resulting in high semantic segmentation uncertainty, unclear boundaries, and low accuracy in polyp image segmentation.
A multi-scale semantic enhancement module is used to enhance deep features, and a cross-layer feature fusion module is used to upsample layer by layer during the decoding stage. By combining the complementarity of deep and shallow features, the semantic segmentation accuracy of polyp images is improved.
By using multi-scale semantic enhancement and cross-layer feature fusion, the segmentation accuracy of polyp images is significantly improved. In particular, when dealing with polyps of different sizes, colors and textures, the internally complete polyp images can be segmented more clearly.
Smart Images
Figure CN116597138B_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The application belongs to the technical field of medical image processing, and relates to a polyp image semantic segmentation method based on a deep convolutional neural network. BACKGROUND
[0002] A neoplasm growing on the surface of the colorectal mucosa and protruding into the intestinal cavity is called colorectal polyp, which is usually divided into adenomatous and non-adenomatous polyps, and the adenomatous polyp has the possibility of transforming into colorectal cancer, so the diagnosis and resection of the adenomatous colorectal polyp is an important measure to prevent colorectal cancer. Colonoscopy and rectoscopy are effective colorectal polyp detection technologies, and the polyps are segmented from the detection images to assist doctors in disease diagnosis and pathological analysis, improve the diagnosis efficiency and accuracy, and have important significance in clinical practice.
[0003] The uncertainty of intestinal polyp image semantic segmentation mainly has two reasons: polyps of the same type have different sizes, colors and textures; and the boundary between the polyp and the surrounding mucosa is not clear. The current popular medical image segmentation network usually relies on U-Net or a network similar to U-Net (for example: U-Net++, ResUNet, etc.). These models are essentially encoding and decoding frameworks, and all levels of features in the encoding stage are used for decoding and segmentation map prediction with the same weight, but compared with deep features, shallow features have a larger spatial resolution and need more computing resources, but contribute less to the performance of polyp segmentation. SUMMARY
[0004] The application aims to overcome the defects of the prior art and provide a polyp image semantic segmentation method based on a deep convolutional neural network, which can focus on deep features in the encoding stage to extract high-level semantic information of the polyp, and adopt a cross-layer fusion method in the decoding stage to comprehensively consider the complementarity and correlation of deep and shallow features, thereby improving the accuracy of polyp image semantic segmentation.
[0005] To solve the above technical problems, the application adopts the following technical scheme.
[0006] A polyp image semantic segmentation method based on a deep convolutional neural network comprises the following steps:
[0007] Step 1: inputting a polyp image for training into a VGG16 encoder, using the encoder to extract features of different levels of the image, and dividing the features into shallow features and deep features according to the depth of the convolutional layer;
[0008] Step 2: applying a multi-scale semantic enhancement module to respectively enhance the features of the deep features X3, X4 and X5, to obtain enhanced features corresponding to each deep feature;
[0009] Step 3, the enhanced features Z3, Z4, Z5 are fused to obtain a global enhanced feature F5;
[0010] Step 4, the global enhanced feature F5 and the deep layer feature X4 are input into a cross-layer feature fusion module to obtain a cross-layer fusion feature F4; F4 and X3 are input into the cross-layer feature fusion module to obtain a cross-layer fusion feature F3; F3 and X2 are input into the cross-layer feature fusion module to obtain a cross-layer fusion feature F2; F2 and X1 are input into the cross-layer feature fusion module to obtain a cross-layer fusion feature F1;
[0011] Step 5, F1 passes through a convolution layer with a convolution kernel number of 1, a padding of 2, and a convolution kernel size of 3x3, and a Sigmoid function to obtain a predicted polyp segmentation map S;
[0012] Step 6, the network is trained to minimize the cross-entropy loss function between the predicted polyp segmentation map S and the ground truth map G, and the network parameters are updated using loss backpropagation;
[0013] Cross-entropy loss function:
[0014] Where N is the number of pixels, y i is the true class of the i-th pixel obtained from the ground truth map G, is the probability that the i-th pixel obtained from the predicted polyp segmentation map S is a polyp;
[0015] Step 7, input the test image into the trained network to obtain a polyp segmentation map, and use mIoU to evaluate the performance of polyp segmentation.
[0016] Specifically, in step 1, the training image is input into the VGG16 encoder, and the features of different levels of the image are extracted using the encoder. The features are divided into shallow layer features and deep layer features according to the depth of the convolution layer, including the following processes:
[0017] 1.1 Scale the training image to a uniform size of 224x224 and input it into the VGG16 encoder;
[0018] 1.2 Divide the VGG16 encoder into 5 modules according to the depth of the convolution layer, each module corresponding to a lateral output. The output results obtained from shallow to deep are denoted as X1, X2, X3, X4, X5, with channel numbers and sizes of 64x224x224, 128x112x112, 256x56x56, 512x28x28, and 512x14x14, respectively;
[0019] 1.3 X1 and X2 are referred to as shallow layer features, and X3, X4, and X5 are referred to as deep layer features.
[0020] Specifically, in step 2, the multi-scale semantic enhancement module is applied to enhance the deep features X3, X4 and X5 respectively, and the enhanced features corresponding to each deep feature are obtained, including:
[0021] 2.1. applying a multi-scale semantic enhancement module to enhance the deep feature X5:
[0022] 2.1.1 inputting the deep feature X5 into a deep convolution layer with a convolution kernel size of 7x7 and a convolution layer with a convolution kernel size of 1x1 in sequence, wherein the number of convolution kernels of the deep convolution layer is 1 and the padding is 3, and the number of convolution kernels of the convolution layer is 128 and the padding is 0, to obtain a feature map with a channel number and a size of 128x14x14;
[0023] 2.1.2 inputting X5 into a deep convolution layer with a convolution kernel size of 5x5 and a convolution layer with a convolution kernel size of 1x1 in sequence, wherein the number of convolution kernels of the deep convolution layer is 1 and the padding is 2, and the number of convolution kernels of the convolution layer is 128 and the padding is 0, to obtain a feature map with a channel number and a size of 128x14x14;
[0024] 2.1.3 inputting X5 into a deep convolution layer with a convolution kernel size of 3x3 and a convolution layer with a convolution kernel size of 1x1 in sequence, wherein the number of convolution kernels of the deep convolution layer is 1 and the padding is 1, and the number of convolution kernels of the convolution layer is 128 and the padding is 0, to obtain a feature map with a channel number and a size of 128x14x14;
[0025] 2.1.4 inputting X5 into a deep convolution layer with a convolution kernel size of 1x1 and a convolution layer with a convolution kernel size of 1x1 in sequence, wherein the number of convolution kernels of the deep convolution layer is 1 and the padding is 0, and the number of convolution kernels of the convolution layer is 128 and the padding is 0, to obtain a feature map with a channel number and a size of 128x14x14;
[0026] 2.1.5 concatenating the four feature maps obtained in 2.1-2.4 in the channel dimension to obtain a feature map with a channel number and a size of 512x14x14, and then passing through a channel attention module, a convolution layer with a convolution kernel size of 1x1, batch normalization and a Relu activation layer to obtain a multi-scale enhanced feature Z5, the channel number and size of Z5 are consistent with X5, which are 512x14x14.
[0027] Specifically, in step 2, the multi-scale semantic enhancement module is applied to enhance the deep features X3, X4 and X5 respectively, and the enhanced features corresponding to each deep feature are obtained, including:
[0028] 2.2. applying a multi-scale semantic enhancement module to enhance the deep feature X4:
[0029] 2.2.1 input the deep feature X4 into a deep convolutional layer with a convolution kernel size of 7x7 and a convolution layer with a convolution kernel size of 1x1 in sequence, wherein the number of convolution kernels of the deep convolutional layer is 1 and the padding is 3, and the number of convolution kernels of the convolution layer is 128 and the padding is 0, to obtain a feature map with a channel number and a size of 128x28x28;
[0030] 2.2.2 input X4 into a deep convolutional layer with a convolution kernel size of 5x5 and a convolution layer with a convolution kernel size of 1x1 in sequence, wherein the number of convolution kernels of the deep convolutional layer is 1 and the padding is 2, and the number of convolution kernels of the convolution layer is 128 and the padding is 0, to obtain a feature map with a channel number and a size of 128x28x28;
[0031] 2.2.3 input X4 into a deep convolutional layer with a convolution kernel size of 3x3 and a convolution layer with a convolution kernel size of 1x1 in sequence, wherein the number of convolution kernels of the deep convolutional layer is 1 and the padding is 1, and the number of convolution kernels of the convolution layer is 128 and the padding is 0, to obtain a feature map with a channel number and a size of 128x28x28;
[0032] 2.2.4 input X4 into a deep convolutional layer with a convolution kernel size of 1x1 and a convolution layer with a convolution kernel size of 1x1 in sequence, wherein the number of convolution kernels of the deep convolutional layer is 1 and the padding is 0, and the number of convolution kernels of the convolution layer is 128 and the padding is 0, to obtain a feature map with a channel number and a size of 128x28x28;
[0033] 2.2.5 concatenate the four feature maps obtained in 2.2.1-2.2.4 in the channel dimension to obtain a feature map with a channel number and a size of 512x28x28, and then pass through a channel attention module, a convolution layer with a convolution kernel size of 1x1, batch normalization, and a Relu activation layer to obtain a multi-scale enhanced feature Z4, which has the same channel number and size as X4, i.e., 512x28x28.
[0034] Further, in step 2, the multi-scale semantic enhancement module is applied to each of the deep features X3, X4, and X5 for feature enhancement to obtain an enhanced feature corresponding to each deep feature, including:
[0035] 2.3 applying a multi-scale semantic enhancement module to the deep feature X3 for feature enhancement:
[0036] 2.3.1 input the deep feature X3 into a deep convolutional layer with a convolution kernel size of 7x7 and a convolution layer with a convolution kernel size of 1x1 in sequence, wherein the number of convolution kernels of the deep convolutional layer is 1 and the padding is 3, the number of convolution kernels of the convolution layer is 128 and the padding is 0, to obtain a feature map with a channel number and a size of 64x56x56;
[0037] 2.3.2 input X3 into a deep convolutional layer with a convolution kernel size of 5x5 and a convolution layer with a convolution kernel size of 1x1 in sequence, wherein the number of convolution kernels of the deep convolutional layer is 1 and the padding is 2, the number of convolution kernels of the convolution layer is 128 and the padding is 0, to obtain a feature map with a channel number and a size of 64x56x56;
[0038] 2.3.3 input X3 into a deep convolutional layer with a convolution kernel size of 3x3 and a convolution layer with a convolution kernel size of 1x1 in sequence, wherein the number of convolution kernels of the deep convolutional layer is 1 and the padding is 1, the number of convolution kernels of the convolution layer is 128 and the padding is 0, to obtain a feature map with a channel number and a size of 64x56x56;
[0039] 2.3.4 input X3 into a deep convolutional layer with a convolution kernel size of 1x1 and a convolution layer with a convolution kernel size of 1x1 in sequence, wherein the number of convolution kernels of the deep convolutional layer is 1 and the padding is 0, the number of convolution kernels of the convolution layer is 128 and the padding is 0, to obtain a feature map with a channel number and a size of 64x56x56;
[0040] 2.3.5 concatenate the four feature maps obtained in 2.3.1-2.3.4 in the channel dimension to obtain a feature map with a channel number and a size of 256x56x56, and then pass through a channel attention module, a convolution layer with a convolution kernel size of 1x1, batch normalization and a Relu activation layer to obtain a multi-scale enhanced feature Z3, the channel number and size of Z3 are consistent with those of X3, i.e., 256x56x56.
[0041] Specifically, in step 3, the enhanced features Z3, Z4 and Z5 are fused to obtain a global enhanced feature F5, and the process includes:
[0042] 3.1 perform 4 times interpolation upsampling on the enhanced feature Z5, and the size becomes 512x56x56;
[0043] 3.2 perform 2 times interpolation upsampling on the enhanced feature Z4, and the size becomes 512x56x56;
[0044] 3.3 Concatenate the enhanced features Z5, Z4 and Z3 in the channel dimension, sequentially pass through a channel attention module, a convolution layer with a kernel size of 3*3, batch normalization and a Relu activation layer to obtain a global enhanced feature F5 with a channel number and size of 512*56*56.
[0045] Specifically, the step 4 comprises the following processes:
[0046] 4.1 Scale F5 to the same size as X4 using an interpolation method, and then input F5 and X4 into a cross-layer feature fusion module;
[0047] 4.2 sequentially pass F5 through a maximum pooling layer, a convolution layer with a kernel size of 1*1 to obtain a feature map with a channel number of 1, i.e., 1*28*28; then pass through batch normalization, a Relu activation layer, a Sigmoid function and channel replication to obtain a feature map A, which has the same channel number and size as X4, i.e., 512*28*28;
[0048] 4.3 sequentially pass F5 through a depth convolution layer with a kernel size of 1*1, a convolution layer with a kernel size of 1*1, batch normalization and a Relu activation layer to obtain a feature map Y, which has the same channel number and size as X4, i.e., 512*28*28;
[0049] 4.4 add corresponding position elements of Y and A, and multiply corresponding position elements of X4 and A to obtain two feature maps, both of which have a size of 512*28*28;
[0050] 4.5 concatenate the two features obtained in 4.4 in the channel dimension, and then pass through a convolution layer with a kernel size of 3*3, batch normalization and a Relu activation layer to obtain a fusion feature F4 with a size of 512*28*28;
[0051] 4.6 scale F4 to the same size as X3 using an interpolation method, input F4 and X3 into a cross-layer feature fusion module, and execute steps 4.2-4.5 to obtain a fusion feature F3 with a channel number and size of 256*56*56;
[0052] 4.7 scale F3 to the same size as X2 using an interpolation method, input F3 and X2 into a cross-layer feature fusion module, and execute steps 4.2-4.5 to obtain a fusion feature F2 with a channel number and size of 128*112*112;
[0053] 4.8 scale F2 to the same size as X1 using an interpolation method, input F2 and X1 into a cross-layer feature fusion module, and execute steps 4.2-4.5 to obtain a fusion feature F1 with a channel number and size of 64*224*224.
[0054] Compared with the prior art, the present application has the following advantages and beneficial effects:
[0055] (1) The application adopts a multi-scale semantic enhancement module, uses deep convolution of different sizes to analyze and fuse deep features, and effectively extracts high-level semantic information of the polyp image, including target object positioning and global semantic information;
[0056] (2) The cross-layer feature fusion module of the application decodes by layer-by-layer upsampling from deep to shallow in the decoding stage, fully utilizes the global semantic information in the deep features, uses the color, texture and other information in the shallow features as auxiliary, and segments the polyp image with complete internal and clear edges, effectively improving the accuracy of polyp image semantic segmentation. BRIEF DESCRIPTION OF DRAWINGS
[0057] Figure 1 is a method flowchart of an embodiment of the application.
[0058] Figure 2 is a polyp segmentation deep convolutional neural network structure diagram of an embodiment of the application.
[0059] Figure 3 is a multi-scale feature enhancement (MFE) module structure diagram of an embodiment of the application.
[0060] Figure 4 is a cross-layer feature fusion (CLFF) module structure diagram of an embodiment of the application.
[0061] Figure 5 is a segmentation result instance comparison diagram. DETAILED DESCRIPTION
[0062] The polyp image semantic segmentation method based on a deep convolutional neural network of the application comprises the following steps: using a deep convolutional neural network model as an encoder to extract image multi-layer features, and dividing the features into shallow features and deep features according to the depth of the convolutional layer; using a multi-scale semantic enhancement module to enhance the semantic of the deep features; fusing the obtained multiple enhanced features into global enhanced features; inputting the obtained global enhanced features into a decoding end, using a cross-layer feature fusion module to decode by layer-by-layer upsampling from deep to shallow, and obtaining a predicted segmentation map. The same type of polyp usually has different sizes, colors and textures, and the boundary between the polyp and the surrounding mucosa is blurred, so it is difficult to correctly segment the target area by using shallow features, and therefore the method focuses on deep features in the encoding stage. The multi-scale semantic enhancement module is used to enhance the deep features, improve the feature expression ability, the cross-layer feature fusion module is used to decode by layer-by-layer upsampling from deep to shallow, fully utilizes the complementarity and correlation of adjacent layer features, and effectively improves the accuracy of polyp image semantic segmentation. The polyp segmentation data set used in the method of the application is from the network.
[0063] The application will be further described in detail below with reference to the accompanying drawings.
[0064] A polyp image semantic segmentation method based on a deep convolutional neural network according to the application is shown in Fig. 1, which uses a deep convolutional neural network model as an encoder, inputs a training image into the encoder, extracts shallow and deep features of the image, enhances the deep features by using a multi-scale feature enhancement module, decodes by upsampling using a cross-layer fusion module until the network output is consistent with the size of the original image, and trains the network to minimize the cross-entropy loss between the segmentation image and the true value image of the training image. The method comprises the following steps: Figure 1 Step 1: input the training image into the VGG16 encoder, extract the features of different levels of the image using the encoder, and divide the features into shallow features and deep features according to the depth of the convolutional layer. As shown in Fig. 2, the specific implementation method of this step is as follows:
[0065] Figure 2
[0066] 1.1: scale the polyp image used for training to a uniform size of 224x224 and input it into the VGG16 encoder;
[0067] 1.2: divide the VGG16 encoder into 5 modules according to the depth of the convolutional layer, each module corresponds to a lateral output, and the output results obtained from shallow to deep are denoted as X1, X2, X3, X4, and X5, with the channel number and size being 64x224x224, 128x112x112, 256x56x56, 512x28x28, and 512x14x14, respectively;
[0068] 1.3: regard X1 and X2 as shallow features, and regard X3, X4, and X5 as deep features.
[0069] Step 2: apply a multi-scale semantic enhancement module to respectively enhance the deep features X3, X4, and X5, and obtain the enhanced features corresponding to each deep feature. As shown in Fig. 3, the specific implementation method of this step is as follows: Figure 3
[0070] 2.1: apply the multi-scale semantic enhancement module to enhance the deep feature X5:
[0071] 2.1.1: input the deep feature X5 into a deep convolutional layer with a convolution kernel size of 7x7 and a convolution layer with a convolution kernel size of 1x1 in sequence, wherein the number of convolution kernels of the deep convolutional layer is 1 and the padding is 3, the number of convolution kernels of the convolution layer is 128 and the padding is 0, and a feature map with a channel number and size of 128x14x14 is obtained;
[0072] 2.1.2 input X5 into a deep convolutional layer with a convolution kernel size of 5x5 and a convolution layer with a convolution kernel size of 1x1 in sequence, wherein the number of convolution kernels of the deep convolutional layer is 1 and the padding is 2, and the number of convolution kernels of the convolution layer is 128 and the padding is 0, to obtain a feature map with a channel number and a size of 128x14x14;
[0073] 2.1.3 input X5 into a deep convolutional layer with a convolution kernel size of 3x3 and a convolution layer with a convolution kernel size of 1x1 in sequence, wherein the number of convolution kernels of the deep convolutional layer is 1 and the padding is 1, and the number of convolution kernels of the convolution layer is 128 and the padding is 0, to obtain a feature map with a channel number and a size of 128x14x14;
[0074] 2.1.4 input X5 into a deep convolutional layer with a convolution kernel size of 1x1 and a convolution layer with a convolution kernel size of 1x1 in sequence, wherein the number of convolution kernels of the deep convolutional layer is 1 and the padding is 0, and the number of convolution kernels of the convolution layer is 128 and the padding is 0, to obtain a feature map with a channel number and a size of 128x14x14;
[0075] 2.1.5 concatenate the four feature maps obtained in 2.1.1-2.1.4 in the channel dimension to obtain a feature map with a channel number and a size of 512x14x14, and then pass the feature map through a channel attention module, a convolution layer with a convolution kernel size of 1x1, a batch normalization layer, and a Relu activation layer to obtain a multi-scale enhanced feature Z5, wherein the channel number and the size of Z5 are consistent with those of X5, i.e., 512x14x14;
[0076] 2.2 apply a multi-scale semantic enhancement module to feature enhance the deep feature X4:
[0077] 2.2.1 input the deep feature X4 into a deep convolutional layer with a convolution kernel size of 7x7 and a convolution layer with a convolution kernel size of 1x1 in sequence, wherein the number of convolution kernels of the deep convolutional layer is 1 and the padding is 3, and the number of convolution kernels of the convolution layer is 128 and the padding is 0, to obtain a feature map with a channel number and a size of 128x28x28;
[0078] 2.2.2 input X4 into a deep convolutional layer with a convolution kernel size of 5x5 and a convolution layer with a convolution kernel size of 1x1 in sequence, wherein the number of convolution kernels of the deep convolutional layer is 1 and the padding is 2, and the number of convolution kernels of the convolution layer is 128 and the padding is 0, to obtain a feature map with a channel number and a size of 128x28x28;
[0079] 2.2.3 input X4 into a deep convolution layer with a convolution kernel size of 3x3 and a convolution layer with a convolution kernel size of 1x1 in sequence, wherein the number of convolution kernels of the deep convolution layer is 1 and the padding is 1, the number of convolution kernels of the convolution layer is 128 and the padding is 0, to obtain a feature map with a channel number and a size of 128x28x28;
[0080] 2.2.4 input X4 into a deep convolution layer with a convolution kernel size of 1x1 and a convolution layer with a convolution kernel size of 1x1 in sequence, wherein the number of convolution kernels of the deep convolution layer is 1 and the padding is 0, the number of convolution kernels of the convolution layer is 128 and the padding is 0, to obtain a feature map with a channel number and a size of 128x28x28;
[0081] 2.2.5 concatenate the four feature maps obtained in 2.2.1-2.2.4 in the channel dimension to obtain a feature map with a channel number and a size of 512x28x28, and then pass through a channel attention module, a convolution layer with a convolution kernel size of 1x1, batch normalization and a Relu activation layer to obtain a multi-scale enhanced feature Z4, the channel number and size of Z4 are consistent with X4, being 512x28x28.
[0082] 2.3 apply a multi-scale semantic enhancement module to feature enhancement on deep feature X3:
[0083] 2.3.1 input deep feature X3 into a deep convolution layer with a convolution kernel size of 7x7 and a convolution layer with a convolution kernel size of 1x1 in sequence, wherein the number of convolution kernels of the deep convolution layer is 1 and the padding is 3, the number of convolution kernels of the convolution layer is 128 and the padding is 0, to obtain a feature map with a channel number and a size of 64x56x56;
[0084] 2.3.2 input X3 into a deep convolution layer with a convolution kernel size of 5x5 and a convolution layer with a convolution kernel size of 1x1 in sequence, wherein the number of convolution kernels of the deep convolution layer is 1 and the padding is 2, the number of convolution kernels of the convolution layer is 128 and the padding is 0, to obtain a feature map with a channel number and a size of 64x56x56;
[0085] 2.3.3 input X3 into a deep convolution layer with a convolution kernel size of 3x3 and a convolution layer with a convolution kernel size of 1x1 in sequence, wherein the number of convolution kernels of the deep convolution layer is 1 and the padding is 1, the number of convolution kernels of the convolution layer is 128 and the padding is 0, to obtain a feature map with a channel number and a size of 64x56x56;
[0086] 2.3.4 Input X3 sequentially into a depthwise convolutional layer with a kernel size of 1×1 and a convolutional layer with a kernel size of 1×1. The depthwise convolutional layer has 1 kernel and 0 padding, and the convolutional layer has 128 kernels and 0 padding, to obtain a feature map with 64×56×56 channels.
[0087] 2.3.5 The four feature maps obtained in 2.3.1 to 2.3.4 are concatenated along the channel dimension to obtain a feature map with a channel number and size of 256×56×56. Then, through a channel attention module, a convolutional layer with a kernel size of 1×1, batch normalization, and a ReLU activation layer, a multi-scale enhanced feature Z3 is obtained. The number of channels and size of Z3 are consistent with X3, which is 256×56×56.
[0088] Step 3: Fuse the enhanced features Z3, Z4, and Z5 to obtain the global enhanced feature F5; the specific implementation method of this step is as follows:
[0089] 3.1 The enhanced feature Z5 is upsampled by 4 times through interpolation, and its size becomes 512×56×56;
[0090] 3.2 The enhanced feature Z4 is upsampled by 2 times through interpolation, and its size becomes 512×56×56;
[0091] 3.3 The enhanced features Z5, Z4, and Z3 are concatenated along the channel dimension and then passed through a channel attention module, a convolutional layer with a kernel size of 3×3, batch normalization, and a ReLU activation layer to obtain the global enhanced feature F5, which has a channel number and size of 512×56×56.
[0092] Step 4: Input the global enhancement feature F5 and the deep feature X4 into the cross-layer feature fusion module to obtain the cross-layer fused feature F4; input F4 and X3 into the cross-layer feature fusion module to obtain the cross-layer fused feature F3; input F3 and X2 into the cross-layer feature fusion module to obtain the cross-layer fused feature F2; input F2 and X1 into the cross-layer feature fusion module to obtain the cross-layer fused feature F1. For example... Figure 4 As shown, the specific implementation method of this step is as follows:
[0093] 4.1 Use interpolation to scale F5 to the same size as X4, and then input F5 and X4 into the cross-layer feature fusion module;
[0094] 4.2 F5 is passed through max pooling and convolutional layers with a kernel size of 1×1 to obtain a feature map with 1 channel, i.e., 1×28×28; then through batch normalization, ReLU activation layer, Sigmoid function, and channel copying, feature map A is obtained. The number of channels and size of A and X4 are the same, i.e., 512×28×28.
[0095] 4.3 F5 is sequentially passed through a deep convolutional layer with a convolution kernel size of 1x1, a convolutional layer with a convolution kernel size of 1x1, batch normalization, and a Relu activation layer to obtain a feature map Y, which is consistent with the channel number and size of X4, and is 512x28x28;
[0096] 4.4 Y and A are added at corresponding positions, and X4 and A are multiplied at corresponding positions to obtain two feature maps, both of which have a size of 512x28x28;
[0097] 4.5 The two features obtained in 4.4 are channel-spliced, and then passed through a convolutional layer with a convolution kernel size of 3x3, batch normalization, and a Relu activation layer to obtain a fused feature F4 with a size of 512x28x28;
[0098] 4.6 F4 is scaled to the same size as X3 using the interpolation method, and F4 and X3 are input into the cross-layer feature fusion module to perform steps 4.2-4.5 to obtain a fused feature F3 with a channel number and size of 256x56x56;
[0099] 4.7 F3 is scaled to the same size as X2 using the interpolation method, and F3 and X2 are input into the cross-layer feature fusion module to perform steps 4.2-4.5 to obtain a fused feature F2 with a channel number and size of 128x112x112;
[0100] 4.8 F2 is scaled to the same size as X1 using the interpolation method, and F2 and X1 are input into the cross-layer feature fusion module to perform steps 4.2-4.5 to obtain a fused feature F1 with a channel number and size of 64x224x224;
[0101] Step 5, as shown in Figure 2 , F1 is passed through a convolutional layer with a convolution kernel number of 1, a padding of 2, a convolution kernel size of 3x3, and a Sigmoid function to obtain a predicted polyp segmentation map S;
[0102] Step 6, the network is trained to minimize the cross-entropy loss function between the predicted polyp segmentation map S and the ground truth map G, and the network parameters are updated using loss backpropagation.
[0103] Cross-entropy loss function:
[0104] where N is the number of pixels, y i is the true class of the i-th pixel obtained from the ground truth map G, is the probability that the i-th pixel obtained from the predicted polyp segmentation map S is a polyp.
[0105] Step 7, input the test image into the trained network to obtain a polyp segmentation graph, and adopt mIoU (Mean Intersection over Union) to evaluate the performance of polyp segmentation.
[0106] The following is an experiment performed according to the method of the present application, which illustrates the effect of the present application.
[0107] Experimental environment: PyTorch 1.9.0; Python 3.8 (ubuntu18.04); Cuda 11.1; GPU RTX3090*1.
[0108] Dataset: The data set used in the experiment is the Kvasir-SEG public data set for intestinal polyp segmentation, of which 880 are used as training data and 120 are used as test data.
[0109] Test index: The present application uses mIoU (Mean Intersection over Union) as the evaluation index.
[0110] Table 1 shows the score comparison of the method of the present application and Unet, Unet++, ResUnet, ResUnet++, PraNet, and MKDCNet methods trained on the Kvasir-SEG data set.
[0111] Figure 5 The segmentation result examples of the method of the present application and Unet, Unet++, PraNet, and MKDCNet methods are shown.
[0112] Table 1
[0113]
[0114] In summary, the test result score of the method of the present application is higher than that of other models in Table 1. Figure 5 In the present application, it can be seen that the method of the present application effectively avoids many loss or misjudgment situations. The present application has more advantages in analyzing complex images and unobservable polyp targets.
Claims
1. A semantic segmentation method for polyp images based on deep convolutional neural networks, characterized in that, Includes the following steps: Step 1: Input the polyp image used for training into the VGG16 encoder, use the encoder to extract features at different levels of the image, and divide the features into shallow features and deep features according to the depth of the convolutional layer. Step 2: Apply the multi-scale semantic enhancement module to enhance the deep features X3, X4, and X5 respectively, and obtain the enhanced features corresponding to each deep feature; Step 3: Fuse the enhanced features Z3, Z4, and Z5 to obtain the global enhanced feature F5; Step 4: Input the global enhancement feature F5 and the deep feature X4 into the cross-layer feature fusion module to obtain the cross-layer fusion feature F4; input F4 and X3 into the cross-layer feature fusion module to obtain the cross-layer fusion feature F3; input F3 and X2 into the cross-layer feature fusion module to obtain the cross-layer fusion feature F2; input F2 and X1 into the cross-layer feature fusion module to obtain the cross-layer fusion feature F1. Step 5: F1 passes through a convolutional layer with 1 kernel, 2 padding, and 3×3 kernel size, followed by a sigmoid function, to obtain the predicted polyp segmentation map S; Step 6: Train the network with the objective of minimizing the cross-entropy loss function between the predicted polyp segmentation map S and the ground truth map G, and update the network parameters using backpropagation of the loss. Cross-entropy loss function: ; Where N is the number of pixels, y i Let ŷ be the true category of the i-th pixel obtained from the truth map G. i Let S be the probability that the i-th pixel obtained from the predicted polyp segmentation map S is a polyp; Step 7: Input the test image into the trained network to obtain the polyp segmentation map, and use mIoU uniform intersection and comparison to evaluate the polyp segmentation performance; In step 2, the multi-scale semantic enhancement module is used to enhance the deep features X3, X4, and X5 respectively, resulting in enhanced features for each deep feature, including: 2.1 Applying a multi-scale semantic enhancement module to enhance the deep feature X5: 2.1.1 Input the deep feature X5 into a depth convolutional layer with a kernel size of 7×7 and a convolutional layer with a kernel size of 1×1 in sequence. The depth convolutional layer has 1 kernel and 3 padding, and the convolutional layer has 128 kernels and 0 padding, to obtain a feature map with 128×14×14 channels. 2.1.2 Input X5 sequentially into a depthwise convolutional layer with a kernel size of 5×5 and a convolutional layer with a kernel size of 1×1. The depthwise convolutional layer has 1 kernel and 2 padding, while the convolutional layer has 128 kernels and 0 padding, resulting in a feature map with 128×14×14 channels. 2.1.3 Input X5 sequentially into a depthwise convolutional layer with a kernel size of 3×3 and a convolutional layer with a kernel size of 1×1. The depthwise convolutional layer has 1 kernel and 1 padding, while the convolutional layer has 128 kernels and 0 padding, resulting in a feature map with 128×14×14 channels. 2.1.4 Input X5 sequentially into a depthwise convolutional layer with a kernel size of 1×1 and a convolutional layer with a kernel size of 1×1. The depthwise convolutional layer has 1 kernel and 0 padding, while the convolutional layer has 128 kernels and 0 padding, resulting in a feature map with 128×14×14 channels. 2.1.5 The four feature maps obtained in 2.1.1 to 2.1.4 are concatenated along the channel dimension to obtain a feature map with a channel number and size of 512×14×14. Then, through a channel attention module, a convolutional layer with a kernel size of 1×1, batch normalization, and a ReLU activation layer, a multi-scale enhanced feature Z5 is obtained. The number of channels and size of Z5 are consistent with X5, which is 512×14×14.
2. The polyp image semantic segmentation method based on a deep convolutional neural network according to claim 1, characterized in that, in step 1, the training image is input into a VGG16 encoder, the encoder is used to extract features at different levels of the image, and the features are divided into shallow features and deep features according to the depth of the convolutional layer, including the following process: 1.1 Scale the training images to a uniform size of 224×224 and input them into the VGG16 encoder; 1.2 The VGG16 encoder is divided into 5 modules according to the depth of the convolutional layer. Each module corresponds to a lateral output. The output results obtained from shallow to deep are denoted as X1, X2, X3, X4, and X5, and their channel numbers and sizes are 64×224×224, 128×112×112, 256×56×56, 512×28×28, and 512×14×14, respectively. 1.3 X1 and X2 are referred to as shallow features, and X3, X4 and X5 are referred to as deep features.
3. The method for semantic segmentation of polyp images based on deep convolutional neural networks according to claim 1, characterized in that, In step 2, the multi-scale semantic enhancement module is used to enhance the deep features X3, X4, and X5 respectively, to obtain the enhanced features corresponding to each deep feature. The process includes: 2.2 Applying a multi-scale semantic enhancement module to enhance the deep feature X4: 2.2.1 Input the deep feature X4 into a depth convolutional layer with a kernel size of 7×7 and a convolutional layer with a kernel size of 1×1 in sequence. The depth convolutional layer has 1 kernel and 3 padding, and the convolutional layer has 128 kernels and 0 padding, to obtain a feature map with 128×28×28 channels. 2.2.2 Input X4 sequentially into a depthwise convolutional layer with a kernel size of 5×5 and a convolutional layer with a kernel size of 1×1. The depthwise convolutional layer has 1 kernel and 2 padding, while the convolutional layer has 128 kernels and 0 padding, resulting in a feature map with 128×28×28 channels. 2.2.3 Input X4 sequentially into a depthwise convolutional layer with a kernel size of 3×3 and a convolutional layer with a kernel size of 1×1. The depthwise convolutional layer has 1 kernel and 1 padding, while the convolutional layer has 128 kernels and 0 padding, resulting in a feature map with 128×28×28 channels. 2.2.4 Input X4 sequentially into a depthwise convolutional layer with a kernel size of 1×1 and a convolutional layer with a kernel size of 1×1. The depthwise convolutional layer has 1 kernel and 0 padding, and the convolutional layer has 128 kernels and 0 padding, to obtain a feature map with 128×28×28 channels. 2.2.5 The four feature maps obtained in 2.2.1 to 2.2.4 are concatenated along the channel dimension to obtain a feature map with a channel number and size of 512×28×28. Then, through the channel attention module, a convolutional layer with a kernel size of 1×1, batch normalization, and ReLU activation layer, a multi-scale enhancement feature Z4 is obtained. The number of channels and size of Z4 are consistent with X4, which is 512×28×28.
4. The method for semantic segmentation of polyp images based on deep convolutional neural networks according to claim 1, characterized in that, In step 2, the multi-scale semantic enhancement module is used to enhance the deep features X3, X4, and X5 respectively, resulting in enhanced features for each deep feature, including: 2.3 Applying a multi-scale semantic enhancement module to enhance the deep feature X3: 2.3.1 Input the deep feature X3 into a depth convolutional layer with a kernel size of 7×7 and a convolutional layer with a kernel size of 1×1 in sequence. The depth convolutional layer has 1 kernel and 3 padding, and the convolutional layer has 128 kernels and 0 padding, to obtain a feature map with 64×56×56 channels. 2.3.2 Input X3 sequentially into a depthwise convolutional layer with a kernel size of 5×5 and a convolutional layer with a kernel size of 1×1. The depthwise convolutional layer has 1 kernel and 2 padding, while the convolutional layer has 128 kernels and 0 padding, resulting in a feature map with 64×56×56 channels. 2.3.3 Input X3 sequentially into a depthwise convolutional layer with a kernel size of 3×3 and a convolutional layer with a kernel size of 1×1. The depthwise convolutional layer has 1 kernel and 1 padding, while the convolutional layer has 128 kernels and 0 padding, resulting in a feature map with 64×56×56 channels. 2.3.4 Input X3 sequentially into a depthwise convolutional layer with a kernel size of 1×1 and a convolutional layer with a kernel size of 1×1. The depthwise convolutional layer has 1 kernel and 0 padding, and the convolutional layer has 128 kernels and 0 padding, to obtain a feature map with 64×56×56 channels. 2.3.5 The four feature maps obtained in 2.3.1 to 2.3.4 are concatenated along the channel dimension to obtain a feature map with a channel number and size of 256×56×56. Then, through a channel attention module, a convolutional layer with a kernel size of 1×1, batch normalization, and a ReLU activation layer, a multi-scale enhanced feature Z3 is obtained. The number of channels and size of Z3 are consistent with X3, which is 256×56×56.
5. The method for semantic segmentation of polyp images based on deep convolutional neural networks according to claim 1, characterized in that, In step 3, the enhanced features Z3, Z4, and Z5 are fused to obtain the global enhanced feature F5. The process includes: 3.1 The enhanced feature Z5 is upsampled by 4 times through interpolation, and its size becomes 512×56×56; 3.2 The enhanced feature Z4 is upsampled by 2 times through interpolation, and its size becomes 512×56×56; 3.3 The enhanced features Z5, Z4, and Z3 are concatenated along the channel dimension and then passed through a channel attention module, a convolutional layer with a kernel size of 3×3, batch normalization, and a ReLU activation layer to obtain the global enhanced feature F5, which has a channel number and size of 512×56×56.
6. The method for semantic segmentation of polyp images based on deep convolutional neural networks according to claim 1, characterized in that, Step 4, Includes the following processes: 4.1 Use interpolation to scale F5 to the same size as X4, and then input F5 and X4 into the cross-layer feature fusion module; 4.2 F5 is passed through max pooling and convolutional layers with a kernel size of 1×1 to obtain a feature map with 1 channel, i.e., 1×28×28; then through batch normalization, ReLU activation layer, Sigmoid function, and channel copying, feature map A is obtained. The number of channels and size of A and X4 are the same, i.e., 512×28×28. 4.3 F5 is passed through a depthwise convolutional layer with a kernel size of 1×1, a convolutional layer with a kernel size of 1×1, a batch normalization, and a ReLU activation layer in sequence to obtain the feature map Y. The number of channels and the size of Y are the same as those of X4, which are 512×28×28. 4.4 Add the corresponding elements of Y to A, and multiply the corresponding elements of X4 to A to obtain two feature maps, each with a size of 512×28×28; 4.5 The two features obtained in 4.4 are concatenated by channels, and then the fused feature F4 is obtained by passing it through a convolutional layer with a kernel size of 3×3, batch normalization, and ReLU activation layer, with a size of 512×28×28. 4.6 Use interpolation to scale F4 to the same size as X3. Input F4 and X3 into the cross-layer feature fusion module and execute steps 4.2 to 4.5 to obtain the fused feature F3, with the number of channels and size of 256×56×56. 4.7 Use interpolation to scale F3 to the same size as X2. Input F3 and X2 into the cross-layer feature fusion module and execute steps 4.2 to 4.5 to obtain the fused feature F2, with the number of channels and size being 128×112×112. 4.8 Use interpolation to scale F2 to the same size as X1. Input F2 and X1 into the cross-layer feature fusion module and execute steps 4.2 to 4.5 to obtain the fused feature F1, with the number of channels and size being 64×224×224.