Image semantic segmentation method based on staggered dilated convolution UNet

Abstract

The image semantic segmentation method based on the interleaved dilated convolution UNet is applicable to the field of computer vision. This method uses an interleaved dilated convolution module and an edge-preserving module. The interleaved dilated convolution module uses an interleaved fusion method to avoid adding features representing different objects and effectively integrate features distributed across different channels. The edge-preserving module obtains edge features by subtracting the outputs of convolutional layers with different convolution kernels, integrating and adding them to the output features. This module has the ability to sharpen feature edge information, thereby enhancing the model's ability to predict edges. The main technical issues addressed by this method include feature fusion in multiple receptive fields and detailed optimization of image semantic segmentation, thereby achieving better semantic segmentation performance.

Description

Technical Field

[0001] The present invention is applicable to image semantic segmentation technology in the field of computer vision. Background Art

[0002] Image semantic segmentation is a core task in computer vision. Its purpose is to classify visual input at the pixel level. It is the foundation of visual analysis for applications such as autonomous driving, medical diagnosis, pedestrian detection, and scene understanding. The quality of image semantic segmentation directly impacts the results of subsequent algorithms. In recent years, with the rapid development of deep learning, a large number of image semantic segmentation algorithms have been proposed. These algorithms can be generally divided into three categories: encoder-decoder-based semantic segmentation networks, residual network-based semantic segmentation networks, and dilated convolution-based semantic segmentation networks.

[0003] (1) Semantic Segmentation Network Based on Encoder-Decoder

[0004] Inspired by the fully convolutional network (FCN), the SegNet network uses a symmetrical encoder-decoder structure. The encoder continuously downsamples the image through a combination of convolution and pooling, resulting in feature maps that, while increasingly smaller in size, possess increasingly stronger semantic resolution. The decoder stage is responsible for restoring the feature maps to the input size and predicting the semantic category of each pixel. To address the significant loss of edge detail caused by upsampling in the decoder stage, the UNet network fuses shallow and deep features through lateral skip connections, restoring object details to a certain extent.

[0005] (2) Semantic Segmentation Network Based on Residual Network

[0006] Full-Resolution Residual Networks (FRRN) have strong positioning and recognition capabilities for targets. The network uses residual flow and pooling flow to combine multi-scale contextual information and pixel-level accuracy. The residual flow carries full-resolution information to achieve accurate segmentation boundaries, and the pooling flow is used to obtain high-level features. However, the network does not process the boundaries of the segmented image finely enough, making the segmentation boundaries of some images blurred or ignored. On this basis, the Global Convolutional Network (GCN) can simultaneously solve the classification and positioning problems of semantic segmentation. When locating the target, the stacking idea is adopted, and the target boundary is further refined by residual-based boundary refinement. After upsampling the positioning results of each layer, they are fused and added with the positioning results of the previous layer to obtain the final pixel-level prediction results.

[0007] (3) Semantic Segmentation Network Based on Dilated Convolution

[0008] Atrous convolution can increase the receptive field without increasing the number of parameters. Stacking multiple convolutional layers with different atrous rates can capture objects at multiple scales and image context information. The semantic segmentation network that uses atrous convolution is mainly the Deeplab series. Deeplabv1 was the first to use atrous convolution, and Deeplabv2 proposed a pyramid pooling module (Atrous Spatial Pyramid Pooling, ASPP) with atrous convolution, and used conditional random fields for post-processing. However, while conditional random fields bring performance improvements, they also bring problems such as increased number of parameters and increased computational complexity. Starting from Deeplab v3, the Deeplab series abandoned the conditional random field (CRF) post-processing module and proposed a more general segmentation framework that is applicable to any network. It copied and cascaded the last block of ResNet, upgraded the ASPP module, and added a batch normalization layer (Batch Normalization, BN). Deeplab v3+ expands and improves upon Deeplab v3. Its main improvement is the use of ASPP in the codec architecture, creating a codec + ASPP architecture. Furthermore, Deeplab v3+ replaces its backbone with Xception, whose depthwise separable convolutional design makes the segmentation network more efficient.

[0009] Due to the problems of large object size range, more details and complex semantics in image semantic segmentation, the existing image semantic segmentation network has the following two shortcomings: (a) it is difficult to integrate the features of different receptive fields well and enrich the semantic information of the features; (b) it is difficult to accurately predict the edge details of the object. Therefore, the present invention proposes two general modules: Alternating Atrous Convolution (AAC) module and Edge Preserve (EP) module. The AAC module better integrates the features of different receptive fields through an interlaced fusion method, thereby enhancing the range of the receptive field that the features can express; and the EP module can enhance the detailed information of the features, thereby improving the network's ability to predict the edge details of objects. Summary of the Invention

[0010] This paper proposes a method for image semantic segmentation based on an interleaved dilated convolutional network (UNet). This method utilizes an interleaved dilated convolution module and an edge-preserving module. Fusion of features from different receptive fields can significantly improve semantic segmentation performance. However, direct addition or concatenation struggles to effectively fuse features from different receptive fields. Direct addition combines features from the same channel with features from different receptive fields, but these features may not represent the same object. Direct concatenation results in similar features with different receptive fields being far apart in the channel dimension, making it difficult to effectively fuse features distributed across different channels. The interleaved dilated convolution module uses an interleaved fusion approach, avoiding the addition of features representing different objects while effectively fusing features distributed across different channels. The edge-preserving module obtains edge features by subtracting the outputs of convolutional layers with different kernels, integrating and summing them to the output features. This module sharpens feature edge information, thereby enhancing the model's edge prediction capabilities. This method primarily addresses the technical challenges of multi-receptive field feature fusion and detail optimization for image semantic segmentation, thereby achieving better semantic segmentation performance.

[0011] The proposed image semantic segmentation method based on staggered dilated convolutional UNet effectively overcomes the shortcomings of traditional semantic segmentation methods. By introducing staggered dilated convolution operations, it effectively fuses features from different receptive fields, enriching the range of object sizes that can be represented by the features. Furthermore, by introducing an edge-preserving module, it enhances the edge details of features, preventing the network from losing a large amount of detail while enhancing the semantic information of the features. Figure 2 The overall structure diagram of the image semantic segmentation network based on staggered hole convolution is given.

[0012] The present invention is achieved through the following technical solutions:

[0013] The input is RGB image data. First, the image data in the dataset is processed for model training. Then the processed image data is sent to the network to train the parameters of the established network. The trained model is tested and the image semantic segmentation label results are output. The image semantic segmentation flow chart is as follows: Figure 1 As shown, the specific steps are as follows:

[0014] Step 1: Build a deep network model based on staggered dilated convolution

[0015] 1. Training data input

[0016] The proposed network uses RGB data as input. Specifically, the Synapse dataset uses 18 CT scan files for training; the ACDC dataset uses 80 files containing three semantic categories: left ventricle, right ventricle, and myocardium; the ISIC2017 dataset uses 2000 skin images as training sets; and the PH2 dataset uses 150 skin images as training sets.

[0017] 2. Design of staggered dilated convolution module

[0018] The receptive field of a feature represents the range of object sizes it can represent. Therefore, the wider the receptive field of a feature, the stronger its expressive power. Fusion of features with different receptive fields is one way to expand the receptive field, but simple concatenation or addition does not effectively fuse these features. Therefore, this paper proposes an interleaved dilated convolution module.

[0019] like Figure 2 As shown in (b), for the input F∈R of the AAC module C*H*W First, it passes through a 1x1 convolution, and then the result passes through two branches, the two branches are respectively the dilated convolutions with dilation rates of 1 and 2, and the size of the convolution kernel is 3x3, so as to obtain similar features F1∈R with different receptive fields. C1*H*W and F2∈R C2*H*W Where C1 and C2 are both 0.5 times C. These features are interleaved in the order of channels to obtain combined features, and 1x1 convolution is used to fuse the features and enrich the receptive field to obtain the output of the model F'∈R C*H*W In addition, the number of channels output by each branch of the module is equal to the number of channels of the input module features divided by the number of branches, which ensures that the features output by the module are consistent with the features input in terms of dimension, thus improving the versatility of the module.

[0020] 3. Design edge retention module

[0021] While downsampling and convolution operations expand the receptive field, they also cause problems such as blurred edges and loss of detailed information. This results in poor prediction results in areas with fine edge information. However, convolution operations are essential to expanding the receptive field and enhancing semantic information.

[0022] Therefore, in order to improve the prediction accuracy of the network for edge details, the present invention proposes an edge preservation module. Figure 2 As shown in (c), we have the input feature Z∈R C*H*W After a 1x1 convolution and a 3x3 convolution respectively, the feature Z1∈R is obtained C*H*W and Z2∈R C*H*W And subtract Z2 from Z1 to get rich detail information Z3∈RC*H*W , after sharpening Z3 through 1x1 convolution, it is added to Z2 to obtain the output Z'∈R C*H*W Through the edge-preserving module, the feature not only enhances the semantic information through the convolution operation, but also strengthens the detail information of the feature.

[0023] 4. Design a semantic segmentation network based on staggered hole convolution UNet

[0024] The UNet network based on the interleaved dilated convolution in this invention adopts a U-shaped encoding-decoding structure, and its overall structure is as follows: Figure 2 (a) As shown. For the input image I∈R 1*224*224 , firstly, it is encoded by the encoder on the left. Specifically, the input image I undergoes three 3x3 convolution and downsampling operations to obtain three features of different scales X1∈R 64*112*112 , X2∈R 256*56*56 and X3∈R 512*28*28 Before entering the decoder, feature X3 first passes through 12 Transformer layers to further enhance the semantic information, and obtains feature X4∈R 512*14*14 The feature X4 is subjected to 3x3 convolution and upsampling operations to obtain the feature X3'∈R 256*28*28 Concatenate features X3 and X3', and after 3x3 convolution and upsampling operations, obtain feature X2'∈R 128 *56*56 The feature X2 is passed through the AAC module and the EP module respectively, and the outputs are concatenated to obtain the feature M2∈R 512 *56*56 , M2 is subjected to 1x1 convolution and 3x3 convolution to obtain the feature M2'∈R 256*56*56 , after 1x1 convolution, 3x3 convolution and 1x1 convolution, we get the feature M2”∈R 256*56*56 , add the feature M2' and the feature M2" to get the feature M2"'∈R 256*56*56 . Concatenate feature M2'' and feature X2' to obtain feature X1'∈R 64*112*112 The feature X1 is passed through the AAC module and the EP module respectively, and the outputs are concatenated to obtain the feature M1∈R 128*112*112 , M1 is subjected to 1x1 convolution and 3x3 convolution to obtain the feature M1'∈R 64*112*112 , after 1x1 convolution, 3x3 convolution and 1x1 convolution, we get the feature M1"∈R 64*112*112 , add the feature M1' and the feature M1" to get the feature M1"'∈R 64*112*112 . Concatenate feature M1'' and feature X1' to obtain feature X∈R 128*112*112Perform 3x3 convolution and upsampling operations on feature X to obtain feature X'∈R 128*224*224 Perform 3x3 convolution on feature X' and semantic segmentation head (1x1 convolution) to get output Y'∈R 8*224*224 .

[0025] Step 2: Adjust network parameters and train and test the neural network

[0026] 1. Parameter settings

[0027] This paper uses the PyTorch framework and is implemented on a single NVIDIA TITAN XP GPU, using the Stochastic Gradient Descent (SGD) optimizer. For the Synapse and ACDC datasets, the initial learning rate is set to 0.01, the momentum is 0.9, the weight decay is 1e-4, the batch size is 12, and the number of iterations is 20k. For the ISIC2017 and PH2 datasets, the initial learning rate is set to 0.005, the momentum is 0.9, the weight decay is 1e-4, the batch size is 12, and the number of iterations is 15k / 7k.

[0028] During model training, the loss function used is:

[0029]

[0030] Where N is the number of semantic categories in the dataset, Y i ′ is the model’s prediction result for the i-th semantic category, Y i is the true value of the i-th semantic category, Represents element-wise multiplication.

[0031] 2. Input test data

[0032] For the Synapse dataset, 12 CT scan files were used as the test set; for the ACDC dataset, 20 files were used as the test set; for the ISIC2017 dataset, 600 skin image images were used as the test set; for the PH2 dataset, 50 skin image images were used as the test set.

[0033] The accompanying figure shows the visualization results of an ablation experiment using the EP module. From left to right, the image shows the label, the result with the EP module, and the result without the EP module. As can be seen from the figure, using the EP module significantly improves the model's prediction accuracy for edge details. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Figure 1 : Image semantic segmentation flowchart

[0035] Figure 2 :The overall structure diagram of the UNet network based on staggered hole convolution

[0036] Figure 3 : Visualization effects on the Synapse dataset

[0037] Figure 4 :Visualization results on the ISIC2017 dataset

[0038] Figure 5 : Visualization results on the PH2 dataset

[0039] Figure 6 :Visualization results of different feature fusion methods on the Synapse dataset

[0040] Figure 7 :Visualization results of EP module ablation experiment on Synapse dataset DETAILED DESCRIPTION

[0041] We experimentally validated the above method and evaluated its performance on image semantic segmentation tasks on four datasets: Synapse, ACDC, ISIC2017, and PH2. We also visualized some of the semantic segmentation results.

[0042] 1. Qualitative evaluation

[0043] Table 1 shows quantitative results on the Synapse dataset. Compared to competing methods, our method achieves a Dice Similarity Coefficient (DSC) of 79.56% on the Synapse dataset, an improvement of at least 2.08%. The Hausdorff Distance (HD) metric is reduced to 24.81, an improvement of at least 6.88. Table 1 also shows the segmentation accuracy of our method across eight categories, demonstrating that our approach achieves optimal performance across most categories.

[0044] Table 2 shows the quantitative results on the ACDC dataset. It can be seen that the performance of our method is superior to that of other comparison methods. In particular, our method significantly improves the segmentation results of the myocardium category.

[0045] Table 3 shows the results for the five metrics of Jaccard index (JA), DSC, pixel accuracy (Pixel-AC), pixel sensitivity (Pixel-SE), and pixel specificity (Pixel-SP) on the ISIC2017 dataset. It can be seen that our method significantly outperforms the other compared methods in these four metrics: JA, DSC, Pixel-AC, and Pixel-SE. Table 4 shows the results for JA, DSC, Pixel-AC, Pixel-SE, and Pixel-SP on the PH2 dataset. It can be seen that our method achieves optimal performance in all three metrics: JA, DSC, and Pixel-SE.

[0046] Table 1 Overall results and results of each category on the Synapse dataset

[0047] Method DSC HD Aorta Gallbladder Kidney(L) Kidney(R) Liver Pancreas Spleen Stomach V-Net 68.81 75.34 51.87 77.10 80.75 87.84 40.05 80.56 56.98 DARR 69.77 74.74 53.77 72.31 73.24 94.08 54.18 89.90 45.96 UNet 74.68 36.87 84.18 62.84 79.19 71.29 93.35 48.23 84.41 73.92 Att-UNet 75.57 36.97 55.92 63.91 79.20 72.71 93.56 49.37 87.19 74.95 TransUNet 77.48 31.69 87.23 63.13 81.87 77.02 94.08 55.86 85.08 75.62 Ours 79.56 24.81 89.03 70.21 80.17 78.2 94.89 61.73 85.05 77.21

[0048] Table 2 Overall results and results for each category on the ACDC dataset

[0049]

[0050] Table 3 Results on the ISIC2017 dataset

[0051] Method JA DSC Pixel-ACC Pixel-SE Pixel-SP CDNN 76.5 84.9 93.4 82.5 97.5 DNN 76.5 86.6 93.9 82..5 98.4 FCN-SSP 77.3 85.7 93.8 85.5 97.3 SLSDeep 78.2 87.8 93.6 81.6 98.3 Swin-Transformer 75.7 86.2 92.4 86.6 96.9 Segementer 75.6 86.1 93.8 83.3 97.2 CCL+MSFA 79.5 87.1 94.3 88.8 96.5 MB-DCNN 80.4 87.8 94.7 87.4 96.8 MT-TransUNet 80.7 88.0 94.9 88.2 96.4 Ours 82.2 89.3 95.2 91.0 96.5

[0052] Table 4 Results on the PH2 dataset

[0053]

[0054] 2. Subjective effect

[0055] Figure 3 The following visualizations are based on the Synapse dataset, showing, from left to right: the original image, the label, the results of our method, and the TransUNet method. Compared to TransUNet, our method avoids mispredicting non-organ regions in rows 1 and 2; our method avoids missing organ regions in rows 3 and 6; and our predictions are more comprehensive and less missing in rows 4 and 5.

[0056] Figure 4 and Figure 5 The visualization results of our method on the ISIC2017 and PH2 datasets are shown in the figure. From top to bottom, the original image, the labels, and the prediction results of our method are shown. As can be seen from the figure, our method has excellent performance.

[0057] Figure 6 This figure illustrates the differences in prediction results and feature maps between different feature fusion methods. Our AAC module uses interleaved fusion, in addition to direct addition and direct concatenation. The figure shows, from top to bottom, direct addition, direct concatenation, and the AAC module. As can be seen, our AAC module can better fuse features from different receptive fields, resulting in more complete predictions and fewer mispredictions.

[0058] Figure 7 This figure shows the visualization results of an ablation experiment on the EP module. From left to right, the image shows the label, the result with the EP module, and the result without the EP module. As can be seen from the figure, using the EP module significantly improves the model's prediction accuracy for edge details.

Claims

1. Image semantic segmentation method based on staggered dilated convolutional UNet, characterized by: Step 1: Build a deep network model based on staggered dilated convolution 1) Training data input Take RGB data as input; 2) Design of staggered dilated convolutional AAC module Input F∈R C*H*W First, it passes through a 1x1 convolution, and then the result passes through two branches, the two branches are respectively the dilated convolutions with dilation rates of 1 and 2, and the size of the convolution kernel is 3x3, so as to obtain similar features F1∈R with different receptive fields. C1*H*W and F2∈R C2*H*W ; C1 and C2 are both 0.5 times C; and these features are staggered in the order of channels to obtain combined features, and 1x1 convolution is used to fuse the features and enrich the receptive field to obtain the output of the model F'∈R C*H*W ; The number of channels output by each branch of the module is equal to the number of channels of the input module features divided by the number of branches, which ensures that the features output by the module are consistent with the features input in terms of dimension. 3). Design edge-keeping EP module An edge-preserving module is proposed; for the input feature Z∈R C*H*W After a 1x1 convolution and a 3x3 convolution respectively, the feature Z1∈R is obtained C*H*W and Z2∈R C*H*W And subtract Z2 from Z1 to get rich detail information Z3∈R C*H*W , after sharpening Z3 through 1x1 convolution, it is added to Z2 to obtain the output Z'∈R C*H*W ; 4) Design a semantic segmentation network based on staggered dilated convolutional UNet Based on the staggered hole convolution UNet network, the U-shaped encoding-decoding structure is used. For the input image I∈R 1*224*224 , first encoded by the encoder on the left; The input image I undergoes three 3x3 convolution and downsampling operations to obtain features X1∈R of three different scales 64*112*112 , X2∈R 256*56*56 and X3∈R 512*28*28 ; Before entering the decoder, feature X3 first passes through 12 Transformer layers to further enhance the semantic information, obtaining feature X4∈R 512*14*14 ; The feature X4 is subjected to 3x3 convolution and upsampling operations to obtain the feature X3'∈R 256*28*28 ; Concatenate features X3 and X3', and after 3x3 convolution and upsampling operations, obtain feature X2'∈R 128*56*56 ; Pass the feature X2 through the AAC module and the EP module respectively, and concatenate the outputs to obtain the feature M2∈R 512*56*56 , M2 is subjected to 1x1 convolution and 3x3 convolution to obtain the feature M2'∈R 256*56*56 , after 1x1 convolution, 3x3 convolution and 1x1 convolution, we get the feature M2”∈R 256*56*56 , add the feature M2' and the feature M2" to get the feature M2"'∈R 256 *56*56 ; Concatenate feature M2'' and feature X2' to obtain feature X1'∈R 64*112*112 ; Pass the feature X1 through the AAC module and the EP module respectively, and concatenate the outputs to obtain the feature M1∈R 128*112*112 , M1 is subjected to 1x1 convolution and 3x3 convolution to obtain the feature M1'∈R 64*112*112 , after 1x1 convolution, 3x3 convolution and 1x1 convolution, we get the feature M1"∈R 64 *112*112 , add the feature M1' and the feature M1" to get the feature M1"'∈R 64*112*112 ; Concatenate feature M1'' and feature X1' to obtain feature X∈R 128*112*112 ; Perform 3x3 convolution and upsampling operations on feature X to obtain feature X'∈R 128*224*224 ; Perform 3x3 convolution on feature X' and 1x1 convolution on semantic segmentation head to get output Y'∈R 8*224*224 ; Step 2: Adjust network parameters and train and test the neural network The Pytorch architecture is used, and the optimizer is the stochastic gradient descent optimizer. For the Synapse dataset and the ACDC dataset, the initial learning rate is set to 0.01, the momentum is 0.9, the weight decay is 1e-4, the batch size is 12, and the number of iterations is 20k. For the ISIC2017 dataset and the PH2 dataset, the initial learning rate is set to 0.005, the momentum is 0.9, the weight decay is 1e-4, the batch size is 12, and the number of iterations is 15k / 7k. During model training, the loss function used is: Where N is the number of semantic categories in the dataset, Y i ′ is the model’s prediction result for the i-th semantic category, Y i is the true value of the i-th semantic category, Represents element-wise multiplication.

Images