Road segmentation method for remote sensing images combining dense attention and parallel upsampling

Abstract

Aiming at the problem of low accuracy of road segmentation algorithm in high-resolution remote sensing images, the present invention proposes a remote sensing image road segmentation method combining dense attention and parallel upsampling. The method comprises the following steps: step 1, designing a dense atrous spatial pyramid attention module (DASPA); step 2, designing a multi-channel parallel upsampling structure (MPUpsample); step 3, building a remote sensing image road segmentation network, wherein the dense atrous spatial pyramid attention structure acts on the central part of the encoder and decoder, and the multi-channel parallel upsampling module acts on the decoder part; step 4, training and testing the proposed network model on a public remote sensing image road information extraction dataset; step 5, comparing and analyzing the segmentation results of the method of the present invention with the currently advanced remote sensing image road segmentation methods. The method of the present invention can significantly improve the remote sensing image road segmentation performance on the DeepGlobe public dataset, and has great application value in the field of remote sensing images.

Description

Technical Field

[0001] The present invention belongs to the field of image processing of computer vision, and in particular relates to a remote sensing image road segmentation method combining dense attention and parallel upsampling. Background Art

[0002] As a typical geographical landmark in remote sensing images, roads have important application value in mapping, environmental monitoring, military applications and other fields. Due to the rapid development of remote sensing technology, the resolution of remote sensing images has been continuously improved, and the interference of noise on images has also increased. How to automatically extract high-precision road information from remote sensing images has become a hot topic and difficulty in recent years. At present, the remote sensing image segmentation method based on Convolutional Neural Network (CNN) has performed particularly well. Compared with the traditional semi-automatic extraction method, this method can effectively suppress the noise generated in the process of road information extraction and reduce the loss of road detail information, which greatly improves the extraction effect. However, there are mainly the following difficulties in automatically extracting road information from remote sensing images: (1) The input image resolution is high and the data volume is large, which requires a sufficiently large receptive field; (2) The roads in remote sensing images are slender and complex, accounting for a small proportion of the entire image; (3) Due to geographical restrictions, some roads are covered by objects such as shadows, clouds, buildings or trees, and the image color contrast is low, making it difficult to extract the area of ​​interest; (4) Roads have natural connectivity, that is, they have the topological characteristics of the image.

[0003] In recent years, many methods have been proposed at home and abroad to solve the problem of how to automatically extract road information from high-resolution remote sensing images. The more common traditional remote sensing image road information extraction methods include pixel-based, object-based, knowledge-based, and machine learning-based methods. Although these methods have made great progress in segmentation performance, they still cannot solve the problem of information loss due to road occlusion. At the same time, due to the presence of background noise information, a large number of difficult-to-handle fine boundary features will be generated during the extraction process. Summary of the invention

[0004] In order to solve the problem of low accuracy of road segmentation algorithm in high-resolution remote sensing images, the present invention proposes a method for extracting road information from remote sensing images by combining dense attention and parallel upsampling. A dense and hollow spatial pyramid attention structure is designed in the middle part of the encoder-decoder. Adaptively fusing the channel attention branch and the spatial attention branch is conducive to extracting rich global context information, screening out useful target features, and suppressing the interference of irrelevant features. A multi-channel parallel upsampling module is designed in the decoder part. By fusing multi-channel feature maps to obtain prediction results with fine location information, the model's ability to retain detailed features is improved. Experimental results show that the model has achieved significant improvement in segmentation performance on the DeepGlobe dataset and can be widely used in various tasks in the field of remote sensing images.

[0005] The technical solution of the present invention includes the following steps.

[0006] Step 1: Design the Dense Atrous Spatial Pyramid Attention (DASPA) module.

[0007] Step 2: Design a multi-channel parallel upsampling structure (MPUpsample).

[0008] Step 3: Build a remote sensing image road segmentation network, in which the dense void spatial pyramid attention structure acts on the central part of the encoder and decoder, and the multi-way parallel upsampling module acts on the decoder part.

[0009] Step 4: Train and test the proposed network model on a public remote sensing image road information extraction dataset.

[0010] Step 5: Compare and analyze the segmentation results of the method of the present invention with the currently advanced remote sensing image road segmentation methods.

[0011] Compared with the prior art, the present invention has the following significant advantages: 1) The dense hollow spatial pyramid attention module designed by the method of the present invention combines parallel spatial information and channel information, readjusts the feature importance, and eliminates the interference of background features while retaining the image detail information, thereby obtaining a more accurate segmentation result; 2) The multi-channel parallel upsampling module designed by the method of the present invention stacks all upsampled feature maps from the decoder to obtain contextual information at different scales, and uses them together as the input of the last layer for image prediction, thereby ensuring accurate positioning while capturing rich detail information, enabling the network model to better aggregate multi-scale features, effectively improving the upsampling repair effect on image detail information, and further improving the road segmentation accuracy; 3) The network model proposed by the present invention can effectively improve the road segmentation accuracy of remote sensing images, and can be extended to the field of remote sensing images or other image segmentation tasks, which has great innovative significance and application value. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Figure 1 This is a structural diagram of the dense hole space pyramid attention module of the present invention;

[0013] Figure 2 It is a structural diagram of a multi-channel parallel up-sampling module of the present invention;

[0014] Figure 3 A network structure diagram of a remote sensing image road segmentation according to the present invention;

[0015] Figure 4 Comparison of prediction results of the method of the present invention with other algorithms on the DeepGlobe dataset. DETAILED DESCRIPTION

[0016] Figure 1 The present invention will be further described below in conjunction with the accompanying drawings and specific embodiments.

[0017] Step 1: Design a dense atrous spatial pyramid attention module (DASPA). In order to better extract the dense context information of pyramid features at different scales and eliminate the interference of background features while retaining image detail information, the present invention proposes a dense atrous spatial pyramid attention module, such as Figure 1 As shown in the figure, we first combine the outputs of the dilated convolutions with dilation rates of 1, 2, 4, and 8 using dense connections, and then concatenate the input feature maps. A The learned dense feature information is weighted pixel by pixel with the original features extracted by 1×1 convolution, and finally the channel attention branch is fused with the dense void space attention branch to obtain the output feature map. ,in , In addition, the channel attention branch obtains a 1×1× c The feature vector is then generated through two fully connected layers to establish the correlation between channels. The first fully connected (FC) operation reduces the feature dimension to 1 / 16 of the input image. After ReLu activation, the second fully connected operation restores the feature dimension to the original dimension. Finally, the normalized weight obtained through the Sigmoid function is weighted to the features of each channel of the input image. The calculation process of the DASPA module is shown in formula (1):

[0018] (1).

[0019] In the formula Represents the input image A The feature map obtained by the spatial attention branch, Represents the feature map obtained by the channel attention branch of the input image:

[0020] (2).

[0021] In the formula Represents the output feature map of the layer in the dense atrous spatial pyramid structure, Conv represents 1×1 convolution, Concat It represents the feature fusion of the output feature maps of the dilated convolutional layers with the hole rates of 1, 2, 4, and 8, where y i It can be formulated as:

[0022] (3).

[0023] H K,di where represents the dilated convolution operation, d i Representative i The void ratio of the layer, K is the filter size, [...] represents a dense connection operation, represents the feature map formed by connecting the outputs of all previous layers. The calculation process of the channel attention branch in formula (1) is:

[0024] (4) .

[0025] In the formula U Represents the input feature map AThe weighted output obtained after weighting the channel attention branch is , weighted U The calculation process is:

[0026] (5).

[0027] In the formula G L represents the global average pooling operation, FC 1 Represents the first full connection operation, FC 2 Represents the second full connection operation, represents the ReLU activation function, Represents the Sigmoid activation function.

[0028] Step 2: Design a multi-channel parallel upsampling module (MPUpsample). In the traditional U-shaped network, high-level abstract semantic information is gradually transferred from the high layer of the decoder to the low layer, and the edge information captured by the deeper layer will be gradually diluted, which is not conducive to the road segmentation of remote sensing images. Therefore, in order to solve the problem that the road edge detail information is easily lost in the top-down path, the present invention further proposes a multi-channel parallel upsampling module, such as Figure 2 As shown in the figure. Compared with the mainstream architecture, this structure not only uses the last layer of the decoder to predict the segmentation mask, but also stacks all the upsampled feature maps from the decoder to obtain contextual information at different scales, and uses them together as the input of the last layer for image prediction, ensuring accurate positioning while capturing rich detail information, so that the network model can better aggregate multi-scale features. In the process of parallel upsampling of feature maps, considering the impact of video memory consumption, this structure first uses 1×1 convolution to reduce the number of feature map channels at each level, and then performs 2x, 4x, 8x, and 16x upsampling to restore the image resolution, to ensure that too many detail features are not lost during the upsampling process. Finally, the multi-branch upsampled feature maps are stacked and passed through 3×3 and 1×1 convolution layers in sequence to achieve image segmentation.

[0029] Step 3: Build a remote sensing image road segmentation network, where the dense void space pyramid attention structure acts on the central part of the encoder and decoder, and the multi-channel parallel upsampling module acts on the decoder. The remote sensing image road segmentation network model is as follows: Figure 3 shown.

[0030] In order to improve the segmentation accuracy of the model for the target of interest and the ability to preserve the road detail features, the dense atrous spatial pyramid attention module in the network uses multiple dilated convolutions and densely connected structures in the spatial attention branch to expand the network receptive field and obtain local and global multi-scale hierarchical features. The channel attention branch readjusts the importance of each channel feature by establishing inter-channel dependencies. Adaptively fusing the channel attention branch and the spatial attention branch is conducive to extracting rich global context information, screening out useful target features, and suppressing the interference of irrelevant features. The multi-channel parallel upsampling module is different from the simple upsampling method of bilinear interpolation or deconvolution to restore the resolution of the feature map. Instead, it fuses multiple feature maps in the decoder to obtain prediction results with fine position information, improving the model's ability to preserve detail features.

[0031] Step 4. The present invention uses the DeepGlobe remote sensing image road extraction dataset to verify the performance of the algorithm. The dataset contains 6226 training images and their corresponding labels. The 6226 images are randomly divided into 4358 training images, 1245 test images and 623 verification images according to 7:2:1. The label is a grayscale binary image with the same height and width as the input image, and the road and non-road pixels are set to 255 and 0 respectively. All images are 1024×1024 in size, and each image is an RGB image with a ground resolution of 0.5m / pixel collected by the DigitalGlobe satellite. The dataset includes complex scenes such as wilderness, cities, villages and rainforests. The proposed network model is trained and tested on a public remote sensing image road information extraction dataset. The experiment of the present invention was carried out on a computing platform with a GPU model of NVIDIA GTX1080Ti, and the deep learning framework used was Pytorch1.2.0. In the training phase, the present invention adopts the data enhancement method of translation, scaling, rotation and flipping, with binary cross entropy and dice coefficient loss as loss functions and Adam as optimizer. The initial learning rate is set to , the training batch size is 4, a total of 300 rounds are trained, and the input image size is . In the test phase, horizontal rotation and [1.25, 1.5, 1.75] scaling ratios are used for multi-scale testing.

[0032] Step 5: In order to accurately evaluate the segmentation accuracy of the remote sensing image road segmentation model, the present invention uses the commonly used indicators in semantic segmentation for evaluation, namely, recall rate ( P r )、Accuracy( P a )、Accuracy( P p )and F 1-score. P rThat is, the ratio of the number of correctly predicted road pixels to the number of all road pixels. P a That is, the ratio of the number of correctly predicted road and background pixels to the number of pixels in the entire remote sensing image. P p That is, the ratio of the number of road pixels predicted correctly to the number of road pixels predicted correctly. F 1-score is a comprehensive evaluation index of recall rate and precision rate. F The higher the 1-score, the more robust the model segmentation performance. P r 、P a 、P p and F The 1-score calculation formula is:

[0033] (6)

[0034] (7)

[0035] (8)

[0036] (9).

[0037] In the formula: TP Predict the correct number of pixels for the road; TN Predict the correct number of pixels for the background; FN The number of pixels for extracting wrong background information; FP is the number of pixels from which incorrect road information is extracted.

[0038] The experimental results are shown in Table 1. The indicators of the method of the present invention are better than those of the current mainstream algorithms. Although DenseASPP in the mainstream algorithm obtains dense multi-scale features, it is difficult to strike a balance between the selection of the expansion rate and the scale change. The U-Net model is a typical U-shaped structure. The network uses skip connections to combine the spatial information of low-level feature maps. Although the final feature map realizes feature fusion at different scales, due to the presence of many interference factors in remote sensing images, the U-Net network cannot suppress noise interference and filter out useful features. At the same time, the high-resolution remote sensing image data set has a large amount of data. Compared with LinkNet, the U-Net model has more parameters and lower accuracy in model training. Although FCN-16s fuses the features output by different downsampling layers, the road detail representation in the segmentation result is not obvious and the accuracy is not high, and there is a phenomenon of missing road structure. The method of the present invention effectively extracts the dense multi-scale spatial features of the highest layer of the network, and fuses the spatial attention branch and the channel attention branch to filter out the image information containing the road, avoiding the interference of useless features. In the decoder part, multi-channel feature maps are used for channel superposition to solve the problem of loss of detail information due to road structure interruption.

[0039] Table 1 Comparison of segmentation indicators of different algorithms on DeepGlobe dataset

[0040] .

[0041] Figure 4 The comparison chart shows the segmentation effect of different algorithms on the DeepGlobe remote sensing image road extraction dataset. Figure 4 It can be seen that the method of the present invention is significantly superior to other algorithms in suppressing the interference of redundant features and dealing with the loss of detail information. Although U-Net has higher image integrity than LinkNet segmentation, it is easy to introduce a lot of useless information and there are cases of misclassification and omission. FCN-16s loses serious road key detail information, and image feature extraction is prone to produce false information due to the influence of other obstructions, resulting in lower segmentation performance. Compared with other algorithms, DenseASPP has the worst segmentation effect. There are large blocks of missing road structures in some complex images, and the boundary information is seriously lost. Compared with the above-mentioned algorithms, the method of the present invention better retains the integrity of road structure features, effectively suppresses the interference of noise on image extraction, and the segmentation effect diagram is closer to the real image label.

Claims

1. A remote sensing image road segmentation method combining dense attention and parallel upsampling includes the following steps: Step 1: Design a dense atrous spatial pyramid attention module (DASPA). Step 2: Design a multi-channel parallel upsampling structure (MPUpsample). Step 3: Build a remote sensing image road segmentation network, where the dense void space pyramid attention structure acts on the central part of the encoder and decoder, and the multi-channel parallel upsampling module acts on the decoder part; Step 4: Train and test the proposed network model on a public remote sensing image road information extraction dataset; Step 5: Compare and analyze the segmentation results of this method with the currently advanced remote sensing image road segmentation methods; in, The dense atrous spatial pyramid attention module in step 1 consists of two parallel branches: spatial attention and channel attention. The spatial attention branch includes a 1×1 convolution and a dense atrous spatial pyramid structure with dilation rates of 1, 2, 4, and 8, which can readjust the importance of features and eliminate the interference of background features while retaining image detail information, thereby obtaining more accurate segmentation results. Among them, the multi-way parallel upsampling structure in step 2 first uses 1×1 convolution to reduce the number of feature map channels at each level; secondly, 2x, 4x, 8x, and 16x upsampling are performed to restore the image resolution to ensure that too many detail features are not lost during the upsampling process; finally, the multi-branch upsampled feature maps are stacked and passed through 3×3 and 1×1 convolution layers in sequence to achieve image segmentation; In step 3, the encoder part uses ResNet34 as the pre-trained model, uses 7×7 convolution to extract the rough features of the original input image, and extracts fine image edge information and position information after 4 downsampling layers; Among them, the dense atrous spatial pyramid attention structure acts on the central part of the encoder and decoder: the dense atrous spatial pyramid attention module first connects the atrous convolutions with different dilation rates in a dense connection manner to increase the receptive field of the highest layer of the network and achieve the fusion of all intermediate features; secondly, it suppresses the irrelevant and easily confused pixel information of multi-scale images by learning weights; finally, it combines the abstract spatial information and channel information of the network to capture rich local and global features, providing sufficiently accurate pixel-level predictions for mask segmentation; The multi-channel parallel upsampling module acts on the decoder part: the multi-channel parallel upsampling module effectively combines the features of different scale layers, which helps to enhance the responsiveness of the target features. At the same time, in order to reduce the number of parameters for feature channel fusion, the structure first reduces the number of channels, then restores the image resolution through upsampling, and stacks the upsampled feature maps as a whole as input to produce the segmentation result.

Images