A deep learning-based remote sensing image road extraction method

Abstract

The application discloses a kind of based on deep learning's remote sensing image road extraction method, the method is by encoder, cross space-scale feature interaction module, decoder three parts, and the road feature of remote sensing image is learned from data set by convolutional neural network so as to classify the road in image and background pixel point.Modeling road feature has better feature expression ability, effectively improves the integrity of model prediction road result.

Description

Technical Field

[0001] This invention belongs to the field of pattern recognition technology, specifically relating to a method for extracting roads from remote sensing images based on deep learning. Background Technology

[0002] Extracting roads from remote sensing images has significant application value in fields such as autonomous driving, geological surveying, and disaster relief. However, the presence of numerous objects with similar road structures in remote sensing images, as well as the occlusion of roads by surrounding pixels, makes extracting complete road structures from remote sensing images extremely challenging. Deep learning networks possess powerful feature extraction and fitting capabilities, and numerous deep learning-based methods for road extraction from remote sensing images have emerged. The skip connections in the encoder-decoder structure can effectively provide the decoder with complete and rich road structures, enabling the model to predict road details more accurately. Previous methods have aggregated features at different scales by employing dense connections, including UNet++ (Zhou, Z., Siddiquee, MMR, Tajbakhsh, N., & Liang, J. (2019). Unet++: Redesigning skip connections to exploit multiscale features in imagesegmentation. IEEE transactions on medical imaging, 39(6), 1856-1867.) and UNet3+ (H. Huang, L. Lin, R. Tong, H. Hu, Q. Zhang, Y. Iwamoto, X. Han, YW Chen, and J. Wu, “Unet 3+: A full-scale connected union for medical image segmentation,” in IEEE Int. Conf. Acoust. Speech Signal Process., 2020, pp. 1055–1059.). These methods can effectively aggregate encoder features from different stages, but they also have some drawbacks: the upsampling and pooling operations on feature maps can destroy the accurate semantic information of the feature maps or cause information loss. Furthermore, the extensive use of convolution and pooling operations increases the model's dependence on device performance. These issues mean that the accuracy of the predicted roads still has room for improvement, and the high computational cost makes them unsuitable for high-resolution remote sensing image road extraction tasks. Summary of the Invention

[0003] To overcome the shortcomings of existing technologies, this invention provides a deep learning-based method for road extraction from remote sensing images. This method consists of three parts: an encoder, a cross-spatial-scale feature interaction module, and a decoder. It learns road features from a dataset using a convolutional neural network, thereby classifying roads and background pixels in the image. The road features modeled by the method have better feature representation capabilities, effectively improving the completeness of the model's predicted road results.

[0004] The technical solution adopted by this invention to solve its technical problem is as follows:

[0005] Step 1: Divide the dataset into a training set and a test set;

[0006] The remote sensing images containing road targets are cropped to 1024×1024 pixels, and then the remote sensing images containing road pixels are manually labeled; the resulting remote sensing images and corresponding road labels are divided into training sets and test sets.

[0007] Step 2: Input the training set into the convolutional neural network and use the ResNet34 network as the visual feature encoder. The encoder is divided into five stages, and finally obtains five different feature maps E1, E2, E3, E4 and E5 with resolutions arranged from large to small.

[0008] Step 3: Input the four feature maps E2, E3, E4, and E5 into the cross-space-scale feature interaction module to achieve the aggregation of features at different scales;

[0009] Step 4: Input the feature map E5 obtained by the encoder into the decoder for feature map prediction;

[0010] Step 5: Input the feature map output by the decoder, the feature map passed by the skip connection, and the supplementary road structure feature map generated by the cross-space and scale feature interaction module into the decoder to generate the output feature map of the decoder at the current stage;

[0011] Step 6: Upsample and convolve the decoder feature maps D5, D4, D3, and D2 from different stages, and fuse them with the feature map E1 passed through the skip connection; then perform a Sigmoid operation on the fused features to generate the predicted road results of the remote sensing image; use a combination of the binary cross-entropy loss function and the Dice coefficient as the model loss function, and train the entire network by calculating the difference between the road label and the predicted result through gradient backpropagation.

[0012] Step 7: Validate the trained model on the test set and generate road prediction results corresponding to the test remote sensing images.

[0013] Further, step 1 specifically includes:

[0014] Prepare the dataset needed for training and testing the road extraction experiment;

[0015] a) Select remote sensing images containing road objects, preprocess the remote sensing images, and crop them into 1024*1024 pixel images; use labelme software to annotate the images, marking road pixels as white and background pixels as black;

[0016] b) Split the labeled remote sensing images into training and testing sets at a ratio of 9:1.

[0017] Furthermore, step 2 specifically includes:

[0018] The training set data is input into the encoder to generate road features at different scales;

[0019] The ResNet34 network is used as the encoder. First, downsampling is performed using max pooling, normalization, and ReLU operations. The remaining stages are composed of {3, 4, 6, 3} residual blocks respectively. The encoder generates five feature maps with different resolutions, named E1, E2, E3, E4, and E5 respectively.

[0020] Furthermore, step 3 specifically includes:

[0021] Step 3-1: Input the feature maps E2, E3, E4, and E5 generated by the encoder network into the cross-space and scale feature fusion module for feature fusion to achieve the fusion of road detail features and semantic information.

[0022] Step 3-2: In the cross-spatial, scale-based feature fusion module, different slicing operations of different sizes are selected according to the size of the input feature map to perform feature slicing. Feature maps of different scales are all transformed into 8*8 feature sequences. H, W, and C represent the width, height, and channel dimensions of the feature map, respectively; the sliced ​​features are then deformed. at this time The channel information of the current feature map is represented by 64, and the position information of the feature map is represented by 64. These deformed feature maps are then concatenated along the channel dimension to achieve feature fusion of feature maps from different stages of the encoder, resulting in a fused feature map F. fuse ;

[0023] Step 3-3: Analyze the fused feature map F fuse Feature extraction was performed using horizontal dilated convolution with a kernel size of 129 to obtain cross-scale feature interaction features of encoder feature maps at different stages; vertical dilated convolution with a kernel size of 2 to obtain cross-spatial feature interaction features of encoder feature maps at different stages; and convolution with a kernel size of 3×3 to obtain cross-spatial and cross-scale feature interaction features of encoder feature maps at different stages.

[0024] Steps 3-4: Channel concatenation is performed on the generated cross-spatial interaction features, cross-scale interaction features, and cross-spatial and scale interaction features to achieve feature fusion. Then, a 1×1 convolution is used to restore the channels of the fused feature map to the same level as the fused feature map F. fuse The number of channels is consistent; the generated feature map is deformed and the feature is split in the channel dimension according to the channel dimension of the original input feature. Then, a 1×1 convolution is used to restore the feature (H,W,C); thus supplementing the decoder with supplementary road structure feature maps S5, S4, S3, and S2 that fuse semantic information and texture features.

[0025] Furthermore, step 4 specifically includes:

[0026] The encoder feature map E5 is input into the decoder and added to the feature map S4 generated by the cross-space, scale feature interaction module to generate feature map D5.

[0027] The feature map D5 is input into the decoder, and feature map upsampling and channel dimension transformation are performed through ConvTrans2D and 1×1Conv operations to obtain feature map D5. ^ The feature map E4, which is passed through the jump connection, is combined with the feature maps S4 and D5 generated by the cross-space, scale feature interaction module. ^ Perform an addition operation to decode the feature map and generate feature map D4; repeat this module to generate feature maps D3 and D2.

[0028] The decoder feature map D2 is input into the decoder and added to the feature map E1 generated by the skip connection to generate feature map D1.

[0029] Furthermore, step 5 specifically includes:

[0030] The feature maps D5, D4, D3, and D2 generated by the decoder are input into upsampling and 1×1 convolution operations to transform the feature maps to have the same channels and size as feature map D1. Then, an addition operation is used to fuse the features.

[0031] A sigmoid operation is performed on the fused feature map to generate a road probability map in the remote sensing image.

[0032] The beneficial effects of this invention are as follows:

[0033] 1. The generated road prediction results have a more complete structure. This network achieves the full integration of deep semantic feature maps and shallow road structure features through the design of feature splitting, feature fusion, feature extraction, and feature distribution strategies, enabling the model to have more complete and accurate road structure features.

[0034] 2. The designed network structure is lightweight. This method uses a sampling-based interaction strategy to aggregate feature maps at different levels, which places lower demands on device performance compared to dense connection methods based on extensive sampling and convolutional operations.

[0035] 3. Effectively handles road occlusion scenarios. This invention efficiently integrates road features at different scales, enabling the model to obtain accurate road structure features and generate road prediction results with good connectivity even when roads are occluded. Attached Figure Description

[0036] Figure 1 This is a flowchart illustrating the specific process of this invention;

[0037] Figure 2 This is a diagram of the overall network structure of the present invention;

[0038] Figure 3 This is a structural diagram of the cross-space-scale feature interaction module of the present invention;

[0039] Figure 4 This is a structural diagram of the decoder module corresponding to the present invention;

[0040] Figure 5 The remote sensing images and corresponding road labels detected in this embodiment of the invention;

[0041] Figure 6 This is a road prediction result diagram according to an embodiment of the present invention. Detailed Implementation

[0042] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0043] This invention proposes a deep learning-based method for road extraction from remote sensing images. The method consists of three parts: an encoder, a cross-spatial-scale feature interaction module, and a decoder. It learns road features from a dataset using a convolutional neural network, thereby classifying roads and background pixels in the image. The road features modeled by the method have better feature representation capabilities, effectively improving the completeness of the model's road prediction results.

[0044] This invention proposes a road extraction method for remote sensing images based on deep learning. The technical solution to achieve the purpose of this invention includes the following steps:

[0045] 1. Divide the dataset into training and testing sets. Crops the remote sensing images containing road targets into 1024 pixels.

[0046] The images were divided into training and testing sets, each containing 1024 pixels. The pixels containing road information were then manually labeled.

[0047] 2. The training set is input into a convolutional neural network, and a ResNet34 network is used as a visual feature encoder. The encoder is divided into five stages, and finally five feature maps with different resolution sizes are obtained.

[0048] 3. Input the four deeper feature maps obtained by the encoder into the cross-space-scale feature interaction module to achieve full aggregation of features at different scales.

[0049] 4. Input the deep features obtained by the encoder into the decoder for feature map prediction.

[0050] 5. Input the feature map output by the decoder in the previous stage, the feature map passed by the skip connection, and the supplementary road structure feature map generated by the cross-space and scale feature interaction module into the decoder to generate the output feature map of the decoder in the current stage.

[0051] 6. Upsample and convolutionally process the feature maps from different stages of the decoder, and then fuse them by adding the feature maps. A sigmoid operation is then performed on the fused features to generate road predictions from the remote sensing image. A combination of binary cross-entropy loss and Dice coefficients is used as the model loss function, and gradient backpropagation is performed to train the entire network by calculating the difference between the road labels and the prediction results.

[0052] 7. The trained model is validated on the test set to generate road prediction results corresponding to the test remote sensing images.

[0053] Reference Figure 1 , Figure 2 , Figure 3 and Figure 4 The implementation steps of the remote sensing image road extraction model of the present invention are as follows:

[0054] Step 1: Prepare the dataset needed for training and testing the road extraction experiment. This method uses a supervised training approach. The model requires a large amount of input data to learn road features.

[0055] a) Select remote sensing images containing road objects, preprocess the images, and crop them to 1024*1024 pixel sizes. Use LabelMe software to annotate the images, marking road pixels as white and background pixels as black.

[0056] b) Split the labeled remote sensing images into training and testing sets at a ratio of approximately 9:1.

[0057] Step 2: Input the training set data into the encoder to generate road features at different scales.

[0058] The ResNet34 network proposed by He et al. in the paper "K.He, X.Zhang, S.Ren and J.Sun, Deep Residual Learning for Image Recognition, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Las Vegas, NV, USA, 2016, pp.770-778, doi:10.1109 / CVPR.2016.90" is used as the encoder. The network first performs downsampling using max pooling, normalization, and ReLU operations. The remaining stages consist of {3, 4, 6, 3} residual blocks, respectively. The encoder can generate five feature maps with different resolutions, named E1, E2, E3, E4, and E5.

[0059] Step 3: Input the feature maps E2, E3, E4, and E5 generated by the encoder network into the cross-space and scale feature fusion module for feature fusion to achieve accurate fusion of road detail features and semantic information.

[0060] In the cross-spatial, scale-based feature fusion module, different slicing operations of different sizes are selected based on the size of the input feature map for feature slicing. Feature maps of different scales are all transformed into 8*8 feature sequences. Transform the slice features in 64 represents the channel information of the feature map, and 64 represents the position information of the feature map. These deformed feature maps are then concatenated along the channel dimension to achieve feature fusion of feature maps from different stages of the encoder.

[0061] For the fused feature map F fuse Feature extraction was performed using horizontally dilated convolutions with a kernel size of 129 to obtain cross-scale feature interactions between feature maps from different stages. Vertically dilated convolutions with a kernel size of 2 were also used to extract cross-spatial feature interactions between feature maps from different stages. Finally, convolutions with a kernel size of 3×3 were used to extract cross-spatial and cross-scale feature interactions between encoder features from different stages.

[0062] The generated cross-spatial interaction features, cross-scale interaction features, and cross-spatial and scale interaction features are concatenated to achieve feature fusion. Then, a 1×1 convolution is used to restore the channels of the fused feature map to the same level as the fused feature map F. fuseThe number of channels is consistent. The generated feature map is deformed, and the features are split according to the channel dimension of the original input features. Then, a 1×1 convolution is used to restore the features (H, W, C). This generates a supplementary road structure feature map for the decoder that integrates accurate semantic information and texture features.

[0063] Step 4: Input the encoder feature map E5 into the decoder. Upsample the feature map D5^ using ConvTrans2D and 1×1Conv operations. Add the feature map E4 passed through the skip connection with the feature map S4 generated by the cross-space, scale feature interaction module and D5^ to achieve feature map decoding. Repeat this module to generate feature maps D3, D2, and D1.

[0064] Step 5: Feature map E5 and feature maps D4, D3 and D2 generated by the decoder are input into upsampling and 1×1 convolution operations to transform these feature maps to have the same channels and size as feature map D1. Then, feature fusion is performed by addition operation.

[0065] Step 6: Perform a Sigmoid operation on the fused feature map to generate a road probability map in the remote sensing image.

[0066] Step 7: Use the combination of binary cross-entropy function and Dice coefficient as the loss function to optimize the loss of the generated road probability map and road labels, and continuously optimize the road extraction model.

[0067] Step 8: Use the trained model to infer on the test set and generate predicted road labels for the remote sensing images.

[0068] Example:

[0069] 1. Experimental conditions;

[0070] This invention utilizes a network built on the PyTorch framework on an RTX 3090 GPU server and CentOS operating system. The dataset used in the experiments comes from the DeepGlobe road data, with each image having a resolution of 1024*1024, and the dataset is divided in a 9:1 ratio. The dataset includes 5500 pairs of remote sensing images and road labels as the training set and 726 pairs of images as the test set. Some roads in the images are obscured by trees or building shadows. Figure 5 As shown.

[0071] 2. Experiment content;

[0072] The remote sensing image road model is trained on the training set, and the trained remote sensing image road prediction model is tested on the road dataset. Road pixels are extracted from the images, and the recall, precision, harmonic mean, and intersection-union ratio of each image with respect to the road target are calculated.

[0073] Table 1 shows the model's performance on the DeepGlobe dataset.

[0074] Unet 83.70 73.22 63.62 76.76 68.19 Deeplabv3+ 84.74 72.68 63.73 75.71 67.91 This method 84.90 79.49 69.61 81.50 74.45

[0075] After multiple experiments, the results show that the proposed method can extract accurate and complete road prediction results from remote sensing images. The road recall rate on the DeepGlobe dataset reaches 84.90%, the intersection-over-union ratio reaches 69.61%, and the APLS index reaches 74.45%, all of which significantly outperform the comparison methods.

[0076] This invention uses a deep learning network to extract road features from remote sensing images. At the same time, it designs a cross-space-scale feature fusion module to effectively fuse road detail texture with accurate semantic information, which can provide the decoder with complete road structure adjustment, thereby improving the model's reasoning ability in scenarios where the road is occluded by surrounding objects. The model can generate road labels with good connectivity. Figure 6 The detection results of the model in the road occlusion scene, from Figure 6 It can be seen that the model is still able to predict the obscured road area even when the road is blocked by trees.

Claims

1. A method for road extraction from remote sensing images based on deep learning, characterized in that, Includes the following steps: Step 1: Divide the dataset into a training set and a test set; The remote sensing images containing road targets are cropped to 1024×1024 pixels, and then the remote sensing images containing road pixels are manually labeled; the resulting remote sensing images and corresponding road labels are divided into training sets and test sets. Step 2: Input the training set into the convolutional neural network and use the ResNet34 network as the visual feature encoder. The encoder is divided into five stages, and finally obtains five different feature maps E1, E2, E3, E4 and E5 with resolutions arranged from large to small. Step 3: Input the four feature maps E2, E3, E4, and E5 into the cross-space-scale feature interaction module to achieve the aggregation of features at different scales; Specifically: Step 3-1: Input the feature maps E2, E3, E4, and E5 generated by the encoder network into the cross-space-scale feature fusion module for feature fusion to achieve the fusion of road detail features and semantic information; Step 3-2: In the cross-space-scale feature fusion module, different slicing operations of different sizes are selected according to the size of the input feature map to perform feature slicing. Feature maps of different scales are all transformed into 8*8 feature sequences. , H, W, C These represent the width, height, and channel dimensions of the feature map, respectively; the sliced ​​features are then deformed. ,at this time The channel information of the current feature map is represented by 64, and the position information of the feature map is represented by 64. These deformed feature maps are concatenated along the channel dimension to achieve feature fusion of feature maps from different stages of the encoder, resulting in a fused feature map. ; Step 3-3: Analyze the fused feature map Feature extraction was performed using horizontal dilated convolution with a kernel size of 129 to obtain cross-scale feature interaction features of encoder feature maps at different stages; vertical dilated convolution with a kernel size of 2 to obtain cross-spatial feature interaction features of encoder feature maps at different stages; and convolution with a kernel size of 3×3 to obtain cross-spatial-scale interaction features of encoder feature maps at different stages. Steps 3-4: Channel concatenation is performed on the generated cross-spatial interaction features, cross-scale interaction features, and cross-spatial-scale interaction features to achieve feature fusion. Then, a 1×1 convolution is used to restore the channels of the fused feature map to match those of the fused feature map. The number of channels is consistent; the generated feature map is deformed, and the features are split in the channel dimension according to the channel dimension of the original input features, and then a 1×1 convolution is used for feature restoration. Thus, the decoder is supplemented with road structure features that fuse semantic information and texture features, resulting in supplementary diagrams S5, S4, S3, and S2. Step 4: Input the feature map E5 obtained by the encoder into the decoder for feature map prediction; input the feature map output by the decoder, the feature map passed by the skip connection, and the supplementary road structure feature map generated by the cross-space-scale feature interaction module into the decoder to generate the output feature map of the decoder at the current stage; Step 5: Upsample and convolve the decoder feature maps D5, D4, D3, and D2 from different stages, and fuse them with the decoder feature map D1 by addition; then perform a Sigmoid operation on the fused features to generate the predicted road results of the remote sensing image; use a combination of the binary cross-entropy loss function and the Dice coefficient as the model loss function, and train the entire network by calculating the difference between the road label and the predicted result through gradient backpropagation. Step 6: Validate the trained model on the test set and generate road prediction results corresponding to the test remote sensing images.

2. The method for road extraction from remote sensing images based on deep learning according to claim 1, characterized in that, Step 1 specifically involves: Prepare the dataset needed for training and testing the road extraction experiment; a) Select remote sensing images containing road objects, preprocess the remote sensing images, and crop them into 1024*1024 pixel images; use labelme software to annotate the images, marking road pixels as white and background pixels as black; b) Split the labeled remote sensing images into training and testing sets at a ratio of 9:

1.

3. The method for road extraction from remote sensing images based on deep learning according to claim 2, characterized in that, Step 2 specifically involves: The training set data is input into the encoder to generate road features at different scales; The ResNet34 network is used as the encoder. First, downsampling is performed using max pooling, normalization, and ReLU operations. The remaining stages are composed of {3, 4, 6, 3} residual blocks respectively. The encoder generates five feature maps with different resolutions, named E1, E2, E3, E4, and E5 respectively.

4. The method for road extraction from remote sensing images based on deep learning according to claim 3, characterized in that, Step 4 specifically involves: The encoder feature map E5 is input into the decoder and added to the feature map S5 generated by the cross-space-scale feature interaction module to generate feature map D5; The feature map D5 is input into the decoder, and feature map upsampling and channel dimension transformation are performed through ConvTrans2D and 1×1Conv operations to obtain feature map D5. ^ The feature map E4, which is passed through the skip connection, is combined with the feature maps S4 and D5 generated by the cross-space-scale feature interaction module. ^ Perform an addition operation to decode the feature map and generate feature map D4; repeat this process to generate feature maps D3 and D2. The decoder feature map D2 is input into the decoder and added to the feature map E1 generated by the skip connection to generate feature map D1.

5. The method for road extraction from remote sensing images based on deep learning according to claim 4, characterized in that, Step 5 specifically involves: The feature maps D5, D4, D3, and D2 generated by the decoder are input into upsampling and 1×1 convolution operations to transform the feature maps to have the same channels and size as feature map D1. Then, an addition operation is used to fuse the features. A sigmoid operation is performed on the fused feature map to generate a road probability map in the remote sensing image.

Images