A road extraction method based on remote sensing satellite images

Abstract

The invention belongs to the field of computer road extraction, and proposes a road extraction method based on remote sensing satellite images, which includes a feature encoder, an iterative feature enhancement subnetwork and a multi-task decoder. The iterative feature enhancement subnetwork unit includes a semantically guided feature enhancement module, a direction-aware feature aggregation module and two side branches; the semantically guided feature enhancement module includes feature selection and feature fusion; the direction-aware feature aggregation module includes a residual structure of multiple branches, each branch includes a direction-aware deformable convolution and a ReLU unit; the proposed semantically guided feature enhancement module, the direction-aware feature aggregation module and the direction-aware deformable convolution better utilize the road direction to automatically align the convolution kernel receptive field and the road area; the connectivity problem of the automated road extraction task in remote sensing satellite images is solved, and the result output by the method has high accuracy and efficiency.

Description

Technical Field

[0001] The present invention relates to the field of road extraction in the field of computer vision, and in particular to a road extraction method based on remote sensing satellite images. Background Art

[0002] The image semantic segmentation task aims to help computers understand the categories and locations of objects in the real environment, and identify the content and corresponding locations in the image according to the user-defined object categories. The remote sensing satellite image road extraction task based on semantic segmentation technology aims to predict the binary road segmentation map and locate the road position in the remote sensing satellite image. In recent years, as machine learning models based on deep neural networks have achieved great success in many computer vision tasks such as image classification, target detection, and semantic segmentation, deep learning-based methods have significantly improved the quality of road extraction from remote sensing satellite images on some large public datasets.

[0003] However, recent studies have found that the task of extracting roads from remote sensing satellite images has more technical challenges than the task of image semantic segmentation. In remote sensing satellite images, road areas are often obscured by clouds or high-rise buildings and their shadows. The appearance of road areas is greatly affected by objective factors such as weather, light intensity and shooting angle, and road areas have highly similar texture characteristics to adjacent non-road areas. Therefore, directly applying image semantic segmentation technology to the task of extracting roads from remote sensing satellite images usually results in a fragmented road network, which is far behind the requirements of industrial applications. At the same time, researchers try to obtain better road extraction results by improving the neural network structure. Abhishek Chaurasia et al. proposed a new network structure LinkNet that strengthens information transmission, increases the information flow between the encoder and the decoder, and directly inputs spatial information from the encoder to the decoder. Based on the results of predecessors, Lichen Zhou et al. designed a dilated convolution module after the encoder, and the dilated module part includes cascade mode and parallel mode dilated convolution. The receptive field of each path is different, so the network can capture multi-scale features to obtain better detection results. Yao Wei et al. used road boundaries as prior information to reduce the deviation along the road edge direction. Anil Batra et al. proposed StackHourglass, a road extraction method for remote sensing images based on multi-task learning. They used road direction as supervision information and preliminarily proved that road direction information can bring benefits to road extraction tasks. These methods have achieved certain improvements in road coherence, but they have not fully utilized the road direction information to assist road extraction tasks. The establishment of road topology always follows the guidance of road direction. Reasonable use of road direction can promote more accurate road extraction results. Road direction is inherently related to road extraction tasks.

[0004] Based on the investigation and analysis of the existing remote sensing image road extraction method StackHourglass, the present invention proposes a road extraction method based on remote sensing satellite images according to the appearance characteristics of remote sensing satellite images and the geometric characteristics of road elements. Summary of the invention

[0005] The technical problem to be solved by the present invention is to overcome the defect of insufficient utilization of road direction information in the existing method StackHourglass, and provide a new method for fully utilizing road direction information to assist road extraction. The purpose of the present invention is to give a binary mask representing road pixels by the proposed method under a given single RGB image, and the present invention is applicable to various remote sensing satellite images.

[0006] The present invention is a road extraction method based on remote sensing satellite images, which is implemented based on convolutional neural networks. The present invention uses road direction as prior knowledge to more accurately infer road areas, especially in occluded and noisy areas, thereby producing more complete and consistent road extraction results. At the same time, the present invention uses the road extraction task to assist the task of road direction prediction. The network proposed by the present invention consists of three components: a feature encoder, an iterative feature enhancement subnetwork, and a multi-task decoder.

[0007] The technical solution of the present invention is as follows: a road extraction method (IterNet) based on remote sensing satellite images, including a feature encoder, an iterative feature enhancement subnetwork and a multi-task decoder; the specific extraction steps are as follows;

[0008] Step 1: The remote sensing satellite image is converted into an RGB image and input into the feature encoder for feature extraction. The output resolution is a feature map that is 4 times the size of the original input image downsampled to extract road semantic features and road direction features;

[0009] Step 2: The features extracted in step 1 are sent to the iterative feature enhancement subnetwork; the iterative feature enhancement subnetwork includes multiple feature enhancement units, each of which contains a semantically guided feature enhancement module (SGFE), an orientation-aware feature aggregation module (OAFA) and two side branches;

[0010] Input the road semantic features and road direction features to the semantic guided feature enhancement module to obtain enhanced road direction features; the enhanced road direction features are divided into two branches, one branch is used for feature fusion, and the other branch is input to a side branch to obtain a preliminary road direction prediction result, and the side branch consists of two convolutional layers;

[0011] The semantic-guided feature enhancement module includes a feature selection part and a feature fusion part; the feature selection part includes a convolutional layer, an activation function layer, and an hourglass-shaped subnetwork;

[0012] In the feature selection part, the input road semantic features and road direction features are merged in a cascade manner, and the merged features are sent to the convolution layer and the activation function layer to obtain the single-channel feature confidence;

[0013] B=Sigmoid(Ψ 1 (cat(F e , F o ), Θ 1 )) (1)

[0014] Among them, B is the single channel feature confidence, cat(·,·) represents the cascade along the channel direction, Ψ 1 (·, θ 1 ) indicates that the parameter is Θ 1 The convolutional layer of F e Represents the semantic features of the road; F o Indicates road direction characteristics;

[0015] The generated single-channel feature confidence selects features in the numerical domains of road semantic features and road direction features respectively, and inputs them into the hourglass subnetwork; the hourglass subnetwork first downsamples the selected features and then upsamples them, which is conducive to expanding the receptive field to better capture the global context information;

[0016] F a =Ψ 2 ((1-B)·F e +B·F o , Θ 2 ) (2)

[0017] Among them Ψ 2 (·, θ 2 ) indicates that the parameter is Θ 2 The convolutional layer of F a It is the feature generated by feature selection;

[0018] The feature fusion part uses the fusion mask to perform weighted feature fusion on the input road semantic features, road direction features and the results of the feature selection part to obtain enhanced road direction features;

[0019] F′ o =(1+M)·F o +(1+M)·F e , (3)

[0020] M=Sigmoid(D(E(F a )))

[0021] Where D(·) and E(·) represent the encoding part and decoding part of the hourglass subnetwork, respectively, and F′o represents the enhanced road direction feature, M represents the fusion mask;

[0022] The preliminary road direction prediction result and the road semantic feature are input into the direction perception feature aggregation module to generate enhanced road semantic features; the enhanced road semantic features are divided into two branches, one for feature fusion and the other for input into another side branch to generate preliminary road segmentation results for supervised network learning;

[0023] The direction-aware feature aggregation module adjusts the shape and direction of the receptive field according to the preliminary road direction prediction results, focusing on the information area used for road extraction; the direction-aware feature aggregation module includes a residual structure of multiple branches, each branch includes a direction-aware deformable convolution (OD-Conv) and a ReLU unit; different branches use convolution kernels of different sizes and shapes; the features obtained by all branches are weightedly combined with the input road semantic features to output the enhanced road semantic features;

[0024] F′ e =F e +λ·∑ i Φ 0 (F e , α, θ i ) (4)

[0025] Among them, F e Represents the input road semantic features, F′ e represents the enhanced road semantic features; α represents the road direction obtained by the side branch from the road direction features, i represents the branch number, Φ 0 Denote the parameter as θ i The direction-aware deformable convolutional layer is used to balance the weights of the aggregated features and the original features.

[0026] W represents the parameters of the conventional 2D convolutional layer. The output of the conventional convolutional layer can be calculated by formula (5).

[0027]

[0028] Where X represents the input feature map, Y(p) represents the output value at position p. R represents the receptive field of 2D convolution. Taking a 3×3 convolution kernel as an example, the two-dimensional grid R can be defined as in formula (6).

[0029] R={(-1,-1), (-1,0),,..., (0,1), (1,1)} (6)

[0030] The convolution kernel of the direction-aware deformable convolution automatically adjusts the receptive field according to the preliminary road direction prediction result; the output of the direction-aware deformable convolution layer is;

[0031]

[0032] Where X represents the input feature map; Y(p) represents the output value at position p; R represents the receptive field; α(p) represents the predicted road direction angle at position p, and r(p, α) represents the angle applied to the coordinate p = [p x , p y ] rotation and translation; W(p n )The weight of the convolution kernel receptive field;

[0033]

[0034] The enhanced direction feature, the enhanced road semantic feature, the original road semantic feature and the original road direction feature are fused, and the fused feature is input into the next feature enhancement unit;

[0035] Step 3: The output features of the final sub-network are sent to the multi-task decoder to obtain road extraction results and road direction prediction results.

[0036] The convolution kernels of different sizes and shapes are 5×1, 1×5 and 3×3 respectively.

[0037] When r(p, α) is a non-integer, bilinear interpolation is used to sample the input feature map X.

[0038] The feature encoder uses ResNet50;

[0039] The beneficial effects of the present invention are as follows: the present invention solves the connectivity problem of the automated road extraction task in remote sensing satellite images, and the result output by the method is highly accurate and efficient; the present invention can make full use of the prior information contained in the road direction to assist in improving the accuracy of the result of the road extraction task; the proposed semantic-guided feature enhancement module can use the road semantic information to supplement and enhance the road direction information; the proposed OD-Conv can better use the road direction to automatically align the convolution kernel receptive field and the road area. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1 This is the overall network structure of the present invention.

[0041] Figure 2 Schematic diagram of the semantically guided feature enhancement module SGFE proposed in the present invention.

[0042] Figure 3 Schematic diagram of the direction-aware deformable convolution proposed in the present invention. The left side shows the receptive field on the input image, and the right side shows the proposed direction-aware deformable convolution.

[0043] Figure 4Schematic diagram of the direction-aware feature aggregation module OAFA proposed in the present invention.

[0044] Figure 5 It is the iterative feature enhancement sub-network proposed in the present invention.

[0045] Figure 6 The experimental results of the present invention are shown in Figure 2. From left to right, they are (a) remote sensing satellite image, (b) road extraction truth, (c) the result of using only multi-task learning, (d) the result of using the SGFE module, (e) the result of using the OAFA module, and (f) the result of the method of the present invention. DETAILED DESCRIPTION

[0046] The specific implementation of the present invention is further described below in conjunction with the accompanying drawings and technical solutions.

[0047] Figure 1 The overall structure of IterNet proposed in the present invention is composed of a remote sensing satellite image feature encoder, an iterative feature enhancement subnetwork and a multi-task decoder. After the remote sensing satellite image is converted into an RGB image, it is sent to the feature encoder proposed in the present invention for feature extraction, and the extracted remote sensing image features are sent to the iterative feature enhancement subnetwork. The iterative feature enhancement subnetwork is composed of multiple feature enhancement units, each of which contains an SGFE module, an OAFA module and two side branches. In each feature enhancement unit, the input features are respectively extracted from the road semantic features and road direction features of the road through convolution operations, and the features are enhanced through the SGFE module and OAFA module proposed in the present invention, and the fused features are output and sent to the next feature unit.

[0048] Figure 2 The SGFE module proposed in the present invention can enhance the road direction features by using the road semantic information. Figure 4 The OAFA module proposed in the present invention can use the preliminary road direction prediction results to adjust the shape and direction of the OD-Conv receptive field and automatically focus on the information area used for road extraction. Figure 3 The OD-Conv proposed in the present invention can adaptively focus on the information area of ​​road extraction, aggregate the information of the road surface and its surrounding environment, and make the road extraction result more accurate. Figure 5 The iterative feature enhancement subnetwork proposed by the present invention uses the output of the previous feature enhancement unit as the input of the next unit. The first feature enhancement unit uses the output of the feature encoder as the input feature to enhance the feature. The last feature enhancement unit inputs its enhanced features into the two-branch multi-task decoder, and after a series of convolution and upsampling operations, the image resolution is restored and the final prediction result is output.

[0049] The present invention uses the DeepGlobe dataset and the Massachusetts dataset to train and test the method. The DeepGlobe dataset includes images from three different regions of Thailand, Indonesia and India, with a total area of ​​2,220 square kilometers. The dataset contains satellite images of cities and suburbs and corresponding pixel-level truth values. The ground resolution of the remote sensing image is 50 cm, and the pixel resolution is 1024×1024. The present invention uses 4,696 images for training and 1,530 images for testing. The Massachusetts dataset consists of 1,171 images of Massachusetts. The size of each image is 1,500×1,500 pixels, and the spatial resolution is 1m per pixel. The dataset is collected from aerial images and contains satellite images of cities and suburbs and corresponding pixel-level truth values. The present invention uses the validation set of the dataset for testing.

[0050] The present invention is mainly divided into two subtasks: data processing and network training. In data processing, the present invention follows the setting of Anil Batra et al. and uses the road surface truth value to generate the truth value of the road direction prediction task. First, the pixel-level annotation is skeletonized to generate the road centerline truth value, and then the road centerline truth value is smoothed using the smoothing algorithm method proposed by David H Douglas et al. For each section of the road centerline, the present invention generates a series of key points using interpolation amplification and calculates the direction angle between each two adjacent key points. In order to simplify the direction estimation task, the present invention sets the interval size to 10° to cluster the direction. The present invention expands the skeletonized road direction truth value outward by 20 pixels to generate a pixel-level road direction truth value for the supervision of the road direction prediction task. Finally, the present invention uses 36-dimensional direction classes to annotate the direction of the road surface in the road area, and the non-road area is marked as a background class, converting the road direction prediction problem into a pixel-level classification problem of converting the road direction into 36 types of angles and 1 type of background.

[0051] In network training, all experiments of the present invention are carried out on a server with 16GB RAM and two NVIDIA 2080Ti GPUs (11GB), using the PyTorch framework to implement training and testing. During training, the present invention uses random cropping operations on images of size 512×512 to generate images of size 256×256 as input for IterNet. In addition, the present invention uses data enhancement operations such as random horizontal flipping, mirroring, and rotation to avoid overfitting. The present invention uses a stochastic gradient descent method to optimize the entire network with a momentum value of 0.9, a batch size of 32, and a total of 160 rounds of training. The initial learning rate is 2e-2, which decreases by 10 times after the 60th round, is set to 5e-4 after the 90th round, and decreases by 10 times after the 120th round. During the inference process, the Deepglobe dataset image size is 512×512, the Massachusetts dataset image size is 320×320, and the image is directly input into the network without further cropping operations.

[0052] The comparison method selected in the specific implementation of the present invention is StckHourglass. The StckHourglass method simply uses road direction information for supervision to obtain better road extraction results. In order to make a fair comparison, the StckHourglass method uses its public code and its recommended parameter settings, and uses the same pre-trained network and is tested on the same test set. From the final experimental results, the IterNet method proposed in the present invention achieved the best performance in all different experimental settings in terms of indicators IoU, F1, Precision and Recall. The higher the indicators such as IoU, F1, Precision and Recall, the higher the accuracy of the method. The specific experimental results on the Deepglobe dataset are shown in Table 1 below:

[0053] Table 1 Specific experimental results on the Deepglobe dataset

[0054]

[0055]

[0056] The specific experimental results on the Massachusetts dataset are shown in Table 2 below:

[0057] Table 2 Specific experimental results on the Massachusetts dataset

[0058]

Claims

1. A road extraction method based on remote sensing satellite images, characterized in that: The road extraction method based on remote sensing satellite images includes a feature encoder, an iterative feature enhancement sub-network and a multi-task decoder; the specific extraction steps are as follows: Step 1: The remote sensing satellite image is converted into an RGB image and input into the feature encoder for feature extraction. The output resolution is a feature map that is 4 times the size of the original input image downsampled to obtain road semantic features and road direction features; Step 2: The features extracted in step 1 are fed into the iterative feature enhancement sub-network; The iterative feature enhancement subnetwork includes multiple feature enhancement units, each of which contains a semantically guided feature enhancement module, a direction-aware feature aggregation module, and two side branches. Inputting the road semantic features and the road direction features into the semantic guided feature enhancement module to obtain enhanced road direction features; The enhanced road direction feature is divided into two branches, one for feature fusion and the other for inputting into a side branch to obtain a preliminary road direction prediction result. The side branch consists of two convolutional layers. The semantic-guided feature enhancement module includes a feature selection part and a feature fusion part; The feature selection part includes convolutional layer, activation function layer and hourglass sub-network; In the feature selection part, the input road semantic features and road direction features are merged in a cascade manner, and the merged features are sent to the convolution layer and the activation function layer to obtain the single-channel feature confidence; B=Sigmoid(Ψ1(cat(F e ,F o ),Θ1)) (1) Among them, B is the single channel feature confidence, cat(·,·) represents the cascade along the channel direction, Ψ1(·,Θ1) represents the convolution layer with parameter Θ1; F e Represents the semantic features of the road; F o Indicates road direction characteristics; The generated single-channel feature confidence scores are used to select features in the numerical domains of road semantic features and road direction features, respectively, and are input into the hourglass subnetwork; the hourglass subnetwork first downsamples the selected features and then upsamples them; F a =Ψ2((1-B)·F e +B·F o ,Θ2) (2) where Ψ2(·,Θ2) represents a convolutional layer with parameter Θ2; F a It is the feature generated by feature selection; The feature fusion part uses the fusion mask to perform weighted feature fusion on the input road semantic features, road direction features and the results of the feature selection part to obtain enhanced road direction features; F′ o =(1+M)·F o +(1+M)·F e , (3) M=Sigmoid(D(E(F a ))) Where D(·) and E(·) represent the encoding part and decoding part of the hourglass subnetwork, respectively, and F′ o represents the enhanced road direction feature, M represents the fusion mask; The preliminary road direction prediction result and the road semantic feature are input into the direction perception feature aggregation module to generate enhanced road semantic features; the enhanced road semantic features are divided into two branches, one for feature fusion and the other for input into another side branch to generate preliminary road segmentation results for supervised network learning; The direction-aware feature aggregation module adjusts the shape and direction of the receptive field according to the preliminary road direction prediction results, focusing on the information area used for road extraction; the direction-aware feature aggregation module includes a residual structure of multiple branches, each branch includes a direction-aware deformable convolution and a ReLU unit; different branches use convolution kernels of different sizes and shapes; the features obtained by all branches are weightedly combined with the input road semantic features to output the enhanced road semantic features; F′ e =F e +λ·∑ i Φ0(F e ,a,i i ) (4) Among them, F e represents the semantic features of the road, F′ e represents the enhanced road semantic features; α represents the road direction obtained by the side branch from the road direction features, i represents the branch number, and Φ0 represents the parameter θ i The direction-aware deformable convolution layer is used, and λ is a hyperparameter of the residual structure, which is used to balance the weights of the aggregated features and the original features; The convolution kernel of the direction-aware deformable convolution automatically adjusts the receptive field according to the preliminary road direction prediction result; the output of the direction-aware deformable convolution layer is; Where X represents the input feature map; Y(p) represents the output value at position p; R represents the receptive field; α(p) represents the predicted road direction angle at position p, and r(p,α) represents the angle applied to the coordinate p = [p x ,p y ] rotation and translation; W(p n )The weight of the convolution kernel receptive field; The enhanced direction feature, the enhanced road semantic feature, the original road semantic feature and the original road direction feature are fused, and the fused feature is input into the next feature enhancement unit; Step 3: The output features of the final sub-network are sent to the multi-task decoder to obtain road extraction results and road direction prediction results.

2. The road extraction method based on remote sensing satellite images according to claim 1 is characterized in that: The convolution kernels of different sizes and shapes are 5×1, 1×5 and 3×3 respectively.

3. The road extraction method based on remote sensing satellite images according to claim 1 or 2, characterized in that: When r(p,α) is a non-integer, bilinear interpolation is used to sample the input feature map X.

Images