Asymmetric Adversarial Learning Method for Multi-Scale Road Network Extraction Based on GF-2 Remote Sensing Images

Abstract

The present invention discloses an asymmetric adversarial learning method for multi-scale road network extraction based on GF-2 remote sensing images. The asymmetric adversarial multi-scale road feature learning method in complex scenarios of this method enhances the road extraction ability from two aspects. On the one hand, an adversarial learning network architecture is adopted to strengthen the continuity of spatial labels. Compared with non-reverse training, more accurate and smooth results can be generated. On the other hand, an asymmetric encoder-decoder structure is used as the segmentation network to reduce the input of redundant features, gradually fuse high-level features with low-level features, and retain more detailed information. Finally, structural supervision is introduced into the discriminator to guide the generator to generate an annotated road network similar to the manually annotated road network structure, introduce road linear features into the extraction process, and solve the connectivity problem of road extraction results. Using the present invention to achieve the extraction of road features can effectively improve the accuracy and connectivity of road feature extraction, and has higher precision.

Description

Technical Field

[0001] The present invention relates to spatio-temporal remote sensing, road image segmentation, road network information extraction, and adversarial learning technologies, and particularly relates to an asymmetric adversarial learning method for multi-scale road network extraction based on GF-2 remote sensing images. Background Art

[0002] As one of the basic tasks in the field of remote sensing, road extraction has wide applications, such as geographical information updating, autonomous driving, urban planning, disaster emergency navigation, etc. Traditional road data is mainly obtained through manual visual recognition, which consumes human resources and far from meets the requirements of data updating. Automatic road extraction based on high-resolution remote sensing images provides an efficient, low-cost, and reliable solution for receiving dynamic road information. Automatic recognition of road networks based on satellite remote sensing data greatly reduces labor costs, helps to achieve the automation and acceleration of road monitoring, and has higher accuracy.

[0003] Therefore, this patent proposes an asymmetric adversarial learning method for multi-scale road network extraction based on GF-2 remote sensing images. To solve the problems of road extraction at different scales and fragmented extraction results in complex scenarios. This method uses a generative adversarial network for road segmentation, improves the connectivity of spatial labels, and refines the segmentation results, making the road extraction results more connected. Road structure information is introduced into the network training process, and by adding a structural similarity loss to the discriminator, the training process of the generator is effectively guided, so that the connectivity problem of road extraction is solved while road segmentation is performed without additional training. The addition of an Atrous Spatial Pyramid Pooling (ASPP) module and partial feature fusion solves the problem of low-dimensional feature information loss, retains more narrow road information, and makes the network applicable not only to coarse-grained road segmentation in complex scenarios but also to fine-grained road segmentation in complex scenarios. The invention can handle interruptions caused by shadows and occlusions, extract roads of different widths and different materials, process roads with incomplete spectral and geometric features, and improve the narrow road extraction ability. Summary of the Invention

[0004] The technical problem to be solved by the present invention is: to propose an asymmetric adversarial learning method for multi-scale road network extraction based on GF-2 remote sensing images, to achieve the extraction of multi-scale road features in complex scenarios, and to ensure accuracy and connectivity.

[0005] The technical solution of the present invention is as follows: An asymmetric adversarial learning method for multi-scale road network extraction based on GF-2 remote sensing images. First, GF-2 remote sensing data is obtained, and the data is stored, preprocessed, and dataset partitioned to form a road extraction sample dataset. Based on the road extraction sample dataset, a road segmentation network - RNet integrating a multi-scale asymmetric encoding and decoding structure is constructed to obtain a preliminary road extraction result. Based on the preliminary road extraction result and the true road labels, a discriminator - DNet of MSC-RoadGAN is constructed. The labels generated by RNet and the true labels are jointly input into DNet, and structural supervision is introduced into DNet. DNet determines whether the incoming label is a true label or a label generated by RNet. Based on the RNet and DNet networks, MSC-RoadGAN is constructed. In the model, RNet and DNet are in mutual confrontation, and DNet guides RNet to generate an annotated road network similar to the manually annotated road network structure to solve the problem of road segmentation connectivity. Based on the constructed MSC-RoadGAN road extraction model, training is carried out on the training set to optimize the model parameters until the MSC-RoadGAN model converges. Based on the converged MSC-RoadGAN road extraction model, prediction is made on the test set, and the road extraction result is obtained using the output probability. The specific steps are as follows:

[0006] (1) Obtain GF-2 remote sensing data, and store, preprocess, and partition the dataset.

[0007] In step (1), for the obtained GF-2 remote sensing data, data reduction is performed using a standard data structure.

[0008] Furthermore, efficient storage of GF-2 remote sensing data is carried out. Either a relational database can be used for logical storage and management of the data in a structured manner, or a NoSQL database can be used in an unstructured manner; either a centralized high-performance storage device can be adopted, or a distributed cluster method can be used for physical storage and optimization of the building remote sensing image data.

[0009] Even further, preprocessing of the road remote sensing image data is carried out. The specific steps include:

[0010] (a) Write spatial information. On the premise of ensuring accuracy, the spatial information of the study area is written into the high-resolution remote sensing image.

[0011] (b) Geometric registration. Use Arcgis tools to perform geometric registration on the road centerline vector map and the high-resolution remote sensing image.

[0012] (c) Image cutting. The registered sample label data and the original remote sensing image are cut into images of 512 * 512 pixels.

[0013] (d) Manually delineate the road vectors in the sample dataset. Open the sample images using the Photoshop software. Based on the road centerline vector map, manually delineate the road vectors. The delineated vector map is saved in the PNG format as the sample label data. The label data images are made to correspond one-to-one with the original data images and given the same name. The sample label data and the original data initially form the road extraction sample database.

[0014] (e) Image filtering. Compare the original data in (d) with the corresponding sample label data, and filter out the images without roads. Thus, the road extraction sample database is constructed.

[0015] (2) Perform road semantic segmentation on the road extraction sample dataset obtained in step (1) to construct the road extraction model - MSC - RoadGAN. The specific steps are as follows:

[0016] (a) Construct the road segmentation network - RNet of MSC - RoadGAN using an asymmetric encoder - decoder structure. The encoder of the RNet network is used to extract multi - scale features of the input image. It is a hierarchical rearrangement version of ResNet. The striding operations in the ResNet network are designed within three convolutional blocks, gradually reducing the feature size. Through 4 - layer convolutional operations, a feature map with a size of 1 / 8 of the original image size is obtained. And the ASPP algorithm is used to extract multi - scale context information through atrous convolutional layers. ASPP contains one 1*1 convolution and three 3*3 atrous convolutions with rates of (6, 12, 18), which can extract multi - scale context information, increase the network receptive field, enhance the network's ability to obtain small - scale context, and help the entire network obtain stronger robustness. Then we fuse the second - layer features and the last - layer features of the RNet network to obtain more road detail features and enhance the RNet network's ability to obtain multi - scale context features. In the decoder stage, three transposed convolutional layers are used to enlarge the feature map and smooth the boundaries, simplifying the number of decoder network layers and shortening the network training time.

[0017] (b) Construct the discriminator network - DNet of MSC - RoadGAN. Its network structure is based on FCN and can accept inputs of different sizes. The input of DNet is the class probability map output by RNet and the ground truth label. It consists of 5 convolutional layers with a convolutional kernel of 4*4, the number of channels being (64, 128, 256, 512, 1), the stride being 2, and there is a Leaky - ReLU after each convolutional layer. The final output is a confidence map with a dimension of H×W×1, where H is the height of the original image and W is the width of the original image. For each pixel x in the confidence map, it is set to 1 if it comes from the ground truth label and 0 if it comes from the class probability map. By using the confidence map, the prediction can be forced to split, making the parameters closer to the ground truth label spatially. The structural similarity (SSIM) loss is applied in DNet to calculate the structural similarity loss between the ground truth label and the 1 / 8 result map of the class probability map, ensuring that the road structure in the image predicted by RNet is more similar to the label image.

[0018] (3) Based on the constructed MSC - RoadGAN road extraction network, train on the training set to optimize the model parameters until the MSC - RoadGAN network converges.

[0019] (a) Road feature model training. Initially, set the model learning rate to 0.01, train the training set and the validation set for 300 rounds, observe the training results of the model such as ACC, precision, recall, F1, etc. On the premise of ensuring that the precision and recall values are not much different, use the method of parameter tuning to improve the precision value.

[0020] (b) Road feature model validation. Retain the optimal model in the training rounds of process (a), use this model to generate a prediction set for the first - round validation of the validation set, view the pictures with lower prediction results, and add the same - class data to the training set so that the model can learn the features of this class.

[0021] (c) Repeat the processes of (a) and (b) until the model accuracy reaches an acceptable level, retain the optimal model, and thus the construction of the road extraction model is completed.

[0022] (4) Based on the converged MSC - RoadGAN road extraction model, make predictions on the test set and obtain the road extraction results using the output probability. The specific steps include: using the road extraction model to extract road features from the test set, storing the extraction result precision set, and the grayscale map of the overall road dataset.

[0023] The advantages of the present invention compared with the prior art are as follows:

[0024] 1. Apply the generative adversarial network to road segmentation to improve the connectivity of spatial labels and refine the segmentation results, making the road extraction results more connected.

[0025] 2. Use an asymmetric encoder-decoder structure to construct the road segmentation part of the network, effectively retaining high-dimensional features, reducing noise interference, and being more suitable for road segmentation in complex scenarios. However, without the addition of low-level features, the network will be unable to recognize narrow roads in complex scenarios. Therefore, ASPP and a layer of feature fusion are added to the last layer of our encoder to solve the recognition of narrow roads in complex scenarios.

[0026] 3. Add the SSIM loss to the discriminator of the network, combining the road pixel information with its position information, which is equivalent to recognizing the road structure and has the same effect as road extraction based on road boundary lines. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 It is the overall flowchart of the present invention.

[0028] Figure 2 It is the network structure diagram of MSC-RoadGAN. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0029] To enable those skilled in the art of the present technology to better understand the solutions of the embodiments of the present invention, the embodiments of the present invention will be further described in detail below with reference to the drawings and embodiments.

[0030] As Figure 1 shown, the present invention includes the following steps:

[0031] 1. Data storage and preprocessing: Store GF-2 remote sensing data, which can be stored using a centralized high-performance computing facility, utilizing a high-performance server, high-speed network, and high-speed storage device cluster for physical storage of high-resolution remote sensing data, or using a distributed cluster method for physical storage of high-resolution remote sensing data of buildings in a distributed computing environment and storage structure. The steps of data preprocessing include: writing spatial information, georegistration, image cutting, manually delineating the road vector of the sample data set, and image filtering.

[0032] Writing spatial information: Write the spatial information of the study area into the high-resolution remote sensing image while ensuring accuracy. Georegistration: Use Arcgis tools to georegister the road centerline vector map and the high-resolution remote sensing image.

[0033] Image cutting: Cut the registered sample label data and the original remote sensing image into images of 512 * 512 pixels. Manually draw the road vector of the sample data set: Open the sample image using the PhotoShop software. Based on the road centerline vector map, manually draw the road vector. The drawn vector map is saved in PNG format as the sample label data. Corresponding the label data image with the original data image one by one and naming them the same, the sample label data and the original data initially form a road extraction sample database.

[0034] Image filtering: Compare the original data in the road extraction sample database with the corresponding sample label data, and filter out the images without roads. Thus, the road extraction sample database is constructed.

[0035] 2. Perform road extraction on the road sample data set to construct a road extraction model. The specific steps include: constructing a road feature learning model, training the road feature model, testing the road feature model, optimizing the model, and storing the optimal model;

[0036] Construct the road feature learning model MSC-RoadGAN: First, construct the road segmentation network - RNet of MSC-RoadGAN. The network encoder is used to extract multi-scale features of the input image. It is a hierarchical rearrangement version of ResNet. The striding operations in the network are designed within three convolutional blocks, gradually reducing the feature size. Through 4-layer convolutional operations, a feature map with a size of 1 / 8 of the original image size is obtained. And the ASPP algorithm is used to extract multi-scale context information through atrous convolutional layers. ASPP contains one 1*1 convolution and three 3*3 atrous convolutions with rates of (6, 12, 18), which can extract multi-scale context information, increase the receptive field of the network, enhance the network's ability to obtain small-scale context, and help the entire network obtain stronger robustness. Then we fuse the features of the second layer and the last layer of the network to obtain more road detail features and enhance the network's ability to obtain multi-scale context features. In the decoder stage, 3 deconvolution layers are used to expand the feature map and smooth the boundaries, simplifying the number of decoder network layers and shortening the network training time. Construct the discriminator network - DNet of MSC-RoadGAN. Its network structure is based on FCN and can accept inputs of different sizes. The input of DNet is the class probability map and the ground truth label output by RNet. It consists of 5 convolutional layers with a convolutional kernel of 4*4 and channels of (64, 128, 256, 512, 1), and a stride of 2. There is a Leaky-ReLU after each convolutional layer. The final output is a confidence map with a dimension of H×W×1. For each pixel x in the confidence map, it is set to 1 if it comes from the ground truth label and 0 if it comes from the class probability map. By using the confidence map, the prediction can be forced to split, making the parameters closer to the ground truth label in space. Apply the SSIM loss in DNet to calculate the structural similarity loss between the ground truth label and the 1 / 8 result map of the class probability map to ensure that the image predicted by RNet is more similar to the road structure in the label image.

[0037] Training of the road feature model: Initially, set the model learning rate to 0.01, and train the training set and the validation set for 300 rounds. Observe the training results of the model, such as ACC, precision, recall, F1, etc. On the premise of ensuring that the precision and recall values are not much different, use the method of parameter tuning to increase the precision value.

[0038] Testing of the road feature model: Retain the optimal model in the training rounds of the road feature model, and use this model to conduct the first round of testing on the test set to generate a prediction set. Check the pictures with lower prediction results, and add data of the same category to the training set so that the model can learn the features of this category.

[0039] Model tuning: Repeat the model training and model testing process until the model accuracy reaches a usable level, retain the optimal model, and the road extraction model is completed.

[0040] Store the optimal model: Use centralized high-performance computing facilities or utilize distributed computing environments and storage structures to retain the last round of model optimization.

[0041] 3. Extract remote sensing image data sets from all roads based on the road model. The specific steps include: extracting all road features and storing prediction results;

[0042] Extraction of all road features: All road remote sensing image datasets are placed in the test set, and the optimal road extraction model is used to extract road features. The precision set of road feature extraction results is the road extracted data.

[0043] Storing prediction results: Using centralized high-performance computing facilities or utilizing distributed computing environments and storage structures to physically store the extracted road data.

[0044] It should be noted that the method of the embodiment of the present invention is suitable for multi-scale road feature extraction in complex scenes.

[0045] The embodiments of the present invention are described in detail above. The present invention is explained in this article using specific implementation methods. The description of the above embodiments is only used to help understand the method of the present invention. At the same time, for those skilled in the art, according to the ideas of the present invention, there will be changes in the specific implementation methods and application scopes. In summary, the content of this specification should not be understood as limiting the present invention.

Claims

1. An asymmetric adversarial learning method for multi-scale road network extraction based on GF-2 remote sensing images, characterized in that The implementation steps of this method are as follows: Step (1): Obtain GF-2 remote sensing data, store and preprocess the GF-2 remote sensing data, mark the roads to construct a road extraction sample database, and split it into a training set and a test set according to the public division method; Step (2): Construct a road segmentation network - RNet that integrates a multi-scale asymmetric encoder-decoder structure. Finally, obtain a road segmentation probability map through a Sigmoid classifier. The asymmetric encoder-decoder structure road segmentation network includes an ASPP module, a feature fusion module, and a decoder module; The ASPP module includes parallel sampling of the input feature map using dilated convolutions with different sampling rates, and then connecting the obtained results together to capture image context information at multiple scales; The feature fusion module includes fusing low-dimensional features with high-dimensional features, enabling the network to retain more high-resolution detail information contained in high-level feature maps and retain more details; The decoder module includes an asymmetric decoder that reduces the repeated operations in the overlapping parts of the fields and improves the network operation efficiency; Step (3): Construct a discriminator - DNet of MSC-RoadGAN, input the labels generated by RNet and the real labels into DNet together, and introduce structural supervision in DNet. DNet determines whether the incoming label is a real label or a label generated by RNet; Step (4): Based on the RNet and DNet networks, construct MSC-RoadGAN. RNet and DNet confront each other. DNet guides RNet to generate an annotated road network similar to the manually annotated road network structure to solve the problem of road segmentation connectivity; Step (5): Based on the constructed MSC-RoadGAN road extraction model, train it on the training set and optimize the model parameters until the MSC-RoadGAN model converges; Step (6): Based on the converged MSC-RoadGAN road extraction model, predict the test set and obtain the road extraction result using the output probability; In step (3), the discriminator training includes adding a structural similarity loss to the discriminator, introducing graphic features into the pixel segmentation process, and guiding the segmentation result of the generator to be more similar to the label contour; In step (5), based on the constructed MSC-RoadGAN road extraction model, training on the training set includes: Step 51: Initialize the parameters of the MSC-RoadGAN road extraction model, fix the cropping size of the training set, the number of input images per batch, and the initial learning rate; Step 52: The road segmentation network is trained using the Adam optimizer. The road segmentation network performs forward propagation, and the output result obtains a road segmentation probability map through a Sigmoid classifier. Calculate the loss value between the real label and the probability map according to the segmentation loss function, where the segmentation loss function is jointly composed of binary cross-entropy loss and mean square loss; Step 53: Input the road segmentation probability map and the ground truth label into the discriminator. The discriminator network adopts the FCN structure and outputs a confidence map with the dimension of H × W × 1, where H is the height of the original image and W is the width of the original image. For each pixel x in the confidence map, it is set to 1 if it comes from the ground truth label and 0 if it comes from the probability map. The network parameters are continuously updated through backpropagation until the network converges.

2. The asymmetric adversarial learning method for multi-scale road network extraction based on GF-2 remote sensing images according to claim 1, wherein, In the said step (1), for the storage of GF-2 remote sensing images, either a centralized high-performance storage device or a distributed cluster method can be adopted for physical storage and optimization of the data. For the processing of GF-2 remote sensing image data, it includes image cutting, position alignment, image filtering, and road annotation. For the division of GF-2 remote sensing image data, in the publicly used division method with a ratio of 5:1, it is split into a training set and a test set.

3. The asymmetric adversarial learning method for multi-scale road network extraction based on GF-2 remote sensing images according to claim 1, characterized in that In the said step (2), an asymmetric encoder-decoder structure is adopted to construct the road segmentation network, including: Step 21: Adopt an asymmetric structure to reduce the repeated operations in the overlapping part of the domain and improve the network operation efficiency. Step 22: Gradually fuse the high-level features with the low-level features to achieve multi-scale recognition of image features, making the output multi-scale information richer. Step 23: Add an ASPP layer to expand the network receptive field, capture multi-scale context information, introduce the feature information at the corresponding scale into the deconvolution process, and provide multi-scale and multi-level information for subsequent image segmentation.

4. The asymmetric adversarial learning method for multi-scale road network extraction based on GF-2 remote sensing images according to claim 1, characterized in that In the said step (4), construct the road extraction model MSC-RoadGAN and adopt the GAN architecture for road network extraction. The MSC-RoadGAN model is divided into two parts: the generator and the discriminator. The road segmentation network is used as the generator, and the segmentation result and the ground truth label are jointly input into the discriminator, which is used to distinguish whether the input sample comes from the generator or the ground truth label.

5. The asymmetric adversarial learning method for multi-scale road network extraction based on GF-2 remote sensing images according to claim 1, characterized in that In the said step (6), based on the converged MSC-RoadGAN road extraction model, predict the test set, including: Step 61: Input the test set images and perform normalization operations on them. Step 62: Input the normalized images into the already trained model to obtain the road segmentation probability map of the test set, and then determine the areas belonging to the road through a threshold of 0.5 to obtain the final road segmentation map.

Images