An intelligent method for extracting road information from optical remote sensing images under cloud influence

Abstract

This paper combines the respective advantages of domestic high-resolution remote sensing imagery and OpenStreetMap data to propose an intelligent method for extracting road information from optical remote sensing images under the influence of clouds. This method facilitates the extraction of road information without cloud removal even when the sample contains clouds that obscure road surface information. The method primarily includes: preprocessing remote sensing imagery and OSM data to create a road dataset and a corresponding segmentation mask dataset; extracting cloud shadow regions from remote sensing images using the Fmask cloud detection algorithm to obtain a potential cloud layer; and then using the fill transformation method from morphological transformations combined with scattering correlation calculations to obtain a potential cloud shadow layer, masking out cloud-containing and cloud-shadow regions; and constructing a road information extraction model for unlabeled samples based on deep learning technology. By combining the advantages of remote sensing imagery and OSM road network data, this model effectively addresses the incomplete road information extraction problem caused by cloud cover in optical remote sensing data.

Description

[0001] The present invention relates to extracting road information from remote sensing images under the influence of clouds based on AI technology, and belongs to the research fields of artificial intelligence, intelligent processing of image information, and intelligent driving. Background Art

[0002] With the rapid development of remote sensing technology, images obtained from remote sensing sensors (mounted on drones or satellites) have made great contributions to disaster / emergency management, urban planning, and target detection. Road networks are also an indispensable part of our daily lives in urban planning, traffic management, GPS navigation, road condition monitoring, etc. In a rapidly developing region, the road network changes frequently. Therefore, it is necessary to extract the latest road network to effectively support space applications. Accurate and up-to-date road network information is the foundation of our daily lives, whether it is used for urban management, traffic planning, vehicle navigation, intelligent transportation systems, etc. Therefore, continuous extraction of road networks is very important, especially for rapidly growing regions. With the rapid development of remote sensing technology, images can be easily acquired by remote sensing sensors mounted on drones or satellites, which makes it possible to update road networks and adopt timely methods for road extraction from remote sensing images.

[0003] Deep learning has received great attention in a range of semantic understanding tasks because of its ability to cover multiple layers of features. In order to reduce the number of network parameters that need to be learned and improve the real-time processing capabilities of the network, the LinkNet architecture was proposed. Leveraging the powerful pre-trained VGG encoder and U-Net architecture, Iglovikov et al. proposed TernausNet, which was part of the winning solution (the first out of 735) in the Carvana image mask challenge. In the field of road extraction, Zhou et al. proposed a VHR remote sensing road extraction method based on D-LinkNet. D-LinkNet was released by LinkNet, which embeds dilated convolutional layers into the middle part to integrate multi-scale features while retaining detailed information. Zhang et al. proposed an FCN-based road extraction framework that adopts an integration strategy called spatial consistency (SC) to address the imbalance problem between road and background areas in aerial images.

[0004] Furthermore, cloud occlusion can severely impact feature extraction in remote sensing images (Lin et al., 2014). Feature extraction is typically performed after cloud removal (Sarukkai et al., 2020). However, for specific roads, such as narrow roads, which only occupy a small pixel fraction in remote sensing images, extracting roads after cloud removal is not only cumbersome but can also easily result in road loss. Current road extraction methods are performed on cloud-free remote sensing images, failing to consider the impact of cloud occlusion on road extraction. Consequently, road extraction performance in cloud-covered remote sensing images is significantly reduced. Li et al. (2021) used Perlin noise (Perlin, 2002) to simulate clouds (Enomoto et al., 2017). They then fused the simulated clouds with an RGB remote sensing image using the alpha coefficient to generate a simulated cloud-occluded image. The inclusion of a cloud mask further evaluates the performance of the network model in road prediction under simulated cloud-occluded areas. Due to the varying transparency of the simulated clouds, the visual effect of varying cloud thickness is produced. The cloud coverage of the entire cloud simulation image dataset is approximately 40% to 70%, with an average coverage of 3% to 23% for simulated thick clouds and 28% to 59% for simulated thin clouds. Using these simulated cloud images to train the network model, we hope that the final model will be robust to cloud occlusion by both thick and thin clouds.

[0005] The aforementioned methods primarily extract road information from cloud-free remote sensing images and achieve satisfactory results. However, for cloud-covered remote sensing images, road information extraction methods based on optical remote sensing images often produce incomplete road information. Therefore, the present invention designs an intelligent road information extraction method for optical remote sensing images under cloud cover. This method is of great significance for improving urban road network extraction, reducing the cost of manual annotation, and intelligently extracting road information under cloud cover. Summary of the Invention

[0006] A method for intelligently extracting road information from optical remote sensing images under the influence of clouds, characterized by comprising the following steps:

[0007] Step 1: Preprocess remote sensing images and Open Street Map data to mark a small amount of segmentation mask samples of road information;

[0008] Step 2: Perform cloud detection on the remote sensing image, mark the cloud-containing part of the image, and set it not to enter the network;

[0009] Step 3, using prior knowledge such as road types and pixel-level similarity to set buffers for the road centerlines obtained from the Open Street Map data, a large number of road information samples;

[0010] Step 4: Build a training sample library. Use the cloud-free remote sensing image data and the large amount of mask data generated in step 3 as input to build a series of training sample data.

[0011] Step 5: Build a SADU-net (Self-attention and Densely connected U-net) network model. This model extracts features from a global-local perspective and uses spatial attention and dense blocks to extract road information from cloud-free remote sensing images.

[0012] Step 6: SADU-net network model training. The SADU-net network model is trained by using spatial attention to focus on the spatial coherence of the road and the transferability of dense blocks to contextual relationships.

[0013] Step 7, SADU-net network model performance evaluation;

[0014] Step 8: Combine the complete road information without clouds with the Open Street Map data to predict the cloud-containing part, and then generate a complete road information segmentation map after splicing.

[0015] In step 1, the remote sensing image is aligned with the road network data of the open street map; the purpose is to obtain the road network data corresponding to the remote sensing image to provide a basis for subsequent cropping.

[0016] In step 2, the Fmask method is used to detect clouds. Rules based on the physical properties of clouds and cloud shadows are used to extract potential cloud layers and potential cloud shadow layers. The segmented potential cloud layers and geometric relationships are then used to match the potential cloud shadow layers to generate the final cloud and cloud shadow masks. Cloud-containing portions are excluded from the neural network prediction process.

[0017] In step 3, the centerline of the road network obtained from the open street map data is combined with prior knowledge to generate reliable road segmentation information. The centerline is used as the axis, combined with the road type information given in the road network, to roughly determine the width of the road. The road and background information are distinguished by the similarity between pixels, forming the boundary of the segmentation information, and obtaining the road and the corresponding segmentation mask image.

[0018] In step 4, the cloud information detected in step 2 is used to divide the image into cloud-containing areas and cloud-free areas, and the remote sensing image data obtained from the cloud-free areas and the mask data generated by the road network data are cut equally, and the size is set to 512×512 pixels. The image is cropped with a repetition rate of 0.5 and a step size of 256. The cropped image is still 512×512 pixels, and 0 is filled in at the edges or in irregularities to meet the size of 512; then the samples are set in a 7:3 ratio and divided into a training set and a prediction set.

[0019] In step 5, the SADU-net model is based on the U-net network, which has three parts. The first part of the network is mainly the encoder, consisting of four dense blocks. Each dense block consists of a batch normalization layer, a Relu activation function, and a 3×3 convolution kernel. Each node comprehensively considers the information of all previous nodes. BN layers, 1×1 convolution layers, and 2×2 average pooling layers are placed between different dense blocks. The second part is the connection part. Self-Attention is applied to the original image, and a 1×1 convolution is performed on each channel. Through different mapping functions, a transposed point multiplication operation is performed. Finally, the result of the third part is added to extract information from the image, considering the global information of the road. The third part is the upsampling process. The encoded output of the corresponding layer is concatenated, the image is decoded, and the size is restored to 512×512, and the result is output.

[0020] In step 6, first set the learning rate, batch size, and training times, and use the stochastic gradient descent method to train SADU-net. The loss function uses the binary cross entropy (BCE) loss combined with the Dice coefficient, and its formula is defined as:

[0021]

[0022] N is the number of images, Y i is the input image, G i It's Y i The corresponding label, F(Y i ) is for Y i The prediction result graph.

[0023] In step 7, sample data is input to test the entire SADU-net network model. When the loss function value tends to be stable and minimized, the network performance is optimized. The optimal SADU-net network is used to extract road information from the input remote sensing image.

[0024] In step 8, the data of the cloud-covered portion of the image is supplemented by using Open Street Map data to obtain the road centerline. The road mask images at both ends of the cloud-covered area are combined with the road width information to complete the segmentation information supplement of the cloud-covered portion.

[0025] This paper combines the advantages of optical remote sensing imagery and Open Street Map data to propose an intelligent road information extraction method for optical remote sensing imagery under cloud conditions. This method facilitates road information extraction without cloud removal even when the sample contains clouds that obscure road information. The method includes preprocessing remote sensing imagery and Open Street Map (OSM) data to obtain data content for the corresponding city; aligning and cropping the two preprocessed data types to generate a road dataset and a corresponding segmentation mask dataset; extracting cloud shadow regions from the remote sensing imagery using the Fmask cloud detection algorithm, performing a series of spectral tests based on spectral characteristics to identify potential cloud layers; and using the fill transform method from morphological transformations combined with scattering correlation calculations to identify potential cloud shadow layers. Cloud-containing regions are then masked out of prediction within the deep learning network. Based on AI technology, a road information extraction model for unlabeled samples is constructed. By combining remote sensing imagery with OSM road network data, this model avoids manual data annotation while leveraging the accuracy of OSM data to predict occluded road information in road extraction tasks and improve the consistency of road extraction. Furthermore, the model improves upon the classic semantic segmentation model, leveraging the trained model to better perform intelligent road information extraction. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Figure 1 This is a flow chart of a method for intelligently extracting road information from optical remote sensing images under the influence of clouds according to an embodiment of the present invention;

[0027] Figure 2 2 is a diagram of the SADU-net neural network structure according to an embodiment of the present invention; DETAILED DESCRIPTION

[0028] This paper utilizes remote sensing imagery and Open Street Map road network data to propose an intelligent method for extracting road information from optical remote sensing images under cloud conditions. This method facilitates road information extraction without cloud removal even when the sample contains clouds that obscure road surface information. The method includes preprocessing remote sensing imagery and Open Street Map (OSM) data to obtain data content for the corresponding city; aligning and cropping the two preprocessed data types to create a road dataset and a corresponding segmentation mask dataset; extracting cloud shadow regions from the remote sensing image using the Fmask cloud detection algorithm, performing a series of spectral tests based on spectral characteristics to identify potential cloud layers; and using the fill transform method from morphological transformations combined with scattering correlation calculations to identify potential cloud shadow layers. Cloud-containing regions are then masked out of prediction within the deep learning network. Based on AI technology, a road information extraction model for unlabeled samples is constructed. By combining remote sensing imagery with OSM road network data, this model avoids manual data annotation while leveraging the accuracy of OSM data to predict occluded road information in road extraction tasks and improve the consistency of road extraction. Furthermore, the model improves upon the classic semantic segmentation model, leveraging the trained model to better complete road information extraction.

[0029] The technical solution of the present invention can be used in the form of computer software to support the automatic operation process. The technical solution of the present invention is described in detail below in conjunction with the embodiments and drawings.

[0030] See also Figure 1 The embodiment of the present invention provides an intelligent method for extracting road information from optical remote sensing images under the influence of clouds. The specific operation steps are as follows:

[0031] Step 1: Preprocess remote sensing images and Open Street Map data to mark a small amount of segmentation mask samples of road information;

[0032] Step 2: Perform cloud detection on the remote sensing image, mark the cloud-containing part of the image, and set it not to enter the network;

[0033] Step 3, using prior knowledge such as road types and pixel-level similarity to set buffers for the road centerlines obtained from the open street map data, a large number of road information samples;

[0034] Step 4: Build a training sample library. Use the cloud-free remote sensing image data and the large amount of mask data generated in step 3 as input to build a series of training sample data.

[0035] Step 5: Build the SADU-net (Self Attention and Densely Connected U-net) network model, which extracts features from a global-local perspective and uses spatial attention and dense blocks to extract road information from cloud-free remote sensing images.

[0036] Step 6: SADU-net network model training. The SADU-net network model is trained by using spatial attention to focus on the spatial coherence of the road and the transferability of dense blocks to contextual relationships.

[0037] Step 7, SADU-net network model performance evaluation;

[0038] Step 8: Combine the complete road information without clouds with the open street map data to predict the cloud-containing part, and then generate a complete road information segmentation map after splicing.

[0039] In the present invention, road information extraction under the cloud layer can be completed, which helps to achieve timely updating of the road network.

[0040] The specific embodiments described herein are merely illustrative of the spirit of the present invention. Persons skilled in the art may make various modifications, additions, or substitutions to the described specific embodiments without departing from the spirit of the present invention or exceeding the scope of the appended claims.

Claims

1. A method for intelligently extracting road information from optical remote sensing images under cloud cover, characterized by: The following steps are included: Step 1: Preprocess the remote sensing image and Open Street Map data and mark a small number of segmentation mask samples of road information; Step 2: perform cloud detection on the remote sensing image containing clouds and mark the cloud-containing parts of the image; Step 3, using prior knowledge of road types and pixel-level similarity to set buffers for the road centerlines obtained from the Open Street Map data, a large number of road information samples; Step 4: Build a training sample library. Use the cloud-free remote sensing image data and the large amount of mask data generated in step 3 as input to build a series of training sample data. Step 5: Build the SADU-net network model (Self Attention and Densely Connected U-net). This model extracts features from a global-local perspective and uses spatial attention and dense blocks to extract road information from cloud-free remote sensing images. Step 6: SADU-net network model training. The SADU-net network model is trained by using spatial attention to focus on the spatial coherence of the road and the transferability of dense blocks to contextual relationships. Step 7, SADU-net network model performance evaluation; Step 8: Combine the complete road information without clouds with the Open Street Map data to predict the cloud-containing part, and then generate the complete road information after splicing. In step 3, the centerline of the road network obtained from the Open Street Map data is combined with prior knowledge to generate reliable road segmentation information. The width of the road is determined based on the centerline and the road type information given in the road network. The road and background information are distinguished based on the similarity between pixels to form the boundary of the segmentation information, and the road and the corresponding segmentation mask image are obtained. In step 5, the SADU-net model is based on the U-net network, which has three parts. The first part of the network includes an encoder, which consists of 4 dense blocks, each of which is composed of a BN layer, a Relu activation function, and a 3×3 convolution kernel. Each node comprehensively considers the information of all previous nodes; a BN layer, a 1×1 convolution layer, and a 2×2 average pooling layer are placed between different dense blocks; the second part is a dense connection part, which uses a self-attention mechanism on the original image, performs a 1×1 convolution on each channel, and performs a transposed point multiplication operation through different mapping functions. Finally, it is added to the result of the third part to extract information from the image, considering the global information of the road; the third part is the upsampling process, which performs a full connection operation on the encoded output of the corresponding layer, decodes the image, restores it to a size of 512×512, and outputs the result.

2. The method for intelligently extracting road information from optical remote sensing images under cloud cover according to claim 1, characterized in that: Step 1: aligning the remote sensing image with the road network data of Open Street Map; The purpose is to obtain road network data corresponding to remote sensing images and provide a basis for subsequent cropping.

3. The method for intelligently extracting road information from optical remote sensing images under cloud cover according to claim 1, characterized in that: In step 2, the Fmask method is used to detect clouds, and potential cloud layers and potential cloud shadow layers are extracted using rules based on the physical properties of clouds and cloud shadows; then the potential cloud layers and geometric relationships are used to match the potential cloud shadows.

4. The method for intelligently extracting road information from optical remote sensing images under cloud cover according to claim 1, characterized in that: In step 4, the cloud results detected in step 2 are used to divide the image into cloud-containing areas and cloud-free areas, and the remote sensing image data obtained from the cloud-free areas and the mask data generated by the road network data are cut equally, and the size is set to 512×512 pixels. The image is cropped with a repetition rate of 0.5 and a step size of 256. The cropped image is still 512×512 pixels, and 0 is filled in at the edges or in irregularities to meet the size of 512; then the samples are set according to a 7:3 ratio and divided into a training set and a prediction set.

5. The method for intelligently extracting road information from optical remote sensing images under cloud cover according to claim 1, characterized in that: In step 6, the learning rate, batch size, and number of training times are first set, and the momentum stochastic gradient descent method is used to train SADU-net; the loss function uses the mean square error loss function, and its formula is defined as: ； N is the number of images, is the input image, yes The corresponding label, Yes The prediction result graph.

6. The method for intelligently extracting road information from optical remote sensing images under cloud cover according to claim 1, characterized in that: In step 7, sample data is input to test the entire SADU-net network model. When the loss function value tends to be stable and minimum, the network performance reaches the optimal level. The optimal SADU-net network is used to extract road information from the input remote sensing image.

7. The method for intelligently extracting road information from optical remote sensing images under cloud cover according to claim 1, characterized in that: In step 8, the data of the cloud-covered portion of the image is supplemented by using the Open Street Map data to obtain the road centerline. Combined with the road mask image of the cloud-covered area, the width information of the road is used for mapping, and the segmentation information of the cloud-covered portion is supplemented.

Images