A method for extracting a road from a remote sensing image
By constructing a multi-scale gated full fusion network, the problem of incomplete road extraction in remote sensing images was solved, achieving more efficient multi-scale feature fusion and more complete road extraction, especially for roads in shadowed areas.
Patent Information
- Application Number
- CN202310184762.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2023-03-01
- Publication Date
- 2025-12-05
- Estimated Expiration
- 2043-03-01
AI Technical Summary
Traditional methods struggle to extract complete roads from remote sensing images, especially those obscured by clouds, trees, and building shadows, and multi-scale feature fusion is inefficient.
A multi-scale gated full fusion network is constructed, including a multi-scale feature extraction layer and a gated full fusion unit. Multi-scale features are screened and fused through multiple convolutional layers with different dilation rates and non-local pooling layers, and useful features are filtered by the gated full fusion unit.
It improves the completeness and accuracy of road extraction in remote sensing images, and can extract features of roads under clouds, trees, and shadows, thus improving the efficiency of multi-scale information fusion.
Smart Images
Figure CN116188986B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of remote sensing image data processing technology, specifically to a method and system for road extraction from remote sensing images. Background Technology
[0002] With the rapid development of remote sensing technology, extracting roads from high-resolution remote sensing images has become a hot topic. Traditional methods use global average pooling to enhance the network's global information extraction capabilities when extracting multi-scale features. While global average pooling provides translation and rotation invariance, it also destroys spatial information in the image. For pixel-by-pixel classification tasks like road extraction, the global information provided by global average pooling is insufficient. Moreover, for the fusion of multi-scale features, these methods typically use stitching or addition, without considering the semantic differences between features at different scales. This leads to useful information being buried in a large amount of useless information, thus affecting the efficiency of multi-scale information fusion. The direct consequence of these methods is the difficulty in extracting complete roads from high-resolution remote sensing images, and poor performance in extracting roads obscured by clouds, trees, and building shadows. Summary of the Invention
[0003] This invention proposes a road extraction method and system for remote sensing images, which solves the problems of incomplete road extraction and difficulty in extracting roads obscured by clouds, trees, and shadows in the traditional road extraction process, and improves the accuracy of road extraction in high-resolution scenes.
[0004] This invention provides a method for road extraction from remote sensing images, comprising the following steps:
[0005] Acquire the remote sensing image to be extracted, and extract the deep semantic feature map from the remote sensing image using a feature extractor;
[0006] A multi-scale gated full fusion network is constructed, wherein the multi-scale gated full fusion network includes a multi-scale feature extraction layer and a gated full fusion unit; the multi-scale feature extraction layer includes multiple convolutional layers with different dilation rates and a non-local pooling layer;
[0007] The deep semantic feature map is input into the trained multi-scale gated full fusion network. After passing through multiple convolutional layers with different dilation rates and a non-local pooling layer, the deep semantic feature map is used to obtain the feature maps of the remote sensing image at different scales. The gated full fusion unit filters the features in the feature maps at different scales to obtain the multi-scale features of the road.
[0008] The multi-scale features of the road are fed into the feature decoder to obtain an image with road markings.
[0009] Furthermore, before extracting the deep semantic feature map from the remote sensing image, the method further includes preprocessing the remote sensing image to be extracted, including the following steps:
[0010] Determine the attribute and size information of the remote sensing data to be extracted;
[0011] Based on the attribute and size information of the remote sensing data, the remote sensing image is converted into a 3-channel image and then cropped. The methods for converting the remote sensing image into a 3-channel image include image augmentation based on PCA principal component analysis and image denoising based on autoencoder.
[0012] Furthermore, the multi-scale gated fully fused network is trained, including the following steps:
[0013] A portion of the preprocessed remote sensing image to be extracted is used as the training set;
[0014] The remote sensing image data in the training set are normalized.
[0015] Deep semantic feature maps of remote sensing image data in the training set after normalization are extracted and input into a multi-scale gated full fusion network to obtain multi-scale road features of remote sensing images in the training set.
[0016] The multi-scale road features of remote sensing images in the training set are decoded using a decoder to obtain the decoded features. Based on the decoded features, the parameters of the multi-scale gated full fusion network are updated through backpropagation, thereby completing the training of the multi-scale gated full fusion network.
[0017] Furthermore, after the deep semantic feature map is processed through multiple convolutional layers with different dilation rates and a nonlocal pooling layer, feature maps of the remote sensing image at different scales are obtained, including:
[0018] Given a deep semantic feature map F D The feature map F is then input into a multi-scale gated fusion network, and dimensionality reduction is performed using multiple convolutional layers with different dilation rates to obtain the feature map F. E F G F H and F W ;
[0019] F G F H F W Remodeled into matrix form, F G and F H The value at each location in the spatial attention map S is calculated using the following formula:
[0020]
[0021] Where N = H × W, H represents the height of the deep semantic feature map, and W represents the width of the deep semantic feature map;
[0022] S ij This represents the value of attention map S in the i-th row and j-th column;
[0023] F Gi F in matrix form G The i-th row;
[0024] F Hj F in matrix form H The j-th column;
[0025] The output of nonlocal pooling can be expressed as F. O The calculation process is as follows:
[0026]
[0027] Where W represents a 1×1 convolution; This indicates a splicing operation.
[0028] Furthermore, the gated full fusion unit filters features in feature maps at different scales, and its implementation process is as follows:
[0029]
[0030] Where · represents multiplication; This represents the output feature map of the gated full fusion unit;
[0031] F i Indicates non-F l Other input feature maps; the value of l ranges from 1 to 5, F l This represents the input feature map of the gated full fusion unit;
[0032] G(F l ) is the calculation function for the gating feature, which is expressed by the following formula:
[0033] G(F l )=σ1(bn(W1(σ0(bn(W0(F l (4)
[0034] Where W1 and W0 represent 1×1 convolution, bn represents batch normalization, σ0 represents the linear rectified function, and σ1 represents the sigmoid function.
[0035] The present invention also provides a road extraction system for remote sensing images, comprising:
[0036] The data acquisition module acquires the remote sensing images to be extracted and extracts deep semantic feature maps from the remote sensing images using a feature extractor.
[0037] The network model construction module is used to construct a multi-scale gated full fusion network, wherein the multi-scale gated full fusion network includes a multi-scale feature extraction layer and a gated full fusion unit; the multi-scale feature extraction layer includes multiple convolutional layers with different dilation rates and a non-local pooling layer;
[0038] The road multi-scale feature extraction module is used to input the deep semantic feature map into the trained multi-scale gated full fusion network. After the deep semantic feature map passes through multiple convolutional layers with different dilation rates and a non-local pooling layer, the feature maps of the remote sensing image at different scales are obtained. The gated full fusion unit filters the features in the feature maps at different scales to obtain the multi-scale features of the road.
[0039] The image output module is used to input the multi-scale features of the road into the feature decoder for decoding, and obtain an image with road markings.
[0040] Compared with the prior art, the beneficial effects of the present invention are as follows:
[0041] This invention utilizes a multi-scale gated full fusion network for road extraction from remote sensing images. The network comprises a multi-scale feature extraction layer and a gated full fusion unit. The multi-scale feature extraction layer includes multiple convolutional layers with varying dilation rates and a non-local pooling layer to obtain features at different scales. These features aid the multi-scale gated full fusion network in extracting roads of different scales from remote sensing images. The use of non-local pooling allows the network to extract rich global contextual information, thus contributing to a more complete extraction of multi-scale road features. The gated full fusion unit further filters useful features from the multi-scale data, improving road extraction efficiency. Attached Figure Description
[0042] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used in conjunction with embodiments of the invention to explain the invention and do not constitute a limitation thereof. In the drawings:
[0043] Figure 1 This is a schematic flowchart of a remote sensing image road extraction method provided in an embodiment of the present invention;
[0044] Figure 2 This is a schematic flowchart of a remote sensing image road extraction method provided in an embodiment of the present invention;
[0045] Figure 3 This is a schematic diagram of a multi-scale gated fully fused network provided in an embodiment of the present invention;
[0046] Figure 4 This is a schematic diagram of a nonlocal pooling layer provided in an embodiment of the present invention;
[0047] Figure 5 This is a schematic diagram of a gated full-fusion unit provided in an embodiment of the present invention. Detailed Implementation
[0048] The technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. However, it should be understood that the scope of protection of the present invention is not limited to the specific implementation.
[0049] Example 1
[0050] Currently, remote sensing technology is developing rapidly, and automatically extracting roads from remote sensing images can be used for disaster emergency management, smart cities, autonomous driving, high-precision map acquisition, and other tasks. However, due to the complex geometric structure and significant geometric changes of roads, as well as their frequent occlusion by objects such as clouds, trees, and building shadows, it may be difficult to extract complete roads from remote sensing images.
[0051] To address the aforementioned problems, this invention discloses a road extraction method for remote sensing images, which can easily extract complete roads from remote sensing images, improving user experience. Furthermore, since this method can run end-to-end, it can significantly reduce labor costs. The embodiments of this invention are described in detail below.
[0052] It should be understood that this disclosure may be provided as a program, and the server providing services to the corresponding application may be a terminal server and / or a cloud server. The methods of this disclosure are applicable to various terminals. It should be understood that the terminals in this disclosure may include, but are not limited to, fixed terminal devices such as smartphones, laptops, tablets, navigation devices, desktop computers, etc.
[0053] like Figure 1 As shown, the present invention provides a road extraction method for remote sensing images, comprising the following steps:
[0054] Step S1: Obtain the remote sensing image to be extracted, and extract the deep semantic feature map from the remote sensing image using a feature extractor;
[0055] Step S2: Construct a multi-scale gated full fusion network, wherein the multi-scale gated full fusion network includes a multi-scale feature extraction layer and a gated full fusion unit; the multi-scale feature extraction layer includes multiple convolutional layers with different dilation rates and a non-local pooling layer;
[0056] Step S3: Input the deep semantic feature map into the trained multi-scale gated full fusion network. After the deep semantic feature map passes through multiple convolutional layers with different dilation rates and a non-local pooling layer, the feature maps of the remote sensing image at different scales are obtained. The gated full fusion unit filters the features in the feature maps at different scales to obtain the multi-scale features of the road.
[0057] Step S4: Input the multi-scale features of the road into the feature decoder to obtain an image with road markings.
[0058] In step S1, before extracting the deep semantic feature map from the remote sensing image, preprocessing of the remote sensing image to be extracted is also included, comprising the following steps:
[0059] Determine the attribute and size information of the remote sensing data to be extracted; when reading remote sensing images, first obtain the attribute information of the remote sensing image, determine the type of the remote sensing image, whether it is a remote sensing image or a normal format image, whether it is an RGB channel, a grayscale image or a remote sensing multi-band channel, and then obtain the size information of the image.
[0060] Based on the attribute and size information of the remote sensing data, the remote sensing image is converted into a 3-channel image and then cropped. The methods for converting the remote sensing image into a 3-channel image include image augmentation based on PCA principal component analysis and image denoising based on autoencoder.
[0061] The cropped size includes, but is not limited to, 128 pixels × 128 pixels, 128 pixels × 256 pixels, 256 pixels × 256 pixels, etc., as long as the size is a multiple of 64 pixels.
[0062] In step S1, a feature extractor is used to extract deep semantic feature maps from the preprocessed remote sensing image. In this embodiment of the invention, the feature extractor may be a backbone network model including but not limited to ResNet18, VGG16, VGG19, MobileNet, and Transformer.
[0063] In step S2, the multi-scale gated fully fused network is trained, including the following steps:
[0064] Step S2.1: Use a portion of the preprocessed remote sensing image data to be extracted as the training set;
[0065] Step S2.2: Normalize the remote sensing image data in the training set;
[0066] Normalized remote sensing image data will be used as training data for a multi-scale gated fusion network, thus making the network training process more stable. One possible normalization method is Z-score standardization, which can transform data of different magnitudes into a uniform Z-score.
[0067] Step S2.3: Extract the deep semantic feature map of the remote sensing image data in the training set after normalization, and input it into the multi-scale gated full fusion network to obtain the road multi-scale features of the remote sensing images in the training set.
[0068] Step S2.4: Decode the multi-scale road features of the remote sensing images in the training set using the decoder to obtain the decoded features; based on the decoded features, update the parameters of the multi-scale gated full fusion network through backpropagation, thereby completing the training of the multi-scale gated full fusion network. The decoder can be one or more convolutions of different sizes.
[0069] In this invention, a multi-scale gated fusion network is trained using a training set, enabling the multi-scale gated fusion network to operate efficiently.
[0070] In step S3, the multi-scale gated full fusion network includes two stages:
[0071] (1) Multi-scale feature extraction stage: Multi-scale features are extracted from the input features using a multi-scale feature extraction layer.
[0072] The input feature map is processed through multiple convolutional layers with different dilation rates and a nonlocal pooling layer to obtain features at different scales. These features help the multi-scale gated fusion network extract roads of different scales from remote sensing images. Due to the use of nonlocal pooling, the multi-scale gated fusion network will extract rich global contextual information at this stage, thus contributing to a more complete extraction of multi-scale road features.
[0073] One option for the hole ratio of each convolutional layer is (1, 12, 24, 36), or any other combination of numbers less than 36.
[0074] (2) Gated full fusion stage: Use the gated full fusion unit to filter useful features from multi-scale features to improve road extraction efficiency.
[0075] Feature maps with different scale features will be sent to the gated full fusion module, where features will be filtered through gating to obtain features more suitable for road extraction.
[0076] In this invention, through the multi-scale extraction stage and the gated full fusion stage, multi-scale and discriminative features will be obtained. These features enable the network to extract complete roads and, to a certain extent, roads under the shadows of clouds, trees, and buildings.
[0077] like Figure 4 A schematic diagram of a nonlocal pooling operation is given. In traditional CNN convolution operations, the output neurons are only related to a portion of the input mapping. Therefore, when faced with these long-term dependencies, they often fail to capture enough information to represent the data. Consequently, global average pooling is frequently used to capture global information. While global average pooling provides translation and rotation invariance, it also destroys image spatial information, which is not ideal for problems like road extraction that classify each pixel of an image. Nonlocal pooling, compared to global average pooling, can extract richer global semantic information and is more suitable for road extraction.
[0078] The deep semantic feature map in this invention, after passing through multiple convolutional layers with different dilation rates and a nonlocal pooling layer, yields feature maps of remote sensing images at different scales, including:
[0079] Given a deep semantic feature map F D The feature map F is then input into a multi-scale gated fusion network, and dimensionality reduction is performed using multiple convolutional layers with different dilation rates to obtain the feature map F. E F G F H and F W ;
[0080] F G F H F W Remodeled into matrix form, F G and F H The value at each location in the spatial attention map S is calculated using the following formula:
[0081]
[0082] Where N = H × W, H represents the height of the deep semantic feature map, and W represents the width of the deep semantic feature map;
[0083] S ij This represents the value of attention map S in the i-th row and j-th column;
[0084] F Gi F in matrix form G The i-th row;
[0085] F Hj F in matrix form HThe j-th column;
[0086] The output of nonlocal pooling can be used with F O The calculation process is as follows:
[0087]
[0088] Where W represents a 1×1 convolution; This indicates a splicing operation.
[0089] Figure 5 A schematic diagram of a gated full fusion unit is presented. Improving road extraction accuracy requires expanding the receptive field and acquiring multi-scale information. Most road extraction methods fuse different multi-scale feature maps by directly concatenating or adding them. This simple combination of multi-scale features is ineffective because semantic gaps exist between them. In the gated full fusion unit, each branch feature is enhanced by other branch features, and information propagation is controlled through a gating mechanism. By gating and filtering features, more effective information can be extracted.
[0090] The gated full fusion unit filters features in feature maps at different scales, and its implementation process is as follows:
[0091]
[0092] Where · represents multiplication; This represents the output feature map of the gated full fusion unit; F i Indicates non-F l Other input feature maps;
[0093] The value of l ranges from 1 to 5, and F l This represents the input feature map of the gated full fusion unit;
[0094] G(F l ) is the calculation function for the gating feature, which is expressed by the following formula:
[0095] G(F l )=σ1(bn(W1(σ0(bn(W0(F l (4)
[0096] Where W1 and W0 represent 1×1 convolution, bn represents batch normalization, σ0 represents the linear rectified function, and σ1 represents the sigmoid function.
[0097] Example 2
[0098] The present invention also provides a road extraction system for remote sensing images, comprising:
[0099] The data acquisition module acquires the remote sensing images to be extracted and extracts deep semantic feature maps from the remote sensing images using a feature extractor.
[0100] The network model building module is used to construct a multi-scale gated full fusion network, which includes a multi-scale feature extraction layer and a gated full fusion unit; the multi-scale feature extraction layer includes multiple convolutional layers with different dilation rates and a non-local pooling layer.
[0101] The road multi-scale feature extraction module is used to input the deep semantic feature map into the trained multi-scale gated full fusion network. After the deep semantic feature map passes through multiple convolutional layers with different dilation rates and a non-local pooling layer, the feature maps of the remote sensing image at different scales are obtained. The gated full fusion unit filters the features in the feature maps at different scales to obtain the multi-scale features of the road.
[0102] The image output module is used to input the multi-scale features of the road into the feature decoder for decoding, and obtain an image with road markings.
[0103] Finally, it should be noted that the above-disclosed embodiment is only one specific embodiment of the present invention. However, the embodiments of the present invention are not limited thereto, and any variations that can be conceived by those skilled in the art should fall within the protection scope of the present invention.
Claims
1. A method of road extraction from a remote sensing image, characterized in that, The method comprises the following steps: obtaining a remote sensing image to be extracted, and extracting a deep semantic feature map in the remote sensing image through a feature extractor; constructing a multi-scale gated all-in-one network, wherein the multi-scale gated all-in-one network comprises a multi-scale feature extraction layer and a gated all-in-one unit; the multi-scale feature extraction layer comprises a plurality of convolution layers with different hole rates and a non-local pooling layer; inputting the deep semantic feature map into the trained multi-scale gated all-in-one network, obtaining feature maps of the remote sensing image at different scales after the deep semantic feature map passes through the plurality of convolution layers with different hole rates and the non-local pooling layer, and filtering features in the feature maps at different scales through the gated all-in-one unit to obtain multi-scale features of a road; decoding the multi-scale features of the road in a feature decoder to obtain an image with road markings.
2. The method of claim 1, wherein: Before extracting the deep semantic feature map in the remote sensing image, the method further comprises pre-processing the remote sensing image to be extracted, comprising the following steps: determining attribute information and size information of the remote sensing data to be extracted; converting the remote sensing image into a 3-channel image according to the attribute information and size information of the remote sensing data, and cropping the image; wherein the method for converting the remote sensing image into a 3-channel image comprises an image expansion method based on a PCA principal component analysis method and an image denoising method based on an autoencoder.
3. The method of claim 2, wherein: training the multi-scale gated all-in-one network, comprising the following steps: taking part of the pre-processed remote sensing image to be extracted as a training set; performing normalization processing on the remote sensing image data in the training set; extracting a deep semantic feature map of the remote sensing image data in the normalized training set, and inputting the deep semantic feature map into the multi-scale gated all-in-one network to obtain road multi-scale features of the remote sensing image in the training set; decoding the road multi-scale features of the remote sensing image in the training set by using a decoder to obtain decoded features; and updating parameters of the multi-scale gated all-in-one network through back propagation according to the decoded features, so as to complete the training of the multi-scale gated all-in-one network.
4. The method of claim 1, wherein: After the deep semantic feature map passes through the plurality of convolution layers with different hole rates and the non-local pooling layer, the remote sensing image obtains feature maps at different scales, comprising: Given a deep semantic feature map F D , it is input into a multi-scale gated all-fusion network, and is respectively reduced in dimension using multiple convolutional layers with different hole rates to obtain feature maps F E , F G , F H , and F W ; F G , F H , F W are reshaped into matrix form, F G and F H are used to generate spatial attention maps S, the value of each position in the spatial attention maps S is calculated using the following formula: wherein N=HxW, H represents the height of the deep semantic feature map, and W represents the width of the deep semantic feature map; S ij denotes the value of the attention map S in the i-th row and j-th column; F Gi represents the i-th row of the matrix form of F G F F Hj represents the jth column of the matrix form of F H ; The output of the non-local pooling is represented by F O with the following calculation process: where W represents a 1x1 convolution; denotes a concatenation operation.
5. The method of claim 1, wherein: the gated all-in-one unit filters features in the feature maps at different scales, and the implementation process is as follows: wherein • denotes multiplication; denotes the output feature map of the gating all-fusion unit. F i represents other input feature maps other than F l ; l is in the range of 1-5, F l represents input feature maps of the gating all-fusion unit G(F l ) is a gating function of the feature, which is represented by the following equation: G(F l ) = σ1(bn(W1(σ0(bn(W0(F l )))))) (4) wherein W1 and W0 represent 1x1 convolution, bn represents batch normalization, σ0 represents a linear rectifier function, and σ1 represents a sigmoid function.
6. A road extraction system of a remote sensing image, characterized by: comprising: a data acquisition module, which obtains a remote sensing image to be extracted, and extracts a deep semantic feature map in the remote sensing image through a feature extractor; a network model construction module, which is configured to construct a multi-scale gated all-in-one network, wherein the multi-scale gated all-in-one network comprises a multi-scale feature extraction layer and a gated all-in-one unit; the multi-scale feature extraction layer comprises a plurality of convolution layers with different hole rates and a non-local pooling layer; The road multi-scale feature extraction module is configured to input a deep semantic feature map into a trained multi-scale gating full fusion network, and obtain feature maps of the remote sensing image at different scales after the deep semantic feature map passes through a plurality of convolution layers with different hole rates and a non-local pooling layer; the gating full fusion unit filters features in the feature maps at different scales to obtain multi-scale features of the road. The image output module is configured to input the multi-scale features of the road into a feature decoder to obtain an image with road markings.