Remote sensing road extraction method based on deep learning

Abstract

The invention provides a remote sensing image road extraction method based on deep learning, a remote sensing image is processed through a trained generator to obtain a road segmentation map, and the generator comprises an encoder module, a decoder module, a multi-scale feature fusion module and an output module; wherein an enhanced decoder module is constructed in a decoder structure, multi-level semantics and information space are fused through an up-sampling path and a transverse connection path, and a channel attention and space attention mechanism is introduced into the decoder to enhance the attention capability to a road area; and unified alignment, channel compression and fusion are carried out on the output of each stage of the decoder through the multi-scale feature fusion module, so that the modeling capability of road structures with different widths and changeable forms is improved. According to the method, high precision and robustness are still kept under the conditions of complex background and weak supervision, the problems of road breakage, adhesion and wrong segmentation can be effectively relieved, and the method has good engineering application value and popularization potential.

Description

Technical Field

[0001] This invention belongs to the field of remote sensing data land cover classification, and specifically relates to a method for road extraction from remote sensing images based on deep learning. Background Technology

[0002] In recent years, the extraction of road information from remote sensing images has received widespread attention and has been widely applied in various fields such as map updating, urban planning, and intelligent transportation. However, due to the characteristics of roads in remote sensing images, such as being long and thin, curved, varying in scale, and easily affected by occlusion and background interference, extracting roads from remote sensing images is a challenging task. Existing methods mostly employ encoder-decoder architectures and convolutional neural networks (CNNs) as their backbone networks. While these methods have improved segmentation accuracy to some extent, they still suffer from the following problems: Firstly, CNNs are limited by the fixed shape of their convolutional kernels, making it difficult to fully model the long-distance continuous structure of roads; secondly, shallow features are sensitive to edge localization, while deep semantic features focus more on the overall structure. Existing methods struggle to achieve a good balance between these two aspects, leading to road fragmentation, adhesion, or missegmentation. Summary of the Invention

[0003] To address the problems existing in the prior art, this invention proposes a remote sensing road extraction method based on deep learning. This method integrates the advantages of convolutional neural networks (CNN), multi-scale modeling mechanisms, and attention mechanisms. It adopts a multi-scale feature attention fusion network (MAF-Net) and is designed for the extraction of road information with narrow structures in remote sensing images, featuring high accuracy and high efficiency.

[0004] The technical solution of this invention is as follows:

[0005] A remote sensing road extraction method based on deep learning is proposed, wherein the remote sensing image to be processed is input into a trained generator, and the final output is a road segmentation map with the same resolution as the input image; the generator includes an encoder module, a decoder module, a multi-scale feature fusion module, and an output module;

[0006] The encoder module includes a multi-layer encoder, wherein the first layer encoder Encoder_0 performs shallow feature extraction and downsampling on the input image, and multiple deep encoders are connected in series after the first layer encoder Encoder_0 to extract deep semantic features step by step.

[0007] The decoder module fuses encoder features with features upsampled layer by layer by the decoder through a lateral connection mechanism. Except for the last encoder layer, the output of each encoder layer is divided into two identical parts: one part is fed into the next encoder layer, and the other part is input into the corresponding decoder as lateral connection features. The decoder integrates a dual attention module consisting of a channel-space attention module and a spatial context attention module. The decoder module gradually restores the spatial resolution of the image through lateral connection and attention mechanisms.

[0008] The multi-scale feature fusion module receives the output of each layer decoder, first performs channel compression and upsampling on the mid-to-high-level feature maps to align them with the low-level features in the spatial dimension, and then concatenates the scale-aligned feature maps with the same number of channels in the channel dimension. The concatenated feature maps are then fused using a 1×1 convolution operation. At the same time, normalization and non-linear activation functions are introduced after fusion to further enhance the features, so as to achieve the integration of multi-scale semantics and edge information.

[0009] Finally, the output module upsamples the feature map output by the multi-scale feature fusion module to restore the size of the output image, ensuring that the prediction result is consistent with the size of the input image.

[0010] Furthermore, in the encoder module, the first layer encoder Encoder_0 performs shallow feature extraction and downsampling on the input image, which is achieved through Conv+BN+ReLU+MaxPooling. After the first layer encoder Encoder_0, multiple deep encoders are connected in series, and the deep encoders are implemented using the residual module ResNet34.

[0011] Furthermore, except for the last encoder layer, the output of each of the other encoder layers is divided into two identical parts: one part is fed into the next encoder layer, and the other part is input into the corresponding decoder as a lateral connection feature.

[0012] Furthermore, the decoder module consists of several decoders, with one fewer decoder than encoder. Each decoder layer has two input branches: one branch receives low-resolution features from the decoder layer above, and the other branch receives features output by the encoder layer above.

[0013] Furthermore, if it is the last layer decoder, one of its branches inputs is the output of the center part, and the other branch input is the feature output of the corresponding layer encoder; the center part is a dilated convolution block.

[0014] Furthermore, in the decoder, low-resolution features are first upsampled through transposed convolution, followed by batch normalization and nonlinear activation, and finally interpolated upsampling to obtain high-resolution features. The features output by the corresponding layer encoder are input through skip connections, and a 1×1 convolution operation is used in the lateral connection path, along with a BatcNorm layer to adjust its channel number, to obtain lateral connection features, achieving alignment between channel number and resolution. The high-resolution features and the lateral connection features are then added element-wise to obtain the fused features.

[0015] Furthermore, in the decoder, the fused features are input into a dual attention module, wherein in the channel-space attention module, channel description vectors are generated through global average pooling and max pooling, and then input into a multilayer perceptron and activated by Sigmoid to obtain channel weights; the output of the channel-space attention module is input into the spatial context attention module, and the long-range dependencies between pixels are modeled in the horizontal and vertical directions respectively through the Criss-Cross mechanism.

[0016] Furthermore, in the output module, transposed convolution and convolution operations are combined with the ReLU activation function to enhance edge and detail features, ultimately generating a segmentation result map of the road region and background region with the same resolution as the input image, achieving pixel-level differentiation between the road region and the background region.

[0017] Furthermore, the generator employs a generative adversarial network and is trained using a training set that includes both labeled and unlabeled parts.

[0018] Beneficial effects

[0019] This invention proposes a deep learning-based road extraction method for remote sensing images called MAF-Net. Based on an encoder-decoder structure, this method integrates multi-scale feature extraction and attention enhancement mechanisms to effectively improve the model's ability to perceive complex road structures in remote sensing images. Specifically, the designed multi-scale feature fusion module (MSFF) fully integrates feature information from different levels of decoders, achieving unified modeling of roads at different scales through upsampling and channel compression operations, thereby enhancing the model's adaptability to scale changes. Furthermore, the enhanced decoder module introduced in the decoding stage, through lateral connections and upsampling fusion, combined with the channel-space attention mechanism (CBAM) and the spatial context attention module (CC_module), achieves multi-dimensional feature enhancement, effectively mitigating problems such as blurred road edges and structural breaks.

[0020] Additional aspects and advantages of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. Attached Figure Description

[0021] The above and / or additional aspects and advantages of the present invention will become apparent and readily understood from the description of the embodiments taken in conjunction with the following drawings, in which:

[0022] Figure 1 The deep learning network training graph for remote sensing road extraction proposed in this invention;

[0023] Figure 2 Generator architecture diagram;

[0024] Figure 3 Decoder module architecture diagram;

[0025] Figure 4 Architecture diagram of multi-scale feature fusion module;

[0026] Figure 5 Output module architecture diagram;

[0027] Figure 6 : Some visualization results of this invention on the DeepGlobe dataset;

[0028] Figure 7 Comparison of extraction results of different algorithms on the DeepGlobe dataset. Detailed Implementation

[0029] The embodiments of the present invention are described in detail below. These embodiments are exemplary and intended to explain the present invention, and should not be construed as limiting the present invention.

[0030] The deep learning-based remote sensing road extraction method proposed in this embodiment is mainly based on, for example, Figure 2 The multi-scale feature attention fusion network MAF-Net shown is used as the generator. The remote sensing image to be processed is input into the trained generator, and the final output is a road segmentation map with the same resolution as the input image, achieving pixel-level differentiation between road areas and background areas.

[0031] like Figure 2 As shown, the generator includes an encoder module, a central part, a decoder module, a multi-scale feature fusion module, and an output module.

[0032] In the encoder module, spatial detail features and semantic abstraction features are extracted layer by layer from the input image through convolution and downsampling operations. Shallow features preserve spatial details such as edges and textures, while deep features contain global semantic information about road categories, providing a multi-layered feature foundation for subsequent decoding. Specifically, as follows... Figure 2As shown, the encoder module includes a multi-layer encoder. The first-layer encoder, Encoder_0, performs shallow feature extraction and downsampling on the input image. In this embodiment, this is mainly achieved through Conv+BN+ReLU+MaxPooling, that is, the first-layer encoder, Encoder_0, sequentially performs convolution, batch normalization, ReLU activation, and max pooling operations to achieve shallow feature extraction and downsampling of the input image. Multiple deep encoders are connected in series after the first-layer encoder, Encoder_0. In this embodiment, ResNet34 residual modules are used as deep encoders Encoder_1~Encoder_4, and deep semantic features are extracted step by step using multiple residual modules. The shallow features proposed by Encoder_0 retain rich spatial details, while the deep features provide more abstract semantic information. Through this hierarchical encoding, the model can construct multi-layer feature representations.

[0033] Furthermore, to implement the lateral connection mechanism during the decoding stage, the encoder features are fused with the features upsampled layer by layer by the decoder. Except for the last encoder layer, Encoder_4, the outputs of the remaining encoder layers are divided into two identical parts: one part is fed into the next encoder layer, and the other part is input into the corresponding decoder as the lateral connection feature. That is, let the first layer be... The layer encoder outputs the feature map as follows ,but:

[0034]

[0035] By using a lateral connection mechanism to fuse semantic information and spatial details at different scales, MAF-Net enhances its ability to extract fine-grained structures, enabling it to focus on spatial details while taking into account semantic information.

[0036] The decoder module consists of several decoders, one fewer than the number of encoders. In the decoding stage, restoring spatial resolution and fusing semantic information from the encoders are crucial for road structure reconstruction. This invention designs an enhanced decoder module that strengthens feature representation by fusing upsampled features and lateral connectivity features, and through a dual attention mechanism. Specifically... Figure 3 As shown, each decoder layer has two input branches, one of which receives low-resolution features from the decoder layer above. The other branch input is the feature output from the corresponding layer encoder. If it is the last layer decoder, one branch input is the output from the central part, and the other branch input is the feature output from the corresponding layer encoder; low-resolution features. First, upsampling (UpConv) is performed using transposed convolution to double the spatial resolution. Following upsampling, batch normalization (BN) and non-linear activation (ReLU) are applied to enhance the stability and expressive power of the features. Finally, interpolation upsampling (F.Interpolate) is performed to further improve the feature map resolution, resulting in high-resolution features. The corresponding layer encoder output features By using skip connections to the input, and considering the dimensionality inconsistency between the features and the corresponding stage in the encoder, a 1×1 convolution operation is used in the lateral connection path, along with a BatcNorm layer to adjust the number of channels, achieving alignment between the number of channels and resolution. This ensures that the feature map is consistent in dimension with the upsampled feature map of the current decoder.

[0037]

[0038] in, This represents the convolution operation. The parameters are 1×1 convolution kernel parameters. To achieve multi-source information fusion, high-resolution features are incorporated into the decoder. Features of lateral connection By adding elements one by one, the fused features are obtained. :

[0039]

[0040] The shallow spatial detail features of the encoder are fused with the deep semantic features upsampled by the decoder through a lateral connection mechanism. The fused features maintain global semantic consistency and supplement spatial detail information, thereby enhancing the continuity and accuracy of road boundaries.

[0041] To suppress the interference of background noise on road prediction, the fused features are input into a dual attention module, which is integrated into the decoder. The dual attention module contains two substructures:

[0042] CBMA (Channel-Spatial Attention) module: Guides the model to automatically focus on the most critical information regions in the channel dimension and spatial dimension. It obtains channel weights through global average pooling and max pooling, learns the importance of different channels, highlights road-related channel features, suppresses irrelevant background features, and improves the response to edges, textures, and elongated structures.

[0043] The CC (Spatial Context Attention) module constructs sparse global attention paths in the horizontal and vertical directions, effectively models long-distance dependencies, captures the continuity of long-distance roads, enhances the contextual understanding of slender continuous roads, and improves the ability to identify broken and occluded areas.

[0044] The features fused after the encoder and decoder feature maps are horizontally concatenated are first fed into the CBMA module, where channel description vectors are generated through global average pooling and max pooling. These vectors are then input into a multilayer perceptron and activated by Sigmoid to obtain channel weights, which are used to highlight road-related channel features and suppress redundant information. Subsequently, the output of the CBMA module is input into the CC module, where the long-range dependencies between pixels are modeled in the horizontal and vertical directions through the Criss-Cross mechanism, expanding the receptive field and enhancing global contextual information, thereby improving the problems of road breakage and adhesion.

[0045] The introduction of the dual attention module enables the network to have stronger feature selection capabilities in both spatial and channel dimensions, improving the completeness and accuracy of road extraction results.

[0046] Overall, the decoder uses an upsampling → alignment → fusion → attention enhancement process to gradually restore the spatial resolution of the image through lateral connections and attention mechanisms. This effectively solves problems such as boundary breaks and structural blurring in road target extraction, and provides a structurally coherent and semantically sufficient feature foundation for subsequent multi-scale fusion and final output.

[0047] like Figure 2 As shown, the decoder module in this embodiment has four decoder layers, Decoder_1 to Decoder_4. The features output by the last encoder layer, Encoder_4, are fed into the center and then fused with the features output by Encoder_3. Similarly, the features output by Decoder_4 are fused with the features output by Encoder_2; the features output by Decoder_3 are fused with the features output by Encoder_1; and the features output by Decoder_2 are fused with the features output by Encoder_0. In this way, the decoder can maintain global semantic awareness while recovering edges and local structures. In this embodiment, the center is a dilated convolutional block, which expands the receptive field and enhances semantic information without increasing the number of parameters. Since roads are typically long, continuous structures, ordinary convolution may only see a local area, while dilated convolution can maintain the continuity of long-distance information, thereby enhancing the ability to perceive global information.

[0048] Except for Decoder_1, the output of each layer of decoders Decoder_2~Decoder_4 is divided into two identical parts. One part is sent to the next layer of decoder, and the other part is input to the multi-scale feature fusion module MSFF. The output of Decoder_1 is directly input to the multi-scale feature fusion module MSFF.

[0049] The Multi-Scale Feature Fusion (MSFF) module extracts image features from different receptive fields by setting parallel convolutional units with different kernel sizes and dilated convolution rates, and connecting convolutional pathways of different scales in parallel. This effectively integrates feature information from shallow and deep layers, enabling adaptive modeling of road structures with varying widths and shapes. Then, it employs concatenation (Concat) and Conv+BN+ReLU to achieve channel and resolution consistency through feature alignment, followed by element-wise weighted fusion to obtain multi-scale fused features. This process simultaneously captures fine-grained road edge information and global connectivity, improving robustness in complex road scenarios. While maintaining a compact network structure, MSFF significantly enhances the model's dual perception of details and semantics, effectively improving the accuracy and continuity of road extraction.

[0050] This module is as follows Figure 4 As shown, the input of the multi-scale feature fusion module (MSFF) comes from the output feature maps of four different decoders: Decoder_1, Decoder_2, Decoder_3, and Decoder_4. The module performs channel compression and upsampling on the mid-to-high-level feature maps of Decoder_2, Decoder_3, and Decoder_4 to align them spatially with the low-level features output by Decoder_1. Then, the four scale-aligned feature maps with the same number of channels are concatenated along the channel dimension to fully integrate semantic information from different scales. The concatenated feature maps are then fused using a 1×1 convolution operation. After fusion, normalization and non-linear activation functions are introduced to further enhance the features, achieving the integration of multi-scale semantics and edge information, thereby enhancing the model's adaptability to changes in road width.

[0051] Unlike stacking multiple convolutional layers in the backbone network to obtain semantic information, this module is structurally lightweight and computationally efficient, improving the model's multi-scale expressive capabilities without significantly increasing computational overhead. By maintaining a lightweight structure, it achieves collaborative modeling of spatial details and high-level semantics, effectively controlling overall computational cost and improving the integrity of road edges. Experiments demonstrate that the introduction of MSFF significantly enhances the model's ability to recognize roads of different widths, especially exhibiting better robustness in complex backgrounds.

[0052] Since the decoder and multi-scale feature fusion module do not completely restore the image size to match the input size, the final output module performs upsampling to restore the output image size, ensuring that the prediction result is consistent with the input image size. This facilitates the calculation of the loss and other evaluation metrics. Figure 5 As shown, the output module combines transposed convolution with convolution operations and the ReLU activation function to enhance edge and detail features, ultimately generating a segmentation result map of the road region and background region with the same resolution as the input image, achieving pixel-level differentiation between the road region and the background region.

[0053] To verify the effectiveness of the method in this embodiment, experiments were conducted using the DeepGlobe dataset, and the results are as follows: Figure 7 As shown, the results demonstrate that our proposed method outperforms existing methods in preserving edge continuity and road connectivity, with significant improvements in evaluation metrics such as F1-score and IoU. Experimental results also indicate that the multi-scale feature fusion and dual attention mechanism proposed in this invention effectively enhances the extraction capability of road details while improving overall segmentation accuracy and stability.

[0054] Furthermore, the training process of the multi-scale feature attention fusion network MAF-Net proposed in this invention will be explained below, such as... Figure 1 As shown, a Generative Adversarial Network (Generative Adversarial Network) is employed, trained using a training set that includes both labeled and unlabeled portions. First, the network reads the labeled images for pre-training. Then, it is trained using the entire dataset. Images are first processed by a generator to produce corresponding predicted images, i.e., pseudo-labels. Then, the labeled images and the real images are fed into a discriminator, which determines whether the label comes from the dataset or from the generator. A loss function is used to optimize the generator's output, ultimately resulting in the trained Multi-Scale Feature Attention Fusion Network (MAF-Net).

[0055] This invention achieves accurate extraction of targets of different scales and road types by introducing multi-scale feature fusion and attention mechanisms. Employing a semi-supervised learning approach, this invention first constructs an enhanced decoder module within the encoder-decoder structure of the generator. This module fuses multi-level semantic and information spaces through upsampling paths and lateral connection paths, and introduces channel attention and spatial attention mechanisms into the decoder to enhance its ability to focus on road regions. Subsequently, to fully integrate the semantic and edge information of multi-level feature maps, this invention proposes a multi-scale feature fusion module. This module unifies, compresses, and fuses the outputs of each stage of the decoder, improving the modeling ability for road structures with varying widths and shapes. Finally, the fused multi-scale features are used to generate road prediction results. This method maintains high accuracy and robustness even under complex backgrounds and weak supervision, effectively mitigating problems such as road breakage, adhesion, and missegmentation, demonstrating significant engineering application value and potential for widespread adoption.

[0056] Although embodiments of the present invention have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting the present invention. Those skilled in the art can make changes, modifications, substitutions and variations to the above embodiments within the scope of the present invention without departing from the principles and spirit of the present invention.

Claims

1. A deep learning based remote sensing road extraction method, characterized in that: The to-be-processed remote sensing image is input into the trained generator, and finally a road segmentation map with a resolution consistent with that of the input image is output; the generator comprises an encoder module, a decoder module, a multi-scale feature fusion module and an output module; The encoder module comprises multiple layers of encoders, wherein the first layer of encoders Encoder_0 extracts shallow features and performs down-sampling on the input image, and multiple deep encoders are connected in series after the first layer of encoders Encoder_0 to gradually extract deep semantic features. The decoder module fuses the encoder features and the features output by the decoder after up-sampling by a layer through a lateral connection mechanism; except for the last layer of encoders, the output of each layer of encoders is divided into two identical parts, one part is input into the next layer of encoders, and the other part is input into the corresponding decoder as lateral connection features. And a double attention module composed of a channel-space attention module and a spatial context attention module is integrated in the decoder. The decoder module gradually restores the spatial resolution of the image through a lateral connection and an attention mechanism. The multi-scale feature fusion module receives the output of each layer of decoders, first performs channel compression and up-sampling on the medium-high layer feature maps to align them with the low layer feature maps in the spatial dimension, then splices the feature maps of the same scale and consistent number of channels in the channel dimension, and finally uses a 1x1 convolution operation to realize feature fusion, and introduces normalization and a nonlinear activation function after the fusion to further enhance the features, so as to realize the integration of multi-scale semantic and edge information. Finally, the output module is used to up-sample the feature map output by the multi-scale feature fusion module to restore the size of the output image, so as to ensure that the prediction result and the input image are of the same size. 2.The deep learning-based remote sensing road extraction method of claim 1, wherein: The first layer of encoders Encoder_0 in the encoder module extracts shallow features and performs down-sampling on the input image, which is realized by Conv+BN+ReLU+MaxPooling, and multiple deep encoders are connected in series after the first layer of encoders Encoder_0, and the deep encoders are realized by using a residual module ResNet34. 3.The deep learning-based remote sensing road extraction method of claim 1, wherein: Except for the last layer of encoders, the output of each layer of encoders is divided into two identical parts, one part is input into the next layer of encoders, and the other part is input into the corresponding decoder as lateral connection features. 4.The deep learning-based remote sensing road extraction method of claim 3, wherein: The decoder module is composed of a plurality of decoders, and the number of decoders is one less than the number of encoders; each layer of decoders has two input branches, one branch input is a low-resolution feature transmitted from the previous layer of decoders, and the other branch input is a feature output by the corresponding layer of encoders. 5.The deep learning based remote sensing road extraction method of claim 3, wherein: If it is the last layer of decoders, one branch input is the output of the center part, and the other branch input is the feature output by the corresponding layer of encoders; the center part Center is a hole convolution block. 6.The method of claim 3, wherein the method comprises: In the decoder, the low-resolution feature is first up-sampled by a transpose convolution operation, and then batch normalization and nonlinear activation are performed after the up-sampling process, and finally interpolation up-sampling is performed to obtain a high-resolution feature; The features output by the corresponding layer encoder are input through a skip connection, are subjected to a 1x1 convolution operation in a lateral connection path, and are adjusted in channel number with a BatchNorm layer to obtain lateral connection features, thereby aligning the channel number and resolution; the high-resolution features and the lateral connection features are added element by element to obtain fused features.

7. The deep learning-based remote sensing road extraction method according to claim 6, characterized in that: In the decoder, the fused features are input into a dual attention module, wherein a channel-spatial attention module generates a channel description vector through global average pooling and maximum pooling, then inputs a multilayer perceptron and obtains a channel weight through Sigmoid activation; the output of the channel-spatial attention module is input into a spatial context attention module, and the Criss-Cross mechanism is used to model the long-range dependency between pixels in the horizontal and vertical directions. 8.The deep learning based remote sensing road extraction method of claim 1, wherein: In the output module, a transpose convolution and a convolution operation are combined with a ReLU activation function to enhance edge and detail features, and finally a segmentation result image of the road region and the background region with a resolution consistent with the input image is generated to realize pixel-level differentiation of the road region and the background region. 9.The deep learning based remote sensing road extraction method of claim 1, wherein: The generator uses a generative adversarial network and is trained using a training set including a labeled part and an unlabeled part.

Images