Road extraction method from high-resolution remote sensing images based on MPResNet

Abstract

The present invention relates to a method for extracting roads from high-resolution remote sensing images based on MPResNet, belonging to the technical field of remote sensing image processing. This method employs an improved MPResNet model, which uses RestNet34 as the network encoder to ensure network depth and robustness. A feature extraction structure based on multi-scale context acquisition is used to extract semantic features in different scenarios. A spatial attention mechanism that fuses channel representations is introduced to calculate sample attention weights at the spatial and channel levels. In the decoder, dense upsampling at different magnifications replaces the original bilinear interpolation, significantly reducing computational complexity while ensuring accuracy.

Description

Technical Field

[0001] The invention belongs to the technical field of remote sensing image processing and relates to a high-resolution remote sensing image road extraction method based on MPResNet. Background Art

[0002] Road extraction from remote sensing imagery is crucial for a wide range of applications, including urban planning, disaster management, vehicle navigation, and autonomous driving. Rapid advances in aerospace and computer vision technologies have enabled airborne Earth observation platforms like aerial drones and high-resolution satellites to interpret high-spatial-resolution surface information. Furthermore, efficient image capture technologies have made it easier to access massive amounts of data. Consequently, interpreting road information from remote sensing imagery has become a hot topic among researchers.

[0003] Early road extraction methods mainly include edge detection, region growing, and template matching. Edge detection extracts roads by identifying edges in images. This method relies on changes in pixel intensity, and commonly used algorithms include Canny edge detection. However, it is sensitive to noise and prone to false detections and missed detections, especially under changing lighting or complex backgrounds. Region growing refers to extracting roads by merging adjacent similar regions starting from a seed point. This method can handle relatively smooth roads, but is prone to inaccurate segmentation when dealing with complex shapes. Template matching matches images using predefined road models. This method is suitable for roads of known shapes, but is sensitive to the selection and location of templates, has high computational complexity, and is difficult to adapt to different scenarios.

[0004] In recent years, deep learning techniques have achieved remarkable success in computer vision and artificial intelligence, and a growing number of researchers are using this approach to extract road information from remote sensing images. A deep convolutional neural network (CNN) and a sliding window strategy are used to extract road objects from remote sensing images. This method significantly outperforms traditional methods for road extraction, but the inefficiency of the interactive window significantly limits computational speed. The fully convolutional network (FCN) proposed by Long et al. uses standard convolutional layers instead of fully connected layers, significantly advancing image segmentation. However, due to the downsampling pooling operation, some spatial information is lost, affecting the ability to recognize small objects, making it difficult to fully restore the resolution of the input feature map. In 2017, Zhang et al. proposed combining residual blocks with the basic U-net network architecture to achieve road segmentation. He et al. integrated Atrous Spatial Pyramid Pooling (ASPP) with an encoder-decoder network to extract fine-scale road features. Yang et al. designed a recurrent CNN (RCNN) unit to explore detailed low-level spatial features. Zhou et al. developed an encoder-decoder network called D-LinkNet, which consists of a LinkNet and dilated convolutions. Zhang and Wang combined effective dense connections with dilated convolutional layers to obtain the largest receptive field. Zhu et al. improved the completeness of predicted roads by capturing the global context of the road. Wang et al. introduced non-local structures to grasp the long-range relationships of roads. Chen et al. designed a dual-branch encoder to extract multi-scale image features. Lu et al. captured long-range dependencies from the spatial and channel dimensions, which enabled the network to overcome the limitations of local receptive field. Zhou et al. used a graph convolutional network to obtain global information of road spatial and channel features, and compared with competing models, the method achieved higher accuracy in complex backgrounds. Attention mechanisms have been widely used in various computer vision tasks. Hu et al. designed an efficient squeeze and excitation (SE) network that can retain the key features of the channel. Huang et al. designed a cross network to solve the problem of high computational cost of non-local structure while capturing dependencies between long-range pixels.

[0005] Recently, various visual transformer networks have emerged in many vision tasks, demonstrating the power of self-attention. Road extraction tasks require capturing multi-scale context. A common approach is to use dilated convolutions to capture long-range dependencies. However, using ordinary N×N convolution kernels to model long and narrow road features always introduces a lot of irrelevant context. In addition, some existing works equipped with attention mechanisms do not consider road shape. At the same time, these methods still have the shortcomings of U-shaped structure networks, that is, during the downsampling process, convolution and pooling operations will lead to the loss of spatial feature information; and this type of network structure lacks consistent global semantic information, resulting in poor results in the model's extraction of road detail information.

[0006] Therefore, a new road extraction method for remote sensing images is urgently needed to improve extraction accuracy and detection efficiency. Summary of the Invention

[0007] In view of this, the purpose of the present invention is to provide a road extraction method for high-resolution remote sensing images based on MPResNet to solve the problems of complex scenes in high-resolution remote sensing images, such as lighting changes, shadows, bad weather, etc., which will reduce the accuracy of road extraction and detection efficiency.

[0008] In order to achieve the above object, the present invention provides the following technical solutions:

[0009] A road extraction method for high-resolution remote sensing images based on MPResNet is proposed. An improved model (MPResNet) is proposed, which integrates the Multi-Scale Context Acquisition (MSCA) module with horizontal and vertical pooling and convolutions with different dilation ratios, and the Parallel Attention Mechanism (PAM). The MPResNet model uses RestNet34 as the network encoder, ensuring the network depth and robustness. The feature extraction structure of the MSCA module with horizontal and vertical pooling and convolutions with different dilation ratios is used to extract semantic features in different scenarios. At the same time, the Spatial Attention Mechanism (PAM) that integrates channel representation is introduced to calculate the sample attention weights at the spatial and channel levels. In the decoder part, dense upsampling with different magnifications is used instead of the original bilinear interpolation, significantly reducing the computational complexity while ensuring accuracy.

[0010] The method specifically comprises the following steps:

[0011] S1: Obtain a high-definition remote sensing image dataset and divide it into a training set and a test set;

[0012] S2: Build and train a segmentation network model based on MSCA and PAM, referred to as the MPResNet model, which consists of three parts: an encoder, a bridge network, and a decoder. ResNet34 is used as the encoder backbone network. The input image is processed through multiple residual blocks and downsampled by a factor of 32. In the bridge network, the MSCA module is used to capture long-range dependencies, effectively capturing long-range road environments. The PAM module is also used to calculate sample attention weights at the spatial and channel levels. In the decoder, the decoder is optimized based on the Unet decoder, replacing bilinear interpolation upsampling with DUCs of varying magnifications.

[0013] S3: Input the test set into the trained MPResNet model and output the road extraction mask.

[0014] Furthermore, in step S2, the MSCA module includes a dilation block, a horizontal pooling block, and a vertical pooling block. The dilation block uses 3x3 convolutional layers with dilation rates of 1, 2, and 4 in cascade and parallel mode to expand the receptive field and preserve spatial information. In order to combine features of different scales, the receptive field of each path is different, which can better improve the recognition ability of complex patterns while capturing details and global information. The use of skip connections alleviates the problem of gradient disappearance and facilitates the training of deeper neural networks. The pooling branch is used to retain the global information of the road in the vertical and horizontal directions. The principal element addition method is used to unify the operation of feature maps with different receptive fields in the same branch. Then, the feature maps output by different branches are combined through connection and convolution operations to obtain a feature map with rich road context.

[0015] Furthermore, in step S2, the PAM module includes three parts: compression, interaction, and separation. The compression part refers to the fact that channel attention and spatial attention have the same operation in the compression part, and both obtain corresponding channel descriptors and spatial descriptors through maximum pooling and average pooling. The interaction part refers to concatenating the channel descriptors and spatial descriptors, and then modeling the correlation between the global spatial features, i.e., the channel descriptors, and the global channel features, i.e., the spatial descriptors, through an MLP. The separation part refers to splitting the global spatial features and the global channel features after they fully interact, restoring them to their original shape size (1×1×C and H×W×1), and then obtaining the weights through the sigmoid function, weighting the inputs respectively, and obtaining the corresponding outputs.

[0016] Furthermore, in step S2, DUC is used to upsample the feature map to any multiple along the spatial dimension.

[0017] The DUC process is as follows: given a feature map of size H×W×C, N 1×1 convolution filters are applied to the feature map to generate a new encoded feature map of size H×W×N, and then the encoded feature map is reshaped to size 2H×2W×N / 4.

[0018] The beneficial effects of the present invention are as follows: the present invention designs an MPResNet model for the road characteristics of high-resolution remote sensing images and improves the traditional UNet network. The MPResNet model uses ResNet34 as a feature extractor for semantic segmentation. In the bridging network, a remote dependency capture module (MSCA) module that integrates horizontal and vertical pooling and convolutions with different expansion ratios is used to capture remote dependencies, which can effectively capture long-distance road environments. In addition, a spatial attention mechanism (PAM) that integrates channel representations is added to enhance the representation capability of features and increase the ability to understand complex scenes. In order to avoid the loss of accuracy due to upsampling and effectively restore feature details, the present invention uses dense upsampling convolution (DUC) to replace the original upsampling operation, thereby improving the accuracy of the model when processing large-size images.

[0019] The encoder, based on ResNet34, possesses excellent hierarchical feature extraction capabilities. When processing high-resolution remote sensing imagery, it can effectively capture road features at different scales. For example, for wider main roads and narrower side roads in remote sensing images, it uses convolution kernels at different levels to extract corresponding texture, shape, and other features. Low-level convolutional layers can capture basic features such as road edges and lines, while deeper convolutional layers can capture more abstract semantic information, such as road connectivity and overall layout.

[0020] The decoder uses dense upsampling instead of traditional bilinear interpolation. Dense upsampling can more accurately restore the size of feature maps, which is critical for extracting detailed objects such as roads. In remote sensing imagery road extraction, road boundaries and detailed information are crucial for accurate road extraction. Dense upsampling can better utilize the feature information in the decoder, avoiding problems such as blurred road boundaries caused by simple interpolation. Simultaneously, the feature maps of the deep network are input into the spatial attention mechanism of the fusion channel. This information, combined with the spatial attention mechanism, can better understand the spatial distribution and semantic relationships of roads in complex environments, enabling the processing of relatively complex remote sensing imagery.

[0021] The bridge network uses the MSCA module to capture long-range dependencies. This module establishes associations between distant road segments, which is crucial for complete road network extraction. For example, in urban remote sensing imagery, a main road may span multiple blocks. The MSCA module helps the network understand the direction and connectivity of this main road within the overall urban layout, avoiding interrupted or incomplete road extraction due to missing long-range information.

[0022] Other advantages, objects, and features of the present invention will be described in part in the following description and, in part, will be apparent to those skilled in the art upon examination of the following description or may be learned from practice of the present invention. The objects and other advantages of the present invention may be realized and obtained through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] In order to make the purpose, technical solutions and advantages of the present invention more clear, the present invention will be described in detail below with reference to the accompanying drawings, in which:

[0024] Figure 1 The overall structure diagram of the MPResNet model provided by the present invention;

[0025] Figure 2 This is the structural block diagram of the Multi-Scale Context Acquisition (MSCA) module;

[0026] Figure 3 This is the structural diagram of the fusion channel spatial attention mechanism (PAM);

[0027] Figure 4 This is the dense upsampling (DUC) flow chart;

[0028] Figure 5 The MPResNet model of the present invention is used with the six existing models to extract the effect diagram of the road. DETAILED DESCRIPTION

[0029] The following describes the embodiments of the present invention by means of specific examples, and those skilled in the art can easily understand other advantages and effects of the present invention from the contents disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that the illustrations provided in the following embodiments are only schematic illustrations of the basic concept of the present invention, and the following embodiments and features in the embodiments can be combined with each other without conflict.

[0030] See also Figures 1 to 5The present invention provides an improved model, MPResNet. This model uses RestNet34 as the network encoder, ensuring network depth and robustness. It uses the feature extraction structure of the Multi-Scale Context Acquisition (MSCA) module to extract semantic features in different scenarios. It also introduces the Spatial Attention Mechanism (PAM) that integrates channel representations to calculate sample attention weights at the spatial and channel levels. In the decoder, dense upsampling at different rates replaces the original bilinear interpolation, significantly reducing computational complexity while ensuring accuracy.

[0031] MPResNet is a segmentation network based on an encoder-decoder structure. The network takes high-definition remote sensing satellite images as input and outputs a road extraction mask. The white area of ​​the mask represents the road, and the black area represents the background.

[0032] The MPResNet model structure is as follows Figure 1 As shown in the figure, it mainly consists of three parts: encoder, bridge network and decoder. ResNet34 is used as the backbone network of the encoder. The input image is processed by multiple residual blocks, which effectively avoids the model degradation problem caused by the increase in network depth. At the same time, the image is downsampled by 32 times, which enhances the network model's ability to extract semantic features. In the decoder part, it is optimized based on the Unet decoder, and dense upsampling at different rates replaces bilinear interpolation upsampling. When processing complex scenes, it can better combine contextual information to restore details, allowing the network to adaptively adjust weights throughout the process, thereby optimizing the upsampling results.

[0033] (1) Long-range dependency capture module (MSCA) that integrates horizontal and vertical pooling and convolutions with different expansion ratios

[0034] In remote sensing images, a large number of roads are obscured by the surrounding environment. Capturing the relationship between roads and the surrounding environment can enrich the overall characteristics of the roads, which is very helpful for improving the connectivity of roads. Unlike ordinary objects, roads naturally have strong connectivity and a wide coverage, often covering the entire image. To address this problem, we designed an MSCA module to introduce rich road environments. Figure 2As shown, MSCA consists of three components: a dilation block, a horizontal pooling branch, and a vertical pooling branch. The dilation block utilizes a cascaded and parallel pattern of 3 x 3 convolutional layers with dilation rates of 1, 2, and 4 to expand the receptive field and preserve spatial information. To combine features at different scales, the receptive field of each path is different, capturing both details and global information while improving the ability to recognize complex patterns. The use of skip connections mitigates the vanishing gradient problem and facilitates the training of deeper neural networks. The pooling branch preserves global information about the road in both the vertical and horizontal directions. We use principal element addition to unify feature maps with different receptive fields within the same branch. Then, concatenation and convolution operations combine the feature maps output by different branches to produce a feature map with rich road context.

[0035] (2) Spatial Attention Mechanism (PAM) Module for Fusion Channel Representation

[0036] Due to the complex background interference of remote sensing images, the diversity and variability of road features, and the imbalance of data information, the spatial distribution and physical meaning of shallow features differ from those of deep features when extracting roads in convolutional neural networks. By introducing the attention mechanism, greater weight is added to key information and more computing resources are allocated.

[0037] The spatial attention mechanism (CBAM) of the fused channel representation proposed in this paper is Figure 3 As shown in the figure, the attention mechanism module combines channel attention and spatial attention. It improves model performance by adaptively learning the importance of different channels and spatial locations in the feature map. Unlike the traditional CBAM, which uses a simple concatenation model, the spatial attention mechanism that integrates channel attention into spatial attention. Instead of analyzing channels and spatial modeling in isolation, the two components are jointly modeled, allowing them to interact. This results in more comprehensive features, which improves model performance to a certain extent.

[0038] The entire attention mechanism is divided into three parts: compression, interaction, and separation. The compression part refers to the fact that channel attention and spatial attention have the same operation in the compression part, and both obtain the corresponding channel descriptors and spatial descriptors through maximum pooling and average pooling. The interaction part refers to the concatenation of channel descriptors and spatial descriptors, and then using an MLP to model the correlation between global spatial features (referring to channel descriptors) and global channel features (referring to spatial descriptors); the separation part refers to the fact that after the global spatial features and global channel features have fully interacted, we split them and restore them to their original shape size (1×1×C and H×W×1), and then use the sigmoid function to obtain weights, weight the inputs respectively, and obtain the corresponding outputs.

[0039] (3) Dense Upsampling (DUC)

[0040] Bilinear interpolation is a common method for upsampling feature maps in decoders. However, this method is very simple and data-independent, which may lead to suboptimal results. In order to achieve better segmentation results, the present invention uses DUC (DUpsampling) instead of bilinear interpolation for upsampling. Compared with bilinear interpolation, DUpsampling has better learning ability and context perception ability, and can adapt to different scenarios and complex input data distribution. The flowchart of DUpsampling is shown in the figure. Figure 4 As shown in the figure, given a feature map of size H×W×C, N 1×1 convolution filters are applied to the feature map to generate a new encoded feature map of size H×W×N. The encoded feature map is then reshaped to a size of 2H×2W×N / 4, which is the output feature map of the DUpsample module. In general, DUpsampling can upsample feature maps to any multiple along the spatial dimension. In the present invention, the feature map in the context branch and the fused feature map are each upscaled by a factor of 2 along the spatial dimension.

[0041] Verification experiment:

[0042] To verify the effectiveness of the MPResNet model proposed by the method of this invention, this experiment was conducted on two public datasets. The first dataset is Deep Globe Road, which is used to train and test the experimental model. The remote sensing images in this dataset are all taken by Digital Globe satellites from six different countries. The training set contains 6226 images, the validation set contains 1243 images, and the test set contains 1093 images. The second dataset is CHN6-CUG, which selects six cities and marks roads including railways, highways, urban roads, and rural roads. The corresponding MIOU are 78.7% and 79.4% respectively. Ablation experiments were conducted on each module separately. Among them, the multi-scale context acquisition module and the spatial attention mechanism module of the fusion channel improved by 0.42% and 0.45% respectively compared with the backbone, greatly improving the road segmentation effect.

[0043] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not limiting. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present invention can be modified or replaced by equivalents without departing from the purpose and scope of the technical solutions, which should all be included in the scope of the claims of the present invention.

Claims

1. A high-resolution remote sensing image road extraction method based on MPResNet, characterized in that: The method specifically comprises the following steps: S1: Obtain a high-definition remote sensing image dataset and divide it into a training set and a test set; S2: Construct and train a segmentation network model based on MSCA and PAM, referred to as the MPResNet model, which includes three parts: encoder, bridge network and decoder. Among them, ResNet34 is used as the backbone network of the encoder, and the input image is processed by multiple residual blocks, and the image is downsampled by 32 times. In the bridge network, the MSCA module is used to capture long-range dependencies. At the same time, the PAM module is used, which connects the last layer of the encoder and the first layer of the decoder to calculate the sample attention weights at the spatial and channel levels. In the decoder part, the Unet decoder is optimized as the basis, and the bilinear interpolation upsampling is replaced by DUC with different magnifications. Among them, MSCA represents the long-range dependency capture module that integrates horizontal and vertical pooling and convolution with different expansion magnifications, PAM represents the parallel channel spatial attention mechanism, and DUC represents dense upsampling. The MSCA module includes three parallel branches, specifically a dilation block, a horizontal pooling branch, and a vertical pooling branch; the dilation block includes three parallel paths, path one is three cascaded 3×3 convolutions, whose dilation rates are 1, 2, and 4, respectively; path two is two cascaded 3×3 convolutions, whose dilation rates are 1 and 2, respectively; path three is a 3×3 convolution with a dilation rate of 4, the outputs of the three paths are element-wise added, and then spliced ​​with the outputs of the two pooling branches. The spliced ​​feature map undergoes a 1×1 convolution and then element-wise addition with the original input feature map to obtain the final output; the vertical pooling branch is used to retain the global information of the road in the vertical direction. It is first pooled on the width W, and then undergoes convolution, BN normalization, ReLU and interpolation operations; the horizontal pooling branch is used to retain the global information of the road in the horizontal direction. It is first pooled on the height H, and then undergoes convolution, BN normalization, ReLU and interpolation operations; The PAM module consists of three parts: compression, interaction, and separation. The compression part refers to the fact that channel attention and spatial attention have the same operation in the compression part, and both obtain corresponding channel descriptors and spatial descriptors through maximum pooling and average pooling. The interaction part refers to splicing the channel descriptor and the spatial descriptor, and then using an MLP to model the correlation between the global spatial feature, i.e., the channel descriptor, and the global channel feature, i.e., the spatial descriptor. The separation part refers to splitting the global spatial feature and the global channel feature after they have fully interacted, restoring them to their original shape size, and then obtaining weights through the sigmoid function, weighting the inputs respectively, and obtaining the corresponding outputs. S3: Input the test set into the trained MPResNet model and output the road extraction mask.

2. The high-resolution remote sensing image road extraction method according to claim 1, characterized in that: In step S2, DUC is used to upsample the feature map to any multiple along the spatial dimension; the DUC process is: given a feature map of size H × W × C, N 1 × 1 convolution filters are applied to the feature map to generate a new encoded feature map of size H × W × N, and then the encoded feature map is reshaped to size 2H × 2W× N / 4.

Images