MPResNet-based high-resolution remote sensing image road extraction method

Abstract

The invention relates to a high-resolution remote sensing image road extraction method based on MPResNet, and belongs to the technical field of remote sensing image processing. According to the method, an improved model MPResNet is adopted, the model uses RestNet34 as an encoder of a network, and the depth and robustness of the network are guaranteed; a feature extraction structure of multi-scale context collection is used to realize semantic feature extraction in different scenes, and meanwhile, a space attention mechanism represented by a fusion channel is introduced to calculate sample attention weights of space and channel levels; in a decoder part, dense up-sampling with different multiplying powers is adopted to replace original bilinear interpolation, so that the calculation complexity is remarkably reduced while the precision is ensured.

Description

Technical Field

[0001] The invention belongs to the technical field of remote sensing image processing and relates to a high-resolution remote sensing image road extraction method based on MPResNet. Background Art

[0002] Road extraction from remote sensing images is of great significance in a wide range of applications, including urban planning, disaster management, vehicle navigation, and autonomous driving. On the one hand, with the rapid development of aerospace technology and computer vision technology, air-based earth observation platforms such as aerial photography drones and high-resolution satellites can interpret surface information with high spatial resolution; on the other hand, efficient image capture technology makes it easier for people to obtain massive data. Therefore, interpreting road information from remote sensing images has become a hot topic for researchers.

[0003] Early road extraction methods mainly include edge detection, region growing and template matching. Edge detection extracts roads by identifying edges in images. This method relies on changes in pixel intensity. Common algorithms include Canny edge detection. However, it is sensitive to noise and prone to false detection and missed detection, especially under changing lighting or complex backgrounds. Region growing refers to extracting roads by merging adjacent similar regions starting from a seed point. This method can handle smoother roads, but it is prone to inaccurate segmentation when dealing with complex shapes. Template matching matches images by using predefined road models. This method is suitable for roads of known shapes, but is sensitive to the selection and location of templates, and has high computational complexity, making it difficult to adapt to different scenarios.

[0004] In recent years, deep learning technology has achieved remarkable achievements in the fields of computer vision and artificial intelligence, and more and more researchers have used this method to extract road information from remote sensing images. Extract road targets from remote sensing images based on deep convolutional neural networks and sliding window strategies. This method is far superior to traditional methods in road extraction, but the inefficiency of interactive windows far limits the calculation speed. The fully convolutional network (FCN) proposed by Long et al. uses standard convolutional layers instead of fully connected layers, which greatly advances image segmentation. However, due to the downsampling pooling operation, some spatial information is lost, which affects the ability to recognize small targets, making it difficult to fully restore the resolution of the input feature map. In 2017, Zhang et al. proposed combining residual blocks with the U-net basic network architecture to achieve road segmentation. He et al. integrated Atrous Spatial Pyramid Pooling (ASPP) with an encoder-decoder network to extract fine features of roads. Yang et al. designed a recurrent CNN (RCNN) unit to explore detailed low-level spatial features. Zhou et al. developed an encoder-decoder network called D-LinkNet, which consists of LinkNet and dilated convolutions. Zhang and Wang combined effective dense connections with dilated convolutional layers to obtain the largest receptive field. Zhu et al. improved the completeness of predicted roads by capturing the global context of roads. Wang et al. introduced non-local structures to grasp the long-range relationships of roads. Chen et al. designed a dual-branch encoder to extract multi-scale image features. Lu et al. captured long-range dependencies from the spatial and channel dimensions, which enabled the network to overcome the limitation of local receptive field. Zhou et al. adopted a graph convolutional network to obtain global information of road spatial and channel features, which achieved higher accuracy in complex backgrounds compared with competing models. Attention mechanisms have been widely used in various computer vision tasks. Hu et al. designed an efficient squeeze and excitation (SE) network that can preserve the key features of channels. Huang et al. designed a cross network to solve the problem of high computational cost of non-local structures while capturing dependencies between remote pixels.

[0005] Recently, various visual transformer networks have appeared in many visual tasks, showing the power of self-attention. Road extraction tasks require capturing multi-scale contexts. A common approach is to use dilated convolutions to capture long-range dependencies. However, using ordinary N×N convolution kernels to model long and narrow road features always introduces a lot of irrelevant context. In addition, some existing works equipped with attention mechanisms do not consider the shape of the road. At the same time, these methods still have the shortcomings of U-shaped structure networks, that is, during the downsampling process, convolution and pooling operations will lead to the loss of spatial feature information; and this type of network structure lacks consistent global semantic information, resulting in poor results in the model's extraction of road detail information.

[0006] Therefore, a new road extraction method for remote sensing images is urgently needed to improve extraction accuracy and detection efficiency. Summary of the invention

[0007] In view of this, the purpose of the present invention is to provide a high-resolution remote sensing image road extraction method based on MPResNet to solve the problems of complex scenes in high-resolution remote sensing images, such as lighting changes, shadows, bad weather, etc., which will reduce the accuracy of road extraction and detection efficiency.

[0008] In order to achieve the above object, the present invention provides the following technical solutions:

[0009] A road extraction method for high-resolution remote sensing images based on MPResNet is proposed. An improved model (MPResNet) is proposed. The model integrates the long-range dependency capture module (Multi-Scale Context Acquisition, MSCA) of horizontal and vertical pooling and convolutions with different expansion ratios and the parallel channel spatial attention mechanism (PAM). The MPResNet model uses RestNet34 as the encoder of the network to ensure the depth and robustness of the network; the feature extraction structure of the long-range dependency capture module (MSCA) integrating horizontal and vertical pooling and convolutions with different expansion ratios is used to realize the semantic feature extraction in different scenes, and the spatial attention mechanism (PAM) integrating channel representation is introduced to calculate the sample attention weights at the spatial and channel levels; in the decoder part, dense upsampling with different ratios is used instead of the original bilinear interpolation, which significantly reduces the computational complexity while ensuring accuracy.

[0010] The method specifically comprises the following steps:

[0011] S1: Obtain a high-definition remote sensing image dataset and divide it into a training set and a test set;

[0012] S2: Construct and train a segmentation network model based on MSCA and PAM, referred to as the MPResNet model, which includes three parts: encoder, bridge network and decoder. ResNet34 is used as the backbone network of the encoder, and the input image is processed by multiple residual blocks, and the image is downsampled by 32 times. In the bridge network, the MSCA module is used to capture long-range dependencies, which can effectively capture long-distance road environments. At the same time, the PAM module is used to calculate the sample attention weights at the spatial and channel levels. In the decoder part, the decoder of Unet is optimized, and the bilinear interpolation upsampling is replaced by DUCs of different magnifications.

[0013] S3: Input the test set into the trained MPResNet model and output the road extraction mask.

[0014] Further, in step S2, the MSCA module includes a dilation block, a horizontal pooling, and a vertical pooling; the dilation block uses 3x3 convolutional layers with dilation rates of 1, 2, and 4 in cascade and parallel mode, in order to expand the receptive field and preserve spatial information; in order to combine features of different scales, the receptive field of each path is different, which can better improve the recognition ability of complex patterns while capturing details and global information. The use of skip connections alleviates the problem of gradient disappearance and facilitates the training of deeper neural networks; the pooling branch is used to retain the global information of the road in the vertical and horizontal directions; the principal element addition method is used to unify the operation of feature maps with different receptive fields in the same branch, and then, the connection and convolution operations are combined to combine the feature maps output by different branches to obtain feature maps with rich road context.

[0015] Further, in step S2, the PAM module includes three parts: compression, interaction and separation; the compression part means that the channel attention and the spatial attention have the same operation in the compression part, and both obtain the corresponding channel descriptors and spatial descriptors through maximum pooling and average pooling; the interaction part means concatenating the channel descriptors and the spatial descriptors, and then modeling the correlation between the global spatial features, i.e., the channel descriptors, and the global channel features, i.e., the spatial descriptors, through an MLP; the separation part means that after the global spatial features and the global channel features fully interact, they are divided and restored to the original shape size (1×1×C and H×W×1), and then the weights are obtained through the sigmoid function, and the inputs are weighted respectively to obtain the corresponding outputs.

[0016] Further, in step S2, DUC is used to upsample the feature map to any multiple along the spatial dimension.

[0017] The process of DUC is: given a feature map of size H×W×C, N 1×1 convolution filters are applied to the feature map to generate a new encoded feature map of size H×W×N, and then the encoded feature map is reshaped to size 2H×2W×N / 4.

[0018] The beneficial effects of the present invention are as follows: the present invention designs an MPResNet model for the road characteristics of high-resolution remote sensing images and improves the traditional UNet network. The MPResNet model uses ResNet34 as a feature extractor for semantic segmentation. In the bridging network, a remote dependency capture module (MSCA) module that integrates horizontal and vertical pooling and convolutions with different expansion ratios is used to capture remote dependencies, which can effectively capture long-distance road environments. In addition, a spatial attention mechanism (PAM) that integrates channel representations is added to enhance the representation capabilities of features and increase the ability to understand complex scenes. In order to avoid the loss of accuracy caused by upsampling and effectively restore feature details, the present invention uses dense upsampling convolution (DUC) to replace the original upsampling operation to improve the accuracy of the model when processing large-size images.

[0019] The encoder, based on ResNet34, has good hierarchical feature extraction capabilities. When processing high-resolution remote sensing images, it can effectively capture road features of different scales. For example, for wider main roads and narrower branch roads in remote sensing images, it can extract corresponding texture, shape and other features through different levels of convolution kernels. The low-level convolution layer can capture basic features such as the edges and lines of the road, while the deep convolution layer can obtain more abstract semantic information, such as the connectivity and overall layout of the road.

[0020] The decoder uses dense upsampling instead of traditional bilinear interpolation. Dense upsampling can restore the size of the feature map more accurately, which is critical for the extraction of detailed targets such as roads. In remote sensing image road extraction, the boundary and detail information of the road are crucial for accurate road extraction. Dense upsampling can better utilize the feature information in the decoder and avoid problems such as blurred road boundaries caused by simple interpolation. At the same time, the feature map of the deep network is input into the spatial attention mechanism of the fusion channel. This information combined with the spatial attention mechanism can better understand the spatial distribution and semantic relationship of roads in complex environments, and can process relatively complex remote sensing images.

[0021] The bridge network uses the MSCA module to capture long-range dependencies. It can establish associations between long-distance road parts, which is very important for the complete extraction of road networks. For example, in urban remote sensing images, a main road may span multiple blocks. The MSCA module can help the network understand the direction and connection relationship of this main road in the entire urban layout, avoiding interruptions or incomplete road extraction due to the lack of long-distance information.

[0022] Other advantages, objectives and features of the present invention will be described in the following description to some extent, and to some extent, will be obvious to those skilled in the art based on the following examination and study, or can be taught from the practice of the present invention. The objectives and other advantages of the present invention can be realized and obtained through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] In order to make the purpose, technical solutions and advantages of the present invention more clear, the present invention will be described in detail below in conjunction with the accompanying drawings, wherein:

[0024] Figure 1 The overall structure diagram of the MPResNet model provided by the present invention;

[0025] Figure 2 This is the structural block diagram of the multi-scale context acquisition (MSCA) module;

[0026] Figure 3 This is the structural diagram of the fusion channel spatial attention mechanism (PAM);

[0027] Figure 4 This is the dense upsampling (DUC) flow chart;

[0028] Figure 5 The MPResNet model of the present invention and the six existing models are used to extract the effect diagram of the road. DETAILED DESCRIPTION

[0029] The following describes the embodiments of the present invention by specific examples, and those skilled in the art can easily understand other advantages and effects of the present invention from the contents disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that the illustrations provided in the following embodiments only illustrate the basic concept of the present invention in a schematic manner, and the following embodiments and features in the embodiments can be combined with each other without conflict.

[0030] See also Figure 1 to Figure 5, the present invention provides an improved model MPResNet. The model uses RestNet34 as the encoder of the network, ensuring the depth and robustness of the network; uses the feature extraction structure of the multi-scale context acquisition (MSCA) module to realize the extraction of semantic features in different scenarios, and introduces the spatial attention mechanism (PAM) of the fusion channel representation to calculate the sample attention weights at the spatial and channel levels; in the decoder part, dense upsampling with different magnifications is used instead of the original bilinear interpolation, which significantly reduces the computational complexity while ensuring accuracy.

[0031] MPResNet is a segmentation network based on an encoder-decoder structure. The network takes high-definition remote sensing satellite images as input and outputs a road extraction mask. The white area of ​​the mask represents the road, and the black area represents the background.

[0032] The MPResNet model structure is as follows Figure 1 As shown in the figure, it mainly consists of three parts: encoder, bridge network and decoder. ResNet34 is used as the backbone network of the encoder. The input image is processed by multiple residual blocks, which effectively avoids the model degradation problem caused by the increase of network depth. At the same time, the image is downsampled by 32 times, which enhances the ability of the network model to extract semantic features. In the decoder part, it is optimized based on the decoder of Unet, and dense upsampling of different rates replaces bilinear interpolation upsampling. When processing complex scenes, it can better combine context information to restore details, so that the network can adaptively adjust weights in the whole process, thereby optimizing the upsampling results.

[0033] (1) Long-range Dependency Capture Module (MSCA) that integrates horizontal and vertical pooling and convolutions with different dilation ratios

[0034] There are a lot of roads in remote sensing images that are blocked by the surrounding environment. Capturing the relationship between the road and the surrounding environment can enrich the overall characteristics of the road, which is very helpful for improving the connectivity of the road. Unlike ordinary objects, roads naturally have strong connectivity and a wide coverage, often covering the entire image. To address this problem, we designed an MSCA module to introduce a rich road environment. Figure 2As shown in the figure, MSCA consists of three parts: a dilation block, a horizontal pooling, and a vertical pooling. The dilation block uses a cascade and parallel mode of 3x 3 convolutional layers with dilation rates of 1, 2, and 4, in order to expand the receptive field and preserve spatial information. In order to combine features of different scales, the receptive field of each path is different, which can better improve the recognition ability of complex patterns while capturing details and global information. The use of skip connections alleviates the problem of gradient disappearance and facilitates the training of deeper neural networks. The pooling branch can retain the global information of the road in the vertical and horizontal directions. We use principal element addition to unify the operation of feature maps with different receptive fields in the same branch, and then, concatenate and convolve to combine the feature maps output by different branches to obtain feature maps with rich road context.

[0035] (2) Spatial Attention Mechanism (PAM) Module for Fusion Channel Representation

[0036] Due to the complex background interference of remote sensing images, the diversity and variability of road features, and the imbalance of data information, the shallow features differ from the deep features in spatial distribution and physical meaning when extracting roads in convolutional neural networks. By introducing the attention mechanism, greater weight is added to key information and more computing resources are allocated.

[0037] The spatial attention mechanism (CBAM) of the fused channel representation proposed in this invention is as follows: Figure 3 As shown in the figure, the attention mechanism module combines channel attention and spatial attention. It improves the performance of the model by adaptively learning the importance of different channels and spatial positions in the feature map. Different from the traditional CBAM through a simple series mode, the spatial attention mechanism of fused channel representation integrates channel attention into spatial attention. It no longer analyzes channel and space modeling in isolation. The two parts are jointly modeled to allow them to interact with each other. In this way, the features obtained are more comprehensive, which improves the performance of the model to a certain extent.

[0038] The entire attention mechanism is divided into three parts: compression, interaction, and separation. The compression part refers to the fact that channel attention and spatial attention have the same operation in the compression part, and both obtain the corresponding channel descriptors and spatial descriptors through maximum pooling and average pooling. The interaction part refers to the concatenation of channel descriptors and spatial descriptors, and then the correlation between global spatial features (referring to channel descriptors) and global channel features (referring to spatial descriptors) is modeled through an MLP; the separation part refers to the fact that after the global spatial features and global channel features fully interact, we split them and restore them to their original shape size (1×1×C and H×W×1), and then obtain the weights through the sigmoid function, weight the inputs respectively, and obtain the corresponding outputs.

[0039] (3) Dense Upsampling (DUC)

[0040] Bilinear interpolation is a common method for upsampling feature maps in decoders. However, this method is very simple and data-independent, which may lead to suboptimal results. In order to achieve better segmentation effect, the present invention uses DUC (DUpsampling) instead of bilinear interpolation for upsampling. Compared with bilinear interpolation, DUpsampling has better learning ability and context perception ability, and can adapt to different scenarios and complex input data distribution. The flowchart of DUpsampling is shown in the figure. Figure 4 As shown, given a feature map of size H×W×C, N 1×1 convolution filters are applied to the feature map to generate a new encoded feature map of size H×W×N. The encoded feature map is then reshaped to a size of 2H×2W×N / 4, which is the output feature map of the DUpsample module. In general, DUpsampling can upsample the feature map to any multiple along the spatial dimension. In the present invention, the feature map in the context branch and the fused feature map are magnified by 2 times along the spatial dimension respectively.

[0041] Verification experiment:

[0042] In order to verify the effectiveness of the MPResNet model proposed by the method of the present invention, this experiment was conducted on two public data sets. The first data set is Deep Globe Road, which is used to train and test the experimental model. The remote sensing images in this data set are all taken by Digital Globe satellites from 6 different countries, including 6226 images in the training set, 1243 images in the validation set, and 1093 images in the test set; the second data set is CHN6-CUG, which selects 6 cities and marks roads including railways, highways, urban roads and rural roads. The corresponding MIOUs are 78.7% and 79.4% respectively, and ablation experiments are performed on each module respectively. The multi-scale context acquisition module and the spatial attention mechanism module of the fusion channel are improved by 0.42% and 0.45% respectively compared with the backbone, and the road segmentation effect is greatly improved.

[0043] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solution of the present invention can be modified or replaced by equivalents without departing from the purpose and scope of the technical solution, which should be included in the scope of the claims of the present invention.

Claims

1. A high-resolution remote sensing image road extraction method based on MPResNet, characterized in that: The method specifically comprises the following steps: S1: Obtain a high-definition remote sensing image dataset and divide it into a training set and a test set; S2: Construct and train a segmentation network model based on MSCA and PAM, referred to as MPResNet model, which includes three parts: encoder, bridge network and decoder; ResNet34 is used as the backbone network of the encoder, the input image is processed by multiple residual blocks, and the image is downsampled by 32 times; in the bridge network, the MSCA module is used to capture long-range dependencies; the PAM module is used to calculate the sample attention weights at the spatial and channel levels; in the decoder part, the Unet decoder is optimized as the basis, and the bilinear interpolation upsampling is replaced by DUC with different magnifications; MSCA represents the long-range dependency capture module that integrates horizontal and vertical pooling and convolution with different expansion magnifications, PAM represents the parallel channel spatial attention mechanism, and DUC represents dense upsampling; S3: Input the test set into the trained MPResNet model and output the road extraction mask.

2. The high-resolution remote sensing image road extraction method according to claim 1, characterized in that: In step S2, the MSCA module includes a dilation block, a horizontal pooling block, and a vertical pooling block; The dilation block uses 3x3 convolutional layers with dilation rates of 1, 2, and 4 in cascade and parallel mode; the pooling branch is used to retain the global information of the road in the vertical and horizontal directions; the principal element addition method is used to unify the operation of feature maps with different receptive fields in the same branch, and then the concatenation and convolution operations are used to combine the feature maps output by different branches to obtain feature maps with rich road context.

3. The high-resolution remote sensing image road extraction method according to claim 1, characterized in that: In step S2, the PAM module includes three parts: compression, interaction and separation; the compression part means that the channel attention and the spatial attention have the same operation in the compression part, and both obtain the corresponding channel descriptors and spatial descriptors through maximum pooling and average pooling; the interaction part means concatenating the channel descriptors and the spatial descriptors, and then modeling the correlation between the global spatial features, i.e., the channel descriptors, and the global channel features, i.e., the spatial descriptors, through an MLP; the separation part means that after the global spatial features and the global channel features fully interact, they are segmented and restored to their original shape size, and then the weights are obtained through the sigmoid function, and the inputs are weighted respectively to obtain the corresponding outputs.

4. The high-resolution remote sensing image road extraction method according to claim 1, characterized in that: In step S2, DUC is used to upsample the feature map to any multiple along the spatial dimension; the process of DUC is: given a feature map of size H×W×C, N 1×1 convolution filters are applied to the feature map to generate a new encoded feature map of size H×W×N, and then the encoded feature map is reshaped to size 2H×2W×N / 4.

Images