A Remote Sensing Image Road Extraction Method Based on a Multi-Dimensional and Multi-Scale U-Net Network

Abstract

The present invention claims protection for a method for extracting roads from remote sensing images based on a multi-dimensional multi-scale U-net network. First, the processed data is input into the encoding network. In the encoding network part, a residual structure and an attention feature fusion mechanism are combined to extract road information, comprehensively considering global and local feature extraction to improve the network's extraction of road detail targets. Secondly, at the bridging network part, an Atrous Spatial Pyramid Pooling (ASPP) module is added to perform multi-scale feature extraction on road information. Finally, at the decoding network part, a feature alignment module is added to adjust the inaccurate corresponding relationship between high-level and low-level features caused by the non-learnability of the upsampling operation and the repeatability of upsampling and downsampling. The present invention calculates the model loss using a composite loss function combining cross-entropy and Dice coefficient, alleviates the phenomenon of imbalance between positive and negative samples existing in the remote sensing road dataset, and improves the road extraction result of the model.

Description

Technical Field

[0001] The invention belongs to the technical field of remote sensing image processing, and specifically designs a remote sensing image road extraction method based on an improved U-net network. Background Art

[0002] Road information is essential to our daily lives and travel, serving as the backbone and fundamental mode of transportation, and providing valuable support for the development of human civilization. Road extraction is extremely valuable in numerous applications, such as autonomous driving, urban planning, intelligent transportation systems, emergency risk management, and geographic information updates.

[0003] Currently, methods for extracting roads from remote sensing images are primarily divided into traditional methods and deep learning-based methods. Traditional methods typically utilize hand-crafted features for road extraction and can be further categorized into pixel-based and object-oriented methods. Pixel-based methods primarily extract roads by analyzing differences in spectral features and can generally be categorized into spectral analysis, threshold segmentation, and edge detection. Pixel-based methods fully utilize the image's grayscale values and are effective in extracting roads from remote sensing images with clear images, simple backgrounds, and sparse road networks. However, these methods are prone to salt-and-pepper effects and are unable to effectively distinguish background interference. Object-oriented methods treat roads as a whole and identify them using information modeling. While they offer good noise immunity and applicability, they can easily lead to confusion and misclassification of spatially adjacent and similarly shaped features.

[0004] With the rapid growth of available data and computing power, the use of deep learning technology has achieved tremendous success in the field of computer vision. Due to its excellent performance and generalization capabilities, deep learning has been increasingly used to extract information from high-resolution remote sensing imagery. Unlike traditional methods that require manual extraction of road features from remote sensing images, deep learning actively learns feature experience through training neural networks. It can automatically learn shallow information features through repeated network iterations and then gradually learn deeper abstract features. On the one hand, deep learning methods can explore high-level features of roads, thereby improving the effectiveness of computer vision tasks. On the other hand, its adaptive learning and feature fitting capabilities are very powerful, and it has significant advantages in the accuracy and automation of road extraction.

[0005] Mnih, Hinton, and others (2013) first applied deep learning techniques to road extraction and constructed a Massachusetts road dataset. Subsequently, Jonathan Long and others proposed the fully convolutional network (FCN). By replacing fully connected layers with standard convolutional layers, they achieved pixel-level classification, while preserving the spatial information of the original input image and significantly improving segmentation. Consequently, FCN-based extended FCNs have been increasingly used, particularly for road extraction. However, with the increasing resolution of remote sensing images, the detailed features of road areas in images have become increasingly complex, and road interference factors (such as buildings and trees) have become increasingly numerous. Many non-road areas (such as urban buildings and vegetable greenhouses) also have features highly similar to roads. Due to the continuous downsampling operations in FCNs, some spatial information is lost, making it difficult to consider multi-scale features. Consequently, the output of these FCN-based extended FCNs struggles to reconstruct accurate detailed information. Consequently, these FCN-based methods no longer meet societal needs. CN110807376A discloses a method and apparatus for extracting roads outside cities based on remote sensing images. The method comprises obtaining GIS image information from digital maps and generating labeled data and training / test data; constructing an initial road extraction network model based on a U-Net network; training the initial network model using the labeled data and training / test data to obtain a road extraction model capable of identifying roads; and using the road extraction model to detect remote sensing images and automatically extract road targets. This application improves the accuracy of extracting roads outside cities using remote sensing images by constructing an improved U-Net network.

[0006] First, the patent uses feature summation to connect the two branches of the residual structure. This fusion method only assigns fixed weights to the features, does not consider changes in feature content, and is inefficient. The present invention uses attention feature fusion to fuse the two branches of the residual structure, which can dynamically and adaptively fuse the received features in a scale-aware manner, bridge the semantic gap between different branches, and enhance the network's ability to learn global and local information of remote sensing images. Secondly, due to the repeated application of up and down sampling, there is an inaccurate correspondence between high-level features and low-level features connected in the decoding network. Connecting unaligned features only through channel superposition may have an adverse effect on subsequent learning. This patent does not take this issue into account. To address this problem, the present invention adds a feature alignment module to the decoding network, and dynamically establishes positional correspondences between features at different levels through the feature alignment module, thereby improving the decoder's ability to reconstruct precise details. Summary of the Invention

[0007] The present invention aims to solve the above problems of the prior art. A method for extracting roads from remote sensing images based on a multi-dimensional multi-scale U-net network is proposed. The technical solution of the present invention is as follows:

[0008] A method for extracting roads from remote sensing images based on a multi-dimensional multi-scale U-net network, comprising the following steps:

[0009] Step 1: Select the publicly available remote sensing road dataset, the Massachusetts road dataset, as the original data, and perform preprocessing steps including cropping and data augmentation;

[0010] Step 2: Input the preprocessed data into the encoding network, and the encoding network combines a residual structure and an attention feature fusion mechanism to perform multi-scale extraction of road feature information;

[0011] Step 3: The output of the encoding network is used as the input of the bridging network. An atrous spatial pyramid pooling (ASPP) module is added to the bridging network part. The ASPP module includes parallel atrous convolutional layers. The parallel atrous convolutional layers in the ASPP module are equivalent to multiple different receptive fields, which are used for parallel sampling at multiple scales to achieve multi-scale feature fusion on deep features;

[0012] Step 4: Decoding network stage: Gradually restore the feature image to the size of the input image through upsampling. A feature alignment module (FAM) is added to the decoding network. This module takes the high-level features and the low-level features of the corresponding layers of the encoding network as inputs to generate a semantic flow, and uses the semantic flow to adjust the feature maps of two adjacent levels to generate a high-resolution and strongly semantic feature output;

[0013] Step 5: Finally, change the number of channels to 2 through a 1×1 convolutional layer, and test the model using the test set in the Massachusetts dataset;

[0014] During model training, a composite loss function combining a cross-entropy loss function and a Dice loss function is used to calculate the loss of the model.

[0015] Further, the specific content of step 1 includes:

[0016] The images in the Massachusetts dataset are all 1500×1500 in size. Set a 256×256 area to crop all the images in the original dataset; Use the high-resolution data of 256×256×3 bands as the input data and input it into the constructed encoding network to extract road information.

[0017] Further, when the preprocessed data is input into the encoding network in step 2, and the encoding network combines a residual structure and an attention feature fusion mechanism to perform multi-scale extraction of road feature information, it specifically includes:

[0018] The encoding network includes a convolutional sequence block (CSB) and an attention residual learning unit (ARLU). The preprocessed RGB image is first transformed into high-dimensional features through the convolutional sequence block, and then multi-scale and multi-level features are generated through the attention residual learning unit. In the attention residual learning unit, residual units are used to replace ordinary neural network units, and then the identity mapping branch and the residual branch in the residual units are fused through an attention feature fusion module. By fusing the two branches of the residual structure through the attention feature fusion method, the network can extract information at multiple scales along the channel dimension of the feature map while maintaining the lightweight of the network.

[0019] Further, in step 3, the ASPP module of the bridging network includes 5 parallel branches, namely: a 1×1 convolutional branch, three 3×3 dilated convolutional branches, and a global average pooling branch. Among them, the 1×1 convolutional branch and the global average pooling branch are equivalent to using the smallest and largest receptive fields respectively to maintain the inherent characteristics of the input. The other three branches are set with different dilation rates to describe the image features at different scales.

[0020] Further, step 4 specifically includes: gradually restoring the feature image to the size of the input image through upsampling in the decoding network stage. Connect the high-level features of the decoding network and the low-level features of the corresponding layers of the encoding network using a feature alignment module. Inside the feature alignment module, first, the high-level features change the image size and the number of channels through transposed convolution, and then the changed high-level features and low-level features are connected and passed through a convolutional operation to generate semantics. According to the semantic flow, the feature alignment module adjusts the inaccurate correspondence between the high-level and low-level features caused by repeated upsampling and downsampling, so that the semantic information in the high-level features can flow better into the low-level features, bridging the semantic and resolution differences between the high-level and low-level features, and guiding the model to better restore to the initial resolution while containing rich semantic information.

[0021] Further, the model prediction stage of step 5 specifically includes: changing the number of channels of the feature map to 2 through a 1×1 convolutional layer to generate the final prediction map; inputting the test images in the Massachusetts dataset into the trained model after preprocessing.

[0022] Further, a composite loss function composed of a cross-entropy loss function and a Dice loss function is used to calculate the model loss. The definitions of the cross-entropy loss function and the Dice loss function are as follows:

[0023]

[0024]

[0025] Where N represents the total number of pixels, and g i represents the true label value of pixel i, and p i represents the predicted value of pixel i;

[0026] The composite loss function is defined as follows:

[0027] L = L BCE + L D .

[0028] The advantages and beneficial effects of the present invention are as follows:

[0029] The innovation of the present invention is mainly the cooperation of steps 2, 3, and 4 of the claims. Step 2 introduces residual learning, making network training easier and largely solving the degradation problem existing in deep networks; at the same time, an attention feature fusion module is used to fuse the two branches of the residual structure, making up for the semantic gap between different branches and enhancing the network's learning ability for global and local information of remote sensing images. Step 3 uses ASPP in the bridging network to increase the receptive field of the convolutional kernel through parallel dilated convolutional layers, further extracting and fusing information at multiple scales for high-level features and improving the connectivity of roads. Step 4 adds a feature alignment module to the decoding network to dynamically establish the position correspondence relationship between features at different levels, solving the misalignment problem existing between features at different levels and improving the ability of the decoding network to reconstruct precise details. Description of the Drawings

[0030] Figure 1 is a framework diagram of a remote sensing image road extraction model based on MMS-UNet provided by a preferred embodiment of the present invention.

[0031] Figure 2 is the convolutional sequence block in the encoding network

[0032] Figure 3 is the attention residual learning unit (ARLU) in the encoding network

[0033] Figure 4 is the attention fusion module (AFF)

[0034] Figure 5 is the ASPP module

[0035] Figure 6 is the feature alignment module (FAM) Detailed Embodiments

[0036] Next, the technical solutions in the embodiments of the present invention will be clearly and detailedly described in conjunction with the drawings in the embodiments of the present invention. The described embodiments are only a part of the embodiments of the present invention.

[0037] The technical solution of the present invention to solve the above technical problems is as follows:

[0038] In practical applications, the present invention aims to solve the problems of limited receptive fields of models, low fusion efficiency of the identity mapping branch and the residual branch in the residual structure, and low fusion efficiency of high-level and low-level features. To address the above problems, we propose a remote sensing image road extraction method based on multi-dimensional multi-scale U-net. This method fully utilizes information from different layers through an attention feature fusion method, uses ASPP to expand the receptive field of the convolutional kernel, then adds a feature module to align and fuse high-level and low-level information, upsamples to the size of the input image, and generates a prediction map. Testing on the Massachusetts publicly available road dataset shows that our method has a good effect on predicting images.

[0039] Figure 1 The following shows the specific implementation steps and structure of the network structure of the present invention, where:

[0040] Step (1) is the preprocessing method for the input data of this model. The images in the Massachusetts dataset are all of size 1500×1500. We set a 256×256 area to crop all the images in the original dataset. The high-resolution data of 256×256×3 bands is used as the input data and input into the MMS-UNet network model we built to extract road information.

[0041] Step (2) is to combine the residual structure and the attention feature fusion module in the encoding network for road feature extraction. The encoding network includes a convolutional sequence block CSB and an attention residual learning unit ARLU. The convolutional sequence block is composed of two stacked convolutional sequences, and each convolutional sequence includes a 3×3 convolutional layer, a batch normalization layer, and a ReLU layer. The input RGB image is first transformed into high-dimensional features through the convolutional sequence block, and then multi-scale and multi-level features are generated through the attention residual learning unit; in the attention residual learning unit, residual units are used to replace ordinary neural network units, and then the identity mapping branch and the residual branch in the residual units are fused through the attention feature fusion module; the attention feature fusion module first outputs the identity mapping branch and the output of the residual branch are fused by feature summation, and then the fused result A is input into the multi-scale channel attention module MS-CAM. One branch G(A) uses global average pooling to obtain global channel context information, and the other branch L(A) directly uses point convolution to obtain local channel context information. The two branches G(A) and L(A) are fused through a summation operation, and the output of MS-CAM is set as M, then:

[0042] G(A) = B(PW2(δ(B(PW1(g(A)))))) (1)

[0043] L(A) = B(PW2(δ(B(PW1(A))))) (2)

[0044] M(A) = σ(G(A) + L(A)) (3)

[0045] where g(·) represents the global average pooling operation (GAP). PW1 and PW2 respectively represent the pointwise convolution operations with convolution kernels of and respectively, r represents the channel reduction rate. B represents the batch normalization operation (BN), δ represents the ReLU activation function, and σ represents the Sigmoid function. Therefore, the output F of the attention feature fusion module AFF can be shown by Equation (4):

[0046]

[0047] where represents element-wise multiplication. This module adds local information to global information through pointwise convolution, enabling the network to extract information at multiple scales along the channel dimension of the feature map while maintaining the lightweight of the network. The attention residual learning unit connects the residual branch and the identity mapping branch through the attention feature fusion module, dynamically and adaptively fusing the received features in a scale-aware manner, bridging the semantic gap between features of different branches, enhancing the network's learning ability for global and local information of remote sensing images, and improving the accuracy of road recognition.

[0048] Step (3) is to introduce the ASPP module in part of the bridging network to achieve multi-scale feature extraction on deep features. The ASPP module contains 5 branches, a 1×1 convolutional branch, three 3×3 dilated convolutional branches, and a global average pooling branch. Among them, the 1×1 convolutional branch and the global average pooling branch are equivalent to using the smallest and largest receptive fields respectively to maintain the inherent characteristics of the input. The other three branches are respectively set with dilation rates of 6, 12, and 18 to perform feature sampling on the feature map in parallel; finally, the outputs of these five branches are fused through feature concatenation, and a 1×1 convolutional layer is used to adjust the number of channels.

[0049] Step (4) is to gradually restore the feature image to the size of the input image through upsampling in the decoding network stage. The output of the bridging network first changes the number of channels and the image size to be consistent with the features of the corresponding layer of the encoding network through transposed convolution, and then fuses F H and F L through concatenation and inputs them into a 3×3 convolutional layer to generate the semantic flow S L , as shown in Equation (1);

[0050] S L = Conv1(Cat(T(F H ), F L )) (1)

[0051] In the formula, Conv1(·) represents a 3×3 convolution operation, Cat(·) represents a concatenation operation, and T(·) represents a transposed convolution operation. The obtained semantic flow is equivalent to the offset in two directions between the high-level features and the low-level features. Each pixel point p L on the low-level feature map is corresponded to the pixel point p H on the high-level feature map through addition operation. Then, linear interpolation is performed on the four neighboring points of p H to achieve the semantic alignment of the high-level features and the low-level features, as shown in formulas (2) and (3):

[0052]

[0053]

[0054] In the formula, Ν(p H ) represents the neighboring points of the pixel point p H in the high-level feature map, and w p represents the bilinear kernel weight estimated by the distance of the warped grid. After changing the number of channels of F L through a 1×1 convolution and summing it with F H (p H ), the output F out of the feature alignment module is obtained, that is:

[0055] F out = Conv2(F L ) + F H (p H ) (4)

[0056] In the formula, Conv2(·) represents a 1×1 convolution operation; the semantic and resolution differences between the high-level and low-level features are bridged by the feature alignment module, guiding the model to better restore to the initial resolution while containing rich semantic information.

[0057] Step (5) is the final prediction stage of the model. The number of channels of the feature map is changed to 2 through a 1×1 convolution layer to generate the final prediction map. The test images in the Massachusetts dataset are preprocessed and then input into the trained model. The experimental results show that the improved U-net network model can achieve better road extraction results.

[0058] Step (6) The road extraction task of the present invention uses a composite loss function composed of a cross-entropy loss function and a Dice loss function. In remote sensing images, roads are narrow areas, and compared with the entire image, the proportion of roads is very small. There is a significant imbalance in the sample categories between roads and the background. The cross-entropy loss function evaluates each pixel in the segmentation result. If there is a class imbalance problem in the image, it will lead to overfitting of the category with more samples. When extracting roads from remote sensing images, the network will tend to learn the background, reducing the network's ability to extract foreground targets. The Dice coefficient takes all the pixels of a category as a whole and calculates the proportion of the intersection of the two categories in the whole, so it is not affected by a large number of background pixels and can achieve better results in the case of sample imbalance. Therefore, the definitions of the cross-entropy loss function and the Dice loss function are as follows:

[0059]

[0060]

[0061] where N represents the total number of pixels, g i represents the true label value of pixel i, and p i represents the predicted value of pixel i;

[0062] The composite loss function is defined as follows:

[0063] L = L BCE + L D (3)

[0064] The systems, devices, modules, or units illustrated in the above embodiments can be specifically implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer. Specifically, the computer can be, for example, a personal computer, a laptop computer, a cellular phone, a camera phone, a smart phone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or any combination of these devices.

[0065] A computer-readable medium includes both permanent and non-permanent, removable and non-removable media that can implement information storage by any method or technology. The information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices, or any other non-transitory medium that can be used to store information that can be accessed by a computing device. As defined herein, a computer-readable medium does not include transitory computer-readable media such as modulated data signals and carrier waves.

[0066] It should also be noted that the term "comprising", "including" or any other variant thereof is intended to cover non-exclusive inclusion, such that a process, method, article or apparatus comprising a series of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article or apparatus. Without further limitation, an element defined by the statement "comprising a..." does not exclude the presence of additional identical elements in the process, method, article or apparatus comprising the element.

[0067] The above embodiments should be understood as being only for illustrative purposes of the present invention and not for limiting the scope of protection of the present invention. After reading the content described in the present invention, those skilled in the art can make various changes or modifications to the present invention, and these equivalent changes and modifications also fall within the scope defined by the claims of the present invention.

Claims

1. A method for extracting roads from remote sensing images based on a multi-dimensional and multi-scale U-net network, characterized in that It includes the following steps: Step 1: Select the publicly available remote sensing road dataset, the Massachusetts road dataset, as the original data, and perform preprocessing steps including cropping and data augmentation; Step 2: Input the preprocessed data into the encoding network, which combines a residual structure and an attention feature fusion mechanism to perform multi-scale extraction of road feature information; Step 3: The output of the encoding network is used as the input of the bridging network. An Atrous Spatial Pyramid Pooling (ASPP) module is added to the bridging network part. The ASPP module includes parallel atrous convolutional layers. The parallel atrous convolutional layers in the ASPP module are equivalent to multiple different receptive fields, which are used for parallel sampling at multiple scales to achieve multi-scale feature fusion on deep features; Step 4: Decoding network stage: Gradually restore the feature image to the size of the input image through upsampling; Add a Feature Alignment Module (FAM) to the decoding network. This module takes the high-level features and the low-level features of the corresponding layers of the encoding network as inputs to generate a semantic flow, and uses the semantic flow to adjust the feature maps of two adjacent levels to generate a high-resolution and strong-semantic feature output; Step 5: Finally, change the number of channels to 2 through a 1×1 convolutional layer, and test the model using the test set in the Massachusetts dataset; During model training, a composite loss function that combines the cross-entropy loss function and the Dice loss function is used to calculate the loss of the model; In Step 2, the preprocessed data is input into the encoding network, which combines a residual structure and an attention feature fusion mechanism to perform multi-scale extraction of road feature information. Specifically, it includes: The encoding network includes a Convolution Sequence Block (CSB) and an Attention Residual Learning Unit (ARLU). The RGB image after preprocessing operations is first transformed into high-dimensional features through the convolution sequence block, and then multi-scale and multi-level features are generated through the attention residual learning unit; In the attention residual learning unit, residual units are used to replace ordinary neural network units, and then the identity mapping branch and the residual branch in the residual units are fused through the attention feature fusion module; The two branches of the residual structure are fused through the attention feature fusion method, enabling the network to extract information on multiple scales along the channel dimension of the feature map while maintaining the lightweight of the network; In Step 3, the ASPP module in the bridging network contains 5 parallel branches, namely: a 1×1 convolutional branch, three 3×3 dilated convolutional branches, and a global average pooling branch. Among them, the 1×1 convolutional branch and the global average pooling branch are equivalent to using the smallest and largest receptive fields respectively to maintain the inherent characteristics of the input, and the other three branches are set with different dilation rates to describe the image features at different scales.

2. A remote sensing image road extraction method based on a multi-dimensional multi-scale U-net network according to claim 1, characterized in that, Step 1 specifically includes: The images in the Massachusetts dataset are all of size 1500×1500. Set a 256×256 area to crop all the images in the original dataset; Use the high-resolution data of 256×256×3 bands as the input data and input it into the constructed encoding network to extract road information.

3. A method for extracting roads from remote sensing images based on a multi-dimensional multi-scale U-net network according to claim 1, characterized in that, The step 4 specifically includes: gradually upsampling and restoring the feature image to the input image size in the decoding network stage, and connecting the high-level features of the decoding network and the low-level features of the corresponding level of the encoding network using a feature alignment module; in the feature alignment module, first, the high-level features are changed to change the image size and the number of channels through inverse convolution, and then the changed high-level features are connected to the low-level features, and a convolution operation is performed to generate a semantic flow; according to the semantic flow, the feature alignment module adjusts the inaccurate correspondence between the high-level and low-level features caused by repeated up and down sampling, so that the semantic information in the high-level features can better flow to the low-level features, bridging the semantic and resolution differences between the high-level and low-level features, and guiding the model to better restore to the initial resolution while containing rich semantic information.

4. A method for extracting roads from remote sensing images based on a multi-dimensional multi-scale U-net network according to claim 3, characterized in that, The model prediction stage in step 5 specifically includes: changing the number of feature map channels to 2 through a 1×1 convolutional layer to generate a final prediction map; and inputting the test image in the Massachusetts dataset into the trained model after preprocessing.

5. A method for extracting roads from remote sensing images based on a multi-dimensional multi-scale U-net network according to claim 4, characterized in that, The model loss is calculated using a composite loss function consisting of the cross entropy loss function and the Dice loss function. The cross entropy loss function and the Dice loss function are defined as follows: where N represents the total number of pixels, and g i represents the true label value of pixel i, and p i represents the predicted value of pixel i; The composite loss function is defined as follows: L=L BCE +L D .

Images