A high-resolution remote sensing image road extraction method

Abstract

The application discloses a high-resolution remote sensing image road extraction method, which combines the shape rule and scale difference of the road to construct a double-task-driven deep convolutional neural network, wherein a main line task is road surface segmentation RSS, the encoding layer of which adopts residual convolution for feature extraction, and the decoding layer is composed of a multi-scale and multi-direction strip convolution module, and the output of the task line is the final extraction result in the paper. A branch task is road center extraction RCE, the input feature of which comes from the encoding layer of the RSS branch, and the intermediate feature will be integrated into the decoding layer of the RSS branch, so that the connectivity of the road surface segmentation result is improved by fully utilizing the road center line. In the experimental part, it is proved by means of the public data set that the method of the application is higher than the comparative method in the quantitative evaluation index, has obvious advantages in the heavy and difficult points such as road covering, road intersection and low-grade road, and has stronger model transplantation and small sample learning ability.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of remote sensing image processing, and particularly relates to a high-resolution remote sensing image road extraction method. BACKGROUND

[0002] Intelligent road extraction technology plays an important role in map updating, automatic driving, city planning, vehicle navigation and other fields. As a product of non-contact information acquisition technology, remote sensing images can make it possible for people to obtain detailed ground information in a short time with the continuous improvement of their spatial resolution and time resolution, so that the road network can be displayed in the visual image. Therefore, remote sensing images can be used as an effective data basis for automatic and real-time identification of road information.

[0003] The types of remote sensing image road extraction methods can be divided into traditional methods and deep learning methods according to their development history. The traditional methods mainly include template matching method, knowledge-driven method and object-oriented method. These methods mainly rely on the shape features, spectral features, texture features and artificial shallow combined features of the roads in remote sensing images. However, with the continuous improvement of the spatial resolution of remote sensing images, their ability to express ground details has been significantly improved. The problem of "same object with different spectrum and same spectrum with different object" is increasingly prominent. The roads are not enough to gather in the shallow feature space and intersect with other ground objects, resulting in poor applicability and stability of the methods.

[0004] In recent years, with the development of deep convolutional neural networks, especially the birth of typical semantic segmentation networks such as FCN, UNet, SegNet, Deeplab series, deep learning methods have been widely used in remote sensing image pixel-level intelligent interpretation. However, it is challenging to extract roads from remote sensing images. Firstly, there are obvious differences in road levels, and small and large targets coexist. Secondly, buildings, trees and other objects form obstacles to the road surface. Thirdly, the similarity between roads and other targets (empty land, ditch, etc.) is a problem. The above difficulties often cause errors, omissions and fragmentation of road extraction results. Recently, researchers have improved existing methods by introducing attention mechanisms, improving network structures, and optimizing supervision forms based on the typical "encoding-decoding" structure and the image features of roads and the application requirements of the extraction results.As in document 1 (Zhou L, Zhang C, Wu M. D-LinkNet: LinkNet with pretrained encoder and dilated convolution for high resolution satellite imagery road extraction [C] / / Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops. 2018: 182-186.), document 2 (Wei Y, Zhang K, Ji S. Simultaneous road surface and centerline extraction from large-scale remote sensing images using CNN-based segmentation and tracing [J]. IEEE Transactions on Geoscience and Remote Sensing, 2020, 58(12): 8919-8931.) added multi-scale dilated convolution between the encoding layer and the decoding layer, and document 3 (Zhou G, Chen W, Gui Q, et al. Split depth-wise separable graph-convolution network for road extraction in complex environments from high-resolution remote-sensing Images [J]. IEEE Transactions on Geoscience and Remote Sensing, 2021, 60: 1-15.) introduced a graph neural network into the convolutional neural network, thereby making up for the defects of the convolutional neural network in the ability to capture global features. Document 4 (Wang Y, Peng Y, Li W, et al. DDU-Net: dual-decoder-U-Net for road extraction using high-resolution remote sensing images [J]. IEEE Transactions on Geoscience and Remote Sensing, 2022, 60: 1-12.) added a decoding branch to the conventional framework to form a double-decoding structure, thereby enhancing the ability to extract detailed information.Document 5 (Zhou M, Sui H, Chen S, et al. BT-RoadNet: A boundary and topologically-aware neural network for road extraction from high-resolution remote sensing imagery [J]. ISPRS Journal of Photogrammetry and Remote Sensing, 2020, 168: 288-306.) builds a rough-to-fine extraction framework to correct the missing extraction and false extraction in the rough extraction process through fine extraction. The above method studies the improvement of the accuracy and stability of the road extraction result from different angles, but does not take into account the shape rule, scale difference and connectivity of the road on the high-resolution remote sensing image at the same time. SUMMARY

[0005] The present application aims at the problem that the existing method studies the improvement of the accuracy and stability of the road extraction result from different angles, but does not take into account the shape rule, scale difference and connectivity of the road on the high-resolution remote sensing image at the same time, and proposes a high-resolution remote sensing image road extraction method.

[0006] In order to achieve the above purpose, the present application adopts the following technical scheme:

[0007] A high-resolution remote sensing image road extraction method comprises:

[0008] A double-task-driven deep convolutional neural network is constructed combining the shape rule and scale difference of the road, and the double-task-driven deep convolutional neural network comprises a road surface segmentation network branch and a road center extraction network branch; the road surface segmentation network branch comprises an encoding layer, an intermediate layer and a decoding layer; the encoding layer adopts residual convolution for feature extraction, and the decoding layer is composed of a multi-scale multi-directional strip convolution module; the input data of the network branch is the original remote sensing image, and the output is the final road extraction result; the input feature of the road center extraction network branch comes from the encoding layer of the road surface segmentation network branch, and the obtained intermediate feature is integrated into the decoding layer of the road surface segmentation network branch, so as to improve the connectivity of the road surface segmentation result by using the road center line;

[0009] High-resolution remote sensing image road extraction is performed based on the constructed double-task-driven deep convolutional neural network.

[0010] Further, in the encoding layer, ResNet34 is adopted as a feature extractor.

[0011] Further, the encoding layer adopts residual convolution for feature extraction, which comprises:

[0012] Firstly, shallow feature extraction is performed by a 7x7 convolution kernel and a 3x3 max pooling layer; then deep feature mining is performed by four residual convolution blocks with a double 3x3 convolution kernel number of 3, 4, 6, and 3; and finally the size of the output feature map is 1 / 32 of the original image and the channel number is 512.

[0013] Further, the decoding layer comprises four multi-scale multi-directional strip convolution modules, and the output features of the four multi-scale multi-directional strip convolution modules are added to the corresponding feature maps of the encoding layer, and the output features of the last multi-scale multi-directional strip convolution module are fused with the intermediate results of the road center extraction network branch in the form of channel stacking.

[0014] Further, the four multi-scale multi-directional strip convolution modules use strip convolution kernels in four directions of 0°, 45°, 90°, and 135° for line feature extraction, and use channel stacking, bilinear upsampling, 1x1 convolution, and channel stacking operations in sequence to complete the fusion of the feature extraction results in each direction.

[0015] Further, the fusion is performed in the following manner:

[0016]

[0017] In the above formula, X and Y represent input and output features respectively, and Concat is a channel stacking operation, is a linear convolution kernel of scale i, k represents the number of scales involved, and * represents a convolution operation.

[0018] Further, the data processing procedure of the road center extraction network branch comprises:

[0019] The multi-scale features in the encoding layer of the road surface segmentation network branch are taken as input data, and then 3x3 convolution and upsampling are used in sequence to unify the channels and scales, and on this basis, the channel stacking of the features in each scale is performed, the size of the stacked and fused feature map is 256x256, and the channel number is 64. The obtained feature map is further processed in two paths, one is to stack the fusion result to the decoding layer of the road surface segmentation network branch to improve the connectivity of the road surface segmentation result, and the other is to obtain a road center line prediction result consistent with the size of the original image by upsampling, convolution, and sigmoid activation operation.

[0020] Further, it further comprises:

[0021] The constructed dual-task-driven deep convolutional neural network is trained.

[0022] Further, when the dual-task-driven deep convolutional neural network is trained, the following loss function is used:

[0023] Loss=Loss seg +Loss cen (2)

[0024] where Loss, Loss seg , Loss cen respectively represent the total loss, the road surface segmentation network branch loss and the road center extraction network branch loss; the sum of the BCE loss and the dice coefficient loss is taken as the loss of each branch.

[0025] Compared with the prior art, the present application has the beneficial effects of:

[0026] (1) A multi-scale multi-directional strip convolution module is proposed for the strip characteristics of road shapes and the width differences of different grades of roads, and the module is used as a basic unit of the decoding part, which uses the strip convolution kernel to capture features of different scales in the specified direction, thereby weakening the interference of non-coherent information and enhancing the extraction ability of multi-grade roads.

[0027] (2) Considering that the road center line is a direct manifestation of road connectivity, under the premise of taking the road surface segmentation network branch as the basic framework, the road center extraction network branch is taken as a supplement, wherein the input features of the road center extraction network branch come from different levels of the encoding layer of the road surface segmentation network branch, and the intermediate features will be integrated into the decoding layer of the road surface segmentation network branch, thereby increasing the attention of the network model to road connectivity. BRIEF DESCRIPTION OF DRAWINGS

[0028] Figure 1 A high-resolution remote sensing image road extraction method framework flowchart of an embodiment of the present application;

[0029] Figure 2 A dual-task-driven deep convolutional neural network architecture diagram constructed by an embodiment of the present application;

[0030] Figure 3 A multi-scale multi-directional strip convolution module architecture diagram constructed by an embodiment of the present application;

[0031] Figure 4 A connectivity evaluation diagram of an embodiment of the present application; wherein (a) and (b) correspond to the Com and Eor evaluation indexes respectively;

[0032] Figure 5 Different method road surface segmentation result examples (CHN6-CUG data set);

[0033] Figure 6 Different method road surface segmentation result examples (DeepGlobe data set);

[0034] Figure 7 Visualize the intermediate features of the network;

[0035] Figure 8 The results are for the model portability experiment; (a) is a comparison chart of the accuracy of model portability experiment 1, and (b) is a comparison chart of the accuracy of model portability experiment 2.

[0036] Figure 9 These are statistical charts for different sample sizes; where (a), (b), (c), and (d) correspond to the F1, IoU, Com, and Eor charts, respectively. Detailed Implementation

[0037] The present invention will be further explained below with reference to the accompanying drawings and specific embodiments:

[0038] like Figure 1 As shown, a method for road extraction from high-resolution remote sensing imagery includes:

[0039] A dual-task driven deep convolutional neural network is constructed by combining the shape regularity and scale differences of roads. The dual-task driven deep convolutional neural network includes a road surface segmentation network branch and a road center extraction network branch. The road surface segmentation network branch includes an encoding layer, an intermediate layer, and a decoding layer. The encoding layer uses residual convolution for feature extraction, and the decoding layer is composed of multi-scale, multi-directional strip convolutional modules. The input data of this network branch is the original remote sensing image, and the output is the final road extraction result. The input features of the road center extraction network branch come from the encoding layer of the road surface segmentation network branch, and the obtained intermediate features are integrated into the decoding layer of the road surface segmentation network branch to improve the connectivity of the road surface segmentation result by utilizing the road centerline.

[0040] Road extraction from high-resolution remote sensing images is performed using a constructed dual-task-driven deep convolutional neural network.

[0041] Specifically, the method framework takes road surface segmentation (RSS) and road centerline extraction (RCE) as two branches, respectively using corresponding label data for synchronous supervised learning, wherein the RSS branch is the basis, and the output data is the final extraction result of the application, and the RCE branch is a supplement, and the input features come from the encoding layer of the RSS branch, and the intermediate features are transmitted to the decoding layer of the RSS branch to improve the connectivity of the road extraction result, and the output data is the road centerline (auxiliary result, not the final result). In addition, for the shape regularity and scale difference of the road, a multi-scale and multi-direction strip convolution module (MSMD-SCM) is proposed to enhance the feature capture capability of the road in the specified direction and multiple scales.

[0042] 1Network structure

[0043] The network structure of the method of the application develops in the form of double tasks combined with main lines and branch lines, wherein the RSS is the main line, and the RCE is the branch line, and in the training process, supervised training is respectively performed according to the respective label data, so that the parameters on the two task lines are constantly updated in the backward propagation process, and the road prediction capability of the network model is gradually improved, and the detailed structure of the network model is as shown in Figure 2 .

[0044] 1.1RSS branch network structure

[0045] The branch mainly includes an encoding layer, an intermediate layer and a decoding layer, the input data is an original image, and the output data is a prediction result of road surface segmentation (i.e. the final road extraction result). In the encoding layer, ResNet34 is introduced as a feature extractor, that is, shallow feature extraction is first performed through a 7*7 convolution kernel and a 3*3 max pooling layer; then four residual convolution blocks with double 3*3 convolution kernels with quantities of 3, 4, 6 and 3 are used for deep feature mining; and finally the size of the feature map is 1 / 32 of the original image, and the channel number is 512. In the decoding layer, four MSMD-SCMs are mainly included, which are used for line feature extraction of roads of different levels and upsample the feature maps to a suitable size. At the same time, in order to alleviate the information loss problem in the up-sampling and down-sampling process, the output features of the four MSMD-SCMs are all added to the corresponding feature maps of the encoding layer. In addition, in order to improve the connectivity of the road extraction result, the output features of the last MSMD-SCM are fused with the intermediate result of the RCE in the form of channel superposition. The last stage of the decoding layer is to obtain the road surface segmentation prediction result consistent with the size of the original image by using up-sampling, convolution and sigmoid activation operations.

[0046] 1.2RCE branch network structure

[0047] The road centerline can reflect the topology of the road in a vector form, and has a direct promoting effect on improving the connectivity of the road. Therefore, based on the RSS branch, an RCE branch is introduced as a supplement to improve the connectivity of the road surface segmentation result. The structure of the RCE branch is relatively simple compared with the RSS branch, and the output data is the prediction result of the road centerline. The corresponding true value is obtained by morphological thinning processing of the label data of the road surface segmentation result. Because the road centerline and the road surface have certain similarity in the topology structure, and considering the complexity of the whole method, the multi-scale features in the RSS encoding layer are directly used as the input data of the branch. Then, 3x3 convolution and up-sampling are used to unify the channels and scales, and on this basis, the channel stacking of the feature maps of each scale is performed. The size of the stacked and fused feature map is 256x256, and the number of channels is 64. After the above processing, the obtained feature map is continued to be pushed forward in two ways. One is to stack the fusion result to the RSS decoding layer to improve the connectivity of the road surface segmentation result, and the other is to obtain the road centerline prediction result consistent with the size of the original image by up-sampling, convolution, sigmoid activation and other operations.

[0048] 1.3Multi-scale and multi-directional strip convolution module

[0049] Roads have strict grade standards in the transportation department and the surveying and mapping department. Different grades often correspond to different widths, which are manifested as different scales on remote sensing images. Therefore, when performing road surface segmentation, which is a dense prediction task, targets with different scales will be faced. Using a fixed-size window for convolution often cannot take into account the scale difference of the target. In addition, unlike building, vegetation or lake and other terrain features, the shape of the road often presents a strip shape. Therefore, using a traditional square convolution kernel will inevitably capture more irrelevant information, while a strip convolution kernel can extract features along a specified direction, and its attention range is more consistent with the shape rule of the road itself.

[0050] Based on the above analysis, the present application proposes MSMD-SCM based on the strip convolution module (SCM), and the specific structure is as shown in Figure 3 That is, the strip convolution kernels in 0°, 45°, 90° and 135° directions are used for line feature extraction respectively, and channel stacking, bilinear up-sampling, 1x1 convolution and channel stacking are used in sequence to complete the fusion of the feature extraction results of each direction. Among them, the strip convolution of each direction includes multiple scales, and the specific multi-scale fusion form is shown in formula (1):

[0051]

[0052] where X and Y represent input and output features respectively, Concat is the channel concatenation operation, is the linear convolution kernel for scale i, k corresponds to the number of scales involved, and * denotes the convolution operation.

[0053] 1.4 Loss function

[0054] The deep learning network of the present application involves two branches, which respectively utilize road surface segmentation labels and road centerline labels for loss function calculation, and the overall loss function is shown in equation (2)

[0055] Loss = Loss seg + Loss cen (2)

[0056] where Loss, Loss seg , Loss cen represent the overall loss, the RSS branch loss and the RCE branch loss respectively.

[0057] Considering that both branches of the label are binary classification labels, and there is an imbalance problem between positive and negative samples, the sum of the BCE loss and the dice coefficient loss is used as the loss of each branch. The BCE loss treats each pixel equally, and when there are fewer positive samples, the network will mainly use negative samples, resulting in a decrease in the recognition ability of positive samples. The dice coefficient loss focuses on the information mining of positive samples (foreground area), which can better solve the imbalance problem between positive and negative samples, but the training loss is unstable, so the combination of the two losses can achieve good results. The calculation formulas of the BCE loss and the dice coefficient loss are shown in equations (3) and (4).

[0058]

[0059]

[0060] where P and Y represent the prediction result and the label data respectively, W and H are the width and height of the image, and the prediction and label of the image (i, j) position are p ij and y ij .

[0061] To verify the effect of the present application, the following experiments are performed:

[0062] 2. Experiment:

[0063] 2.1 Experimental data

[0064] (1) CHN6-CUG dataset: This dataset is from Google Earth, including highways, urban roads and rural roads in Beijing, Wuhan, Shenzhen, Shanghai, Hong Kong, Macau and other regions. The dataset contains 3608 training images and 903 test images, with a spatial resolution of 0.5m and a size of 512x512.

[0065] (2) DeepGlobe dataset: This dataset includes urban, suburban and rural areas in Thailand, India and Indonesia, with a total of 6226 labeled images, a spatial resolution of 0.5 meters, and a size of 1024x1024. To improve the training efficiency of the model, we perform synchronous division operations in the width and height directions on the original image and corresponding label data to generate a dataset with a size of 512x512, and divide the training data and test data according to a ratio of 3:1, finally obtaining 18784 training data and 6120 test data.

[0066] 2.2 Evaluation index

[0067] (1) Pixel-level evaluation index

[0068] In order to comprehensively evaluate the method at the pixel level, the evaluation index selects the accuracy (P), recall (R), F1 score, overall accuracy (OA) and intersection over union (IoU), and the specific calculation formula is as follows:

[0069]

[0070]

[0071]

[0072] In the formula, TP, FP, TN and FN represent the number of correctly extracted positive samples, incorrectly extracted positive samples, correctly extracted negative samples and incorrectly extracted negative samples, respectively.

[0073] (2) Connectivity evaluation index

[0074] In order to verify the connectivity of the road extraction result, the present application designs two evaluation indexes, completeness (Com) and error rate (Eor), to measure the specific, as shown in Figure 4 Figure 4 (a) The medium dark tone buffer is the prediction result, and the light tone and red line segment are the morphological thinning results of the label data, wherein the light tone line segment is located in the prediction result buffer with a length of l1, and the red line segment is located outside the prediction result buffer with a length of l2; Figure 4 ​(b) The light color buffer is the label data, the dark color and blue line segments are the morphological thinning results of the prediction results, wherein the dark color line segment is located in the label data buffer and has a length of l3, and the blue line segment is located outside the label data buffer and has a length of l4. The specific calculation formulas of Com and Eor are as follows:

[0075]

[0076] 2.3 Experimental details

[0077] The core hardware configuration of the experimental environment is 2 NVIDIA Tesla V100 graphics cards with a total of 64G of video memory. Adam is selected as the optimizer for network training, the initial learning rate is set to 2e-4, the learning rate is reduced by 5 times whenever the loss value is higher than the current optimal loss value for 3 consecutive times, the data block size for training is 32, and the iteration epoch value is 150. At the same time, in order to enhance the samples, the training data is randomly (50%) flipped vertically, horizontally, diagonally, and radiated.

[0078] 2.4 Experimental results

[0079] 2.4.1. Method comparison experiment

[0080] This part of the experiment mainly uses the above two experimental data sets to compare and analyze the extraction results and accuracy of the method of the application and typical methods (UNet, D-LinkNet (literature 1), Deeplabv3+).

[0081] Figure 5are the road surface segmentation results of part of the test images in the CHN6-CUG dataset, five images are selected from different cities, different scenes, and basically cover the difficult areas of road extraction, and the comparative analysis has certain representativeness. The right lower corner of image (a) is located in a region with lush vegetation, the overall road width is relatively narrow, and there is a vegetation covering phenomenon on part of the road section. From the extraction result, U-Net and D-LinkNet have a missing extraction problem for the small road section, DeepLabv3+ extracts a small section but the road extraction result is obviously incomplete due to the vegetation covering, the method of the application basically extracts the road section completely, but the nearby part of the empty land is misidentified as a road problem to be improved in the next step; image (b) is an image of a high-rise residential area in the city, the road has a lot of shadow covering phenomenon, and the shadow color is deep, causing the spectral characteristics of the road surface itself not to correspond to the actual ground objects, such as the east-west oriented road in the figure is severely covered by the building shadow, the completeness of the three comparative methods for extracting the road section is seriously insufficient, the method of the application is better adapted to the shadow covering phenomenon, and the missing extraction problem is significantly improved; image (c) is an image containing multiple road intersections, part of the road section has a large width, and there are ground objects such as parking lots with similar spectral characteristics to roads. From the extraction result, except for U-Net and the method of the application, there is basically no mis-extraction problem, but the missing extraction problem of the comparative method is obviously more serious than the method of the application; in image (d), part of the east-west oriented road section is shadowed and has obvious moving carrier (vehicle) interference, the extraction effect of UNet is almost blank in this road section, D-LinkNet and DeepLabv3+ have slightly better effect, but there are still continuous missing extraction parts, causing errors in road connectivity, while the method of the application restores the actual situation of the road completely, and the effect is the best; image (f) is affected by factors such as building shadow, shooting environment, and containing water area and large proportion of vegetation, so that the color of the image is dark, UNet and DeepLabv3+ have serious missing extraction problems, UNet and D-LinkNet have serious mis-extraction problems, and the comprehensive effect of the method of the application is the best.

[0082] Figure 6 are the surface segmentation results of part of the test images in the DeepGlobe dataset, and Figure 5Similarly, in order to make the comparative analysis representative, the extraction effects of each method are discussed in detail for the current road extraction difficulties. Image (a) is an image of farmland area, which contains a rural road with a small width and a road segment covered with vegetation. The road mining effects of D-LinkNet and Deeplabv3+ on this image are almost zero, UNet extracts most of the road segments but the extraction result is incomplete due to the influence of vegetation coverage, and the method of the present application overcomes the above difficulties, and the completeness and accuracy of the extraction result are better; the spectral features of the square open space in image (b) are basically the same as those of the road, and part of the road has only a small section on the image. From the final extraction result, each method has different degrees of missing extraction problems, the method of the present application is relatively good, but the extraction ability of small roads with no outstanding overall length still needs to be improved; image (c) is an image of farmland area, the overall tone of the image is dark, but the tone of the road with north-south trend is high, which is quite different from other roads in the area. UNet determines this road as a background area, D-LinkNet and Deeplabv3+ recognize part of the road but the missing extraction problem is very prominent, and the method of the present application has high completeness of the extraction result and no obvious mis-extraction problem; image (d) contains two roads with bidirectional intersection. Due to the existence of the isolation belt, the "cross" intersection is actually a combination of "L" type intersection and "T" type intersection. From the final result, three comparison methods cannot correctly restore the actual situation of the road intersection, and the method of the present application effectively distinguishes the road, isolation belt and other ground objects, and obtains a good extraction result; image (f) is an image of dense residential area, and the road network environment has high complexity. Buildings, vegetation, shadows and even mobile carriers all form obstacles on the road surface, which brings difficulties to the correct extraction of the road. From the extraction result, the extraction results of the four methods all have a certain degree of mis-extraction, but the method of the present application has the best effect on the recovery of road network connectivity, and the other methods all have the problem of missing extraction of the main road.

[0083] In addition to the above extraction effect analysis of typical example images based on road extraction difficulties, in order to further comprehensively quantify the extraction ability of each method, 7 evaluation indexes are used for specific analysis. The values of each evaluation index are the average values of all test images in CHN6-CUG data set and DeepGlobe data set.

[0084] P, R, F1, OA, IoU are pixel-level precision evaluation for road extraction results, the larger the P value represents the higher the accuracy of the extracted results, the larger the R value represents the larger the proportion of the real road extracted, and F1, OA, IoU are comprehensive evaluation indexes combined with the extraction of positive and negative samples. Com, Eor are connectivity evaluation indexes for road extraction results, the larger the Com value represents the higher the completeness of the extracted road connectivity, and the smaller the Eor value represents the lower the error rate of the extracted road connectivity.

[0085] In summary, the following conclusions can be drawn in combination with Table 1: (1) Compared with UNet, D-LinkNet and Deeplabv3+, the method of the present application is optimal in each index in the two road data sets; (2) Compared with the DeepGlobe data set, the method of the present application has more obvious advantages in the CHN6-CUG data set, which proves that the road extraction capability of the method of the present application is more prominent in the case of more complex environment and smaller sample library; (3) In the two data sets, the method of the present application has more advantages in the R and Com evaluation indexes than in the P and Eor evaluation indexes, which proves that the road extraction completeness of the method of the present application is obviously better than that of the comparative methods, which is also consistent with the final effect of the road extraction difficulty in Figure 5 、 Figure 6 .

[0086] Table 1 Precision comparison of different methods of road surface segmentation results

[0087]

[0088] Note: bold underlined indicates optimal index

[0089] 2.4.2. Ablation experiment

[0090] This part takes the CHN6-CUG data set as an example to conduct an ablation experiment on the double-task form and MSMD-SCM designed by the present application, and the specific cases mainly include the following four kinds, wherein case 1 and case 3 only contain the RSS branch, and case 1 replaces the MSMD-SCM with a 3*3 convolution kernel in the decoding process, case 2 adds the RCE branch on the basis of case 1, and case 4 is the method of the present application. The specific precision statistics of the four cases are shown in Table 2.

[0091] It can be seen from Table 2 that the RCE branch and the MSMD-SCM have obvious promoting effects on the road extraction results. Specifically, by comparing scenario 1 and scenario 3, and scenario 2 and scenario 4, it can be found that the addition of the MSMD-SCM improves the indicators in the pixel-level evaluation, and the improvement of R is particularly significant, proving that the module can more fully and completely extract road information and reduce the omission problem in road extraction, which also makes the Com in the connectivity evaluation index significantly improve. By comparing scenario 1 and scenario 2, and scenario 3 and scenario 4, it can be found that the addition of the RCE branch makes most of the indicators have certain progress (except P and Eor in scenario 3 and scenario 4), and the Com in the connectivity evaluation index improves significantly, proving that the RCE branch can improve the connectivity of the road extraction results and effectively suppress the mis-extraction problem of negative samples.

[0092] Table 2 Ablation experiment precision statistics table

[0093]

[0094] In addition to the above precision analysis, in order to more intuitively express the role of MSMD-SCM and RCE in road extraction, the output features of the first four modules (block1, block2, block3, block4) in the decoding process are visualized and analyzed, wherein Baseline represents the basic network framework of the method of the present application, but does not contain MSMD-SCM and RCE, +MSMD-SCM and +RCE respectively represent the addition of the corresponding module or branch. By comparing the visualized results of block1-block4 in the three conditions, it can be found that the visualized results of block1-block4 in the three conditions are more and more close to the actual situation of the road. In addition, after adding the MSMD-SCM, the roads with north-south trend are more prominent, and the small planar buildings around are inhibited to a certain extent in block4. After further adding the RCE, it can be found through the comparison of block3 that the separation speed of road features and other ground features is significantly accelerated, and the high-light features in block4 are basically only roads, further excluding the interference of other ground objects. In summary, MSMD-SCM and RCE can help to improve the efficiency and accuracy of separating roads and other ground objects in the feature space, thereby optimizing the final road extraction results. Figure 7 It can be found that in the calculation process of block1-block4, the visualized results in the three conditions are more and more close to the actual situation of the road. In addition, after adding the MSMD-SCM, the roads with north-south trend are more prominent, and the small planar buildings around are inhibited to a certain extent in block4. After further adding the RCE, it can be found through the comparison of block3 that the separation speed of road features and other ground features is significantly accelerated, and the high-light features in block4 are basically only roads, further excluding the interference of other ground objects. In summary, MSMD-SCM and RCE can help to improve the efficiency and accuracy of separating roads and other ground objects in the feature space, thereby optimizing the final road extraction results.

[0095] 3 DISCUSSION

[0096] 3.1 Model portability discussion

[0097] The current sample demand for manual labor is the main factor restricting the full application of deep learning, while the strong portability of network models provides a basic learning framework for transfer learning, thereby reducing the dependence on samples and improving the reliability of "cross-domain supervision". Therefore, the portability of each method is analyzed in this section. The training data in the data is the DeepGlobe dataset, the validation set and test set in the Massachusetts road dataset are the test data in experiment 1, and the test set in the CHN6-CUG dataset is the test data in experiment 2. Considering that the ground features and background characteristics of the training data and test data are quite different, the test results can be used as an evaluation standard for the portability of the model. In order to intuitively and comprehensively evaluate the portability of the model, the comprehensive pixel-level evaluation indicators F1 and IoU, and the connectivity evaluation indicators Com and Eor are selected for analysis, and the specific results are shown in Figure 8

[0098] As can be seen from Figure 8 , in the pixel-level evaluation indicators, the method of the present application obtains the optimal result in experiment 1 and experiment 2; in the connectivity evaluation indicators, the method of the present application is optimal in the Eor indicator, that is, the case of misidentifying other ground objects as roads is the least, the Com indicator in experiment 2 is optimal, but it is slightly lower than D-LinkNet and Deeplabv3+ in experiment 1, which may be related to the low detail presentation of ground objects in the Massachusetts road dataset (low spatial resolution) and the relatively concentrated regional range. In summary, the portability of the method of the present application is the strongest when there is a significant difference between the training data and the test data, and the deep learning network designed by the method can provide a more reliable and more generalizable model framework for transfer learning.

[0099] 3.2 Training set small sample discussion

[0100] ​Similar to the model portability discussion, small sample as a form of weakly supervised learning can effectively alleviate the demand for samples of deep learning, thereby improving the automation and intelligence level of the whole process, and providing a basic network framework and design idea for special weakly supervised learning methods. In this subsection, the accuracy and stability of each method under different sample sizes are analyzed with the DeepGlobe dataset as an example. OTS (Original Training Sets) indicates that the training data and test data involved in the experiment are consistent with 2.4.1, 8000, 6000, 4000, 2000, 1000 represent the number of samples randomly extracted from the OTS training data (the same sample number is extracted from the training data, and the test data is the same as OTS). Since the experimental variables of this analysis only involve methods and sample sizes, the road extraction capability of each method under different sample sizes can be directly measured. The evaluation indicators selected are consistent with 3.1, and the specific results are shown in Table 2. Figure 9

[0101] As can be seen from Figure 9 , the F1, IoU, Com three evaluation indicators gradually decrease with the decrease of the sample size, and the Eor increases with the decrease of the sample size. In terms of the degree of change, UNet has the largest change in the four evaluation indicators, and the method of the present application has the most gentle change, so the method of the present application has the greatest ability to maintain road extraction efficiency as the sample size decreases. In addition, from OTS to sample sizes of 8000, 6000, 4000, 2000, 1000, the gap between the method of the present application and the three comparison methods in the four evaluation indicators basically presents an expanding trend, especially when the sample size is 1000, the method of the present application has obvious advantages. In summary, the extraction capability of the method of the present application is more prominent than the comparison methods under the condition of small sample, and has better application prospect in terms of both precision and efficiency.

[0102] Considering that the overall idea of the present application adopts a double-task form, the efficiency under the same training condition is not outstanding, but combined with the above small sample analysis experimental results, when the extraction results with approximately the same precision are obtained, the efficiency of the method of the present application has certain advantages. As shown in Table 3, Num represents the number of samples in the training set, and Time represents the average training time of each epoch under the corresponding training set (the hardware environment and parameter settings are consistent). From Table 3, it can be seen that the precision obtained by the four methods selected has basically the same pixel-level evaluation indicators (the method of the present application is relatively the highest, and UNet is poor in connectivity evaluation indicators), but in terms of training time, the training time of the method of the present application is basically the same as that of D-LinkNet, but is better than that of UNet and Deeplabv3+. Therefore, in summary, the method of the present application can balance precision and efficiency, and has better practical value. ​

[0103] Table 3 training efficiency comparison table

[0104]

[0105] 4 Conclusion

[0106] Road as an important topographic feature, its shape, size has its own regularity, the above-mentioned law also helps in high-resolution remote sensing image in-depth, accurate extraction of road image features; In addition, as the basic framework of transportation facilities, the extraction result of the connected relationship constructed will directly reflect the topological structure of the traffic network, and its transportation function has an important role in practical application. The method of the present application carries out targeted research on road extraction from the above three aspects, taking the end-to-end network as the basic framework, by increasing the multi-scale multi-direction feature extraction module, improving the extraction strategy, etc. A double task driven road extraction method is proposed.

[0107] (1) Aiming at the strip characteristics of road shape and the width difference of different grades of roads, a multi-scale multi-direction strip convolution module is proposed, and the module is used as the basic unit of the decoding part. The strip convolution kernel is used to capture features of different scales in the specified direction, so as to weaken the interference of non-coherent information and enhance the extraction ability of multi-grade roads.

[0108] (2) Considering that the road centerline is a direct manifestation of road connectivity, under the premise of taking RSS branch as the basic framework, RCE branch is taken as a supplement. The input features of RCE branch come from different levels of the encoding layer of RSS branch. The intermediate features will be integrated into the decoding layer of RSS branch, so as to increase the attention of the network model to the road connectivity.

[0109] (3) Considering the development bottleneck of large proportion of artificial intervention in deep learning sample collection, the generalization ability and practical value of the method of the present application are discussed. In addition to the traditional precision comparison and ablation experiment, the portability of the method and the road extraction ability in the small sample environment are analyzed in detail, and the potential of the method in practical application is discussed.

[0110] The above only shows the preferred embodiments of the present application, and it should be noted that for ordinary skilled in the art, without departing from the principles of the present application, a number of improvements and refinements can be made, which improvements and refinements should also be considered as the protection scope of the present application.

Claims

1. A high-resolution remote sensing image road extraction method, characterized in that, The application relates to a method for extracting a road from a high-resolution remote sensing image. The method comprises the following steps: A deep convolutional neural network driven by double tasks is constructed according to the shape rule and size difference of a road, wherein the deep convolutional neural network driven by double tasks comprises a road surface segmentation network branch and a road center extraction network branch; the road surface segmentation network branch comprises an encoding layer, an intermediate layer and a decoding layer; the encoding layer adopts residual convolution for feature extraction, and the decoding layer is composed of multi-scale and multi-directional strip convolution modules; the input data of the network branch is an original remote sensing image, and the output is a final road extraction result; the input feature of the road center extraction network branch comes from the encoding layer of the road surface segmentation network branch, and the obtained intermediate feature is fused into the decoding layer of the road surface segmentation network branch, so that the connectivity of the road surface segmentation result is improved by using the road center line; The method comprises the following steps: The decoding layer comprises four multi-scale and multi-directional strip convolution modules, the output features of the four multi-scale and multi-directional strip convolution modules are added to the corresponding feature maps of the encoding layer, and the output feature of the last multi-scale and multi-directional strip convolution module is fused with the intermediate result of the road center extraction network branch in the form of channel superposition; The data processing process of the road center extraction network branch comprises the following steps: 2.The method of claim 1, wherein, The multi-scale features in the encoding layer of the road surface segmentation network branch are taken as input data, then 3*3 convolution and up-sampling are used to unify the channels and scales, and on this basis, the channel superposition of the feature maps of various scales is carried out; the size of the superposition and fusion feature map is 256*256, the number of channels is 64, the obtained feature map is continuously pushed forward in a double-line mode, one is that the fusion result is superimposed to the decoding layer of the road surface segmentation network branch to improve the connectivity of the road surface segmentation result, and the other is that the fusion result is subjected to up-sampling, convolution and sigmoid activation operation to obtain a road center line prediction result which is consistent with the size of the original image. 3.The method of claim 2, wherein, In the encoding layer, ResNet34 is used as a feature extractor. The encoding layer adopts residual convolution for feature extraction, which comprises the following steps:

4. The method of claim 1, wherein the high-resolution remote sensing image is a satellite image. Firstly, shallow layer feature extraction is carried out through a 7*7 convolution kernel and a 3*3 maximum pooling layer; then deep layer feature mining is carried out through four residual convolution blocks with a double-layer 3*3 convolution kernel and a number of 3, 4, 6 and 3; finally, the size of the output feature map is 1 / 32 of the original image, and the number of channels is 512.

5. The method of claim 4, wherein the road extraction method is a high-resolution remote sensing image road extraction method. The four multi-scale and multi-directional strip convolution modules utilize strip convolution kernels in four directions of 0 DEG, 45 DEG, 90 DEG and 135 DEG for line feature extraction, and then utilize channel superposition, bilinear up-sampling, 1*1 convolution and channel superposition operation to complete fusion of the feature extraction results in various directions. In the above equation, X and Y represent input and output features, respectively, and Concat is a channel concatenation operation. is a linear convolution kernel of size i, k represents the number of scales involved, and * denotes a convolution operation.

6. The method of claim 1, wherein the method further comprises: The fusion is carried out in the following manner: The application further relates to a method for training the deep convolutional neural network driven by double tasks.

7. The method of claim 6, wherein the road extraction method is a high-resolution remote sensing image road extraction method. When the deep convolutional neural network driven by double tasks is trained, the following loss function is adopted: Loss = Loss seg + Loss cen (2) where Loss, Loss seg , Loss cen denote the total loss, the road surface segmentation network branch loss and the road center extraction network branch loss, respectively; the sum of the BCE loss and the dice coefficient loss is taken as the loss of each branch.

Images