Remote sensing image road extraction method

Abstract

The invention discloses a remote sensing image road extraction method. The method comprises the following steps: acquiring a remote sensing road image and cutting to obtain a region of interest; utilizing a backbone network of an improved baseline model RNGDet to extract depth features of the region of interest, and obtaining a first feature tensor; based on the first feature tensor, performing road segmentation prediction by using an improved feature pyramid network structure to obtain two distribution diagrams; cutting and rasterization are carried out after a historical track graph is obtained, and a rasterization graph is obtained; inputting the two distribution maps and the rasterization map into a backbone network to obtain a second feature tensor; and carrying out splicing operation on the first feature tensor and the second feature tensor to obtain a target feature tensor, sending the target feature tensor as a feature extraction result of the backbone network into a Transform structure for prediction, and updating the road network map according to a prediction result. According to the method, the robustness and the accuracy of a road extraction result on small-scale objects and complex scenes are improved.

Description

Technical Field

[0001] The invention belongs to the technical field of road extraction, and in particular relates to a remote sensing image road extraction method. Background Art

[0002] As an important means of obtaining road information, road extraction from remote sensing images has broad application prospects. With the development of deep learning technology and the increasing abundance of remote sensing image data, road extraction methods based on deep learning have gradually become an important research direction for road extraction from remote sensing images. Road extraction methods based on deep learning can be divided into two categories: segmentation-based methods and graph generation-based methods. The former can accurately extract roads at the pixel level, but there is a problem of poor topology of the extraction results; the latter has better topology than the former, but there is a problem of difficulty in end-to-end optimization. In addition, the current mainstream road extraction methods based on deep learning generally have the problem of difficulty in extracting complex road structures such as overpasses and multi-section intersections.

[0003] To solve these problems, the prior art proposes a graph-based end-to-end road network graph detection method (Road Network Graph Detection by Transformer, RNGDet). RNGDet first predicts a set of candidate initial vertices, and then iteratively generates a road network graph point by point starting from each candidate initial vertex; due to the use of Transformer and deep query structure, RNGDet can handle intersections with a large number of road segments. RNGDet was evaluated on public datasets and showed good performance in many evaluation indicators.

[0004] From the overall network architecture, RNGDet can be divided into feature extraction and inference prediction. The feature extraction part of RNGDet starts with the input of remote sensing images and ends with the generation of fused feature vector F. RNGDet receives remote sensing images I A As input, the final output graph structure G = (V, E). Where E = {e i}, is the set of edges in the graph, where each edge e i Represents a road segmentation; V = {v j} is the set of vertices in the graph, where each vertex v j It may represent the intersection of the road or the end point of the broken road. The RNGDet network is inspired by the DETR network structure and detects the road network graph through iteration. Starting from the candidate initial vertices predicted in the initial vertex set C, RNGDet iteratively generates the road network graph by controlling an agent to explore the road network. During the iteration process, the historical trajectory of the agent is recorded through the graph G. At each step, the agent's current position v tAs the center, RNGDet is in I A A region of interest (ROI) is cropped from the image and G is rasterized into H inside the ROI. A ResNet backbone network is used to extract the deep visual features F of the ROI. I , the feature is sent to the segmentation head to predict the road segmentation map S and the road intersection segmentation map I. By finding the local peak points of map I, the candidate initial vertices in C can be obtained. After connecting H with S and I, the ResNet backbone network is used to extract the deep feature F H Then, F H and F I The fused data is used as the input feature vector of Transformer and sent to the inference prediction part of RNGDet.

[0005] The reasoning prediction part of RNGDet starts from fusing the feature vector F with the position code and feeding it into the Transformer encoder, and ends with predicting valid vertices and updating the road network graph. Input N vertex query Q = {q i} i＝1 N , the Transformer decoder directly predicts N vertex embeddings, encoding the effective probability and coordinates of the vertex. Each vertex query is a learned embedding that can be viewed as a slot used by RNGDet to predict a vertex. Therefore, the number of vertex queries N must be greater than the maximum number of vertices that need to be predicted. After filtering out vertices with low probability, M (M≤N) valid vertices are obtained, which are adjacent to v t , forming a set γ={v t+1 i} i＝1 M If M = 0, RNGDet obtains a new candidate initial vertex from the candidate initial vertex set C and repeats the above process; if M = 1, the agent moves to the predicted coordinates and repeats the above process; if M>1, the agent sends all predicted vertices to the candidate initial vertex set C, obtains a vertex from C, and then repeats the above process. When the set C is empty, the algorithm stops and outputs the final road network graph.

[0006] Although RNGDet is superior to previous methods, it still cannot handle some very complex road conditions well, such as obscured overpasses. One reason for this problem is that the model is not capable of extracting features from complex roads. Therefore, it is urgent to propose a more powerful feature extraction backbone network to improve this problem. Summary of the invention

[0007] In view of the common problems of different road extraction methods at present, such as poor topology of extraction results, difficulty in end-to-end training of models, and poor extraction effect on complex road structures, the present invention improves on the basis of the baseline model RNGDet to solve the shortcomings that still exist in the current mainstream road extraction methods. Specifically, in terms of the feature extraction module of the road extraction model, the present invention introduces the atrous space pyramid pooling module (ASPP) to solve the problem that the existing methods are difficult to obtain road information of different scales when extracting features, expands the receptive field of extracted image features, and captures road information of different scales of the image; in terms of the feature fusion module of the road extraction model, the present invention introduces an improved feature pyramid network structure (MFPN) to solve the problem that the existing methods are difficult to fully utilize road feature information of different scales, processes input features of different sizes, and realizes full utilization of the extracted road feature information, thereby improving the model's extraction performance for road objects of different scales and complex road conditions.

[0008] To achieve the above object, the present invention provides a method for extracting roads from remote sensing images, comprising:

[0009] Acquire a remote sensing road image, and crop the remote sensing road image to obtain an area of ​​interest;

[0010] Extracting the deep features of the region of interest using the backbone network of the improved baseline model RNGDet to obtain a first feature tensor;

[0011] Based on the first feature tensor, an improved feature pyramid network structure is used to perform road segmentation prediction to obtain a road section distribution map and a road intersection and broken road endpoint distribution map;

[0012] Iteratively generate a road network map by detecting vertices and connecting vertices according to historical information, iteratively obtain a historical trajectory map according to the road network map, crop the historical trajectory map to an area of ​​interest, and rasterize the cropped historical trajectory map to obtain a rasterized map;

[0013] Input the road segment distribution map, the road intersection and broken road endpoint distribution map, and the rasterized map into the backbone network, and output a second feature tensor;

[0014] The first feature tensor and the second feature tensor are concatenated to obtain a target feature tensor, and the target feature tensor is sent to the Transformer structure as a feature extraction result of the backbone network for prediction, and the road network graph is updated according to the prediction result.

[0015] Preferably, the process of cropping the remote sensing road image to obtain the region of interest includes:

[0016] Select the current vertex v from the candidate vertex set C t, with the current vertex v t Centered on the image, a square area of ​​size 3×L×L is cropped, where L is 256.

[0017] Preferably, before sending the target feature tensor as the feature extraction result of the backbone network into the Transformer structure for prediction, the method further includes: converting the target feature tensor into a smaller tensor sequence;

[0018] The process of converting the target feature tensor into a sequence of smaller tensors includes:

[0019] By utilizing the permutation invariance of the Transformer architecture, the target feature tensor is fused with a fixed position code, and the fused tensor sequence is fed into the Transformer encoder structure, wherein the output tensor scale of the Transformer encoder is the same as the input tensor scale;

[0020] The output tensor is fed into the Transformer decoder structure, which receives the encoder's output sequence and a set of vertex queries Q = {q i} i＝1 N , and predict the embedding tensors of N adjacent vertices; each vertex query q i is a learned embedding tensor and produces a predicted adjacent vertex. The Transformer outputs the maximum a posteriori estimate of the vertex at the next time step. Each output tensor is decoded into an effective probability p through a feedforward network. t+1 i and the two-dimensional vertex coordinates v t+1 i , p t+1 i Indicates v t+1 i The probability of being valid, valid means it should be added to the road network graph;

[0021] If there are M (M≤N) valid prediction vertices, the set γ={v t+1 i} i＝1 M , then for each valid prediction vertex v t+1 i , by changing v t+1 i Add to the vertex set V and connect the newly added t+1 i With v t The edge is added to the edge set E to update the road network graph.

[0022] Preferably, the process of sending the target feature tensor as the feature extraction result of the backbone network into the Transformer structure for prediction includes:

[0023] The tensor output by the Transformer decoder is fed into the prediction module to finally determine the next iteration vertex v of the intelligent agent. t+1 First, we initialize the candidate vertex set C by finding the local peaks of the distribution graph of road intersections and broken road endpoints, and then improve the baseline model RNGDet to take a vertex v from the candidate vertex set C. t , with vertex v t The input image is cropped to the center, and in the next step, a valid vertex set γ is predicted, where the number of valid vertices in γ |γ| = M;

[0024] Depending on the value of M, different methods are used to update the road network graph.

[0025] Preferably, according to different values ​​of M, the process of taking different methods to update the road network graph includes:

[0026] When M = 0, it means there is no road ahead. Therefore, if the candidate vertex set C is not empty, the model will stop processing the current road and turn to process other roads. The model will re-take a vertex v from the vertex set C. t , re-predict;

[0027] When M = 1, it means that there is only one possible prediction point in front of the model, and the model predicts forward along a road; at this time, the model will v t+1 1 Add the vertex set V and connect the edge (v t , v t+1 1 ) joins the edge set E, and then the intelligent agent moves to v t+1 1 , with v t+1 1 Make predictions for the center and iteratively update the road network graph until an intersection M>1 or a disconnected road M=0 is encountered;

[0028] When M>1, it means that the model encounters a road intersection and needs to generate new predicted candidate vertices in multiple directions. At this time, the model will update the road network graph, put all vertices in the predicted vertex set γ into the selected vertex set C, and reselect an initial vertex v from the selected vertex set C. t Make predictions; in the inference stage, the model continues to run the iterative algorithm to generate a road network graph point by point. If and only if the candidate initial vertex set C is empty, the model ends running the algorithm and outputs the generated road network graph G = (V, E).

[0029] Preferably, the backbone network of the improved baseline model RNGDet includes ResNet-101 and a dilated space pyramid pooling module;

[0030] The atrous spatial pyramid pooling module uses 3×3 atrous convolutional layers with dilation rates of 6, 12, 18, and 24 in parallel to process the same input feature map. The features extracted by each convolutional layer are further processed in a separate branch and fused to generate the final result.

[0031] Preferably, the improved feature pyramid network structure includes three parts: bottom-up, top-down, and lateral connection;

[0032] The bottom-up structure consists of layers 51 to 67. The model receives input features from the backbone through layers 51, 55, and 67, with corresponding sizes of 20×20×1024, 40×40×256, and 80×80×128, respectively.

[0033] The 51st layer of the model receives 20×20×1024 feature inputs, and after passing through a cross-stage feature pyramid pooling module SPPCSPC consisting of a Conv convolution module, an MP maximum pooling module, and a Concat splicing module, the size becomes 20×20×512;

[0034] After 52 layers of 1×1 convolution, the size becomes 20×20×256. After 53 layers of 2x upsampling module, the size becomes 40×40×256. The 40×40×256 features are concatenated with the 40×40×256 input features from the backbone layer 55 in the channel dimension to obtain features of size 40×40×512.

[0035] The feature with a size of 40×40×512 passes through a multi-gradient fusion feature extraction module ELAN, and its size becomes 40×40×256. Then the feature undergoes the same 1×1 convolution and 2x upsampling as the convolution and upsampling operations of layers 52-53, and the size of the feature first becomes 40×40×128 and then becomes 80×80×128. At this time, the 80×80×128 feature is concat-joined with the 67-layer input feature 80×80×128 from the backbone in the channel dimension to obtain a feature with a size of 80×80×256.

[0036] Preferably, the improved feature pyramid network structure includes three parts: bottom-up, top-down, and lateral connection;

[0037] The top-down structure consists of layers 68 to 101, where features of size 80×80×256 first pass through an ELAN module, changing the size to 80×80×128, and then pass through an MP maximum pooling module, which also receives features of size 40×40×256 output by the ELAN modules of layers 56-63, and outputs features of size 40×40×512;

[0038] The features of size 40×40×512 continue to pass through an ELAN module, the size becomes 40×40×256, and then pass through an MP maximum pooling module, the MP maximum pooling module also receives the features of size 20×20×512 output by the 51-layer SPPCSPC module, and the MP maximum pooling module outputs features of size 20×20×1024;

[0039] The feature with the size of 20×20×1024 passes through an ELAN module again and becomes 20×20×512;

[0040] The top-down structure includes three parts that directly output to the detection head, namely, the ELAN modules of layers 68-75, which output features with a size of 80×80×128; the ELAN modules of layers 81-88, which output features with a size of 40×40×256; and the ELAN modules of layers 94-101, which output features with a size of 20×20×512.

[0041] Preferably, the SPPCSPC module includes a Conv convolution module, an MP maximum pooling module and a Concat splicing module;

[0042] The Conv convolution module is a collection of convolution operations of corresponding sizes, batch normalization operations, and SiLU activation function operations, and the convolution kernel sizes include 1×1 and 3×3;

[0043] The pooling sizes of the MP maximum pooling module include 5×5, 9×9 and 13×13;

[0044] The input feature size of the SPPCSPC module is 20×20×1024. The processing of the features is divided into two paths. One path first undergoes 1×1 convolution, 3×3 convolution, and 1×1 convolution, and the size becomes 20×20×512. At this time, the number of feature channels is halved. The features at this time are respectively sent to the 5×5 pooling layer, the 9×9 pooling layer, and the 13×13 pooling layer. Then, the three features with a size of 20×20×512 that pass through different pooling layers and the 20×20×512 features that do not pass through the pooling layer are concat-joined in the channel dimension to obtain features with a size of 20×20×2048.

[0045] The feature with a size of 20×20×2048 is further reduced in number of channels through a 1×1 convolution and a 3×3 convolution to become a feature with a size of 20×20×512;

[0046] At this time, the input feature undergoes a 1×1 convolution in another way, and its size also becomes 20×20×512. The two features of size 20×20×512 are concat-concatenated in the channel dimension to obtain a feature of 20×20×1024. Finally, it undergoes a 1×1 convolution reduction channel and outputs a feature of size 20×20×512.

[0047] Preferably, the ELAN module includes a Conv convolution module and a Concat splicing module;

[0048] The Conv convolution module is a collection of convolution operations of corresponding sizes, batch normalization, and SiLU activation function operations; the convolution kernel sizes include 1×1 and 3×3;

[0049] The input feature size of the ELAN module is 40×40×512. The processing of the feature is divided into two paths. One path undergoes 1×1 convolution and its size becomes 40×40×256. No more convolution operation is performed and it waits for the final Concat operation with other features. This feature is recorded as feature 1.

[0050] The other path first also undergoes a 1×1 convolution, and its size becomes 40×40×256. At this time, it is divided into two paths again. One path no longer undergoes convolution and waits for the final Concat splicing operation with other features, recorded as feature 2, and the other path undergoes a 3×3 convolution, and its size becomes 40×40×128; at this time, it is divided into two paths again, one path waits for the final Concat splicing, recorded as feature 3, and the other path continues to undergo 3×3 convolution, and the size remains unchanged;

[0051] Similarly, after three more 3×3 convolutions, features 4, 5, and 6 are generated at the same time; the sizes of features 1 and 2 are 40×40×256, and the sizes of features 3 to 6 are 40×40×128. The six features are concat-concatenated in the channel dimension to obtain features of size 40×40×1024; the features of size 40×40×1024 are further subjected to a 1×1 convolution to reduce the number of channels, and the output features of size 40×40×256 are obtained.

[0052] Compared with the prior art, the present invention has the following advantages and technical effects:

[0053] The present invention introduces the Atrous spatial pyramid pooling module (Atrous spatial pyramid pooling module). Al pyramid pooling (ASPP) improves the feature extraction part of the RNGDet model, which can expand the receptive field of the model to extract image features, capture road information of different scales in the image without adding additional parameters, and improve the robustness and accuracy of the model extraction results on small-scale objects and complex scenes;

[0054] The present invention introduces an improved feature pyramid network structure (ModiF I The feature fusion part of the RNGDet model is enhanced by using convolution, pooling, upsampling and other operations to process the input features of different scales, and the feature information extracted by the atrous space pyramid pooling module is fully utilized to improve the model's prediction performance for road segmentation maps and intersection segmentation maps. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] The drawings constituting a part of the present application are used to provide a further understanding of the present application. The illustrative embodiments and descriptions of the present application are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0056] Figure 1 A schematic diagram of a method flow of an embodiment of the present invention;

[0057] Figure 2 A schematic diagram of an algorithm for updating a road network diagram according to an embodiment of the present invention;

[0058] Figure 3 A schematic diagram of a dilated spatial pyramid pooling network according to an embodiment of the present invention;

[0059] Figure 4 A schematic diagram of an improved feature pyramid structure according to an embodiment of the present invention;

[0060] Figure 5 Schematic diagram of the structure of a cross-stage feature pyramid pooling module (SPPCSPC) according to an embodiment of the present invention;

[0061] Figure 6 Schematic diagram of the structure of a multi-gradient fusion feature extraction module (ELAN) according to an embodiment of the present invention;

[0062] Figure 7 Schematic diagram of the structure of the maximum pooling module (MP) according to an embodiment of the present invention. DETAILED DESCRIPTION

[0063] It should be noted that, in the absence of conflict, the embodiments and features in the embodiments of the present application can be combined with each other. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0064] It should be noted that the steps shown in the flowcharts of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and that, although a logical order is shown in the flowcharts, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0065] like Figure 1-7 As shown, this embodiment provides a remote sensing image road extraction method, including:

[0066] Acquire remote sensing road images, and crop the remote sensing road images to obtain the area of ​​interest;

[0067] The backbone network of the improved baseline model RNGDet is used to extract the deep features of the region of interest and obtain the first feature tensor;

[0068] Based on the first feature tensor, an improved feature pyramid network structure is used to perform road segmentation prediction to obtain a road section distribution map and a road intersection and broken road endpoint distribution map;

[0069] A road network diagram is generated by detecting vertices and connecting vertices iteratively according to historical information, a historical trajectory diagram is obtained iteratively according to the road network diagram, an area of ​​interest is cropped on the historical trajectory diagram, and the cropped historical trajectory diagram is rasterized to obtain a rasterized diagram;

[0070] Inputting the road segment distribution map, the road intersection and broken road endpoint distribution map, and the rasterized map into the backbone network, and outputting a second feature tensor;

[0071] The first feature tensor and the second feature tensor are concatenated to obtain the target feature tensor, which is sent to the Transformer structure as the feature extraction result of the backbone network for prediction, and the road network graph is updated according to the prediction result.

[0072] Furthermore, the process of cropping the remote sensing road image to obtain the region of interest includes:

[0073] Select the current vertex v from the candidate vertex set C t , with the current vertex v t Centered on the image, a square area of ​​size 3×L×L is cropped, where L is 256.

[0074] Further optimization scheme, such as Figure 1 As shown in FIG. 1 , the architecture of the model of this embodiment is divided into a feature extraction and fusion part and an inference and prediction part. The feature extraction and fusion part starts with the input of the remote sensing image. The input of the model is a color remote sensing image of size 3×4096×4096 (with RGB three channels), denoted as I AThe model iteratively generates a road network graph based on historical information, so the model includes a graph G that records the historical trajectory of the intelligent agent. The model iteratively generates a road network graph by detecting vertices and connecting vertices, so the model includes recording a set of candidate vertices C.

[0075] Because I A The size of is large, and the effect of directly extracting global features is poor. Therefore, the model first A A region of interest (ROI) is cut out from the top. The principle of selecting ROI is based on the current vertex v t (v t Select from the candidate vertex set C) as the center and crop a square area of ​​size 3×L×L (L is 256). Then use the backbone network to extract the deep features of ROI, denoted as F I .

[0076] The backbone network of the improved model consists of ResNet-101 and ASPP modules. Based on the extracted deep features F I , the MFPN structure is used to predict the segmentation maps S and I, where S represents the distribution map of road segments, and I represents the distribution map of road intersections and broken road endpoints. Both segmentation tasks are binary segmentation.

[0077] In order to obtain the historical trajectory information of the intelligent agent to assist road network detection, we use the clipping I A The image G is cropped in the form of ROI, and the cropped G is rasterized into an image H of size 1×L×L for subsequent splicing operations. With S, I and H as input, the backbone network outputs the feature tensor F H . I and F H The two feature tensors are concatenated to obtain the target feature tensor F, which is sent to the subsequent Transformer structure as the result of feature extraction by the backbone network. At this point, the model enters the reasoning and prediction part.

[0078] Furthermore, before the target feature tensor is sent as the feature extraction result of the backbone network into the Transformer structure for prediction, the following steps are also included: converting the target feature tensor into a smaller tensor sequence;

[0079] The process of converting the target feature tensor into a sequence of smaller tensors includes:

[0080] Using the permutation invariance of the Transformer architecture, the target feature tensor is fused with the fixed position code, and the fused tensor sequence is fed into the Transformer encoder structure. The output tensor scale of the Transformer encoder is the same as the input tensor scale.

[0081] The output tensor is fed into the Transformer decoder structure, which receives the encoder's output sequence and a set of vertex queries Q = {q i} i＝1 N , and predict the embedding tensors of N adjacent vertices; each vertex query q i is a learned embedding tensor and produces a predicted adjacent vertex. The Transformer outputs the maximum a posteriori estimate of the vertex at the next time step. Each output tensor is decoded into an effective probability p through a feedforward network. t+1 i and the two-dimensional vertex coordinates v t+1 i , p t+1 i Indicates v t+1 i The probability of being valid, valid means it should be added to the road network graph;

[0082] If there are M (M≤N) valid prediction vertices, the set γ={v t+1 i} i＝1 M , then for each valid prediction vertex v t+1 i , by changing v t+1 i Add to the vertex set V and connect the newly added t+1 i With v t The edge is added to the edge set E to update the road network graph.

[0083] Furthermore, the process of sending the target feature tensor as the feature extraction result of the backbone network into the Transformer structure for prediction includes:

[0084] The tensor output by the Transformer decoder is fed into the prediction module to finally determine the next iteration vertex v of the intelligent agent. t+1 First, we initialize the candidate vertex set C by finding the local peaks of the distribution graph of road intersections and broken road endpoints, and then improve the baseline model RNGDet to take a vertex v from the candidate vertex set C. t , with vertex v t The input image is cropped to the center, and in the next step, a valid vertex set γ is predicted, where the number of valid vertices in γ |γ| = M;

[0085] Depending on the value of M, different methods are used to update the road network graph.

[0086] Furthermore, if Figure 2 As shown in FIG. 1 , according to different values ​​of M, the process of updating the road network graph by different methods includes:

[0087] When M = 0, it means there is no road ahead. Therefore, if the candidate vertex set C is not empty, the model will stop processing the current road and turn to process other roads. The model will re-take a vertex v from the vertex set C. t , re-predict;

[0088] When M = 1, it means that there is only one possible prediction point in front of the model, and the model predicts forward along a road; at this time, the model will v t+1 1 Add the vertex set V and connect the edge (v t , v t+1 1 ) joins the edge set E, and then the intelligent agent moves to v t+1 1 , with v t+1 1 Make predictions for the center and iteratively update the road network graph until an intersection M>1 or a disconnected road M=0 is encountered;

[0089] When M>1, it means that the model encounters a road intersection and needs to generate new predicted candidate vertices in multiple directions. At this time, the model will update the road network graph, put all vertices in the predicted vertex set γ into the selected vertex set C, and reselect an initial vertex v from the selected vertex set C. t Make predictions; in the inference stage, the model continues to run the iterative algorithm to generate a road network graph point by point. If and only if the candidate initial vertex set C is empty, the model ends running the algorithm and outputs the generated road network graph G = (V, E).

[0090] The atrous convolution principle used in this embodiment is to insert some holes in the traditional convolution kernel so that the convolution kernel can span a larger area and increase the receptive field without increasing parameters and computational complexity. It not only does not reduce the resolution of the feature map, but also reduces the loss of detail information and the decrease in positioning accuracy. It can also obtain multi-scale contextual information and improve the prediction performance of small-scale objects.

[0091] Furthermore, the backbone network of the improved baseline model RNGDet includes ResNet-101 and atrous spatial pyramid pooling modules;

[0092] like Figure 3As shown in the figure, the Atrous spatial pyramid pooling (ASPP) module uses multiple atrous convolutional layers with different dilation rates in parallel to extract features. The features extracted by each convolutional layer are further processed in a separate branch and fused to generate the final result. In this model, 3×3 atrous convolutional layers with dilation rates of 6, 12, 18, and 24 are used to process the same input feature map, and these outputs are concatenated to form a richer and more diverse feature representation, which improves the robustness and accuracy of the segmentation results on small objects and complex scenes.

[0093] Furthermore, if Figure 4 As shown in the figure, the improved feature pyramid network structure includes three parts: bottom-up, top-down, and lateral connection; the structure contains many operation modules such as convolution and upsampling.

[0094] The bottom-up structure consists of layers 51 to 67. The model receives input features from the backbone through layers 51, 55, and 67, with corresponding sizes of 20×20×1024, 40×40×256, and 80×80×128, respectively.

[0095] The 51st layer of the model receives 20×20×1024 feature inputs, and after passing through a cross-stage feature pyramid pooling module SPPCSPC consisting of a Conv convolution module, an MP maximum pooling module, and a Concat splicing module, the size becomes 20×20×512;

[0096] After 52 layers of 1×1 convolution, the size becomes 20×20×256. After 53 layers of 2x upsampling module, the size becomes 40×40×256. The 40×40×256 features are concatenated with the 40×40×256 input features from the backbone layer 55 in the channel dimension to obtain features of size 40×40×512.

[0097] The features with a size of 40×40×512 are transformed into 40×40×256 through a multi-gradient fusion feature extraction module ELAN. After that, the features are subjected to the same 1×1 convolution and 2x upsampling as the convolution and upsampling operations of layers 52-53. The feature size is first transformed into 40×40×128 and then into 80×80×128. The 80×80×128 features at this time are concat-joined with the 67-layer input features 80×80×128 from the backbone in the channel dimension to obtain features with a size of 80×80×256.

[0098] Furthermore, the improved feature pyramid network structure includes three parts: bottom-up, top-down, and lateral connection;

[0099] The top-down structure consists of layers 68 to 101. The features of size 80×80×256 first pass through an ELAN module, and the size becomes 80×80×128, and then pass through an MP maximum pooling module. The MP maximum pooling module integrates the maximum pooling operation with multiple convolution operations and Concat splicing operations, including Conv convolution module, MP maximum pooling module and Concat splicing module. The MP maximum pooling module also receives the features of size 40×40×256 output by the ELAN modules of layers 56-63, and the MP maximum pooling module outputs features of size 40×40×512;

[0100] The features of size 40×40×512 continue to pass through an ELAN module, and the size becomes 40×40×256, and then pass through an MP maximum pooling module. The MP maximum pooling module also receives the features of size 20×20×512 output by the 51-layer SPPCSPC module. The MP maximum pooling module outputs features of size 20×20×1024;

[0101] The features with the size of 20×20×1024 are passed through an ELAN module again and the size becomes 20×20×512;

[0102] The top-down structure includes three parts that directly output to the detection head, namely the ELAN modules of layers 68-75, which output features of size 80×80×128; the ELAN modules of layers 81-88, which output features of size 40×40×256; and the ELAN modules of layers 94-101, which output features of size 20×20×512.

[0103] The MFPN structure can be divided into bottom-up, top-down, and horizontally connected structures, which are typical FPN feature pyramid structures. By performing convolution, pooling, upsampling of features of different sizes, and fusing features from different layers, the model's ability to extract features is continuously enhanced.

[0104] Furthermore, if Figure 5 As shown, the SPPCSPC module includes the Conv convolution module, the MP maximum pooling module and the Concat splicing module;

[0105] The Conv convolution module is a collection of convolution operations of corresponding sizes, batch normalization operations (Batch Normalization, BN) and SiLU activation function operations. The convolution kernel sizes include 1×1 and 3×3;

[0106] The pooling sizes of the MP maximum pooling module include 5×5, 9×9, and 13×13;

[0107] The input feature size of the SPPCSPC module is 20×20×1024. The feature processing is divided into two paths. One path first passes through 1×1 convolution, 3×3 convolution, and 1×1 convolution, and the size becomes 20×20×512. At this time, the number of feature channels is halved. The features at this time are sent to the 5×5 pooling layer, 9×9 pooling layer, and 13×13 pooling layer respectively. Then, the three features with a size of 20×20×512 that pass through different pooling layers and the 20×20×512 features that do not pass through the pooling layer are concat-concatenated in the channel dimension to obtain features with a size of 20×20×2048.

[0108] The feature with a size of 20×20×2048 is further reduced through a 1×1 convolution and a 3×3 convolution to reduce the number of channels and becomes a feature with a size of 20×20×512.

[0109] At this time, the input feature undergoes a 1×1 convolution in another way, and its size also becomes 20×20×512. The two features of size 20×20×512 are concat-concatenated in the channel dimension to obtain a feature of 20×20×1024. Finally, it undergoes a 1×1 convolution reduction channel and outputs a feature of size 20×20×512.

[0110] Furthermore, if Figure 6 As shown, the ELAN module includes a Conv convolution module and a Concat splicing module;

[0111] Similar to the SPPCSPC module, the Conv convolution module is a collection of convolution operations of corresponding sizes, batch normalization operations (BatchNormalization, BN) and SiLU activation function operations; the convolution kernel sizes include 1×1 and 3×3;

[0112] The input feature size of the ELAN module is 40×40×512. The feature processing is divided into two paths. One path undergoes 1×1 convolution and the size becomes 40×40×256. The convolution operation is no longer performed and it waits for the final concat operation with other features. This feature is recorded as feature 1.

[0113] The other path first also undergoes a 1×1 convolution, and its size becomes 40×40×256. At this time, it is divided into two paths again. One path no longer undergoes convolution and waits for the final Concat splicing operation with other features, recorded as feature 2, and the other path undergoes a 3×3 convolution, and its size becomes 40×40×128; at this time, it is divided into two paths again, one path waits for the final Concat splicing, recorded as feature 3, and the other path continues to undergo 3×3 convolution, and the size remains unchanged;

[0114] Similarly, after three more 3×3 convolutions, features 4, 5, and 6 are generated at the same time; the sizes of features 1 and 2 are 40×40×256, and the sizes of features 3 to 6 are 40×40×128. The six features are concat-concatenated in the channel dimension to obtain features of size 40×40×1024; the features of size 40×40×1024 are further subjected to a 1×1 convolution to reduce the number of channels, and the output features of size 40×40×256 are obtained.

[0115] Furthermore, if Figure 7 As shown in the figure, the MP Max Pooling module (MP) includes a Conv convolution module (the convolution kernel size includes 1×1 and 3×3), an MP Max Pooling unit (the pooling size is 2×2), and a Concat splicing module. Similar to the SPPCSPC module, the Conv convolution module in MP refers to a collection of convolution operations of corresponding sizes and batch normalization operations (Batch Normalization, BN) and SiLU activation function operations. In the MP Max Pooling unit, the input feature size is 80×80×128, and the processing of this feature is divided into two paths. One path passes through a 2×2 pooling layer, and the feature size is reduced to 40×40×128, and then passes through a 1×1 convolution, and the feature size remains unchanged. The feature obtained at this time is recorded as feature 1; the other path passes through a 1×1 convolution and a 3×3 convolution successively. The size remains unchanged after the 1×1 convolution, and the size is reduced to 40×40×128 after the 3×3 convolution. The feature obtained at this time is recorded as feature 2; a feature of size 40×40×256 is input from the backbone, recorded as feature 3; Concat splicing operation is performed on feature 1, feature 2, and feature 3 in the channel dimension to obtain a feature of size 40×40×512 and output it.

[0116] The MFPN structure composed of SPPCSPC, ELAN, MP and other modules performs convolution, pooling, upsampling and other operations on the input features, fully utilizes the feature information extracted by the atrous spatial pyramid pooling module, and improves the model's prediction performance for road segmentation maps and intersection segmentation maps.

[0117] The above are only preferred specific implementations of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed in the present application should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A method for extracting roads from remote sensing images, characterized in that: include: Acquire a remote sensing road image, and crop the remote sensing road image to obtain an area of ​​interest; Extracting the deep features of the region of interest using the backbone network of the improved baseline model RNGDet to obtain a first feature tensor; Based on the first feature tensor, an improved feature pyramid network structure is used to perform road segmentation prediction to obtain a road section distribution map and a road intersection and broken road endpoint distribution map; Iteratively generate a road network map by detecting vertices and connecting vertices according to historical information, iteratively obtain a historical trajectory map according to the road network map, crop the historical trajectory map to an area of ​​interest, and rasterize the cropped historical trajectory map to obtain a rasterized map; Input the road section distribution map, the road intersection and broken road endpoint distribution map, and the rasterized map into the backbone network, and output a second feature tensor; The first feature tensor and the second feature tensor are concatenated to obtain a target feature tensor, and the target feature tensor is sent to the Transformer structure as a feature extraction result of the backbone network for prediction, and the road network graph is updated according to the prediction result.

2. The method for extracting roads from remote sensing images according to claim 1, characterized in that: The process of clipping the remote sensing road image to obtain the region of interest includes: Select the current vertex v from the candidate vertex set C t , with the current vertex v t Centered on the image, a square area of ​​size 3×L×L is cropped, where L is 256.

3. The method for extracting roads from remote sensing images according to claim 1, characterized in that: Before sending the target feature tensor as the feature extraction result of the backbone network into the Transformer structure for prediction, the method further includes: converting the target feature tensor into a smaller tensor sequence; The process of converting the target feature tensor into a sequence of smaller tensors includes: By utilizing the permutation invariance of the Transformer architecture, the target feature tensor is fused with a fixed position code, and the fused tensor sequence is fed into the Transformer encoder structure, wherein the output tensor scale of the Transformer encoder is the same as the input tensor scale; The output tensor is fed into the Transformer decoder structure, which receives the encoder's output sequence and a set of vertex queries Q = {q i } i＝1 N , and predict the embedding tensors of N adjacent vertices; each vertex query q i is a learned embedding tensor and produces a predicted adjacent vertex. The Transformer outputs the maximum a posteriori estimate of the vertex at the next time step. Each output tensor is decoded into an effective probability p through a feedforward network. t+1 i and the two-dimensional vertex coordinates v t+1 i , p t+1 i Indicates v t+1 i The probability of being valid, valid means it should be added to the road network graph; If there are M (M≤N) valid prediction vertices, the set γ={v t+1 i } i＝1 M , then for each valid prediction vertex v t+1 i , by changing v t+1 i Add to the vertex set V and connect the newly added t+1 i With v t The edge is added to the edge set E to update the road network graph.

4. The method for extracting roads from remote sensing images according to claim 1, characterized in that: The process of sending the target feature tensor as the feature extraction result of the backbone network into the Transformer structure for prediction includes: The tensor output by the Transformer decoder is fed into the prediction module to finally determine the next iteration vertex v of the intelligent agent. t+1 First, we initialize the candidate vertex set C by finding the local peaks of the distribution graph of road intersections and broken road endpoints, and then improve the baseline model RNGDet to take a vertex v from the candidate vertex set C. t , with vertex v t The input image is cropped to the center, and in the next step, a valid vertex set γ is predicted, where the number of valid vertices in γ |γ| = M; Depending on the value of M, different methods are used to update the road network graph.

5. The method for extracting roads from remote sensing images according to claim 1, characterized in that: Depending on the value of M, the process of updating the road network graph using different methods includes: When M = 0, it means there is no road ahead. Therefore, if the candidate vertex set C is not empty, the model will stop processing the current road and turn to process other roads. The model will re-take a vertex v from the vertex set C. t , re-predict; When M = 1, it means that there is only one possible prediction point in front of the model, and the model predicts forward along a road; at this time, the model will v t+1 1 Add the vertex set V and connect the edge (v t , v t+1 1 ) joins the edge set E, and then the intelligent agent moves to v t+1 1 , with v t+1 1 Make predictions for the center and iteratively update the road network graph until an intersection M>1 or a disconnected road M=0 is encountered; When M>1, it means that the model encounters a road intersection and needs to generate new predicted candidate vertices in multiple directions. At this time, the model will update the road network graph, put all vertices in the predicted vertex set γ into the selected vertex set C, and reselect an initial vertex v from the selected vertex set C. t Make predictions; in the inference stage, the model continues to run the iterative algorithm to generate a road network graph point by point. If and only if the candidate initial vertex set C is empty, the model ends running the algorithm and outputs the generated road network graph G = (V, E).

6. The method for extracting roads from remote sensing images according to claim 1, characterized in that: The backbone network of the improved baseline model RNGDet includes ResNet-101 and a dilated spatial pyramid pooling module; The atrous spatial pyramid pooling module uses 3×3 atrous convolutional layers with dilation rates of 6, 12, 18, and 24 in parallel to process the same input feature map. The features extracted by each convolutional layer are further processed in a separate branch and fused to generate the final result.

7. The method for extracting roads from remote sensing images according to claim 1, characterized in that: The improved feature pyramid network structure includes three parts: bottom-up, top-down, and lateral connection; The bottom-up structure consists of layers 51 to 67. The model receives input features from the backbone through layers 51, 55, and 67, with corresponding sizes of 20×20×1024, 40×40×256, and 80×80×128, respectively. The 51st layer of the model receives 20×20×1024 feature inputs, and after passing through a cross-stage feature pyramid pooling module SPPCSPC consisting of a Conv convolution module, an MP maximum pooling module, and a Concat splicing module, the size becomes 20×20×512; After 52 layers of 1×1 convolution, the size becomes 20×20×256. After 53 layers of 2x upsampling module, the size becomes 40×40×256. The 40×40×256 features are concatenated with the 40×40×256 input features from the backbone layer 55 in the channel dimension to obtain features of size 40×40×512. The feature with a size of 40×40×512 passes through a multi-gradient fusion feature extraction module ELAN, and its size becomes 40×40×256. Then the feature undergoes the same 1×1 convolution and 2x upsampling as the convolution and upsampling operations of layers 52-53, and the size of the feature first becomes 40×40×128 and then becomes 80×80×128. At this time, the 80×80×128 feature is concat-joined with the 67-layer input feature 80×80×128 from the backbone in the channel dimension to obtain a feature with a size of 80×80×256.

8. The method for extracting roads from remote sensing images according to claim 1, characterized in that: The improved feature pyramid network structure includes three parts: bottom-up, top-down, and lateral connection; The top-down structure consists of layers 68 to 101, where features of size 80×80×256 first pass through an ELAN module, changing the size to 80×80×128, and then pass through an MP maximum pooling module, which also receives features of size 40×40×256 output by the ELAN modules of layers 56-63, and outputs features of size 40×40×512; The features of size 40×40×512 continue to pass through an ELAN module, the size becomes 40×40×256, and then pass through an MP maximum pooling module, the MP maximum pooling module also receives the features of size 20×20×512 output by the 51-layer SPPCSPC module, and the MP maximum pooling module outputs features of size 20×20×1024; The feature with the size of 20×20×1024 passes through an ELAN module again and becomes 20×20×512; The top-down structure includes three parts that directly output to the detection head, namely, the ELAN modules of layers 68-75, which output features of size 80×80×128; The ELAN modules of layers 81-88 output features of size 40×40×256, and the ELAN modules of layers 94-101 output features of size 20×20×512.

9. The method for extracting roads from remote sensing images according to claim 8, characterized in that: The SPPCSPC module includes a Conv convolution module, an MP maximum pooling module and a Concat splicing module; The Conv convolution module is a collection of convolution operations of corresponding sizes, batch normalization operations, and SiLU activation function operations, and the convolution kernel sizes include 1×1 and 3×3; The pooling sizes of the MP maximum pooling module include 5×5, 9×9 and 13×13; The input feature size of the SPPCSPC module is 20×20×1024. The processing of the features is divided into two paths. One path first undergoes 1×1 convolution, 3×3 convolution, and 1×1 convolution, and the size becomes 20×20×512. At this time, the number of feature channels is halved. The features at this time are respectively sent to the 5×5 pooling layer, the 9×9 pooling layer, and the 13×13 pooling layer. Then, the three features with a size of 20×20×512 that pass through different pooling layers and the 20×20×512 features that do not pass through the pooling layer are concat-joined in the channel dimension to obtain features with a size of 20×20×2048. The feature with a size of 20×20×2048 is further reduced in number of channels through a 1×1 convolution and a 3×3 convolution to become a feature with a size of 20×20×512; At this time, the input feature undergoes a 1×1 convolution in another way, and its size also becomes 20×20×512. The two features of size 20×20×512 are concat-concatenated in the channel dimension to obtain a feature of 20×20×1024. Finally, it undergoes a 1×1 convolution reduction channel and outputs a feature of size 20×20×512.

10. The method for extracting roads from remote sensing images according to claim 8, characterized in that: The ELAN module includes a Conv convolution module and a Concat splicing module; The Conv convolution module is a collection of convolution operations of corresponding sizes, batch normalization, and SiLU activation function operations; the convolution kernel sizes include 1×1 and 3×3; The input feature size of the ELAN module is 40×40×512. The processing of the feature is divided into two paths. One path undergoes 1×1 convolution and its size becomes 40×40×256. No more convolution operation is performed and it waits for the final Concat operation with other features. This feature is recorded as feature 1. The other path first also undergoes a 1×1 convolution, and its size becomes 40×40×256. At this time, it is divided into two paths again. One path no longer undergoes convolution and waits for the final Concat splicing operation with other features, recorded as feature 2, and the other path undergoes a 3×3 convolution, and its size becomes 40×40×128; at this time, it is divided into two paths again, one path waits for the final Concat splicing, recorded as feature 3, and the other path continues to undergo 3×3 convolution, and the size remains unchanged; Similarly, after three more 3×3 convolutions, features 4, 5, and 6 are generated at the same time; the sizes of features 1 and 2 are 40×40×256, and the sizes of features 3 to 6 are 40×40×128. The six features are concat-concatenated in the channel dimension to obtain features of size 40×40×1024; the features of size 40×40×1024 are further subjected to a 1×1 convolution to reduce the number of channels, and the output features of size 40×40×256 are obtained.

Images