A method for road extraction from remote sensing images

Abstract

The present invention discloses a method for extracting roads from remote sensing images, comprising: acquiring a remote sensing road image and cropping it to obtain a region of interest; extracting deep features of the region of interest using a backbone network of an improved baseline model RNGDet to obtain a first feature tensor; based on the first feature tensor, using an improved feature pyramid network structure to perform road segmentation prediction to obtain two distribution maps; obtaining a historical trajectory map, cropping and rasterizing it to obtain a rasterized map; inputting the two distribution maps and the rasterized map into the backbone network to obtain a second feature tensor; concatenating the first and second feature tensors to obtain a target feature tensor, feeding the target feature tensor as the feature extraction result of the backbone network into a Transformer structure for prediction, and updating the road network map based on the prediction result. The present invention improves the robustness and accuracy of road extraction results for small-scale objects and complex scenes.

Description

Technical Field

[0001] The present invention belongs to the technical field of road extraction, and in particular relates to a method for extracting roads from remote sensing images. Background Art

[0002] As an important means of obtaining road information, road extraction from remote sensing images has broad application prospects. With the development of deep learning technology and the increasing abundance of remote sensing image data, deep learning-based road extraction methods have gradually become an important research direction for remote sensing image road extraction. Deep learning-based road extraction methods can be divided into two categories: segmentation-based methods and graph generation-based methods. The former can accurately extract roads at the pixel level, but suffers from poor topological properties of the extracted results; the latter produces road extraction results with better topological properties than the former, but suffers from the difficulty of end-to-end optimization. Furthermore, current mainstream deep learning-based road extraction methods generally have difficulty extracting complex road structures such as overpasses and multi-segment intersections.

[0003] To address these issues, a graph-based, end-to-end road network graph detection method (Road Network Graph Detection by Transformer, RNGDet) has been proposed. RNGDet first predicts a set of candidate initial vertices and then iteratively generates a road network graph point by point starting from each candidate initial vertex. Due to its use of the Transformer and deep query architecture, RNGDet can handle intersections with a large number of road segments. RNGDet has been evaluated on public datasets and demonstrated good performance across many evaluation metrics.

[0004] From the perspective of the overall network architecture, RNGDet can be divided into the feature extraction part and the inference prediction part. The feature extraction part of RNGDet starts with the input of remote sensing images and ends with the generation of fusion feature vector F. RNGDet receives remote sensing images I A As input, the final output graph structure G = (V, E). Where E = {e i}, is the set of edges in the graph, where each edge e i Represents a road segmentation; V={v j} is a set of vertices in the graph, where each vertex v j It may represent a road intersection or the end point of a broken road. The RNGDet network is inspired by the DETR network structure and detects the road network graph through iteration. Starting from the candidate initial vertices predicted in the initial vertex set C, RNGDet iteratively generates a road network graph by controlling an agent to explore the road network. During the iteration process, the historical trajectory of the agent is recorded through the graph G. At each step, the agent's current position v is tAs the center, RNGDet is A A region of interest (ROI) is cropped from the image and G is rasterized into H inside the ROI. A ResNet backbone network is used to extract the deep visual features F of the ROI. I , the feature is sent to the segmentation head to predict the road segmentation map S and the road intersection segmentation map I. By finding the local peak points of map I, the candidate initial vertices in C can be obtained. After connecting H with S and I, the ResNet backbone network is used to extract the deep feature F H Then, F H and F I The fusion is used as the input feature vector of Transformer and sent to the inference prediction part of RNGDet.

[0005] The reasoning prediction part of RNGDet starts from fusing the feature vector F with the position code and feeding it into the Transformer encoder, and ends with predicting the valid vertices and updating the road network graph. Input N vertex query Q = {q i} i＝1 N , the Transformer decoder directly predicts N vertex embeddings, encoding the effective probability and coordinates of the vertex. Each vertex query is a learned embedding that can be regarded as a slot used by RNGDet to predict a vertex. Therefore, the number of vertex queries N must be greater than the maximum number of vertices that need to be predicted. After filtering out vertices with low probability, M (M≤N) valid vertices are obtained, which are adjacent to v t , forming a set γ={v t+1 i} i＝1 M If M = 0, RNGDet obtains a new candidate initial vertex from the candidate initial vertex set C and repeats the above process. If M = 1, the agent moves to the predicted coordinates and repeats the above process. If M > 1, the agent feeds all predicted vertices into the candidate initial vertex set C, obtains a vertex from C, and repeats the above process. When the set C is empty, the algorithm stops and outputs the final road network graph.

[0006] Although RNGDet is superior to previous methods, it still cannot handle some very complex road conditions well, such as obscured overpasses. One reason for this problem is that the model is not capable of extracting features from complex roads. Therefore, it is urgent to propose a more powerful feature extraction backbone network to improve this problem. Summary of the Invention

[0007] In response to the common problems of different road extraction methods at present, such as poor topology of extraction results, difficulty in end-to-end training of models, and poor extraction effect on complex road structures, the present invention improves on the basis of the baseline model RNGDet to solve the shortcomings that still exist in the current mainstream road extraction methods. Specifically, in terms of the feature extraction module of the road extraction model, the present invention introduces the atrous spatial pyramid pooling module (ASPP) to solve the problem that existing methods have difficulty in obtaining road information of different scales when extracting features, expands the receptive field of extracted image features, and captures road information of different scales of the image; in terms of the feature fusion module of the road extraction model, the present invention introduces an improved feature pyramid network structure (MFPN) to solve the problem that existing methods have difficulty in fully utilizing road feature information of different scales, processes input features of different sizes, realizes full utilization of the extracted road feature information, and improves the model's extraction performance for road objects of different scales and complex road conditions.

[0008] To achieve the above object, the present invention provides a method for extracting roads from remote sensing images, comprising:

[0009] Acquiring a remote sensing road image, and cropping the remote sensing road image to obtain a region of interest;

[0010] Extracting the deep features of the region of interest using the backbone network of the improved baseline model RNGDet to obtain a first feature tensor;

[0011] Based on the first feature tensor, an improved feature pyramid network structure is used to perform road segmentation prediction to obtain a road section distribution map and a road intersection and broken road endpoint distribution map;

[0012] Iteratively generating a road network graph by detecting vertices and connecting vertices according to historical information, iteratively obtaining a historical trajectory graph according to the road network graph, clipping an area of ​​interest on the historical trajectory graph, and rasterizing the clipped historical trajectory graph to obtain a rasterized graph;

[0013] Inputting the road segment distribution map, the road intersection and broken road endpoint distribution map, and the rasterized map into the backbone network, and outputting a second feature tensor;

[0014] The first feature tensor and the second feature tensor are concatenated to obtain a target feature tensor, which is used as the feature extraction result of the backbone network and sent to the Transformer structure for prediction. The road network graph is updated according to the prediction result.

[0015] Preferably, the process of cropping the remote sensing road image to obtain the region of interest includes:

[0016] Select the current vertex v from the candidate vertex set C t, taking the current vertex v t As the center, crop a square area of ​​size 3×L×L, where L is 256.

[0017] Preferably, before sending the target feature tensor as the feature extraction result of the backbone network into the Transformer structure for prediction, the method further includes: converting the target feature tensor into a smaller tensor sequence;

[0018] The process of converting the target feature tensor into a sequence of smaller tensors includes:

[0019] Utilizing the permutation invariance of the Transformer architecture, the target feature tensor is fused with a fixed positional encoding, and the fused tensor sequence is fed into a Transformer encoder structure, where the output tensor scale of the Transformer encoder is the same as the input tensor scale;

[0020] The output tensor is fed into the Transformer decoder structure, which receives the encoder’s output sequence and a set of vertex queries Q = {q i} i＝1 N , and predict the embedding tensors of N adjacent vertices; each vertex query q i is a learned embedding tensor and produces a predicted adjacent vertex. Transformer outputs the maximum a posteriori estimate of the vertex at the next time step. Each output tensor is decoded into an effective probability p through the feedforward network. t+1 i and the two-dimensional vertex coordinates v t+1 i , p t+1 i Indicates v t+1 i The probability of being valid, valid means it should be added to the road network graph;

[0021] If there are M (M≤N) valid prediction vertices, the set γ={v t+1 i} i＝1 M , then for each valid prediction vertex v t+1 i , by changing v t+1 i Add to the vertex set V and connect the newly added t+1 i With v t The edge of is added to the edge set E to update the road network graph.

[0022] Preferably, the process of sending the target feature tensor as the feature extraction result of the backbone network into the Transformer structure for prediction includes:

[0023] The tensor output by the Transformer decoder is fed into the prediction module to ultimately determine the next iteration vertex v of the intelligent agent. t+1 First, we initialize the candidate vertex set C by finding the local peaks of the distribution graph of road intersections and broken road endpoints, and then improve the baseline model RNGDet to extract a vertex v from the candidate vertex set C. t , with vertex v t Crop the input image to the center and predict the valid vertex set γ in the next step, where the number of valid vertices in γ |γ| = M;

[0024] According to the different values ​​of M, different methods are adopted to update the road network graph.

[0025] Preferably, according to different values ​​of M, the process of updating the road network graph by adopting different methods includes:

[0026] When M=0, it means there is no road ahead. Therefore, if the candidate vertex set C is not empty, the model will stop processing the current road and turn to process other roads. The model will re-take a vertex v from the vertex set C. t , re-predict;

[0027] When M=1, it means that there is only one possible prediction point in front of the model, and the model predicts along a road; at this time, the model will v t+1 1 Add the vertex set V and connect the edge (v t , v t+1 1 ) joins the edge set E, and then the intelligent agent moves to v t+1 1 , with v t+1 1 Make predictions for the center and iteratively update the road network graph until an intersection M>1 or a disconnected road M=0 is encountered;

[0028] When M>1, it means that the model encounters a road intersection and needs to generate new predicted candidate vertices in multiple directions. At this time, the model will update the road network graph, put all vertices in the predicted vertex set γ into the selected vertex set C, and reselect an initial vertex v from the selected vertex set C. t Make predictions; in the inference stage, the model continues to run the iterative algorithm to generate a road network graph point by point. If and only if the candidate initial vertex set C is empty, the model ends running the algorithm and outputs the generated road network graph G = (V, E).

[0029] Preferably, the backbone network of the improved baseline model RNGDet includes ResNet-101 and a dilated spatial pyramid pooling module;

[0030] The atrous spatial pyramid pooling module uses 3×3 atrous convolutional layers with dilation rates of 6, 12, 18, and 24 in parallel to process the same input feature map. The features extracted by each convolutional layer are further processed in a separate branch and fused to generate the final result.

[0031] Preferably, the improved feature pyramid network structure includes three parts: bottom-up, top-down, and lateral connection;

[0032] The bottom-up structure consists of layers 51 to 67. The model receives input features from the backbone through layers 51, 55, and 67, with corresponding sizes of 20×20×1024, 40×40×256, and 80×80×128, respectively.

[0033] The 51st layer of the model receives a 20×20×1024 feature input, which is then transformed into a 20×20×512 size after passing through a cross-stage feature pyramid pooling module SPPCSPC consisting of a Conv convolution module, an MP maximum pooling module, and a Concat splicing module.

[0034] After 52 layers of 1×1 convolution, the size becomes 20×20×256. After 53 layers of 2x upsampling module, the size becomes 40×40×256. The 40×40×256 features at this time are concatenated with the 40×40×256 input features from the backbone layer 55 in the channel dimension to obtain features of size 40×40×512.

[0035] The features with a size of 40×40×512 are transformed into 40×40×256 by a multi-gradient fusion feature extraction module ELAN. After that, the features are subjected to the same 1×1 convolution and 2x upsampling as the convolution and upsampling operations of layers 52-53. The size of the features is first transformed into 40×40×128 and then into 80×80×128. The 80×80×128 features at this time are concat-ed with the 80×80×128 input features of layer 67 from the backbone in the channel dimension to obtain features with a size of 80×80×256.

[0036] Preferably, the improved feature pyramid network structure includes three parts: bottom-up, top-down, and lateral connection;

[0037] The top-down structure consists of layers 68 to 101. Features of size 80×80×256 first pass through an ELAN module to become 80×80×128, and then pass through an MP maximum pooling module. The MP maximum pooling module also receives the features of size 40×40×256 output by the ELAN modules of layers 56-63. The MP maximum pooling module outputs features of size 40×40×512.

[0038] The features of size 40×40×512 continue to pass through an ELAN module, and the size becomes 40×40×256, and then pass through an MP maximum pooling module. The MP maximum pooling module also receives the features of size 20×20×512 output by the 51-layer SPPCSPC module, and the MP maximum pooling module outputs features of size 20×20×1024;

[0039] The feature with the size of 20×20×1024 passes through an ELAN module again and becomes 20×20×512;

[0040] The top-down structure includes three parts that directly output to the detection head: the ELAN modules of layers 68-75, which output features of size 80×80×128; the ELAN modules of layers 81-88, which output features of size 40×40×256; and the ELAN modules of layers 94-101, which output features of size 20×20×512.

[0041] Preferably, the SPPCSPC module includes a Conv convolution module, an MP maximum pooling module and a Concat splicing module;

[0042] The Conv convolution module is a collection of convolution operations of corresponding sizes, batch normalization operations, and SiLU activation function operations, and the convolution kernel sizes include 1×1 and 3×3;

[0043] The pooling sizes of the MP maximum pooling module include 5×5, 9×9 and 13×13;

[0044] The input feature size of the SPPCSPC module is 20×20×1024. The processing of the features is divided into two paths. One path first undergoes 1×1 convolution, 3×3 convolution, and 1×1 convolution, and the size becomes 20×20×512. At this time, the number of feature channels is halved. The features at this time are respectively sent to the 5×5 pooling layer, the 9×9 pooling layer, and the 13×13 pooling layer. Then, the three features of size 20×20×512 that have passed through different pooling layers are concat-ed with the 20×20×512 features that have not passed through the pooling layer in the channel dimension to obtain features of size 20×20×2048.

[0045] The feature with a size of 20×20×2048 is further reduced in number of channels through a 1×1 convolution and a 3×3 convolution to become a feature with a size of 20×20×512;

[0046] At this time, the input feature undergoes 1×1 convolution in another path, and the size becomes 20×20×512. The two features of size 20×20×512 are concat-concatenated in the channel dimension to obtain 20×20×1024 features. Finally, a 1×1 convolution is performed to reduce the channel and the output size is 20×20×512.

[0047] Preferably, the ELAN module includes a Conv convolution module and a Concat splicing module;

[0048] The Conv convolution module is a collection of convolution operations of corresponding sizes, batch normalization, and SiLU activation function operations; the convolution kernel sizes include 1×1 and 3×3;

[0049] The input feature size of the ELAN module is 40×40×512. The processing of the feature is divided into two paths. One path undergoes 1×1 convolution and the size becomes 40×40×256. No more convolution operation is performed and it waits for the final concat operation with other features. This feature is recorded as feature 1.

[0050] The other path first also undergoes a 1×1 convolution, and its size becomes 40×40×256. At this time, it is divided into two paths again. One path no longer undergoes convolution and waits for the final concat splicing operation with other features, recorded as feature 2, and the other path undergoes a 3×3 convolution, and its size becomes 40×40×128; at this time, it is divided into two paths again, one path waits for the final concat splicing operation, recorded as feature 3, and the other path continues to undergo 3×3 convolution, and its size remains unchanged;

[0051] Similarly, after three more 3×3 convolutions, features 4, 5, and 6 are generated at the same time; the sizes of features 1 and 2 are 40×40×256, and the sizes of features 3 to 6 are 40×40×128. The six features are concat-concatenated in the channel dimension to obtain features of size 40×40×1024; the features of size 40×40×1024 are further subjected to a 1×1 convolution to reduce the number of channels, and the output features of size 40×40×256 are obtained.

[0052] Compared with the prior art, the present invention has the following advantages and technical effects:

[0053] The present invention introduces the Atrous spatial pyramid pooling module (Atrous spatial pyramid pooling module). Al pyramid pooling (ASPP) improves the feature extraction part of the RNGDet model, which can expand the receptive field of the model's extracted image features, capture road information at different scales in the image without adding additional parameters, and improve the robustness and accuracy of the model's extraction results on small-scale objects and complex scenes;

[0054] The present invention introduces an improved feature pyramid network structure (ModiF I The ed Feature Pyramid Network (MFPN) enhances the feature fusion part of the RNGDet model, uses convolution, pooling, upsampling and other operations to process input features of different scales, and makes full use of the feature information extracted by the void space pyramid pooling module to improve the model's prediction performance for road segmentation maps and intersection segmentation maps. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] The accompanying drawings, which constitute part of this application, are intended to provide a further understanding of this application. The exemplary embodiments and descriptions of this application are intended to explain this application and do not constitute an improper limitation on this application. In the accompanying drawings:

[0056] Figure 1 Schematic diagram of a method flow in an embodiment of the present invention;

[0057] Figure 2 A schematic diagram of an algorithm for updating a road network graph according to an embodiment of the present invention;

[0058] Figure 3 Schematic diagram of a dilated spatial pyramid pooling network according to an embodiment of the present invention;

[0059] Figure 4 Schematic diagram of an improved feature pyramid structure according to an embodiment of the present invention;

[0060] Figure 5 Schematic diagram of the structure of the cross-stage feature pyramid pooling module (SPPCSPC) according to an embodiment of the present invention;

[0061] Figure 6 Schematic diagram of the structure of a multi-gradient fusion feature extraction module (ELAN) according to an embodiment of the present invention;

[0062] Figure 7 Schematic diagram of the maximum pooling module (MP) structure of an embodiment of the present invention. DETAILED DESCRIPTION

[0063] It should be noted that, in the absence of conflict, the embodiments and features of the embodiments in this application can be combined with each other. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0064] It should be noted that the steps shown in the flowcharts of the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and that, although a logical order is shown in the flowcharts, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0065] like Figure 1-7 As shown, this embodiment provides a method for extracting roads from remote sensing images, including:

[0066] Acquire remote sensing road images and crop the remote sensing road images to obtain regions of interest;

[0067] The backbone network of the improved baseline model RNGDet is used to extract the deep features of the region of interest and obtain the first feature tensor;

[0068] Based on the first feature tensor, an improved feature pyramid network structure is used to perform road segmentation prediction, and a distribution map of road sections, road intersections, and broken road endpoints is obtained;

[0069] A road network graph is generated by detecting vertices and connecting vertices iteratively according to historical information, a historical trajectory graph is obtained iteratively according to the road network graph, the historical trajectory graph is clipped to an area of ​​interest, and the clipped historical trajectory graph is rasterized to obtain a rasterized graph;

[0070] Input the road segment distribution map, the road intersection and broken road endpoint distribution map, and the rasterized map into the backbone network, and output a second feature tensor;

[0071] The first feature tensor and the second feature tensor are concatenated to obtain the target feature tensor, which is then sent to the Transformer structure as the feature extraction result of the backbone network for prediction, and the road network map is updated according to the prediction results.

[0072] Furthermore, the process of cropping the remote sensing road image to obtain the region of interest includes:

[0073] Select the current vertex v from the candidate vertex set C t , with the current vertex v t As the center, crop a square area of ​​size 3×L×L, where L is 256.

[0074] Further optimization scheme, such as Figure 1 As shown in Figure 1, the architecture of the model in this embodiment is divided into a feature extraction and fusion part, and an inference and prediction part. The feature extraction and fusion part starts with the input of the remote sensing image. The input of the model is a color remote sensing image of size 3×4096×4096 (with RGB three channels), denoted as I AThe model iteratively generates a road network graph based on historical information, so the model includes a graph G that records the historical trajectory of the intelligent agent. The model iteratively generates a road network graph by detecting and connecting vertices, so the model includes a set of candidate vertices C.

[0075] Because I A The size of is large, and the effect of directly extracting global features is poor, so the model first A A region of interest (ROI) is cropped on the top. The principle of selecting ROI is based on the current vertex v t (v t Select from the candidate vertex set C) as the center and crop a square area of ​​size 3×L×L (L is 256). Then use the backbone network to extract the deep features of ROI, denoted as F I .

[0076] The backbone network of the improved model consists of ResNet-101 and ASPP modules. Based on the extracted deep features F I , the MFPN structure is used to predict the segmentation maps S and I, where S represents the distribution map of road segments, and I represents the distribution map of road intersections and broken road endpoints. Both segmentation tasks are binary segmentation.

[0077] In order to obtain the historical trajectory information of the intelligent agent to assist the road network detection, we use the clipping A The image G is cropped in the form of ROI, and the cropped G is rasterized into an image H of size 1×L×L for subsequent splicing operations. With S, I and H as input, the backbone network outputs the feature tensor F H 。 I and F H The two feature tensors are concatenated to obtain the target feature tensor F. This target feature tensor F is then fed into the Transformer structure as the result of feature extraction by the backbone network. At this point, the model enters the inference and prediction phase.

[0078] Furthermore, before the target feature tensor is fed into the Transformer structure as the feature extraction result of the backbone network for prediction, the following steps are also included: converting the target feature tensor into a sequence of smaller tensors;

[0079] The process of converting the target feature tensor into a sequence of smaller tensors includes:

[0080] Utilizing the permutation invariance of the Transformer architecture, the target feature tensor is fused with a fixed positional encoding, and the fused tensor sequence is fed into the Transformer encoder structure. The output tensor scale of the Transformer encoder is the same as the input tensor scale.

[0081] The output tensor is fed into the Transformer decoder structure, which receives the encoder’s output sequence and a set of vertex queries Q = {q i} i＝1 N , and predict the embedding tensors of N adjacent vertices; each vertex query q i is a learned embedding tensor and produces a predicted adjacent vertex. Transformer outputs the maximum a posteriori estimate of the vertex at the next time step. Each output tensor is decoded into an effective probability p through the feedforward network. t+1 i and the two-dimensional vertex coordinates v t+1 i , p t+1 i Indicates v t+1 i The probability of being valid, valid means it should be added to the road network graph;

[0082] If there are M (M≤N) valid prediction vertices, the set γ={v t+1 i} i＝1 M , then for each valid prediction vertex v t+1 i , by changing v t+1 i Add to the vertex set V and connect the newly added t+1 i With v t The edges are added to the edge set E to update the road network graph.

[0083] Furthermore, the process of sending the target feature tensor as the feature extraction result of the backbone network into the Transformer structure for prediction includes:

[0084] The tensor output by the Transformer decoder is fed into the prediction module to ultimately determine the next iteration vertex v of the intelligent agent. t+1 First, we initialize the candidate vertex set C by finding the local peaks of the distribution graph of road intersections and broken road endpoints, and then improve the baseline model RNGDet to extract a vertex v from the candidate vertex set C. t , with vertex v t Crop the input image to the center and predict the valid vertex set γ in the next step, where the number of valid vertices in γ |γ| = M;

[0085] According to the different values ​​of M, different methods are adopted to update the road network graph.

[0086] Further, if Figure 2 As shown in Figure 2, depending on the value of M, the process of updating the road network graph using different methods includes:

[0087] When M=0, it means there is no road ahead. Therefore, if the candidate vertex set C is not empty, the model will stop processing the current road and turn to process other roads. The model will re-take a vertex v from the vertex set C. t , re-predict;

[0088] When M=1, it means that there is only one possible prediction point in front of the model, and the model predicts along a road; at this time, the model will v t+1 1 Add the vertex set V and connect the edge (v t , v t+1 1 ) joins the edge set E, and then the intelligent agent moves to v t+1 1 , with v t+1 1 Make predictions for the center and iteratively update the road network graph until an intersection M>1 or a disconnected road M=0 is encountered;

[0089] When M>1, it means that the model encounters a road intersection and needs to generate new predicted candidate vertices in multiple directions. At this time, the model will update the road network graph, put all vertices in the predicted vertex set γ into the selected vertex set C, and reselect an initial vertex v from the selected vertex set C. t Make predictions; in the inference stage, the model continues to run the iterative algorithm to generate a road network graph point by point. If and only if the candidate initial vertex set C is empty, the model ends running the algorithm and outputs the generated road network graph G = (V, E).

[0090] The atrous convolution principle used in this embodiment is to insert some holes into the traditional convolution kernel, so that the convolution kernel can span a larger area and increase the receptive field without increasing the parameters and computational complexity. It not only does not reduce the resolution of the feature map, reduce the loss of detail information and the decline in positioning accuracy, but also can obtain multi-scale context information and improve the prediction performance of small-scale objects.

[0091] Furthermore, the backbone network of the improved baseline model RNGDet includes ResNet-101 and the dilated spatial pyramid pooling module;

[0092] like Figure 3As shown in the figure, the Atrous Spatial Pyramid Pooling (ASPP) module uses multiple dilated convolutional layers with different dilation rates in parallel to extract features. The features extracted by each convolutional layer are further processed in a separate branch and fused to generate the final result. This model simultaneously processes the same input feature map using 3×3 dilated convolutional layers with dilation rates of 6, 12, 18, and 24. These outputs are concatenated to form a richer and more diverse feature representation, improving the robustness and accuracy of segmentation results for small objects and complex scenes.

[0093] Further, if Figure 4 As shown in the figure, the improved feature pyramid network structure includes three parts: bottom-up, top-down, and horizontal connection; the structure contains many operation modules such as convolution and upsampling.

[0094] The bottom-up structure consists of layers 51 to 67. The model receives input features from the backbone through layers 51, 55, and 67, with corresponding sizes of 20×20×1024, 40×40×256, and 80×80×128, respectively.

[0095] The 51st layer of the model receives a 20×20×1024 feature input, which is then transformed into a 20×20×512 size after passing through a cross-stage feature pyramid pooling module SPPCSPC consisting of a Conv convolution module, an MP maximum pooling module, and a Concat splicing module.

[0096] After 52 layers of 1×1 convolution, the size becomes 20×20×256. After 53 layers of 2x upsampling module, the size becomes 40×40×256. The 40×40×256 features at this time are concatenated with the 40×40×256 input features from the backbone layer 55 in the channel dimension to obtain features of size 40×40×512.

[0097] The features of size 40×40×512 are transformed into 40×40×256 through a multi-gradient fusion feature extraction module ELAN. After that, the features are subjected to the same 1×1 convolution and 2x upsampling as the convolution and upsampling operations of layers 52-53. The feature size is first transformed into 40×40×128 and then into 80×80×128. The 80×80×128 features at this time are concatenated with the 80×80×128 input features of layer 67 from the backbone in the channel dimension to obtain features of size 80×80×256.

[0098] Furthermore, the improved feature pyramid network structure includes three parts: bottom-up, top-down, and horizontal connection;

[0099] The top-down structure consists of layers 68 to 101. Features of size 80×80×256 first pass through an ELAN module, reducing their size to 80×80×128. They then pass through an MP max pooling module, which integrates max pooling with multiple convolutional and concatenated operations. The MP max pooling module includes a Conv convolution module, an MP max pooling module, and a Concat concatenation module. The MP max pooling module also receives features of size 40×40×256 output by the ELAN modules in layers 56-63. The MP max pooling module outputs features of size 40×40×512.

[0100] The features of size 40×40×512 continue to pass through an ELAN module, and the size becomes 40×40×256, and then pass through an MP maximum pooling module. The MP maximum pooling module also receives the features of size 20×20×512 output by the 51-layer SPPCSPC module. The MP maximum pooling module outputs features of size 20×20×1024;

[0101] The features with the size of 20×20×1024 are passed through an ELAN module again and the size becomes 20×20×512;

[0102] The top-down structure includes three parts that directly output to the detection head: the ELAN modules at layers 68-75, which output features of size 80×80×128; the ELAN modules at layers 81-88, which output features of size 40×40×256; and the ELAN modules at layers 94-101, which output features of size 20×20×512.

[0103] The MFPN architecture can be categorized into bottom-up, top-down, and horizontally connected structures, representing a typical FPN feature pyramid structure. By performing convolution, pooling, and upsampling on features of different sizes, and fusing features from different layers, the model's feature extraction capabilities are continuously enhanced.

[0104] Further, if Figure 5 As shown in the figure, the SPPCSPC module includes the Conv convolution module, the MP maximum pooling module and the Concat splicing module;

[0105] The Conv convolution module is a collection of convolution operations of corresponding sizes, batch normalization operations (BN), and SiLU activation function operations. The convolution kernel sizes include 1×1 and 3×3.

[0106] The pooling sizes of the MP maximum pooling module include 5×5, 9×9, and 13×13;

[0107] The input feature size of the SPPCSPC module is 20×20×1024. The feature processing is divided into two paths. One path first undergoes 1×1 convolution, 3×3 convolution, and 1×1 convolution, and the size becomes 20×20×512. At this time, the number of feature channels is halved. The features at this time are sent to the 5×5 pooling layer, 9×9 pooling layer, and 13×13 pooling layer respectively. Then, the three features of size 20×20×512 that have passed through different pooling layers are concat-ed with the 20×20×512 features that have not passed through the pooling layer in the channel dimension to obtain features of size 20×20×2048.

[0108] The feature with a size of 20×20×2048 is further reduced through a 1×1 convolution and a 3×3 convolution to reduce the number of channels and become a feature with a size of 20×20×512;

[0109] At this time, the input feature undergoes 1×1 convolution in another path, and the size becomes 20×20×512. The two features of size 20×20×512 are concat-concatenated in the channel dimension to obtain 20×20×1024 features. Finally, a 1×1 convolution is performed to reduce the channel and the output size is 20×20×512.

[0110] Further, if Figure 6 As shown, the ELAN module includes the Conv convolution module and the Concat splicing module;

[0111] Similar to the SPPCSPC module, the Conv convolution module is a collection of convolution operations of corresponding sizes, batch normalization operations (BN), and SiLU activation function operations; the convolution kernel sizes include 1×1 and 3×3;

[0112] The input feature size of the ELAN module is 40×40×512. The feature processing is divided into two paths. One path undergoes 1×1 convolution and the size becomes 40×40×256. The convolution operation is no longer performed and waits for the final concat operation with other features. This feature is recorded as feature 1.

[0113] The other path first also undergoes a 1×1 convolution, and its size becomes 40×40×256. At this time, it is divided into two paths again. One path no longer undergoes convolution and waits for the final concat splicing operation with other features, recorded as feature 2, and the other path undergoes a 3×3 convolution, and its size becomes 40×40×128; at this time, it is divided into two paths again, one path waits for the final concat splicing operation, recorded as feature 3, and the other path continues to undergo 3×3 convolution, and its size remains unchanged;

[0114] Similarly, after three more 3×3 convolutions, features 4, 5, and 6 are generated at the same time; the sizes of features 1 and 2 are 40×40×256, and the sizes of features 3 to 6 are 40×40×128. The six features are concat-concatenated in the channel dimension to obtain features of size 40×40×1024; the features of size 40×40×1024 are further subjected to a 1×1 convolution to reduce the number of channels, and the output features of size 40×40×256 are obtained.

[0115] Further, if Figure 7 As shown in the figure, the MP Max Pooling module (MP) includes a Conv convolution module (with kernel sizes of 1×1 and 3×3), an MP Max Pooling unit (with a pooling size of 2×2), and a Concat module. Similar to the SPPCSPC module, the Conv convolution module in MP refers to a collection of convolution operations of corresponding sizes, batch normalization (BN), and SiLU activation function operations. In the MP Max Pooling unit, the input feature size is 80×80×128, and the processing of this feature is divided into two paths. One path passes through a 2×2 pooling layer, reducing the feature size to 40×40×128, and then passes through a 1×1 convolution, keeping the feature size unchanged. The feature obtained at this time is recorded as feature 1; the other path passes through a 1×1 convolution and a 3×3 convolution successively. The size remains unchanged after the 1×1 convolution, and the size is reduced to 40×40×128 after the 3×3 convolution. The feature obtained at this time is recorded as feature 2; a feature of size 40×40×256 is input from the backbone, recorded as feature 3; Concat splicing operation is performed on feature 1, feature 2, and feature 3 in the channel dimension to obtain a feature of size 40×40×512 and output it.

[0116] The MFPN structure, which consists of modules such as SPPCSPC, ELAN, and MP, fully utilizes the feature information extracted by the dilated spatial pyramid pooling module by performing convolution, pooling, and upsampling on the input features, thereby improving the model's prediction performance for road segmentation maps and intersection segmentation maps.

[0117] The above are merely preferred embodiments of the present application, but the scope of protection of the present application is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in this application should be included in the scope of protection of the present application. Therefore, the scope of protection of the present application should be based on the scope of protection of the claims.

Claims

1. A method for extracting roads from remote sensing images, characterized in that: include: Acquiring a remote sensing road image, and cropping the remote sensing road image to obtain a region of interest; Extracting the deep features of the region of interest using the backbone network of the improved baseline model RNGDet to obtain a first feature tensor; Based on the first feature tensor, an improved feature pyramid network structure is used to perform road segmentation prediction to obtain a road section distribution map and a road intersection and broken road endpoint distribution map; Iteratively generating a road network graph by detecting vertices and connecting vertices according to historical information, iteratively obtaining a historical trajectory graph according to the road network graph, clipping an area of ​​interest on the historical trajectory graph, and rasterizing the clipped historical trajectory graph to obtain a rasterized graph; Inputting the road segment distribution map, the road intersection and broken road endpoint distribution map, and the rasterized map into the backbone network, and outputting a second feature tensor; Concatenate the first feature tensor and the second feature tensor to obtain a target feature tensor, send the target feature tensor as the feature extraction result of the backbone network into the Transformer structure for prediction, and update the road network graph according to the prediction result; The backbone network of the improved baseline model RNGDet includes ResNet-101 and a dilated spatial pyramid pooling module; The dilated spatial pyramid pooling module uses 3×3 dilated convolutional layers with dilation rates of 6, 12, 18, and 24 in parallel to process the same input feature map. The features extracted by each convolutional layer are further processed in a separate branch and fused to generate the final result. The improved feature pyramid network structure includes three parts: bottom-up, top-down, and horizontal connection; The bottom-up structure consists of layers 51 to 67. The model receives input features from the backbone through layers 51, 55, and 67, with corresponding sizes of 20×20×1024, 40×40×256, and 80×80×128, respectively. The 51st layer of the model receives a 20×20×1024 feature input, which is then transformed into a 20×20×512 size after passing through a cross-stage feature pyramid pooling module SPPCSPC consisting of a Conv convolution module, an MP maximum pooling module, and a Concat splicing module. After 52 layers of 1×1 convolution, the size becomes 20×20×256. After 53 layers of 2x upsampling module, the size becomes 40×40×256. The 40×40×256 features at this time are concatenated with the 40×40×256 input features from the backbone layer 55 in the channel dimension to obtain features of size 40×40×512. The 40×40×512 feature is passed through a multi-gradient fusion feature extraction module ELAN and its size is reduced to 40×40×256. The feature is then subjected to the same 1×1 convolution and 2x upsampling as the convolution and upsampling operations of layers 52-53. The feature size is first reduced to 40×40×128 and then to 80×80×128. The 80×80×128 feature at this time is concat-ed with the 80×80×128 input feature from the 67th layer of the backbone in the channel dimension to obtain a feature of size 80×80×256. The improved feature pyramid network structure includes three parts: bottom-up, top-down, and horizontal connection; The top-down structure consists of layers 68 to 101. Features of size 80×80×256 first pass through an ELAN module to become 80×80×128, and then pass through an MP maximum pooling module. The MP maximum pooling module also receives the features of size 40×40×256 output by the ELAN modules of layers 56-63. The MP maximum pooling module outputs features of size 40×40×512. The features of size 40×40×512 continue to pass through an ELAN module, and the size becomes 40×40×256, and then pass through an MP maximum pooling module. The MP maximum pooling module also receives the features of size 20×20×512 output by the 51-layer SPPCSPC module, and the MP maximum pooling module outputs features of size 20×20×1024; The feature with the size of 20×20×1024 passes through an ELAN module again and becomes 20×20×512; The top-down structure includes three parts that directly output to the detection head: the ELAN modules at layers 68-75, which output features of size 80×80×128; The ELAN modules of layers 81-88 output features of size 40×40×256; and the ELAN modules of layers 94-101 output features of size 20×20×512.

2. The method for extracting roads from remote sensing images according to claim 1, wherein: The process of cropping the remote sensing road image to obtain the region of interest includes: Select the current vertex v from the candidate vertex set C t , taking the current vertex v t As the center, crop a square area of ​​size 3×L×L, where L is 256.

3. The method for extracting roads from remote sensing images according to claim 1, wherein: Before sending the target feature tensor as the feature extraction result of the backbone network into the Transformer structure for prediction, the method further includes: converting the target feature tensor into a smaller tensor sequence; The process of converting the target feature tensor into a sequence of smaller tensors includes: Utilizing the permutation invariance of the Transformer architecture, the target feature tensor is fused with a fixed positional encoding, and the fused tensor sequence is fed into a Transformer encoder structure, where the output tensor scale of the Transformer encoder is the same as the input tensor scale; The output tensor is fed into the Transformer decoder structure, which receives the encoder's output sequence and a set of vertex queries Q = {q i } i=1 N , and predict the embedding tensors of N adjacent vertices; each vertex query q i is a learned embedding tensor and produces a predicted adjacent vertex. Transformer outputs the maximum a posteriori estimate of the vertex at the next time step. Each output tensor is decoded into an effective probability p through the feedforward network. t+1 i and the two-dimensional vertex coordinates v t+1 i , p t+1 i Indicates v t+1 i The probability of being valid, valid means it should be added to the road network graph; If there are M (M≤N) valid prediction vertices, the set γ={v t+1 i } i=1 M , then for each valid prediction vertex v t+1 i , by changing v t+1 i Add to the vertex set V and connect the newly added t+1 i With v t The edge of is added to the edge set E to update the road network graph.

4. The method for extracting roads from remote sensing images according to claim 1, wherein: The process of sending the target feature tensor as the feature extraction result of the backbone network into the Transformer structure for prediction includes: The tensor output by the Transformer decoder is fed into the prediction module to ultimately determine the next iteration vertex v of the intelligent agent. t+1 First, we initialize the candidate vertex set C by finding the local peaks of the distribution graph of road intersections and broken road endpoints, and then improve the baseline model RNGDet to extract a vertex v from the candidate vertex set C. t , with vertex v t Crop the input image to the center and predict the valid vertex set γ in the next step, where the number of valid vertices in γ is |γ|=M; According to the different values ​​of M, different methods are adopted to update the road network graph.

5. The method for extracting roads from remote sensing images according to claim 1, wherein: Depending on the value of M, the process of updating the road network graph using different methods includes: When M=0, it means there is no road ahead. Therefore, if the candidate vertex set C is not empty, the model will stop processing the current road and turn to process other roads. The model will re-take a vertex v from the vertex set C. t , re-predict; When M=1, it means that there is only one possible prediction point in front of the model, and the model predicts forward along a road; at this time, the model will v t+1 1 Add the vertex set V and connect the edge (v t , v t+1 1 ) joins the edge set E, and then the intelligent agent moves to v t+1 1 , with v t+1 1 Make predictions for the center and iteratively update the road network graph until an intersection M>1 or a disconnected road M=0 is encountered; When M>1, it means that the model encounters a road intersection and needs to generate new predicted candidate vertices in multiple directions. At this time, the model will update the road network graph, put all vertices in the predicted vertex set γ into the selected vertex set C, and reselect an initial vertex v from the selected vertex set C. t Make predictions; in the inference stage, the model continues to run the iterative algorithm to generate a road network graph point by point. If and only if the candidate initial vertex set C is empty, the model ends running the algorithm and outputs the generated road network graph G=(V, E).

6. The method for extracting roads from remote sensing images according to claim 1, wherein: The SPPCSPC module includes a Conv convolution module, an MP maximum pooling module and a Concat splicing module; The Conv convolution module is a collection of convolution operations, batch normalization operations, and SiLU activation function operations of corresponding sizes, and the convolution kernel sizes include 1×1 and 3×3; The pooling sizes of the MP maximum pooling module include 5×5, 9×9 and 13×13; The input feature size of the SPPCSPC module is 20×20×1024. The processing of the features is divided into two paths. One path first undergoes 1×1 convolution, 3×3 convolution, and 1×1 convolution, and the size becomes 20×20×512. At this time, the number of feature channels is halved. The features at this time are respectively sent to the 5×5 pooling layer, the 9×9 pooling layer, and the 13×13 pooling layer. Then, the three features of size 20×20×512 that have passed through different pooling layers are concat-ed with the 20×20×512 features that have not passed through the pooling layer in the channel dimension to obtain features of size 20×20×2048. The feature with a size of 20×20×2048 is further reduced in number of channels through a 1×1 convolution and a 3×3 convolution to become a feature with a size of 20×20×512; At this time, the input feature undergoes 1×1 convolution in another path, and the size becomes 20×20×512. The two features of size 20×20×512 are concat-concatenated in the channel dimension to obtain 20×20×1024 features. Finally, a 1×1 convolution is performed to reduce the channel and the output size is 20×20×512.

7. The method for extracting roads from remote sensing images according to claim 1, wherein: The ELAN module includes a Conv convolution module and a Concat splicing module; The Conv convolution module is a collection of convolution operations of corresponding sizes, batch normalization, and SiLU activation function operations; the convolution kernel sizes include 1×1 and 3×3; The input feature size of the ELAN module is 40×40×512. The processing of the feature is divided into two paths. One path undergoes 1×1 convolution and the size becomes 40×40×256. No more convolution operation is performed and it waits for the final concat operation with other features. This feature is recorded as feature 1. The other path first also undergoes a 1×1 convolution, and its size becomes 40×40×256. At this time, it is divided into two paths again. One path no longer undergoes convolution and waits for the final concat splicing operation with other features, recorded as feature 2, and the other path undergoes a 3×3 convolution, and its size becomes 40×40×128; at this time, it is divided into two paths again, one path waits for the final concat splicing operation, recorded as feature 3, and the other path continues to undergo 3×3 convolution, and its size remains unchanged; Similarly, after three more 3×3 convolutions, features 4, 5, and 6 are generated at the same time; the sizes of features 1 and 2 are 40×40×256, and the sizes of features 3 to 6 are 40×40×128. The six features are concat-concatenated in the channel dimension to obtain features of size 40×40×1024; the features of size 40×40×1024 are further subjected to a 1×1 convolution to reduce the number of channels, and the output features of size 40×40×256 are obtained.

Images