Building vector end-to-end extraction method and system based on state updating

Abstract

The invention discloses a building vector end-to-end extraction method and system based on state updating, and relates to the field of data models.The method comprises the steps that building features in a to-be-detected remote sensing image are extracted, and an initial feature map containing multi-scale information is obtained; based on the initial feature map, calculating a building center point by using a center point prediction head, and calculating a relative offset between a building contour and the building center point by using an offset prediction head to obtain an initial contour; on the basis of the initial feature map, the initial contour and ConvGRU, continuously iteratively updating the building contour and the contour state by adopting feature sampling, contour updating and multi-scale feature fusion modes, and determining a final contour; and carrying out vertex classification and point attention calculation on the final contour in sequence, and determining a building vector in the remote sensing image to be detected. According to the method, the building vector in the remote sensing image is accurately extracted, and the shape of the extracted building vector is more regular.

Description

Technical Field

[0001] The present application relates to the technical field of data models, and in particular to a method and system for end-to-end extraction of building vectors based on status updating. Background Art

[0002] Building extraction is one of the key tasks in the field of remote sensing and is of great significance in application areas such as urban planning, disaster management and population estimation. Traditional methods mainly rely on image segmentation and regularization techniques to generate building vectors. Given the complexity of building structures and the limitations of imaging conditions, segmentation models often produce irregular results and it is difficult to accurately generate building vectors.

[0003] At present, the research on building extraction also includes semantic segmentation methods and instance segmentation methods. The semantic segmentation method distinguishes between building and non-building areas by classifying image pixels. The result can effectively cover the building area, but it is difficult to accurately distinguish connected building instances; the instance segmentation method locates a single building through the target detection box and generates the corresponding building segmentation area. The result is closer to the building vector in form. Although the semantic segmentation method and the instance segmentation method have achieved excellent results in building extraction, since these methods are segmented at the image pixel level, it is easy to produce jagged and irregular shapes in the building boundary area, so the final building vector is not accurate. Summary of the invention

[0004] The purpose of this application is to provide a state-updating based end-to-end building vector extraction method and system, which accurately extracts building vectors in remote sensing images, and the extracted building vectors have more regular shapes.

[0005] To achieve the above objectives, this application provides the following solutions.

[0006] In a first aspect, the present application provides an end-to-end method for extracting building vectors based on status update, and the end-to-end method for extracting building vectors based on status update is as follows.

[0007] Extract the building features in the remote sensing image to be detected and obtain an initial feature map containing multi-scale information.

[0008] Based on the initial feature map, the center point of the building is calculated using a center point prediction head, and the relative offset between the building outline and the building center point is calculated using an offset prediction head to obtain an initial outline.

[0009] Based on the initial feature map, the initial contour and ConvGRU, feature sampling, contour updating and multi-scale feature fusion are adopted to continuously iteratively update the building contour and contour state, and determine the final contour.

[0010] The final contour is sequentially subjected to vertex classification and point attention calculation to determine the building vector in the remote sensing image to be detected.

[0011] In a second aspect, the present application further provides a state-updating based end-to-end building vector extraction system, and the state-updating based end-to-end building vector extraction system is as follows.

[0012] The initial feature extraction module is used to extract the building features in the remote sensing image to be detected and obtain an initial feature map containing multi-scale information.

[0013] The contour initialization module is used to calculate the building center point based on the initial feature map using the center point prediction head, and calculate the relative offset between the building contour and the building center point using the offset prediction head to obtain the initial contour.

[0014] The ConvGRU-based contour updating module is used to iteratively update the building contour and contour state based on the initial feature map, the initial contour and the ConvGRU, using feature sampling, contour updating and multi-scale feature fusion methods, and determine the final contour.

[0015] The point attention guided vertex classification module is used to perform vertex classification and point attention calculation on the final contour in sequence to determine the building vector in the remote sensing image to be detected.

[0016] According to the specific embodiments provided in this application, this application discloses the following technical effects.

[0017] This application uses the Convolutional Gated Recurrent Unit (ConvGRU) to update the building outline as a whole, learns long-range dependency information through its gating mechanism, makes full use of the global feature information of the building outline as a whole and the long-range dependency information before and after the update, and establishes the connection between the building outline states before and after the update, thereby avoiding the phenomenon of oscillation of the building vertices at the edge of the outline. At the same time, this application also proposes a vertex classification method guided by point attention, which guides the model output from straight lines to corner points through point attention, enhances the focus on building corner points, and facilitates the formation of more regular and accurate building vectors. Therefore, this application can accurately and regularly extract building vectors in remote sensing images. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0019] Figure 1 A flowchart of an end-to-end building vector extraction method based on status update provided in an embodiment of the present application.

[0020] Figure 2 A schematic diagram of profile initialization provided in an embodiment of the present application.

[0021] Figure 3 A schematic diagram of a contour update based on ConvGRU provided in an embodiment of the present application.

[0022] Figure 4 A schematic diagram of the ConvGRU operation provided in an embodiment of the present application.

[0023] Figure 5 A schematic diagram of vertex classification guided by point attention provided in an embodiment of the present application.

[0024] Figure 6 A schematic diagram of an end-to-end building vector extraction system based on state update provided in an embodiment of the present application. DETAILED DESCRIPTION

[0025] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0026] There are also end-to-end methods for extracting building vectors from remote sensing images, but they generally only use local features to generate building vectors, ignoring global information, which can easily lead to shape distortion.

[0027] Specifically, according to the different ways of generating building vectors, end-to-end extraction can be divided into corner point sorting method and contour regression method. The corner point sorting method regards the building corner points as a directed sequence, extracts features through the Convolutional Neural Network (CNN), generates corner point positions and judges the relationship between corner points to obtain building vectors; the contour regression method predicts the building vector as a fixed number of vertices, obtains the building corner points through vertex classification, and generates the building vector. However, the corner point sorting method only considers the use of previously predicted corner points and directions to generate subsequent corner points, and the contour regression method generally uses ring convolution or one-dimensional convolution to extract local features for contour update, which cannot fully perceive the global information of the building, resulting in distortion of the final overall shape. Therefore, it is still a challenge to design an accurate and efficient end-to-end extraction method for building vectors from remote sensing images.

[0028] The purpose of this application is to provide a state-updating based end-to-end building vector extraction method and system, which accurately extracts building vectors in remote sensing images, and the extracted building vectors have more regular shapes.

[0029] In order to make the above-mentioned objects, features and advantages of the present application more obvious and easy to understand, the present application is further described in detail below in conjunction with the accompanying drawings and specific implementation methods.

[0030] In an exemplary embodiment, Figure 1 As shown, a method for extracting building vectors from end to end based on state updating is provided. The method for extracting building vectors from end to end based on state updating is specifically as follows.

[0031] Step S1: extracting building features from the remote sensing image to be detected, and obtaining an initial feature map containing multi-scale information.

[0032] In this embodiment, the extraction of architectural features refers to the Deepsnake framework, and the specific backbone network used is DLA-34. DLA-34 is used to obtain pyramid-level architectural features to obtain an initial feature map containing multi-scale information.

[0033] Step S2: Based on the initial feature map, the center point prediction head is used to calculate the center point of the building, and the offset prediction head is used to calculate the relative offset between the building outline and the building center point to obtain the initial outline (see Figure 2 ).

[0034] In this embodiment, the center point prediction head includes a 3×3 convolution layer, a ReLU activation function, a 1×1 convolution layer and a Sigmoid activation function; the offset prediction head includes a 3×3 convolution layer, a ReLU activation function and a 1×1 convolution layer. The calculation method of the initial contour is as follows.

[0035] .

[0036] In the formula, is the initial contour, is the coordinate of the building center point, For the building outline i The relative offset coordinates of the points and the center point of the building, and the building outline is N It consists of points, is an empirical parameter.

[0037] Step S3: Based on the initial feature map, initial contour and ConvGRU, feature sampling, contour updating and multi-scale feature fusion are used to iteratively update the building contour and contour status, and determine the final contour.

[0038] like Figure 3 As shown, step S3 is specifically as follows.

[0039] Step S31: Use bilinear interpolation to project the initial contour onto the initial feature map and perform feature sampling. The corresponding position features are spliced ​​to generate spliced ​​features. .

[0040] Step S32: Use ring convolution to concatenate features Generating Vertex Features .

[0041] Step S33: Vertex features And the initial contour state is input into ConvGRU to get the next contour state and building outline offset .

[0042] like Figure 4 As shown, the next contour state The calculation method of is as follows.

[0043] .

[0044] .

[0045] .

[0046] .

[0047] In the formula, is the previous contour state, is the candidate contour state, To update the gate, To reset the gate, , and are the corresponding weights, is the convolution operation, is the element-wise multiplication, and is the activation function.

[0048] Building outline offset It is calculated by the Feedforward Neural Network (FNN) in ConvGRU. The initial contour state is obtained by activating the initial vertex features through the tanh function.

[0049] Step S34: Based on the building outline offset For the initial contour Update the contour to get the next contour .

[0050] .

[0051] Step S35: Use the edge-aware head to extract geometric information of the initial feature map to obtain a refined contour feature map.

[0052] The edge-aware head consists of a 3×3 convolutional layer, a ReLU activation function and a 1×1 convolutional layer. The edge-aware head can extract richer geometric information and refine the edge contours of the building.

[0053] Step S36: Perform multi-scale feature fusion on the initial feature map and the refined contour feature map to obtain the next feature map F new .

[0054] Multi-scale feature fusion draws on the architecture of feature pyramid to deeply fuse the initial feature map with the refined contour feature map generated by the edge perception head.

[0055] Repeat the above steps S31 to S36, continuously update the initial contour with the next contour, update the initial feature map with the next feature map, and update the initial contour state with the next contour state, until the maximum number of iterations is reached and the final contour is output. .

[0056] Step S4: Final contour Vertex classification and point attention calculation are performed in sequence to determine the building vector in the remote sensing image to be detected.

[0057] like Figure 5 As shown, step S4 is specifically as follows.

[0058] Step S41: Use a three-layer fully connected network to align the final contour Perform vertex classification to obtain initial classification results .

[0059] Step S42: Initial classification results Perform point attention weighting to obtain the final classification result .

[0060] The final classification result is calculated as follows.

[0061] .

[0062] .

[0063] Where Q, K and V are query vector, key vector and value vector respectively. for The dimension of is the activation function. The vertex classification process in the above process is actually the process of removing redundant points.

[0064] Step S43: constructing a building vector in the remote sensing image to be detected based on the points classified as corner points in the final classification result.

[0065] In another exemplary embodiment, Figure 6 As shown, a state-updating based end-to-end building vector extraction system is provided, and the state-updating based end-to-end building vector extraction system includes the following modules.

[0066] The initial feature extraction module is used to extract the building features in the remote sensing image to be detected and obtain an initial feature map containing multi-scale information.

[0067] The contour initialization module is used to calculate the building center point based on the initial feature map using the center point prediction head, and calculate the relative offset between the building contour and the building center point using the offset prediction head to obtain the initial contour.

[0068] The ConvGRU-based contour update module is used to iteratively update the building contour and contour status based on the initial feature map, initial contour and ConvGRU, using feature sampling, contour updating and multi-scale feature fusion methods, and determine the final contour.

[0069] The vertex classification module guided by point attention is used to perform vertex classification and point attention calculation on the final contour in sequence to determine the building vector in the remote sensing image to be detected.

[0070] Furthermore, the ConvGRU-based contour update module specifically includes the following modules.

[0071] The feature sampling module is used to project the initial contour onto the initial feature map using bilinear interpolation and perform feature sampling. The corresponding position features are spliced ​​to generate spliced ​​features; the spliced ​​features are generated into vertex features using ring convolution.

[0072] The ConvGRU module is used to obtain the next contour state and the building contour offset according to the vertex features and the initial contour state; and to update the initial contour based on the building contour offset to obtain the next contour.

[0073] The multi-scale feature fusion module is used to extract the geometric information of the initial feature map using the edge perception head to obtain a refined contour feature map; and perform multi-scale feature fusion on the initial feature map and the refined contour feature map to obtain the next feature map.

[0074] Furthermore, the point attention-guided vertex classification module specifically includes the following modules.

[0075] The classification module is used to classify the vertices of the final contour using a three-layer fully connected network to obtain the initial classification result.

[0076] The attention calculation module is used to perform point attention weighting on the initial classification results to obtain the final classification results; based on the points classified as corner points in the final classification results, the building vectors in the remote sensing image to be detected are constructed.

[0077] In summary, the present application has the following advantages and effects.

[0078] (1) It is proposed to use ConvGRU to update the building contour as a whole, learn long-range dependency information through a gating mechanism, and then optimize the global shape of the building vector, avoiding the problem that the building vertices cannot converge near the contour.

[0079] (2) A vertex classification method guided by point attention is proposed. By guiding the output of building vectors from straight lines to corner points, more attention is paid to building corner points, thus forming more regular and accurate building vectors.

[0080] All actions to obtain signals, information or data in this application are carried out in compliance with the relevant data protection laws and policies of the country where they are located, and with the authorization given by the owner of the corresponding device.

[0081] In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the system disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part.

[0082] This article uses specific examples to illustrate the principles and implementation methods of this application. The description of the above embodiments is only used to help understand the method and core ideas of this application. At the same time, for those skilled in the art, according to the ideas of this application, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as limiting this application.

Claims

1. A method for extracting building vectors end-to-end based on state updating, characterized in that: The end-to-end building vector extraction method based on state update includes: Extract the building features in the remote sensing image to be detected and obtain an initial feature map containing multi-scale information; Based on the initial feature map, a center point prediction head is used to calculate the center point of the building, and an offset prediction head is used to calculate the relative offset between the building outline and the center point of the building to obtain an initial outline; Based on the initial feature map, the initial contour and ConvGRU, feature sampling, contour updating and multi-scale feature fusion are adopted to continuously iteratively update the building contour and contour state, and determine the final contour; The final contour is sequentially subjected to vertex classification and point attention calculation to determine the building vector in the remote sensing image to be detected.

2. The end-to-end building vector extraction method based on state update according to claim 1 is characterized in that: The center point prediction head includes a 3×3 convolution layer, a ReLU activation function, a 1×1 convolution layer and a Sigmoid activation function; the offset prediction head includes a 3×3 convolution layer, a ReLU activation function and a 1×1 convolution layer.

3. The end-to-end building vector extraction method based on state update according to claim 1 is characterized in that: The initial contour is calculated as follows: ； In the formula, is the initial contour, is the coordinate of the building center point, For the building outline i The relative offset coordinates of the points and the center point of the building, and the building outline is N It consists of points, is an empirical parameter.

4. The end-to-end building vector extraction method based on state update according to claim 1 is characterized in that: Based on the initial feature map, the initial contour and ConvGRU, feature sampling, contour updating and multi-scale feature fusion are adopted to continuously iteratively update the building contour and contour state, and determine the final contour, specifically including: The initial contour is projected onto the initial feature map by bilinear interpolation and feature sampling is performed, and features at corresponding positions are spliced ​​to generate spliced ​​features; Using ring convolution to generate vertex features from the splicing features; Input the vertex features and the initial contour state into ConvGRU to obtain the next contour state and the building contour offset; Performing contour updating on the initial contour based on the building contour offset to obtain a next contour; Extracting geometric information of the initial feature map using an edge-aware head to obtain a refined contour feature map; Performing multi-scale feature fusion on the initial feature map and the refined contour feature map to obtain a next feature map; The above steps are repeated to continuously update the initial contour, the initial feature map and the initial contour state until the maximum number of iterations is reached and the final contour is output.

5. The end-to-end building vector extraction method based on state update according to claim 4 is characterized in that: The next contour state is calculated as follows: ； ； ； ； In the formula, is the next contour state, is the previous contour state, is the candidate contour state, is the feature of the previous vertex, To update the gate, To reset the gate, , and are the corresponding weights, is the convolution operation, is the element-wise multiplication, and is the activation function.

6. The end-to-end building vector extraction method based on state update according to claim 4 is characterized in that: The edge-aware head includes a 3×3 convolutional layer, a ReLU activation function, and a 1×1 convolutional layer.

7. The end-to-end building vector extraction method based on state update according to claim 1 is characterized in that: The final contour is sequentially subjected to vertex classification and point attention calculation to determine the building vector in the remote sensing image to be detected, specifically including: Using a three-layer fully connected network to perform vertex classification on the final contour to obtain an initial classification result; Performing point attention weighting on the initial classification result to obtain a final classification result; Based on the points classified as corner points in the final classification result, a building vector in the remote sensing image to be detected is constructed.

8. The end-to-end building vector extraction method based on state update according to claim 7 is characterized in that: The final classification result is calculated as follows: ； ； In the formula, is the initial classification result, Q, K and V are the query vector, key vector and value vector respectively. for The dimension of is the final classification result, is the activation function.

9. An end-to-end building vector extraction system based on state update, characterized in that: The state-updating based building vector end-to-end extraction system comprises: The initial feature extraction module is used to extract the building features in the remote sensing image to be detected and obtain the initial feature map containing multi-scale information; A contour initialization module is used to calculate the center point of the building based on the initial feature map using a center point prediction head, and calculate the relative offset between the building contour and the building center point using an offset prediction head to obtain an initial contour; A ConvGRU-based contour updating module is used to iteratively update the building contour and contour state based on the initial feature map, the initial contour and the ConvGRU, using feature sampling, contour updating and multi-scale feature fusion methods, and determine the final contour; The point attention guided vertex classification module is used to perform vertex classification and point attention calculation on the final contour in sequence to determine the building vector in the remote sensing image to be detected.

10. The end-to-end building vector extraction system based on state update according to claim 9, characterized in that: The ConvGRU-based contour update module includes: A feature sampling module is used to project the initial contour onto the initial feature map using bilinear interpolation and perform feature sampling, and corresponding position features are spliced ​​to generate spliced ​​features; and the spliced ​​features are generated into vertex features using ring convolution; A ConvGRU module is used to obtain a next contour state and a building contour offset according to the vertex features and the initial contour state; and to update the initial contour based on the building contour offset to obtain a next contour; The multi-scale feature fusion module is used to extract the geometric information of the initial feature map by using the edge perception head to obtain a refined contour feature map; and perform multi-scale feature fusion on the initial feature map and the refined contour feature map to obtain a next feature map.

Images