An end-to-end extraction method and system for building vectors based on status update

Abstract

The present application discloses a building vector end-to-end extraction method and system based on state update, which relates to the field of data models. The method includes: extracting building features in the remote sensing image to be detected to obtain an initial feature map containing multi-scale information; based on the initial feature map, using a center point prediction head to calculate the building center point, and using an offset prediction head to calculate the relative offset between the building contour and the building center point to obtain an initial contour; based on the initial feature map, the initial contour and ConvGRU, adopting feature sampling, contour update and multi-scale feature fusion methods, continuously iteratively update the building contour and the contour state, and determine the final contour; sequentially performing vertex classification and point attention calculation on the final contour to determine the building vector in the remote sensing image to be detected. The present application accurately extracts the building vector in the remote sensing image, and the extracted building vector has a more regular shape.

Description

Technical Field

[0001] This application relates to the technical field of data models, and particularly to an end-to-end extraction method and system for building vectors based on state update. Background Art

[0002] Building extraction is one of the key tasks in the field of remote sensing and is of great significance in application fields such as urban planning, disaster management, and population estimation. Traditional methods mainly rely on image segmentation and regularization techniques to generate building vectors. Given the complexity of building structures and the limitations of imaging conditions, segmentation models often produce irregular results and it is difficult to accurately generate building vectors.

[0003] Currently, research on building extraction also includes semantic segmentation methods and instance segmentation methods. Semantic segmentation methods distinguish building and non-building areas by classifying image pixels. The results can effectively cover the building area, but it is difficult to accurately distinguish connected building instances; instance segmentation methods locate individual buildings through object detection boxes and generate corresponding building segmentation areas. The results are formally closer to building vectors. Although semantic segmentation methods and instance segmentation methods have achieved excellent results in building extraction, since these methods are all segmented at the image pixel level, it is easy to generate jagged and irregular shapes in the building boundary area, so the final obtained building vectors are not accurate. Summary of the Invention

[0004] The purpose of this application is to provide an end-to-end extraction method and system for building vectors based on state update, which accurately extracts building vectors in remote sensing images, and the extracted building vectors have more regular shapes.

[0005] To achieve the above purpose, this application provides the following solutions.

[0006] In the first aspect, this application provides an end-to-end extraction method for building vectors based on state update, and the end-to-end extraction method for building vectors based on state update is as follows.

[0007] Extract building features in the remote sensing image to be detected to obtain an initial feature map containing multi-scale information.

[0008] Based on the initial feature map, use the center point prediction head to calculate the building center point, and use the offset prediction head to calculate the relative offset between the building contour and the building center point to obtain an initial contour.

[0009] Based on the initial feature map, the initial contour and ConvGRU, adopt feature sampling, contour update and multi-scale feature fusion methods to continuously iteratively update the building contour and contour state, and determine the final contour.

[0010] Perform vertex classification and point attention calculation on the final contour in sequence to determine the building vector in the remote sensing image to be detected.

[0011] In a second aspect, the present application also provides an end-to-end building vector extraction system based on state update, and the end-to-end building vector extraction system based on state update is as follows.

[0012] An initial feature extraction module, configured to extract building features in the remote sensing image to be detected, and obtain an initial feature map containing multi-scale information.

[0013] A contour initialization module, configured to calculate the building center point by using a center point prediction head based on the initial feature map, and calculate the relative offset between the building contour and the building center point by using an offset prediction head, so as to obtain an initial contour.

[0014] A contour update module based on ConvGRU, configured to continuously iterate and update the building contour and contour state by using feature sampling, contour update and multi-scale feature fusion methods based on the initial feature map, the initial contour and ConvGRU, and determine the final contour.

[0015] A vertex classification module guided by point attention, configured to perform vertex classification and point attention calculation on the final contour in sequence to determine the building vector in the remote sensing image to be detected.

[0016] According to the specific embodiments provided by the present application, the following technical effects are disclosed in the present application.

[0017] The present application uses a Convolutional Gated Recurrent Unit (ConvGRU) to globally update the building contour, learns long-range dependence information through its gating mechanism, fully utilizes the global feature information of the entire building contour and the long-range dependence information before and after the update, establishes the contour state connection of the building before and after the update, and thus avoids the phenomenon that building vertices oscillate at the contour edge. At the same time, the present application also proposes a vertex classification method guided by point attention, which refines the model output from a straight line to a corner point through point attention, enhances the attention to building corner points, and facilitates the formation of a more regular and accurate building vector. Therefore, the present application can accurately and regularly extract the building vector in the remote sensing image. Description of the Drawings

[0018] In order to more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the following will briefly introduce the drawings required to be used in the embodiments. Obviously, the drawings in the following description are only some embodiments of the present application. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0019] Figure 1 This is a flowchart of the building vector end-to-end extraction method based on status update provided by an embodiment of the present application.

[0020] Figure 2 This is a schematic diagram of contour initialization provided by an embodiment of the present application.

[0021] Figure 3 This is a schematic diagram of contour update based on ConvGRU provided by an embodiment of the present application.

[0022] Figure 4 This is a schematic diagram of ConvGRU operation provided by an embodiment of the present application.

[0023] Figure 5 This is a schematic diagram of vertex classification guided by point attention provided by an embodiment of the present application.

[0024] Figure 6 This is a schematic diagram of the building vector end-to-end extraction system based on status update provided by an embodiment of the present application. Detailed implementation manners

[0025] Next, the technical solutions in the embodiments of the present application will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present application without creative efforts shall fall within the protection scope of the present application.

[0026] Currently, there are also methods for extracting building vectors from remote sensing images in an end-to-end manner, but they generally only use local features to generate building vectors, ignoring global information and easily resulting in shape distortion.

[0027] Specifically, according to the different ways of generating building vectors, end-to-end extraction can be divided into the corner point sorting method and the contour regression method. The corner point sorting method regards building corner points as a directed sequence, extracts features through a Convolutional Neural Network (CNN), generates the positions of corner points and judges the relationships between corner points to obtain building vectors; the contour regression method predicts building vectors as a fixed number of vertices, obtains building corner points through vertex classification, and generates building vectors. However, the corner point sorting method only considers using previously predicted corner points and directions to generate subsequent corner points, and the contour regression method generally uses circular convolution or one-dimensional convolution to extract local features for contour update, which cannot comprehensively perceive the global information of the building and leads to distortion of the final overall shape. Therefore, designing an accurate and efficient end-to-end extraction method for building vectors in remote sensing images remains a challenge.

[0028] The purpose of this application is to provide a method and system for end-to-end extraction of building vectors based on state update, which accurately extracts building vectors in remote sensing images, and the extracted building vectors have more regular shapes.

[0029] To make the above objects, features, and advantages of this application more obvious and understandable, the following further detailed description of this application will be given in conjunction with the accompanying drawings and specific embodiments.

[0030] In an exemplary embodiment, as Figure 1 shown, a method for end-to-end extraction of building vectors based on state update is provided, and the method for end-to-end extraction of building vectors based on state update is as follows.

[0031] Step S1: Extract building features in the remote sensing image to be detected to obtain an initial feature map containing multi-scale information.

[0032] In this embodiment, the extraction of building features refers to the Deepsnake framework, and the backbone network specifically used is DLA-34. DLA-34 is used to obtain building features at pyramid levels, and an initial feature map containing multi-scale information is obtained.

[0033] Step S2: Based on the initial feature map, use the center point prediction head to calculate the building center point, and use the offset prediction head to calculate the relative offset between the building contour and the building center point to obtain an initial contour (see Figure 2 ).

[0034] In this embodiment, the center point prediction head includes a 3×3 convolutional layer, a ReLU activation function, a 1×1 convolutional layer, and a Sigmoid activation function; the offset prediction head includes a 3×3 convolutional layer, a ReLU activation function, and a 1×1 convolutional layer. Among them, the calculation method of the initial contour is as follows.

[0035] .

[0036] In the formula, is the initial contour, is the building center point coordinate, is the relative offset coordinate between the i th point on the building contour and the building center point, the building contour consists of N points, is an empirical parameter.

[0037] Step S3: Based on the initial feature map, the initial contour, and ConvGRU, adopt feature sampling, contour update, and multi-scale feature fusion methods to continuously iteratively update the building contour and contour state, and determine the final contour.

[0038] AsFigure 3 As shown in the figure, step S3 is specifically as follows.

[0039] Step S31: Project the initial contour onto the initial feature map using bilinear interpolation and perform feature sampling, and the features at corresponding positions are concatenated to generate concatenated features .

[0040] Step S32: Use circular convolution to generate vertex features from the concatenated features . .

[0041] Step S33: Input the vertex features and the initial contour state into ConvGRU to obtain the next contour state and the building contour offset .

[0042] As Figure 4 shown, the calculation method of the next contour state is as follows.

[0043] .

[0044] .

[0045] .

[0046] .

[0047] In the formula, is the previous contour state, is the candidate contour state, is the update gate, is the reset gate, , and are the corresponding weights respectively, is the convolution operation, is the element-wise multiplication, and are the activation functions.

[0048] The building contour offset is calculated through the feedforward neural network (FNN) in ConvGRU. The initial contour state is obtained after being activated by the tanh function from the initial vertex features.

[0049] Step S34: Update the initial contour based on the building contour offset to obtain the next contour .

[0050] 。

[0051] Step S35: Extract the geometric information of the initial feature map using an edge perception head to obtain a refined contour feature map.

[0052] The edge perception head includes a 3×3 convolutional layer, a ReLU activation function, and a 1×1 convolutional layer. Using the edge perception head can extract richer geometric information and refine the edge contour of the building.

[0053] Step S36: Perform multi-scale feature fusion on the initial feature map and the refined contour feature map to obtain the next feature map F new 。

[0054] The multi-scale feature fusion draws on the architecture of the feature pyramid and is used to deeply fuse the initial feature map and the refined contour feature map generated by the edge perception head.

[0055] Repeat the above steps S31 - S36, continuously update the initial contour using the next contour, update the initial feature map using the next feature map, and update the initial contour state using the next contour state until the final contour is output after reaching the maximum number of iterations 。

[0056] Step S4: For the final contour Perform vertex classification and point attention calculation in sequence to determine the building vector in the remote sensing image to be detected.

[0057] As Figure 5 shown, step S4 is specifically as follows.

[0058] Step S41: Use a three-layer fully connected network to perform vertex classification on the final contour to obtain an initial classification result 。

[0059] Step S42: Perform point attention weighting on the initial classification result to obtain the final classification result 。

[0060] The calculation method of the final classification result is as follows.

[0061] 。

[0062] 。

[0063] In the formula, Q, K, and V are the query vector, key vector, and value vector respectively, is the dimension of, is the activation function. In the above process, the vertex classification process is actually a process of removing redundant points.

[0064] Step S43: Based on the points classified as corner points in the final classification result, construct the building vector in the remote sensing image to be detected.

[0065] In another exemplary embodiment, as Figure 6 shown, a state-update-based end-to-end building vector extraction system is provided. The state-update-based end-to-end building vector extraction system includes the following modules.

[0066] Initial feature extraction module, used to extract the building features in the remote sensing image to be detected, and obtain an initial feature map containing multi-scale information.

[0067] Contour initialization module, used to calculate the building center point based on the initial feature map by using the center point prediction head, and calculate the relative offset between the building contour and the building center point by using the offset prediction head, so as to obtain the initial contour.

[0068] Contour update module based on ConvGRU, used to continuously iterate and update the building contour and contour state, and determine the final contour by adopting feature sampling, contour update and multi-scale feature fusion methods based on the initial feature map, initial contour and ConvGRU.

[0069] Vertex classification module guided by point attention, used to perform vertex classification and point attention calculation on the final contour in sequence, and determine the building vector in the remote sensing image to be detected.

[0070] Furthermore, the contour update module based on ConvGRU specifically includes the following modules.

[0071] Feature sampling module, used to project the initial contour onto the initial feature map by using bilinear interpolation and perform feature sampling, and the corresponding position features are spliced to generate spliced features; use circular convolution to generate vertex features from the spliced features.

[0072] ConvGRU module, used to obtain the next contour state and the building contour offset according to the vertex features and the initial contour state; update the initial contour based on the building contour offset to obtain the next contour.

[0073] Multi-scale feature fusion module, used to extract the geometric information of the initial feature map by using the edge perception head to obtain a refined contour feature map; perform multi-scale feature fusion on the initial feature map and the refined contour feature map to obtain the next feature map.

[0074] Furthermore, the vertex classification module guided by point attention specifically includes the following modules.

[0075] A classification module for classifying vertices of the final contour using a three-layer fully connected network to obtain an initial classification result.

[0076] An attention calculation module for performing point attention weighting on the initial classification result to obtain a final classification result; based on the points classified as corner points in the final classification result, a building vector in the remote sensing image to be detected is formed.

[0077] In summary, the present application has the following advantages and effects.

[0078] (1) It is proposed to use ConvGRU to globally update the building contour, learn long-range dependence information through the gating mechanism, and then optimize the global shape of the building vector, avoiding the problem that building vertices cannot converge near the contour.

[0079] (2) A vertex classification method guided by point attention is proposed. Through point attention, the output of the building vector is refined from a straight line to corner points, paying more attention to building corner points, so that a more regular and accurate building vector can be formed.

[0080] All actions of obtaining signals, information or data in this application are carried out on the premise of complying with the corresponding data protection regulations and policies of the country where it is located and with the authorization given by the owner of the corresponding device.

[0081] In this specification, the various embodiments are described in a progressive manner. Each embodiment focuses on the differences from other embodiments. The same or similar parts among the various embodiments can be referred to each other. For the system disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the description of the method part.

[0082] Specific examples are used in this article to elaborate on the principle and implementation manner of the present application. The description of the above embodiments is only used to help understand the method and its core idea of the present application; at the same time, for those of ordinary skill in the art, based on the idea of the present application, there will be changes in the specific implementation manner and application scope. In summary, the content of this specification should not be construed as a limitation to the present application.

Claims

1. An end-to-end extraction method for building vectors based on status update, characterized in that The state-update-based end-to-end building vector extraction method includes: Extract building features from the remote sensing image to be detected to obtain an initial feature map containing multi-scale information; Based on the initial feature map, use the center point prediction head to calculate the building center point, and use the offset prediction head to calculate the relative offset between the building contour and the building center point to obtain an initial contour; Based on the initial feature map, the initial contour, and ConvGRU, adopt feature sampling, contour update, and multi-scale feature fusion methods to continuously iterate and update the building contour and contour state, and determine the final contour. Specifically, it includes: project the initial contour onto the initial feature map using bilinear interpolation and perform feature sampling, and the corresponding position features are concatenated to generate concatenated features; use circular convolution to generate vertex features from the concatenated features; input the vertex features and the initial contour state into ConvGRU to obtain the next contour state and the building contour offset; update the initial contour based on the building contour offset to obtain the next contour; use the edge perception head to extract the geometric information of the initial feature map to obtain a refined contour feature map; perform multi-scale feature fusion on the initial feature map and the refined contour feature map to obtain the next feature map; repeat the above steps to continuously update the initial contour, the initial feature map, and the initial contour state until the maximum number of iterations is reached and then output the final contour; the calculation method of the next contour state is: ； ； ； ； In the formula, is the next contour state, is the previous contour state, is the candidate contour state, is the previous vertex feature, is the update gate, is the reset gate, , and are the corresponding weights respectively, is the convolution operation, is the element-wise multiplication, and are the activation functions, and the building contour offset is calculated by the feed-forward neural network in ConvGRU, and the initial contour state is obtained by activating the initial vertex feature through the tanh function; Perform vertex classification and point attention calculation on the final contour in sequence to determine the building vector in the remote sensing image to be detected. Specifically, it includes: use a three-layer fully connected network to perform vertex classification on the final contour to obtain an initial classification result; perform point attention weighting on the initial classification result to obtain the final classification result; based on the points classified as corner points in the final classification result, form the building vector in the remote sensing image to be detected.

2. The method for end-to-end extraction of building vectors based on status update according to claim 1, wherein The center point prediction head includes a 3×3 convolutional layer, a ReLU activation function, a 1×1 convolutional layer, and a Sigmoid activation function; the offset prediction head includes a 3×3 convolutional layer, a ReLU activation function, and a 1×1 convolutional layer.

3. The method for end-to-end extraction of building vectors based on status update according to claim 1, characterized in that The calculation method of the initial contour is: ； In the formula, is the initial contour, is the coordinate of the building center point, is the relative offset coordinate of the i -th point on the building contour with respect to the building center point. The building contour consists of N points, is the empirical parameter.

4. The method for end-to-end extraction of building vectors based on status update according to claim 1, wherein, The edge perception head includes a 3×3 convolutional layer, a ReLU activation function, and a 1×1 convolutional layer.

5. The method for end-to-end extraction of building vectors based on status update according to claim 1, characterized in that The calculation method of the final classification result is: ； ； In the formula, is the initial classification result, where Q, K, and V are the query vector, key vector, and value vector respectively, is the dimension of is the final classification result, is the activation function.

6. An end-to-end extraction system for building vectors based on state update, characterized in that, The state-update-based end-to-end building vector extraction system includes: An initial feature extraction module for extracting building features from the remote sensing image to be detected to obtain an initial feature map containing multi-scale information; A contour initialization module for calculating the building center point using the center point prediction head and calculating the relative offset between the building contour and the building center point using the offset prediction head based on the initial feature map to obtain an initial contour; The contour update module based on ConvGRU is used to continuously iteratively update the building contour and contour state and determine the final contour based on the initial feature map, the initial contour, and ConvGRU by means of feature sampling, contour update, and multi-scale feature fusion. Specifically, it includes: projecting the initial contour onto the initial feature map using bilinear interpolation and performing feature sampling, and splicing the features at corresponding positions to generate spliced features; generating vertex features from the spliced features using circular convolution; inputting the vertex features and the initial contour state into ConvGRU to obtain the next contour state and the building contour offset; updating the initial contour based on the building contour offset to obtain the next contour; extracting the geometric information of the initial feature map using an edge-aware head to obtain a refined contour feature map; performing multi-scale feature fusion on the initial feature map and the refined contour feature map to obtain the next feature map; repeating the above steps to continuously update the initial contour, the initial feature map, and the initial contour state until the maximum number of iterations is reached and then outputting the final contour; the calculation method of the next contour state is: ； ； ； ； In the formula, is the next contour state, is the previous contour state, is the candidate contour state, is the previous vertex feature, is the update gate, is the reset gate, , and are the corresponding weights respectively, is the convolution operation, is the element-wise multiplication, and are the activation functions, and the building contour offset is calculated through the feedforward neural network in ConvGRU, and the initial contour state is obtained by activating the initial vertex feature through the tanh function; The vertex classification module guided by point attention is used to sequentially perform vertex classification and point attention calculation on the final contour to determine the building vector in the remote sensing image to be detected. Specifically, it includes: performing vertex classification on the final contour using a three-layer fully connected network to obtain an initial classification result; performing point attention weighting on the initial classification result to obtain a final classification result; and forming the building vector in the remote sensing image to be detected based on the points classified as corner points in the final classification result.

Images