Building vector outline extraction method, device, equipment and storage medium

CN118736240BActive Publication Date: 2024-11-26AEROSPACE INFORMATION RES INST CAS
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202411207233.9
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-08-30
Publication Date
2024-11-26
Estimated Expiration
2044-08-30

Smart Images

  • Figure CN118736240B_ABST
    Figure CN118736240B_ABST
Patent Text Reader

Abstract

The present invention provides a method, device, equipment and storage medium for extracting vector contours of buildings, and relates to the technical field of image processing. The method comprises the following steps: obtaining a first feature map of a building instance, wherein the first feature map is obtained by applying a first remote sensing image to an image encoder and a first RoIAlign model, or by applying a first bounding box to the first remote sensing image and applying the image encoder; obtaining a prompt coding vector of the building instance, wherein the prompt coding vector is obtained by applying the first remote sensing image to an image encoder, a bounding box prompt generator and a prompt encoder, or by applying the first bounding box and prompt points to the prompt encoder; inputting the first feature map and the prompt coding vector into a multi-task decoder to obtain a vertex confidence map, a vertex bias map and a mask; extracting a vertex coordinate sequence based on the vertex confidence map, the vertex bias map and the mask, thereby improving the generalization and accuracy of the extraction of vector contours of buildings.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to a method, device, equipment and storage medium for extracting vector outlines of buildings. Background Art

[0002] Building vector outline extraction refers to extracting the vector outline polygons of individual buildings from remote sensing images, which helps to achieve application scenarios such as large-scale map production and smart city management. Building vector outlines are usually represented by vertex coordinate sequences, and vertices are usually the corner points of the building outline. Therefore, the core of the problem is to find the vertices and connect them.

[0003] Deep learning-based building vector contour extraction methods can be divided into two categories according to the way of representing vertices: pixel-level methods, which represent vertices as vertex confidence maps; and contour-based methods, which directly predict or optimize the vertex coordinate sequence on the contour.

[0004] However, the generalization of current pixel-level methods and contour-based methods is not strong. When the trained model is tested on data from different regions than the training dataset, the building extraction accuracy will decrease. In addition, the semantic segmentation method used by the current pixel-level method will cause two nearby buildings to be adhered and thus mistakenly regarded as one building. The contour-based method has poor adaptability to contours with different numbers of vertices because the predicted contour is set to have a fixed number of vertices. Summary of the invention

[0005] The present invention provides a method, device, equipment and storage medium for extracting vector outlines of buildings, which are used to solve the defects of the methods for extracting vector outlines of buildings in the prior art.

[0006] The present invention provides a method for extracting a building vector outline, the method comprising:

[0007] Obtain a first feature map of a building instance, wherein, in the case of a first extraction mode, the first feature map is obtained based on a first remote sensing image of the building instance, an image encoder applied to a SAM model, and a first RoIAlign model, and in the case of a second extraction mode, the first feature map is obtained based on a first bounding box of the building instance and the first remote sensing image, and an image encoder applied to the SAM model;

[0008] Obtaining a hint coding vector for a building instance, wherein the hint coding vector is obtained based on the first remote sensing image, an image encoder, a bounding box hint generator, and a hint encoder of the SAM model applied to the SAM model, in a first extraction mode, and the hint coding vector is obtained based on a first bounding box and hint points of the building instance, and a hint encoder of the SAM model applied to the SAM model, in a second extraction mode;

[0009] Inputting the first feature map and the hint encoding vector into a multi-task decoder to obtain a vertex confidence map, a vertex bias map and a mask of the building instance output by the multi-task decoder;

[0010] Based on the vertex confidence map, the vertex bias map and the mask, a vertex coordinate sequence of the building instance is obtained.

[0011] According to a building vector outline extraction method provided by the present invention, in the case of being in a first extraction mode, the hint coding vector is obtained in the following manner:

[0012] Inputting the first remote sensing image into the image encoder of the SAM model to obtain a plurality of second feature maps output by the image encoder of the SAM model;

[0013] Inputting the plurality of second feature maps into a bounding box hint generator to obtain at least one second bounding box output by the bounding box hint generator;

[0014] The at least one second bounding box and a center point corresponding to the at least one second bounding box are input into a prompt encoder of the SAM model to obtain a prompt encoding vector.

[0015] According to a method for extracting a building vector outline provided by the present invention, the bounding box prompt generator includes a feature aggregator, a region generation network, a second RoIAlign model and a bounding box prediction head, and the inputting of the plurality of second feature maps into the bounding box prompt generator to obtain at least one second bounding box output by the bounding box prompt generator includes:

[0016] Inputting the plurality of second feature maps into the feature aggregator to obtain a plurality of third feature maps output by the feature aggregator;

[0017] Inputting the plurality of third feature maps into the region generation network to obtain a plurality of candidate bounding boxes output by the region generation network;

[0018] Input the plurality of candidate bounding boxes and the plurality of third feature maps into the second RoIAlign model to obtain a plurality of fourth feature maps output by the second RoIAlign model;

[0019] The plurality of fourth feature maps are input into the bounding box prediction head to obtain at least one second bounding box.

[0020] According to a method for extracting building vector contours provided by the present invention, the multi-task decoder includes a bidirectional transformer module, an upsampling convolution module, and a first multi-layer perceptron, and the vertex confidence map and vertex bias map of the building instance are obtained by the following method:

[0021] Inputting the first feature map, the hint encoding vector and the first token into the bidirectional transformer module, obtaining a fifth feature map and a second token output by the bidirectional transformer module, wherein the first token includes a first polygon token and the second token includes a second polygon token;

[0022] Inputting the fifth feature map into the upsampling convolution module to obtain a sixth feature map output by the upsampling convolution module;

[0023] The second polygon token is input into the first multi-layer perceptron and then subjected to a dot product operation with the fifth feature map to obtain a vertex confidence map and a vertex bias map of the building instance.

[0024] According to a method for extracting a building vector outline provided by the present invention, the multi-task decoder further includes a deconvolution module, a second multi-layer perceptron and a third multi-layer perceptron, the first token further includes a first mask token and a first cross-parameter token, the second token further includes a second mask token and a second cross-parameter token, and the mask of the building instance is obtained by:

[0025] Inputting the fifth feature map into the deconvolution module to obtain a seventh feature map;

[0026] In the case of being in the first extraction mode, inputting the second mask token into a second multi-layer perceptron and performing a dot product operation with the seventh feature map to obtain a mask of the building instance;

[0027] In the second extraction mode, the second mask token is input into a second multi-layer perceptron and then subjected to a dot product operation with the seventh feature map to obtain a candidate mask of the building instance;

[0028] Inputting the second parallel cross ratio token into the third multi-layer perceptron to obtain a prediction score;

[0029] A mask of the building instance is obtained from the candidate masks based on the prediction score.

[0030] According to a method for extracting a building vector outline provided by the present invention, the step of obtaining a vertex coordinate sequence of the building instance based on the vertex confidence map, the vertex bias map and the mask comprises:

[0031] After performing a non-maximum suppression operation on the vertex confidence map, a plurality of candidate vertices are selected from the vertex confidence map;

[0032] Based on the vertex bias map, calibrating the coordinates of each candidate vertex;

[0033] Based on the mask and the plurality of candidate vertices, a vertex coordinate sequence of the building instance is obtained.

[0034] The present invention also provides a device for extracting building vector outlines, the device comprising:

[0035] A first building vector contour extraction module is used to obtain a first feature map of a building instance, wherein, in the case of a first extraction mode, the first feature map is obtained based on a first remote sensing image of the building instance, an image encoder applied to a SAM model and a first RoIAlign model, and in the case of a second extraction mode, the first feature map is obtained based on a first bounding box of the building instance and the first remote sensing image, an image encoder applied to the SAM model;

[0036] A second building vector outline extraction module is used to obtain a hint coding vector of a building instance, wherein, in the case of being in a first extraction mode, the hint coding vector is obtained based on the first remote sensing image, the image encoder, the bounding box hint generator and the hint encoder of the SAM model applied to the SAM model, and in the case of being in a second extraction mode, the hint coding vector is obtained based on the first bounding box and hint points of the building instance, the hint encoder applied to the SAM model;

[0037] A third building vector contour extraction module, used for inputting the first feature map and the hint coding vector into a multi-task decoder to obtain a vertex confidence map, a vertex bias map and a mask of the building instance output by the multi-task decoder;

[0038] The fourth building vector outline extraction module is used to obtain a vertex coordinate sequence of the building instance based on the vertex confidence map, the vertex bias map and the mask.

[0039] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, any one of the above-mentioned building vector outline extraction methods is implemented.

[0040] The present invention also provides a non-transitory computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the building vector outline extraction method described in any one of the above methods is implemented.

[0041] The present invention also provides a computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the computer program implements any of the above-mentioned building vector outline extraction methods.

[0042] The method, device, equipment and storage medium for extracting building vector outlines provided by the present invention include: obtaining a first feature map of a building instance, wherein, in a first extraction mode, the first feature map is obtained by applying an image encoder of a SAM model and a first RoIAlign model based on a first remote sensing image of the building instance, and in a second extraction mode, the first feature map is obtained by applying an image encoder of the SAM model based on a first bounding box of the building instance and a first remote sensing image; obtaining a prompt coding vector of the building instance, wherein, in the first extraction mode, the prompt coding vector is obtained by applying an image encoder, a bounding box prompt generator and a prompt encoder of the SAM model based on a first remote sensing image, and in a second extraction mode, the prompt coding vector is obtained by applying a prompt encoder of the SAM model based on a first bounding box and prompt points of the building instance; inputting the first feature map and the prompt coding vector into a multi-task decoder to obtain a vertex confidence map, a vertex bias map and a mask of the building instance output by the multi-task decoder; extracting a vertex coordinate sequence of the building instance based on the vertex confidence map, the vertex bias map and the mask. Thus, the present invention improves the generalization performance of building vector contour extraction by introducing the SAM model and its prompt learning mechanism. On the other hand, by adopting the instance segmentation method, the vertex confidence map, vertex bias map and mask of the building instance are first located by using the first extraction mode or the second extraction mode, and then the vector contour of the building instance is extracted, thereby improving the accuracy of building vector contour extraction. In addition, the present invention also adopts the vertex representation method of the vertex confidence map, so as to realize the flexible extraction of vector polygon contours with any number of vertices. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0044] Figure 1It is a schematic diagram of the flow of the building vector outline extraction method provided by the present invention;

[0045] Figure 2 This is one of the scene schematic diagrams of the building vector outline extraction method provided by the present invention;

[0046] Figure 3 This is the second scene schematic diagram of the building vector outline extraction method provided by the present invention;

[0047] Figure 4 This is the third scene schematic diagram of the building vector outline extraction method provided by the present invention;

[0048] Figure 5 It is a structural schematic diagram of a building vector outline extraction device provided by the present invention;

[0049] Figure 6 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0050] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0051] Figure 1 FIG. 1 is a flow chart of the method for extracting building vector outlines provided by the present invention. Figure 1 As shown, the method includes:

[0052] Step 110, obtaining a first feature map of a building instance, wherein, in the case of a first extraction mode, the first feature map is obtained based on a first remote sensing image of the building instance, an image encoder applied to a SAM model and a first RoIAlign model, and in the case of a second extraction mode, the first feature map is obtained based on a first bounding box of the building instance and the first remote sensing image, an image encoder applied to the SAM model;

[0053] Here, the first extraction mode is an automatic mode, and the second extraction mode is a manual prompt mode.

[0054] Specifically, in the automatic mode, the first remote sensing image of the building instance is input into the image encoder of the SAM model (Segment Anything Model) to obtain the final stage image feature map output by the image encoder of the SAM model. Then, the final stage image feature map is input into a RoIAlign model (i.e., the first RoIAlign model), and after being processed by the RoIAlign model, the required first feature map can be obtained.

[0055] In the manual prompt mode, the user manually inputs the first bounding box of the building instance, then crops the first remote sensing image of the building instance by the first bounding box of the building instance, and finally inputs the cropped remote sensing image into the image encoder of the SAM model to obtain the final stage image feature map output by the image encoder of the SAM model as the required first feature map.

[0056] It can be understood that the image encoder of the SAM model in this embodiment can be constructed based on ViT (Vision Transformer) as a basic network unit.

[0057] Step 120, obtaining a hint coding vector of the building instance, wherein, in the case of being in a first extraction mode, the hint coding vector is obtained based on the first remote sensing image, the image encoder, the bounding box hint generator and the hint encoder of the SAM model applied to the SAM model, and in the case of being in a second extraction mode, the hint coding vector is obtained based on the first bounding box and hint points of the building instance, the hint encoder applied to the SAM model;

[0058] Specifically, in automatic mode, after the first remote sensing image of the building instance is input into the image encoder of the SAM model (Segment Anything Model), in addition to obtaining the final stage image feature map output by the image encoder of the SAM model, at least one intermediate stage image feature map output by the image encoder of the SAM model is also obtained, and then the bounding box prompt generator is combined with the prompt encoder of the SAM model to generate the final prompt encoding vector.

[0059] In the manual mode, the user will not only manually input the first bounding box of the building instance, but also input the corresponding cue points, which can be key points of the building outline, such as corner points and inflection points. Then, the first bounding box of the building instance and the cue points are input to the cue encoder of the SAM model to obtain the cue encoding vector output by the cue encoder of the SAM model.

[0060] It should be understood here that the prompt encoding vector in this embodiment includes key information extracted from the user input, such as the location, shape, outline, etc. of the building.

[0061] Step 130, inputting the first feature map and the hint coding vector into a multi-task decoder, and obtaining a vertex confidence map, a vertex bias map and a mask of the building instance output by the multi-task decoder;

[0062] Here, the vertex confidence map represents the confidence of the vertices on the building instance contour. Each pixel in the map corresponds to a confidence value, indicating the possibility of whether the pixel is a vertex on the building instance contour.

[0063] The vertex bias map provides more fine-tuned information about vertex positions, encoding the offset of each vertex to guide the model to locate vertex positions more accurately, resulting in a more precise outline of the building instance.

[0064] A mask is a binary image that indicates which areas of an image belong to building instances. In the mask, areas that are usually building instances are marked as foreground (usually white or 1), while other areas are marked as background (usually black or 0).

[0065] Step 140: Obtain a vertex coordinate sequence of the building instance based on the vertex confidence map, the vertex bias map and the mask.

[0066] Here, we can first select pixel points with confidence higher than a certain threshold as candidate vertices based on the vertex confidence map, and then use the vertex bias map to fine-tune the initially selected candidate vertices. After obtaining the precise vertex coordinates of the candidate vertices, we can use masks to connect these candidate vertices in a certain order to form an ordered vertex coordinate sequence.

[0067] The building vector contour extraction method proposed in this embodiment, on the one hand, improves the generalization performance of building vector contour extraction by introducing the SAM model and its prompt learning mechanism. On the other hand, by adopting the instance segmentation method, the vertex confidence map, vertex bias map and mask of the building instance are first located using the first extraction mode or the second extraction mode, and then the vector contour of the building instance is extracted, thereby improving the accuracy of building vector contour extraction. In addition, the present invention also adopts the vertex representation method of the vertex confidence map, so as to realize the flexible extraction of vector polygon contours with any number of vertices.

[0068] It should be noted that each implementation method of the present application can be freely combined, the order can be changed, or it can be executed separately, and does not need to rely on or depend on a fixed execution order.

[0069] In some embodiments, in the case of being in the first extraction mode, the hint encoding vector is obtained by:

[0070] Inputting the first remote sensing image into the image encoder of the SAM model to obtain a plurality of second feature maps output by the image encoder of the SAM model;

[0071] Inputting the plurality of second feature maps into a bounding box hint generator to obtain at least one second bounding box output by the bounding box hint generator;

[0072] The at least one second bounding box and a center point corresponding to the at least one second bounding box are input into a prompt encoder of the SAM model to obtain a prompt encoding vector.

[0073] Here, the multiple second feature maps refer to feature maps output after the first remote sensing image of the building instance is input into the image encoder of the SAM model and is extracted and converted at different levels or stages.

[0074] Specifically, after the multiple second feature maps are input into the bounding box hint generator, the bounding box hint generator can use the information in the feature map to predict multiple second bounding boxes of the building instance. Then, the multiple second bounding boxes and their corresponding center points are input into the hint encoder of the SAM model, and the hint encoder can encode this information into a hint encoding vector to provide useful information such as the location, size, shape, etc. of the building instance.

[0075] In some embodiments, the bounding box hint generator includes a feature aggregator, a region generation network, a second RoIAlign model, and a bounding box prediction head, and the inputting the plurality of second feature maps into the bounding box hint generator to obtain at least one second bounding box output by the bounding box hint generator includes:

[0076] Inputting the plurality of second feature maps into the feature aggregator to obtain a plurality of third feature maps output by the feature aggregator;

[0077] Inputting the plurality of third feature maps into the region generation network to obtain a plurality of candidate bounding boxes output by the region generation network;

[0078] Input the plurality of candidate bounding boxes and the plurality of third feature maps into the second RoIAlign model to obtain a plurality of fourth feature maps output by the second RoIAlign model;

[0079] The plurality of fourth feature maps are input into the bounding box prediction head to obtain at least one second bounding box.

[0080] refer to Figure 2, the feature aggregator first extracts the image from the SAM model’s image encoder (i.e. Figure 2 Multiple second feature maps are extracted from multiple intermediate ViT modules in the SAM image encoder shown in , and then these second feature maps from different layers are fused or aggregated in the feature aggregator to generate a set of third feature maps with multi-scale information.

[0081] Next, the multiple third feature maps output by the feature aggregator are input to the Region Proposal Network (RPN), which can identify candidate bounding boxes that may contain the outline of building instances on the third feature map. Figure 1 And then input it into a RoIAlign model (i.e., the second RoIAlign model), the second RoIAlign model can sample in the third feature map by bilinear interpolation combined with the candidate bounding box to generate a fixed-size feature map, i.e., the fourth feature map.

[0082] Finally, the plurality of fourth feature maps are input into a bounding box prediction head, and the bounding box prediction head is used to identify the regression bounding box of the building, that is, the second bounding box.

[0083] Here, the bounding box prediction head consists of 4 shared convolutional layers, a category prediction fully connected layer, and a bounding box prediction fully connected layer. Specifically, the 4 shared convolutional layers are mainly used to extract features from the input image, and the category prediction fully connected layer is responsible for mapping the features extracted by the 4 shared convolutional layers to the category space, that is, outputting the predicted probability of each category (building or background). The bounding box prediction fully connected layer is responsible for predicting the bounding box.

[0084] During training, bounding box prediction and single building extraction are jointly trained. For a detected bounding box, if the intersection over union (IoU) with the closest annotation box is greater than 0.5, it is considered a positive sample instance; otherwise, if the IoU is less than 0.5, the detection box area is not considered a valid training sample.

[0085] Here, the total training loss, including the loss of the region generation network, is as follows Add the average loss of the n candidate samples.

[0086] ;

[0087] in, It is used to indicate whether the jth candidate sample is a positive sample instance, if yes, it is 1, otherwise it is 0. Calculate the loss of the predicted category (building or background) and the target class (i.e. building) for the jth candidate sample. Calculate the loss of the predicted bounding box coordinates of the jth candidate sample relative to the offset of the anchor box and the label relative to the offset of the anchor box, is the total loss of the single building vector extraction of the jth candidate sample, which will be described in detail in the following embodiment.

[0088] In some embodiments, the multi-task decoder includes a bidirectional transformer module, an upsampling convolution module, and a first multi-layer perceptron, and the vertex confidence map and the vertex bias map of the building instance are obtained by:

[0089] Inputting the first feature map, the hint encoding vector and the first token into the bidirectional transformer module, obtaining a fifth feature map and a second token output by the bidirectional transformer module, wherein the first token includes a first polygon token and the second token includes a second polygon token;

[0090] Inputting the fifth feature map into the upsampling convolution module to obtain a sixth feature map output by the upsampling convolution module;

[0091] The second polygon token is input into the first multi-layer perceptron and then subjected to a dot product operation with the fifth feature map to obtain a vertex confidence map and a vertex bias map of the building instance.

[0092] It should be noted here that the polygon token can be used to extract the vertex confidence map, vertex bias map and boundary map of the building instance. In this embodiment, the vertex confidence map and vertex bias map of the building instance are obtained only through the polygon token.

[0093] refer to Figure 3 ,The bidirectional transformer module consists of two bidirectional transformer blocks connected in series.

[0094] The first feature map, the hint encoding vector and the first token (here, the first polygon token) are fed into the bidirectional transformer module. The bidirectional transformer module uses its self-attention mechanism to realize the attention interaction between the first token (here, the first polygon token), the hint encoding vector and the first feature map, thereby updating their respective feature representations and obtaining the corresponding fifth feature map and the second token (here, the second polygon token).

[0095] Afterwards, the fifth feature map is sent to the upsampling convolution module for upsampling convolution processing to obtain the sixth feature map. The second polygon token is first processed by the first multi-layer perceptron (here MLPs), and then dot-producted with the sixth feature map to generate the vertex confidence map and vertex bias map of the building instance.

[0096] In this embodiment, by utilizing a bidirectional transformer module, an upsampling convolution module, and a first multi-layer perceptron, and combining information of polygon tokens and the fifth feature map, the vertex confidence map and the vertex bias map of the building instance are accurately extracted.

[0097] In some embodiments, the multi-task decoder further includes a deconvolution module, a second multi-layer perceptron, and a third multi-layer perceptron, the first token further includes a first mask token and a first cross-parameter token, the second token further includes a second mask token and a second cross-parameter token, and the mask of the building instance is obtained by:

[0098] Inputting the fifth feature map into the deconvolution module to obtain a seventh feature map;

[0099] In the case of being in the first extraction mode, inputting the second mask token into a second multi-layer perceptron and performing a dot product operation with the seventh feature map to obtain a mask of the building instance;

[0100] In the second extraction mode, the second mask token is input into a second multi-layer perceptron and then subjected to a dot product operation with the seventh feature map to obtain a candidate mask of the building instance;

[0101] Inputting the second parallel cross ratio token into the third multi-layer perceptron to obtain a prediction score;

[0102] A mask of the building instance is obtained from the candidate masks based on the prediction score.

[0103] It should be noted here that the mask token is used to extract the mask, and the intersection-over-union token is used to predict the intersection-over-union ratio of each mask.

[0104] Continue to refer Figure 3 , the multi-task decoder also includes a deconvolution module, a second multi-layer perceptron (here MLPs) and a third multi-layer perceptron (here MLP).

[0105] It should be noted that the deconvolution module includes two layers of deconvolution, with LayerNorm and Gelu activation functions embedded in the middle. MLP is the superposition of three linear layers and Relu nonlinear activation function, and MLPs refers to the plural form of MLP, that is, multiple MLPs.

[0106] Specifically, the first feature map, the hint encoding vector and the first token (here, the first mask token and the first cross-parallel token) are sent to the bidirectional transformer module. The bidirectional transformer module realizes the attention interaction between the first token (here, the first mask token and the first cross-parallel token), the hint encoding vector and the first feature map through its self-attention mechanism, thereby updating their respective feature representations to obtain the corresponding fifth feature map and the second token (here, the second mask token and the second cross-parallel token). After that, the fifth feature map is sent to the deconvolution module for deconvolution processing to obtain the seventh feature map.

[0107] It should be noted here that when extracting the mask of the building instance, in the automatic mode, after the second mask token is input into the second multi-layer perceptron and the dot product operation is performed with the seventh feature map, only one predicted mask can be obtained, so in this embodiment, this predicted mask is used as the mask of the building instance. In the manual mode, after the second mask token is input into the second multi-layer perceptron and the dot product operation is performed with the seventh feature map, multiple predicted masks can be obtained. In this embodiment, the predicted mask is used as a candidate mask, and the second parallel cross ratio token is used to input into the third multi-layer perceptron to obtain the predicted score of each candidate mask, and finally the candidate mask with the highest predicted score is selected as the mask of the building instance.

[0108] Here, during training, the total loss of a single building vector extracted from the jth candidate sample is It can be obtained by the following formula:

[0109] ;

[0110] in, is the vertex confidence graph loss of the j-th candidate sample, is the vertex bias graph loss of the j-th candidate sample, is the boundary map loss of the j-th candidate sample, is the mask loss of the jth candidate sample, is the intersection-and-joint loss of the jth candidate sample.

[0111] In some embodiments, obtaining the vertex coordinate sequence of the building instance based on the vertex confidence map, the vertex bias map and the mask includes:

[0112] After performing a non-maximum suppression operation on the vertex confidence map, a plurality of candidate vertices are selected from the vertex confidence map;

[0113] Based on the vertex bias map, calibrating the coordinates of each candidate vertex;

[0114] Based on the mask and the plurality of candidate vertices, a vertex coordinate sequence of the building instance is obtained.

[0115] First, perform non-maximum suppression on the vertex confidence map. For example, use the maxpooling technique with a 3×3 neighborhood to remove non-local maximum points, that is, set the confidence values of these points to zero.

[0116] Next, after performing non-maximum suppression, select the local maximum points in the vertex confidence map whose confidence values are greater than the activation threshold (such as 0.1) as candidate vertices. Further, for each candidate vertex, read the corresponding offset value from the vertex offset map and add these offset values to the image coordinates of the candidate vertex to accurately determine the actual position of the candidate vertex.

[0117] Finally, use the mask to guide the connection between multiple candidate vertices to form an ordered sequence of vertex coordinates.

[0118] Specifically, for each contour point c on the mask i , first find and record the candidate vertex v closest to it i , to obtain the first vertex matching list Vmatch. Then remove the continuously repeated candidate vertices v from the first vertex matching list Vmatch i , to obtain the second vertex matching list Vmatch_u. For each candidate vertex v in the second vertex matching list Vmatch_u i , find and record the contour point c with the closest distance i , to form the contour point list Cclosest. For each pair of candidate vertices v i and contour point c i in the second vertex matching list Vmatch_u and the contour point list Cclosest i , calculate the angle between the edge formed by contour point c i+1 , contour point c i , and the edge formed by candidate vertex v i+1 , and the distance between contour point c i and candidate vertex v i . If angle / 10 + distance < Th (such as 12), then add the candidate vertex vi to the third vertex matching list Vfinal in order. Finally, if the included angle between the adjacent edges formed by the vertices in the third vertex matching list Vfinal is less than 10 degrees, then delete the vertex in the middle of the adjacent edges, that is, obtain the vertex coordinate sequence of the required building instance.

[0119] In this embodiment, the mask is used to guide the connection of candidate vertices to form an ordered sequence of vertex coordinates. This ensures that the vertices are not only close to the boundary of the mask but also consistent with the boundary direction of the mask.

[0120] Further, for the sake of understanding, refer to Figure 4In the stage of obtaining the hint coding vector and the RoI feature map, two acquisition modes are provided, namely the automatic mode and the hint mode. For example, in the automatic mode, the remote sensing image can be input into the image encoder, and the image encoder outputs the multi-stage image features and the final stage image features. Then, the final stage image features output by the image encoder are input into RoIAlign to obtain the output RoI feature map, and the multi-stage image features are input into the bounding box hint generator to obtain the RoI bounding box output by the bounding box hint generator. Then, the RoI bounding box and the corresponding RoI center point are input into the hint encoder to obtain the hint coding vector output by the hint encoder.

[0121] Afterwards, the hint coding vector and the RoI feature map obtained in the automatic mode or the hint mode are input into the multi-task decoder to obtain the vertex confidence map, vertex bias map (not shown in the figure), mask and boundary map output by the multi-task decoder. Here, in this embodiment, the vertex bias map is used to calibrate the coordinates of the vertex confidence map. The vertex confidence map after mask and coordinate calibration is combined with the mask-guided vertex connection algorithm to extract the vertex coordinate sequence of the required building instance, that is, the vector contour.

[0122] Based on any of the above embodiments, the present invention further provides a device for extracting building vector contours. Figure 5 Schematic diagram of the structure of the building vector contour extraction device provided by the present invention. Figure 5 As shown, the device includes: a first building vector outline extraction module 510 , a second building vector outline extraction module 520 , a third building vector outline extraction module 530 , and a fourth building vector outline extraction module 540 .

[0123] A first building vector contour extraction module 510 is used to obtain a first feature map of a building instance, wherein, in the case of a first extraction mode, the first feature map is obtained based on a first remote sensing image of the building instance, an image encoder applied to a SAM model and a first RoIAlign model, and in the case of a second extraction mode, the first feature map is obtained based on a first bounding box of the building instance and the first remote sensing image, an image encoder applied to the SAM model;

[0124] A second building vector outline extraction module 520 is used to obtain a hint coding vector of a building instance, wherein, in the case of being in the first extraction mode, the hint coding vector is obtained based on the first remote sensing image, the image encoder, the bounding box hint generator and the hint encoder of the SAM model applied to the SAM model, and in the case of being in the second extraction mode, the hint coding vector is obtained based on the first bounding box and hint points of the building instance, the hint encoder applied to the SAM model;

[0125] A third building vector outline extraction module 530 is used to input the first feature map and the hint coding vector into a multi-task decoder to obtain a vertex confidence map, a vertex bias map and a mask of the building instance output by the multi-task decoder;

[0126] The fourth building vector outline extraction module 540 is configured to obtain a vertex coordinate sequence of the building instance based on the vertex confidence map, the vertex bias map and the mask.

[0127] The device provided by the embodiment of the present invention, on the one hand, improves the generalization performance of building vector contour extraction by introducing the SAM model and its prompt learning mechanism. On the other hand, by adopting the instance segmentation method, the vertex confidence map, vertex bias map and mask of the building instance are first located using the first extraction mode or the second extraction mode, and then the vector contour of the building instance is extracted, thereby improving the accuracy of building vector contour extraction. In addition, the present invention also adopts the vertex representation method of the vertex confidence map, so as to realize the flexible extraction of vector polygon contours with any number of vertices.

[0128] The building vector outline extraction device described in this embodiment and the building vector outline extraction method provided by the present invention described above can correspond to each other and will not be described in detail herein.

[0129] Figure 6 An example of a physical structure diagram of an electronic device is shown in FIG. Figure 6 As shown, the electronic device may include: a processor 610, a communication interface 620, a memory 630 and a communication bus 640, wherein the processor 610, the communication interface 620 and the memory 630 communicate with each other through the communication bus 640. The processor 610 may call the logic instructions in the memory 630 to execute the building vector outline extraction method, which includes:

[0130] Obtain a first feature map of a building instance, wherein, in the case of a first extraction mode, the first feature map is obtained based on a first remote sensing image of the building instance, an image encoder applied to a SAM model, and a first RoIAlign model, and in the case of a second extraction mode, the first feature map is obtained based on a first bounding box of the building instance and the first remote sensing image, and an image encoder applied to the SAM model;

[0131] Obtaining a hint coding vector for a building instance, wherein the hint coding vector is obtained based on the first remote sensing image, an image encoder, a bounding box hint generator, and a hint encoder of the SAM model applied to the SAM model, in a first extraction mode, and the hint coding vector is obtained based on a first bounding box and hint points of the building instance, and a hint encoder of the SAM model applied to the SAM model, in a second extraction mode;

[0132] Inputting the first feature map and the hint encoding vector into a multi-task decoder to obtain a vertex confidence map, a vertex bias map and a mask of the building instance output by the multi-task decoder;

[0133] Based on the vertex confidence map, the vertex bias map and the mask, a vertex coordinate sequence of the building instance is obtained.

[0134] In addition, the logic instructions in the above-mentioned memory 630 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0135] On the other hand, the present invention further provides a computer program product, the computer program product includes a computer program, the computer program can be stored in a non-transitory computer-readable storage medium, when the computer program is executed by a processor, the computer can execute the building vector outline extraction method provided by the above methods, the method includes:

[0136] Obtain a first feature map of a building instance, wherein, in the case of a first extraction mode, the first feature map is obtained based on a first remote sensing image of the building instance, an image encoder applied to a SAM model, and a first RoIAlign model, and in the case of a second extraction mode, the first feature map is obtained based on a first bounding box of the building instance and the first remote sensing image, and an image encoder applied to the SAM model;

[0137] Obtaining a hint coding vector for a building instance, wherein the hint coding vector is obtained based on the first remote sensing image, an image encoder, a bounding box hint generator, and a hint encoder of the SAM model applied to the SAM model, in a first extraction mode, and the hint coding vector is obtained based on a first bounding box and hint points of the building instance, and a hint encoder of the SAM model applied to the SAM model, in a second extraction mode;

[0138] Inputting the first feature map and the hint encoding vector into a multi-task decoder to obtain a vertex confidence map, a vertex bias map and a mask of the building instance output by the multi-task decoder;

[0139] Based on the vertex confidence map, the vertex bias map and the mask, a vertex coordinate sequence of the building instance is obtained.

[0140] In another aspect, the present invention further provides a non-transitory computer-readable storage medium having a computer program stored thereon, which is implemented when the computer program is executed by a processor to perform the building vector outline extraction method provided by the above methods, the method comprising:

[0141] Obtain a first feature map of a building instance, wherein, in the case of a first extraction mode, the first feature map is obtained based on a first remote sensing image of the building instance, an image encoder applied to a SAM model, and a first RoIAlign model, and in the case of a second extraction mode, the first feature map is obtained based on a first bounding box of the building instance and the first remote sensing image, and an image encoder applied to the SAM model;

[0142] Obtaining a hint coding vector for a building instance, wherein the hint coding vector is obtained based on the first remote sensing image, an image encoder, a bounding box hint generator, and a hint encoder of the SAM model applied to the SAM model, in a first extraction mode, and the hint coding vector is obtained based on a first bounding box and hint points of the building instance, and a hint encoder of the SAM model applied to the SAM model, in a second extraction mode;

[0143] Inputting the first feature map and the hint encoding vector into a multi-task decoder to obtain a vertex confidence map, a vertex bias map and a mask of the building instance output by the multi-task decoder;

[0144] Based on the vertex confidence map, the vertex bias map and the mask, a vertex coordinate sequence of the building instance is obtained.

[0145] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0146] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0147] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for extracting building vector contours, characterized in that: The method comprises: Obtain a first feature map of a building instance, wherein, in the case of a first extraction mode, the first feature map is obtained based on a first remote sensing image of the building instance, an image encoder applied to a SAM model, and a first RoIAlign model, and in the case of a second extraction mode, the first feature map is obtained based on a first bounding box of the building instance and the first remote sensing image, and an image encoder applied to the SAM model; Obtaining a hint coding vector for a building instance, wherein the hint coding vector is obtained based on the first remote sensing image, an image encoder, a bounding box hint generator, and a hint encoder of the SAM model applied to the SAM model, in a first extraction mode, and the hint coding vector is obtained based on a first bounding box and hint points of the building instance, and a hint encoder of the SAM model applied to the SAM model, in a second extraction mode; Inputting the first feature map and the hint encoding vector into a multi-task decoder to obtain a vertex confidence map, a vertex bias map and a mask of the building instance output by the multi-task decoder; Based on the vertex confidence map, the vertex bias map and the mask, obtaining a vertex coordinate sequence of the building instance; The multi-task decoder includes a bidirectional transformer module, an upsampling convolution module, and a first multi-layer perceptron. The vertex confidence map and the vertex bias map of the building instance are obtained by: Inputting the first feature map, the hint encoding vector and the first token into the bidirectional transformer module, obtaining a fifth feature map and a second token output by the bidirectional transformer module, wherein the first token includes a first polygon token and the second token includes a second polygon token; Inputting the fifth feature map into the upsampling convolution module to obtain a sixth feature map output by the upsampling convolution module; Inputting the second polygon token into the first multi-layer perceptron and performing a dot product operation with the fifth feature map to obtain a vertex confidence map and a vertex bias map of the building instance; The multi-task decoder further includes a deconvolution module, a second multi-layer perceptron and a third multi-layer perceptron, the first token further includes a first mask token and a first cross-parameter token, the second token further includes a second mask token and a second cross-parameter token, and the mask of the building instance is obtained by: Inputting the fifth feature map into the deconvolution module to obtain a seventh feature map; In the case of being in the first extraction mode, inputting the second mask token into a second multi-layer perceptron and performing a dot product operation with the seventh feature map to obtain a mask of the building instance; In the second extraction mode, the second mask token is input into a second multi-layer perceptron and then subjected to a dot product operation with the seventh feature map to obtain a candidate mask of the building instance; Inputting the second parallel cross ratio token into the third multi-layer perceptron to obtain a prediction score; A mask of the building instance is obtained from the candidate masks based on the prediction score.

2. The building vector outline extraction method according to claim 1, characterized in that: In the case of the first extraction mode, the hint encoding vector is obtained by: Inputting the first remote sensing image into the image encoder of the SAM model to obtain a plurality of second feature maps output by the image encoder of the SAM model; Inputting the plurality of second feature maps into a bounding box hint generator to obtain at least one second bounding box output by the bounding box hint generator; The at least one second bounding box and a center point corresponding to the at least one second bounding box are input into a prompt encoder of the SAM model to obtain a prompt encoding vector.

3. The building vector outline extraction method according to claim 2, characterized in that: The bounding box hint generator includes a feature aggregator, a region generation network, a second RoIAlign model and a bounding box prediction head, and the inputting the plurality of second feature maps into the bounding box hint generator to obtain at least one second bounding box output by the bounding box hint generator includes: Inputting the plurality of second feature maps into the feature aggregator to obtain a plurality of third feature maps output by the feature aggregator; Inputting the plurality of third feature maps into the region generation network to obtain a plurality of candidate bounding boxes output by the region generation network; Input the plurality of candidate bounding boxes and the plurality of third feature maps into the second RoIAlign model to obtain a plurality of fourth feature maps output by the second RoIAlign model; The plurality of fourth feature maps are input into the bounding box prediction head to obtain at least one second bounding box.

4. The building vector outline extraction method according to claim 1, characterized in that: The step of obtaining a vertex coordinate sequence of the building instance based on the vertex confidence map, the vertex bias map and the mask comprises: After performing a non-maximum suppression operation on the vertex confidence graph, a plurality of candidate vertices are selected from the vertex confidence graph; Based on the vertex bias map, calibrating the coordinates of each candidate vertex; Based on the mask and the plurality of candidate vertices, a vertex coordinate sequence of the building instance is obtained.

5. A building vector contour extraction device, characterized in that: The device comprises: A first building vector contour extraction module is used to obtain a first feature map of a building instance, wherein, in the case of a first extraction mode, the first feature map is obtained based on a first remote sensing image of the building instance, an image encoder applied to a SAM model and a first RoIAlign model, and in the case of a second extraction mode, the first feature map is obtained based on a first bounding box of the building instance and the first remote sensing image, an image encoder applied to the SAM model; A second building vector outline extraction module is used to obtain a hint coding vector of a building instance, wherein, in the case of being in a first extraction mode, the hint coding vector is obtained based on the first remote sensing image, the image encoder, the bounding box hint generator and the hint encoder of the SAM model applied to the SAM model, and in the case of being in a second extraction mode, the hint coding vector is obtained based on the first bounding box and hint points of the building instance, the hint encoder applied to the SAM model; A third building vector contour extraction module, used for inputting the first feature map and the hint coding vector into a multi-task decoder to obtain a vertex confidence map, a vertex bias map and a mask of the building instance output by the multi-task decoder; a fourth building vector outline extraction module, configured to obtain a vertex coordinate sequence of the building instance based on the vertex confidence map, the vertex bias map and the mask; The multi-task decoder includes a bidirectional transformer module, an upsampling convolution module, and a first multi-layer perceptron. The vertex confidence map and the vertex bias map of the building instance are obtained by: Inputting the first feature map, the hint encoding vector and the first token into the bidirectional transformer module, obtaining a fifth feature map and a second token output by the bidirectional transformer module, wherein the first token includes a first polygon token and the second token includes a second polygon token; Inputting the fifth feature map into the upsampling convolution module to obtain a sixth feature map output by the upsampling convolution module; Inputting the second polygon token into the first multi-layer perceptron and performing a dot product operation with the fifth feature map to obtain a vertex confidence map and a vertex bias map of the building instance; The multi-task decoder further includes a deconvolution module, a second multi-layer perceptron and a third multi-layer perceptron, the first token further includes a first mask token and a first cross-parameter token, the second token further includes a second mask token and a second cross-parameter token, and the mask of the building instance is obtained by: Inputting the fifth feature map into the deconvolution module to obtain a seventh feature map; In the case of being in the first extraction mode, inputting the second mask token into a second multi-layer perceptron and performing a dot product operation with the seventh feature map to obtain a mask of the building instance; In the second extraction mode, the second mask token is input into a second multi-layer perceptron and then subjected to a dot product operation with the seventh feature map to obtain a candidate mask of the building instance; Inputting the second parallel cross ratio token into the third multi-layer perceptron to obtain a prediction score; A mask of the building instance is obtained from the candidate masks based on the prediction score.

6. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the building vector outline extraction method according to any one of claims 1 to 4 is implemented.

7. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the building vector outline extraction method according to any one of claims 1 to 4 is implemented.

8. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the building vector outline extraction method according to any one of claims 1 to 4 is implemented.