A method and apparatus for extracting building footprints

Abstract

This invention relates to the field of building outline extraction technology, providing a method and apparatus for building footprint extraction, comprising: inputting remote sensing imagery into an image encoder of an offset building model to obtain image-coded features, and inputting these features into a proposal network to generate automatic prompts. Sampling and encoding the input prompts and automatic prompts to obtain various markers. Based on the image-coded features and various markers, mask prediction and offset prediction are performed using a mask prediction stream and an offset prediction stream, respectively. Using the mask prediction results and offset prediction results, a ground footprint of the building is generated; the various markers include prompt markers, mask markers, vertex markers, and offset markers. The mask prediction stream generates a preliminary building outline, and the offset prediction stream further adjusts the boundaries, enabling the offset building model to handle buildings with complex shapes. The model exhibits higher stability and adaptability when processing remote sensing images with different resolutions, lighting conditions, and architectural styles.

Description

Technical Field

[0001] This invention relates to the field of building outline extraction technology, and in particular to a method and apparatus for extracting building footprints. Background Technology

[0002] Building Footprint Extraction (BFE) is an important research area in remote sensing and computer vision. Its core objective is to accurately extract the outline information of buildings from remote sensing images, which is of great significance for tasks such as 3D building reconstruction and building change detection.

[0003] Traditional BFE (Browsing Explanation) methods are mostly based on geometric features and conventional machine learning techniques. These methods rely on hand-designed features and often suffer from insufficient generalization ability in complex scenes. Furthermore, they are poorly adaptable to diverse building types and inefficient in production applications. With the rapid development of deep learning technology, deep learning-based BFE methods have significantly improved extraction accuracy, with offset-based techniques becoming a research hotspot. These methods achieve efficient processing of off-nadir remote sensing images by predicting the offset between the building roof and its outline. Offset-based models, for example, use U-Net and ResNet structures to predict the offset direction and length of the image, thereby generating the building outline. However, these methods suffer from insufficient generalization ability in practical applications. Summary of the Invention

[0004] This invention provides a method and apparatus for extracting building footprints, addressing the shortcomings of insufficient generalization ability in existing building outline extraction techniques. It improves the generalization ability of the model, exhibiting higher stability and adaptability when handling diverse building types and complex scenes. The technical solution proposed by this invention is as follows:

[0005] In a first aspect, the present invention provides a method for extracting building footprints, comprising:

[0006] Acquire remote sensing images and offset building models to be processed; wherein, the offset building model includes an image encoder, a proposal network, a mask prediction stream, and an offset prediction stream;

[0007] The remote sensing image is input into the image encoder to obtain image coding features;

[0008] The image encoding features are input into the proposal network to generate automatic suggestions;

[0009] Obtain input prompts, and sample and encode the input prompts and the automatic prompts to obtain multiple tags;

[0010] Based on the image coding features and the various markers, mask prediction and offset prediction are performed using the mask prediction stream and the offset prediction stream, respectively, to obtain mask prediction results and offset prediction results;

[0011] The ground footprint of the building is generated using the mask prediction results and offset prediction results;

[0012] The various markers include hint markers, mask markers, vertex markers, and offset markers.

[0013] Optionally, the image encoder includes a pre-trained Vision Transformer encoder; the step of inputting the remote sensing image into the image encoder to obtain image coding features includes:

[0014] The remote sensing image is divided into fixed-size, non-overlapping patches, and each patch is flattened into a one-dimensional vector.

[0015] The one-dimensional vector is mapped to a high-dimensional vector space through linear transformation, and the positional encoding is added to each patch before being input into a pre-trained Vision Transformer encoder to obtain the image encoding features.

[0016] Optionally, the proposal network includes a semantic segmentation head and a mask segmenter; the step of inputting the image-encoded features into the proposal network to generate automatic prompts includes:

[0017] The image encoding features are input into the semantic segmentation head to obtain preliminary segmentation results;

[0018] The automatic prompt is obtained by upsampling the preliminary segmentation result based on the mask segmenter.

[0019] Optionally, the offset building model further includes a cue sampler and a cue encoder; the input cue and the automatic cue are sampled and encoded to obtain various tags, including:

[0020] The input prompts and the automatic prompts are fed into the prompt sampler for prompt evaluation to obtain initial prompts;

[0021] The initial prompt is input into the prompt encoder for feature extraction to generate the various tags.

[0022] Optionally, the offset building model further includes a bidirectional Transformer; both the mask prediction stream and the offset prediction stream include a feedforward neural network, and the offset prediction stream further includes a self-offset attention module;

[0023] Based on the image coding features and the various markers, the mask prediction stream and the offset prediction stream are used to perform mask prediction and offset prediction respectively, to obtain mask prediction results and offset prediction results, including:

[0024] The image encoding features and the various markers are input into a bidirectional Transformer to perform preliminary estimation of the mask and offset, thereby obtaining the preliminary mask and preliminary offset.

[0025] In the mask prediction stream, the mask markers and vertex markers are merged along the channel dimension and then input together with the preliminary mask into the feedforward neural network in the mask prediction stream for processing, and the mask prediction result is output.

[0026] The initial offset is input into the offset prediction stream, processed in the feedforward neural network of the offset prediction stream to generate an encoded offset, and the encoded offset is input into the self-offset attention module for offset adjustment, and the offset prediction result is output.

[0027] Optionally, the mask prediction results include roof masks, building masks, and vertex masks; generating the building's ground footprint using the mask prediction results and offset prediction results includes:

[0028] Obtain the offset matrix, and offset the roof mask according to the offset matrix to obtain the offset roof mask;

[0029] The target offset angle is determined based on the roof mask, the building mask, and the offset roof mask;

[0030] At the target offset angle, a binary search method is used to search the offset prediction results to determine the target movement distance;

[0031] Based on the target offset angle and the target movement distance, the roof mask in the mask prediction result is offset to obtain the ground footprint of the building.

[0032] Secondly, the present invention also provides a building footprint extraction device, comprising the following modules:

[0033] An acquisition module is used to acquire remote sensing images and offset building models to be processed; wherein, the offset building model includes an image encoder, a proposal network, a mask prediction stream, and an offset prediction stream;

[0034] The encoding module is used to input the remote sensing image into the image encoder to obtain image encoding features;

[0035] The proposal module is used to input the image encoding features into the proposal network to generate automatic prompts;

[0036] A sampling module is used to acquire input prompts, and to sample and encode the input prompts and the automatic prompts to obtain multiple tags;

[0037] The prediction module is used to perform mask prediction and offset prediction based on the image coding features and the various markers, using the mask prediction stream and the offset prediction stream respectively, to obtain the mask prediction result and the offset prediction result;

[0038] The generation module is used to generate the ground footprint of the building using the mask prediction results and offset prediction results;

[0039] The various markers include hint markers, mask markers, vertex markers, and offset markers.

[0040] Thirdly, the present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and running on the processor, wherein the processor executes the computer program to implement the building footprint extraction method as described in the first aspect above.

[0041] Fourthly, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the building footprint extraction method as described in the first aspect above.

[0042] Fifthly, the present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the building footprint extraction method as described in the first aspect above.

[0043] Based on the above technical solution, the beneficial effects of the present invention compared with the prior art are as follows:

[0044] The building footprint extraction method and apparatus provided by this invention can extract multi-level features from remote sensing images through an image encoder, including local details and global contextual information. This multi-scale feature extraction enables the model to adapt to remote sensing images of different resolutions, lighting conditions, and architectural styles. The proposal network generates automatic cues based on image-encoded features, reducing reliance on manual annotation. The automatic cues can capture the initial location and shape of buildings, enabling the model to learn diverse architectural features from the data. By combining input cues and automatic cues, various markers (such as cue markers, mask markers, vertex markers, and offset markers) are generated, providing rich architectural feature representations. This multi-marker design enhances the model's adaptability to complex scenes. The mask prediction stream generates the initial building outline, while the offset prediction stream further adjusts the boundaries, enabling the model to handle buildings with complex shapes. This dual-stream design improves the model's adaptability to diverse building types. The model exhibits higher stability and adaptability when processing remote sensing images of different resolutions, lighting conditions, and architectural styles.

[0045] Other features and advantages of the invention will be set forth in the description which follows, and will be apparent in part from the description, or may be learned by practicing the invention. The objects and other advantages of the invention are realized and obtained in accordance with the structures particularly pointed out in the description, claims and drawings.

[0046] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, preferred embodiments are described below in detail with reference to the accompanying drawings. Attached Figure Description

[0047] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0048] Figure 1 This is a flowchart illustrating the building footprint extraction method provided by the present invention.

[0049] Figure 2 This is a schematic diagram of the offset building model provided by the present invention.

[0050] Figure 3 This is a schematic diagram of the self-offset attention module provided by the present invention.

[0051] Figure 4a and Figure 4b This is a schematic diagram of the visualization results of existing models and the model of this invention on the dataset.

[0052] Figure 5 This is a schematic diagram of the building footprint visualization results extracted by the model of this invention.

[0053] Figure 6 This is a schematic diagram of the structure of the building footprint extraction device provided by the present invention.

[0054] Figure 7 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0055] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0056] The following is combined Figures 1-6 The present invention describes a method and apparatus for extracting building footprints.

[0057] This invention proposes an improved offset-based model—Offset-Building Model Version 2 (OBMv2). This model innovatively introduces a Self-Offset Attention (SOFA) mechanism and a Multi-level Information System (MISS) to achieve high-precision, polygonal extraction of building outlines. Compared to traditional methods, this invention can directly generate vectorized building outline boundaries, thereby significantly improving data editability and productivity in practical applications.

[0058] Reference Figure 1 As shown, the method includes the following:

[0059] Step S110: Obtain the remote sensing image to be processed and the offset building model; wherein, the offset building model includes an image encoder, a proposal network, a mask prediction stream, and an offset prediction stream.

[0060] Remote sensing imagery is high-resolution imagery data acquired from remote sensing platforms such as satellites or drones. This data forms the basis for building footprint extraction. Specifically, remote sensing imagery can be high-resolution satellite or aerial imagery containing information on features such as buildings, roads, and vegetation.

[0061] The remote sensing image (Img) has dimensions of h × w × c, where h is the image height, w is the image width, and c is the number of image channels. The image has undergone preprocessing, including spatial registration and relative radiometric correction. Before inputting the remote sensing image into the model, it is cropped into multiple small multispectral image patches (Patms), each with dimensions of s × s × c. The cropped image patches have a size of 1024 × 1024, facilitating subsequent model processing.

[0062] The cropping process is implemented using a non-overlapping sliding window, with a window size of s×s. The sliding window moves across the image with a fixed step size s, ensuring no overlap between windows. For each sliding window position, an image block of size s×s×c is cropped. The top-left corner coordinates of the image block are (i×s, j×s), where i and j are the row and column indices of the sliding window. If the image height h or image width w is not an integer multiple of s, the image block in the last row or last column may be smaller than s×s. For these insufficient portions, zero values ​​are used for padding (i.e., filling missing pixels with 0) to ensure that all image blocks are of consistent size. The output is a set of cropped image blocks. Each image block has a size of s×s×c. The image blocks are arranged in the row and column order of the sliding window for easy location of their positions in the original image during subsequent processing.

[0063] The image cropping process divides large remote sensing images into multiple smaller image blocks using a non-overlapping sliding window, ensuring that each image block is the same size to facilitate subsequent model processing. Boundary regions are filled with zero values ​​to ensure that the image blocks in these areas are the same size and to avoid information loss. This step provides standardized input data for subsequent building footprint extraction and is crucial for the method's efficiency and accuracy.

[0064] The Misaligned Building Model is a deep learning model for the accurate extraction of building footprints. It consists of multiple modules, including a cue sampler, a cue encoder, a bidirectional Transformer, a mask prediction stream, and an offset prediction stream. These modules work together to accomplish the task of extracting building footprints from remotely sensed imagery.

[0065] Image encoders can employ convolutional neural network (CNN) architectures to extract features from remotely sensed images. A CNN consists of multiple convolutional layers, pooling layers, and fully connected layers, capable of extracting low-level and high-level features from the image layer by layer. Based on the output features of the image encoder, a proposal network generates preliminary predictions of building locations, producing initial location and shape cues. The proposal network is a lightweight neural network, such as a region proposal network (RPN) or a similar architecture.

[0066] Step S120: Input the remote sensing image into the image encoder to obtain image coding features.

[0067] In this step, the remote sensing image is input into the image encoder, where it undergoes processing through multiple convolutional and pooling layers to ultimately obtain image-coded features. These image-coded features contain semantic information about the image, such as edges and textures, which aids in subsequent building detection and segmentation. The features extracted by the image encoder provide rich contextual information for building detection and segmentation.

[0068] Step S130: Input the image encoding features into the proposal network to generate automatic prompts.

[0069] The proposal network receives image-encoded features as input and generates preliminary predictions of building locations based on these features, producing preliminary location and shape cues (i.e., auto-cues). These predictions can be represented as bounding boxes, points, rough outlines, or other forms to guide subsequent mask and offset predictions. The auto-cues generated by the proposal network reduce the need for manual annotation and increase automation.

[0070] Step S140: Obtain input prompts, and sample and encode the input prompts and the automatic prompts to obtain multiple tags.

[0071] In this step, user-provided input prompts (such as manually labeled building locations and shape information) and automatically generated prompts from the proposal network are combined to generate various markers. These markers include prompt markers, mask markers, vertex markers, and offset markers. By combining input prompts and automatically generated prompts, the generated markers provide more accurate guidance for subsequent mask prediction and offset prediction.

[0072] Cue markers are used to clearly indicate the approximate location or extent of a building, providing the model with initial positioning information. Mask markers provide a precise representation of the building's shape, typically a binary matrix the same size as the image data, where building pixels are marked as 1 and pixels in other areas are marked as 0. Vertex markers are used to locate key vertices of the building, such as corners or key points on edges, helping the model to more accurately understand the building's geometry. Offset markers are used to fine-tune the precise location of building boundaries, adjusting the boundaries of the mask markers by providing small offsets to further improve the accuracy of footprint extraction.

[0073] Step S150: Based on the image coding features and the various markers, perform mask prediction and offset prediction using the mask prediction stream and the offset prediction stream respectively to obtain the mask prediction result and the offset prediction result.

[0074] Using image-coded features and mask markers, an accurate segmentation mask for the building is generated. The building boundaries are then adjusted using image-coded features and offset markers to more closely approximate their true location. The combination of mask prediction and offset prediction enables more accurate extraction of the building's outline and position.

[0075] The Mask Prediction Stream generates accurate segmentation masks for buildings based on image-encoded features and various markers. The Mask Prediction Stream can be a decoder architecture (such as U-Net or a feedforward neural network FPN) that maps high-dimensional features back to the original image space. Resolution is progressively restored using convolutional layers and upsampling operations. The output generates a probabilistic mask through an activation function (such as Sigmoid). The Mask Prediction Stream provides accurate contour information for buildings, laying the foundation for generating ground footprints.

[0076] The Offset Prediction Stream predicts the offsets of building boundaries based on offset markers, resulting in a mask prediction. The Offset Prediction Stream is also a decoder structure that outputs an offset map with the same resolution as the input image.

[0077] Step S160: Using the mask prediction results and offset prediction results, generate the ground footprint of the building.

[0078] By combining the mask prediction results and the offset prediction results, the final ground footprint of the building is generated, outputting the building's precise outline and location information, which can be used for applications such as urban planning and disaster assessment.

[0079] The final generated building ground footprint has high precision and high reliability, which can meet the needs of practical applications.

[0080] This invention extracts multi-level features from remote sensing imagery through an image encoder, including local details and global contextual information. This multi-scale feature extraction enables the model to adapt to remote sensing imagery with varying resolutions, lighting conditions, and architectural styles. The proposal network generates automatic cues based on image-encoded features, reducing reliance on manual annotation. Automatic cues capture the initial location and shape of buildings, allowing the model to learn diverse architectural features from the data. By combining input cues and automatic cues, various labels (such as cue labels, mask labels, vertex labels, and offset labels) are generated, providing rich representations of architectural features. This multi-label design enhances the model's adaptability to complex scenes. The mask prediction stream generates initial building outlines, while the offset prediction stream further refines the boundaries, enabling the model to handle buildings with complex shapes. This dual-stream design improves the model's adaptability to diverse building types. The model exhibits higher stability and adaptability when processing remote sensing imagery with varying resolutions, lighting conditions, and architectural styles. This invention can be widely applied in remote sensing data processing, building change detection, and 3D building reconstruction.

[0081] The prompt markers, mask markers, vertex markers, and offset markers generated by this invention provide rich information support for subsequent editing. The building outline boundaries generated through mask prediction and offset prediction can be directly output as vector data, facilitating editing and adjustment in GIS software. It supports user input prompts (such as points, boxes, and doodles), allowing flexible adjustment of prediction results to meet diverse needs. The generated building outline boundaries are easy to edit and can quickly respond to user requirements.

[0082] The building footprint extraction method of this invention introduces a Multi-level Information System (MISS). This is because the invention integrates multi-level, multi-source information such as building roof segmentation information, building volume segmentation information, and offset information, thereby improving the accuracy and robustness of building contour extraction. Building roof segmentation information describes the contour and shape of the building roof, providing a precise outline of the building's top. Building volume segmentation information describes the overall three-dimensional structure of the building. Offset information describes the fine adjustments to the building boundaries, further refining boundary details and making the contour more accurate. By integrating multi-level information, the building contour can be globally optimized, avoiding local errors. Through multi-level information, this invention can effectively utilize building roof segmentation, building volume segmentation, and offset information to achieve globally optimized extraction of the building contour. This fusion of multi-source information not only improves the model's extraction accuracy but also provides the model with richer input features.

[0083] In an optional embodiment, the present invention uses a SAM (Segment Anything Model) encoder (or a similar Vision Transformer variant) as an image encoder. The SAM encoder is a pre-trained Vision Transformer (ViT) that has been trained on a large amount of image data and has learned how to effectively extract image features. This fully leverages the advantages of pre-training to improve the model's feature extraction capability and generalization performance. The SAM encoder extracts image features through patch partitioning, linear transformation, positional encoding, and the Vision Transformer encoder. The pre-trained weights of the SAM encoder enable it to quickly adapt to new remote sensing image data, reducing training time and data requirements. The global context modeling capability of the SAM encoder makes it more robust when handling complex scenes. Therefore, the present invention uses a pre-trained Vision Transformer to extract features from remote sensing images. The image is divided into fixed-size patches, and high-dimensional features are extracted using the Vision Transformer encoder. Step S120 above, which involves inputting the remote sensing image into the image encoder to obtain image encoded features, includes:

[0084] S1201. Divide the remote sensing image into fixed-size, non-overlapping patches, and flatten each patch into a one-dimensional vector.

[0085] S1202. The one-dimensional vector is mapped to a high-dimensional vector space through linear transformation, and position encoding is added to each patch before being input into a pre-trained Vision Transformer encoder to obtain the image encoding features.

[0086] First, the remote sensing image is divided into a series of fixed-size, non-overlapping patches. These patches are the basic processing units of the image, and their size is typically determined by the model's pre-training settings. Each patch is then flattened into a one-dimensional vector. This step converts the two-dimensional image patches into one-dimensional sequence data that the Transformer can process. Next, these one-dimensional vectors are mapped to a high-dimensional vector space through a linear transformation. This high-dimensional space can better capture the complex features in the image. Since the Transformer itself does not have the ability to process positional information in sequence data, position encoding is added to each patch. Position encoding can be fixed or learnable. They provide the model with information about the patch's location in the image. Finally, the high-dimensional vectors with position encoding are input into the Vision Transformer encoder. The Vision Transformer encoder iteratively processes these vectors using components such as self-attention and multilayer perceptrons (MLPs), ultimately outputting image-encoded features. These features contain key information such as the shape, texture, and contextual information of buildings in the image.

[0087] Taking a specific implementation as an example, the remote sensing image is divided into fixed-size, non-overlapping patches. Each patch has a size of p × p × c, where p is the patch size (e.g., p = 16) and c is the number of image channels. Each patch is flattened into a one-dimensional vector of length p × p × c. For example, if p = 16 and c = 3, the flattened vector length of each patch is 16 × 16 × 3 = 768. A linear transformation (fully connected layer) maps the one-dimensional vector of each patch to a high-dimensional vector space. The mapped vector has a dimension of d, where d is the embedding dimension of the Vision Transformer encoder (e.g., d = 768). Positional encoding is added to each patch to preserve its spatial location information in the original image. The positional encoding can be generated using sine and cosine functions and added to the high-dimensional vector of the patch.

[0088] The sequence of patch vectors, after linear transformation and positional encoding, serves as the input to the Vision Transformer encoder. The Vision Transformer encoder consists of multiple Transformer blocks, each including a Multi-Head Self-Attention (MHSA) mechanism and a Feed-Forward Network (FFN). The MHSA mechanism captures the global relationships between patches. The Feed-Forward Network performs a non-linear transformation on the features of each patch. Through multi-layer stacking, the Vision Transformer encoder can extract global contextual information from the image, outputting a high-dimensional feature vector sequence, i.e., image-encoded features.

[0089] This invention divides images into patches and processes the sequential data of these patches, using a SAM encoder as a feature extractor for remote sensing images to capture both global and local features. Through patch division and the multi-level structure of the SAM encoder, the model can extract multi-scale features of the image, adapting to remote sensing images of different resolutions. The introduction of positional encoding preserves the spatial location information of the patches, enabling the model to accurately understand the positional relationships of the patches within the image, which is crucial for subsequent building outline extraction. The parallel computing capabilities of the SAM encoder allow it to efficiently process large-scale remote sensing images.

[0090] In an optional embodiment, the proposal network in step S130 above comprises two main components: a semantic segmentation head and a mask decoder. The semantic segmentation head is used to generate preliminary segmentation results for the buildings. It is implemented based on the segmenter and the Segment AnythingModel-High Quality (SAM-HQ) model. The mask decoder upsamples the preliminary segmentation results to generate high-quality auto-suggestions.

[0091] Step S130 above, which involves inputting the image-encoded features into the proposal network to generate automatic prompts, includes:

[0092] S1301. Input the image encoding features into the semantic segmentation head to obtain preliminary segmentation results.

[0093] The semantic segmentation heads of Segmenter and SAM-HQ can efficiently generate preliminary segmentation masks for buildings. Image-encoded features are first input into the semantic segmentation head, which processes the input features and outputs a preliminary segmentation result. The preliminary segmentation result is a low-resolution binary mask representing the approximate location and shape of the building. This result is a feature map of the same size as the input image, where each pixel value represents the probability or category label that the location belongs to a building.

[0094] S1302. The initial segmentation result is upsampled based on the mask segmenter to obtain the automatic prompt.

[0095] The initial segmentation results are input into a mask segmenter for further processing. The mask segmenter upsamples the initial segmentation results by 4x to generate high-resolution auto-cues. The upsampling process uses interpolation or deconvolution operations to recover the detailed information of the segmentation mask. The upsampled high-quality embeddings are output as auto-cues. These high-quality embeddings not only preserve the key contour information of the building but also enhance the expression of details. These embeddings will be used later to regress the roof and outline of the building. The auto-cue is a segmentation mask with a size close to or the same as the original image, used for subsequent building outline extraction and editing.

[0096] This invention utilizes a combination of a semantic segmentation head and a mask segmenter to enable the model to more accurately capture the outlines and details of buildings. The initial segmentation results provide a general framework for the model, while upsampling further refines this framework, improving segmentation accuracy. Since both the semantic segmentation head and the mask segmenter are built using deep learning technology, they can learn general feature representations and segmentation strategies from large amounts of data. This allows the model to generate reasonable segmentation results for different scenes and building types, thereby enhancing its generalization ability. Automatic prompts facilitate subsequent building outline extraction and editing. Users can directly fine-tune or correct based on these prompts to meet the needs of practical applications. Simultaneously, high-quality segmentation masks provide strong support for integration and analysis with other Geographic Information System (GIS) data. The proposed network design fully considers efficiency and scalability. Both the semantic segmentation head and the mask segmenter can be customized and adjusted according to actual needs to adapt to different application scenarios and requirements. Furthermore, due to the rapid development of deep learning technology, the performance and effectiveness of these components can continue to improve with the emergence of new technologies.

[0097] In an optional embodiment, the aforementioned offset building model further includes a cue sampler and a cue encoder. The cue sampler receives two types of cues simultaneously: one is a cue manually entered by the user or provided based on prior knowledge (referred to as "input cue"), and the other is a cue automatically generated by the proposal network based on remote sensing imagery (referred to as "automatic cue"). The cue sampler intelligently evaluates these cues. This evaluation process is based on various factors, such as the accuracy of the cue, its matching degree with the current imagery data, and its potential positive impact on the model's subsequent predictions. Through this evaluation, the cue sampler can select the most suitable cue that is most helpful for the model to make accurate predictions. Once the most suitable cue is selected, the cue sampler outputs it to the subsequent cue encoder. The cue encoder is a deep learning module specifically designed to extract cue features and generate various labels. It receives cue information from the cue sampler and uses advanced deep learning techniques (such as convolutional neural networks or other feature extraction methods) to extract key features from these cuees. Based on these features, the cue encoder further generates four important labels.

[0098] The sampling and encoding of the input prompts and the automatic prompts described in S140 above to obtain multiple tags includes:

[0099] S1401. Input the input prompt and the automatic prompt into the prompt sampler for prompt evaluation to obtain an initial prompt.

[0100] First, user-input prompts and auto-prompts generated by the proposal network are fed into the prompt sampler. The main task of the prompt sampler is to evaluate these input prompts. The evaluation criteria are based on multiple dimensions such as the relevance, accuracy, and originality of the prompts. Through this evaluation process, the prompt sampler can select the most suitable prompts, which are most likely to guide the model to make accurate and efficient predictions in the next step. After completing the evaluation, the prompt sampler outputs the selected optimal prompts to the subsequent prompt encoder.

[0101] Specifically, the prompt sampler is responsible for sampling key points or regions from the auto-cues generated by the proposal network, using them as input for subsequent processing. The input to the prompt sampler is the auto-cues generated by the proposal network (such as a rough location or bounding box of a building). The output is a set of sampled points or regions representing the potential location and shape of the building. The sampler can employ random sampling, attention-based sampling, or rule-based sampling strategies. The sampled points or regions need to cover key parts of the building (such as corners, edges, etc.) to reduce redundant information, focus on key areas of the building, and improve the efficiency of subsequent processing.

[0102] S1402. Input the initial prompt into the prompt encoder for feature extraction to generate the various tags.

[0103] After receiving the optimal cues from the cue sampler, the cue encoder begins its work. Its main task is feature extraction from these cues, i.e., analyzing and identifying the key information contained within them. Specifically, the cue encoder encodes the cue information (such as points or regions) output by the sampler into a high-dimensional feature vector. The input to the cue encoder is the sampled points or regions generated by the cue sampler. The output is the encoded cue feature vector (i.e., the various labels mentioned above). The cue encoder is a lightweight neural network (such as a multilayer perceptron (MLP) or a small CNN). The encoding process fuses spatial information (such as coordinates) and semantic information (such as cue type) into a unified feature representation. The cue encoder converts the cue information into a format suitable for subsequent processing, facilitating its combination with image-encoded features. Based on the feature extraction results, the cue encoder further generates various labels, including cue labels, mask labels, vertex labels, and offset labels, providing information on how different parts of the model should be offset or adjusted to achieve more refined adjustments to the building model. Cue labels are used to clearly indicate the approximate location or extent of the building, providing preliminary positioning information for the model. Mask markers provide a precise representation of the building's shape, typically a binary matrix the same size as the image data, where pixels of the building are marked as 1 and pixels of other areas are marked as 0. Vertex markers are used to locate key vertices of the building, such as corners or key points on edges, helping the model to more accurately understand the building's geometry. Offset markers are used to fine-tune the precise position of the building's boundaries.

[0104] This invention significantly improves prediction accuracy by using a prompt sampler to evaluate and filter prompts, enabling the model to make predictions based on the most relevant and accurate prompts. The combination of automatically generated prompts from the proposal network and user-input prompts provides the model with diverse information sources. This not only enhances the model's flexibility but also allows it to meet more diverse user needs. The various markers generated by the prompt encoder (especially offset markers) provide detailed adjustment guidance for the model, enabling more refined and accurate adjustments to the building model. The entire process not only improves the model's predictive power and adjustment accuracy but also provides a more personalized and efficient user experience by combining user input and automatically generated prompts.

[0105] Furthermore, the cue sampler and cue encoder further optimize cue information, improving processing efficiency. The modular design of the offset building model (e.g., image encoder, proposal network, mask prediction stream, offset prediction stream) allows each module to be processed in parallel or in stages, improving computational efficiency. Generating multiple labels through sampling and encoding reduces redundant information, focusing on key areas of the building and improving processing efficiency. The model can quickly process large-scale remote sensing imagery, meeting the high-efficiency requirements of practical applications. The cue sampler and cue encoder can generate a large number of training samples using less labeled data, thus reducing dependence on large amounts of labeled data and lowering annotation costs.

[0106] This invention automatically generates preliminary location and shape cues for buildings through a proposal network, reducing the need for manual intervention. A cue sampler and cue encoder further optimize the cue information, improving processing efficiency. The modular design of the Offset Building Model 2.0 allows for parallel or phased processing of modules, enhancing computational efficiency. Directly generating vectorized building outline boundaries avoids the traditional raster-to-vector data conversion step, improving production efficiency. The model can rapidly process large-scale remote sensing imagery, meeting the high-efficiency requirements of practical applications.

[0107] In an optional embodiment, the above-described offset building model further includes a bidirectional Transformer; both the mask prediction stream and the offset prediction stream include a feedforward neural network, and the offset prediction stream further includes a self-offset attention module (SOFA).

[0108] The above-described S150, based on the image coding features and the various markers, utilizes the mask prediction stream and the offset prediction stream to perform mask prediction and offset prediction respectively, to obtain mask prediction results and offset prediction results, includes:

[0109] S1501. Input the image encoding features and the various markers into the bidirectional Transformer to perform preliminary estimation of the mask and offset, and obtain the preliminary mask and preliminary offset.

[0110] The bidirectional Transformer is used to fuse image-encoded features and cue-encoded features, capturing global contextual information. The input to the bidirectional Transformer consists of image-encoded features from the image encoder and multiple tags from the cue encoder. The output of the bidirectional Transformer is a preliminary mask and a preliminary offset, containing global information from both the image and the cue. The Transformer achieves feature fusion through self-attention and cross-attention mechanisms. The bidirectional design allows image features and cue features to complement each other, enhancing the model's expressive power. The bidirectional Transformer improves the accuracy of building detection and segmentation through global context modeling.

[0111] Specifically, the image encoding features generated in step S120 and the four types of markers (mask marker, vertex marker, offset marker, and cue marker) generated in step S140 are input into the bidirectional Transformer. The bidirectional Transformer utilizes its powerful encoding capabilities to process this input data, thereby obtaining preliminary estimates of the mask and offset. These preliminary estimates provide the foundation for subsequent mask and offset predictions. The introduction of the bidirectional Transformer enables the model to better capture global and local information in the input data, providing more accurate and reliable preliminary estimates for subsequent prediction tasks.

[0112] S1502. In the mask prediction stream, the mask marker and vertex marker are merged in the channel dimension and then input together with the preliminary mask into the feedforward neural network in the mask prediction stream for processing, and the mask prediction result is output.

[0113] In the mask prediction stream, the mask labels and vertex labels are first merged along the channel dimension. This step aims to fuse the information from both types of labels so that the subsequent feedforward neural network can utilize both types of information simultaneously. The merged labels, along with the initial mask, are input into the feedforward neural network in the mask prediction stream. The feedforward neural network processes these input data through multiple layers of nonlinear transformations to output the final mask prediction results. The mask prediction results include roof masks, building masks, and vertex masks. These masks accurately identify the positions of roofs, buildings, and vertices in the image, providing crucial information for subsequent offset prediction and building model reconstruction.

[0114] S1503. Input the preliminary offset into the offset prediction stream, process it in the feedforward neural network of the offset prediction stream to generate the encoded offset, input the encoded offset into the self-offset attention module for offset adjustment, and output the offset prediction result.

[0115] In the offset prediction stream, offset labels are first fed into a feedforward neural network. The feedforward network encodes this input data, generating encoded offsets. These encoded offsets are then fed into a Self-Offset Attention (SOFA) module. The SOFA module uses an attention mechanism to fine-tune the encoded offsets. The purpose of this step is to ensure that short-range offsets are predicted more accurately, thereby improving the overall prediction accuracy of the model. After adjustments by the SOFA module, the final offset prediction results are output. These offset prediction results accurately describe the offsets of buildings and roofs in the image, providing crucial information for the reconstruction of the building model.

[0116] The SOFA module utilizes Nadaraya-Watson kernel regression. Its formula is as follows:

[0117]

[0118] In the above formula, It is the output offset. Unit offset, Indicates the SOFA module. Indicates the length of the offset. Indicates the number of offsets. Indicates learnable parameters, This represents a long offset mask. This represents the normalization function. Indicates the first The length of each offset. The specific derivation of this formula is as follows:

[0119] Starting with the Nadaraya-Watson kernel pooling formula:

[0120] .

[0121] In the above formula, This represents the predicted value at x. Represents the kernel function. The input feature representing the prediction point is the offset. Indicates input features, Represents arbitrary input features, express The corresponding target variable, Indicates the number of offsets.

[0122] Prior knowledge of the BFE problem indicates that longer offsets tend to lead to better directions. Therefore, the above equation can be transformed into the following equation:

[0123]

[0124] In the above formula, This indicates the corrected offset angle. Offset query. Represented in polar coordinates . Indicates the number of offsets. Indicates the length of the offset. This represents the length of the i-th offset. Indicates the length of any offset. Indicates the offset angle. This represents the offset angle of the i-th offset. The kernel function is a Gaussian kernel with learnable parameters added. The final formula is as follows:

[0125]

[0126] Since longer offsets are generally better than shorter offsets in the BFE problem, a "long offset mask" is designed for the shorter offsets. .

[0127]

[0128] The final vector form of the SOFA module formula is as follows:

[0129]

[0130] The mask prediction stream generates an initial building outline, while the offset prediction stream further refines the boundaries, making the outline more accurate. This two-stream design can capture subtle boundary changes in buildings. Various generated markers (such as vertex markers and offset markers) provide rich feature information for both mask and offset predictions, optimizing the outline extraction results. Mask and offset predictions are optimized simultaneously as multiple tasks, mutually reinforcing each other and improving overall accuracy. The model generates more accurate building outline boundaries, making it particularly suitable for buildings with complex shapes.

[0131] This invention, by introducing a bidirectional Transformer and a self-offset attention module, enables the model to better capture global and local information in the input data, thereby improving the prediction accuracy of masks and offsets. The model can handle various types of labeled information and enhances its robustness and generalization ability through fusion and adjustment via a feedforward neural network and the self-offset attention module. Accurate mask and offset prediction results provide reliable information support for the reconstruction of complex architectural models, enabling the model to better adapt to architectural models of different scenarios and complexities.

[0132] The building footprint extraction method of this invention significantly improves the model's generalization ability, accuracy, and editability by introducing Offset Building Model 2.0 (OBMv2) and combining Self-Offset Attention (SOFA) and Multilayer Information System (MISS). SOFA dynamically adjusts attention weights to capture local details and global structure of building outlines, adapting to buildings of different shapes, sizes, and styles. This mechanism can handle complex scenes (such as densely built areas and areas with blurred boundaries), enhancing the model's adaptability to diverse data.

[0133] In an optional embodiment, the present invention utilizes mask prediction results (including roof masks, building masks, and vertex masks) and offset prediction results to generate the ground footprint of a building. This process incorporates the concept of polygon footprint generation, that is, using offset regression and building roof vector point prediction to connect vector points and directly calculate the polygon of the building base. Simultaneously, it also incorporates external knowledge, including different hinting methods and the building's own characteristics. The mask prediction results mentioned in S160 above include roof masks, building masks, and vertex masks; generating the ground footprint of the building using the mask prediction results and offset prediction results includes:

[0134] S1601. Obtain the offset matrix, and offset the roof mask according to the offset matrix to obtain the offset roof mask.

[0135] Let the unit length of the offset be... The roof segmentation result is Building segmentation results The unit length of the offset is the fixed step size used for offset during the search process. The roof segmentation result, also known as the roof mask, identifies the location of the roof in the image. The building segmentation result, also known as the building mask, identifies the overall location of the buildings in the image.

[0136] Traversal offset angle The offset angle is iterated from 0° to 360° to find the optimal offset direction. For each offset angle, a corresponding offset matrix is ​​calculated, which is used to offset points in the roof mask by a specified angle and unit length. Offset matrix for:

[0137]

[0138] S1602. Determine the target offset angle based on the roof mask, the building mask, and the offset roof mask.

[0139] The offset roof mask is compared with the building mask to evaluate the offset effect. This can be achieved by calculating metrics such as overlap area and IoU (Intersection over Union). Based on the evaluation results, the offset angle that maximizes the overlap area or maximizes the IoU is selected as the optimal offset angle, i.e., the target offset angle mentioned above.

[0140] Offset roof mask Then the aforementioned target offset angle for:

[0141]

[0142] in, This refers to finding a function within the range of 0 to 360 degrees that satisfies the condition. Angle that achieves the maximum value . It is about The function that calculates a specific angle. A certain quantity below ( and The intersection and (ratio). Indicated from different angles The roof mask obtained after offset.

[0143] S1603. At the target offset angle, a binary search method is used to search the offset prediction results to determine the target movement distance.

[0144] For the target offset angle, a binary search method is used to determine the optimal moving distance. Binary search is an efficient search algorithm that finds the target value by continuously narrowing the search range within the search interval. First, a reasonable search range is set based on the actual situation, which should include the possible optimal moving distance. Within the search range, the interval is continuously divided into two halves, and the half-interval containing the better result is selected for further searching based on evaluation metrics (such as overlapping area, IoU, etc.). When the search interval shrinks to a certain extent (i.e., reaching the preset accuracy requirement), it is considered to have converged to the optimal moving distance. That is, the distance the target moved.

[0145]

[0146] In the above formula, This represents a feasible solution within the feasible region.

[0147] S1604. Based on the target offset angle and the target movement distance, the roof mask in the mask prediction result is offset to obtain the ground footprint of the building.

[0148] The roof mask is offset based on the target offset angle and target movement distance to obtain the offset roof mask. Vector points (i.e., roof edge points) in the offset roof mask are then connected to form a polygon. This polygon, obtained by connecting the vector points, is the predicted ground footprint of the building. This polygon accurately reflects the shape and position of the building's base.

[0149] This invention uses an offset matrix to initially adjust the roof mask, providing a more realistic starting point for determining the target offset angle and improving the accuracy of subsequent steps. After obtaining the offset roof mask, the target offset angle is determined by combining the original roof mask, the building mask, and the offset roof mask. This target offset angle is the main offset direction of the building roof relative to the ground footprint. By comprehensively considering multiple mask information to determine the offset angle, the shape and position of the building in reality can be more accurately reflected, providing a foundation for precise offsetting in subsequent steps. After determining the target offset angle, a binary search method is used to search the offset prediction results at this angle to determine the target movement distance. The binary search method can efficiently find the optimal movement distance that meets the conditions, maximizing the matching degree between the offset roof mask and the building mask. The application of the binary search method improves search efficiency, enabling the rapid finding of the optimal movement distance, thereby ensuring the accuracy and efficiency of building ground footprint generation. Finally, based on the determined target offset angle and target movement distance, the roof mask in the mask prediction results is subjected to roof offset processing to obtain the building's ground footprint. This ground footprint is the projected shape of the building on the ground and is the basis for subsequent GIS processing or other applications. Through precise offset processing, ground footprints that closely match the actual shape of buildings were obtained. This not only improves the accuracy of building footprint extraction but also provides reliable basic data support for subsequent applications such as GIS processing, urban planning, and disaster assessment.

[0150] Reference Figure 2 As shown, the model first receives an image containing buildings as input. The encoding layer in the image encoder is responsible for image embedding, converting the image into a series of embedding vectors (such as ×NCopy). These vectors capture key information in the image, obtaining image encoded features, which provide the basis for subsequent processing steps.

[0151] The high-quality forward transform (HQ Former) in the semantic segmentation head is used to further process the image-encoded features using high-quality image embedding (HQ Image Embedding) to extract higher-quality features. This step enhances the representation of building features in the image, which helps in subsequent more accurate footprint extraction. The semantic decoder in the semantic segmentation head is then used to decode the extracted features to recover the semantic information of the buildings, resulting in the preliminary segmentation result described above. This step is crucial for footprint extraction because it accurately identifies the different parts of the buildings. The preliminary segmentation result is then input into a mask segmenter for further processing. The mask segmenter upsamples the preliminary segmentation result by a factor of 4, and the upsampled high-quality embedding (HQ Image Embedding) is used as the automatic prompt output.

[0152] By combining algorithms such as Mask R-CNN (Region Convolutional Neural Network) or HTC (Hybrid Task Concatenation) from external model libraries, input cues (including external and human cues) are added to the input image. The input cues and automatic cues are sampled by a cue sampler and then input into a cue encoder for encoding to obtain various tags. The decoder layers (Decoder Layer 1 and Decoder Layer 2) align the image encoding features with the various tags. The image encoding features and various tags are input into a bidirectional Transformer (including N sets of terms, each including cue terms, mask terms, vertex terms, and offset terms) for preliminary mask and offset estimation, resulting in preliminary masks and offsets. In the mask prediction stream (including N sets of terms, each including cue terms, mask terms, and vertex terms), the mask tags and vertex tags are merged along the channel dimension and input together with the preliminary mask into a feedforward neural network in the mask prediction stream for processing. Decoding converts the output of the feedforward neural network into mask prediction results (including roof masks, building masks, and vertex masks). The initial offset is input into the offset prediction stream (including 4N offset terms), and the feedforward neural network of the offset prediction stream is processed to generate the encoded offset. The encoded offset is input into the self-offset attention (SOFABlock) module for offset adjustment, and the offset prediction result (×N offset) is output.

[0153] The SOFA module comprises a Base Head FFN and an Adaptive Head FFN. Both the Base Head FFN and Adaptive Head FFN perform nonlinear transformations on the initial input offsets to enhance feature representation. Specifically, they use a Multilayer Perceptron (MLP) or Convolutional Neural Network (CNN) to extract features from the offset information. Nonlinearity is introduced through activation functions (such as ReLU) to further enhance feature representation. The Base Head Encoder and Adaptive Head Encoder encode and decode the offset information processed by the FFN, generating accurate offset predictions. Specifically, they encode the offset information into high-dimensional feature vectors, capturing local details and global structure of the building boundaries. The encoded feature vectors are then decoded into offset predictions, representing the precise adjustment amount of the building boundaries. The offset predictions, adjusted by the SOFA module, are used to further optimize the building outline.

[0154] Using the mask prediction results and offset prediction results, the roof mask is offset according to the offset to generate the ground footprint of the building, i.e., the footprint polygon. After extracting the building footprint, the model decodes the vertices of the footprint using a vertex decoder to determine the precise outline of the building. The decoded vertex information is converted into a vector format for subsequent Geographic Information System (GIS) processing or other applications.

[0155] Figure 2 Modules marked "Frozen" have their parameters frozen during training, while modules marked "Trainable" have their parameters updated during training.

[0156] Reference Figure 3 As shown, in the SOFA module described above, the initial offset is first input into multiple feedforward neural networks (FFNs). Figure 3The offset encoder in the model performs processing. The Initial Fine-Tuning Network (FFN) performs a non-linear transformation on the initial offsets to enhance feature representation. Each FFN extracts features from the offsets using a Multilayer Perceptron (MLP) or Convolutional Neural Network (CNN). Activation functions (such as ReLU) are used to introduce non-linearity, further enhancing feature representation. The output is the coded offset processed by the FFN, containing richer feature information. Coordinate system projection maps the coded offsets to a specific coordinate system, facilitating subsequent footprint extraction and vectorization. Specifically, the coded offsets are projected from the original coordinate system to a new coordinate system, potentially involving transformations such as rotation and scaling. This projection operation helps capture local details and global structure of building boundaries. The projected coded offsets are output, ready to be input into the self-offset attention module. The self-attention mechanism adjusts the coded offsets to generate accurate offset predictions. Specifically, the self-attention mechanism calculates attention weights between coded offsets, capturing local details and global structure. By dynamically adjusting the attention weights, the model's understanding of building boundaries is enhanced. Look-longer Masking is an attention masking mechanism used to control the attention range and avoid overfitting. The unit offset is used to normalize the adjusted offset, generating the unit offset. In the SOFA module, offset length and offset angle are two core parameters. The offset length represents the distance a data point needs to move in a specific direction, while the offset angle specifies the direction of this movement. These two parameters together determine the new position of the data point in the feature space, thereby optimizing the feature distribution and improving model performance. By calculating the offset length and offset angle, the SOFA module can guide the model to focus more precisely on features crucial to the final output. This capability enables the model to extract and utilize key information more efficiently when processing complex images or data. Finally, the SOFA module outputs the offset prediction result based on the calculated offset length and offset angle.

[0157] Throughout the model's processing stages, cue words, mask words, vertex words, and offset words are introduced to guide the learning process. These cues and masks help the model better understand building features in the image and accurately extract footprints.

[0158] This invention trains the model by minimizing the loss during the training process of the offset building model. The loss calculation is divided into segmentation loss and offset loss.

[0159] S1701, Offset Loss Calculation. Offset loss refers to the loss of the Reference OffsetAugment Module (ROAM). The ROAM module aims to generate more accurate offset predictions by further processing the initial offsets. To measure the performance of this module, this invention uses Smooth L1 Loss to calculate for each offset head.

[0160] S1702, Segmentation Loss Calculation. Segmentation loss is divided into semantic segmentation loss. and vertex segmentation loss Semantic segmentation loss Cross-entropy loss is used. For vertex segmentation loss... Since the vertex graph encompasses the entire input image, this means that most pixels will be classified as negative samples. Therefore, the Dynamic Range Binary Cross-Entropy Loss (DS-BCE Loss) was proposed. DS-BCE Loss improves upon the traditional binary cross-entropy loss by dynamically adjusting the weights of positive and negative samples to alleviate the sample imbalance problem. Specifically, it automatically adjusts the weights based on the ratio of positive to negative samples in the current batch, allowing the model to focus more on the minority class (i.e., the vertex class) during training.

[0161]

[0162] In the above formula, For vertex segmentation loss, and It is located in The pixels on the actual ground map and the predicted map at that location. This is the original prompt area. It is a small random domain in this area.

[0163] S1703, Overall Loss Calculate and combine the above offset losses. And the segmentation loss, the final loss is as follows

[0164]

[0165] Among them, parameters and It is a parameter that can be either 0 or 1, used to control whether the semantic head is trained simultaneously.

[0166] The overall loss is a weighted sum of the offset loss and the segmentation loss. When the parameters... and When the value is 1, it indicates that both the semantic head and the offset head are trained simultaneously; when the value is 0, it indicates that only the offset head is trained without considering the semantic segmentation task. By minimizing the overall loss, the model can simultaneously optimize the offset prediction and segmentation performance, thereby improving the overall performance.

[0167] The performance of existing models (PANet, MRCNN, MTBR-Net, LOFT, Cas.LOFT*, MLS-BRN, p.LOFT, p.Cas.LOFT*, OBM, OBM+) and the models Ours(mask) and Ours(poly.) of this invention on the BONAI and OmniCity-View3 datasets are shown in Tables 1 and 2. The visualization results are as follows: Figure 4a and Figure 4b As shown, Input represents the input remote sensing image, and Ground Truth represents the ground truth. F1 in Tables 1 and 2 is the harmonic mean of precision and recall. Precision is precision, Recall is recall, and EPE is endpoint error. mVL, mLL, mAL, aVL, aLL, aAL, R.IoU, and R.BIoU are the mean vector loss, mean length loss, mean angle loss, average length loss, average vector loss, average length loss, average angle loss, roof IoU, and roof boundary IoU, respectively.

[0168] Table 1 - Model performance on the BONAI dataset

[0169]

[0170] Table 2 - Model performance on the OmniCity-view3 dataset

[0171]

[0172] In terms of automated extraction, the model of this invention can also provide reliable results, and its visualization is as follows: Figure 5 As shown, the footprint extraction results (Ours) obtained by using the model of this invention to extract footprints from the original remote sensing image are highly consistent with the ground truth.

[0173] As shown in Table 1 (BONAI dataset), the proposed method (Ours) outperforms existing methods in several key metrics (such as F1, Precision, Recall, mAP, etc.), demonstrating higher detection accuracy and recall. Especially in evaluation metrics with higher accuracy requirements, such as EPE (Endpoint Error), angle, and orientation, the proposed method also achieves significantly superior performance, indicating better robustness and accuracy in target localization and pose estimation. These results demonstrate that the detection performance and localization accuracy of this invention on the BONAI dataset are at an advanced level.

[0174] In Table 2 (OmniCity-view3 dataset), the method of this invention consistently maintains the highest or near-highest values ​​in mainstream metrics such as F1, Precision, Recall, and mAP, and continues to lead other comparative methods in positioning accuracy evaluation, such as endpoint error. These results further demonstrate the cross-dataset generalization ability and stability of this invention, proving its outstanding performance not only in a single scenario but also adaptability to data requirements under different perspectives or resolutions, thus providing more reliable technical support for practical applications.

[0175] The building footprint extraction device provided by the present invention is described below. The building footprint extraction device described below and the building footprint extraction method described above can be referred to in correspondence.

[0176] The building footprint extraction device provided by this invention refers to... Figure 6 As shown, it includes:

[0177] The acquisition module 210 is used to acquire the remote sensing image to be processed and the offset building model; wherein, the offset building model includes an image encoder, a proposal network, a mask prediction stream and an offset prediction stream;

[0178] Encoding module 220 is used to input the remote sensing image into the image encoder to obtain image encoding features;

[0179] Proposal module 230 is used to input the image encoding features into the proposal network to generate automatic prompts;

[0180] Sampling module 240 is used to acquire input prompts, sample and encode the input prompts and the automatic prompts to obtain multiple tags;

[0181] Prediction module 250 is used to perform mask prediction and offset prediction based on the image coding features and the multiple markers, using the mask prediction stream and the offset prediction stream respectively, to obtain mask prediction results and offset prediction results;

[0182] The generation module 260 is used to generate the ground footprint of the building using the mask prediction results and the offset prediction results;

[0183] The various markers include hint markers, mask markers, vertex markers, and offset markers.

[0184] Figure 7 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 7 As shown, the electronic device may include a processor 310, a communications interface 320, a memory 330, and a communication bus 340. The processor 310, communications interface 320, and memory 330 communicate with each other via the communication bus 340. The processor 310 can call logical instructions in the memory 330 to execute the building footprint extraction method.

[0185] Furthermore, the logical instructions in the aforementioned memory 330 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, essentially, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0186] On the other hand, the present invention also provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer is able to execute the building footprint extraction method provided by the above methods.

[0187] In another aspect, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to perform the building footprint extraction methods provided by the above methods.

[0188] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0189] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0190] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for extracting building footprints, characterized in that, include: Acquire remote sensing images and offset building models to be processed; wherein, the offset building model includes an image encoder, a proposal network, a mask prediction stream, and an offset prediction stream; The remote sensing image is input into the image encoder to obtain image coding features; The image encoding features are input into the proposal network to generate automatic suggestions; Obtain input prompts, and sample and encode the input prompts and the automatic prompts to obtain multiple tags; Based on the image coding features and the various markers, mask prediction and offset prediction are performed using the mask prediction stream and the offset prediction stream, respectively, to obtain mask prediction results and offset prediction results; The ground footprint of the building is generated using the mask prediction results and offset prediction results; The various markers include prompt markers, mask markers, vertex markers, and offset markers; The offset building model also includes a bidirectional Transformer; both the mask prediction stream and the offset prediction stream include a feedforward neural network, and the offset prediction stream also includes a self-offset attention module; Based on the image coding features and the various markers, the mask prediction stream and the offset prediction stream are used to perform mask prediction and offset prediction respectively, to obtain mask prediction results and offset prediction results, including: The image encoding features and the various markers are input into a bidirectional Transformer to perform preliminary estimation of the mask and offset, thereby obtaining the preliminary mask and preliminary offset. In the mask prediction stream, the mask markers and vertex markers are merged along the channel dimension and then input together with the preliminary mask into the feedforward neural network in the mask prediction stream for processing, and the mask prediction result is output. The initial offset is input into the offset prediction stream, processed in the feedforward neural network of the offset prediction stream to generate an encoded offset, and the encoded offset is input into the self-offset attention module for offset adjustment, and the offset prediction result is output.

2. The method for extracting building footprints according to claim 1, characterized in that, The image encoder includes a pre-trained Vision Transformer encoder; the step of inputting the remote sensing image into the image encoder to obtain image coding features includes: The remote sensing image is divided into fixed-size, non-overlapping patches, and each patch is flattened into a one-dimensional vector. The one-dimensional vector is mapped to a high-dimensional vector space through linear transformation, and the positional encoding is added to each patch before being input into a pre-trained Vision Transformer encoder to obtain the image encoding features.

3. The method for extracting building footprints according to claim 1, characterized in that, The proposal network includes a semantic segmentation head and a mask segmenter; the step of inputting the image encoded features into the proposal network to generate automatic prompts includes: The image encoding features are input into the semantic segmentation head to obtain preliminary segmentation results; The automatic prompt is obtained by upsampling the preliminary segmentation result based on the mask segmenter.

4. The method for extracting building footprints according to claim 1, characterized in that, The offset building model also includes a cue sampler and a cue encoder; The input prompts and the auto-prompts are sampled and encoded to obtain various tags, including: The input prompts and the automatic prompts are fed into the prompt sampler for prompt evaluation to obtain initial prompts; The initial prompt is input into the prompt encoder for feature extraction to generate the various tags.

5. The method for extracting building footprints according to claim 1, characterized in that, The mask prediction results include roof masks, building masks, and vertex masks; the process of generating the building's ground footprint using the mask prediction results and offset prediction results includes: Obtain the offset matrix, and offset the roof mask according to the offset matrix to obtain the offset roof mask; The target offset angle is determined based on the roof mask, the building mask, and the offset roof mask; At the target offset angle, a binary search method is used to search the offset prediction results to determine the target movement distance; Based on the target offset angle and the target movement distance, the roof mask in the mask prediction result is offset to obtain the ground footprint of the building.

6. A device for extracting building footprints, characterized in that, include: An acquisition module is used to acquire remote sensing images and offset building models to be processed; wherein, the offset building model includes an image encoder, a proposal network, a mask prediction stream, and an offset prediction stream; The encoding module is used to input the remote sensing image into the image encoder to obtain image encoding features; The proposal module is used to input the image encoding features into the proposal network to generate automatic prompts; A sampling module is used to acquire input prompts, and to sample and encode the input prompts and the automatic prompts to obtain multiple tags; The prediction module is used to perform mask prediction and offset prediction based on the image coding features and the various markers, using the mask prediction stream and the offset prediction stream respectively, to obtain the mask prediction result and the offset prediction result; The generation module is used to generate the ground footprint of the building using the mask prediction results and offset prediction results; The various markers include prompt markers, mask markers, vertex markers, and offset markers; The offset building model also includes a bidirectional Transformer; both the mask prediction stream and the offset prediction stream include a feedforward neural network, and the offset prediction stream also includes a self-offset attention module; Based on the image coding features and the various markers, the mask prediction stream and the offset prediction stream are used to perform mask prediction and offset prediction respectively, to obtain mask prediction results and offset prediction results, including: The image encoding features and the various markers are input into a bidirectional Transformer to perform preliminary estimation of the mask and offset, thereby obtaining the preliminary mask and preliminary offset. In the mask prediction stream, the mask markers and vertex markers are merged along the channel dimension and then input together with the preliminary mask into the feedforward neural network in the mask prediction stream for processing, and the mask prediction result is output. The initial offset is input into the offset prediction stream, processed in the feedforward neural network of the offset prediction stream to generate an encoded offset, and the encoded offset is input into the self-offset attention module for offset adjustment, and the offset prediction result is output.

7. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that, When the processor executes the computer program, it implements the building footprint extraction method as described in any one of claims 1 to 5.

8. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the building footprint extraction method as described in any one of claims 1 to 5.

9. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the building footprint extraction method as described in any one of claims 1 to 5.

Images