Building information extraction method, device and equipment based on monocular side-looking satellite image and storage medium

Abstract

The invention discloses a building information extraction method, device and equipment based on a monocular side-looking satellite image. The method comprises the following steps: performing multi-scale feature extraction and enhancement on an inclined satellite image of a building by using an image encoding-decoding device to obtain multi-scale pixel-level features; gradually fusing the features by utilizing a query Transform module so as to extract attribute information of the building instance; segmenting the building instance by using a multi-layer sensing mechanism, generating key attributes of the building instance, and generating a contour boundary and a category label of the building according to the key attributes of the building instance; height features are extracted from the key attributes of the building instances through the attention mechanism in the instances and the attention mechanism between the instances, and then the height features are utilized to predict the height of the building through an adaptive binning regression strategy. According to the method, high-precision extraction of the contour and the height of the building is realized through an instance-level multi-task learning framework, and an economical and efficient solution is provided for urban three-dimensional modeling.

Description

Technical Field

[0001] The present invention relates to the technical field of remote sensing image processing, and in particular to a method, device, equipment and storage medium for extracting building information based on a monocular side-view satellite image. Background Art

[0002] Three-dimensional building information extraction technology is the core link of urban digital modeling and has important application value in smart city management, real estate appraisal, environmental monitoring and other fields.

[0003] The technology for extracting three-dimensional architectural information has undergone a significant evolution from traditional measurement to deep learning. From the perspective of data sources, five main technical routes have been formed: (1) LiDAR technology provides high-precision three-dimensional spatial data, but is limited by high costs; (2) Multi-view optical image analysis can reconstruct detailed urban models, but faces occlusion and matching problems in densely built areas; (3) Synthetic aperture radar technology has all-weather imaging capabilities, but is affected by complex signal phenomena and changes in the reflective properties of building materials in urban environments; (4) Monocular imaging methods can be divided into two categories based on the acquisition angle: orthographic and side-view. The former lacks vertical dimension information, while the latter reveals building facades but is accompanied by perspective distortion; (5) Multi-source data fusion methods can theoretically provide the most comprehensive information, but in practical applications they face challenges such as data heterogeneity and complex alignment processing. Among them, methods based on monocular images have attracted much attention due to their cost-effectiveness and scalability.

[0004] With the development of remote sensing technology, building height estimation based on monocular satellite imagery has become a key breakthrough in reducing the cost of three-dimensional modeling, but it still faces severe challenges in feature representation and spatial relationship modeling. Summary of the Invention

[0005] In order to solve the above technical problems, the present invention provides a method, device and equipment for extracting building information based on monocular side-view satellite images.

[0006] First, a method for extracting building information based on monocular side-view satellite images is provided, the method comprising: performing multi-scale feature extraction and enhancement on inclined satellite images of buildings using an image encoding-decoding module to obtain multi-scale pixel-level features; gradually fusing the multi-scale pixel-level features by querying a Transformer module to extract attribute information of building instances; segmenting the building instances based on the attribute information of the building instances using a multi-layer perception mechanism to generate key attributes of the building instances, and generating outline boundaries and category labels of the buildings based on the key attributes of the building instances; extracting height features from the key attributes of the building instances through an intra-instance attention mechanism and an inter-instance attention mechanism, and then predicting the building height using the height features through an adaptive binning regression strategy; outputting the building outline boundaries, category labels, and building height prediction values to complete the extraction of three-dimensional information of the buildings.

[0007] In some embodiments, the process of extracting attribute information of a building instance by querying a Transformer module specifically includes: initializing the query Transformer module to generate a learnable query and query position embedding; calculating masked cross-attention between the query and the pixel-level features output by the pixel decoder, while masking out areas with a low probability of containing the target, so that each query focuses on relevant local features; calculating self-attention between queries to capture global features with semantic meaning; and generating attribute information of the building instance through a feedforward network.

[0008] In some embodiments, the key attributes of the building instance include instance-level mask embedding, classification prediction logits, offset vector embedding, height embedding, and bin embedding, etc.

[0009] In some embodiments, the process of generating the outline boundary of a building includes: converting pixel-level features into pixel embeddings through a multi-layer perceptron; fusing instance-level mask embeddings with pixel embeddings to generate a spatially consistent representation, capturing the building instances segmented in the image to generate the outline boundary; wherein the mask embeddings are obtained after segmenting the building instances through a multi-layer perceptron.

[0010] In some embodiments, before extracting height features from key attributes of building instances through intra-instance attention and inter-instance attention mechanisms, the location attribute of the building instance and the scale attribute of the image are introduced; wherein the location attribute is represented by the geometric center coordinates of the predicted building instance outline boundary; the scale attribute information of the image is quantized by a scale factor to provide accurate location information of the building instance in the image, and the scale factor is calculated by the following formula:

[0011]

[0012] in, Refers to the benchmark scale, Refers to standard resolution images, r spatial Indicates the spatial resolution of the current image, The width ratio is W / W0, where W represents the actual width of the image and W0 is the standard width.

[0013] In some embodiments, a method is provided for extracting height features from key attributes of building instances through intra-instance attention mechanism and inter-instance attention mechanism, and then using height features to predict building height through an adaptive binning regression strategy. Specifically, the method includes: recalibrating key attribute features of building instances using intra-instance attention mechanism; modeling spatial relationships between buildings through inter-instance attention map, and recalibrating key attribute features between building instances to obtain height features; and predicting building height based on height features using an adaptive binning height regressor.

[0014] In some embodiments, a method for modeling spatial relationships between buildings through an inter-instance attention map specifically includes: constructing an adjacency matrix based on the semantic similarity of key attribute features of building instances; calculating the relationship strength between the attribute feature vectors of any two building instances; and gradually transmitting and fusing information between nodes through a graph neural network using the constructed adjacency matrix, thereby integrating the spatial and attribute relationships between nodes.

[0015] In some embodiments, a method for predicting building height based on height features using an adaptive binned height regressor specifically includes: calculating the center width of each bin based on an adaptive binned embedding, and the calculation formula is as follows:

[0016]

[0017] in, Refers to the normalized center width of each bin, ∈ is a small constant to avoid division by zero, Refers to adaptive binning embedding;

[0018] Then calculate the height of the center of each bin using the following formula:

[0019]

[0020] in, Refers to the height of the center of each bin, h min and h max Refers to the preset minimum and maximum heights;

[0021] The predicted height of the building is calculated by the height feature of the building instance and the corresponding bin center height. The calculation formula is as follows:

[0022]

[0023] in, refers to the predicted height of the building, Refers to the height of the center of each bin, Refers to the height characteristics of an architectural instance.

[0024] In the second aspect, a building information extraction device based on monocular side-view satellite images is provided, which includes: an image encoding-decoding module, which is used to extract multi-scale features from inclined satellite images through an image encoder, and enhance the multi-scale features through a pixel decoder to obtain multi-scale pixel-level features; a query Transformer module, which is used to gradually fuse multi-scale pixel-level features to extract attribute information of building instances; a multi-task decoder, which is used to segment building instances based on the attribute information of building instances using a multi-layer perception mechanism to generate key attributes of building instances, and generate the outline boundary of the building based on the key attributes of the building instances; wherein the key attributes of the building instances include category labels; a height decoder, which is used to extract height features from the key attributes of building instances through an intra-instance attention mechanism and an inter-instance attention mechanism, and then use the height features to predict the building height through an adaptive binning regression strategy; a building information output module, which is used to output the outline boundary, category label and building height of the building to complete the extraction of three-dimensional information of the building.

[0025] In a third aspect, an electronic device is provided, which includes a memory and a processor, wherein the memory stores a computer program that can be run on the processor, and when the processor executes the computer program, the method described in the first aspect is implemented.

[0026] The solution provided in the embodiments of the present application constructs an instance-level multi-task learning framework based on side-view satellite imagery, overcoming the fundamental limitation of traditional orthophotos that lack direct height cues; adopts instance segmentation methods to replace traditional pixel-level semantic segmentation to more accurately characterize the overall characteristics of buildings as independent entities; the dual attention mechanism inside and outside the instance fully utilizes the contextual information of building groups through feature recalibration within the instance and modeling of spatial relationships between instances to address the challenge of insufficient feature interaction; by quantifying the displacement relationship between the roof of the building and the corresponding bottom surface in the side-view image, physically interpretable height cues are provided for monocular height estimation. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0028] Figure 1 A schematic diagram of a building information extraction model architecture based on a monocular side-view satellite image provided by an embodiment of this specification is shown;

[0029] Figure 2 A schematic diagram of the CSA unit architecture in a query Transformer module provided in an embodiment of this specification is shown;

[0030] Figure 3 A schematic diagram of an inter-instance attention graph model architecture provided by an embodiment of this specification is shown;

[0031] Figure 4 A schematic flow chart of a method for extracting building information based on a monocular side-view satellite image provided in an embodiment of this specification is shown;

[0032] Figure 5 A schematic diagram showing a flow chart of a building height prediction method provided in an embodiment of this specification;

[0033] Figure 6 A comparison diagram showing a method for extracting building information from an orthophoto image and a side view image provided by an embodiment of this specification is shown;

[0034] Figure 7 A schematic structural diagram of a building information extraction device based on a monocular side-view satellite image provided in an embodiment of this specification is shown. DETAILED DESCRIPTION

[0035] The solution provided in this specification is described below in conjunction with the accompanying drawings.

[0036] In order to make the purpose, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be described below with reference to the accompanying drawings.

[0037] In the description of the embodiments of the present application, words such as "exemplary," "for example," or "for example" are used to indicate examples, illustrations, or descriptions. Any embodiment or design described as "exemplary," "for example," or "for example" in the embodiments of the present application should not be construed as being preferred or advantageous over other embodiments or designs. Rather, the use of words such as "exemplary," "for example," or "for example" is intended to present the relevant concepts in a concrete manner.

[0038] In the description of the embodiments of this application, the term "and / or" is merely a description of an association relationship between associated objects, indicating that three relationships may exist. For example, A and / or B can represent the following three situations: A exists alone, B exists alone, and A and B exist at the same time. In addition, unless otherwise specified, the term "plurality" means two or more.

[0039] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly identifying the technical features being referred to. Thus, features specified as "first" or "second" may explicitly or implicitly include one or more of such features. The terms "include," "comprising," "having," and their variations all mean "including but not limited to," unless otherwise specifically emphasized.

[0040] Before describing the technical solution of the present application, it is necessary to explain the existing technology that is different from this solution.

[0041] Traditional monocular height estimation methods are primarily based on orthophotos, which inherently lack direct three-dimensional height cues and must rely on indirect proxy information such as shadow length and texture features, resulting in limited estimation accuracy. While side-view images contain a wealth of critical height proxy information, namely, roof-to-base offsets, accurately extracting this proxy information presents significant challenges. Traditional pixel-level methods struggle to accurately segment and align building roofs with their corresponding bases. Establishing roof-to-base correspondences and accurately quantifying offset vectors remain key challenges, particularly in densely built areas and complex geometric structures. Furthermore, existing methods often employ pixel-level single-task or multi-task learning paradigms. While these methods have achieved some success in understanding the semantics of individual pixels, they inherently lack the ability to structure and understand buildings as holistic instances. This pixel-level feature extraction strategy struggles to effectively distinguish the boundaries of adjacent building instances in densely built areas and fails to capture the overall correlation of a building's internal structure, resulting in incomplete information in the extracted features used to represent the building's three-dimensional geometric characteristics.

[0042] Existing methods often treat buildings as independent individuals, ignoring the spatial organization patterns and morphological correlations of urban building clusters. This isolated treatment approach cannot effectively utilize the contextual information between buildings, resulting in poor performance in complex urban environments. Especially in densely populated areas, due to partial occlusion caused by viewing angle limitations, shadow interference, and complex building stacking structures, it is difficult to achieve accurate height estimation relying solely on individual building features. The lack of an effective inter-instance feature interaction mechanism prevents the model from obtaining supplementary information from surrounding buildings, which not only reduces the estimation accuracy of individual buildings, but also affects the consistency and rationality of the overall urban morphological reconstruction.

[0043] To address these challenges, this chapter proposes a building information extraction method based on monocular side-view satellite imagery. This method leverages the height information contained in the offsets between building roofs and floors in side-view imagery. Through innovative network design and optimization strategies, it achieves low-cost, high-precision building height estimation.

[0044] Figure 1The following is a diagram showing the architecture of a building information extraction model based on monocular side-view satellite images provided in an embodiment of this specification. The model includes at least an image encoding-decoding module, a query transformer module, a multi-task decoder, and a height decoder.

[0045] like Figure 1 As shown, the image encoder-decoder module may include an image encoder and a pixel decoder. The image encoder is used to capture features of multiple resolution scales of the building-slanted satellite image, and the pixel decoder is used to decode the features of multiple resolution scales of the building-slanted satellite image to generate a multi-scale pixel-level feature map with the same size as the original building-slanted satellite image.

[0046] In one embodiment, the backbone network of the image encoder can be a SwinTransformer pre-trained on ImageNet. The backbone network can be pre-trained on the ImageNet model using a large amount of oblique satellite imagery data and used to extract features at multiple resolutions of oblique satellite imagery of buildings.

[0047] The pixel image decoder can be combined with a feature pyramid network and a deformable detection transformer. Specifically, the feature pyramid network can be used to perform top-down and lateral connections on the features at multiple resolution scales output by the encoder to generate fused feature maps. These feature maps are further processed by a deformable detection transformer to capture more complex spatial relationships and contextual information. To be compatible with the original image size, the pixel decoder upsamples these feature maps and fuses information from different levels as needed to generate multi-scale pixel-level feature maps of the same size as the original oblique satellite image of the building.

[0048] Continue to refer Figure 1 The query Transformer module is used to receive the multi-scale pixel-level feature map output by the image encoder-decoder module, and extract building instances from the multi-scale pixel-level feature map, and refine instance-specific attributes such as category, building boundary, height and offset vector by processing pixel-level features.

[0049] The query transformer module can be composed of a query unit and multiple cascaded cross-self-attention (CSA) units. Its query unit is used to generate learnable query and query position embeddings. Its multiple cascaded cross-self-attention (CSA) units are used to gradually pixel-level features to extract attribute information of building instances.

[0050] In one embodiment, reference Figure 2,Each cross-self-attention unit can be equipped with masked cross-attention, self-attention and feed-forward network, thus achieving detailed interaction of instance-level features of buildings.

[0051] Specifically, we first generate learnable queries through the query module and query position embedding Masked cross-attention is then calculated between the query and the pixel-level features output by the pixel decoder, masking out regions with low probability of containing the target, allowing each query to focus on relevant local features. Furthermore, self-attention is calculated between queries to capture semantically meaningful global features. Finally, attribute information for building instances is generated through the self-attention mechanism and a feedforward network.

[0052] Back to Figure 1 The multi-task encoder receives the attribute information of the building instance extracted by the query transformer module through the multi-layer perceptron, and separates the building instance from the background or other instances based on its attribute information to generate the key attributes of the building instance. In addition, the building outline boundary and category label can be generated based on the key attributes of the building instance.

[0053] Specifically, the multi-task decoder can be composed of a multi-layer perceptron. Each instance query vector output by its query Transformer module is a global semantic representation of the building instance, containing information such as its shape, location, and context (i.e., the attribute information of the building instance). Through the multi-layer nonlinear transformation (fully connected layer + activation function) of the multi-layer perceptron, the high-dimensional vector is gradually mapped to the target space, and the building information is decoded into a pixel-level probability distribution. This distribution separates the building instance from the background and other instances through thresholding or competition mechanisms, and at the same time predicts the key attributes of the building instance (such as height, type, etc.) through the multi-layer perceptron branch.

[0054] Continue to refer Figure 1 ,The height decoder is used to receive the key attributes of the building instance output by the ,multitask decoder, and extract the height features based on the key attributes of the ,building instance, and then use the height features to predict the building height.

[0055] Specifically, the height decoder can include intra-instance attention, inter-instance attention maps, and an adaptive binned height regressor. The intra-instance attention is used to recalibrate key attribute features of building instances. The inter-instance attention map is used to model the spatial relationship between buildings and recalibrate inter-instance features to obtain height features. The adaptive binned height regressor is used to predict building heights based on the height features output by the inter-instance attention map.

[0056] In one embodiment, Figure 3This paper presents an inter-instance attention graph architecture, which includes a multi-layer graph neural network. This multi-layer graph neural network is used to comprehensively understand the complex relationships between buildings and the spatial dynamics.

[0057] Specifically, continue to refer to Figure 3 , two multi-layer perceptrons are used to generate query and key vectors for the two building instances, respectively. These vectors are then used to calculate the relationship strength between the instances and ultimately normalize them into adjacency weights, forming an adjacency matrix. This constructed adjacency matrix is then used to gradually transfer and fuse information between nodes through a graph neural network, thereby integrating the spatial and attribute relationships between nodes and modeling the spatial relationships between buildings.

[0058] Next, based on the above content, a building information extraction method based on monocular side-view satellite imagery provided in an embodiment of the present application is introduced.

[0059] Figure 4 FIG2 shows a flow chart of a method for extracting building information based on a monocular side-view satellite image provided by an embodiment of this specification. Figure 4 As shown, the method includes the following steps:

[0060] In step S101 , an image encoding-decoding module is used to extract and enhance multi-scale features of the inclined satellite image of the building to obtain multi-scale pixel-level features.

[0061] Specifically, the image encoder can use a CNN encoder (such as VGG, ResNet, DenseNet), a multi-scale feature encoder (such as FPN, HRNet, UNet) or a Transformer encoder (such as Vision Transformer, SwinTransformer). The image encoder can be used to extract multi-level features from oblique satellite images of buildings, retaining high-resolution details (such as edges) in the shallow layer and capturing high-semantic information (such as building outlines) in the deep layer. Multi-scale feature enhancement modules (such as Atrous Spatial Pyramid Pooling (ASPP), Pyramid Pooling Module (PPM), Feature Pyramid Network (FPN), etc.) can be used to enhance multi-level features. Pixel decoding is then performed through a pixel decoder to obtain multi-scale pixel-level features.

[0062] Optionally, the image encoder can use a Swin Transformer backbone network, which can be pre-trained on the ImageNet model using a large amount of oblique satellite imagery data and used to extract features at multiple resolution scales of oblique satellite imagery of buildings.

[0063] Furthermore, the pixel decoder can be combined with the feature pyramid network and the deformable detection transformer. Specifically, the pixel decoder processes the oblique satellite image of the building to generate multiple feature maps at different scales, such as and In order to be compatible with the original image size, the pixel decoder upsamples these feature maps to obtain multi-scale pixel-level feature maps F′={F′1,F′2,F′3,F′4} and mask features

[0064] Step S102 : gradually fuse multi-scale pixel-level features by querying the Transformer module to extract attribute information of the building instance.

[0065] Specifically, in the query transformer module-based pixel-level feature fusion model, multiple cascaded cross-scale self-attention (CSA) units form the core architecture of the query transformer module. This design adopts a coarse-to-fine progressive processing strategy, first modeling the global context of low-resolution features, and then gradually transferring semantic information to high-resolution features for local refinement.

[0066] The model processing pipeline begins with the multi-scale feature pyramid extracted by the backbone network in the previous step. The lowest-resolution features are first processed by a cross-scale self-attention (CSA) unit to generate a global contextual representation that includes the overall distribution and category information of buildings. This contextual information, acting as prior knowledge, is then injected into adjacent higher-resolution feature maps via a cross-scale attention mechanism. In this process, high-level features provide semantic guidance, while low-level features provide geometric details, forming a two-way information exchange. As the processing scale progresses from low to high, the global building layout information is gradually refined into the attributes of specific building instances.

[0067] Step S103 : Based on the attribute information of the building instance, the building instance is segmented using a multi-layer perception mechanism to generate key attributes of the building instance, and the outline boundary and category label of the building are generated according to the key attributes of the building instance.

[0068] The output of the query Transformer module in the previous step generates building instance attribute information, enriched with local and global context. This attribute information is then processed through different multi-layer perceptron (MLP) segmentation processes to generate key building instance attributes. Finally, the building's outline boundary and category label are generated based on the key building instance attributes.

[0069] In one embodiment, the key attributes of building instances may include multiple embeddings, such as instance-level mask embedding, classification prediction logits (logits represent the model's raw prediction scores for input samples (unnormalized output)), offset vector embedding, height embedding, and binned embedding. The classification prediction logits may be normalized to obtain the building's category label.

[0070] It's worth noting that the offset vector (i.e., the roof-to-base offset vector) describes the spatial displacement between a building pixel in an image and its true orthogonal position (i.e., its correct coordinates from the top view). This offset vector quantifies the displacement between the building roof and its corresponding base in the side-view image, providing physically interpretable height cues for monocular height estimation.

[0071] In step S104, height features are extracted from the key attributes of building instances through the intra-instance attention mechanism and the inter-instance attention mechanism, and then the height features are used to predict the building height through the adaptive binning regression strategy.

[0072] Specifically, the key attribute information of building instances generated by the cascaded cross-attention units and multi-layer perceptron segmentation contains rich information, but may contain redundant data. To streamline the information and focus on truly relevant content, this embodiment applies an intra-instance attention mechanism to the key attribute features of building instances.

[0073] The relationship between building instances is crucial for monocular height estimation, and even more critical for side-view monocular height estimation. Therefore, this embodiment extracts height features from key attribute information of building instances through an inter-instance attention mechanism.

[0074] Finally, an adaptive binning regression strategy is used to predict building heights using height features extracted from key building instance attributes. This strategy combines classification and regression methods to achieve accurate building height prediction. The core idea is to transform the continuous height prediction problem into a joint optimization problem involving discrete binning classification and regression of within-bin offsets.

[0075] Step S105: Output the outline boundary, category label, and building height of the building to complete the extraction of the three-dimensional information of the building.

[0076] As can be seen from this embodiment, the instance-level multi-task learning framework of side-view satellite imagery can overcome the fundamental limitation of traditional orthophotos that lack direct height clues. The multi-layer perceptron segmentation method for building instances replaces the traditional pixel-level semantic segmentation to more accurately characterize the overall characteristics of the building as an independent entity, which meets the actual application requirements that building products are usually organized in the form of instances (instance-based organization means that building data is required to have independent identification: each building has an independent ID, and it also has complete encapsulation: a single instance contains all its own information (geometry + attributes)). Through intra-instance feature recalibration and inter-instance spatial relationship modeling, the contextual information of the building group is fully utilized to address the challenge of insufficient feature interaction.

[0077] In one embodiment, the process of extracting attribute information of a building instance by querying the Transformer module may include:

[0078] First, learnable query and query position embeddings are generated.

[0079] Exemplary, reference Figure 2 , first generate learnable queries through the query module and query position embedding where N Q Indicates the number of queries, that is, the number of target objects that the model can detect simultaneously. Optional, set to N Q =200. E Q Indicates the feature dimension used to characterize the instance, optional, set to E Q =256.

[0080] Since the self-attention mechanism in Transformer does not essentially process the order of elements in the sequence, that is, it calculates the relationship between elements without knowing the arrangement of the elements, it creates the same Query position embedding with dimension matching These embeddings are designed to provide N Q The order of the instances and the E Q The order of dimension features.

[0081] Then, we compute masked cross-attention between the query and the pixel-level features output by the pixel decoder, while masking out regions with low probability of containing the target, so that each query focuses on relevant local features.

[0082] For example, continue to refer to Figure 2 , through a multi-layer perceptron (the multi-layer perceptron is part of the query Transformer module and is different from the multi-layer perceptron described in the multi-task decoder) from the query Generate pixel-level mask embeddings Then, through the mask feature F mask Embedding with pixel-level masks The product of creates an attention map where N h represents the number of attention heads, mask feature F mask The attention map AM can be obtained by upsampling the tilted satellite image feature map of the building by the pixel decoder in step S101. i Sized to fit F' i The dimensions of the mask are flattened. i By focusing on the attention map AM i The sigmoid activation function is applied and then thresholded using a standard threshold of 0.5 to generate a binary mask.

[0083] Furthermore, self-attention between queries is calculated to capture global features with semantic meaning.

[0084] Specifically, the masked crisscross attention process computes three key components: query key Sum To adapt to the permutation invariance property of transformer, position embeddings need to be integrated before computing the attention weights between query and key.

[0085] For example, continue to refer to Figure 2 , embed the location Add to initial query Similarly, embed the position Add to pixel-level feature map F i Generate a key In addition, we introduce scale-level embedding level To accurately assign each query pixel to its corresponding feature layer. The specific process can also be expressed by the following formula:

[0086]

[0087] This comprehensive approach enriches queries with masked attention maps of building instances, optimizing attention to relevant local features. The calculation is as follows:

[0088]

[0089] Finally, the attribute information of the building instance is generated through the self-attention mechanism and feedforward network.

[0090] For example, continue to refer to Figure 2 , mask query After processing by the self-attention mechanism and feedforward neural network, an updated query is generated Then from the updated query The attribute information of the image can be obtained.

[0091] In one embodiment, the process of generating the outline boundary of a building based on key attributes of a building instance may include:

[0092] First, pixel-level features are converted into pixel embeddings through a multi-layer perceptron.

[0093] For example, back to Figure 1 , the initial pixel-level features F′ can be converted into pixel embeddings through a multi-layer perceptron (MLP)

[0094] The instance-level mask embedding is then fused with the pixel embedding to produce a spatially consistent representation that captures the segmented building instances in the image and generates contour boundaries. Specifically, the instance-level mask embedding is obtained by segmenting the building instances using a multi-layer perceptron. In other words, the instance-level mask embedding is one of the key attributes of the building instances.

[0095] For example, continue to refer to Figure 1 , in order to calculate the boundary prediction mask The instance-level mask embedding can be combined with the pixel embedding F ebed Perform a matrix multiplication operation. This operation effectively fuses the instance-level mask embedding with the pixel embedding to produce a spatially consistent representation that captures the building instances segmented in the image. Through this process, the model successfully transforms the abstract, learned features into an actionable and interpretable instance segmentation output (in this example, the instance segmentation output consists of multiple boundary prediction masks, each corresponding to an independent object instance), and then obtains the building outline boundary from the instance segmentation output.

[0096] In one embodiment, before extracting height features from key attributes of building instances through intra-instance attention and inter-instance attention mechanisms, the location attributes of building instances and the scale attributes of images can be introduced.

[0097] Specifically, the location attribute can use the geometric center coordinates of the building instance boundary prediction To express.

[0098] In addition, the scale attribute information of the image can be expressed by the following formula:

[0099]

[0100] in, Refers to the benchmark scale, Refers to standard resolution images, r spatial Indicates the spatial resolution of the current image, The width ratio is W / W0, where W represents the actual width of the image and W0 is the standard width.

[0101] For example, regarding scale attributes, the base scale Can be 1.0, and this base scale corresponds to the spatial resolution The standard image size is 1024×1024. Its standard width can be W0=1024.

[0102] In one embodiment, a method is used to extract height features from key attributes of building instances through intra-instance attention mechanism and inter-instance attention mechanism, and then use the height features to predict building height through an adaptive binning regression strategy, such as Figure 5 As shown, specifically including:

[0103] In step S201, the key attribute features of the building instance are recalibrated using the instance internal attention mechanism.

[0104] Specifically, its input features (i.e., key attribute features of building instances) including highly embedded Category prediction Instance-level mask embedding Offset vector embedding Etc., these diverse components together constitute the input features Optionally, the instance center coordinates in the previous embodiment may also be included and scale ratio

[0105] To recalibrate and optimize the above features, the intra-instance attention module processes the input features as follows

[0106] First, the features are normalized, and then the intermediate features are generated by the multi-layer perceptron (MLP), which is recorded as

[0107]

[0108] in, Refers to the key attribute characteristics of a building instance.

[0109] Subsequently, these intermediate features are recalibrated based on the attention between different feature heads through the Multi-Head Attention mechanism to generate recalibrated features.

[0110]

[0111] Through this intra-instance attention mechanism, the model can effectively focus on the key features in each building instance, optimize the representation of instance-specific attributes, and prepare for inter-instance attention.

[0112] The core goal of the intra-instance attention mechanism is to identify and extract key features from building instance attributes. While the instance attributes generated by the multi-task decoder and query Transformer decoder are rich in information, they may contain redundant or irrelevant information. For example, among building boundary features, the geometry of the roof outline is more critical for height estimation, while texture details are less important. Through the attention mechanism, the model can adaptively strengthen the weights of key features. In-instance attention optimizes the information extraction process by selectively focusing on the most relevant features.

[0113] In step S202, the spatial relationship between buildings is modeled through the inter-instance attention map, and the inter-instance features are recalibrated to obtain height features.

[0114] For example, the inter-instance attention map can be expressed as in represents the node set, and ε represents the edge set. Each node Corresponding to a building instance, it contains multimodal aggregate features such as instance mask, label, location, roof-to-base offset and scale. Each edge e ij ∈ε represents two building instances v i and v j The relationship between them can be expressed as:

[0115]

[0116] R i,j =f r1 (v i )·f r2 (v j ) T

[0117] where f r1 and f r2 are two learnable MLPs, R i,j It is v i and v j The relationship score between them.

[0118] Then, the graph neural network propagates information along the edges through a message passing paradigm, synthesizing the spatial and attribute relationships between nodes. Finally, the output of the inter-instance attention map is processed to achieve recalibration of inter-instance features.

[0119] By using inter-instance attention maps and graph neural networks, we can gain a deeper understanding of the spatial and relational context surrounding each building instance. Furthermore, modeling the relationships between building instances is crucial for monocular height estimation, and even more so for side-view monocular height estimation. This is because occlusion caused by foreground buildings or shadows can prevent the accurate extraction of offset vector information for some buildings. In such cases, leveraging the principle that buildings in the same area or in close proximity have similar heights can significantly aid in estimating the height of the occluded building.

[0120] In one embodiment, a method for modeling the spatial relationship between buildings using an inter-instance attention map is as follows:

[0121] First, an adjacency matrix is constructed based on the semantic similarity of key attribute features of building instances.

[0122] Exemplary, reference Figure 3 , two multi-layer perceptrons can be used to generate query vectors and key vectors of two building instances respectively, and then the relationship strength between instances is calculated based on these vectors, and finally normalized into adjacency weights to form an adjacency matrix.

[0123] For any two instance i and j, the feature vector x i and x j , calculate the relationship strength:

[0124] q i =W q x i ,k j =W k x

[0125]

[0126] Among them, W q and W k is the learnable projection matrix, q i and k j are the query vector and key vector of instance i and j respectively, s ij is the relationship strength between instances i and j.

[0127] Finally, it is normalized to the adjacency weight and forms the adjacency matrix:

[0128]

[0129] Among them, k represents each instance in the instance set, a ij is the normalized adjacency weight, forming the adjacency matrix A.

[0130] Then, the constructed adjacency matrix gradually transmits and fuses the information between nodes through the graph neural network, thereby integrating the spatial and attribute relationships between nodes to complete the modeling of the spatial relationship between buildings.

[0131] For example, continue to refer to Figure 3 Using the constructed adjacency matrix, the three-layer graph convolution operation of the graph neural network can gradually transmit and fuse information between nodes. Through multiple layers of graph convolution, each node can gradually obtain information from directly adjacent and indirectly connected nodes, forming a multi-scale receptive field. The graph neural network then propagates information along the edges through a message passing paradigm, integrating the spatial and attribute relationships between nodes.

[0132] As can be seen from this implementation, the inter-instance feature recalibration is achieved through the inter-instance attention map. Leveraging the power of graph neural networks, a deeper understanding of the spatial and relational context surrounding each building instance is achieved.

[0133] Step S203: Based on the height feature, the building height is predicted using an adaptive binned height regressor.

[0134] Specifically, the building height features are input into the adaptive binning height regressor, which adaptively generates binning intervals and predicts the probability score and offset of the height belonging to each bin, thereby outputting the final height.

[0135] In one embodiment, this solution proposes a new method for predicting building height using an adaptive binned height regressor. The method specifically includes:

[0136] First, the center width of each bin is calculated based on the adaptive bin embedding, and the calculation formula is as follows:

[0137]

[0138] in, Refers to the normalized center width of each bin, ∈ is a small constant to avoid division by zero, Refers to Adaptive Binned Embedding.

[0139] It is worth noting that its adaptive binning embedding Generated by the query Transformer module and processed by a multi-layer perceptron binning regressor, they are then activated by the LeakyReLU function and normalized (the processing process refers to the above formula calculation process) to generate the final bin

[0140] Then, calculate the height of the center of each bin using the following formula:

[0141]

[0142] in, Refers to the height of the center of each bin, h min and h max Refers to the preset minimum and maximum heights.

[0143] Finally, the predicted height of the building is calculated by combining the height feature of the building instance with the corresponding bin center height. The calculation formula is as follows:

[0144]

[0145] in, refers to the predicted height of the building, Refers to the height of the center of each bin, Refers to the height characteristics of an architectural instance.

[0146] As can be seen from this example, the proposed new solution eliminates the negative impact of using the softmax function on instance-level accuracy on height features, such as error amplification caused by softmax, suppression of neighboring relationships by competition between bins, and sensitivity to small instances. By accurately aligning the binning strategy with the unique feature attributes of each instance, the refined method proposed in this example can significantly improve the accuracy and adaptability of the model in height estimation.

[0147] The results of building information extraction using this solution show that there are differences between the orthophoto method and the side view image method in terms of feature extraction and height information acquisition. Figure 6 Left: Orthophoto using pixel-level feature extraction, including (top) identification of pixel features and pixel relationships and (bottom) pixel-level height labels; reference Figure 6 Right: Side-view images utilize instance-level feature extraction, including (top) modeling of building instance features and relationships between them, and (bottom) instance-level height labeling and annotation. This side-view image approach captures roof-to-base offset information, providing crucial clues for height estimation.

[0148] Corresponding to the above method provided by the present invention, the present invention also provides a device. Figure 7 FIG. 1 shows a schematic diagram of a building information extraction device based on a monocular side-view satellite image provided by an embodiment of this specification. Figure 7 As shown, the device includes:

[0149] The image encoding-decoding module 301 is used to extract and enhance multi-scale features of inclined satellite images to obtain multi-scale pixel-level features.

[0150] The Transformer module 302 is queried to gradually fuse multi-scale pixel-level features to extract attribute information of building instances.

[0151] The multi-task decoder 303 is configured to segment the building instances using a multi-layer perception mechanism based on the building instance attribute information to generate key attributes of the building instances and generate the building outline based on the key attributes of the building instances. The key attributes of the building instances include category labels.

[0152] The height decoder 304 is used to extract height features from the key attributes of building instances through the intra-instance attention mechanism and the inter-instance attention mechanism, and then use the height features to predict the building height through the adaptive binning regression strategy.

[0153] The building information output module 305 is used to output the outline boundary, category label and building height of the building to complete the extraction of the three-dimensional information of the building.

[0154] It is worth mentioning that Figure 7 The device shown is Figure 4 The method embodiments shown correspond to and can be applied to Figure 4 The multi-scale pixel-level feature extraction, attribute information extraction of building instances, extraction of key attributes of building instances, prediction of building heights, and output of building outline boundaries, category labels, and building height prediction values in the illustrated method embodiments will not be described in detail here. Those skilled in the art should easily appreciate that, in combination with the units and algorithm steps of the various examples described in the embodiments disclosed herein, the present application can be implemented in the form of hardware or a combination of hardware and computer software. Whether a function is executed in hardware or in a computer software-driven hardware manner depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0155] According to another embodiment, a computing device is provided, including a memory and a processor, wherein the memory stores executable code, and when the processor executes the executable code, the system realizes the combination of Figure 4 The method described.

[0156] According to another embodiment, there is also provided a computer readable storage medium having a computer program stored thereon, which, when executed in a computer, causes the computer to execute a combination of Figure 4 The method described.

[0157] Those skilled in the art will appreciate that, in one or more of the above examples, the functions described herein may be implemented using hardware, software, firmware, or any combination thereof. When implemented using software, these functions may be stored in a computer-readable medium or transmitted as one or more instructions or codes on a computer-readable medium.

[0158] The specific implementation methods described above further illustrate the objectives, technical solutions and beneficial effects of the present invention in detail. It should be understood that the above description is only a specific implementation method of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent replacements, improvements, etc. made on the basis of the technical solution of the present invention should be included in the scope of protection of the present invention.

Claims

1. A method for extracting building information based on monocular side-view satellite images, characterized in that: The method comprises: The image encoding-decoding module is used to extract and enhance multi-scale features from oblique satellite images of buildings to obtain multi-scale pixel-level features. By querying the Transformer module, multi-scale pixel-level features are gradually integrated to extract the attribute information of the building instance; Based on the attribute information of building instances, a multi-layer perception mechanism is used to segment building instances to generate key attributes of building instances, and then the outline boundary and category label of the building are generated according to the key attributes of the building instances; We extract height features from key attributes of building instances through intra-instance and inter-instance attention mechanisms, and then use the height features to predict building heights through an adaptive binning regression strategy. Output the building's outline boundary, category label, and building height prediction value to complete the extraction of the building's three-dimensional information.

2. The method according to claim 1, characterized in that The process of extracting attribute information of building instances by querying the Transformer module includes: Initialize the query Transformer module to generate learnable query and query position embeddings; Compute masked cross-attention between the query and the pixel-level features output by the pixel decoder, while masking out areas with low probability of containing the target, so that each query focuses on relevant local features; Compute self-attention between queries to capture semantically meaningful global features; Generate attribute information of building instances through a feed-forward network.

3. The method according to claim 1, characterized in that The key attributes of the building instances include instance-level mask embedding, classification prediction logits, offset vector embedding, height embedding, and bin embedding.

4. The method according to claim 1, wherein The process of generating the building's outline boundary includes: Convert pixel-level features into pixel embeddings through a multi-layer perceptron; Instance-level mask embeddings are fused with pixel embeddings to generate a spatially consistent representation that captures the segmented building instances in the image to generate contour boundaries; among them, the instance-level mask embeddings are obtained by segmenting the building instances through a multi-layer perceptron.

5. The method according to claim 1, wherein Before extracting height features from key attributes of building instances through intra-instance attention and inter-instance attention mechanisms, the location attributes of building instances and the scale attributes of images are introduced; The location attribute is represented by the geometric center coordinates of the building instance outline boundary prediction; The scale attribute information of the image is quantified by a scale factor to provide accurate location information of the building instance in the image. The scale factor is calculated by the following formula: in, Refers to the benchmark scale, Refers to standard resolution images, r spatial Indicates the spatial resolution of the current image, The width ratio W / W0, W represents the actual width of the image, and W0 is the standard width.

6. The method according to claim 1, characterized in that The method extracts height features from key attributes of building instances through intra-instance attention mechanism and inter-instance attention mechanism, and then uses height features to predict building height through adaptive binning regression strategy, specifically including: Adopting the instance-internal attention mechanism to recalibrate the key attribute features of building instances; The spatial relationship between buildings is modeled through the inter-instance attention map, and the key attribute features between building instances are recalibrated to obtain height features; Based on the height features, an adaptive binned height regressor is used to predict building heights.

7. The method according to claim 6, characterized in that The method of modeling the spatial relationship between buildings through the inter-instance attention map includes: Construct an adjacency matrix based on the semantic similarity of key attribute features of building instances; For the attribute feature vectors of any two building instances, calculate their relationship strength; The constructed adjacency matrix gradually transmits and fuses the information between nodes through the graph neural network, thereby integrating the spatial and attribute relationships between nodes.

8. The method according to claim 6, characterized in that Based on the height feature, the method of predicting building height using adaptive binned height regressor includes: The center width of each bin is calculated based on the adaptive bin embedding, and the calculation formula is as follows: in, Refers to the normalized center width of each bin, ∈ is a small constant to avoid division by zero, Refers to adaptive binning embedding; Then calculate the height of the center of each bin using the following formula: in, Refers to the height of the center of each bin, h min and h max Refers to the preset minimum and maximum heights; The predicted height of the building is calculated by the height feature of the building instance and the corresponding bin center height. The calculation formula is as follows: in, refers to the predicted height of the building, Refers to the height of the center of each bin, Refers to the height characteristics of an architectural instance.

9. A building information extraction device based on monocular side-view satellite images, characterized in that: Building information extraction is performed by the method according to any one of claims 1 to 8, wherein the device comprises: The image encoding-decoding module is used to extract multi-scale features from oblique satellite images through an image encoder and enhance the multi-scale features through a pixel decoder to obtain multi-scale pixel-level features; The query Transformer module is used to gradually fuse multi-scale pixel-level features to extract attribute information of building instances; A multi-task decoder is used to segment building instances based on their attribute information using a multi-layer perception mechanism to generate key attributes of the building instances and generate building outlines based on the key attributes of the building instances; the key attributes of the building instances include category labels; Height decoder, which extracts height features from key attributes of building instances through intra-instance and inter-instance attention mechanisms, and then uses the height features to predict building heights through an adaptive binning regression strategy; The building information output module is used to output the building's outline boundary, category label and building height to complete the extraction of the building's three-dimensional information.

10. An electronic device comprising a memory and a processor, wherein the memory stores a computer program that can be run on the processor, wherein: When the processor executes the computer program, the method according to any one of claims 1 to 8 is implemented.

Images