Urban building proxy reconstruction method based on aerial images

Abstract

The application relates to a kind of city building agent reconstruction methods based on aerial image.The method comprises: obtaining the multi-view image corresponding to target building;Multi-view image includes near vertical view image and inclined view image;According to near vertical view image, the bottom contour of target building is determined;Determine the first depth difference between each pixel point in inclined view image;According to the camera pose parameter of multi-view image and the current predicted height of target building, rendering is carried out, and the rendering depth map of target building under the current predicted height is obtained;With the distance between the second depth difference of each pixel point in the rendering depth map and the first depth difference as the optimization target, the current predicted height is optimized, and according to the bottom contour and target measurement height, the three-dimensional building model corresponding to target building is constructed.Using the method can improve the efficiency and accuracy of city building agent reconstruction.

Description

Technical Field

[0001] This application relates to the field of computer data processing technology, and in particular to a method for urban building proxy reconstruction based on aerial images. Background Technology

[0002] With the development of applications such as digital twin cities, urban planning simulation, drone path planning, and augmented reality, how to construct three-dimensional models of buildings in urban scenarios has become a key technical issue in the field of intelligent urban modeling.

[0003] In related technologies, the mainstream method for reconstructing 3D building models from aerial images is based on dense 3D reconstruction. However, this method has significant limitations in practical applications. Specifically, it is complex, computationally expensive, requires high image quality, and has low resource utilization.

[0004] Therefore, the key challenge is to simplify the 3D reconstruction process, reduce computational complexity, and improve robustness to image quality while ensuring the accuracy of the building's main volume information. Summary of the Invention

[0005] Therefore, it is necessary to provide a method for urban building proxy reconstruction based on aerial images that can improve modeling efficiency and accuracy, addressing the aforementioned technical problems.

[0006] In a first aspect, this application provides a method for urban building proxy reconstruction based on aerial images, the method comprising:

[0007] Acquire multi-view images of the target building; the multi-view images include near-vertical view images and oblique view images;

[0008] The bottom outline of the target building is determined based on the near-vertical view image;

[0009] Determine the first depth difference between each pixel in the tilted view image;

[0010] Rendering is performed based on the camera pose parameters of the multi-view image and the current predicted height of the target building to obtain a rendering depth map of the target building at the current predicted height;

[0011] With the goal of reducing the distance between the second depth difference and the first depth difference between each pixel in the rendered depth map, the current predicted height is optimized to obtain the target measured height of the target building;

[0012] Based on the bottom outline and the target measured height, a three-dimensional building model corresponding to the target building is constructed.

[0013] Secondly, this application also provides a device for urban building proxy reconstruction based on aerial images, the device comprising:

[0014] The acquisition module is used to acquire multi-view images of the target building; the multi-view images include near-vertical view images and oblique view images.

[0015] The first determining module is used to determine the bottom outline of the target building based on the near-vertical view image;

[0016] The second determining module is used to determine the first depth difference between each pixel in the tilted view image;

[0017] The rendering module is used to render the target building based on the camera pose parameters of the multi-view image and the current predicted height of the target building, so as to obtain a rendering depth map of the target building at the current predicted height.

[0018] The optimization module is used to optimize the current predicted height with the goal of reducing the distance between the second depth difference and the first depth difference between each pixel in the rendered depth map, so as to obtain the target measured height of the target building.

[0019] A construction module is used to construct a three-dimensional building model corresponding to the target building based on the bottom outline and the target measured height.

[0020] Thirdly, this application also provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps included in any of the foregoing method embodiments.

[0021] Fourthly, this application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps included in any of the foregoing method embodiments.

[0022] Fifthly, this application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps included in any of the foregoing method embodiments.

[0023] The aforementioned urban building proxy reconstruction method, apparatus, computer equipment, computer-readable storage medium, and computer program product based on aerial imagery transforms the complex problem of "dense reconstruction + geometric abstraction" into a low-dimensional parameter solution problem of "contour extraction + height optimization" by representing buildings as parametric volumetric models formed by stretching the bottom contour along the vertical direction. Furthermore, it directly extracts the bottom contour from near-vertical view images and optimizes the height parameters using depth order information from tilted view images. The entire process avoids feature matching and dense reconstruction, effectively mitigating the failure risks caused by image quality issues in related technologies. This embodiment achieves precise matching of computational resources and modeling objectives, significantly reducing unnecessary computational overhead while ensuring the accuracy of the building's main volumetric information. Attached Figure Description

[0024] To more clearly illustrate the technical solutions in the embodiments of this application or related technologies, the drawings used in the description of the embodiments of this application or related technologies will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0025] Figure 1 This is an application environment diagram of an urban building proxy reconstruction method based on aerial images in one embodiment;

[0026] Figure 2 This is a flowchart illustrating a method for urban building proxy reconstruction based on aerial images in one embodiment;

[0027] Figure 3 This is a structural block diagram of an urban building proxy reconstruction device based on aerial images in one embodiment;

[0028] Figure 4 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0029] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0030] Before describing the embodiments of this application, the relevant technologies and their existing problems will be further explained:

[0031] Aerial imagery offers advantages such as wide coverage, high acquisition efficiency, and repeatable observation, making it a crucial data source for 3D reconstruction of urban scenes. In many applications, it is typically only necessary to acquire information on the spatial layout and main mass of buildings (i.e., 3D proxy models of buildings), without needing to focus on the fine geometric details of building facades.

[0032] In related technologies, dense point clouds or 3D meshes are typically generated from multi-view aerial images using Structure-from-Motion (SfM) and Multi-View Stereo (MVS). Then, based on these meshes, operations such as plane detection, geometric fitting, or structured abstraction are used to extract proxy models of buildings. For example, some methods first reconstruct the dense point cloud of the scene using SfM and MVS, then use plane segmentation and fitting algorithms to identify individual facets of the building, and finally combine them to form a parametric model of the building. These methods achieve good reconstruction results when the point cloud quality is high and the building structure is regular.

[0033] The aforementioned methods suffer from several problems: they require multiple processing steps, including feature extraction, feature matching, adjustment calculation, dense matching, mesh generation, and geometric fitting, resulting in a complex overall process with high computational costs. Processing a medium-sized urban scene often takes several hours or even longer. Furthermore, they are highly demanding in terms of image quality. When image overlap is insufficient, feature matching quality is poor (e.g., glass curtain walls, areas with repetitive textures), or occlusion exists, the generated dense point cloud is prone to holes, noise, or even reconstruction failure, leading to a lack of reliable data foundation for subsequent proxy extraction processes. Moreover, for applications requiring only building mass information, reconstructing a high-precision dense point cloud or mesh before abstraction is essentially a redundant computational pattern of "fine-first, then simplification," wasting significant computational resources on subsequently discarded geometric details.

[0034] Therefore, how to simplify the 3D reconstruction process, reduce computational complexity, and improve robustness to image quality while ensuring the accuracy of the building's main volume information is a problem that needs to be solved in the field of architectural modeling.

[0035] It should be noted that the terms "first," "second," etc., used in this application can be used to describe various elements, but these elements are not limited by these terms. These terms are only used to distinguish the first element from the second element. The terms "comprising" and "having," and any variations thereof, used in this application, are intended to cover non-exclusive inclusion. The term "multiple" used in this application refers to two or more. The term "and / or" used in this application refers to one of the embodiments, or any combination of multiple embodiments.

[0036] The urban building proxy reconstruction method based on aerial images provided in this application can be applied to, for example... Figure 1 In the application environment shown, terminal 102 communicates with server 104 via a network. A data storage system can store the data that server 104 needs to process. The data storage system can be integrated onto server 104 or located on the cloud or other network servers. Terminal 102 can be, but is not limited to, various personal computers, laptops, smartphones, tablets, drones, low-altitude aircraft, IoT devices, and portable wearable devices. IoT devices can include smart speakers, smart TVs, smart air conditioners, smart in-vehicle devices, projection devices, etc. Portable wearable devices can include smartwatches, smart bracelets, head-mounted devices, etc. Head-mounted devices can be virtual reality (VR) devices, augmented reality (AR) devices, smart glasses, etc. Server 104 can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing cloud computing services.

[0037] In one exemplary embodiment, such as Figure 2 As shown, a method for urban building proxy reconstruction based on aerial images is provided, which can be applied to... Figure 1 Taking server 104 or terminal 102 as an example, the following steps are included:

[0038] Step 202: Obtain multi-view images corresponding to the target building; the multi-view images include near-vertical view images and oblique view images.

[0039] The target building can be at least one building in the target urban scene. Multi-view images refer to a collection of aerial images acquired from different shooting angles of the same target building. These aerial image sets are acquired by low-altitude flight platforms such as drones equipped with imaging sensors, and their shooting angles cover a variety of angles, from approximately vertical downwards to angles significantly tilted to the horizontal. For example, multi-view images can include orthophotos taken by a drone during vertical descent, facade images taken at an angle during hovering, and overlay images taken at different flight paths.

[0040] A near-vertical view image refers to an image where the angle between the imaging optical axis and the vertical direction (i.e., the direction of gravity) is less than a preset angle threshold (e.g., 30°). In a near-vertical view image, the roof plane of a building is approximately orthographically projected with minimal geometric distortion, accurately reflecting the planar boundary position where the building contacts the ground. Correspondingly, an oblique view image refers to an image where the angle between the imaging optical axis and the vertical direction is greater than or equal to the preset angle threshold. In an oblique view image, the building facade is clearly presented, providing geometric information in the lateral contour and height directions of the building. This embodiment simultaneously acquires both near-vertical and oblique view images, providing information sources for subsequent building base contour extraction and height measurement. Near-vertical view images, with their smaller perspective distortion, are suitable for determining the two-dimensional planar range of a building on the ground, while oblique view images, with their full representation of the building facade, are suitable for providing depth cues related to building height. This clear division of labor and complementary information between the two types of images avoids the problem of insufficient information when using a single type of image.

[0041] Step 204: Determine the bottom outline of the target building based on the near-vertical view image.

[0042] The bottom outline refers to the two-dimensional projection boundary of the target building on the ground (or a reference horizontal plane), which can be represented as a closed vector polygon. This vector polygon describes the area occupied by the building on the horizontal plane, including its spatial location and planar shape. For example, for an office building with a rectangular plan, its bottom outline can be a rectangle composed of four vertices; for a commercial building with an L-shaped plan, its bottom outline is an L-shaped polygon composed of multiple vertices.

[0043] In this embodiment, considering the high geometric correspondence between the roof boundary and the actual ground contact boundary of the building in the near-vertical view image, and the small perspective distortion of the image, the contour extracted from the near-vertical view image can accurately reflect the actual planar area occupied by the building. When constructing the 3D building model, the bottom contour of the building defines the horizontal boundary of the building and can serve as the "bottom surface" constraint for subsequently determining the 3D volume of the building. Unlike extracting contours directly from tilted view images, this embodiment utilizes near-vertical view images to avoid boundary offset and perspective distortion problems caused by tilted viewpoints, thereby obtaining a more reliable bottom contour.

[0044] Specifically, given one or more near-vertical view images as input, a pre-trained deep learning segmentation model can be used to extract building area masks from the images. This segmentation model has been fine-tuned using a large number of remote sensing images labeled with building boundaries, enabling it to adapt to building extraction tasks under different urban styles and lighting conditions.

[0045] After obtaining the building area mask, the mask is subjected to contour extraction and vectorization. For example, a boundary tracking algorithm can be used to extract the pixel-level contour boundary of the building area from the binary mask image, and then the pixel boundary is fitted into a closed vector polygon composed of several vertices. This polygon is the preliminary expression of the two-dimensional bottom contour of the building.

[0046] Considering that the original outline may have jagged edges or irregularities, the vector polygon is further regularized. Regularization may include simplifying and orthogonalizing the polygon vertices according to the shortest side length and angular constraints, ensuring that adjacent sides are perpendicular or parallel as much as possible, thus obtaining a bottom outline (footprint) that better conforms to the characteristics of an artificial building structure. The final output bottom outline is represented as a set of two-dimensional polygons on the ground reference plane (Z=0 plane), where each polygon corresponds to an independent building instance, containing the building's spatial location and boundary shape information.

[0047] Step 206: Determine the first depth difference between each pixel in the tilted view image.

[0048] In this context, depth refers to the distance from the scene's 3D point corresponding to each pixel in the tilted view image to the camera's optical center. Correspondingly, the first depth difference refers to the difference in depth values ​​between any two different pixels within the same tilted view image. This difference can be a signed numerical value: a positive difference indicates that the scene point corresponding to the first pixel is farther from the camera than the scene point corresponding to the second pixel; a negative difference indicates they are closer; and a zero difference indicates they have the same depth. The first depth difference reflects the relative order of different spatial points within the tilted view image in the depth dimension, representing a relative geometric constraint independent of absolute scale information.

[0049] For example, in a street image taken at an angle, a pixel A on the facade of a foreground building has a smaller depth value (closer to the camera), while a pixel B on the facade of a background building has a larger depth value (farther from the camera). The first depth difference between pixel pairs (A, B) can be negative (if calculated as B minus A) or positive (if calculated as A minus B), the sign depending on the calculation order. This depth difference information can characterize the occlusion relationship between different objects in the scene.

[0050] Unlike directly using absolute depth values ​​(such as the raw values ​​output by monocular depth estimation), this embodiment introduces a first depth difference, focusing on the relative order and magnitude of the differences between depths. This relative relationship is inherently robust to the scale uncertainty commonly present in monocular depth estimation. In other words, even if monocular depth estimation cannot accurately output "whether the building is 10 meters or 15 meters," it can still reliably determine "building A is farther than building B" or "point P on the wall is closer than point Q." Therefore, this embodiment extracts the relative depth relationship between pixels from the tilted view image, thus providing a stable constraint for subsequent height measurement that is unaffected by scale drift.

[0051] The calculation of the first depth difference may include: estimating the depth of each tilted view image using a monocular depth estimation model. In this embodiment, the monocular depth estimation model can be a pre-trained model based on deep learning. The tilted view image is input into the model, and the model outputs a prior depth map with the same resolution as the input image, where the value of each pixel represents the normalized depth value of the scene point corresponding to that pixel relative to the camera. It should be noted that this prior depth map only reflects the relative order of depths, and its absolute values ​​exhibit scale uncertainty.

[0052] Step 208: Render the target building based on the camera pose parameters of the multi-view image and the current predicted height of the target building to obtain a rendering depth map of the target building at the current predicted height.

[0053] Camera pose parameters refer to the camera's internal geometric parameters (such as focal length, principal point coordinates, distortion coefficients, etc.) and external spatial parameters (such as the camera's three-dimensional position coordinates and three-dimensional attitude angles in the world coordinate system) at each image acquisition moment. Camera pose parameters determine how points in three-dimensional space are projected onto the two-dimensional image plane.

[0054] The current predicted height is a numerical parameter representing the current estimate of the target building's height during the current optimization iteration. Its initial value can be set to a default value (e.g., 5 meters based on scene statistics) or obtained through other initialization methods (e.g., estimation from an image). The rendering depth map refers to projecting a 3D model of the target building with a specific predicted height onto the image plane based on known camera pose parameters. During projection, the distance from each pixel on the building model's surface to the camera's optical center is recorded, resulting in a 2D image. The value of each pixel in the rendering depth map represents the depth value of the building surface point corresponding to that pixel's location. For pixels not covered by any building model (e.g., background areas), their depth value can be set to a preset background value (e.g., infinity or a constant much larger than the scene depth).

[0055] In this embodiment, rendering technology is used to convert the currently assumed building height into visual depth information that can be compared with the actual image. Specifically, given a building's bottom outline and an assumed height value, an extruded volume model can be uniquely determined. Rendering this model onto the imaging plane of an image from a tilted viewpoint yields the depth values ​​that the building model "thinks" each pixel should have under the assumed height. This rendered depth map reflects the geometric inference results under the current height assumption. By comparing the rendered depth map with the depth information extracted from the actual image in subsequent steps, it is possible to determine whether the current height assumption is reasonable and adjust the height accordingly. This is equivalent to establishing a computable mapping from "height parameters" to "image depth representation," allowing the height parameters to be optimized using image information.

[0056] Specifically, the target building can be represented as a parametric 3D volumetric model. This model, with the aforementioned bottom contour as its base, is stretched vertically to the height determined by the current predicted height, forming a prism (for simple flat-roofed buildings) or an extruded volume (for complex buildings). Each building instance corresponds to an independent height parameter, which can be updated during subsequent optimization. Then, a differentiable renderer is used to render this parametric volumetric model. For each tilted view image participating in the optimization, based on the camera pose parameters (intrinsic and extrinsic) corresponding to the image, the building proxy model at the current height is projected onto the image's imaging plane, generating a rendered depth map with the same resolution as the image. The value of each pixel in the rendered depth map represents the distance from the corresponding point on the building model surface to the camera's optical center. For pixels not covered by any building model (such as background areas like the sky and ground), their depth value is set to the background depth (e.g., infinity or a preset large value).

[0057] Optionally, the differentiable renderer can also generate a binary mask to mark which pixels in the rendered depth map belong to the visible building area, i.e., the building surface corresponding to that pixel is not occluded by other buildings or part of itself from that viewpoint. This mask will be used to limit the effective range of depth comparison in subsequent optimizations. It should be noted that the differentiable renderer used in this embodiment supports gradient backpropagation, that is, the gradient of the optimization target relative to the rendered depth map can be further backpropagated to the building's height parameters, thereby achieving end-to-end optimization of the height parameters.

[0058] Step 210: Optimize the current predicted height by reducing the distance between the second depth difference and the first depth difference between each pixel in the rendered depth map, and obtain the target measured height of the target building.

[0059] The second depth difference refers to the difference between the depth values ​​of any two different pixels in the rendered depth map. The definition and calculation method of the second depth difference can refer to the first depth difference in the aforementioned steps, and it is used to represent the relative depth relationship (including order and difference magnitude) between two pixels. It can be understood that the first depth difference in this embodiment comes from the actual tilted view image (obtained through monocular depth estimation and other means), representing the depth relationship in the real scene; the second depth difference comes from the rendering result, representing the depth relationship of the building model under the current height assumption.

[0060] In this embodiment, the current predicted height is adjusted so that the relative depth relationship (i.e., the second depth difference) between any two pixels in the rendered depth map is as close as possible to or matches the relative depth relationship (i.e., the first depth difference) obtained from the actual image (i.e., the aforementioned tilted view image). It should be noted that for each pixel pair, the distance between the second depth difference and the first depth difference between the two pixels in that pair can be a generalized metric, including the closeness of the depth difference values ​​and the consistency of the depth order sign. For example, when the first depth difference is positive (indicating that point q is farther than point p in the actual scene) and the second depth difference is also positive (indicating that point q is also farther than point p in the rendered result), their directions are consistent; if their values ​​are also close, the distance is smaller. The optimization goal is to minimize this distance. Specifically, an iterative optimization method can be used to adjust the current predicted height. In each iteration, the second depth difference and the first depth difference at the current predicted height are compared, the degree of difference between them (i.e., the loss value) is calculated, and then the current predicted height is updated based on this loss value, so that the updated height produces a second depth difference that is closer to the first depth difference in the new rendering. Repeat this process until the loss value converges or the preset number of iterations is reached. The height value at this point is the target measured height.

[0061] Since the first depth difference reflects the relative front-to-back relationship between pixels in the real scene, this relationship is robust to the scale uncertainty of monocular depth estimation; even if the absolute depth value is inaccurate, the relative order is usually reliable. By using the depth order information contained in the actual image as a "reference frame," geometric calibration of the building height is performed, ensuring that the relative depth relationship of the rendered depth map is consistent with the first depth difference. This allows the building height to converge to a value consistent with image observations without requiring the absolute depth ground truth, avoiding the dilemma of needing precise depth ground truth or a large amount of labeled data.

[0062] For example, the optimization process may include the following steps: First, randomly sample multiple sets of pixel pairs within the visible building area of ​​the tilted view image. Let the sampled set of pixel pairs be P. To avoid using pixel pairs with unclear depth relationships (i.e., two pixels with too similar depths), remove pixel pairs whose absolute value of the first depth difference is less than a preset depth threshold difference τ_d, resulting in a target set of pixel pairs P*. Second, for each pixel pair (p,q) in the target set, calculate its second depth difference Δ_r(p,q) = R'(q) - R'(p), where R' is the rendered depth value after being normalized in the same way as the first depth difference. Then, construct a depth difference consistency loss function, which constrains the second depth difference to be consistent with the first depth difference in sign, and the absolute value of the second depth difference is not less than a preset interval parameter m (e.g., a typical value of m could be 0.1). Specifically, the mathematical meaning of this loss function is: when the second depth difference has the same sign as the first depth difference and its absolute value is greater than or equal to m, the loss is 0; otherwise, the loss is positive, and its magnitude is proportional to the degree of violation.

[0063] The loss values ​​of all valid pixel pairs are summed or averaged to obtain the loss value for the current view. For multiple tilted view images involved in the optimization, their loss values ​​are aggregated to obtain the overall optimization objective function L. A gradient descent-type optimizer is used to iteratively update the building height parameters. In each iteration, the gradient of the overall loss function L with respect to the current predicted height is calculated. This gradient is backpropagated to the height parameters through a differentiable renderer, and then the height values ​​are adjusted according to the optimizer's update rules. This process is repeated until the loss function converges or the preset maximum number of iterations is reached. The converged height value is the target measured height of the building.

[0064] Step 212: Based on the bottom outline and the target measured height, construct a three-dimensional building model corresponding to the target building.

[0065] In this context, a 3D building model refers to the geometric representation of the target building in 3D space. This embodiment employs a parametric representation: using the bottom contour as the base and the target measured height obtained from the aforementioned optimization steps as the height, the base is stretched vertically to this height, forming a prism-shaped or stretched 3D volumetric model. This 3D volumetric model retains the building's main spatial occupancy information, namely its position and shape on the horizontal plane (described by the bottom contour) and its height in the vertical direction (described by the target measured height), while discarding fine geometric details on the building facade (such as concavity / convexity, decoration, etc.). For most regular buildings, this proxy model using a single stretched volume representation can meet application requirements. However, for buildings with complex roof structures or multi-level heights, the building can be further decomposed into multiple sub-parts, each of which independently undergoes bottom contour extraction and height optimization, and then combined to form a multi-level stretched composite proxy model. The final output 3D building proxy model can be stored or exported in various formats, including but not limited to: 3D mesh file formats (such as OBJ, PLY), GIS vector data formats (such as Shapefile with height attributes), or parametric description formats (such as JSON describing the contour vertex sequence and height values). This model can be directly used in applications such as digital twin cities, urban planning simulation, UAV path planning, and augmented reality.

[0066] This embodiment integrates the separately acquired two-dimensional planar information (bottom outline) and vertical information (target measurement height) to form a complete three-dimensional geometric representation. This "outline + height" modeling method directly addresses the needs of applications such as digital twin cities and urban planning simulation for building volume information, avoiding the redundant process of first reconstructing dense point clouds or meshes and then extracting buildings. Compared with methods that directly output complex three-dimensional mesh models, the parametric volume model output by this embodiment has advantages such as concise expression, small data volume, and ease of editing and storage. Compared with methods that only output two-dimensional building outlines, this embodiment adds a height dimension, enabling the model to have complete three-dimensional spatial expression capabilities. This model can be directly used for subsequent applications such as rapid construction of city-level three-dimensional scenes, building volume statistical analysis, and UAV flight path planning.

[0067] In some embodiments, acquiring multi-view images corresponding to the target building includes:

[0068] Based on real-time dynamic positioning technology, the UAV performs oblique photography to acquire images of the target urban scene, obtaining multi-view images covering the target buildings and the camera pose parameters; the camera pose parameters include pose information acquired by the UAV during the acquisition process.

[0069] The multi-view images are divided into near-vertical view images and oblique view images according to the image shooting angle; wherein, the near-vertical view images are used to represent the building roof and planar distribution information of the target building, and the oblique view images are used to represent the building facade and lateral structure information of the target building.

[0070] Real-Time Kinematic (RTK) is a high-precision positioning technology for Global Navigation Satellite Systems (GNSS). RTK establishes a data link between a base station with known coordinates and a rover (in this embodiment, a UAV). The base station transmits its observed carrier phase and coordinate information to the rover in real time. The rover performs differential calculations based on its own observations and those of the base station, thereby eliminating common errors such as satellite clock errors, orbital errors, ionospheric delays, and tropospheric delays, achieving real-time positioning accuracy at the centimeter or even millimeter level. In this embodiment, the UAV is equipped with an RTK positioning module, enabling it to acquire its own high-precision three-dimensional spatial coordinates in real time during flight and synchronously record these coordinates along with the exposure time of each frame of image.

[0071] Oblique photogrammetry refers to the use of a drone equipped with one or more cameras to acquire multi-angle, high-overlapping images of a target urban scene along a pre-defined flight path (such as a zigzag, five-way, or circular path). During the acquisition process, the drone's flight altitude, lateral overlap, and side overlap can be configured according to the scene scale and accuracy requirements. The introduction of RTK technology allows each acquired image to directly obtain its precisely corresponding camera pose parameters. These parameters can include the camera's three-dimensional position coordinates (X, Y, Z) and three-dimensional attitude angles (pitch, roll, yaw) in the world coordinate system, eliminating the need for subsequent Structure-from-Motion (SfM) processing to calculate the pose.

[0072] This embodiment uses RTK UAV for oblique photogrammetry acquisition. On the one hand, the centimeter-level positioning accuracy provided by RTK technology ensures the high reliability of camera pose parameters, providing an accurate geometric reference for subsequent rendering and optimization. On the other hand, by directly using the pose information obtained during the acquisition process as input, the computational overhead and accumulated errors caused by relying on SfM to solve pose in traditional methods are avoided, thereby simplifying the data preprocessing process and improving the overall modeling efficiency.

[0073] After obtaining multi-view images and their corresponding camera pose parameters, the images are further segmented based on the shooting angle. The shooting angle can be characterized by the angle between the imaging optical axis and the vertical direction (i.e., the direction of gravity). For each image, the angle between its imaging optical axis and the vertical direction is calculated based on the attitude angle information in its camera pose parameters. When this angle is less than a preset angle threshold (e.g., 30°), the image is classified as a near-vertical view image; when the angle is greater than or equal to the preset angle threshold, the image is classified as a tilted view image.

[0074] Near-vertical view images refer to images where the angle between the imaging optical axis and the vertical direction is less than a preset angle threshold (e.g., 30°). In these images, the imaging optical axis is nearly vertically downward, the roof plane of the building is approximately an orthographic projection, geometric distortion is small, and the top boundary of the building has a high geometric correspondence with its actual projected boundary on the ground. Therefore, near-vertical view images mainly reflect the roof and planar distribution information of a building and are suitable for extracting the two-dimensional bottom contour of a building. Correspondingly, oblique view images refer to images where the angle between the imaging optical axis and the vertical direction is greater than or equal to the preset angle threshold. In these images, the imaging optical axis deviates significantly from the vertical direction, the facade of the building is prominently displayed, and geometric information in the lateral contour and height direction of the building can be provided, making them suitable for extracting depth cues related to the building's height.

[0075] In this embodiment, two types of images are used separately based on their respective imaging characteristics. Near-vertical view images are used to solve the problem of planar contour extraction, while oblique view images are used to solve the problem of height estimation. This division of labor avoids the problem of insufficient information inherent in single-type images and also avoids mutual interference caused by differences in imaging characteristics when processing images from two different perspectives together. Furthermore, unlike methods that uniformly input all images into the same processing flow, this embodiment, through clear viewpoint division and specialized use, effectively leverages the information advantages of images from different perspectives.

[0076] In some embodiments, determining the bottom outline of the target building based on the near-vertical view image includes:

[0077] The building region mask is extracted from the near-vertical view image using a preset building segmentation model;

[0078] The building area mask is subjected to contour extraction and vectorization to obtain a closed vector polygon;

[0079] The vector polygon is regularized to obtain the bottom outline of the target building.

[0080] The pre-set building segmentation model refers to a pre-trained deep learning model used to identify and separate building regions from remote sensing or aerial images. This model can employ a semantic segmentation architecture based on Convolutional Neural Networks (CNNs) (such as U-Net and DeepLab series) or an instance segmentation architecture (such as MaskR-CNN). Unlike general image segmentation models, the building segmentation model in this embodiment is specifically fine-tuned for building targets in aerial images: using a large number of high-resolution remote sensing images labeled with building boundaries as training samples, the model learns the typical visual features of buildings in aerial images through supervised learning, including roof texture patterns, edge gradient features, shadow features, and contrast features with the surrounding environment.

[0081] A building region mask is a binary image with the same spatial resolution as the input image, where the value of each pixel indicates whether the pixel belongs to a building region. Typically, pixels with a value of 1 (or 255) represent building regions, while pixels with a value of 0 represent non-building regions (such as ground, roads, vegetation, water bodies, etc.). Essentially, a mask preserves the pixel classification results in the image as a single image, providing pixel-level region indications for subsequent contour extraction.

[0082] This embodiment utilizes deep learning technology to automatically and efficiently identify building regions from near-vertical view images, avoiding the missed detections and false detections that easily occur in complex urban scenes with traditional segmentation methods based on manual annotation or handcrafted features (such as edges, colors, and textures). The finely tuned segmentation model can adapt to building extraction tasks under different urban styles (such as dense urban areas, industrial areas, and residential areas), different building types (such as flat-roofed buildings, pitched-roofed buildings, and high-rise towers), and different lighting conditions, demonstrating good generalization ability. Unlike directly extracting building regions from tilted view images, this embodiment specifically utilizes near-vertical view images for segmentation, resulting in smaller geometric deformation and higher consistency between the segmentation boundaries and the actual ground boundaries of the buildings.

[0083] After obtaining the building area mask, contour extraction and vectorization are performed on the mask. Contour extraction refers to extracting the boundary pixels between the building area and the background area from the binary mask image. Specifically, boundary tracking algorithms (such as the Suzuki algorithm and the Moore-Neighbor algorithm) can be used to traverse the mask image and extract a pixel-level contour chain representing the boundary of the building area. This contour chain consists of a series of continuous pixel coordinate points, and its form is a raster representation, that is, each boundary point is represented by its row and column coordinates (or pixel coordinates) in the image.

[0084] Vectorization refers to converting the pixel-level raster outline into a vector polygon representation consisting of vertices and line segments connecting them. Specifically, the Douglas-Peucker algorithm or a similar polygon fitting algorithm can be used to simplify the pixel sequence on the raster outline: this algorithm recursively removes intermediate points whose distance from the endpoints is less than a preset threshold, retaining feature points that contribute significantly to the outline shape, thus compressing a large number of dense pixels into a smaller number of vector vertices. The resulting vector polygon is defined by several ordered vertex coordinates (in the image coordinate system), with adjacent vertices connected by line segments.

[0085] This embodiment transforms the raster mask output from the segmentation model into geometrically meaningful vector polygons through contour extraction and vectorization, giving it a clear mathematical expression and geometric operability. Raster masks retain complete information about the building area, but they are large in data volume, have low storage efficiency, and are difficult to use directly for 3D modeling (e.g., stretching operations require explicit boundary segments and vertices). In contrast, vector polygons can accurately describe the building contour with a small number of vertices, significantly compressing the data volume and facilitating subsequent geometric transformations (such as projection transformations and coordinate transformations) and 3D stretching operations. Unlike directly using a raster mask as a contour representation, this embodiment obtains a more compact and operable geometric expression through vectorization. After obtaining the vector polygons, they are further regularized. Regularization refers to adjusting and optimizing the boundaries of the vectorized polygons to better conform to the geometric characteristics of a building as a man-made structure. Specifically, regularization processing can include at least one of the following operations: vertex simplification, which removes redundant vertices that contribute little to the overall shape based on the side length of the polygon and the included angle between adjacent sides, further reducing the complexity of the polygon; orthogonalization adjustment, which adjusts the included angle between adjacent sides to close to 90° (perpendicular) or 180° (parallel), making the polygon boundary more consistent with the geometric features of common building plans such as rectangles or L-shapes; edge alignment, which adjusts the edges of the polygon to align with the main direction of the scene (such as the direction of the urban road network), improving the regularity of the model in the global coordinate system; and smoothing filtering, which performs low-pass filtering on the coordinates of the polygon vertex to eliminate the tiny jagged edges and burrs caused by segmentation noise.

[0086] The result of regularization is a regular, concise bottom outline polygon that conforms to the characteristics of an artificial architectural structure. This polygon can usually be accurately described by a small number of vertices (e.g., 4 vertices for a rectangle, 6 vertices for an L-shape), with the included angle between adjacent sides close to 90° or an integer multiple thereof, and the direction of each side line consistent with the main direction of the scene.

[0087] Considering that the original segmentation results and the polygons obtained through vectorization may have irregularities such as jagged edges, local protrusions, or depressions due to image noise, segmentation errors, or perspective distortion, directly using these irregular polygons for 3D modeling may result in a visually rough and irregular architectural model that is difficult to maintain consistent orientation with adjacent buildings. Therefore, in this embodiment, regularization processing is used to improve the geometric regularity and visual quality of the architectural model. Furthermore, vertex simplification and edge alignment further reduce data storage, facilitating subsequent scene fusion and application. Unlike directly using the unprocessed original segmented contours, this embodiment obtains a bottom contour that better conforms to the artificial structural characteristics of the building through regularization processing.

[0088] The bottom profile can be represented as a set of two-dimensional polygons on a ground reference plane (such as the Z=0 plane in the world coordinate system). Each polygon corresponds to an independent building instance, containing the building's spatial location (determined by the coordinates of the polygon's vertices) and boundary shape information. This bottom profile serves as the "base" constraint for subsequent 3D model construction, and together with the target measured height obtained through height optimization, determines the building's 3D volume.

[0089] In some embodiments, determining the first depth difference between pixels within the tilted view image includes:

[0090] For each of the aforementioned tilted view images, the visible building information corresponding to the current tilted view image is calculated based on the camera pose parameters and the current predicted height; the visible building information includes the number of visible buildings and the projected area of ​​the visible buildings;

[0091] The tilted view images are filtered based on the visible building conditions to obtain a first tilted view image set;

[0092] Based on the spatial distribution uniformity of each image in the first tilted view image set, the first tilted view image set is filtered to obtain the second tilted view image set.

[0093] The second tilted-view image set is supplemented with images until the target building is observed by a preset number of images, and the tilted-view images are replaced according to the supplemented second tilted-view image set.

[0094] The visible building situation refers to the statistical information of building instances that can be observed within the field of view of the current tilted view image, based on the current building proxy model (with the current predicted height) and known camera pose parameters. Specifically, the visible building situation includes the following indicators: the number of visible buildings and the projected area of ​​visible buildings.

[0095] The number of visible buildings refers to the number of building instances in the current building proxy model within the current tilted view image where at least one pixel is projected onto the image plane, and that pixel is not occluded by other buildings or parts of the building itself. Calculating the number of visible buildings involves projecting the 3D models of each building onto the image plane using camera pose parameters and performing depth testing to determine if each projected pixel is truly visible, i.e., whether there are other objects occluding the building surface point corresponding to that pixel from the camera. The visible building projection area refers to the number of pixels corresponding to all visible buildings in the current tilted view image. Specifically, the visible pixels of each building are summed to obtain the total number of visible pixels. This value characterizes the degree to which buildings cover the image in the current view. A larger visible building projection area indicates richer architectural geometry information in the view; conversely, a smaller area suggests that the view may primarily consist of sky, ground, or non-building areas, contributing less to the reconstruction of urban buildings based on aerial imagery.

[0096] Considering that in actual oblique photogrammetry acquisition, factors such as UAV flight path design, changes in building height, and scene occlusion can affect the accuracy of data, some oblique view images may contain limited effective building information. For example, some images may primarily capture the sky and distant mountains, with only a few buildings in the edge areas; or in some images, due to the shooting angle, most buildings are severely obscured by foreground buildings. Including all these low-information images in the subsequent optimization process would not only increase unnecessary computational overhead but may also introduce noise interference, affecting the stability and accuracy of height optimization. Therefore, this embodiment quantifies and filters out views with higher information content by calculating the number of visible buildings and their projected area.

[0097] After obtaining the visible building information for each tilted-view image, the images are filtered based on the statistical distribution of these indicators to obtain the first tilted-view image set. Specifically, the statistical distribution of the number of visible buildings (e.g., median, quartiles) and the statistical distribution of the projected area of ​​visible buildings (e.g., median, quartiles) can be calculated separately, and then low-quality views are eliminated according to preset filtering rules. For example, the filtering thresholds can be set as follows: images with fewer than 50% of the median number of visible buildings in all images are eliminated; images with a projected area of ​​visible buildings less than 50% of the median projected area of ​​visible buildings in all images are eliminated. Images that meet both of the above conditions (or at least one condition, depending on the specific configuration) are retained, forming the first tilted-view image set. The first tilted-view image set is a collection of effective views after preliminary quality filtering, eliminating low-quality images with significantly insufficient information.

[0098] Considering that in typical oblique photogrammetry datasets, the number of original images can reach hundreds or even thousands, a considerable portion of these images have limited observational value for specific buildings, this embodiment, through the aforementioned filtering, can quickly filter out a large number of redundant views with low computational cost (requiring only projection and visibility calculations, without depth estimation or depth map generation). This can significantly reduce the number of images involved in subsequent processing without significant information loss, thereby reducing overall computational overhead.

[0099] After obtaining the first set of tilted-view images, a second set of tilted-view images is obtained by further filtering based on the spatial distribution uniformity of each image. Spatial distribution uniformity refers to whether the tilted-view images are evenly distributed in space, that is, whether the shooting positions (camera optical center positions) of each image are too concentrated or too dispersed in three-dimensional space. Spatial distribution uniformity can be quantified and controlled using the Farthest Point Sampling (FPS) method. FPS is an iterative selection strategy: first, an image is randomly selected from the first set of tilted-view images as the starting point; then, in each iteration, the image with the largest spatial distance (i.e., the Euclidean distance between camera optical centers) to all images in the selected set is added to the set; this process is repeated until a preset number of samples is reached or the spatial distribution uniformity threshold is met. By using FPS, a subset of images that are spatially dispersed as much as possible can be selected from the first set of tilted-view images, avoiding the viewpoint redundancy problem caused by a large number of images being shot at similar positions. This embodiment, through the aforementioned spatial distribution screening, ensures that the views participating in height optimization are diverse in perspective. Considering that if only images with high information content are retained, and these images are captured from highly concentrated locations (e.g., all retained images come from the same direction), the geometric constraints provided by these images are essentially correlated, making it difficult to impose comprehensive constraints on the building's height. Therefore, this embodiment selects spatially uniformly distributed views through farthest point sampling, ensuring that the building is observed from multiple different directions, thereby enhancing the constraint strength and convergence stability of the height optimization problem.

[0100] After obtaining the second tilted-view image set, further image supplementation is performed until each target building is observed in a preset number of images. The preset number is a configurable hyperparameter, typically ranging from 3 to 5, indicating that each building must be observed in at least this many images from different perspectives to ensure sufficient constraints for height optimization. The specific method for image supplementation is as follows: for buildings with insufficient observations in the current second tilted-view image set, images with high information content that can be observed from the remaining images that were never selected for the second tilted-view image set are prioritized and added to the set. This supplementation process is iterated until all buildings have reached the preset observation count requirement, or no more images are available for supplementation. After supplementation, the final filtered tilted-view image set is obtained, and the original tilted-view images are replaced based on this image set. That is, in subsequent height optimization steps, only the filtered image set is used in the calculation, and the original full set of tilted-view images is no longer used.

[0101] In real-world scenarios, due to uneven spatial distribution of buildings (e.g., buildings in edge areas) or occlusion, some buildings may only appear in a few images. Without supplementation, the height parameters of these buildings may lack sufficient gradient information during optimization, leading to slow convergence or convergence to incorrect values. This embodiment supplements images to ensure each building has sufficient viewpoint constraints, avoiding insufficient data support for height optimization due to some buildings appearing too infrequently in the view. Furthermore, by forcing each building to be observed in a predetermined number of images, the well-stressed nature of the height optimization problem is ensured.

[0102] In some embodiments, determining the first depth difference between pixels within the tilted view image includes:

[0103] Depth estimation is performed on each tilted view image using a monocular depth estimation model to obtain the prior depth map corresponding to each tilted view image.

[0104] For each tilted view image, differentiable rendering is performed based on the camera pose parameters and the current predicted height to obtain the rendering depth map corresponding to the tilted view image and the visible building area in the rendering depth map.

[0105] The prior depth map and the rendered depth map are aligned within the visible building area. The pixel depth values ​​of the aligned prior depth map and the rendered depth map within the visible building area are then normalized to obtain the processed prior depth map and the processed rendered depth map. The normalization process includes normalizing the depth value of each pixel based on the median of all pixel depth values ​​within the visible building area and the depth deviation between each pixel and the median.

[0106] The first depth difference is determined based on the processed prior depth map, and the second depth difference is determined based on the processed rendered depth map.

[0107] Monocular depth estimation models are pre-trained deep learning models capable of inferring the scene depth value corresponding to each pixel from a single 2D image. Unlike binocular stereo matching or multi-view stereo vision, monocular depth estimation requires only a single image as input, offering advantages such as high computational efficiency and independence from feature matching across multiple views. The monocular depth estimation model used in this embodiment can be a pre-trained deep learning model, such as DPT (DensePrediction Transformer), MiDaS, or AdaBins architectures. These models are pre-trained on large-scale depth datasets (such as MegaDepth and DIW) and can learn rich depth prior knowledge from natural images, including perspective relationships, texture gradients, shadow cues, and relative object scales. After inputting a tilted view image into the model, the model outputs a prior depth map with the same spatial resolution as the input image.

[0108] A priori depth map is a two-dimensional matrix with the same width and height as the tilted view image. Each element (pixel) in the matrix represents the depth value from the corresponding 3D point in the scene to the camera's optical center. It's important to note that the depth values ​​output by a monocular depth estimation model are typically normalized values ​​at a relative scale, rather than absolute depth values ​​at the true physical scale. In other words, the priori depth map accurately reflects the relative depth relationships between different objects in the scene; for example, the depth value of a foreground object is less than that of a background object, and there is no fixed linear mapping between the specific numerical range (such as minimum and maximum values) and the actual physical distance. This scale uncertainty is an inherent problem in monocular depth estimation, and this embodiment addresses this specifically through subsequent normalization processing.

[0109] Monocular depth estimation models can predict the depth of the entire image in a single forward propagation, with computational efficiency far exceeding traditional multi-view geometric methods. Furthermore, the depth priors provided by this model contain rich scene depth order information, which can serve as a valuable reference for subsequent optimization. Unlike stereo vision methods that require feature matching and triangulation from multiple images, this embodiment leverages the advantages of monocular depth estimation in terms of efficiency and information density.

[0110] While obtaining the prior depth map, for each tilted view image, differentiable rendering is performed based on the camera pose parameters and the current predicted height to obtain the corresponding rendered depth map and the visible building areas within that rendered depth map. Differentiable rendering refers to a rendering technique where all computational operations during the rendering process are differentiable, meaning the gradient of the rendered result (such as the rendered depth map) relative to scene parameters (such as building height) can be automatically calculated and backpropagated. Unlike traditional graphics rendering pipelines, differentiable renderers record the derivative information of the depth value of each pixel with respect to the geometric parameters in the scene while generating the image.

[0111] The specific process of generating the rendering depth map is as follows: First, the target building is represented as a parametric 3D volumetric model. Using the bottom contour as the base, it is stretched vertically to the height determined by the current predicted height, forming a prism or extruded volume. Then, using a differentiable renderer, based on the camera's intrinsic and extrinsic parameters corresponding to the tilted view image, the parametric volumetric model is projected onto the image plane. During rendering, for each pixel, the differentiable renderer calculates the first intersection point between the line of sight from the camera's optical center passing through that pixel and the surface of the building model. The distance from this intersection point to the camera's optical center is the rendering depth value of that pixel. For pixels where the line of sight does not hit any building model (such as the sky, ground, or occluded areas), their depth value is set to a preset background depth value (e.g., a constant much larger than the scene depth or infinity). The final generated rendering depth map is a two-dimensional matrix with the same resolution as the input image, where the value of each pixel represents the depth value of the point on the building model surface corresponding to that pixel's location under the current height assumption.

[0112] The visible building region refers to the area formed by pixels whose pixel values ​​in the rendered depth map are not background depth values. The viewpoints corresponding to these pixels have successfully hit the surface of the building model, indicating that the surface of the building model is visible from that viewpoint. The visible building region can be represented by a binary mask: pixels with a value of 1 (or non-zero) in the mask belong to the visible building region, and pixels with a value of 0 belong to the background region. In actual calculations, the visible building region can be extracted by comparing the depth map generated during the rendering process with a background depth threshold.

[0113] Understandably, the rendered depth map reflects the depth structure the model expects the scene to present under the current height assumption. Extracting visible building areas limits the effective range of subsequent depth comparisons, avoiding the inclusion of depth information from background areas (sky, ground, etc.) as constraints. The prediction results for these areas in monocular depth estimation are typically unreliable and irrelevant to building height optimization. Therefore, unlike traditional rendering methods used solely for visualization, the differentiable rendering technology employed in this embodiment creates a calculable derivative relationship between the rendering results and model parameters, thereby supporting gradient-based optimization.

[0114] After obtaining the prior depth map and the rendered depth map, they are aligned within the visible building area. Alignment means matching the prior depth map and the rendered depth map in spatial location, ensuring that pixels with the same coordinates in both point to the same scene space location. Since both the prior depth map and the rendered depth map are generated from the same tilted viewpoint image, they naturally have the same image coordinate system and resolution. Therefore, the alignment operation mainly confirms that the visible building area ranges of both are consistent and excludes unreliable pixels (such as the background area in the rendered depth map, which may correspond to unreliable depth predictions in the prior depth map). The result of alignment is to determine a common and valid set of pixels, that is, at the pixel location, the prior depth map has a valid depth prediction value, and the rendered depth map belongs to the visible building area.

[0115] After alignment, the pixel depth values ​​within the visible building area of ​​both the prior depth map and the rendered depth map are normalized. The purpose of normalization is to eliminate scale differences between the two depth maps. Specifically, the prior depth map comes from monocular depth estimation, and its depth values ​​are at an uncertain relative scale; the rendered depth map comes from geometric rendering, and its depth values ​​have a true physical scale (the unit is consistent with the distance unit in the camera pose parameters, usually meters). The two may differ by orders of magnitude in their numerical range, making direct numerical comparison impossible. Normalization transforms both to the same reference scale, making them comparable.

[0116] The normalization process used in this embodiment is as follows: the depth value of each pixel is normalized based on the median and median absolute deviation (MAD) of the depth values ​​of all pixels within the visible building area. The median is the value at the middle position after sorting the depth values ​​of all pixels within the visible building area by size; it represents the central trend of depth values ​​in that area. Compared to the mean, the median is less sensitive to outliers (such as abnormal values ​​in depth estimation) and has better robustness. The median absolute deviation is the median of the absolute value of the difference between the depth value of each pixel and the median, i.e., MAD = median(|D_i - median(D)|). MAD reflects the degree of dispersion of depth values ​​around the median and can be considered a robust scale estimator.

[0117] The normalization formula is: D' = (D - median(D)) / (MAD(D) + ε), where ε is a very small positive number (e.g., 1e-8) to prevent the denominator from being zero. After this normalization process, the distribution of the normalized depth value D' has the following properties: its center is shifted to near 0, and its scale is scaled to a median absolute deviation of 1. After performing the above normalization process on the prior depth map and the rendered depth map respectively, the depth value distributions of both are adjusted to the same statistical scale, the median of both becomes 0, and the median absolute deviation of both becomes 1. This makes numerical comparisons between the two meaningful: for example, in the prior depth map, if the normalized depth value of pixel p is -0.3 and the normalized depth value of pixel q is 0.5, it indicates that p is closer to the camera than q in the prior depth map (because a smaller depth value indicates closer); in the rendered depth map, if the normalized depth values ​​of p and q at the same position also show a similar relationship, it means that the depth order under the current height assumption is consistent with the prior depth order.

[0118] This embodiment uses the median and median absolute deviation for normalization, which can robustly eliminate the scale uncertainty of monocular depth estimation and the scale differences between different depth maps. Compared with normalization using the mean and standard deviation, the median and median absolute deviation are insensitive to outliers and local anomalies in depth estimation, and can more stably reflect the overall characteristics of the depth distribution. Through normalization, the prior depth map (from monocular estimation) and the rendered depth map (from geometric rendering) are mapped to the same statistical space, and their values ​​can be directly compared, laying the foundation for subsequent depth difference consistency constraints. Unlike directly comparing absolute depth values, this embodiment transforms the comparison of absolute depths into a comparison of relative depth distributions through normalization, thereby avoiding the influence of scale uncertainty in monocular depth estimation.

[0119] After normalization, a first depth difference is determined based on the processed prior depth map, and a second depth difference is determined based on the processed rendered depth map. The first depth difference refers to the difference between the normalized depth values ​​of any two different pixels in the processed prior depth map. Let the normalized depth values ​​of pixels p and q in the processed prior depth map be D'_p and D'_q, respectively. Then, the first depth difference Δ_d(p,q) = D'_q - D'_p. When Δ_d(p,q) is positive, it indicates that the scene point corresponding to pixel q is farther from the camera than the scene point corresponding to pixel p in the processed prior depth map; when Δ_d(p,q) is negative, it indicates they are closer; and when Δ_d(p,q) is zero, it indicates the same depth. The definition and calculation method of the second depth difference are exactly the same as those of the first depth difference. The only difference is that the second depth difference is calculated based on the processed rendered depth map: Let the normalized depth values ​​of pixels p and q in the processed rendered depth map be R'_p and R'_q, respectively. Then the second depth difference Δ_r(p,q) = R'_q - R'_p.

[0120] Understandably, the first depth difference reflects the scene depth order information extracted from the actual image, while the second depth difference reflects the depth order information presented by the building proxy model under the current height assumption. When the two converge in sign and value, it indicates that the current height assumption matches the image observation. By updating the height parameter in subsequent steps with the optimization objective of reducing the distance between the two, accurate estimation of the building height can be achieved.

[0121] To eliminate the problem of inconsistent depth scales between different images, this embodiment normalizes the prior depth map. Specifically, firstly, based on the current building proxy model and camera pose parameters, a rendered depth map corresponding to the tilted view image is generated using a differentiable renderer, and the visible building region (i.e., the pixel region of the building proxy model that is not occluded under this view) is extracted from it. Then, the prior depth map and the rendered depth map are spatially aligned within the visible building region. After alignment, the pixel depth values ​​of the prior depth map within the visible building region are normalized by median: the median and median absolute deviation (MAD) of all pixel depth values ​​in this region are calculated, and then the depth value of each pixel is normalized by subtracting the median and dividing by the median absolute deviation. This normalization process eliminates the scale uncertainty of monocular depth estimation and the scale differences between different images, making depth values ​​from different sources comparable. Finally, based on the normalized prior depth map, the depth difference between any two pixels within the visible building region is calculated and denoted as the first depth difference. The sign of the first depth difference indicates the relative front-to-back relationship between the two pixels (positive sign means point q is farther than point p, negative sign means point q is closer than point p), and its absolute value indicates the magnitude of the relative depth difference.

[0122] In some embodiments, optimizing the current predicted height to obtain the target measured height of the target building by reducing the distance between the second depth difference and the first depth difference between each pixel in the rendered depth map as the optimization objective includes:

[0123] Multiple sets of pixel pairs are randomly sampled within the target area of ​​the tilted view image, and pixel pairs whose absolute value of the first depth difference is less than a preset depth threshold difference are removed to obtain a target pixel pair set; the target area includes visible building areas;

[0124] For each pixel pair in the target pixel pair set, the depth order matching degree of the current pixel pair in the prior depth map and the rendered depth map is determined based on the difference between the second depth difference of the current pixel pair and the preset interval threshold, the consistency of the numerical signs of the first depth difference and the second depth difference, and the correlation between the change in the distance between the first depth difference and the second depth difference and the change in the current predicted height.

[0125] With the goal of minimizing the sum of the depth order matching degrees corresponding to the target pixel pair set, the current predicted height is iteratively optimized to obtain the target measured height;

[0126] The step of constructing a three-dimensional building model corresponding to the target building based on the bottom contour and the target measured height includes:

[0127] The bottom contour is stretched vertically according to the target measured height to obtain a parametric three-dimensional volume model, which serves as the three-dimensional building model corresponding to the target building.

[0128] The target region refers to the effective pixel range in the tilted view image used for depth comparison. In this embodiment, the target region includes the visible building area, i.e., the pixel area visible to the building proxy model from this viewpoint, as determined in the aforementioned differentiable rendering process. Limiting the depth comparison to the visible building area avoids incorporating the depth information of background areas (such as the sky, ground, and non-building structures) into the optimization, as depth predictions for these areas are typically unreliable and irrelevant to building height optimization. Random sampling refers to randomly selecting a certain number of pixel pairs within the target region. Sampling can employ a uniform random distribution to ensure that pixels at each location within the target region have an equal probability of being selected. The number of sampled pixel pairs can be configured according to actual computing resources and accuracy requirements; typical values ​​are 2048 pairs or 4096 pairs. On the one hand, performing a full pixel pair depth comparison within the target region is computationally intensive; random sampling can significantly reduce computational complexity. On the other hand, random sampling preserves the statistical characteristics of depth comparison; as long as the number of samples is sufficient, the resulting set of sampled pixel pairs can well represent the depth distribution characteristics of the entire target region.

[0129] After sampling to obtain the set of pixel pairs, pixel pairs whose absolute value of the first depth difference is less than a preset depth threshold difference are further removed. The preset depth threshold difference τ_d is a preset positive number, which is usually set to a small value in the normalized depth space, such as 0.05 or 0.1. The absolute value of the first depth difference represents the magnitude of the depth difference between two pixels in the prior depth map; when this absolute value is less than τ_d, it indicates that the depths of the two pixels are very close, and their relative front-to-back relationship in the prior depth map is not clear (the order may be reversed due to depth estimation noise). Removing such pixel pairs with unclear depth relationships can avoid using fuzzy constraints in optimization, thereby improving the stability and convergence speed of optimization. By filtering out pixel pairs with unreliable depth relationships in the prior depth map, only pixel pairs with significant depth differences and clear order relationships are retained as effective optimization constraints. After the above sampling and removal operations, the retained pixel pairs constitute the target pixel pair set. Each pixel pair in this set has a clear depth order relationship (the absolute value of the first depth difference is not less than τ_d) and is located within the target region.

[0130] For each pixel pair in the target pixel pair set, it is necessary to determine the depth order matching degree of the pixel pair in the prior depth map and the rendered depth map at the current predicted height. Depth order matching degree is a quantitative metric used to measure the consistency of the depth order of the rendered depth map (second depth difference) and the prior depth map (first depth difference) for that pixel pair under the current height assumption. The matching degree can be determined by considering the following factors: First, the difference between the second depth difference of the current pixel pair and a preset interval threshold. The preset interval threshold m is a preset positive number, typically 0.1 (in normalized depth space). This threshold requires that the absolute value of the second depth difference is not less than m, meaning the depth difference between the two pixels in the rendered depth map needs to reach a certain level of significance. If the absolute value of the second depth difference is less than m, even if the sign is correct, it indicates that the depth difference between the two pixels is not significant enough under the current height assumption, failing to form a sufficiently distinguishable depth hierarchy, resulting in a low matching degree. The preset interval threshold m encourages the model to not only be correct in order but also to reach a certain standard in the magnitude of the difference, thereby avoiding degradation caused by excessively small depth differences. Secondly, the numerical signs of the first and second depth differences are consistent. The sign of the first depth difference indicates the relative order of two pixels in the prior depth map (positive sign means point q is farther than point p, negative sign means point q is closer than point p). The sign of the second depth difference indicates the relative order of two pixels in the currently rendered depth map. When the two signs are the same, it means that the depth order in the rendered depth map is consistent with the depth order in the prior depth map, and this factor contributes positively to the matching accuracy. When the two signs are opposite, it means that the order is inconsistent, and this factor contributes negatively to the matching accuracy. This ensures that the depth order rendered by the architectural proxy model is consistent with the depth order of the real scene.

[0131] Furthermore, there is the correlation between the change in the distance between the first and second depth differences and the change in the current predicted height. This factor involves the gradient propagation characteristics of differentiable rendering: by analyzing the gradient of the loss function relative to the current predicted height, it can be determined how adjusting the height parameter affects the distance between the second and first depth differences. Specifically, when the gradient direction indicates that increasing the height reduces the distance, the optimizer will update the parameters in the direction of increasing the height; conversely, it will update the parameters in the direction of decreasing the height. By utilizing the gradient information of the differentiable renderer, the height parameter is guided to iteratively update in the direction that maximizes the depth order matching.

[0132] Considering the above factors, a quantization function for depth order matching can be constructed. In a typical implementation, this function can be expressed as follows: for a set of pixel pairs (p, q), when the second depth difference has the same sign as the first depth difference and the absolute value of the second depth difference is greater than or equal to m, the matching degree is 0 (indicating a perfect match, with a loss of 0); when the signs are different or the absolute value of the second depth difference is less than m, the matching degree is positive, and its magnitude is proportional to the degree of sign inconsistency or the difference in the interval threshold. In this way, the matching degree is mapped to a non-negative loss value; the higher the matching degree (i.e., the better the depth order consistency), the smaller the corresponding loss value.

[0133] After obtaining the depth order matching degree for each pixel pair, the matching degrees of all pixel pairs in the target pixel pair set are accumulated or averaged to obtain the sum (or average matching degree) of the matching degree corresponding to the current tilted view image. For multiple tilted view images involved in optimization, the matching degrees of all images are further aggregated (e.g., summed) to obtain the overall optimization objective. The current predicted height is iteratively optimized with the goal of minimizing this overall optimization objective (i.e., minimizing the sum of the depth order matching degrees of all pixel pairs). Iterative optimization refers to repeatedly executing the following process until convergence: First, at the current predicted height, the value of the overall optimization objective is calculated; then, the gradient of the optimization objective relative to the current predicted height is calculated (this gradient is obtained through backpropagation using a differentiable renderer); next, based on the gradient direction and a preset learning rate, the current predicted height is updated using a gradient descent optimizer (such as the Adam optimizer, stochastic gradient descent optimizer, etc.); finally, it is determined whether the convergence condition is met (e.g., the change in the optimization objective is less than a preset threshold, or the preset maximum number of iterations has been reached). If not, the next iteration continues; if met, the iteration stops, and the current predicted height is output as the target measured height.

[0134] This embodiment employs an iterative optimization approach for height estimation, transforming the height measurement problem into a differentiable numerical optimization problem. The gradient descent method automatically finds the height value that optimizes the consistency of the depth order. Compared to methods that directly solve closed-form solutions, iterative optimization can handle nonlinear depth comparison relationships and is insensitive to initial values. Compared to exhaustive search methods, gradient descent can efficiently converge to the vicinity of the optimal solution with significantly less computation. Using gradient information provided by the differentiable renderer, the optimization process automatically learns how to adjust the height parameters to match the depth order constraints of multiple views, eliminating the need for manual adjustment rules. After obtaining the target measured height, a 3D building model is constructed. Specifically, the bottom contour is stretched vertically according to the target measured height to obtain a parametric 3D volume model, which serves as the 3D building model corresponding to the target building.

[0135] The specific implementation of the stretching operation is as follows: The bottom contour (a two-dimensional polygon) is used as the base, keeping its vertex coordinates on the horizontal plane (XY plane) unchanged. A vertical coordinate component (Z-axis direction) is added to each vertex. Specifically, let the vertex coordinates of the bottom contour be (x_i, y_i, 0) (located in the Z=0 plane), and the target measured height be h. Then, the vertex coordinates of the top contour obtained after stretching are (x_i, y_i, h). Next, the corresponding vertices of the bottom and top contours are connected sequentially to form the lateral surfaces of the prism. The bottom and top surfaces each constitute polygonal facets. The final resulting geometry is the parametric three-dimensional volume model.

[0136] A parametric 3D volumetric model is a 3D geometric model defined parametrically, whose geometry is entirely determined by two parameters: the vertex sequence of the bottom contour and the target measured height. Unlike mesh models that explicitly store all vertices and faces, parametric models only need to store the key vertices and height values ​​of the bottom contour, allowing for the dynamic generation of the complete mesh geometry during rendering or export. This representation offers advantages such as small data size, ease of editing, and lossless precision.

[0137] This embodiment uses an extrusion method to construct a 3D building model, which can meet the core requirements of urban building proxy modeling with a concise geometric expression, namely the building's spatial location, plan shape, and main height. The extruded volume model discards fine geometric details on the building facade (such as textures, decorative components, etc.), while fully preserving the building's spatial occupancy information and volumetric characteristics, which is sufficient for applications such as digital twin cities, urban planning simulation, and UAV path planning. Compared with methods that output complex mesh models, the extruded volume model significantly reduces storage space and computational overhead; compared with methods that only output two-dimensional contours, the extruded volume model adds a height dimension, possessing complete 3D representation capabilities. By simplifying the 3D modeling problem into a parametric expression of "two-dimensional contour plus one-dimensional height," this embodiment maximizes computational efficiency and data compactness while maintaining sufficient modeling accuracy.

[0138] The final output 3D building model can be stored and used in various formats. For example, it can be exported as a common 3D mesh format such as OBJ or PLY for use by mainstream 3D rendering engines; it can also be exported as a Shapefile format with a height field added to the attribute table for use by GIS systems. Optionally, it can also be exported as a JSON or GeoJSON format for storage in the form of a parametric description, facilitating dynamic model generation during network transmission or front-end rendering.

[0139] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages in other steps. It is understood that the steps in different embodiments can be freely combined as needed, and all non-contradictory solutions formed by such combinations are within the scope of protection of this application.

[0140] Based on the same inventive concept, this application also provides an aerial image-based urban building proxy reconstruction device for implementing the above-mentioned urban building proxy reconstruction method based on aerial images. The solution provided by this device is similar to the implementation described in the above method. Therefore, the specific limitations of one or more aerial image-based urban building proxy reconstruction device embodiments provided below can be found in the limitations of the aerial image-based urban building proxy reconstruction method described above, and will not be repeated here.

[0141] In one exemplary embodiment, such as Figure 3 As shown, a city building proxy reconstruction device 300 based on aerial imagery is provided, comprising:

[0142] The acquisition module 302 is used to acquire multi-view images corresponding to the target building; the multi-view images include near-vertical view images and oblique view images.

[0143] The first determining module 304 is used to determine the bottom outline of the target building based on the near-vertical view image;

[0144] The second determining module 306 is used to determine the first depth difference between each pixel in the tilted view image;

[0145] The rendering module 308 is used to render the target building based on the camera pose parameters of the multi-view image and the current predicted height of the target building, so as to obtain a rendering depth map of the target building at the current predicted height.

[0146] The optimization module 310 is used to optimize the current predicted height with the goal of reducing the distance between the second depth difference and the first depth difference between each pixel in the rendered depth map, so as to obtain the target measured height of the target building.

[0147] The construction module 312 is used to construct a three-dimensional building model corresponding to the target building based on the bottom outline and the target measured height.

[0148] The modules in the aforementioned urban building reconstruction device based on aerial imagery can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device as software, so that the processor can call and execute the corresponding operations of each module.

[0149] In one exemplary embodiment, a computer device is provided, which may be a terminal, and its internal structure diagram may be as follows: Figure 4 As shown, the computer device includes a processor, memory, input / output interfaces, a communication interface, a display unit, and an input device. The processor, memory, and input / output interfaces are connected via a system bus, and the communication interface, display unit, and input device are also connected to the system bus via the input / output interfaces. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The input / output interfaces are used for exchanging information between the processor and external devices. The communication interface is used for wired or wireless communication with external terminals; wireless communication can be achieved through Wi-Fi, mobile cellular networks, Near Field Communication (NFC), or other technologies. When executed by the processor, the computer program implements a method for urban building proxy reconstruction based on aerial imagery. The display unit is used to form a visually visible image and can be a display screen, a projection device, or a virtual reality imaging device. The display screen can be an LCD screen or an e-ink screen. The input device of the computer device can be a touch layer covering the display screen, or buttons, trackballs, or touchpads set on the casing of the computer device, or external keyboards, touchpads, or mice, etc.

[0150] Those skilled in the art will understand that Figure 4 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0151] In one exemplary embodiment, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps included in any of the foregoing method embodiments.

[0152] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the steps included in any of the foregoing method embodiments.

[0153] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps included in any of the foregoing method embodiments.

[0154] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of the relevant data must comply with relevant regulations.

[0155] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, artificial intelligence (AI) processors, etc., and are not limited to these.

[0156] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0157] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A method for urban building proxy reconstruction based on aerial images, characterized in that, The method includes: Acquire multi-view images of the target building; the multi-view images include near-vertical view images and oblique view images; The bottom outline of the target building is determined based on the near-vertical view image; Determine the first depth difference between each pixel in the tilted view image; Rendering is performed based on the camera pose parameters of the multi-view image and the current predicted height of the target building to obtain a rendering depth map of the target building at the current predicted height; With the goal of reducing the distance between the second depth difference and the first depth difference between each pixel in the rendered depth map, the current predicted height is optimized to obtain the target measured height of the target building; Based on the bottom outline and the target measured height, a three-dimensional building model corresponding to the target building is constructed.

2. The method according to claim 1, characterized in that, The acquisition of multi-view images corresponding to the target building includes: Based on real-time dynamic positioning technology, the UAV performs oblique photography to acquire images of the target urban scene, obtaining multi-view images covering the target buildings and the camera pose parameters; the camera pose parameters include pose information acquired by the UAV during the acquisition process. The multi-view images are divided into near-vertical view images and oblique view images according to the image shooting angle; wherein, the near-vertical view images are used to represent the building roof and planar distribution information of the target building, and the oblique view images are used to represent the building facade and lateral structure information of the target building.

3. The method according to claim 1, characterized in that, Determining the bottom outline of the target building based on the near-vertical view image includes: The building region mask is extracted from the near-vertical view image using a preset building segmentation model; The building area mask is subjected to contour extraction and vectorization to obtain a closed vector polygon; The vector polygon is regularized to obtain the bottom outline of the target building.

4. The method according to claim 1, characterized in that, Determining the first depth difference between pixels within the tilted view image includes: For each of the aforementioned tilted view images, the visible building information corresponding to the current tilted view image is calculated based on the camera pose parameters and the current predicted height; the visible building information includes the number of visible buildings and the projected area of ​​the visible buildings; The tilted view images are filtered based on the visible building conditions to obtain a first tilted view image set; Based on the spatial distribution uniformity of each image in the first tilted view image set, the first tilted view image set is filtered to obtain the second tilted view image set. The second tilted-view image set is supplemented with images until the target building is observed by a preset number of images, and the tilted-view images are replaced according to the supplemented second tilted-view image set.

5. The method according to claim 1, characterized in that, Determining the first depth difference between pixels within the tilted view image includes: Depth estimation is performed on each tilted view image using a monocular depth estimation model to obtain the prior depth map corresponding to each tilted view image. For each tilted view image, differentiable rendering is performed based on the camera pose parameters and the current predicted height to obtain the rendering depth map corresponding to the tilted view image and the visible building area in the rendering depth map. The prior depth map and the rendered depth map are aligned within the visible building area. The pixel depth values ​​of the aligned prior depth map and the rendered depth map within the visible building area are then normalized to obtain the processed prior depth map and the processed rendered depth map. The normalization process includes normalizing the depth value of each pixel based on the median of all pixel depth values ​​within the visible building area and the depth deviation between each pixel and the median. The first depth difference is determined based on the processed prior depth map, and the second depth difference is determined based on the processed rendered depth map.

6. The method according to claim 5, characterized in that, The optimization of the current predicted height, with the goal of reducing the distance between the second depth difference and the first depth difference between pixels in the rendered depth map, to obtain the target measured height of the target building includes: Multiple sets of pixel pairs are randomly sampled within the target area of ​​the tilted view image, and pixel pairs whose absolute value of the first depth difference is less than a preset depth threshold difference are removed to obtain a target pixel pair set; the target area includes visible building areas; For each pixel pair in the target pixel pair set, the depth order matching degree of the current pixel pair in the prior depth map and the rendered depth map is determined based on the difference between the second depth difference of the current pixel pair and the preset interval threshold, the consistency of the numerical signs of the first depth difference and the second depth difference, and the correlation between the change in the distance between the first depth difference and the second depth difference and the change in the current predicted height. With the goal of minimizing the sum of the depth order matching degrees corresponding to the target pixel pair set, the current predicted height is iteratively optimized to obtain the target measured height; The step of constructing a three-dimensional building model corresponding to the target building based on the bottom contour and the target measured height includes: The bottom contour is stretched vertically according to the target measured height to obtain a parametric three-dimensional volume model, which serves as the three-dimensional building model corresponding to the target building.

7. A device for urban building proxy reconstruction based on aerial images, characterized in that, The device includes: The acquisition module is used to acquire multi-view images of the target building; the multi-view images include near-vertical view images and oblique view images. The first determining module is used to determine the bottom outline of the target building based on the near-vertical view image; The second determining module is used to determine the first depth difference between each pixel in the tilted view image; The rendering module is used to render the target building based on the camera pose parameters of the multi-view image and the current predicted height of the target building, so as to obtain a rendering depth map of the target building at the current predicted height. The optimization module is used to optimize the current predicted height with the goal of reducing the distance between the second depth difference and the first depth difference between each pixel in the rendered depth map, so as to obtain the target measured height of the target building. A construction module is used to construct a three-dimensional building model corresponding to the target building based on the bottom outline and the target measured height.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 6.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 6.

Images