A method and device for generating and reconstructing a city instance based on sparse view angles

Abstract

The application belongs to the technical field of urban model three-dimensional reconstruction and generation, and discloses a kind of urban instance generation and reconstruction method and device based on sparse view, method includes extracting the geometric prior of each aerial image and two-dimensional mask;Initialize the global building ID of two-dimensional mask in reference view, determine the global building ID of two-dimensional mask for the rest aerial image using voting mechanism;The geometric prior is injected into multi-view 3D generation model, and the building monomer grid with texture of each building instance is output;Get the aligned building monomer grid;The environment grid of background area in aerial image is constructed, and after completing texture mapping for the environment grid, it is merged with the aligned building monomer grid, and a complete urban 3D scene is obtained.The application generates and accurately aligns the instance grid of urban monomer building under the condition of sparse view, combines generative prior knowledge with physical observation data.

Description

Technical Field

[0001] This invention belongs to the field of urban model 3D reconstruction and generation technology, specifically involving a method and apparatus for urban instance generation and reconstruction based on a sparse perspective. Background Technology

[0002] Constructing semantically meaningful, instance-level three-dimensional (3D) city models is a crucial foundation for next-generation urban applications such as digital twins, smart city management, and emergency response. Currently, high-fidelity city reconstruction primarily relies on traditional multi-view stereo (MVS) heavy photogrammetry workflows, or implicit neural rendering techniques such as Neural Radiance Field (NeRF) and 3D Gaussian Splatting (3DGS) that have emerged in recent years. However, in practical applications such as rapid post-disaster drone mapping and crowdsourced data integration, data often originates from sparse viewpoints with wide baselines. Under sparse viewpoint conditions, existing technologies face the following significant technical challenges:

[0003] The traditional Structure from Motion (SfM) / MVS method suffers from degradation: due to large baselines and occlusion, it lacks sufficient cross-view feature matching, leading to camera registration failure or pose drift. The reconstructed geometry often exhibits large-area holes, thin structural gaps, and broken boundaries.

[0004] Topological and generalization limitations of implicit neural representations: NeRF and 3DGS lack the explicit mesh topology required by standard Geographic Information System (GIS) workflows, and typically model the scene holistically, making it difficult to decouple individual building instances. The lack of geometric priors in sparse viewpoints makes them highly susceptible to severe artifacts or "floating objects" in unobserved areas.

[0005] Existing 3D generative artificial intelligence (AI) is difficult to apply directly: Although large-scale pre-trained models, such as the Segment Anything Model 3D-Object (SAM3D-Object), can generate reasonable 3D structures from a single image, they are mostly limited to generating single objects in a local coordinate system and lack measurement alignment with the real-world geographic spatial coordinate system, which cannot meet the strict accuracy requirements of surveying and GIS. Summary of the Invention

[0006] To address the issues of pose instability and geometric fragmentation (deficiencies of traditional methods) in the reconstruction of individual urban buildings from a sparse perspective, as well as the lack of metric accuracy and multi-view support in generative models. Figure 1 To address the technical problem of consistency (deficiencies in generative methods), the present invention aims to provide a method for generating and reconstructing urban instances based on a sparse perspective. Under the condition of a sparse perspective (limited number of perspectives, insufficient overlap, and large baseline), this method combines generative prior knowledge with physical observation data to generate and accurately align grids of individual urban building instances.

[0007] To achieve the above objectives, the technical solution adopted by the present invention is as follows:

[0008] Firstly, a method for generating and reconstructing city instances based on a sparse perspective is provided, including the following steps:

[0009] Given a sequence of aerial images, extract the geometric prior and two-dimensional mask for each aerial image, and construct a data tuple containing the aerial image, geometric prior, and two-dimensional mask;

[0010] Select one frame of aerial image from the aerial image sequence as the reference view, initialize the global building ID of the two-dimensional mask in the reference view, use geometric prior as anchor point constraint, and use a voting mechanism to determine the global building ID for the two-dimensional mask of the remaining aerial images;

[0011] The geometric prior is injected into the multi-view 3D generation model. The multi-view 3D generation model is used to process aerial images and determine the two-dimensional mask after determining the global building ID, and output textured building unit meshes for each building instance.

[0012] Register the building unit grid with the observation point cloud corresponding to the aerial image sequence to obtain the inverse transformation matrix, and apply the inverse transformation matrix to the building unit grid to obtain the aligned building unit grid.

[0013] An environmental mesh is constructed for the background area in the aerial image. After texture mapping is performed on the environmental mesh, it is merged with the aligned individual building meshes to obtain a complete 3D city scene.

[0014] Several alternative methods are provided below, but they are not intended as additional limitations on the overall solution above. They are merely further additions or optimizations. Provided there are no technical or logical contradictions, each alternative method can be combined individually with respect to the overall solution above, or multiple alternative methods can be combined with each other.

[0015] Preferably, the step of using geometric priors as anchor point constraints and employing a voting mechanism to determine the global building ID for the two-dimensional mask of the remaining aerial images includes:

[0016] The geometric prior of each aerial image is back-projected into 3D space to obtain the depth point cloud of each aerial image;

[0017] The depth point cloud of the reference view is projected onto the two-dimensional mask of each of the other aerial images. Combined with depth consistency constraints, a 2D-3D mapping relationship between the reference view and the other aerial images is established.

[0018] Based on the 2D-3D mapping relationship, calculate multiple repetition indicators of each global building ID corresponding to the point cloud on the two-dimensional mask of the aerial image;

[0019] Voting is conducted based on the repetition index, and the two-dimensional masks in the remaining aerial images are assigned a known global building ID or a new global building ID according to the voting results.

[0020] Preferably, assigning a known global building ID or a new global building ID to the two-dimensional mask in the remaining aerial images based on the voting results includes:

[0021] If all repeating metrics meet the tracking threshold, then the known global building ID is assigned to the corresponding two-dimensional mask;

[0022] If all duplicate metrics fail to meet the tracking threshold and reach the weak overlap threshold, then a new global building ID is assigned to the corresponding two-dimensional mask.

[0023] Otherwise, mark the current 2D mask in the current aerial image as a candidate mask, and assign the candidate mask a known global building ID or a new global building ID based on the voting results of the candidate mask in other aerial images.

[0024] Preferably, assigning a known global building ID or a new global building ID to a candidate mask based on the voting results of the candidate mask in other aerial images includes:

[0025] Based on the spatial location and projection range of the candidate mask, determine the corresponding two-dimensional mask in other aerial images. If no corresponding two-dimensional mask exists in other aerial images, discard the candidate mask; otherwise, perform the following operations:

[0026] If all repeatability metrics of at least one determined two-dimensional mask satisfy the tracking threshold, then the known global building ID is assigned to the candidate mask.

[0027] Otherwise, assign a new global building ID to the candidate mask.

[0028] Preferably, the repeatability metrics include purity, coverage, and F1 score.

[0029] Preferably, the multi-view 3D generation model independently predicts the noise of each aerial image during the denoising generation stage, and updates the shared 3D latent variables through weighted aggregation.

[0030] Preferably, the registration of the building unit grid and the observation point cloud corresponding to the aerial image sequence to obtain the inverse transformation matrix includes:

[0031] Based on the geometric priors of each aerial image, the building pixels of each aerial image are back-projected into 3D space to generate observation point clouds;

[0032] For individual building grids and observation point clouds, the iterative nearest point algorithm for point-to-surface error measurement is used to calculate the rigid transformation matrix;

[0033] Take the inverse matrix of the rigid transformation matrix as the inverse transformation matrix.

[0034] Preferably, the construction of the environmental grid for the background region in the aerial image includes:

[0035] For non-building background areas in all aerial images, the ground point cloud is extracted by filtering building masks and using a random sampling consensus algorithm to fit the ground plane. The depth projection is then triangulated into a regular digital surface model, resulting in a structured environmental mesh.

[0036] Secondly, a city instance generation and reconstruction apparatus based on a sparse perspective is provided, including a processor and a memory storing a plurality of computer instructions, wherein the computer instructions, when executed by the processor, implement the steps of the city instance generation and reconstruction method based on a sparse perspective.

[0037] The present invention provides a method and apparatus for generating and reconstructing city instances based on a sparse perspective, which has the following advantages compared with the prior art:

[0038] (1) Breaking through the reconstruction limit of sparse view: effectively overcoming the problem that traditional MVS is prone to large-area holes and geometric fragmentation under wide baseline and sparse view; by introducing generative structural priors, it perfectly repairs the severely obscured areas such as the back and bottom of the building.

[0039] (2) It achieves the unity of visual realism and geographic measurement accuracy: it utilizes the topological completion capability of the generated large model and forces alignment to the real physical coordinates of photogrammetry through the ICP registration mechanism, thus eliminating the problems of "coordinate system drift" and size distortion caused by the pure generated model.

[0040] (3) Output explicit object-level meshes for GIS applications: Unlike the implicit field or point cloud output by NeRF / 3DGS, this solution directly outputs textured, entity-decoupled (Instance-level) triangular meshes, which are naturally compatible with downstream GIS applications such as smart cities, physical simulation, and volume measurement. Attached Figure Description

[0041] Figure 1 This is a flowchart of the city instance generation and reconstruction method based on a sparse perspective according to the present invention.

[0042] Figure 2 This is a framework diagram of the city instance generation and reconstruction method based on sparse perspective of the present invention. Detailed Implementation

[0043] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0044] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. The terminology used herein in the description of the invention is for the purpose of describing particular embodiments only and is not intended to limit the invention.

[0045] To address the issues of pose instability, geometric fragmentation, and lack of metric accuracy and multi-view capability in the reconstruction of individual urban buildings from a sparse perspective. Figure 1 To address the technical problem of consistency, this invention proposes an end-to-end generation-reconstruction fusion framework, specifically a method for generating and reconstructing city instances based on a sparse perspective, such as... Figure 1 and Figure 2 As shown, it includes the following steps:

[0046] Step 1: Given a sparse sequence of aerial images, extract the geometric prior and 2D mask of each aerial image, and construct a data tuple containing the aerial image, geometric prior, and 2D mask.

[0047] To leverage the efficient processing capabilities of a basic large-scale vision model, this embodiment utilizes this model to extract two-dimensional semantics and geometric priors. Specifically, a segmentation model (such as Segment Anything Model 3, SAM3) is used to extract two-dimensional masks of building instances in the image; a depth estimation model (such as Depth Anything 3, DA3) is used to predict the depth map, pose, and intrinsic / extrinsic camera parameters for each viewpoint, thereby bypassing the vulnerability of traditional feature matching under wide baselines and constructing a data tuple containing aerial images, two-dimensional masks, and geometric priors (depth and pose).

[0048] In other embodiments, physical sensors can be used as alternatives: in addition to using depth estimation models to predict depth and camera pose, sparse point clouds obtained by airborne LiDAR (Light Detection and Ranging) scanning can be directly replaced as observation point clouds, and camera extrinsic parameters directly recorded by Real-Time Kinematic-Global Positioning System (RTK-GPS) / Inertial Measurement Unit (IMU) sensors can be used to provide real physical measurement constraints for subsequent modules.

[0049] In other embodiments, alternatives to instance mask acquisition can be used: in addition to using the SAM3 base model, a combination of two-dimensional (2D) object detection models (such as the YOLO series) and traditional image segmentation algorithms (such as GrabCut image segmentation algorithm or conditional random fields) can be used, or instance segmentation networks specifically designed for fine-tuning remote sensing images (such as Mask Region-based Convolutional Neural Network, Mask R-CNN) can be used, which can also achieve the extraction of individual building masks.

[0050] Step 2: Select one frame of aerial image from the aerial image sequence as the reference view, initialize the global building identifier (ID) of the two-dimensional mask in the reference view, use geometric prior as anchor point constraint, and use a voting mechanism to determine the global building ID for the two-dimensional mask of the remaining aerial images.

[0051] To determine a globally unique ID for the same building across multiple sparse views, this embodiment employs a method based on 3D point overlap voting:

[0052] Step 2.1: Backproject the geometric prior of each aerial image into 3D space to obtain the depth point cloud of each aerial image.

[0053] Step 2.2: Project the depth point cloud of the reference view onto the 2D masks of the remaining aerial images. Combined with depth consistency constraints (eliminating occluded invalid points), establish a 2D-3D mapping relationship between the reference view and the remaining aerial images. This embodiment uses depth consistency visibility judgment and 3D point identification overlap index to stably propagate individual building IDs under sparse viewpoints, solving the target identity ambiguity problem under wide baselines.

[0054] Step 2.3, Voting Mechanism: In subsequent views, the global building ID carried by the visible 3D points within each candidate mask is counted. The voting score is calculated by calculating the purity, coverage, and F1 score of the 2D mask for a given global building ID.

[0055]

[0056]

[0057]

[0058] in, For purity, For coverage, The harmonic mean of purity and coverage (F1 score). The two-dimensional mask contains global buildings. (No. The number of 3D points (IDs) This represents the total number of 3D points within the 2D mask. Global buildings in the aerial image from the current perspective The total number of three-dimensional points.

[0059] This embodiment calculates multiple repetition indicators of each global building ID corresponding to the point cloud on the two-dimensional mask of the aerial image based on the 2D-3D mapping relationship; and performs voting based on the repetition indicators, and assigns a known (initialized) global building ID or a new global building ID to the two-dimensional mask in the remaining aerial images according to the voting results.

[0060] Step 2.4: When all three repeating indicators exceed the set threshold, the mask is determined to belong to the corresponding global building ID; if it does not meet the requirements and has very little overlap with the existing ID, a new instance discovery is triggered and a new ID is assigned; otherwise, a secondary confirmation process is performed. This achieves stable instance tracking across different perspectives.

[0061] Step 2.4.1: If all duplicate metrics meet the tracking threshold, then assign the known global building ID to the corresponding two-dimensional mask. In this embodiment, the tracking threshold for all duplicate metrics is set to 80%~100%. In other embodiments, an independent tracking threshold can be set for each duplicate metric.

[0062] Step 2.4.2: If all duplicate indicators fail to meet the tracking threshold and reach the weak overlap threshold, then assign a new global building ID to the corresponding two-dimensional mask. In this embodiment, the weak overlap threshold for all duplicate indicators is set to 5%~10%. In other embodiments, an independent weak overlap threshold can also be set for each duplicate indicator.

[0063] Step 2.4.3 Otherwise (when steps 2.4.1 and 2.4.2 are not satisfied), mark the current two-dimensional mask in the current aerial image as a candidate mask, and assign a known global building ID or a new global building ID to the candidate mask according to the voting results of the candidate mask in other aerial images, which is a two-stage confirmation process.

[0064] In this embodiment, a two-dimensional mask that cannot be directly determined is pre-associated using a two-dimensional confirmation process. In the two-dimensional confirmation process, the two-dimensional mask corresponding to the candidate mask in other aerial images is first determined based on the spatial location and projection range of the candidate mask. If no corresponding two-dimensional mask exists in other aerial images, the candidate mask is discarded; otherwise, the following operations are performed:

[0065] If all repetition metrics of at least one determined two-dimensional mask meet the tracking threshold, then the known global building ID is assigned to the candidate mask; otherwise, a new global building ID is assigned to the candidate mask, and the two-dimensional mask corresponding to the candidate mask is also assigned the new global building ID.

[0066] In this embodiment, a two-dimensional mask is initially associated based on its spatial location and projection range to supplement the confirmation of two-dimensional masks for which the global building ID cannot be identified in the current aerial image. Only when the spatial location and projection range of the mask are within preset change thresholds is the candidate mask considered to match the corresponding two-dimensional mask.

[0067] Step 3: Inject geometric priors into the multi-view 3D generation model, use the multi-view 3D generation model to process aerial images and determine the two-dimensional mask after determining the global building ID, and output textured individual building meshes for each building instance.

[0068] After obtaining building instances with consistent global building IDs, a high-fidelity 3D mesh is synthesized using a mechanism that combines generation and reconstruction to address occlusion and topological loss caused by sparse viewpoints.

[0069] This embodiment extends the single-view 3D generation model to a multi-view environment, employing an Adaptive Multi-View Fusion for Layout-Aware 3D Generation (MV-SAM3D) model. This model introduces an adaptive multi-diffusion strategy during testing in the denoising generation stage (flow matching process): independently predicting noise for each viewpoint and updating shared 3D latent variables through weighted aggregation, as expressed in the following formula:

[0070]

[0071] in, for At each time step, the final denoised prediction after multi-view adaptive weighted fusion is... This represents the total number of aerial images input. Indicates the first A weighted factor for each viewpoint (aerial image), which measures the importance or visibility contribution of different viewpoints in the current generation step (in the simplest case, an average is taken). Indicates the first A noise prediction network with multiple perspectives. Indicates the current denoising time step The latent feature states of the underlying three-dimensional geometry or texture. For time step index, Indicates the first Prior data (image, depth, mask) extracted from the frame view. This process integrates multi-view features and utilizes generative priors (completing the reasonable structure of invisible areas) to output a topologically complete building unit mesh.

[0072] In this embodiment, aerial images and two-dimensional masks after determining the global building ID are used as inputs to the MV-SAM3D model. Furthermore, the relative depth estimation within the model is discarded, and the high-precision physical depth map obtained in the geometric prior is directly injected to ensure that the generated 3D latent variables are at the same scale as the real camera pose.

[0073] In this embodiment, the relative depth is replaced with high-precision physical depth during the denoising process, and a multi-diffusion weighting strategy is used to aggregate multi-view noise, so as to achieve strong consistency between cross-view geometry and texture generation.

[0074] In other embodiments, the building unit mesh can also be replaced with the following generation scheme to complete the geometry:

[0075] (1) Latent space fusion replacement based on cross-attention mechanism: In the denoising U-Net network of the generative model, instead of using noise summation averaging, a cross-view attention layer is introduced. The features of the reference view are used as keys and values, and the features of the current generated view are used as queries. The geometric consistency between multiple views is directly learned in the latent space, thereby generating a topologically complete 3D mesh.

[0076] (2) Alternative based on deformation template library: Instead of using diffusion models to generate meshes, a computer-aided design (CAD) model library for 3D urban architecture is pre-built. Based on the observed multi-view contours and depths, graph convolutional networks or traditional Laplacian deformation algorithms are used to stretch and deform the vertices of the most similar CAD base mesh to fit the current physical observations. This also allows for the use of library priors to complete occluded areas.

[0077] Step 4: Register the building unit grid and the observation point cloud corresponding to the aerial image sequence to obtain the inverse transformation matrix, and apply the inverse transformation matrix to the building unit grid to obtain the aligned building unit grid.

[0078] To address the issue of the generated mesh being located in a dimensionless canonical coordinate system, an iterative closest point (ICP) registration strategy, proceeding from coarse to fine, is employed for physical spatial alignment.

[0079] First, using the depth map and camera parameters from step 1, all building pixels corresponding to the views are back-projected into the real 3D space to form a sparse observation point cloud with absolute measurement accuracy.

[0080] Then, the individual building grids are registered with the observation point cloud, and the rigid transformation matrix is ​​calculated using the ICP algorithm with point-to-plane error metric.

[0081] Finally, the inverse transformation matrix of the rigid transformation matrix is ​​obtained and applied to the generated mesh to strictly register it to the real-world geospatial coordinate system, resulting in an aligned building unit mesh that combines topological integrity and physical measurement accuracy. The formula is as follows:

[0082]

[0083] in, Represents the inverse transformation matrix. Represents the rigid transformation matrix. Represents the grid of individual buildings. This represents the aligned grid of individual building units.

[0084] In the grid registration and correction mechanism from the generated space to the metric space in this embodiment, the sparse point cloud projected by the real physical camera parameters is used as the source data, and the generated grid in the normal coordinate system is used as the target data. The inverse transformation matrix is ​​calculated based on the point-to-surface ICP algorithm, and the generated grid is forcibly pulled back to the real geographic mapping coordinate system.

[0085] This embodiment uses physical observation (point cloud obtained from depth / camera parameters) as anchor constraints and the output of the 3D generated large model as geometric completion, achieving the combination of physical mapping and AI generation in the same end-to-end process.

[0086] In other embodiments, during camera pose correction and registration in metric space, the iterative nearest-point algorithm used to pull the dimensionless generated mesh back to the real physical world coordinate system can be replaced by: pose optimization based on differentiable rendering: instead of using direct geometric registration between the 3D point cloud and the mesh, the generated mesh is reprojected onto the viewpoints of each physical camera through a differentiable renderer to obtain the rendering mask and rendering depth. The loss function between this and the actual mask and actual depth observed in the actual image is calculated, and the 6-DOF pose parameters and global scaling scale of the generated mesh are directly optimized using gradient descent until the rendering result coincides with the observation result.

[0087] Step 5: Construct an environmental mesh for the background area in the aerial image. After completing texture mapping for the environmental mesh, merge it with the aligned individual building meshes to obtain a complete 3D city scene.

[0088] For non-building background areas in all aerial images, building masks are filtered out and ground point clouds are extracted. The Random Sample Consensus (RANSAC) algorithm is used to fit the ground plane, and the depth projection is triangulated into a regular digital surface model (DSM) environment mesh. Specifically, the image depth information of the ground plane is back-projected into 3D space to generate a point cloud. A continuous surface is constructed through triangulation, and further interpolated into a regular grid digital surface model (DSM) to obtain a structured environment mesh representation. In the texture generation of the environment mesh, background pixels are first projected onto the DSM mesh, and areas without texture are repaired using a diffusion model.

[0089] Then, the textured environment mesh is merged with the precisely aligned building unit mesh to output the final complete 3D city scene.

[0090] In other embodiments, the environmental mesh can also be constructed using the following methods: for background point clouds where buildings have been removed, a continuous terrain mesh can be constructed directly using the Delaunay triangulation algorithm; or a lightweight multilayer perceptron can be used to fit the implicit height field of the ground (i.e., input x, y coordinates, output height z), and then the smooth environmental surface can be extracted through meshing.

[0091] To verify the advantages of the method of the present invention, the following experiments were conducted.

[0092] The experimental data for this invention were collected in a typical urban scene. The data collection area is approximately 4.72 km², which includes residential areas, high-rise buildings, dense building clusters, and complex urban scenes with occlusion and repetitive textures. This data can effectively reflect typical problems in urban building instance recognition, cross-view association, and 3D reconstruction.

[0093] Experimental data were acquired using a DJI M300RTK drone platform equipped with a Zenmuse L1 LiDAR sensor. This sensor integrates laser scanning and high-precision positioning capabilities, enabling simultaneous acquisition of image data and high-density 3D point cloud data. A total of 17,910 drone images were obtained across the entire acquisition area. Within this complete acquisition area, five representative urban building scenes were selected as experimental cases, designated as Case 1, Case 2, Case 3, Case 4, and Case 5. The selected cases cover different building layouts, scales, heights, and surrounding environments, including dense residential areas, high-rise building areas, and areas with complex occlusion and repetitive textures. These cases were used to verify the effectiveness of the present invention in building instance association, generative reconstruction, and 3D model updating under different sparse viewpoint conditions.

[0094] To obtain a quantifiable and comparable reference model, a conventional oblique photogrammetry method with high overlap was used in the data acquisition phase. A high-quality real-world 3D model was obtained through dense multi-view image reconstruction. Subsequently, a sparse-view experimental input was constructed based on this densely acquired data, and the densely reconstructed model was used as the accuracy evaluation benchmark, thereby ensuring that the experimental results have repeatability and quantifiable evaluation criteria.

[0095] The experimental computing environment included a 13th generation Intel Core i9-13900K processor and an NVIDIA GeForce RTX4090 graphics card with 24GB of video memory. This hardware environment could support computational processes such as multi-view geometry processing, point cloud processing, generative prior fusion, and 3D model reconstruction, and ensured that each group of comparative experiments was conducted under the same hardware conditions.

[0096] This embodiment uses four different methods as comparison objects: Neural Implicit Surface Reconstruction from SparseViews with Feature and Depth Consistencies (SparseRecon), using 4 images as input; Object-aware Scene Reconstruction and Scene Understanding via Gaussian Splatting (ObjectGS), using 4 images as input; the ObjectGS multi method, which uses 20 images as input; and Oblique Photography (Oblique), using more than 20 images as input. Ours is used as the experimental subject, using 4 images as input. The reconstruction results show that with a small number of images as input, ours achieves the best reconstruction effect, most closely resembling the original input image.

[0097] In addition, to verify the 3D reconstruction effect of this invention under sparse viewpoint conditions, four sparse input viewpoints were set up, and 3D models generated by different methods were rendered per viewpoint using uniform camera parameters and consistent rendering settings. Using the input image as a reference, three metrics were used to evaluate the consistency between the rendered result and the reference image: Peak Signal-to-Noise Ratio (PSNR), Structural Similarity Index Measure (SSIM), and Learned Perceptual Image Patch Similarity (LPIPS). PSNR measures pixel-level error, with higher values ​​being better; SSIM evaluates the ability to preserve image structure and texture, with higher values ​​being better; and LPIPS measures visual similarity at the perceptual level, with lower values ​​being better. The results are shown in Table 1.

[0098] Table 1. 3D Reconstruction Results

[0099]

[0100] Experimental results show that, under strictly sparse viewpoint input conditions, the proposed method achieves superior overall performance, with a PSNR of 13.80, SSIM of 0.238, and LPIPS reduced to 0.298. Compared with the ObjectGS method, the proposed method significantly improves upon these three metrics. Compared with ObjectGS multi, the proposed method maintains higher pixel consistency, structural similarity, and perceptual quality, indicating that the proposed building instance association and generation-reconstruction fusion strategy can effectively alleviate problems such as texture blurring, structural instability, and geometric missingness under sparse viewpoint conditions. As a reference method under dense image conditions, the Oblique method achieves the highest results across the three metrics, with a PSNR of 16.21, an SSIM of 0.359, and LPIPS of 0.202. This is mainly due to its reliance on highly overlapping imagery and traditional oblique photogrammetry procedures, providing more comprehensive multi-viewpoint observation constraints. In contrast, this invention uses only a small number of sparse viewpoints as input. Although the overall performance is slightly lower than that of dense image reconstruction, it is more stable than other sparse viewpoint reconstruction methods, indicating that this invention has good practicality and reconstruction reliability under low-cost, low-overlap image acquisition conditions.

[0101] Therefore, quantitative results show that the present invention can improve the visual consistency and structural stability of 3D reconstruction of urban building instances from sparse perspectives without relying on dense image acquisition and scene-level training. It is suitable for applications such as rapid updating of urban real-scene 3D models, building instance completion, and low-cost digital twin scene construction.

[0102] In another embodiment, the present invention also provides a city instance generation and reconstruction apparatus based on a sparse perspective, namely a computer device, including a processor and a memory storing a plurality of computer instructions, wherein the computer instructions are executed by the processor to implement the steps of the city instance generation and reconstruction method based on a sparse perspective.

[0103] The memory and processor are electrically connected directly or indirectly to enable data transmission or interaction. For example, these components can be electrically connected to each other via one or more communication buses or signal lines. The memory stores a computer program that can run on the processor, and the processor implements the method of the present invention by running the computer program stored in the memory.

[0104] The memory can be, but is not limited to, Random Access Memory (RAM), Read Only Memory (ROM), Programmable Read-Only Memory (PROM), Erasable Programmable Read-Only Memory (EPROM), and Electrically Erasable Programmable Read-Only Memory (EEPROM). The memory stores the program, which the processor executes upon receiving execution instructions.

[0105] A processor can be an integrated circuit chip with data processing capabilities. The processor can be a general-purpose processor, including a Central Processing Unit (CPU), a Network Processor (NP), etc. It can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this invention.

[0106] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this invention.

[0107] The above embodiments merely illustrate several implementation methods of the present invention, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the invention. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention, and these all fall within the protection scope of the present invention. Therefore, the protection scope of the present invention should be determined by the appended claims.

Claims

1. A method for generating and reconstructing city instances based on a sparse perspective, characterized in that, Includes the following steps: Given a sequence of aerial images, extract the geometric prior and two-dimensional mask for each aerial image, and construct a data tuple containing the aerial image, geometric prior, and two-dimensional mask; Select one frame of aerial image from the aerial image sequence as the reference view, initialize the global building ID of the two-dimensional mask in the reference view, use geometric prior as anchor point constraint, and use a voting mechanism to determine the global building ID for the two-dimensional mask of the remaining aerial images; The geometric prior is injected into the multi-view 3D generation model. The multi-view 3D generation model is used to process aerial images and determine the two-dimensional mask after determining the global building ID, and output textured building unit meshes for each building instance. Register the building unit grid with the observation point cloud corresponding to the aerial image sequence to obtain the inverse transformation matrix, and apply the inverse transformation matrix to the building unit grid to obtain the aligned building unit grid. An environmental mesh is constructed for the background area in the aerial image. After texture mapping is performed on the environmental mesh, it is merged with the aligned individual building meshes to obtain a complete 3D city scene.

2. The method for generating and reconstructing city instances based on a sparse perspective according to claim 1, characterized in that, The method of using geometric priors as anchor point constraints and employing a voting mechanism to determine the global building ID for the two-dimensional mask of the remaining aerial images includes: The geometric prior of each aerial image is back-projected into 3D space to obtain the depth point cloud of each aerial image; The depth point cloud of the reference view is projected onto the two-dimensional mask of each of the other aerial images. Combined with depth consistency constraints, a 2D-3D mapping relationship between the reference view and the other aerial images is established. Based on the 2D-3D mapping relationship, calculate multiple repetition indicators of each global building ID corresponding to the point cloud on the two-dimensional mask of the aerial image; Voting is conducted based on the repetition index, and the two-dimensional masks in the remaining aerial images are assigned a known global building ID or a new global building ID according to the voting results.

3. The method for generating and reconstructing city instances based on a sparse perspective according to claim 2, characterized in that, Assigning a known global building ID or a new global building ID to the two-dimensional mask in the remaining aerial images based on the voting results includes: If all repeating metrics meet the tracking threshold, then the known global building ID is assigned to the corresponding two-dimensional mask; If all duplicate metrics fail to meet the tracking threshold and reach the weak overlap threshold, then a new global building ID is assigned to the corresponding two-dimensional mask. Otherwise, mark the current 2D mask in the current aerial image as a candidate mask, and assign the candidate mask a known global building ID or a new global building ID based on the voting results of the candidate mask in other aerial images.

4. The method for generating and reconstructing city instances based on a sparse perspective according to claim 3, characterized in that, The step of assigning a known global building ID or a new global building ID to a candidate mask based on the voting results of the candidate mask in other aerial images includes: Based on the spatial location and projection range of the candidate mask, determine the corresponding two-dimensional mask in other aerial images. If no corresponding two-dimensional mask exists in other aerial images, discard the candidate mask; otherwise, perform the following operations: If all repeatability metrics of at least one determined two-dimensional mask satisfy the tracking threshold, then the known global building ID is assigned to the candidate mask. Otherwise, assign a new global building ID to the candidate mask.

5. The method for generating and reconstructing city instances based on a sparse perspective according to claim 2, characterized in that, The replication metrics include purity, coverage, and F1 score.

6. The method for generating and reconstructing city instances based on a sparse perspective according to claim 1, characterized in that, The multi-view 3D generation model independently predicts the noise of each aerial image during the denoising generation stage, and updates the shared 3D latent variables through weighted aggregation.

7. The method for generating and reconstructing city instances based on a sparse perspective according to claim 1, characterized in that, The registration of the observation point cloud corresponding to the individual building grid and the aerial image sequence to obtain the inverse transformation matrix includes: Based on the geometric priors of each aerial image, the building pixels of each aerial image are back-projected into 3D space to generate observation point clouds; For individual building grids and observation point clouds, the iterative nearest point algorithm for point-to-surface error measurement is used to calculate the rigid transformation matrix; Take the inverse matrix of the rigid transformation matrix as the inverse transformation matrix.

8. The method for generating and reconstructing city instances based on a sparse perspective according to claim 1, characterized in that, The construction of the environmental mesh for the background region in the aerial image includes: For non-building background areas in all aerial images, the ground point cloud is extracted by filtering building masks and using a random sampling consensus algorithm to fit the ground plane. The depth projection is then triangulated into a regular digital surface model, resulting in a structured environmental mesh.

9. A device for generating and reconstructing city instances based on a sparse perspective, comprising a processor and a memory storing a plurality of computer instructions, characterized in that, When the computer instructions are executed by the processor, they implement the steps of the sparse perspective-based city instance generation and reconstruction method according to any one of claims 1 to 8.

Images