A 3D reconstruction method for unbounded scenes based on drone aerial photography

Abstract

The present invention discloses a method for 3D reconstruction of unbounded scenes based on drone aerial photography, which relates to the field of image processing technology. The technical solution includes the following steps: obtaining a multi-view image set of the target area and constructing a spatial topological association network; extracting image features based on the Hessian matrix and the SIFT operator, and generating scene partitions through Euclidean distance matching; performing 3D Gaussian sputtering optimization on each partition, and synchronously generating reusable 3D basic assets, wherein the basic assets include structural components and texture features; dynamically combining the basic assets based on the spatial transformation rules defined by the programmatic code to generate a hierarchical 3D model; performing geometric consistency verification in the latent space through joint optimization of the diffusion model and the differentiable renderer; assembling the optimized hierarchical model into an unbounded 3D scene according to the spatial layout parameters. It aims to solve the problem of high consumption of 3D modeling resources and insufficient geometric accuracy in large-scale scenes.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to a method for three-dimensional reconstruction of an unbounded scene based on drone aerial photography. Background Art

[0002] With the rapid development of science and technology, 3D reconstruction has become a core technology in many fields. This technology is particularly valuable in applications such as urban planning, topographic mapping, disaster assessment, and environmental monitoring. Accurate 3D modeling of scenes provides decision makers with more accurate geospatial information, improving work efficiency, reducing manual intervention, and saving time and costs.

[0003] However, traditional UAV 3D modeling methods typically rely on image matching and point cloud data processing, resulting in poor modeling accuracy. This is particularly true in scenes with complex details or occlusions, making high-precision 3D models impossible to guarantee. Data optimization and correction also require extensive manual intervention. Furthermore, traditional 3D reconstruction methods require significant computational time, especially for large-scale scenes. This often requires lengthy computational time, making it impractical for practical applications.

[0004] To overcome these challenges, image-based 3D reconstruction methods have gradually become a research hotspot in the field of drone modeling. By combining efficient and convenient image acquisition from drones with deep learning and computer vision technologies, more accurate 3D reconstruction can be achieved. Deep learning methods can automatically extract feature information from images through training on large-scale datasets, optimizing model accuracy, reducing manual intervention, and improving the degree of modeling automation. Furthermore, image-based 3D reconstruction methods can significantly reduce the reliance on point cloud data used in traditional modeling methods when processing certain scenes, thereby reducing computational time and memory consumption and improving modeling efficiency. However, as the scale of scenes continues to increase, the computational cost and memory space requirements of 3D reconstruction also increase.

[0005] Therefore, how to avoid high consumption of 3D modeling resources and insufficient geometric accuracy in large-scale scenarios remains a technical challenge that needs to be solved urgently. Summary of the Invention

[0006] The main purpose of the present invention is to provide a 3D reconstruction method for unbounded scenes based on drone aerial photography, aiming to solve the problems of high consumption of 3D modeling resources and insufficient geometric accuracy in large-scale scenes.

[0007] To achieve the above objectives, the present invention proposes a method for 3D reconstruction of an unbounded scene based on drone aerial photography, comprising the following steps:

[0008] Obtain a multi-view image set of the target area and construct a spatial topological association network;

[0009] Image features are extracted based on the Hessian matrix and SIFT operator, and scene partitions are generated through Euclidean distance matching;

[0010] Perform 3D Gaussian sputtering optimization on each partition and simultaneously generate a reusable 3D base asset containing structural components and texture features;

[0011] Based on the spatial transformation rules defined by programmatic code, the underlying assets are dynamically combined to generate a hierarchical three-dimensional model;

[0012] We perform geometric consistency verification in the latent space by jointly optimizing a diffusion model with a differentiable renderer.

[0013] The optimized hierarchical model is assembled into an unbounded three-dimensional scene according to the spatial layout parameters.

[0014] In one embodiment of the present application, the 3D Gaussian sputtering optimization includes:

[0015] Initialize programmatic code for each partition, defining the layout constraints of the underlying asset, including spatial boundaries, rotation parameters, and scaling;

[0016] During 3D Gaussian sputtering optimization, the Gaussian distribution range is limited by bounding box adaptive constraints, including:

[0017] (a) Gaussian center Beyond the soft bounding box , forcing its position to align to the boundary;

[0018] (b) If the Gaussian center exceeds the boundary, the threshold is expanded according to the boundary Update its location;

[0019] The formula is as follows:

[0020]

[0021] Synchronously update the Gaussian parameters of repeated underlying assets and achieve parameter sharing through gradient backpropagation; represents the adjusted Gaussian center position, represents the original x-axis lower bound of partition i; represents the original x-axis upper bound of partition i; Indicates the boundary extension alignment threshold; Soft boundary relaxation amount; represents the original position of the Gaussian center; Represents the center coordinate of the kth Gaussian distribution on the x-axis.

[0022] In one embodiment of the present application, the programmatic code generation includes:

[0023] Segment repetitive structural components from multi-view images and reconstruct their geometric and texture features through 3D Gaussian sputtering optimization;

[0024] Encode the transformation rules of components into a programmatic instruction set, including layer expansion, axial arrangement, and adaptive size adjustment;

[0025] Automatically generate asset distribution rules based on spatial layout parameters;

[0026] When the unbounded scene is expanded, scene continuity is achieved through block loading and real-time splicing.

[0027] In one embodiment of the present application, the diffusion model is combined with a differentiable renderer for optimization, including:

[0028] In the latent space of the diffusion model, the underlying asset texture is enhanced with multi-scale denoising;

[0029] Project the optimized asset to multiple views through a differentiable renderer and calculate the geometric consistency loss:

[0030]

[0031] in, represents the geometric consistency loss; is the number of partitions, Reconstruct the ground truth depth for the partition, For rendering depth maps; supports users to modify programmed code parameters in real time, dynamically triggering local Gaussian parameter re-optimization.

[0032] In one embodiment of the present application, performing geometric consistency verification in the latent space includes the following steps:

[0033] The rendering diffusion model uses the partitioned 3D reconstruction result as input and performs a diffusion process in a low-resolution latent space. Given a camera pose obtained by spherical linear interpolation, a differentiable renderer generates an initial image based on the camera pose as a condition for the diffusion process. The low-resolution latent space is a low-resolution partial region or sub-region. The differentiable renderer is a renderer whose output can be differentiated with respect to the input parameters.

[0034] The initial image is encoded in a low-resolution latent space using a variational autoencoder, and feature reverse sampling and extraction is performed in the latent space. The encoding is the process of iteratively adding Gaussian noise to the initial image. The feature reverse sampling and extraction is the process of extracting features similar to the original ordered image from the chaotic and disordered image through a denoising operation to generate a denoised image.

[0035] Decoding the denoised image in pixel space using a decoder in a variational autoencoder, and converting the decoded denoised image into a pixel space image, where the pixel space is the original representation space of the image, in which each pixel has its specific color value and grayscale value;

[0036] Pixel loss is used to maintain the three-dimensional consistency between the generated pixel space image and the input partitioned three-dimensional reconstruction result, and the three-dimensional network reconstruction optimization result is output.

[0037] In one embodiment of the present application, assembling the optimized hierarchical model according to the spatial layout parameters includes:

[0038] Input the boundary point clouds of adjacent partitions into the graph convolutional network to calculate the feature similarity of the overlapping areas;

[0039] When the overlap exceeds a threshold, partitions are automatically merged using programmatic rules, including:

[0040] (a) Align the partition coordinate system to eliminate the stitching gap;

[0041] (b) Smooth interpolation of Gaussian parameters of the merged area;

[0042] Generate seamless global point cloud maps, supporting dynamic loading and memory compression.

[0043] In one embodiment of the present application, the feature similarity of the overlapping area is calculated using the graph attention mechanism:

[0044] ;

[0045] in, , E is the edge set of the overlapping area, for point and The eigenvector of Represents the two point clouds whose similarity is to be calculated; Represents a pair of adjacent points (nodes) in an edge set; Represents the feature vector The cosine similarity of Attention weight; and Represents learnable query and key transformation matrices; The dimension of the feature vector;

[0046] The combined Gaussian interpolation formula is:

[0047]

[0048] in and The Gaussian parameters to be merged, the weight w is calculated based on the attenuation of the distance from the overlapping center; Gaussian parameters after merging; and represents the weight coefficient; Represents Gaussian distribution The center coordinates of Indicates the center coordinates of the overlapping area.

[0049] In one embodiment of the present application, the SIFT operator extracts image features, specifically including the following steps:

[0050] The SIFT operator is used to perform multi-scale Gaussian difference filtering on the input image, detecting extreme points in the image as key points. The key points are located in the scale space and local neighborhood, and the main direction is assigned to them based on the gradient direction. The graph convolutional neural network is used to sample the key point neighborhood, aggregate the local features of the key point neighborhood, and complete feature extraction.

[0051] For each pixel point in the target image, the disparity value is obtained by subtracting the pixel coordinates of the image features in the adjacent images in the two partitions, thereby constructing the corresponding disparity map between the two images;

[0052] The depth value corresponding to each pixel in the target image is calculated using the camera parameters and the disparity value in the disparity map, thereby constructing a depth map of the target image, where the pixel value of the pixel corresponds to the depth value of the pixel;

[0053] Through the depth value in the depth map, each pixel is converted into a three-dimensional coordinate in the real world to obtain a point cloud;

[0054] The above steps are repeated to obtain the point cloud data containing texture feature information in each partition.

[0055] In one embodiment of the present application, the calculation formula using the SIFT operator is:

[0056]

[0057]

[0058] in, represents the accuracy of the gradient vector, represents the direction of the gradient vector, represents the coordinates of each sampling point, , Indicates the horizontal and vertical coordinates of the coordinate point.

[0059] In one embodiment of the present application, the calculation formula for calculating the Hessian matrix of each image in the image set is as follows:

[0060] ;

[0061] in = , represents the Hessian matrix, Represents each sampling point, express, express, express, They all represent the values ​​of the second-order derivatives, , Indicates the horizontal and vertical coordinates of the coordinate point.

[0062] This technical solution, using drone aerial photography to capture multi-perspective images and combining them with advanced image processing and 3D reconstruction techniques, can accurately restore the 3D structure of unbounded scenes. Efficient image feature matching and partition generation reduce computational complexity and improve modeling efficiency. The simultaneous generation of reusable 3D base assets further enhances the model's flexibility and scalability, allowing the resulting 3D scene to be flexibly adapted to different application scenarios. This solves the problem of high 3D modeling resource consumption and insufficient geometric accuracy in large-scale scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0063] The present invention will be described in detail below with reference to specific embodiments and accompanying drawings, wherein:

[0064] Figure 1 This is a schematic diagram of the process structure of the first embodiment of the present invention. DETAILED DESCRIPTION

[0065] In order to make the purpose, technical solutions and advantages of the present invention more clear, the present invention is described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the following specific embodiments are only used to explain the present invention and do not constitute a limitation of the present invention.

[0066] like Figure 1 To achieve the above objectives, the present invention proposes a method for 3D reconstruction of an unbounded scene based on drone aerial photography, comprising the following steps:

[0067] Obtain a multi-view image set of the target area and construct a spatial topological association network;

[0068] Image features are extracted based on the Hessian matrix and SIFT operator, and scene partitions are generated through Euclidean distance matching;

[0069] Perform 3D Gaussian sputtering optimization on each partition and simultaneously generate a reusable 3D base asset containing structural components and texture features;

[0070] Based on the spatial transformation rules defined by programmatic code, the underlying assets are dynamically combined to generate a hierarchical three-dimensional model;

[0071] We perform geometric consistency verification in the latent space by jointly optimizing a diffusion model with a differentiable renderer.

[0072] The optimized hierarchical model is assembled into an unbounded three-dimensional scene according to the spatial layout parameters.

[0073] Specifically, the target area is first photographed using a drone. A multi-view image set is acquired from various angles and viewing distances. This image set should cover all key locations and features of the target area to ensure sufficient detail in the reconstructed scene. Next, image processing algorithms are used to extract spatial relationships between images and construct a spatial topological association network. This network describes the geometric relationships between images from different perspectives.

[0074] The Hessian matrix is ​​used to detect and describe feature points in the acquired image. The Hessian matrix helps detect structural features such as edges and corners in the image. The SIFT (Scale-Invariant Feature Transform) operator is then used to further extract local feature points from the image and generate corresponding feature descriptors. The extracted feature descriptors are used to match images within the image collection. The degree of association between images is determined by calculating the Euclidean distance between image features. Images with a high degree of association indicate strong similarity or spatial overlap.

[0075] Based on the image correlation, an image partitioning algorithm is used to partition the scene of the image collection. Each partition represents a sub-region or perspective of the target area, and can classify related images into the same group.

[0076] For each partition, 3D reconstruction optimization is performed using a 3D Gaussian sputtering optimization algorithm, leveraging the geometric information from multi-view images to restore the scene's 3D structure. During this process, reusable 3D base assets are generated simultaneously. These base assets include structural components and texture features. Structural components refer to fixed geometric parts of the scene, while texture features refer to surface details within the scene, such as the ground, walls, and vegetation. This base asset can be reused to reduce recalculation and improve reconstruction efficiency.

[0077] Rules are used to adjust spatial parameters such as the position, rotation, and scale of each underlying asset to ensure their rational layout and efficient combination within the overall 3D model. This approach allows for the automated generation of 3D models at different levels, making the entire 3D scene more expressive and diverse. Hierarchical 3D models clearly demonstrate the structure and details of the scene, and models at different levels can be refined or simplified as needed.

[0078] The generated 3D model is further optimized using a joint optimization technique involving a diffusion model and a differentiable renderer. The diffusion model is used to explore diverse geometric shapes within the latent space, and combined with the differentiable renderer, detailed corrections are performed on the geometric model to ensure visual consistency within the resulting 3D model. Geometric consistency verification ensures that all parts of the 3D model are spatially aligned, reducing geometric errors and ensuring the realism and accuracy of the scene.

[0079] By parameterizing the spatial layout of the optimized hierarchical model, the individual 3D components are assembled into an unbounded 3D scene according to specific spatial relationships based on the actual needs of the scene. An unbounded scene is one with unlimited boundaries and can be freely expanded or scaled as needed, making it suitable for a variety of virtual reality and augmented reality applications. Through precise spatial layout and rational model assembly, the resulting 3D scene provides a flexible, efficient, and realistic virtual experience in a variety of application scenarios.

[0080] This technical solution, using drone aerial photography to capture multi-perspective images and combining them with advanced image processing and 3D reconstruction techniques, can accurately restore the 3D structure of unbounded scenes. Efficient image feature matching and partition generation reduce computational complexity and improve modeling efficiency. The simultaneous generation of reusable 3D base assets further enhances the model's flexibility and scalability, allowing the resulting 3D scene to be flexibly adapted to different application scenarios. This solves the problem of high 3D modeling resource consumption and insufficient geometric accuracy in large-scale scenarios.

[0081] In one embodiment of the present application, the 3D Gaussian sputtering optimization includes:

[0082] Initialize programmatic code for each partition, defining the layout constraints of the underlying asset, including spatial boundaries, rotation parameters, and scaling;

[0083] During 3D Gaussian sputtering optimization, the Gaussian distribution range is limited by bounding box adaptive constraints, including:

[0084] (a) Gaussian center Beyond the soft bounding box , forcing its position to align to the boundary;

[0085] (b) If the Gaussian center exceeds the boundary, the threshold is expanded according to the boundary Update its location;

[0086] The formula is as follows:

[0087]

[0088] Synchronously update the Gaussian parameters of repeated underlying assets and achieve parameter sharing through gradient backpropagation; represents the adjusted Gaussian center position, represents the original x-axis lower bound of partition i; represents the original x-axis upper bound of partition i; Indicates the boundary extension alignment threshold; Soft boundary relaxation amount; represents the original position of the Gaussian center; Represents the center coordinate of the kth Gaussian distribution on the x-axis.

[0089] Specifically, the base assets within each partition are assigned spatial boundaries, rotation parameters, and scale factors based on the target scene's requirements. Spatial boundaries define the area within which the asset can be placed in three-dimensional space, rotation parameters determine the asset's orientation, and scale factors define its size. These constraints ensure that the asset layout meets the predetermined design requirements and maintain the rationality and consistency of the scene during the reconstruction process.

[0090] In 3D Gaussian sputtering optimization, a bounding box adaptive constraint is used to limit the range of the Gaussian distribution to avoid the distribution points exceeding the target area or producing unreasonable distribution. The bounding box defines the spatial range of each partition. Through this constraint, the center point of the Gaussian distribution can be limited ( ) within a specific range. Specific constraint operations include:

[0091] (a) If the center point of the Gaussian distribution ( ) is outside the defined soft bounding box (i.e. ), its position is forced to be aligned to the bounding box. This means that if the Gaussian center point is outside the soft boundary in the x-axis direction, the position will be adjusted to the inside of the bounding box to keep the point within the reasonable range.

[0092] (b) If the Gaussian center point is outside the hard bounding box, the boundary will be extended by the threshold ( ) adjusts its position so that it can return to the valid area. This operation ensures that even in extreme cases, the center point of the Gaussian distribution remains within an acceptable range, thus avoiding uncontrollable distribution deviation.

[0093] The formula is as follows:

[0094]

[0095] in, represents the adjusted Gaussian center position, represents the original x-axis lower bound of partition i; represents the original x-axis upper bound of partition i; Indicates the boundary extension alignment threshold; Soft boundary relaxation amount; represents the original position of the Gaussian center; Represents the center coordinate of the kth Gaussian distribution on the x-axis;

[0096] The Gaussian parameters of repeated underlying assets within the same partition are updated synchronously through gradient backpropagation. For multiple underlying assets of the same type, their Gaussian distribution parameters, such as the center position, are shared and adjusted through an optimization algorithm. This parameter sharing effectively reduces redundant computation, improves reconstruction efficiency, and ensures a more consistent and coordinated 3D scene.

[0097] The above technical solution, using Gaussian sputtering optimization techniques combined with bounding box adaptive constraints, effectively limits the distribution of underlying assets in 3D space, preventing assets from exceeding the predefined boundaries of the scene. Sharing Gaussian parameters through gradient backpropagation enables efficient updates and consistency control of underlying assets, ensuring consistency in both detail and overall layout of the 3D scene. This solution significantly improves the accuracy and efficiency of 3D reconstruction.

[0098] In one embodiment of the present application, the programmatic code generation includes:

[0099] Segment repetitive structural components from multi-view images and reconstruct their geometric and texture features through 3D Gaussian sputtering optimization;

[0100] Encode the transformation rules of components into a programmatic instruction set, including layer expansion, axial arrangement, and adaptive size adjustment;

[0101] Automatically generate asset distribution rules based on spatial layout parameters;

[0102] When the unbounded scene is expanded, scene continuity is achieved through block loading and real-time splicing.

[0103] Specifically, repetitive structural components are segmented from multi-view images. Using image analysis and segmentation algorithms, repetitive parts of the target scene, such as buildings, roads, and vegetation, are identified. These repetitive components are then reconstructed in 3D using a 3D Gaussian sputtering optimization algorithm. This process not only reconstructs their geometric structure but also extracts and reconstructs their texture features to ensure that the reconstruction is visually consistent with the original image. 3D Gaussian sputtering optimization effectively repairs and refines the geometric and texture features of repetitive components, improving the scene's realism and detail.

[0104] The transformation rules for repeating structural components are encoded into a procedural instruction set. The procedural code defines how to operate on these components and includes three main transformation rules:

[0105] Layer expansion: Adjust the number of component layers based on scene requirements to increase or decrease the vertical height or depth of the component so that it can adapt to spatial layouts at different levels.

[0106] Axial Arrangement: Determines the axial arrangement of components in space, such as arranging them along the x-axis, y-axis, or z-axis. By using transformation rules, ensure the proper placement of components in the scene to accommodate different scene structures.

[0107] Adaptive size adjustment: Based on the density and spatial layout of the target area, the size of the component is adjusted to adapt it to the proportion of the surrounding environment, ensuring that the scene meets the design requirements both visually and spatially.

[0108] The procedural code generates asset distribution rules based on spatial layout parameters, including topological boundaries and area density.

[0109] Topological boundaries: By defining the spatial boundaries of the scene, determine the area where each asset should be placed and its location range, ensuring that each component is laid out in the scene according to the set boundaries to avoid duplication and redundancy.

[0110] Regional Density: Based on the overall needs and design of the scenario, determine the density of each region, that is, how many assets to place in a specific area. By calculating the density within the region, adjust the number and arrangement of each asset to achieve a reasonable distribution of assets.

[0111] When expanding, the scene is divided into multiple small blocks, and the content of each block is loaded separately and stitched together in real time. Since the scene is divided into multiple small blocks, only the portion currently needed for display is loaded into memory, reducing the consumption of computing resources. When loading new scene blocks, a real-time stitching algorithm seamlessly connects the new blocks with the existing scene data, ensuring continuity and stability during scene expansion. This method enables seamless and smooth scene expansion without boundary restrictions, making it suitable for application scenarios such as virtual reality and augmented reality.

[0112] This technical solution, by segmenting and reconstructing repetitive structural components from multi-view images and generating transformation and asset distribution rules through programmatic code, enables the efficient construction of consistent and highly accurate 3D scenes. In particular, the application of block loading and real-time splicing ensures scene continuity and fluidity during scene expansion. This technical solution improves the efficiency of large-scale 3D scene reconstruction while ensuring scene stability and scalability, adapting to dynamic environments and real-time updates.

[0113] In one embodiment of the present application, the diffusion model is combined with a differentiable renderer for optimization, including:

[0114] In the latent space of the diffusion model, the underlying asset texture is enhanced with multi-scale denoising;

[0115] Project the optimized asset to multiple views through a differentiable renderer and calculate the geometric consistency loss:

[0116]

[0117] in, represents the geometric consistency loss; is the number of partitions, Reconstruct the ground truth depth for the partition, For rendering depth maps; supports users to modify programmed code parameters in real time, dynamically triggering local Gaussian parameter re-optimization.

[0118] Specifically, a diffusion model is used to perform multi-scale denoising and enhancement on the texture of the underlying asset. The core concept of the diffusion model is to gradually remove noise and enhance texture detail by processing the texture at multiple levels and scales. Texture information is mapped into the latent space of the diffusion model, and denoising is applied within this space. The denoising process processes the texture through multi-scale convolution or diffusion, ensuring that the final output texture retains detail at different scales and effectively suppresses noise.

[0119] The optimized assets are rendered using a differentiable renderer, projecting them into a depth map from multiple perspectives. The differentiable renderer enables rendering based on the depth map to obtain depth information consistent with the real scene. The rendered results are compared with the ground truth depth values ​​reconstructed by the partitions to calculate the geometric consistency loss. The difference between the rendered depth map and the reconstructed depth truth is measured. The specific calculation formula is as follows:

[0120]

[0121] in, represents the geometric consistency loss, is the number of partitions, Reconstruct the ground truth depth for the partition, For rendering depth maps; supports users to modify programmed code parameters in real time, dynamically triggering local Gaussian parameter re-optimization.

[0122] The system provides a user interface that allows users to modify the parameters of the programmatic code in real time. By dynamically modifying parameters such as spatial layout and component transformation rules, users can adjust the details and structure of the scene as needed. When users modify parameters, the system automatically triggers reoptimization of local Gaussian parameters. As the programmatic code executes, the Gaussian distribution parameters of the underlying assets associated with the user's modifications are recalculated and updated to ensure that these changes are correctly reflected in the 3D model. Through this dynamic optimization mechanism, users can control the scene generation process in real time, ensuring that the final 3D scene meets the user's actual needs and design intent.

[0123] The above technical solution significantly improves the texture quality and geometric accuracy of the underlying assets in 3D scenes through multi-scale denoising enhancement of the diffusion model and geometric consistency optimization of the differentiable renderer. The calculation and optimization of the geometric consistency loss ensures a high degree of consistency between the rendered results and the real-world depth information, thereby improving the realism and usability of the model. In addition, the ability to modify programmatic code parameters in real time and trigger local Gaussian parameter reoptimization allows users to flexibly adjust the details of the 3D scene, further enhancing the applicability of this technical solution for dynamic scene generation and personalized needs.

[0124] In one embodiment of the present application, performing geometric consistency verification in the latent space includes the following steps:

[0125] The rendering diffusion model uses the partitioned 3D reconstruction result as input and performs a diffusion process in a low-resolution latent space. Given a camera pose obtained by spherical linear interpolation, a differentiable renderer generates an initial image based on the camera pose as a condition for the diffusion process. The low-resolution latent space is a low-resolution partial region or sub-region. The differentiable renderer is a renderer whose output can be differentiated with respect to the input parameters.

[0126] The initial image is encoded in a low-resolution latent space using a variational autoencoder, and feature reverse sampling and extraction is performed in the latent space. The encoding is the process of iteratively adding Gaussian noise to the initial image. The feature reverse sampling and extraction is the process of extracting features similar to the original ordered image from the chaotic and disordered image through a denoising operation to generate a denoised image.

[0127] Decoding the denoised image in pixel space using a decoder in a variational autoencoder, and converting the decoded denoised image into a pixel space image, where the pixel space is the original representation space of the image, in which each pixel has its specific color value and grayscale value;

[0128] Pixel loss is used to maintain the three-dimensional consistency between the generated pixel space image and the input partitioned three-dimensional reconstruction result, and the three-dimensional network reconstruction optimization result is output.

[0129] Specifically, the results of the partitioned 3D reconstruction are used as input and rendered to perform the diffusion process. To verify geometric consistency, the diffusion model first operates in a low-resolution latent space. The low-resolution latent space refers to simplified regions or subregions of the 3D reconstruction results. These regions contain the most critical geometric information but are presented at a low resolution. During this process, a reasonable camera pose is calculated using spherical linear interpolation. The camera pose determines the perspective from which the 3D scene is observed and can affect the image rendering effect. Next, a differentiable renderer (i.e., a renderer that can output and differentiate its input parameters) is used to generate an initial image based on the camera pose. This initial image serves as a condition for the diffusion process and provides the basis for subsequent denoising operations.

[0130] A variational autoencoder is used to encode the initial image in a low-resolution latent space. This encoding process is achieved by iteratively adding Gaussian noise to the initial image, gradually disrupting the image's details and introducing noise. This process helps the model learn how to recover important geometric and texture information from the noisy image. By performing feature reverse sampling in the latent space, the variational autoencoder is able to extract features similar to the original, ordered image from the chaotic and disordered image during the denoising operation, ultimately generating a denoised image. The core of this denoising process is to remove noise from the image and restore its true structure and details.

[0131] The denoised image is decoded using the decoder in the variational autoencoder. The decoder converts the denoised latent space image into a pixel space image. In pixel space, each image pixel has a specific color and grayscale value, representing the original representation of the image. The decoding process restores a clear, realistic image by mapping the image from latent space to pixel space. These images provide more intuitive visual effects and can be further used for geometric consistency verification.

[0132] A pixel-wise loss is used to ensure 3D consistency between the generated pixel-space image and the input partitioned 3D reconstruction. Specifically, the pixel-wise loss is applied by comparing the pixel differences between the denoised image and the 3D reconstructed partitions. By minimizing this loss function, the generated image accurately reflects the geometric structure of the partitioned 3D reconstruction. In this process, the pixel-wise loss ensures high consistency between image details and the 3D model, thereby optimizing the 3D network reconstruction results.

[0133] The above technical solution effectively improves the reconstruction accuracy and consistency of 3D models by performing geometric consistency verification in the latent space. By combining a diffusion model with a variational autoencoder, not only can denoising be performed in the low-resolution latent space, but pixel-wise loss can also be used to accurately compare the difference between the generated image and the actual reconstruction, thereby optimizing the 3D reconstruction process. Ultimately, the output of the optimized 3D network reconstruction better preserves the geometric consistency of the 3D structure, ensuring the realism and accuracy of the scene.

[0134] In one embodiment of the present application, assembling the optimized hierarchical model according to the spatial layout parameters includes:

[0135] Input the boundary point clouds of adjacent partitions into the graph convolutional network to calculate the feature similarity of the overlapping areas;

[0136] When the overlap exceeds a threshold, partitions are automatically merged using programmatic rules, including:

[0137] (a) Align the partition coordinate system to eliminate the stitching gap;

[0138] (b) Smooth interpolation of Gaussian parameters of the merged area;

[0139] Generate seamless global point cloud maps, supporting dynamic loading and memory compression.

[0140] In one embodiment of the present application, the feature similarity of the overlapping area is calculated using the graph attention mechanism:

[0141] ;

[0142] in, , E is the edge set of the overlapping area, for point and The eigenvector of Represents the two point clouds whose similarity is to be calculated; Represents a pair of adjacent points (nodes) in an edge set; Represents the feature vector The cosine similarity of Attention weight; and Represents learnable query and key transformation matrices; The dimension of the feature vector;

[0143] The combined Gaussian interpolation formula is:

[0144]

[0145] in and The Gaussian parameters to be merged, the weight w is calculated based on the attenuation of the distance from the overlapping center; Gaussian parameters after merging; and represents the weight coefficient; Represents Gaussian distribution The center coordinates of Indicates the center coordinates of the overlapping area.

[0146] Specifically, the point cloud to be processed is calculated through the graph attention mechanism and The graph attention mechanism can assign different attention weights to each pair of adjacent nodes through the connection relationship between nodes (i.e., edge set E), thereby accurately measuring the similarity of different parts in the point cloud. The specific calculation process is:

[0147]

[0148] Through the graph attention mechanism, the system can automatically adjust the feature similarity of each pair of adjacent nodes to obtain accurate similarity in overlapping areas. This mechanism can effectively process local feature relationships in point cloud data and help optimize data matching and fusion during the reconstruction process.

[0149] When merging overlapping areas in point clouds, the Gaussian interpolation method is used to weight the Gaussian parameters to be merged. This process helps to smooth the transition and ensure the geometric consistency of the merged area. Specifically, the two Gaussian distributions to be merged are and The parameters of are combined by the following formula:

[0150]

[0151] is the parameter of the merged Gaussian distribution, which realizes the smooth transition of the two Gaussian distributions during the Gaussian interpolation process, ensuring the natural fusion of the geometric and texture features of the point cloud in the overlapping area.

[0152] This technical solution, using a graph attention mechanism to calculate feature similarity in overlapping regions of point clouds, enables more accurate assessment of similarity between different point clouds and optimizes point cloud alignment and matching. A merging method based on Gaussian interpolation ensures geometric consistency in overlapping regions of point clouds and smoothes the transition between different distributions using a distance decay strategy. Ultimately, the resulting Gaussian parameters maintain high accuracy and naturalness during point cloud synthesis, thereby improving the quality of 3D reconstructed models.

[0153] In one embodiment of the present application, the SIFT operator extracts image features, specifically including the following steps:

[0154] The SIFT operator is used to perform multi-scale Gaussian difference filtering on the input image, detecting extreme points in the image as key points. The key points are located in the scale space and local neighborhood, and the main direction is assigned to them based on the gradient direction. The graph convolutional neural network is used to sample the key point neighborhood, aggregate the local features of the key point neighborhood, and complete feature extraction.

[0155] For each pixel point in the target image, the disparity value is obtained by subtracting the pixel coordinates of the image features in the adjacent images in the two partitions, thereby constructing the corresponding disparity map between the two images;

[0156] The depth value corresponding to each pixel in the target image is calculated using the camera parameters and the disparity value in the disparity map, thereby constructing a depth map of the target image, where the pixel value of the pixel corresponds to the depth value of the pixel;

[0157] Through the depth value in the depth map, each pixel is converted into a three-dimensional coordinate in the real world to obtain a point cloud;

[0158] The above steps are repeated to obtain the point cloud data containing texture feature information in each partition.

[0159] Specifically, the image is processed into multiple scale levels, and each scale is smoothed using a Gaussian filter. Then, using the Difference of Gaussian method, the changes in local image features are calculated across the scales, thereby detecting extreme points in the image at different scales. Extreme points are points where the image intensity reaches a local maximum or minimum at the current scale and adjacent scales. These points typically correspond to key features in the image (such as corners and edges).

[0160] Locate the detected extreme points. First, the keypoints need to be precisely located in scale space, and each keypoint is more precisely located within its local neighborhood. By analyzing the gradient direction of the local region where each keypoint resides, the principal direction of that keypoint is calculated. The principal direction refers to the prevailing direction of the pixel gradient within the local region, and it is used to ensure scale invariance under different rotation angles. Therefore, assigning the principal direction can improve the stability and consistency of feature points under rotational transformations.

[0161] A graph convolutional neural network (GCN) is used to sample the neighborhood of each keypoint. A GCN is a deep learning model capable of processing graph-structured data. By sampling image information from the neighborhood surrounding a keypoint, it aggregates local features within that neighborhood. This process effectively extracts texture features around the keypoint and captures detailed information about the local region in the image. These extracted local features provide essential feature support for subsequent matching, recognition, and reconstruction.

[0162] For each pixel in the target image, the disparity value is calculated by subtracting the pixel coordinates in the adjacent image. The disparity value represents the spatial offset between the same scene point in the two images under varying perspectives. By matching feature points in the images and calculating the difference in their pixel positions under different perspectives, a disparity map is constructed. The disparity map provides a disparity value for each pixel, which is used to infer the depth of an object in stereo vision.

[0163] Based on the acquired disparity map, the depth value of each pixel is calculated using the camera's internal and external parameters (such as focal length and camera baseline length). The depth value represents the physical distance between the pixel and the camera. Based on the camera's geometric model, the disparity value can be used to calculate the depth of each pixel, thus generating a complete depth map. Depth maps are an important form of 3D information representation, in which the value of each pixel corresponds to the depth information of a point in the scene, used to represent the position of objects in 3D space.

[0164] Using the depth information in the depth map, the pixel coordinates (i.e., x and y in the image coordinate system) are combined with the depth value and back-projected using the camera's internal and external parameters to calculate the actual position of each pixel in 3D space. Finally, the 3D coordinates of all pixels are combined to form a complete point cloud.

[0165] Repeat the above steps for each partition in the target image. Within each partition, the features extracted by the SIFT operator, along with the calculated disparity and depth values, help generate point cloud data containing texture features within that partition. By looping through each partition, we ultimately obtain point cloud data for the entire scene. These point clouds contain not only spatial information but also texture features from the image, more realistically reflecting the scene's structure and details.

[0166] This technical solution uses the SIFT operator to extract multi-scale features from images, combined with deep learning graph convolutional neural networks for feature extraction. This accurately captures and effectively aggregates local features in the image, providing stable and reliable texture information for point cloud generation. The construction of disparity and depth maps enables the accurate calculation of the 3D spatial position of each pixel, generating high-quality point cloud data.

[0167] In one embodiment of the present application, the calculation formula using the SIFT operator is:

[0168]

[0169]

[0170] in, represents the accuracy of the gradient vector, represents the direction of the gradient vector, represents the coordinates of each sampling point, , Indicates the horizontal and vertical coordinates of the coordinate point.

[0171] Using the above technical solution, by calculating the accuracy of the gradient vector and direction The SIFT operator accurately captures local features in an image and provides a foundation for subsequent feature description. The gradient magnitude reflects the intensity of detail changes in the image, while the gradient direction describes the direction of these changes. These two quantities are central to feature point extraction, matching, and rotational invariance. This information helps identify stable and unique local features in different image regions, providing high-quality feature data for applications such as 3D reconstruction and object recognition.

[0172] In one embodiment of the present application, the calculation formula for calculating the Hessian matrix of each image in the image set is as follows:

[0173] ;

[0174] in = , represents the Hessian matrix, Represents each sampling point, express, express, express, They all represent the values ​​of the second-order derivatives, , Indicates the horizontal and vertical coordinates of the coordinate point.

[0175] Using this technical solution, the Hessian matrix can accurately capture areas of image variation, particularly for detecting structural features such as edges and corners. Edges and corners are often the areas of most dramatic change in an image, and the second-order derivative can effectively reflect these changes. The second-order derivative is robust to scale variations and noise. This enables the Hessian matrix to maintain relatively stable performance in multi-scale image analysis.

[0176] The above description is only a preferred embodiment of the present invention and does not limit the patent scope of the present invention. Any equivalent structural transformation made by using the contents of the present description and drawings under the inventive concept of the present invention, or directly / indirectly applied in other related technical fields, is included in the patent protection scope of the present invention.

Claims

1. A method for 3D reconstruction of unbounded scenes based on drone aerial photography, characterized in that: The following steps are involved: Obtain a multi-view image set of the target area and construct a spatial topological association network; Image features are extracted based on the Hessian matrix and SIFT operator, and scene partitions are generated through Euclidean distance matching; Perform 3D Gaussian sputtering optimization on each partition to simultaneously generate reusable structural components and texture features; Based on the spatial transformation rules defined by programmatic code, structural components and texture features are dynamically combined to generate a hierarchical three-dimensional model; We perform geometric consistency verification in the latent space by jointly optimizing a diffusion model with a differentiable renderer. Assemble the optimized hierarchical model into an unbounded three-dimensional scene according to the spatial layout parameters; The 3D Gaussian sputtering optimization includes: Initializing programmatic code for each partition to define layout constraints for structural components and texture features, the layout constraints including spatial boundaries, rotation parameters, and scaling; During 3D Gaussian sputtering optimization, the Gaussian distribution range is limited by bounding box adaptive constraints, including: (a) Gaussian center Beyond the soft bounding box , forcing its position to align to the boundary; (b) If the Gaussian center exceeds the boundary, the threshold is expanded according to the boundary Update its location; The formula is as follows: Synchronously update the Gaussian parameters of the repeated structural components and texture features, and achieve parameter sharing through gradient back propagation; represents the adjusted Gaussian center position, represents the original x-axis lower bound of partition i; represents the original x-axis upper bound of partition i; Indicates the boundary extension alignment threshold; Soft boundary relaxation amount; represents the original position of the Gaussian center; Represents the center coordinate of the kth Gaussian distribution on the x-axis.

2. The method for 3D reconstruction of an unbounded scene based on drone aerial photography according to claim 1, wherein: The programmatic code generation includes: Segment repetitive structural components from multi-view images and reconstruct their geometric and texture features through 3D Gaussian sputtering optimization; Encode the transformation rules of components into a programmatic instruction set, including layer expansion, axial arrangement, and adaptive size adjustment; Automatically generate structural components and texture feature distribution rules based on spatial layout parameters; When the unbounded scene is expanded, scene continuity is achieved through block loading and real-time splicing.

3. The method for 3D reconstruction of unbounded scenes based on drone aerial photography according to claim 1, wherein: Joint optimization via diffusion model and differentiable renderer, including: In the latent space of the diffusion model, multi-scale denoising and enhancement are performed on the structural components and texture features; The optimized structural components and texture features are projected to multiple perspectives through a differentiable renderer to calculate the geometric consistency loss: in, represents the geometric consistency loss; is the number of partitions, Reconstruct the ground truth depth for the partition, For rendering depth maps; supports users to modify programmed code parameters in real time, dynamically triggering local Gaussian parameter re-optimization.

4. The method for 3D reconstruction of an unbounded scene based on drone aerial photography according to claim 3, wherein: Performing geometric consistency verification in the latent space includes the following steps: The rendering diffusion model uses the partitioned 3D reconstruction result as input and performs a diffusion process in a low-resolution latent space. Given a camera pose obtained by spherical linear interpolation, a differentiable renderer generates an initial image based on the camera pose as a condition for the diffusion process. The low-resolution latent space is a low-resolution partial region or sub-region. The differentiable renderer is a renderer whose output can be differentiated with respect to the input parameters. The initial image is encoded in a low-resolution latent space using a variational autoencoder, and feature reverse sampling and extraction is performed in the latent space. The encoding is the process of iteratively adding Gaussian noise to the initial image. The feature reverse sampling and extraction is the process of extracting features similar to the original ordered image from the chaotic and disordered image through a denoising operation to generate a denoised image. Decoding the denoised image in pixel space using a decoder in a variational autoencoder, and converting the decoded denoised image into a pixel space image, where the pixel space is the original representation space of the image, in which each pixel has its specific color value and grayscale value; Pixel loss is used to maintain the three-dimensional consistency between the generated pixel space image and the input partitioned three-dimensional reconstruction result, and the three-dimensional network reconstruction optimization result is output.

5. The method for 3D reconstruction of unbounded scenes based on drone aerial photography according to claim 1, wherein: The step of assembling the optimized hierarchical model according to the spatial layout parameters includes: Input the boundary point clouds of adjacent partitions into the graph convolutional network to calculate the feature similarity of the overlapping areas; When the overlap exceeds a threshold, partitions are automatically merged using programmatic rules, including: (a) Align the partition coordinate system to eliminate the stitching gap; (b) Smooth interpolation of Gaussian parameters of the merged area; Generate seamless global point cloud maps, supporting dynamic loading and memory compression.

6. The method for 3D reconstruction of unbounded scenes based on drone aerial photography according to claim 1, wherein: The graph attention mechanism is used to calculate the feature similarity of overlapping areas: ； in, , E is the edge set of the overlapping area, for point and The eigenvector of Represents the two point clouds whose similarity is to be calculated; represents a pair of adjacent points in an edge set; Represents the feature vector The cosine similarity of Attention weight; and Represents learnable query and key transformation matrices; The dimension of the feature vector; The combined Gaussian interpolation formula is: in and Gaussian parameters to be merged; Gaussian parameters after merging; and represents the weight coefficient; Represents Gaussian distribution The center coordinates of Indicates the center coordinates of the overlapping area.

7. The method for 3D reconstruction of unbounded scenes based on drone aerial photography according to claim 1, wherein: The SIFT operator extracts image features, which specifically includes the following steps: The SIFT operator is used to perform multi-scale Gaussian difference filtering on the input image, detecting extreme points in the image as key points. The key points are located in the scale space and local neighborhood, and the main direction is assigned to them based on the gradient direction. The graph convolutional neural network is used to sample the key point neighborhood, aggregate the local features of the key point neighborhood, and complete feature extraction. For each pixel point in the target image, the disparity value is obtained by subtracting the pixel coordinates of the image features in the adjacent images in the two partitions, thereby constructing the corresponding disparity map between the two images; The depth value corresponding to each pixel in the target image is calculated using the camera parameters and the disparity value in the disparity map, thereby constructing a depth map of the target image, where the pixel value of the pixel corresponds to the depth value of the pixel; Through the depth value in the depth map, each pixel is converted into a three-dimensional coordinate in the real world to obtain a point cloud; The above steps are repeated to obtain the point cloud data containing texture feature information in each partition.

8. The method for 3D reconstruction of unbounded scenes based on drone aerial photography according to claim 1, wherein: The calculation formula using the SIFT operator is: in, represents the accuracy of the gradient vector, represents the direction of the gradient vector, represents the coordinates of each sampling point, , Indicates the horizontal and vertical coordinates of the coordinate point.

9. The method for 3D reconstruction of unbounded scenes based on drone aerial photography according to claim 1, wherein: The calculation formula for calculating the Hessian matrix of each image in the image set is as follows: ； in = , represents the Hessian matrix, Represents each sampling point, 、 、 、 They all represent the values ​​of the second-order derivatives, , Indicates the horizontal and vertical coordinates of the coordinate point.

Images