A method for generating orthophotos based on load-balanced 3D Gaussian sputtering

Abstract

This invention discloses an orthorectified image generation method based on load-balanced 3D Gaussian sputtering, belonging to the field of computer vision technology. By designing a joint global SfM strategy to quickly recover camera pose and sparse point clouds, it solves the problem of low initialization efficiency of 3D Gaussian sputtering in large-scale scenes. An adaptive spatial partitioning strategy based on point cloud density is proposed to dynamically balance the computational load between different blocks, enabling block-based parallel training. Combined with view allocation and auxiliary point strategies, it effectively eliminates boundary artifacts of blocks, promoting seamless fusion of blocks. Orthogonal rasterization rendering technology is used to directly generate orthorectified images. This invention significantly shortens processing time while ensuring the geometric accuracy of orthorectified images, making it suitable for large-scale urban mapping and emergency response scenarios.

Description

Technical Field

[0001] This invention belongs to the field of computer vision technology, specifically relating to a method for generating orthophotos based on load-balanced 3D Gaussian sputtering. Background Technology

[0002] Traditional orthophoto generation methods primarily involve recovering camera pose from SfM (Simultaneous Method Modeling), then generating DSM (Distributed Modeling Model) using MVS (Multi-View Spectroscopy), followed by complex geometric differential correction and occlusion detection based on the DSM. However, this cascaded process has significant drawbacks; the final image quality heavily depends on the accuracy of the DSM, and insufficient DSM accuracy can easily lead to edge stretching and facade distortion. In recent years, 3D Gaussian sputtering, as an explicit and efficient scene representation method, models the scene using a 3D Gaussian ellipsoid and combines it with a differentiable rasterization process. This approach maintains the continuity of the radiation field while achieving rapid training and real-time rendering, opening up new avenues for orthophoto generation.

[0003] Currently, applying 3D Gaussian sputtering to generate orthophotos of large-scale scenes still faces two major challenges, which existing frameworks have not fully resolved. First, there is an efficiency bottleneck in the initialization phase. 3D Gaussian sputtering algorithms heavily rely on high-quality SfM results to obtain camera pose and sparse point clouds. Current mainstream incremental SfM algorithms (such as COLMAP) are extremely slow in large-scale aerial mapping scenarios, limiting their practical application. Second, when processing large-scale scenes, most existing 3D Gaussian sputtering algorithms use a simple uniform grid partitioning method. However, the density distribution of point cloud data in real scenes is extremely uneven. This partitioning method leads to a severe imbalance in computational load and memory usage between different computational blocks, resulting in a decrease in overall training efficiency. Summary of the Invention

[0004] The purpose of this invention is to provide a method for generating orthophotos based on load-balanced three-dimensional Gaussian sputtering, so as to solve the above-mentioned problems.

[0005] To achieve the above objectives, the present invention provides the following technical solution: a method for generating orthophotos based on load-balanced three-dimensional Gaussian sputtering, the specific steps of which are as follows: S1 acquires aerial images of the target scene and quickly generates camera pose and sparse point cloud by combining global SfM, providing geometric initialization for 3D Gaussian sputtering; S2 divides the scenario into multiple sub-blocks for computational load balancing: S21. The sparse point cloud of the scene is obtained by joint global SfM, the normal vector of the ground plane is estimated by the Manhattan hypothesis, it is aligned to the z-axis, and the horizontal structure of the scene is projected onto the xy plane for subsequent region division. S22, first manually define the rectangular target region Bs and set it as the initial root node for recursive binary search; given the maximum depth M of the binary tree and the threshold number of leaf nodes N. th Perform the following operations on a node of any depth d: Calculate the number N sparse points falling into this node. b ; If d ≤ M and N b >N th Then B will be along the longer side of the rectangular region. s Divide into two sub-regions B s1 and B s2 Then repeat the above process for each sub-region; otherwise, retain it as a leaf node. S3 assigns views based on point visibility for each sub-block and introduces an auxiliary point strategy to eliminate block boundary artifacts, promoting seamless merging of blocks. S4 uses orthogonal rasterization rendering technology to generate high-precision digital orthophotos.

[0006] Preferably, S1 specifically comprises: S101. Use drones to acquire aerial images of the target area, then extract and match features from the acquired drone images. Specifically, use RootSIFT features for feature extraction and description, and combine Bag-of-Words image retrieval technology to filter image pairs to be matched. Then, complete feature matching through brute-force matching. S102. Perform global rotation averaging. We minimize the relative rotation error while simultaneously solving for the global rotation matrix of all cameras. ={Ri}; To mitigate the impact of noise and outliers, we employ a robust global optimization formula: ; Where R ij Indicates relative rotation; Represents the cycle consistency error term, where I is the identity matrix; The rotation group is defined. Geodetic distance on; It is a robust loss function, using the L1 norm; S103. Using the normalized orientation difference as the error metric, simultaneously estimate the centers of all cameras. and three-dimensional points The objective function to be optimized is: ; in Indicates from camera Observation point The global rotation of the camera's light; and The normalization factor is initialized to 1, and the weights of cameras involving unknown intrinsic parameters are reduced by a factor of 2 to mitigate their impact; Huber is used as the robust function. Levenberg-Marquardt was used as the optimizer. S104. To further optimize, multiple rounds of global bundle adjustment were performed. In each round, the camera rotation was first fixed, and then joint optimization was performed together with intrinsic parameters and points. Then, the trajectory was filtered according to the reprojection error in the image space. The iteration stopped when the proportion of the filtered trajectory was less than 0.1%.

[0007] Preferably, S3 specifically includes: S301. A view allocation strategy based on point visibility is adopted. To ensure that each block only receives supervised views that significantly contribute to its geometry and texture, we evaluate the correlation between views and blocks; let... This represents the number of visible points in the training view V throughout the entire scene. This indicates the number of visible points of view V within block b; if If the value is greater than the threshold of 0.3, then the view is selected as the supervisory view for the current block; S302. Design an auxiliary point strategy to introduce a set of auxiliary points for each block. These auxiliary points are extracted from the sparse point cloud corresponding to the block's supervised view, but their range extends beyond the block boundary. This provides correct geometric initialization for the entire field of view and effectively eliminates artifacts that may affect visual quality. After optimization, these auxiliary Gaussians will be discarded. In addition, since adjacent blocks share the same training view in our partitioning scheme, seamless fusion of blocks can be achieved.

[0008] Preferably, S4 specifically includes: S401. Gaussian sputtering rasterization rendering is improved by replacing standard perspective projection with orthographic projection to generate orthographic images; the orthographic projection matrix used is: ; in: ; ; near and far represent the near clipping plane and the far clipping plane, respectively; θx and θy represent the horizontal and vertical fields of view, respectively; top, bottom, left, and right represent the upper, lower, left, and right boundaries of the view frustum, respectively. S402. During the projection process, the mean and covariance Σ undergo different transformations; the mean represents the point coordinates and is directly projected; while the covariance matrix Σ defines the spatial distribution; in order to accurately calculate the transformed shape... Use the following formula: ; Where W is the view transformation matrix, J is the Jacobian matrix, and W T and J T This represents the corresponding transpose matrix; When perspective projection is replaced by orthographic projection, the originally complex and depth-dependent Jacobian matrix becomes a simple constant matrix J. ; S403: By replacing the perspective projection in the 3D Gaussian sputtering rendering process with orthographic projection and setting a uniform height and viewpoint, all Gaussian ellipsoids are projected onto a 2D plane and their opacity is blended, thereby directly generating a high-quality orthographic image.

[0009] The technical effects and advantages of this invention are as follows: By designing a joint global SfM strategy to quickly recover camera pose and sparse point clouds, the problem of low initialization efficiency of 3D Gaussian sputtering in large-scale scenes is solved. An adaptive spatial partitioning strategy based on point cloud density is proposed to dynamically balance the computational load between different blocks, enabling block-based parallel training; combined with view allocation and auxiliary point strategies, boundary artifacts of blocks are effectively eliminated, promoting seamless fusion of blocks; orthogonal rasterization rendering technology is used to directly generate orthorectified images; this invention significantly shortens processing time while ensuring the geometric accuracy of orthorectified images, making it suitable for large-scale urban mapping and emergency response scenarios. Attached Figure Description

[0010] Figure 1 This is a flowchart of the present invention; Figure 2 This invention provides an adaptive block division based on point cloud density, including view allocation and the process of adding auxiliary points. Figure 3 This is a schematic diagram of perspective projection correction orthographic projection according to the present invention; Figure 4 This is an orthophoto image generated using the method of the present invention. Detailed Implementation

[0011] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0012] This invention provides, for example Figures 1-4 The method for generating orthorectified images based on load-balanced 3D Gaussian sputtering, as shown in the figure, includes the following specific steps: S1 acquires aerial images of the target scene and quickly generates camera pose and sparse point cloud by combining global SfM, providing geometric initialization for 3D Gaussian sputtering; S2 divides the scenario into multiple sub-blocks for computational load balancing: S21. The sparse point cloud of the scene is obtained by joint global SfM, the normal vector of the ground plane is estimated by the Manhattan hypothesis, it is aligned to the z-axis, and the horizontal structure of the scene is projected onto the xy plane for subsequent region division. S22, first manually define the rectangular target region Bs and set it as the initial root node for recursive binary search; given the maximum depth M of the binary tree and the threshold number of leaf nodes N. th Perform the following operations on a node of any depth d: Calculate the number N sparse points falling into this node. b ; If d ≤ M and N b >N th Then, along the longer side of the rectangular region, Bs is divided into two sub-regions B. s1 and B s2 Then repeat the above process for each sub-region; otherwise, retain it as a leaf node. S3 assigns views based on point visibility for each sub-block and introduces an auxiliary point strategy to eliminate block boundary artifacts, promoting seamless merging of blocks. S4 uses orthogonal rasterization rendering technology to generate high-precision digital orthophotos.

[0013] Specifically, S1 is as follows: S101. Use drones to acquire aerial images of the target area, then extract and match features from the acquired drone images. Specifically, use RootSIFT features for feature extraction and description, and combine Bag-of-Words image retrieval technology to filter image pairs to be matched. Then, complete feature matching through brute-force matching. S102. Perform global rotation averaging. We minimize the relative rotation error while simultaneously solving for the global rotation matrix of all cameras. ={Ri}; To mitigate the impact of noise and outliers, we employ a robust global optimization formula: ; Where R ij Indicates relative rotation; Represents the cycle consistency error term, where I is the identity matrix; The rotation group is defined. Geodetic distance on; It is a robust loss function, using the L1 norm; S103. Using the normalized orientation difference as the error metric, simultaneously estimate the centers of all cameras. and three-dimensional points The objective function to be optimized is: ; in Indicates from camera Observation point The global rotation of the camera's light; and The normalization factor is initialized to 1, and the weights of cameras involving unknown intrinsic parameters are reduced by a factor of 2 to mitigate their impact; Huber is used as the robust function. Levenberg-Marquardt was used as the optimizer. S104. To further optimize, multiple rounds of global bundle adjustment were performed. In each round, the camera rotation was first fixed, and then joint optimization was performed together with intrinsic parameters and points. Then, the trajectory was filtered according to the reprojection error in the image space. The iteration stopped when the proportion of the filtered trajectory was less than 0.1%.

[0014] Specifically, S3 includes: S301. A view allocation strategy based on point visibility is adopted. To ensure that each block only receives supervised views that significantly contribute to its geometry and texture, we evaluate the correlation between views and blocks; let... This represents the number of visible points in the training view V throughout the entire scene. This indicates the number of visible points of view V within block b; if If the value is greater than the threshold of 0.3, then the view is selected as the supervisory view for the current block; S302. Design an auxiliary point strategy to introduce a set of auxiliary points for each block. These auxiliary points are extracted from the sparse point cloud corresponding to the block's supervised view, but their range extends beyond the block boundary. This provides correct geometric initialization for the entire field of view and effectively eliminates artifacts that may affect visual quality. After optimization, these auxiliary Gaussians will be discarded. In addition, since adjacent blocks share the same training view in our partitioning scheme, seamless fusion of blocks can be achieved.

[0015] Specifically, S4 includes: S401. Gaussian sputtering rasterization rendering is improved by replacing standard perspective projection with orthographic projection to generate orthographic images; the orthographic projection matrix used is: ; in: ; ; near and far represent the near clipping plane and the far clipping plane, respectively; θx and θy represent the horizontal and vertical fields of view, respectively; top, bottom, left, and right represent the upper, lower, left, and right boundaries of the view frustum, respectively. S402. During the projection process, the mean and covariance Σ undergo different transformations; the mean represents the point coordinates and is directly projected; while the covariance matrix Σ defines the spatial distribution; in order to accurately calculate the transformed shape... Use the following formula: ; Where W is the view transformation matrix, J is the Jacobian matrix, and W T and J T This represents the corresponding transpose matrix; When perspective projection is replaced by orthographic projection, the originally complex and depth-dependent Jacobian matrix becomes a simple constant matrix J. ; S403: By replacing the perspective projection in the 3D Gaussian sputtering rendering process with orthographic projection and setting a uniform height and viewpoint, all Gaussian ellipsoids are projected onto a 2D plane and their opacity is blended, thereby directly generating a high-quality orthographic image.

[0016] Working principle: Step S1: Acquire aerial images of the target scene using a UAV, and quickly generate camera pose and sparse point cloud through joint global SfM to provide geometric initialization for 3D Gaussian sputtering. Specific steps include: S101. Use drones to acquire aerial images of the target area, then extract and match features from the acquired drone images. Specifically, RootSIFT features are used for feature extraction and description, and Bag-of-Words image retrieval technology is combined to filter image pairs to be matched. Finally, feature matching is completed through brute-force matching.

[0017] S102. Perform global rotation averaging. We minimize the relative rotation error while simultaneously solving for the global rotation matrix of all cameras. ={Ri}. To mitigate the impact of noise and outliers, we employ a robust global optimization formula: ; Where R ij Indicates relative rotation. This represents the cyclic consistency error term, which ideally should be equal to the identity matrix I. The rotation group is defined. Geodesic distance. It is a robust loss function, using the L1 norm.

[0018] S103. We construct a joint optimization problem, using the normalized direction difference as the error metric, and simultaneously estimate the centers of all cameras. and three-dimensional points The objective function to be optimized is: ; in Indicates from camera Observation point The global rotation of the camera's light. The normalization factor is initialized to 1, and the weights of cameras with unknown intrinsic parameters are reduced by a factor of 2 to mitigate their impact. Huber is used as the robust function. We used Levenberg-Marquardt as the optimizer.

[0019] S104. To further optimize, multiple rounds of global bundle adjustment were performed. In each round, the camera rotation was first fixed, and then joint optimization was performed together with intrinsic parameters and points. Then, the trajectory was filtered based on the reprojection error in the image space. The iteration stopped when the proportion of the filtered trajectory was less than 0.1%.

[0020] Step S2: An adaptive spatial partitioning strategy based on point cloud density was designed to divide the scene into multiple computationally balanced sub-blocks, successfully solving the problem of computational load and memory imbalance in large-scale scenes, thereby improving training efficiency. Specific steps include: S201. Obtain the sparse point cloud of the scene by jointly using global SfM, such as Figure 2 As shown in (a), the normal vector of the ground plane is estimated using the Manhattan hypothesis, aligned to the z-axis, and the horizontal structure of the scene is projected onto the xy plane for subsequent region division.

[0021] S202. Design an adaptive partitioning algorithm based on point cloud density. For example... Figure 2 As shown in (b), the rectangular target region Bs is first manually defined and set as the initial root node for recursive binary search. Given the maximum depth M of the binary tree and the threshold number of leaf nodes... Perform the following operations on a node of any depth d: Calculate the number of sparse points falling into this node. ; If d ≤ M and > Then along the longer side of the rectangular region will Divided into two sub-regions and Then repeat the above process for each sub-region; otherwise, keep it as a leaf node.

[0022] By recursively subdividing each region based on a point density threshold, we achieved load balancing across computational blocks, thereby significantly reducing the training time for the entire scene in a multi-GPU environment.

[0023] Step S3: For each block, view allocation is performed based on point visibility, and an auxiliary point strategy is introduced to eliminate block boundary artifacts, thereby promoting seamless merging of blocks. Specific steps include: S301, Adopt a view assignment strategy based on point visibility, such as Figure 2 As shown in (c). To ensure that each block only receives supervised views that significantly contribute to its geometry and texture, we perform a correlation assessment between views and blocks. Let... This represents the number of visible points in the training view V throughout the entire scene. This indicates the number of visible points of view V within block b. If If the value is greater than the threshold of 0.3, then the view is selected as the supervisory view for the current block.

[0024] S302, Design auxiliary point strategies, such as Figure 2 As shown in (d), during training, the training view typically captures scene content beyond the block space. Artifacts often occur in the outer regions of these blocks due to the lack of sparse point initialization. To address this, we introduce a set of auxiliary points for each block. These auxiliary points are extracted from the sparse point cloud corresponding to the block's supervisory view, but their extent extends beyond the block boundaries. This strategy provides correct geometric initialization for the entire field of view, effectively eliminating artifacts that could affect visual quality. After optimization, these auxiliary Gaussians are discarded; otherwise, they might appear as floating objects in other blocks. Furthermore, since adjacent blocks share the same training view in our partitioning scheme, seamless merging of blocks is achieved.

[0025] Step S4: Use orthogonal rasterization rendering technology to directly generate high-precision digital orthophotos. Specific steps include: S401, such as Figure 3 As shown, Gaussian sputtering rasterization rendering is improved by replacing standard perspective projection with orthographic projection to generate orthographic images. The orthographic projection matrix used is: ; in ; ; near and far represent the near clipping plane and the far clipping plane, respectively. θx and θy represent the horizontal and vertical fields of view, respectively. top, bottom, left, and right represent the upper, lower, left, and right boundaries of the view frustum, respectively.

[0026] S402. During the projection process, the mean and covariance Σ undergo different transformations. The mean represents the point coordinates and is directly projected. The covariance matrix Σ, however, defines the spatial distribution. To accurately calculate the transformed shape... Use the following formula: ; Where W is the view transformation matrix, J is the Jacobian matrix, and W T and J T This represents the corresponding transpose matrix.

[0027] When perspective projection is replaced by orthographic projection, the originally complex and depth-dependent Jacobian matrix becomes a simple constant matrix J.

[0028] ; S403: By replacing perspective projection with orthographic projection in the 3D Gaussian sputtering rendering process and setting a uniform height and viewpoint, all Gaussian ellipsoids are projected onto a 2D plane and their opacity is blended, thereby directly generating a high-quality orthographic image. The generated orthographic image is as follows: Figure 4 As shown.

[0029] Finally, it should be noted that the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments or make equivalent substitutions for some of the technical features. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for generating orthophotos based on load-balanced 3D Gaussian sputtering, characterized in that: The specific steps are as follows: S1 acquires aerial images of the target scene and quickly generates camera pose and sparse point cloud by combining global SfM, providing geometric initialization for 3D Gaussian sputtering; S2 divides the scenario into multiple sub-blocks for computational load balancing: S21. The sparse point cloud of the scene is obtained by joint global SfM, the normal vector of the ground plane is estimated by the Manhattan hypothesis, it is aligned to the z-axis, and the horizontal structure of the scene is projected onto the xy plane for subsequent region division. S22, first manually draw a rectangular target region Bs, and set it as the initial root node for recursive bisection; given the maximum depth M of the binary tree and the point threshold N of the leaf node th perform the following operations on any node with depth d: Compute the number of sparse points N falling into this node b ; If d ≤ M and N b >N th Then B will be along the longer side of the rectangular region. s Divide into two sub-regions B s1 and B s2 Then repeat the above process for each sub-region; otherwise, retain it as a leaf node. S3 assigns views based on point visibility for each sub-block and introduces an auxiliary point strategy to eliminate block boundary artifacts, promoting seamless merging of blocks. S4 uses orthogonal rasterization rendering technology to generate high-precision digital orthophotos.

2. The method for generating orthophotos based on load-balanced three-dimensional Gaussian sputtering according to claim 1, characterized in that: Specifically, S1 is: S101. Use drones to acquire aerial images of the target area, then extract and match features from the acquired drone images. Specifically, RootSIFT features are used for feature extraction and description, and Bag-of-Words image retrieval technology is combined to filter image pairs to be matched. Finally, feature matching is completed through a brute-force matching algorithm. S102. Perform global rotation averaging, minimizing the relative rotation error while simultaneously solving for the global rotation matrix of all cameras. ={Ri}; To mitigate the impact of noise and outliers, a robust global optimization formula is adopted: ； Where R ij Indicates relative rotation; Represents the cycle consistency error term, where I is the identity matrix; The rotation group is defined. Geodetic distance on; It is a robust loss function, using the L1 norm; S103. Using the normalized orientation difference as the error metric, simultaneously estimate the centers of all cameras. and three-dimensional points The objective function to be optimized is: ； in Indicates from camera Observation point The global rotation of the camera's light; and The normalization factor is initialized to 1, and the weights of cameras involving unknown intrinsic parameters are reduced by a factor of 2 to mitigate their impact; Huber is used as the robust function. Levenberg-Marquardt was used as the optimizer. S104. To further optimize, multiple rounds of global bundle adjustment were performed. In each round, the camera rotation was first fixed, and then joint optimization was performed together with intrinsic parameters and points. Then, the trajectory was filtered according to the reprojection error in the image space. The iteration stopped when the proportion of the filtered trajectory was less than 0.1%.

3. The method for generating orthophotos based on load-balanced three-dimensional Gaussian sputtering according to claim 1, characterized in that: S3 specifically includes: S301. A view allocation strategy based on point visibility is adopted. To ensure that each block only receives supervised views that significantly contribute to its geometry and texture, the correlation between the view and the block is evaluated. This represents the number of visible points in the training view V throughout the entire scene. This indicates the number of visible points of view V within block b; if If the value is greater than the threshold of 0.3, then the view is selected as the supervisory view for the current block; S302. Design an auxiliary point strategy to introduce a set of auxiliary points for each block. These auxiliary points are extracted from the sparse point cloud corresponding to the block's supervised view, but their range extends beyond the block boundary. This provides correct geometric initialization for the entire field of view and effectively eliminates artifacts that may affect visual quality. After optimization, these auxiliary Gaussians will be discarded. In addition, since adjacent blocks share the same training view in our partitioning scheme, seamless fusion of blocks can be achieved.

4. The method for generating orthophotos based on load-balanced three-dimensional Gaussian sputtering according to claim 1, characterized in that: S4 specifically includes: S401. Gaussian sputtering rasterization rendering is improved by replacing standard perspective projection with orthographic projection to generate orthographic images; the orthographic projection matrix used is: ； in: ； ； near and far represent the near clipping plane and the far clipping plane, respectively; θx and θy represent the horizontal and vertical fields of view, respectively; top, bottom, left, and right represent the upper, lower, left, and right boundaries of the view frustum, respectively. S402. During the projection process, the mean and covariance Σ undergo different transformations; the mean represents the point coordinates and is directly projected; while the covariance matrix Σ defines the spatial distribution; in order to accurately calculate the transformed shape... Use the following formula: ； Where W is the view transformation matrix, J is the Jacobian matrix, and W T and J T This represents the corresponding transpose matrix; When perspective projection is replaced by orthographic projection, the originally complex and depth-dependent Jacobian matrix becomes a simple constant matrix J. ； S403: By replacing the perspective projection in the 3D Gaussian sputtering rendering process with orthographic projection and setting a uniform height and viewpoint, all Gaussian ellipsoids are projected onto a 2D plane and their opacity is blended, thereby directly generating a high-quality orthographic image.

Images