A large scene parallel processing method based on 3D Gaussian

Abstract

The present invention provides a large scene parallel processing method based on 3D Gauss, including data preprocessing, data partitioning; training sub-regions; processing the trained sub-regions; performing interactive rendering, determining the intersecting bounding box and the appropriate LoD level according to the position and angle of the current viewpoint, and the rendering scheduler is responsible for taking out the corresponding three-dimensional Gauss data, moving it to the video memory, handing it over to CUDA for parallel processing, and inferring pictures at different angles. A large scene parallel processing method based on 3D Gauss proposed by the present invention, based on LoD, improves the training algorithm and rendering technology, and realizes 3DGS efficient modeling and interactive rendering of large scenes.

Description

Technical Field

[0001] The invention relates to the technical field of three-dimensional scene modeling, in particular to a large scene parallel processing method based on 3D Gaussian. Background Art

[0002] With the rise of smart city construction, large-scale 3D modeling has wide application value in many fields, including city roaming, autonomous driving, and navigation. Traditional 3D modeling methods, such as oblique photography modeling, use display methods to express scenes and describe scene information through elements such as geometry, texture, and material. This method has a long modeling process, generates a large amount of data, and often requires a lot of manual editing work to achieve high fidelity, resulting in low data production efficiency.

[0003] 3DGS is a new breakthrough in the field of multi-view synthesis, which represents the geometry and scenes in 3D space as a large number of Gaussian distributions, and then uses the parallel computing power of GPU to project and render pixel colors of these Gaussian distributions to generate realistic 2D images. There are many problems in large scenes.

[0004] 1) The partition training algorithm needs to use the point cloud results of colmap matching, but for large scenes with hundreds of thousands of photos, the matching speed is slow and the matching program may crash, resulting in the inability to provide the initial point cloud;

[0005] 2) The original 3DGS algorithm stores the three-dimensional Gaussian data in the video memory at the same time. However, for large scenes with tens of millions of three-dimensional Gaussians, the RAM or video memory cannot accommodate all the data at the same time.

[0006] 3) LoD level switching is a fixed value and is not universal;

[0007] 4) A large number of 3DGS are used to achieve high-quality rendering, and the huge number of 3DGS requires large video memory and large memory, which causes the rendering cost to soar sharply.

[0008] The 2024 IEEE / CVF Conference on Computer Vision and Pattern Recognition published an article by J Lin, Z Li, XTang, J Liu, S Liu, J Liu, Y Lu, X Wu, S Xu, Y Yan titled "Pan-Gaussian: A Pan-3D Gaussian Model for Large Scene Reconstruction", which published: Existing NeRF-based large scene reconstruction methods often have limitations in terms of visual quality and rendering speed. While recent 3D Gaussian speckles perform well in small-scale and object-centric scenes, scaling them up to large scenes poses challenges due to limited video memory, long optimization time, and significant appearance variations. To address these challenges, we present VastGaussian, the first 3D Gaussian-based approach for high-quality reconstruction and real-time rendering of large scenes. We propose a progressive partitioning strategy to partition a large scene into multiple cells, where training cameras and point clouds are appropriately distributed according to a spatially aware visibility criterion. These cells are merged into a complete scene after parallel optimization. We also introduce decoupled appearance modeling into the optimization process to reduce appearance variations in the rendered image. Our method outperforms existing NeRF-based methods and achieves state-of-the-art results on multiple large scene datasets, enabling fast optimization and high-fidelity real-time rendering. The disadvantage of this method is that the partition training algorithm requires the use of colmap matched point cloud results, but for large scenes with hundreds of thousands of photos, the matching speed is slow and the matching program may crash, resulting in the inability to provide an initial point cloud.

[0009] The 2024 JACM Journal of Graphics (TOG) published an article titled "Hierarchical 3D Gaussian Representations for Real-time Rendering of Very Large Datasets" by Kerbl Bernhard, Meuleman Andreas, Kopanas Georgios, Wimmer Michael, Lanvin Alexandre, and Drettakis George, which stated: In recent years, new view synthesis has made significant progress, and 3D Gaussians provide excellent visual quality, fast training, and real-time rendering. However, the resources required for training and rendering inevitably limit the size of captured scenes that can be represented with good visual quality. We introduce a 3D Gaussian hierarchy that preserves visual quality for very large scenes while providing an efficient LOD solution that enables efficient rendering of distant content through effective level selection and smooth transitions between levels. The disadvantage of this method is that the 3D Gaussian data is stored in RAM at the same time, and the data is retrieved to the video memory as needed; however, for large scenes with tens of millions of 3D Gaussians, RAM or video memory cannot accommodate all the data at the same time. The algorithm is developed for street view data and is not suitable for large scenes, especially those shot by drones. Summary of the invention

[0010] In order to solve the above technical problems, the present invention proposes a large scene parallel processing method based on 3D Gaussian, based on LoD, improved training algorithm and rendering technology, to achieve large scene modeling and interactive rendering.

[0011] The purpose of the present invention is to provide a large scene parallel processing method based on 3D Gaussian, including data preprocessing, and further comprising the following steps:

[0012] Step 1: partition data;

[0013] Step 2: Train the sub-region;

[0014] Step 3: Process the trained sub-regions;

[0015] Step 4: Perform interactive rendering. According to the position and angle of the current viewpoint, determine the intersecting bounding box and the appropriate LoD level. The rendering scheduler is responsible for fetching the corresponding 3D Gaussian data, moving it to the video memory, and handing it over to CUDA for parallel processing to infer images from different angles.

[0016] Preferably, the data preprocessing includes downsampling the photo and estimating camera intrinsic and extrinsic parameters.

[0017] In any of the above solutions, preferably, step 1 includes partitioning the data according to the location of the camera, and expanding the partition range to N %, N >100.

[0018] In any of the above schemes, preferably, step 2 includes using a random point cloud as the first input, performing multiple self-learning of the point cloud, and performing formal training after obtaining a good initial point cloud.

[0019] In any of the above solutions, preferably, in step 3, the processing method includes LoD construction, clipping and calculation of bounding box information.

[0020] In any of the above schemes, preferably, step 4 includes the following sub-steps:

[0021] Step 41: Use the intersection algorithm to quickly determine whether the view cone intersects with the bounding box based on the CUDA code;

[0022] Step 42: When the view cone intersects the bounding box, the corresponding LoD level is selected according to the distance from the center point of the object to the camera or the percentage of the intersecting bounding box;

[0023] Step 43: The rendering scheduler loads the specified LoD level data from the hard disk according to the intersection result and the LoD level;

[0024] Step 44: Project the 3D Gaussian distribution onto the 2D image plane using the internal and external parameters of the camera, use the 𝛂 blending technique to render the image, correctly blend the colors and transparency of multiple Gaussian distributions on the 2D plane, and use the GPU to calculate the color of each pixel in parallel to synthesize a new perspective image;

[0025] Step 45: Determine whether all the three-dimensional Gaussian data have been processed. If all the three-dimensional Gaussian data have been processed, execute step 47;

[0026] Step 46: Determine whether the view cone moves. If the view cone moves, re-execute step 41; if the view cone does not move, re-execute step 44;

[0027] Step 47: Output images from different angles.

[0028] In any of the above solutions, preferably, step 41 includes loading the data of the viewing cone and the bounding box onto the GPU, utilizing the parallel computing characteristics of CUDA to allocate a thread to each bounding box for processing, and each thread executes a separating axis algorithm to determine whether the bounding box intersects the viewing cone;

[0029] If the bounding box intersects with the viewing cone, each 3DGS point in the bounding box is deemed to intersect with the viewing cone, that is, all 3DGS points participate in rendering; if the bounding box does not intersect with the viewing cone, all 3DGS points in the bounding box are deemed to not intersect with the viewing cone, that is, all 3DGS points do not participate in rendering.

[0030] Preferably, in any of the above schemes, step 41 also includes storing the result in a result array after each thread completes the intersection test, transmitting the result array back to the CPU, making a rough estimate through fast calculation, eliminating non-intersecting partitions, and only rendering the partitions that intersect with the viewing cone.

[0031] In any of the above solutions, preferably, step 42 includes setting the following intersection ratio thresholds a0%, a1%, ..., a according to the LoD principle: max-1 %,a max %, where a0 <a1<…<a max-1 max ;

[0032] With the z-axis as the rotation axis, a vertical viewing cone is constructed. The initial vertex of the viewing cone is (0,0,0), the sight vector is (0,0,-1), and the viewing cone moves along the positive direction of the z-axis at specified intervals.

[0033] ​When the ratio of the bounding box that intersects the viewing cone is equal to a0%, get the height at this time, that is, the z coordinate value h0;

[0034] When the ratio of the bounding box that intersects the viewing cone is equal to a1%, obtain the height at this time, that is, the z coordinate value h1;

[0035] And so on;

[0036] When the frustum intersects the bounding box, the value of the bounding box is equal to a max %, get the height at this time, that is, the z coordinate value hmax;

[0037] Among them, LoD0 is the most detailed data of the scene, LoD1 is the relatively detailed data of the scene, and LoD max+1 is the coarsest data of the scene, h0, h1, ..., hmax is the ratio of the bounding box that intersects with the view cone, equal to a0%, a1%, ..., a max % The z coordinate value at that time is the height of the viewing cone.

[0038] In any of the above schemes, preferably, step 42 also includes calculating the coordinates of the intersection of the camera and the xy plane, connecting the camera and the intersection to form a vector v, and calculating the angle θ between the z axis and the vector v. If the angle θ is greater than or equal to 0° and less than or equal to 45°, it is case 1; if the angle θ is greater than 45° and less than or equal to 90°, it is case 2; if the angle θ is greater than 90°, it is case 3.

[0039] In any of the above solutions, preferably, for case 1, when the proportion of the bounding box intersecting with the viewing cone is less than a0%, the LoD0 level data is called;

[0040] When the ratio of the bounding box that intersects with the view cone is ≥ a0% and < a1%, the LoD1 level data is called;

[0041] Similarly, when the ratio of the bounding box where the frustum intersects is ≥ a max-1 % and < a max % when calling LoD max Hierarchical data;

[0042] When the ratio of the bounding box of the intersecting frustum is ≥ a max % when calling LoD max+1 Hierarchical data.

[0043] In any of the above solutions, for case 2, the maximum xmax and minimum xmin of the x coordinate of the center point of the bounding box currently involved in rendering, as well as the maximum ymax and minimum ymin of the y coordinate are taken, and then the following circles are divided:

[0044] First layer: the upper limit of the x coordinate is xmax+h0, and the lower limit is xmax-h0; the upper limit of the y range is ymax+h0, and the lower limit is ymax-h0, that is, the x coordinate range is [xmin-h0,xmax+h0], and the y coordinate range is [ymin-h0,ymax+h0], and the same below;

[0045] Second layer: the x-coordinate range is [xmin-h1, xmin-h0] and [xmin+h0, xmax+h1], and the y-coordinate range is [ymin-h1, ymin-h0] and [ymax+h0, ymax+h1];

[0046] And so on;

[0047] The second to last layer: the x coordinate range is [xmin-hmax, xmin-h max-1 ] and [xmin+h max-1 ,xmax+hmax], the y coordinate range is [ymin-hmax,ymin-h max-1 ] and [ymin+h max-1 ,ymax+hmax];

[0048] The last layer: the x-coordinate range is [-∞,xmin-hmax] and [xmax+hmax,+∞], and the y-coordinate range is [-∞,ymin-hmax] and [ymax+hmax,+∞].

[0049] In any of the above schemes, it is preferred that the number of layers to which the view cone vertex belongs is calculated. When the coordinates of the view cone vertex belong to the first layer, the distance between the bounding box currently involved in rendering and the view cone vertex is calculated and sorted in ascending order; according to the LoD principle: take the first a% bounding boxes of the sorted data and call LoD0; continue to take b% bounding boxes and call LoD1; and so on, take the last z% bounding boxes and call LoD max+1 ;

[0050] When the coordinates of the pyramid vertices belong to the second layer, all LoD1 layer data are called;

[0051] And so on;

[0052] When the coordinates of the pyramid vertices belong to the second-to-last layer, all LoDs are called maxLayer data.

[0053] When the coordinates of the pyramid vertices belong to the last layer, all LoDs are called max+1 Layer data.

[0054] In any of the above solutions, it is preferred that, for case 3, LoD is directly called max+1 Layer data.

[0055] The present invention proposes a large scene parallel processing method based on 3D Gaussian, which aims at high-definition image data collected by UAV, takes into account the technical advantages of 3DGS (high fidelity, arbitrary perspective synthesis, and saving labor costs), and realizes large scene modeling and interactive browsing.

[0056] 3DGS: 3D Gaussian or three-dimensional Gaussian, both refer to three-dimensional Gaussian splashing technology. By inputting photos and point cloud data, a three-dimensional model is generated, and images from various perspectives can be rendered.

[0057] LoD: Level of detail technology optimizes rendering efficiency by simplifying model details in stages and dynamically loading geometry or texture data of different precisions according to the distance to the scene.

[0058] Bounding box: A bounding box is used to describe the three-dimensional boundaries of an object. It is generally the smallest rectangular box and is used for collision detection, acceleration calculations, etc.

[0059] GPU: Graphics Processing Unit, designed for parallel computing, good at parallel computing tasks of large amounts of data.

[0060] CUDA: A general-purpose parallel computing architecture launched by NVIDIA that allows developers to efficiently run general computing tasks on GPUs.

[0061] View cone: A geometric body that describes the field of view of a camera. It is pyramid-shaped and represents the area that the camera can see. It consists of a near clipping plane, a far clipping plane, four side faces, and a vertex. The camera is located at the vertex of the view cone. BRIEF DESCRIPTION OF THE DRAWINGS

[0062] Figure 1 The flowchart is a preferred embodiment of the 3D Gaussian-based large scene parallel processing method according to the present invention.

[0063] Figure 2 The flowchart is another preferred embodiment of the 3D Gaussian-based large scene parallel processing method according to the present invention.

[0064] Figure 3 It is a schematic diagram of an embodiment of the training data partitioning of the 3D Gaussian-based large scene parallel processing method according to the present invention.

[0065] Figure 4The flowchart is a flowchart of an embodiment of a separation axis algorithm of a large scene parallel processing method based on 3D Gaussian according to the present invention.

[0066] Figure 5 It is a schematic diagram of an embodiment of the intersection determination method according to the 3D Gaussian-based large scene parallel processing method of the present invention.

[0067] Figure 6 It is a schematic diagram of an embodiment of calculating the LoD switching threshold according to the 3D Gaussian-based large scene parallel processing method of the present invention.

[0068] Figure 7 Schematic diagram of the intersection of a camera and an xy plane according to an embodiment of the 3D Gaussian-based large scene parallel processing method of the present invention.

[0069] Figure 8 Schematic diagram of the angle between the z-axis and the vector v according to a preferred embodiment of the 3D Gaussian-based large scene parallel processing method of the present invention.

[0070] Fig. 9 FIG. 1 is a schematic diagram of an embodiment of calling the LoD level according to case 1 of the 3D Gaussian-based large scene parallel processing method of the present invention.

[0071] Fig.10 It is a schematic diagram of an embodiment of the circle layer division of the large scene parallel processing method based on 3D Gaussian according to the present invention.

[0072] Fig.11 It is a schematic diagram of an embodiment of calling LoD in the first layer of the 3D Gaussian-based large scene parallel processing method according to the present invention. DETAILED DESCRIPTION

[0073] The present invention is further described below in conjunction with the accompanying drawings and specific embodiments.

[0074] Example 1

[0075] like Figure 1 As shown, step 100 is executed to perform data preprocessing, wherein the data preprocessing includes downsampling the photo, estimating the camera intrinsic parameters and extrinsic parameters, etc.

[0076] Execute step 110 to partition the data according to the location of the camera, and expand the partition range to N %, N >100.

[0077] Execute step 120 to train the sub-region. A random point cloud is used as the first input. The point cloud is self-learned multiple times. After a good initial point cloud is obtained, formal training is performed.

[0078] Execute step 130 to process the trained sub-regions, wherein the processing method includes LoD construction, cutting, and calculation of bounding box information.

[0079] Execute step 140 to perform interactive rendering. According to the position and angle of the current viewpoint, determine the intersection bounding box and the appropriate LoD level. The rendering scheduler is responsible for taking out the corresponding three-dimensional Gaussian data, moving it to the video memory, and handing it over to CUDA for parallel processing to infer pictures at different angles, including the following sub-steps:

[0080] Execute step 141, use the intersection algorithm to quickly determine whether the view cone intersects with the bounding box based on the CUDA code, load the data of the view cone and the bounding box onto the GPU, use the parallel computing feature of CUDA to assign a thread to each bounding box for processing, and each thread executes the separating axis algorithm to determine whether the bounding box intersects with the view cone;

[0081] For a single partition, if the bounding box intersects with the viewing cone, each 3DGS point in the bounding box is deemed to intersect with the viewing cone, that is, the intersection with the viewing cone participates in rendering; if the bounding box does not intersect with the viewing cone, all 3DGS points in the bounding box are deemed to not intersect with the viewing cone, that is, all 3DGS points do not participate in rendering.

[0082] After each thread completes the intersection test, it stores the result in a result array and transfers the result array back to the CPU. A rough estimate is made by fast calculation, and non-intersecting partitions are eliminated, and only the partitions that intersect with the view frustum are rendered.

[0083] Execute step 142. When the view cone intersects the bounding box, select the corresponding LoD level according to the distance from the center point of the object to the camera or the percentage of the bounding box that intersects. Set the following intersection ratio thresholds a0%, a1%, ..., a according to the LoD principle. max-1 %,a max %, where a0 <a1<…<a max-1 max ;

[0084] With the z-axis as the rotation axis, a vertical viewing cone is constructed. The initial vertex of the viewing cone is (0,0,0), the sight vector is (0,0,-1), and the viewing cone moves along the positive direction of the z-axis at specified intervals.

[0085] When the ratio of the bounding box that intersects the viewing cone is equal to a0%, get the height at this time, that is, the z coordinate value h0;

[0086] ​When the ratio of the bounding box that intersects the viewing cone is equal to a1%, obtain the height at this time, that is, the z coordinate value h1;

[0087] And so on;

[0088] When the frustum intersects the bounding box, the value of the bounding box is equal to a max %, get the height at this time, that is, the z coordinate value hmax;

[0089] Among them, LoD0 is the most detailed data of the scene, LoD1 is the relatively detailed data of the scene, and LoD max+1 is the coarsest data of the scene, h0, h1, ..., hmax is the ratio of the bounding box that intersects with the view cone, equal to a0%, a1%, ..., a max % The z coordinate value at that time is the height of the viewing cone.

[0090] Calculate the coordinates of the intersection of the camera and the xy plane, connect the camera and the intersection to form a vector v, calculate the angle θ between the z axis and the vector v, if the angle θ is greater than or equal to 0° and less than or equal to 45°, it is case 1; if the angle θ is greater than 45° and less than or equal to 90°, it is case 2; if the angle θ is greater than 90°, it is case 3.

[0091] 1) For case 1, when the ratio of the bounding box that intersects with the view cone is less than a0%, the LoD0 level data is called;

[0092] When the ratio of the bounding box that intersects with the view cone is ≥ a0% and < a1%, the LoD1 level data is called;

[0093] Similarly, when the ratio of the bounding box where the frustum intersects is ≥ a max-1 % and < a max % when calling LoD max Hierarchical data;

[0094] When the ratio of the bounding box of the intersecting frustum is ≥ a max % when calling LoD max+1 Hierarchical data.

[0095] 2) For case 2, take the maximum x-coordinate value xmax and the minimum xmin of the center point of the bounding box currently involved in rendering, as well as the maximum ymax and the minimum ymin of the y-coordinate, and then divide it into the following circles:

[0096] First layer: the upper limit of the x coordinate is xmax+h0, and the lower limit is xmax-h0; the upper limit of the y range is ymax+h0, and the lower limit is ymax-h0, that is, the x coordinate range is [xmin-h0,xmax+h0], and the y coordinate range is [ymin-h0,ymax+h0], and the same below;

[0097] Second layer: the x-coordinate range is [xmin-h1, xmin-h0] and [xmin+h0, xmax+h1], and the y-coordinate range is [ymin-h1, ymin-h0] and [ymax+h0, ymax+h1];

[0098] And so on;

[0099] The second to last layer: the x coordinate range is [xmin-hmax, xmin-h max-1 ] and [xmin+h max-1 ,xmax+hmax], the y coordinate range is [ymin-hmax,ymin-h max-1 ] and [ymin+h max-1 ,ymax+hmax];

[0100] The last layer: the x-coordinate range is [-∞,xmin-hmax] and [xmax+hmax,+∞], and the y-coordinate range is [-∞,ymin-hmax] and [ymax+hmax,+∞].

[0101] Calculate the circle layer to which the view cone vertex belongs. When the coordinates of the view cone vertex belong to the first layer, calculate the distance between the bounding box currently involved in rendering and the pyramid vertex, and sort them in ascending order; according to the LoD principle: take the first a% of the sorted data and call LoD0; continue to take b% of the bounding box and call LoD1; and so on, take the last z% of the bounding box and call LoD max+1 ;

[0102] When the coordinates of the pyramid vertices belong to the second layer, all LoD1 layer data are called;

[0103] And so on;

[0104] When the coordinates of the pyramid vertices belong to the second-to-last layer, all LoDs are called max Layer data.

[0105] When the coordinates of the pyramid vertices belong to the last layer, all LoDs are called max+1 Layer data.

[0106] 3) For case 3, call LoD max+1 Layer data.

[0107] Execute step 143 , the rendering scheduler loads the specified LoD level data from the hard disk according to the intersection result and the LoD level.

[0108] Execute step 144, project the 3D Gaussian distribution onto the 2D image plane through the internal and external parameters of the camera, use the 𝛂 blending technique to render the image, correctly mix the colors and transparency of multiple Gaussian distributions on the 2D plane, and use the GPU to calculate the color of each pixel in parallel to synthesize a new perspective image.

[0109] Execute step 145 to determine whether all the three-dimensional Gaussian data has been processed. If all the three-dimensional Gaussian data has been processed, execute step 147.

[0110] Execute step 146 to determine whether the viewing cone moves. If the viewing cone moves, re-execute step 141; if the viewing cone does not move, re-execute step 144.

[0111] Execute step 147: output pictures at different angles.

[0112] Example 2

[0113] In order to solve the problems existing in the prior art, the present invention proposes a large scene parallel processing method based on 3D Gaussian, based on LoD, improves the training algorithm and rendering technology, and realizes efficient 3DGS modeling and interactive rendering of large scenes:

[0114] 1) A 3D Gaussian scene reconstruction method based on self-learning can provide a good initial point cloud through multiple self-learning of point clouds;

[0115] 2) The 3DGS involved in rendering is transferred from the hard disk to the video memory, and the 3DGS points not rendered do not occupy the video memory, and no unnecessary RAM and video memory consumption is generated;

[0116] 3) The threshold of level switching changes automatically according to the scenario and is universal;

[0117] 4) Large scene efficient rendering algorithm: partition the 3DGS and create a Bounding Box, write efficient intersection determination code based on CUDA, and realize real-time judgment of whether the view cone intersects with each Bounding Box, quickly determine the 3DGS partitions involved in rendering, exclude the 3DGS partitions that do not participate in rendering, and only store the 3DGS partitions involved in rendering in the video memory and render to generate new perspective images;

[0118] 5) Use the LightGaussian compression algorithm developed by the University of Texas at Austin to achieve lightweight 3DGS points at small scales, making disk space usage, rendering efficiency and effects controllable.

[0119] like Figure 2 As shown, the technical solution of the present invention is as follows:

[0120] Step 1: Data Preprocessing

[0121] Data preprocessing includes downsampling photos (according to actual hardware equipment and rendering requirements), estimating camera intrinsic and extrinsic parameters, etc.

[0122] Step 2: Data Partitioning

[0123] Data is partitioned according to the location of the camera, and the partition range is reasonably expanded according to the actual hardware equipment, such as expanding it by 10% to ensure the modeling quality of the central area. Figure 3 shown.

[0124] Step 3: Sub-region training

[0125] In the 3D Gaussian reconstruction task, the quality of the initial point cloud has a significant impact on the training effect and final performance of the model. This paper proposes a strategy based on multiple self-learning of point clouds, which optimizes the initial point cloud multiple times to make it more consistent with the geometric characteristics of the target scene, thereby improving the training efficiency and reconstruction effect.

[0126] First, perform random point cloud initialization. At the beginning of training, use a randomly distributed point cloud as the initial input. Random point clouds are uniformly distributed, do not contain local details or global structures of the scene, and lack scene geometric characteristics. Therefore, they have limited effect in the early stages of model training, but provide a basis for multiple self-learning.

[0127] Then, perform multiple self-learning of the point cloud. Export the training results in a loop, generate the scene point cloud, and input the algorithm to continue training. Repeat the self-learning process of the point cloud many times to gradually approach a good initial state of the point cloud. After the self-learning is completed, the obtained point cloud has higher geometric accuracy and adaptability to the target scene.

[0128] Finally, formal training is performed. The point cloud optimized through self-learning is used as the initial input of the model for formal training. Due to the high quality of the point cloud, the model can learn the geometric characteristics of the scene more efficiently, thereby improving the training effect.

[0129] Experiments show that the SSIM index is improved by 6.75%, the PSNR index is improved by 1.19dB, and the LPIPS index is reduced by 18.68% (the larger the SSIM, the larger the PSNR, and the smaller the LPIPS, the better the modeling effect).

[0130] Step 4: Post-training processing

[0131] LoD construction (reference 5 for this step: LightGaussian): Use the LightGaussian compression algorithm developed by the University of Texas at Austin to delete redundant Gaussians and construct LOD0, LOD1, …, LOD max ,LOD max+1 , the max value is set by the user, which reduces the number of over-parameterized points while retaining the original accuracy.

[0132] Cropping: To ensure the consistency of the scene and avoid interference between overlapping parts of the boundaries in step 2, the extended parts should be cropped to keep only the well-modeled central area.

[0133] Calculate bounding box information: Based on the partition information, calculate the maximum and minimum values ​​of the xyz coordinates of the 3DGS points in the partition, which is used to quickly determine the intersection of the view frustum and the bounding box during rendering.

[0134] Step 5: Interactive Rendering

[0135] During the rendering process, the intersecting bounding box and the appropriate LoD level are determined based on the position and angle of the current viewpoint. The rendering scheduler is responsible for fetching the corresponding 3D Gaussian data, moving it to the video memory, and handing it over to CUDA for parallel processing, and finally inferring images from different angles.

[0136] The details are as follows:

[0137] Step 5.1 Determine intersection

[0138] Using the intersection algorithm, based on CUDA code, quickly determine which bounding boxes the view cone intersects with:

[0139] The data of the view cone and bounding box are loaded onto the GPU. Using the parallel computing feature of CUDA, a thread is assigned to each bounding box for processing. Each thread executes the separating axis algorithm to determine whether the bounding box intersects with the view cone, such as Figure 4 Shown is the Separating Axis Algorithm Flowchart.

[0140] For a single partition, if the bounding box intersects with the viewing cone, each 3DGS point in the bounding box is deemed to intersect with the viewing cone, that is, the intersection with the viewing cone participates in rendering; if the bounding box does not intersect with the viewing cone, all 3DGS points in the bounding box are deemed to not intersect with the viewing cone, that is, all 3DGS points do not participate in rendering.

[0141] After each thread completes the intersection test, it stores the result in a result array and transmits the result array back to the CPU. Through fast calculation, a rough estimate is made, and non-intersecting partitions are eliminated (3DGS points in non-intersecting partitions will not participate in rendering), and only the partitions that intersect with the view cone are rendered (3DGS points in intersecting partitions will participate in rendering), such as Figure 5 What is shown is a schematic diagram of determining intersection.

[0142] Step 5.2 Determine the LoD level

[0143] If the view frustum does not intersect the bounding box, nothing is done, that is, the subsequent rendering steps are skipped, thus saving computing resources.

[0144] If the view cone intersects the bounding box, the corresponding LoD level is selected based on the distance from the center point of the object to the camera or the percentage of the intersecting bounding box. The process is as follows:

[0145] According to the LoD principle: fine display when close to the scene, rough display when far away from the scene, so if the view cone is close to the scene, the number of intersecting bounding boxes is small and the ratio is small, if the view cone is far away from the scene, the number of intersecting bounding boxes is large and the ratio is large, so the following intersection ratio thresholds are set: a0%, a1%, ..., a max-1 %,a max %, where a0 <a1<…<a max-1 max (This result will be used for case 1 of step 5.2. When the ratio of the bounding box that intersects with the viewing cone is less than a0%, the LoD0 level data is called; when the ratio of the bounding box that intersects with the viewing cone is ≥ a0% and less than a1%, the LoD1 level data is called; and so on. max-1 % and < a max % when calling LoD max Hierarchical data; when the ratio of the bounding box of the cone intersection is ≥ a max % when calling LoD max+1 Hierarchical data.

[0146] ​Then, with the z-axis as the rotation axis, a vertical cone is constructed, with the initial vertex of the cone being (0,0,0) and the sight vector being (0,0,-1). Subsequently, the cone moves along the positive direction of the z-axis at a specified interval. When the ratio of the bounding box that intersects with the cone is equal to a0%, the height at that time, i.e., the z-coordinate value h0, is obtained; when the ratio of the bounding box that intersects with the cone is equal to a1%, the height at that time, i.e., the z-coordinate value h1, is obtained; and so on. max %, obtain the height at this time, that is, the z coordinate value hmax (this result will be used in case 2 of step 5.2).

[0147] Among them, LoD0 is the most detailed data of the scene, suitable for viewing at close range, LoD1 is the relatively detailed data of the scene, suitable for viewing at a closer distance, and LoD max+1 It is the coarsest data of the scene, suitable for viewing from a distance. h0,h1,…,hmax are the proportions of the bounding box that intersects with the viewing cone, equal to a0%,a1%,…,a max % The z coordinate value at that time is the height of the viewing cone.

[0148] Then, a LoD level switching threshold table is generated, as shown in Table 1.

[0149] Table 1 LoD level switching threshold table

[0150]

[0151] Then, calculate the coordinates of the intersection of the camera and the xy plane, such as Figure 7 shown.

[0152] Connect the camera and the intersection point to form a vector v, calculate the angle θ between the z axis and the vector v, if the angle θ is greater than or equal to 0° and less than or equal to 45°, it is case 1; if the angle θ is greater than 45° and less than or equal to 90°, it is case 2; if the angle θ is greater than 90°, it is case 3, such as Figure 8 shown.

[0153] For case 1, when the ratio of the bounding box that intersects with the viewing cone is less than a0%, the LoD0 layer data is called; when the ratio of the bounding box that intersects with the viewing cone is greater than or equal to a0% and less than a1%, the LoD1 layer data is called; and so on. max-1 % and less than a max % when calling LoD max Layer data; when the ratio of the bounding box of the cone intersection is greater than or equal to a max % when calling LoD max+1Layer data, such as Fig. 9 shown.

[0154] For case 2, take the maximum x-coordinate value xmax and the minimum xmin of the center point of the bounding box currently involved in rendering, as well as the maximum ymax and the minimum ymin of the y-coordinate, and then divide it into the following circles, such as Fig.10 As shown:

[0155] First layer: the upper limit of the x coordinate is xmax+h0, and the lower limit is xmax-h0; the upper limit of the y range is ymax+h0, and the lower limit is ymax-h0, that is, the x coordinate range is [xmin-h0,xmax+h0], and the y coordinate range is [ymin-h0,ymax+h0], and the same below;

[0156] Second layer: the x-coordinate range is [xmin-h1, xmin-h0] and [xmin+h0, xmax+h1], and the y-coordinate range is [ymin-h1, ymin-h0] and [ymax+h0, ymax+h1];

[0157] And so on;

[0158] The second to last layer: the x coordinate range is [xmin-hmax, xmin-h max-1 ] and [xmin+h max-1 ,xmax+hmax], the y coordinate range is [ymin-hmax,ymin-h max-1 ] and [ymin+h max-1 ,ymax+hmax];

[0159] The last layer: the x-coordinate range is [-∞, xmin-hmax] and [xmax+hmax, +∞], and the y-coordinate range is [-∞, ymin-hmax] and [ymax+hmax, +∞];

[0160] Among them, h0,h1,…,hmax are the proportions of the bounding box that intersects with the viewing cone, equal to a0%,a1%,…,a max % The z coordinate value at that time is the height of the viewing cone.

[0161] Calculate the circle layer to which the view cone vertex (camera) belongs: If the coordinates of the view cone vertex (camera) belong to the first layer, calculate the distance between the bounding box currently involved in rendering and the pyramid vertex (camera), and sort them in ascending order. According to the LoD principle: take the first a% bounding boxes of the sorted data and call LoD0; continue to take b% bounding boxes and call LoD1; and so on, take the last z% bounding boxes and call LoDmax+1 ;like Fig.11 shown.

[0162] When the coordinates of the pyramid vertices belong to the second layer, all LoD1 layer data are called;

[0163] And so on;

[0164] When the coordinates of the pyramid vertices belong to the second-to-last layer, all LoDs are called max Layer data.

[0165] When the coordinates of the pyramid vertices belong to the last layer, all LoDs are called max+1 Layer data.

[0166] For case 3, call LoD directly max+1 Layer data.

[0167] Step 5.3 Read data

[0168] The rendering scheduler loads the specified LoD level data from the hard disk based on the intersection results and LoD level, ensuring that only necessary partitions are loaded into the video memory, reducing unnecessary RAM and video memory usage.

[0169] Step 5.4 New view synthesis

[0170] The 3D Gaussian distribution is projected onto the 2D image plane through the internal and external parameters of the camera, and the 𝛂 blending technique is used to render the image. The colors and transparency of multiple Gaussian distributions are correctly mixed on the 2D plane. The GPU calculates the color of each pixel in parallel to synthesize a new perspective image.

[0171] Step 5.5: Determine completion status

[0172] Determine whether all the three-dimensional Gaussian data have been processed. If all the processing has been completed, execute step 5.7;

[0173] Step 5.6: Moving the View Frustum

[0174] If the view cone moves, jump to step 5.1; if the view cone does not move, jump to step 5.4.

[0175] Step 5.7 Output image

[0176] Output pictures from different angles.

[0177] The purpose of the present invention is to introduce 3DGS technology into large scene modeling, especially for high-definition image data collected by drones, taking into account the technical advantages of 3DGS (high fidelity, arbitrary perspective synthesis, saving labor costs, etc.), while realizing interactive browsing of large scenes. LoD is an essential acceleration technology in traditional modeling and three-dimensional scene browsing. Therefore, the present invention adopts horizontal divide-and-conquer and vertical LoD strategies to solve the problems of large scene modeling and interactive rendering.

[0178] 1) Partition training stage: A 3D Gaussian scene reconstruction method based on self-learning, i.e., multiple self-learning of point clouds; through self-learning, a good initial point cloud is obtained to optimize the modeling effect.

[0179] 2) Real-time rendering stage

[0180] a) An efficient filtering method for the current visible area. Through CUDA parallel computing acceleration, rapid judgment is achieved and the partitions involved in rendering are quickly determined;

[0181] b) Real-time scheduling method for rendering data. Read data from the hard disk and store it in the video memory, reducing RAM usage and unnecessary video memory consumption.

[0182] c) Universal method for rendering level decision. The threshold of level switching changes automatically according to the scene, which is universal. According to the specific division of the scene, the algorithm can automatically determine the threshold of each LoD level, which is applicable to various scenes.

[0183] In order to better understand the present invention, the above is described in detail in conjunction with the specific embodiments of the present invention, but it is not intended to limit the present invention. Any simple modifications made to the above embodiments based on the technical essence of the present invention still fall within the scope of the technical solution of the present invention. Each embodiment in this specification focuses on the differences from other embodiments, and the same or similar parts between the embodiments can be referenced to each other. For the system embodiment, since it basically corresponds to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

Claims

1. A large scene parallel processing method based on 3D Gaussian, including data preprocessing, characterized in that: The following steps are also included: Step 1: partition data; Step 2: Train the sub-region; Step 3: Process the trained sub-regions; Step 4: Perform interactive rendering. According to the position and angle of the current viewpoint, determine the intersection bounding box and the appropriate LoD level. The rendering scheduler is responsible for taking out the corresponding 3D Gaussian data, moving it to the video memory, and handing it over to CUDA for parallel processing to infer images from different angles. This includes the following sub-steps: Step 41: Use the intersection algorithm to quickly determine whether the view cone and the bounding box intersect based on the CUDA code; load the data of the view cone and the bounding box onto the GPU, and use the parallel computing feature of CUDA to assign a thread to each bounding box for processing, and each thread executes the separating axis algorithm to determine whether the bounding box intersects with the view cone; If the bounding box intersects with the viewing cone, each 3DGS point in the bounding box is considered to intersect with the viewing cone, that is, all 3DGS points participate in rendering; if the bounding box does not intersect with the viewing cone, all 3DGS points in the bounding box are considered to not intersect with the viewing cone, that is, all 3DGS points do not participate in rendering; Step 42: When the view cone intersects the bounding box, the corresponding LoD level is selected according to the distance from the center point of the object to the camera or the percentage of the intersecting bounding box; Step 43: The rendering scheduler loads the specified LoD level data from the hard disk according to the intersection result and the LoD level; Step 44: Project the 3D Gaussian distribution onto the 2D image plane using the internal and external parameters of the camera, use the 𝛂 blending technique to render the image, correctly blend the colors and transparency of multiple Gaussian distributions on the 2D plane, and use the GPU to calculate the color of each pixel in parallel to synthesize a new perspective image; Step 45: Determine whether all the three-dimensional Gaussian data have been processed. If all the three-dimensional Gaussian data have been processed, execute step 47; Step 46: Determine whether the view cone moves. If the view cone moves, re-execute step 41; if the view cone does not move, re-execute step 44; Step 47: Output images from different angles.

2. The 3D Gaussian large scene parallel processing method according to claim 1, characterized in that: The step 1 includes partitioning the data according to the location of the camera and expanding the partition range to N %, N >100.

3. The 3D Gaussian large scene parallel processing method according to claim 2, characterized in that: The step 2 includes using a random point cloud as the first input, performing multiple self-learning of the point cloud, and performing formal training after obtaining a good initial point cloud.

4. The 3D Gaussian large scene parallel processing method according to claim 3, characterized in that: In step 3, the processing method includes LoD construction, clipping, and calculation of bounding box information.

5. The 3D Gaussian-based large scene parallel processing method according to claim 4, characterized in that: The step 41 also includes storing the result in an array after each thread completes the intersection test, transmitting the result array back to the CPU, performing a rough estimate through fast calculation, eliminating non-intersecting partitions, and only rendering the partitions intersecting with the viewing cone.

6. The 3D Gaussian large scene parallel processing method according to claim 5, characterized in that: The step 42 includes setting the following intersection ratio thresholds a0%, a1%, ..., a1% according to the LoD principle: max-1 %,a max %, where a0 <a1<…<a max-1 max ;​ With the z-axis as the rotation axis, a vertical viewing cone is constructed. The initial vertex of the viewing cone is (0,0,0), the sight vector is (0,0,-1), and the viewing cone moves along the positive direction of the z-axis at specified intervals. When the ratio of the bounding box that intersects the viewing cone is equal to a0%, get the height at this time, that is, the z coordinate value h0; When the ratio of the bounding box that intersects the viewing cone is equal to a1%, obtain the height at this time, that is, the z coordinate value h1; And so on; When the frustum intersects the bounding box, the value of the bounding box is equal to a max %, obtain the height at this time, that is, the z coordinate value hmax; Among them, LoD0 is the most detailed data of the scene, LoD m1 For more detailed data of the scene, LoD max+1 is the coarsest data of the scene, h0, h1, ..., hmax is the ratio of the bounding box that intersects with the view cone, equal to a0%, a1%, ..., a max % The z coordinate value at that time is the height of the viewing cone.

7. The 3D Gaussian-based large scene parallel processing method according to claim 6, characterized in that: The step 42 also includes calculating the coordinates of the intersection of the camera and the xy plane in the three-dimensional coordinate system, connecting the camera and the intersection to form a vector v, and calculating the angle θ between the z axis and the vector v. If the angle θ is greater than or equal to 0° and less than or equal to 45°, it is case 1; if the angle θ is greater than 45° and less than or equal to 90°, it is case 2; if the angle θ is greater than 90°, it is case 3.

8. The 3D Gaussian large scene parallel processing method according to claim 7, characterized in that: For case 1, when the ratio of the bounding box that intersects with the view cone is less than a0%, the LoD0 level data is called; When the ratio of the bounding box that intersects with the view cone is ≥ a0% and < a1%, the LoD1 level data is called; Similarly, when the ratio of the bounding box where the frustum intersects is ≥ a max-1 % and < a max % when calling LoD max Hierarchical data; When the ratio of the bounding box of the intersecting frustum is ≥ a max % when calling LoD max+1 Hierarchical data.

9. The 3D Gaussian-based large scene parallel processing method according to claim 8, characterized in that: For case 2, take the maximum x-coordinate value xmax and the minimum xmin of the center point of the bounding box currently involved in rendering, as well as the maximum ymax and the minimum ymin of the y-coordinate, and then divide it into the following circles: First layer: the upper limit of the x coordinate is xmax+h0, and the lower limit is xmax-h0; the upper limit of the y range is ymax+h0, and the lower limit is ymax-h0, that is, the x coordinate range is [xmin-h0,xmax+h0], and the y coordinate range is [ymin-h0,ymax+h0], and the same below; Second layer: the x-coordinate range is [xmin-h1, xmin-h0] and [xmin+h0, xmax+h1], and the y-coordinate range is [ymin-h1, ymin-h0] and [ymax+h0, ymax+h1]; And so on; The second to last layer: the x coordinate range is [xmin-hmax, xmin-h max-1 ] and [xmin+h max-1 ,xmax+hmax], the y coordinate range is [ymin-hmax,ymin-h max-1 ] and [ymin+h max-1 ,ymax+hmax]; The last layer: the x-coordinate range is [-∞,xmin-hmax] and [xmax+hmax,+∞], and the y-coordinate range is [-∞,ymin-hmax] and [ymax+hmax,+∞].

Images