Large-scale scene efficient reconstruction method based on 3D Gaussian sputtering

Abstract

The invention provides a large-scene efficient reconstruction method based on 3D Gaussian sputtering. The method comprises the following steps: collecting a multi-view large-scene image and carrying out image preprocessing; camera pose data and point cloud data are acquired; dividing the scene ground plane into a plurality of grid units; determining a contribution camera of each grid unit; obtaining extended point cloud data outside the boundary of each grid unit and 3D Gaussian distribution inside and outside the boundary of each grid unit; dynamically adjusting the 3D Gaussian distribution in each grid unit and deleting the 3D Gaussian distribution outside the boundary of each grid unit; and determining weights of all adjacent grid units, and carrying out weighted average on the weights of all adjacent grid units and the optimized 3D Gaussian distribution parameters to obtain 3D Gaussian distribution of a merged complete scene and generate a three-dimensional model. According to the sparse multi-view image 3D Gaussian reconstruction method, the reconstruction efficiency of a large-scale scene can be improved, and the memory overhead can be reduced.

Description

Technical Field

[0001] The present invention relates to the field of 3D Gaussian reconstruction, and in particular to a large scene efficient reconstruction method based on 3D Gaussian sputtering. Background Art

[0002] With the booming development of technologies such as virtual reality, aerial mapping, smart cities, and autonomous driving, the demand for efficient reconstruction of large-scale scenes is growing. Existing methods such as Neural Radiance Field (NeRF) have achieved remarkable results in reconstruction quality, but their huge computational and memory overhead lead to low reconstruction efficiency. In contrast, 3D Gaussian Sputtering (3DGS), as an emerging solution, shows good potential in reconstruction quality and reconstruction efficiency. However, 3DGS still faces challenges such as high memory overhead, long training time, and slow rendering speed when applied to large-scale scenes.

[0003] Therefore, it becomes an urgent problem to propose an efficient reconstruction method for large-scale scenes based on 3D Gaussian sputtering to reduce memory overhead and improve reconstruction efficiency. Summary of the invention

[0004] In view of this, the present invention provides a large scene efficient reconstruction method based on 3D Gaussian sputtering to solve the problems existing in the prior art.

[0005] The present invention provides a large scene efficient reconstruction method based on 3D Gaussian sputtering, comprising:

[0006] Collecting a large scene image from multiple perspectives and performing image preprocessing to obtain a preprocessed large scene image from multiple perspectives, wherein images from adjacent perspectives overlap;

[0007] According to the preprocessed multi-view large scene image, COLMAP is used to obtain the camera pose data and point cloud data corresponding to the multi-view large scene image;

[0008] Divide the scene ground plane into multiple grid cells, where each grid cell has a clear boundary and coordinate range;

[0009] Determine the contributing camera for each grid cell;

[0010] Expand the point cloud points within the boundary of each grid unit outward to obtain the extended point cloud data outside the boundary of each grid unit;

[0011] Refine the point cloud points within the boundary of each grid unit to obtain a more refined point cloud representation;

[0012] Use the pre-trained point cloud segmentation model to segment the refined point cloud data within each grid cell boundary and the extended point cloud data outside each grid cell boundary, calculate the Gaussian center position and covariance matrix, and obtain the 3D Gaussian distribution within and outside the boundary of each grid cell;

[0013] Dynamically adjust the 3D Gaussian distribution within each grid cell boundary using a Gaussian refinement module, wherein the Gaussian refinement module is used to dynamically allocate the number of Gaussians according to local geometric complexity, allocate more Gaussians to areas with complex geometric details, and delete redundant Gaussians;

[0014] Delete the 3D Gaussian distribution outside the boundary of each grid cell to obtain the optimized 3D Gaussian distribution of each grid cell;

[0015] The weights of all adjacent grid cells are determined according to the contribution camera of each grid cell, and the weights of all adjacent grid cells and the optimized 3D Gaussian distribution parameters are weighted averaged to obtain the 3D Gaussian distribution of the merged complete scene;

[0016] A 3D Gaussian rendering technique is used to generate a three-dimensional model according to the 3D Gaussian distribution of the complete scene.

[0017] Preferably, the image preprocessing includes denoising, normalization, and cropping and scaling.

[0018] Further preferably, the steps of determining the contributing camera of any grid unit A are as follows:

[0019] Projecting the boundary of the grid unit A onto the image plane of the camera, and calculating the projected area of ​​the grid unit A in each camera viewing angle;

[0020] Obtain the total number of pixels based on the camera resolution and calculate the total area of ​​the camera image;

[0021] Calculate the ratio of the projected area of ​​grid cell A to the total area of ​​the camera image in each camera view;

[0022] A plurality of cameras with the highest ratios are selected as contributing cameras of the grid unit A.

[0023] Further preferably, the method of dynamically adjusting the 3D Gaussian distribution within each grid cell boundary using the Gaussian refinement module is as follows:

[0024] Using a key point scorer, a key point score map of each input view is calculated from the image features, wherein the key point score map is used to reflect the geometric complexity of different regions in the image and indicate the importance of each region in the image;

[0025] In the key point score map, for areas with higher scores, the Gaussian center is further subdivided into multiple smaller centers, and for areas with lower scores, the transparency and scaling of the corresponding Gaussian center are gradually reduced until it is completely deleted.

[0026] Further preferably, the weights of adjacent grid units are determined according to the number of cameras shared between the adjacent grid units. The more cameras shared between adjacent grid units, the higher the weights. If the contributing cameras of two adjacent grid units both include camera A, camera A is the shared camera of the two adjacent grid units.

[0027] The large-scene efficient reconstruction method based on 3D Gaussian sputtering provided by the present invention can improve the reconstruction efficiency of large-scale scenes and reduce memory overhead. The method first divides the large scene into multiple grid units, then calculates the 3D Gaussian distribution of each grid unit and refines the 3D Gaussian distribution through a Gaussian refinement module. By dynamically increasing or decreasing the Gaussian according to the complexity of the area, the quality of the 3D Gaussian can be improved and redundant information can be reduced. Finally, the 3D Gaussian distributions of all grid units are merged into a complete large-scene 3D Gaussian distribution by weighted averaging, and finally a complete large-scene three-dimensional model is generated. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0029] Figure 1 It is a flow chart of a large scene efficient reconstruction method based on 3D Gaussian sputtering provided by the present invention. DETAILED DESCRIPTION

[0030] In order to make the purpose, technical scheme and advantages of the present invention clearer, the present invention is described in detail below in conjunction with the accompanying drawings and specific embodiments. It should be noted that, in order to avoid blurring the present invention due to unnecessary details, only the processing steps closely related to the scheme of the present invention are shown in the accompanying drawings, and other details that are not closely related to the present invention are omitted.

[0031] like Figure 1 As shown, the present invention provides a large scene efficient reconstruction method based on 3D Gaussian sputtering, comprising the following steps:

[0032] S1: collecting a large scene image from multiple perspectives and performing image preprocessing to obtain a preprocessed large scene image from multiple perspectives, wherein images from adjacent perspectives overlap;

[0033] Among them, multi-perspective large scene images can be collected by drone cameras. Specifically, the drone is controlled to fly around the center of the large scene at a constant speed, and the drone camera is controlled to capture scene images from multiple angles, and ensure that there is enough overlap between adjacent perspective images;

[0034] The image preprocessing includes denoising, normalization, and cropping and scaling;

[0035] The denoising process is as follows: first, select a suitable denoising algorithm (such as Gaussian filtering, median filtering, bilateral filtering) according to the noise type of the image; then set the parameters of the denoising algorithm (such as filter size, number of iterations) according to the noise level and resolution of the image; finally, use an image processing library (such as OpenCV or Pillow) to denoise the image;

[0036] The normalization process is as follows: the pixel values ​​of the denoised image are normalized to the range of [0,1] using the image processing library;

[0037] The steps of cropping and scaling are as follows: Use the image processing library to crop and scale the normalized image to a fixed size, such as 256×256 pixels;

[0038] S2: Using COLMAP to obtain the camera pose data and point cloud data corresponding to the multi-view large scene image based on the preprocessed multi-view large scene image;

[0039] S3: Divide the scene ground plane into a plurality of grid cells, for example, using m×n grids, where each grid cell has a clear boundary and coordinate range;

[0040] The specific steps of S3 are as follows:

[0041] S31: Select a suitable grid division scheme according to the size and complexity of the scene to ensure that the size of each grid unit is moderate, neither too large to affect the calculation nor too small to affect the efficiency;

[0042] S32: Divide the scene ground plane into a plurality of grid units according to the selected grid division scheme, each grid unit having a clear boundary and coordinate range;

[0043] S4: determining the contributing camera of each grid unit, wherein for any grid unit, a group of cameras that contribute the most to it is selected as the contributing cameras of the grid unit according to a visibility criterion;

[0044] The steps for determining the contributing camera of any grid cell A are as follows:

[0045] S41: Projecting the boundary of the grid unit A onto the image plane of the camera, and calculating the projection area of ​​the grid unit A in each camera viewing angle;

[0046] S42: Obtain the total number of pixels according to the camera resolution and calculate the total area of ​​the camera image;

[0047] S43: Calculate the ratio of the projection area of ​​the grid unit A to the total area of ​​the camera image in each camera viewing angle;

[0048] S44: selecting a plurality of cameras (the number of which can be adjusted according to the size of the scene) with the highest ratio as contributing cameras of the grid unit A;

[0049] S5: Expand the point cloud points within the boundary of each grid unit outward according to a certain ratio, for example, 20%, to obtain extended point cloud data outside the boundary of each grid unit;

[0050] S6: Use a deep learning-based point cloud refinement method to refine the point cloud points within the boundary of each grid cell to obtain a more refined point cloud representation;

[0051] S7: Use the pre-trained point cloud segmentation model to segment the refined point cloud data within the boundary of each grid cell and the extended point cloud data outside the boundary of each grid cell, calculate the Gaussian center position and covariance matrix, and obtain the 3D Gaussian distribution within and outside the boundary of each grid cell;

[0052] S8: dynamically adjusting the 3D Gaussian distribution within the boundary of each grid unit using a Gaussian refinement module, wherein the Gaussian refinement module is used to dynamically allocate the number of Gaussians according to local geometric complexity, allocate more Gaussians to areas with complex geometric details, and delete redundant Gaussians;

[0053] The specific steps of S8 are as follows:

[0054] S81: using a key point scorer, calculating a key point score map of each input view from the image features, wherein the key point score map is used to reflect the geometric complexity of different regions in the image and can indicate the importance of each region in the image;

[0055] S82: In the key point score map, for a region with a higher score, further subdivide the Gaussian center into a plurality of smaller centers to more accurately capture local geometric details and subtle changes in the image, and for a region with a lower score, gradually reduce the transparency and scaling of the corresponding Gaussian center until it is completely deleted, wherein deleting redundant information from the Gaussian set can avoid Gaussian center overlap and improve the efficiency and accuracy of the model;

[0056] S9: Delete the 3D Gaussian distribution outside the boundary of each grid unit to obtain the optimized 3D Gaussian distribution of each grid unit to prevent boundary artifacts and ensure the quality of the merged scene;

[0057] S10: determining the weights of all adjacent grid cells according to the contributing camera of each grid cell, and performing weighted averaging on the weights of all adjacent grid cells and the optimized 3D Gaussian distribution parameters to obtain the 3D Gaussian distribution of the merged complete scene;

[0058] The weights of adjacent grid units are determined according to the number of cameras shared between the adjacent grid units. The more cameras shared between the adjacent grid units, the higher the weights. If the contributing cameras of two adjacent grid units both include camera A, camera A is the shared camera of the two adjacent grid units.

[0059] S11: Generate a three-dimensional model according to the 3D Gaussian distribution of the complete scene using a 3D Gaussian rendering technique.

[0060] The large-scene efficient reconstruction method based on 3D Gaussian sputtering provided by the present invention can improve the reconstruction efficiency of large-scale scenes and reduce memory overhead. The method first divides the large scene into multiple grid units, then calculates the 3D Gaussian distribution of each grid unit and refines the 3D Gaussian distribution through a Gaussian refinement module. By dynamically increasing or decreasing the Gaussian according to the complexity of the area, the quality of the 3D Gaussian can be improved and redundant information can be reduced. Finally, the 3D Gaussian distributions of all grid units are merged into a complete large-scene 3D Gaussian distribution by weighted averaging, and finally a complete large-scene three-dimensional model is generated.

[0061] It should be noted that the purpose of publishing the embodiments is to help further understand the present invention, but those skilled in the art can understand that various substitutions and modifications are possible without departing from the spirit and scope of the present invention and the appended claims. Therefore, the present invention should not be limited to the contents disclosed in the embodiments, and the scope of protection claimed by the present invention shall be subject to the scope defined in the claims.

Claims

1. A large scene efficient reconstruction method based on 3D Gaussian sputtering, characterized in that: include: Collecting a large scene image from multiple perspectives and performing image preprocessing to obtain a preprocessed large scene image from multiple perspectives, wherein images from adjacent perspectives overlap; According to the preprocessed multi-view large scene image, COLMAP is used to obtain the camera pose data and point cloud data corresponding to the multi-view large scene image; Divide the scene ground plane into multiple grid cells, where each grid cell has a clear boundary and coordinate range; Determine the contributing camera for each grid cell; Expand the point cloud points within the boundary of each grid unit outward to obtain the extended point cloud data outside the boundary of each grid unit; Refine the point cloud points within the boundary of each grid unit to obtain a more refined point cloud representation; Use the pre-trained point cloud segmentation model to segment the refined point cloud data within each grid cell boundary and the extended point cloud data outside each grid cell boundary, calculate the Gaussian center position and covariance matrix, and obtain the 3D Gaussian distribution within and outside the boundary of each grid cell; Dynamically adjust the 3D Gaussian distribution within each grid cell boundary using a Gaussian refinement module, wherein the Gaussian refinement module is used to dynamically allocate the number of Gaussians according to local geometric complexity, allocate more Gaussians to areas with complex geometric details, and delete redundant Gaussians; Delete the 3D Gaussian distribution outside the boundary of each grid cell to obtain the optimized 3D Gaussian distribution of each grid cell; The weights of all adjacent grid cells are determined according to the contribution camera of each grid cell, and the weights of all adjacent grid cells and the optimized 3D Gaussian distribution parameters are weighted averaged to obtain the 3D Gaussian distribution of the merged complete scene; A 3D Gaussian rendering technique is used to generate a three-dimensional model according to the 3D Gaussian distribution of the complete scene.

2. The large scene efficient reconstruction method based on 3D Gaussian sputtering according to claim 1 is characterized by: The image preprocessing includes denoising, normalization, and cropping and scaling.

3. The large scene efficient reconstruction method based on 3D Gaussian sputtering according to claim 1 is characterized by: The steps to determine the contributing camera for any grid cell A are as follows: Projecting the boundary of the grid unit A onto the image plane of the camera, and calculating the projected area of ​​the grid unit A in each camera viewing angle; Obtain the total number of pixels based on the camera resolution and calculate the total area of ​​the camera image; Calculate the ratio of the projected area of ​​grid cell A to the total area of ​​the camera image in each camera view; A plurality of cameras with the highest ratios are selected as contributing cameras of the grid unit A.

4. The large scene efficient reconstruction method based on 3D Gaussian sputtering according to claim 1 is characterized by: The method of using the Gaussian refinement module to dynamically adjust the 3D Gaussian distribution within each grid cell boundary is as follows: Using a key point scorer, a key point score map of each input view is calculated from the image features, wherein the key point score map is used to reflect the geometric complexity of different regions in the image and indicate the importance of each region in the image; In the key point score map, for areas with higher scores, the Gaussian center is further subdivided into multiple smaller centers, and for areas with lower scores, the transparency and scaling of the corresponding Gaussian center are gradually reduced until it is completely deleted.

5. The large scene efficient reconstruction method based on 3D Gaussian sputtering according to claim 1 is characterized in that: The weights of adjacent grid cells are determined according to the number of cameras shared between the adjacent grid cells. The more cameras shared between adjacent grid cells, the higher the weights. If the contributing cameras of two adjacent grid cells both include camera A, camera A is the shared camera of the two adjacent grid cells.

Images