Extensible dynamic loading large-scene three-dimensional reconstruction method

Abstract

The invention relates to an extensible three-dimensional reconstruction method for dynamically loading a large scene. The method comprises the following steps: S1, constructing a training scene; s2, the training scene is divided into blocks, overlapped data sets and parameters exist between the blocks, a current block is selected, and the parameters of the first current block are loaded; s3, loading the data set of the current block, training the data set on a GPU stream, and starting the GPU stream responsible for preloading to load parameters of a non-intersecting part of a next block and the current block; and S4, unloading the parameters belonging to the current block but not belonging to the next block, keeping the intersection part of the parameters of the current block and the next block unchanged, combining the parameters of the non-intersection part of the next block and the current block and the intersection part, taking the next block as a new current block, returning to S3 until all blocks are traversed, and then repeating S3 and S4 again until the training is finished. And obtaining a reconstructed scene. Compared with the prior art, the method has the advantages of reducing repeated loading cost, improving training efficiency and the like.

Description

Technical Field

[0001] The present invention relates to the technical field of three-dimensional reconstruction, and in particular to an expandable dynamic loading large-scene three-dimensional reconstruction method. Background Art

[0002] Thanks to the development of GPU hardware, in the current 3D reconstruction field, a single GPU card can be used to reconstruct objects and small scenes such as indoor scenes. However, as the scenes expand, it involves large-scale 3D reconstruction of city-level scenes. Due to the problems of large data set size and parameter quantity, poor convergence effect, etc., a single GPU card cannot load such a large scene, so it has always been a big challenge.

[0003] There are two main types of algorithms currently used in 3D reconstruction, namely Neural Radiance Field (NeRF) and 3D Gaussian Splatting (3DGS). The NeRF algorithm uses a fully connected deep network to describe the scene, converting the input 5D coordinates (spatial position (x, y, z) and viewing direction (θ, φ)) into volume density and view-related emission rays at the spatial position. The perspective is synthesized by querying the 5D coordinates along the camera ray, and the output color and density are projected into the image using classic volume rendering techniques. The 3DGS algorithm starts with the sparse points generated during the camera calibration process, represents the scene with a 3D Gaussian sphere, retains the required properties of the continuous volume radiation field for scene optimization, and avoids unnecessary calculations in the blank space. Then, during the training process, the density of the 3D Gaussian sphere is controlled, and the anisotropic covariance is optimized to achieve accurate representation of the scene. In addition, 3DGS also designs a fast visibility-aware rendering algorithm that supports anisotropic sputtering, which accelerates training and allows real-time rendering.

[0004] Block training schemes such as MegaNeRF, BlockNeRF, VastGS, and HierarchicalGS are representative works of independent training. They divide the scene into blocks, and train the blocks independently and in parallel. NeRF-XL adopts a joint training method, which distributes NeRF parameters into non-overlapping areas, and improves the algorithm to jointly train them on multiple GPUs. GrendelGS bypasses the concept of block by taking advantage of the characteristics of the Gaussian sputtering algorithm, and randomly and evenly distributes the Gaussian balls storing model parameters to multiple GPUs for joint training. Zero-Offload and a large model training method based on improved ZeRO-Offload technology (CN117992220A) apply the idea of ​​dynamic loading to large language model training, and allow a single card to train more parameters by periodically loading and unloading parameters. The specific approach is as follows: half-precision model parameters are stored in the GPU, while half-precision gradients, full-precision parameters, and all optimizer states are stored on the CPU. During training, the loss is calculated through forward propagation on the GPU, while during backpropagation, the gradients are offloaded to the CPU memory. After that, the full-precision parameters and optimizer state are directly updated on the CPU, and the updated full-precision parameters are copied from the CPU memory to the GPU memory.

[0005] Existing technologies can be independent training, which divides the scene into blocks and trains the blocks independently and in parallel. For example, MegaNeRF, BlockNeRF, VastGS, and HierarchicalGS are not optimized for single-card training, and do not consider the possible optimization points when training continuously between blocks, resulting in complete blocks being loaded each time, and the cost of repeated loading is high. There are also joint training such as VastGS and NeRF-XL. Because the parameters are distributed on multiple GPUs, it is necessary to count the parameter information related to global synchronization during calculation, which introduces additional global message synchronization overhead. And Zero-Offload applied to large language models splits the entire model, and the parameters are still updated in a holistic manner during training. However, in the scene of three-dimensional reconstruction, this splitting method is not designed in combination with the characteristics of spatial distribution, and is not efficient enough compared to the block splitting method. Summary of the invention

[0006] The purpose of the present invention is to provide an extensible dynamic loading large scene 3D reconstruction method in order to reduce repeated loading costs while improving training efficiency.

[0007] The purpose of the present invention can be achieved by the following technical solutions:

[0008] A scalable dynamic loading large scene 3D reconstruction method, the method comprising the following steps:

[0009] S1. Obtain a large scene data set to be trained and construct the entire training scene based on the data set distribution;

[0010] S2, evenly divide the training scene into blocks, divide the large scene data set into corresponding blocks, there are overlapping data sets and parameters between each block, select a block as the current block, and load the parameters of the first current block;

[0011] S3, load the data set of the current block on a GPU stream for training, and start the GPU stream responsible for preloading to load the parameters of the next block that does not intersect with the current block;

[0012] S4. After the training of the current block is completed, the parameters that belong to the current block but not to the next block are unloaded, the intersecting part of the parameters of the current block and the next block is kept unchanged, the parameters of the non-intersecting part of the next block and the intersecting part of the current block are merged, and the merged parameters are used as the parameters of the next block that has been loaded. Then the next block is used as the new current block, and S3 is returned until all blocks are traversed. Then S3 and S4 are repeated again until the training is completed to obtain the reconstructed scene.

[0013] Further, the parameters of the non-intersecting portion between the next block and the current block are parameters that belong to the next block but do not belong to the current block.

[0014] Furthermore, the method further includes: taking all blocks as current blocks, and executing steps S3 and S4 in parallel on multiple GPUs for all current blocks.

[0015] Furthermore, the step of executing S3 and S4 in parallel on multiple GPUs for all current blocks specifically includes: evenly distributing all current blocks to multiple GPUs, and executing S3 and S4 in parallel.

[0016] Furthermore, S3 and S4 are repeated again until the training is completed. Specifically,

[0017] After traversing all blocks, the number of iterations is set to be increased by 1, and S3 and S4 are repeated again until the number of iterations reaches the threshold.

[0018] Furthermore, all blocks are traversed according to the route.

[0019] Furthermore, the route is zigzag.

[0020] Furthermore, large scene datasets consist of scene images captured by cameras.

[0021] Furthermore, the loading of parameters and data sets of the current block for training on one GPU stream and the starting of a GPU stream responsible for preloading to load parameters of a portion of the next block that does not intersect with the current block are performed simultaneously.

[0022] Furthermore, the entire training scenario is constructed based on the data set distribution as follows:

[0023] Record the position of the camera center in the large scene dataset, and build the entire training scene based on the position distribution of the camera center.

[0024] Compared with the prior art, the present invention has the following beneficial effects:

[0025] The independent training method of the present invention avoids the global synchronization overhead and can optimize a single block independently. Compared with traditional independent training, the dynamic loading provided by this patent is for single-card training. Each loading and unloading is the parameter of the difference set between the two blocks, and the loading and calculation are overlapping, which has lower overall costs. In theory, it can support the training and reconstruction of scenes of infinite area on a single GPU card. Compared with the dynamic loading method of large language models, the dynamic loading of this patent focuses more on three-dimensional scenes, and is designed in combination with spatial distribution characteristics, which has higher training efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Figure 1 is a flow chart of the present invention;

[0027] Figure 2 This is a schematic diagram of the dynamic loading three-dimensional reconstruction of the present invention. DETAILED DESCRIPTION

[0028] The present invention is described in detail below in conjunction with the accompanying drawings and specific embodiments. This embodiment is implemented based on the technical solution of the present invention, and provides a detailed implementation method and specific operation process, but the protection scope of the present invention is not limited to the following embodiments.

[0029] Benefiting from the hardware development of GPU, in the current field of 3D reconstruction, a single GPU card can be used to reconstruct objects and small scenes such as indoor scenes. However, as the scene expands, it involves the 3D reconstruction of large scenes at the city level. Because of the large data set size and parameter quantity, poor convergence effect and other problems, a single GPU card cannot load scenes of such a scale, so it has always been a big challenge. The present invention aims to propose an extensible parameter dynamic loading training method to solve the problems of high training cost and poor training effect in the above-mentioned large-scene 3D reconstruction.

[0030] The present invention proposes an extensible dynamic loading large scene 3D reconstruction method, the flow chart is as follows Figure 1 As shown, the method comprises the following steps:

[0031] S1. Obtain a large scene data set to be trained, and construct the entire training scene based on the distribution of the data set;

[0032] S2. Evenly divide the training scene into blocks, divide the large scene data set into corresponding blocks, and there are overlapping data sets and parameters between each block. Select a block as the current block and load the parameters of the first current block;

[0033] S3, load the data set of the current block on a GPU stream for training, and start the GPU stream responsible for preloading to load the parameters of the next block that does not intersect with the current block;

[0034] S4. After the training of the current block is completed, the parameters that belong to the current block but not to the next block are unloaded, the intersecting part of the parameters of the current block and the next block is kept unchanged, the parameters of the non-intersecting part of the next block and the intersecting part of the current block are merged, and the merged parameters are used as the parameters of the next block that has been loaded, and then the next block is used as the new current block. Return to S3 until all blocks are traversed, and then S3 and S4 are repeated again until the training is completed to obtain the reconstructed scene.

[0035] The present invention divides the entire three-dimensional scene into blocks. During training, training is performed by unloading and loading each block. When all blocks are loaded and trained, one round is considered to be completed. Both unloading and loading only operate on the non-intersecting parts between blocks. When the current GPU stream is training the previous block, the new GPU stream will pre-load the part of the next block that does not intersect with the training block. Based on the above ideas, the present invention further proposes a method for expanding from a single GPU to multiple GPUs, that is, by setting multiple blocks and multiple starting points to achieve inter-block hybrid parallelism (model parallelism plus data parallelism), and by training multiple camera inputs simultaneously within the block to achieve intra-block data parallelism.

[0036] In the present invention, when loading the data set required for training, the position of the camera center is recorded, and the position distribution of the camera center of the data set constructs the entire training scene.

[0037] When training officially starts, first load the parameters and data set of the current block on a GPU stream for training, and at the same time start the GPU stream responsible for preloading to load the parameters of the next block that does not intersect with the current training block (the part that belongs to the preloaded block and not the current training block). The two can overlap and execute simultaneously.

[0038] When a block is trained, unload the parameters that belong to the current block but not the next block, keep the intersecting part of the parameters of the current block and the next block unchanged, merge the parameters and the intersecting part of the next block that do not intersect with the current block, and start training the next block. The GPU stream responsible for preloading also starts the prefetch operation of the next block.

[0039] Repeat the above steps until all blocks have been loaded and trained, which is considered a round. The entire training is terminated when the number of training rounds reaches the specified number of training rounds.

[0040] The present invention also extends the dynamic loading of the above-mentioned single card in parallel. All blocks of the scene can be evenly distributed to multiple GPUs for parallel execution, achieving the effect of model parallelism plus data parallelism. Within the blocks, the present invention can further split the data set to achieve the effect of data parallelism, further solving the problem of low training efficiency of large scenes and large data sets.

[0041] The present invention has been verified in both NeRF and 3DGS algorithms. Based on NeRF algorithm, the modeling of 4K15 square kilometers scene was completed, and based on 3DGS algorithm, the modeling of 25 square kilometers was completed. Figure 2 As shown in the figure, the entire scene is divided into blocks, and the data set is allocated to the corresponding sub-blocks according to the location of the camera center. The training is carried out in a zigzag traversal manner, and parameters are dynamically loaded and unloaded, and data parallel training is supported.

[0042] The preferred specific embodiments of the present invention are described in detail above. It should be understood that a person skilled in the art can make many modifications and changes based on the concept of the present invention without creative work. Therefore, any technical solution that can be obtained by a person skilled in the art through logical analysis, reasoning or limited experiments based on the concept of the present invention on the basis of the prior art should be within the scope of protection determined by the claims.

Claims

1. A scalable dynamic loading large scene 3D reconstruction method, characterized in that: The method comprises the following steps: S1. Obtain a large scene data set to be trained and construct the entire training scene based on the data set distribution; S2, evenly divide the training scene into blocks, divide the large scene data set into corresponding blocks, there are overlapping data sets and parameters between each block, select a block as the current block, and load the parameters of the first current block; S3, load the data set of the current block on a GPU stream for training, and start the GPU stream responsible for preloading to load the parameters of the next block that does not intersect with the current block; S4. After the training of the current block is completed, the parameters that belong to the current block but not to the next block are unloaded, the intersecting part of the parameters of the current block and the next block is kept unchanged, the parameters of the non-intersecting part of the next block and the intersecting part of the current block are merged, and the merged parameters are used as the parameters of the next block that has been loaded. Then the next block is used as the new current block, and S3 is returned until all blocks are traversed. Then S3 and S4 are repeated again until the training is completed to obtain the reconstructed scene.

2. The scalable dynamic loading large scene 3D reconstruction method according to claim 1, characterized in that: The parameters of the non-intersecting portion between the next block and the current block are parameters that belong to the next block but do not belong to the current block.

3. The scalable dynamic loading large scene 3D reconstruction method according to claim 1, characterized in that: The method further includes: treating all blocks as current blocks, and executing steps S3 and S4 in parallel on multiple GPUs for all current blocks.

4. The scalable dynamic loading large scene 3D reconstruction method according to claim 3, characterized in that: The step of executing S3 and S4 in parallel on multiple GPUs for all current blocks is specifically: all current blocks are evenly distributed to multiple GPUs, and steps S3 and S4 are executed in parallel.

5. The scalable dynamic loading large scene 3D reconstruction method according to claim 1, characterized in that: Repeat S3 and S4 again until the training is completed. Specifically: After traversing all blocks, the number of iterations is set to be increased by 1, and S3 and S4 are repeated again until the number of iterations reaches the threshold.

6. The scalable dynamic loading large scene 3D reconstruction method according to claim 1, characterized in that: Follow the route when traversing all blocks.

7. The scalable dynamic loading large scene 3D reconstruction method according to claim 6, characterized in that: The route is zigzag shaped.

8. The scalable dynamic loading large scene 3D reconstruction method according to claim 1, characterized in that: The Big Scene dataset consists of scene images captured by cameras.

9. The scalable dynamic loading large scene 3D reconstruction method according to claim 1, characterized in that: The loading of the parameters and data set of the current block is performed on a GPU stream for training and the starting of a GPU stream responsible for preloading to load the parameters of the next block that does not intersect with the current block are performed simultaneously.

10. The scalable dynamic loading large scene 3D reconstruction method according to claim 1, characterized in that: The entire training scenario is constructed based on the data set distribution as follows: Record the position of the camera center in the large scene dataset, and build the entire training scene based on the position distribution of the camera center.

Images