An efficient modeling and visualization method for city-level scenes based on NeRF

Abstract

The present invention provides a method for efficient modeling and visualization of city-level scenes based on NeRF, including training data partitioning, and also including the following steps: parallel training of sub-regions; generating a sparse octree and storing weight data of each node; and performing interactive rendering. The present invention proposes a method for efficient modeling and visualization of city-level scenes based on NeRF, which introduces NeRF implicit neural radiation field technology into city-level three-dimensional scene modeling, and realizes interactive browsing of city-level three-dimensional scenes while taking into account the advantages of NeRF technology for high-definition image data collected by drones, and adopts a sparse octree structure to solve the problems of city-level three-dimensional scene modeling and interactive rendering.

Description

Technical Field

[0001] The present invention relates to the technical field of three-dimensional scene modeling, and in particular to a NeRF-based city-level scene efficient modeling and visualization method. Background Art

[0002] With the rise of smart city construction, 3D modeling of large city-level scenes has wide application value in many fields, including city roaming, autonomous driving, and navigation. Traditional 3D modeling methods, such as oblique photography modeling, use display methods to express scenes and describe scene information through elements such as geometry, texture, and material. This method has a long modeling process, generates a large amount of data, and often requires a lot of manual editing work to achieve high fidelity, resulting in low data production efficiency.

[0003] Problems in large city-level scenarios:

[0004] Block-NeRF is mainly aimed at large amounts of street scene data, where the perspective is mainly pedestrians or cars, and the amount of scene data involved in the field of vision is basically constant; however, in large city-level scenes, as the perspective is adjusted, such as when the scene is zoomed in and out, the amount of data involved may fluctuate from dozens of pictures to pictures of the entire scene, which Block-NeRF cannot process efficiently. In addition, in order to have a good transition effect between blocks, 75% of the data overlap is required between blocks, which will lead to a significant increase in the amount of calculation.

[0005] Similarly, since CityNeRF uses a dual-branch model, additional computing power is required to support the training of the Grid branch, which requires a huge amount of computation.

[0006] Mega-NeRF relies on a good data structure to solve the problem of efficient modeling and browsing of campus-level scenes. Its efficiency mainly lies in loading all octree leaf nodes into the video memory during rendering initialization, and completing node scheduling and dynamic segmentation in the video memory. When the scene scope is expanded to the city level, the initial octree level cannot meet the requirements, which is reflected in:

[0007] 1. If the number of octree layers is increased, the total space occupied by the leaf nodes will be too large to be loaded. To ensure the resolution of the initial octree leaf nodes, the depth of the tree must be increased, and the space occupied by the leaf nodes will increase dramatically. For example, a 500*250m 2 In the scenario where the tree depth increases from 8 to 9 levels, the storage occupied by leaf nodes will increase from 12GB to 55GB, which is beyond the processing range of any GPU.

[0008] 2. If the number of octree layers is kept at a level with a suitable total size, the resolution of the leaf nodes will be lost. In addition, during dynamic rendering, the number of layers that need to be dynamically segmented will increase further, and the amount of calculation required to calculate the weight of the subdivided leaf nodes will increase, thus affecting the efficiency of interactive rendering. In addition, during rendering initialization, the process is slow because the leaf nodes of the entire tree need to be loaded.

[0009] The paper “Mega-NeRF: A Scalable Approach to Building Large-Scale NeRF for Virtual Flying” by H Turki, D Ramanan, and M Satyanarayanan was presented at the 2022 CVPR conference and uses Neural Radiance Fields (Nerf) to build interactive 3D environments from large visual captures spanning buildings and even multiple city blocks collected primarily from drones. Compared to single-object scenes (where Nerf is traditionally evaluated), our scale poses multiple challenges, including (1) the need to model thousands of images with varying lighting conditions, each capturing only a small portion of the scene, (2) prohibitively large model capacity that makes training on a single GPU infeasible, and (3) significant challenges in fast rendering, which would enable interactive flying. To address these challenges, we first analyze visibility statistics for large-scale scenes, motivating sparse network structures where parameters are specialized for different regions of the scene. We introduce a simple geometric clustering algorithm for data parallelism that partitions the training images (or more precisely, pixels) into distinct NeRF submodules that can be trained in parallel. We evaluated our approach on existing datasets (Quad 6k and UrbanScene3D) as well as our own drone footage, speeding up training by 3x and improving PSNR by 12%. We also evaluated the recent NeRF fast renderer on top of Mega-NeRF and introduced a new method to exploit temporal consistency. Our technique achieves a 40x speedup over traditional NeRF rendering while keeping PSNR quality within 0.8db, exceeding the fidelity of existing fast renderers. The disadvantage of the article is that when rendering is initialized, the process is slow because the leaf nodes of the entire tree need to be loaded, and as the depth of the tree increases, the storage occupied by the leaf nodes increases dramatically, which will exceed the processing range of any GPU.

[0010] Linning Xu, Yuanbo Xiangli, Sida Peng, XingangPan, Nanxuan Zhao, Christian Theobalt, Bo Dai, and Dahua Lin, titled “Grid-Guided Neural Radiance Fields for Large Urban Scenes,” was presented at the 2022 CVPR conference. MLP-based neural radiance fields (nerf-based approaches) often suffer from underfitting with blurry renderings in large-scale scenes due to limited model capacity. Recent approaches propose to partition the scene geographically and adopt multiple sub-nerfs to model each region individually, resulting in linear growth in training cost and number of sub-nerfs as the scene expands. Another solution is to use a feature grid representation, which is computationally efficient and can naturally scale to large scenes with higher grid resolutions. However, feature grids are often less constrained and often reach suboptimal solutions, producing noisy artifacts in renderings, especially in areas with complex geometry and textures. In this work, we propose a new framework that achieves high-fidelity rendering of large urban scenes while being computationally efficient. We propose to use a compact multi-resolution ground feature plane representation to roughly capture the scene, and complement the position encoding input through another NeRF branch for rendering in a jointly learned manner. We show that such an integration can take advantage of the advantages of two alternative solutions: under the guidance of the feature grid representation, lightweight NeRF is sufficient to render realistic novel views with fine details; while the jointly optimized ground feature plane can be further refined to form a more accurate and compact feature space, outputting more natural rendering results. The disadvantage of this article is that it uses a two-branch model, which requires additional computing power to support the training of the Grid branch, and the computational complexity is extremely large. Summary of the invention

[0011] In order to solve the above technical problems, the present invention proposes an efficient modeling and visualization method for city-level scenes based on NeRF, which introduces NeRF implicit neural radiation field technology into city-level three-dimensional scene modeling. For high-definition image data collected by drones, while taking into account the advantages of NeRF technology, interactive browsing of city-level three-dimensional scenes is realized, and a sparse octree structure is used to solve the problems of city-level three-dimensional scene modeling and interactive rendering.

[0012] The purpose of the present invention is to provide a method for efficient modeling and visualization of city-level scenes based on NeRF, including training data partitioning, and further comprising the following steps:

[0013] Step 1: Parallel training of sub-regions;

[0014] Step 2: Generate a sparse octree and store the weight data of each node;

[0015] Step 3: Perform interactive rendering.

[0016] Preferably, the training data partitioning includes partitioning the entire scene, with the center point of each zone being denoted as C i , N training images, the image width and height are (W, H) pixels, the estimated closest distance is Near, the farthest distance is Far, and each training image is processed as follows:

[0017] For W*H pixels, each pixel to C i The light samples 256 points between (Near, Far) and calculates W*H*256 points to C i The minimum distance is MinDistance;

[0018] Pixel to C i The actual distance Dis / MinDistance is less than the threshold, then the pixel participates in C i For training, mask is recorded as 1, otherwise it is recorded as 0;

[0019] After processing W*H pixels, a Mask (C i ,Image j ); process each partition in turn and get the total Mask.

[0020] In any of the above schemes, preferably, step 1 includes training NeRF-SHs in each sub-region until a convergence state is reached.

[0021] In any of the above schemes, preferably, step 2 includes dividing the entire scene range into an octree, with the voxels in each layer having the same size, and the division is gradually refined from top to bottom to obtain a sparse octree according to the actual distribution.

[0022] In any of the above schemes, preferably, step 2 comprises the following sub-steps:

[0023] Step 21: Calculate the octree level;

[0024] Step 22: For L l Layer voxel, sub-region model calculates density and spherical harmonic coefficients;

[0025] Step 23: storing the calculation result in step 22 according to the coding rule;

[0026] Step 24: Repeat steps 22 and 23 for the nodes at each layer of the octree to generate the octree.

[0027] In any of the above solutions, preferably, step 21 includes top-down and regular division of the octree, and the division is terminated when the level L satisfies the following conditions:

[0028] G / (S*2 L ) <N 1 / 2

[0029] Where G = (G x +G y ) / 2 and (G x ,G y ) is the length and width of the geographical range of the urban scene, S is the spatial resolution of the training image, and N is the threshold of the average number of pixels in the grid.

[0030] In any of the above solutions, preferably, step 22 includes the following sub-steps:

[0031] Step 221: Use 3D grid uniform sampling points to obtain the volume density ρ of the area where each voxel is located;

[0032] Step 222: Render the alpha map using the viewing angles of all training images, record the weight of each voxel on the ray, and delete the voxels with weights lower than the threshold. The voxels that are not deleted serve as nodes of the octree at this layer.

[0033] Step 223: Randomly sample 256 points in each remaining voxel, and use the average weight value of the sampled points as the value k of the node. The (ρ, k) corresponding to each node is the weight value of the voxel.

[0034] In any of the above solutions, preferably, the encoding rules include:

[0035] 1) The unique identifier of each node is represented by layer L, row R, column C and height H. In order to ensure the retrieval efficiency after storage in the database, it needs to be encoded. The specific method is: L and H are represented by 8 bits, R and C are represented by 24 bits, a total of 64 bits;

[0036] 2) The weight data (ρ, k) of each node is actually a float array, where ρ has only one value and the spherical harmonic weight k is an array. To improve storage efficiency, it is packaged into a binary stream storage. Let the size of array k be S(k), and len be the length of the entire array, that is, len = S(k) + 1;

[0037] 3) In the Key-Value database, the weight data of each node in each layer of the octree is stored.

[0038] In any of the above schemes, it is preferred that step 3 includes adapting the appropriate level according to the position and angle of the current viewpoint during the rendering process, and calculating the layer-row-column-height range of each level of grid related to the current viewport. The rendering scheduler is responsible for retrieving the corresponding grid weight data, handing it over to CUDA for parallel processing, and finally inferring images at different angles.

[0039] In any of the above schemes, preferably, step 3 includes the following sub-steps:

[0040] Step 31: According to the position pos and angle dir of the camera in the scene, and the width and height of the viewport, calculate the position of the viewport center and the viewport boundary in the world coordinate system, and then project it into the two-dimensional screen coordinate system to obtain the number of pixels (V w ,V h );

[0041] Step 32: Calculate the octree level L;

[0042] Step 33: Extract the row, column, and height range of the L-layer grid;

[0043] Step 34: The rendering scheduler reads the mesh weight data from the database;

[0044] Step 35: Synthesize a new view;

[0045] Step 36: During the interactive rendering process, the rendering scheduler is responsible for maintaining the mesh list in the memory. For the requested mesh, it first searches the memory. If it is not found in the memory, it is loaded from the database according to the key and the local mesh list is updated. At the same time, it is responsible for maintaining the local mesh queue and ensuring the stability of the local memory according to the first-in-first-out principle.

[0046] In any of the above solutions, preferably, the calculation formula of the octree level L is:

[0047] L = log2 (G / (S*V))

[0048] V=(V w +V h ) / 2

[0049] Among them, G / S represents the number of pixels of the original image in one dimension, and V is the average number of pixels in the width and height of the image.

[0050] In any of the above solutions, preferably, step 33 includes calculating the intersection of the octree Tree and the bounding box range View of the viewport in the world coordinate system:

[0051] W cut =W tree ∩W View

[0052] W cut Convert the local coordinate system to obtain L cut :

[0053] L cut =W cut *T -1

[0054] Among them, W cut is the bounding box after cropping, W tree is the bounding box of the octree, W View is the bounding box of the current viewport, T is the matrix of octree to world coordinates, L cut is the range to be extracted from the octree (T x ,T y ,T z ), and then according to the level L, the row, column, and height range (R, C, H) of the L-layer grid are obtained, and the number of grids is n.

[0055] In any of the above schemes, preferably, step 34 includes encoding n grids according to the encoding rule, and querying Binary data from a database table as a Key, and obtaining the weight value (ρ, k) of each grid after deserialization.

[0056] In any of the above schemes, preferably, step 35 includes performing parallel calculations on a pixel-by-pixel basis on the newly generated image through CUDA, and for each calculation task, accumulating the color value of the current pixel according to the direction of the current light and the grid weight (ρ, k), and finally obtaining a composite image.

[0057] The present invention proposes an efficient modeling and visualization method for city-level scenes based on NeRF, which minimizes additional rendering work while ensuring rendering clarity, improves rendering efficiency, and solves the storage problem of octrees for ultra-large scenes.

[0058] NeRF: Neural Radiance Field.

[0059] CUDA: Graphics Processing Unit.

[0060] NeRF-SHs: is the collection of NeRF-SHs in each sub-region.

[0061] Mask: is the image mask, with width and height equal to the image, and the value is 0 or 1. 1 means that the pixel participates in training, and 0 means that the pixel does not participate in training. BRIEF DESCRIPTION OF THE DRAWINGS

[0062] Figure 1 The flowchart is a preferred embodiment of the NeRF-based city-level scene efficient modeling and visualization method according to the present invention.

[0063] Figure 2 It is a flowchart of another preferred embodiment of the NeRF-based city-level scene efficient modeling and visualization method according to the present invention.

[0064] Figure 3It is a schematic diagram of an embodiment of training data partitioning according to the NeRF-based city-level scene efficient modeling and visualization method of the present invention.

[0065] Figure 4 It is a schematic diagram of an embodiment of sub-region parallel training of the NeRF-based city-level scene efficient modeling and visualization method according to the present invention.

[0066] Figure 5 It is a schematic diagram of an embodiment of sparse octree hierarchical division and construction of the NeRF-based city-level scene efficient modeling and visualization method according to the present invention.

[0067] Figure 6 It is a schematic diagram of an embodiment of the Key encoding sequence of the NeRF-based city-level scene efficient modeling and visualization method according to the present invention.

[0068] Figure 7 It is a schematic diagram of an embodiment of the storage sequence of Binary according to the NeRF-based city-level scene efficient modeling and visualization method of the present invention.

[0069] Figure 8 It is a schematic diagram of an embodiment of an octree-based rendering process of the NeRF-based city-level scene efficient modeling and visualization method according to the present invention. DETAILED DESCRIPTION

[0070] The present invention is further described below in conjunction with the accompanying drawings and specific embodiments.

[0071] Embodiment 1

[0072] like Figure 1 As shown, step 1000 is executed to partition the training data. The training data partitioning includes partitioning the entire scene, and the center point of each zone is recorded as C i , N training images, the image width and height are (W, H) pixels, the estimated closest distance is Near, the farthest distance is Far, and each training image is processed as follows:

[0073] For W*H pixels, each pixel to C i The light samples 256 points between (Near, Far) and calculates W*H*256 points to C i The minimum distance is MinDistance;

[0074] Pixel to C i The actual distance Dis / MinDistance is less than the threshold, then the pixel participates in C i For training, mask is recorded as 1, otherwise it is recorded as 0;

[0075] After processing W*H pixels, a Mask (C i ,Image j ); process each partition in turn and get the total Mask.

[0076] Execute step 1100, the sub-regions are trained in parallel, and each sub-region trains NeRF-SHs until a convergence state is reached.

[0077] Execute step 1200 to generate a sparse octree and store the weight data of each node. The entire scene range is divided into octrees. The size of each layer of voxels is consistent. From top to bottom, the octree is gradually divided into fine parts to obtain a sparse octree according to the actual distribution, including the following sub-steps:

[0078] Execute step 1210 to calculate the octree level. The octree is divided from top to bottom in a regular manner. The division is terminated when the level L meets the following conditions:

[0079] G / (S*2 L ) <N 1 / 2

[0080] Where G = (G x +G y ) / 2 and (G x ,G y ) is the length and width of the geographical range of the urban scene, S is the spatial resolution of the training image, and N is the threshold of the average number of pixels in the grid.

[0081] Execute step 1220, for the Lth l The layer voxel, sub-region model calculates density and spherical harmonic coefficients, including the following sub-steps:

[0082] Execute step 1221, use the 3D grid uniform sampling points to obtain the volume density ρ of the area where each voxel is located;

[0083] Execute step 1222, render the alpha map using the viewing angles of all training images, record the weight of each voxel on the ray, and delete the voxels with weights lower than the threshold, and the voxels that are not deleted are used as nodes of the octree at this layer;

[0084] Execute step 1223 to randomly sample 256 points in each remaining voxel, and use the average weight value of the sampled points as the value k of the node. The (ρ, k) corresponding to each node is the weight value of the voxel.

[0085] Execute step 1230 to store the calculation result in step 22 according to the encoding rule, where the encoding rule includes:

[0086] 1) The unique identifier of each node is represented by layer L, row R, column C and height H. In order to ensure the retrieval efficiency after storage in the database, it needs to be encoded. The specific method is: L and H are represented by 8 bits, R and C are represented by 24 bits, a total of 64 bits;

[0087] 2) The weight data (ρ, k) of each node is actually a float array, where ρ has only one value and the spherical harmonic weight k is an array. To improve storage efficiency, it is packaged into binary stream storage. Let the size of array k be S(k), and len be the length of the entire array, that is, len = S(k) + 1;

[0088] 3) In the Key-Value database, the weight data of each node in each layer of the octree is stored.

[0089] Execute step 1240, repeat steps 22 and 23 for the nodes of each layer of the octree to generate the octree.

[0090] Execute step 1300 to perform interactive rendering. During the rendering process, according to the position and angle of the current viewpoint, adapt the appropriate level, and calculate the layer-row-column-height range of each level of grid related to the current viewport. The rendering scheduler is responsible for taking out the corresponding grid weight data and handing it over to CUDA for parallel processing. Finally, pictures at different angles are inferred, including the following sub-steps:

[0091] Execute step 1310, calculate the position of the viewport center and the viewport boundary in the world coordinate system according to the position pos and angle dir of the camera in the scene, and the width and height of the viewport, and then project it into the two-dimensional screen coordinate system to obtain the number of pixels (V ) of the width and height of the image to be rendered to the screen. w ,V h ).

[0092] Execute step 1320 to calculate the octree level L. The calculation formula of the octree level L is:

[0093] L = log2 (G / (S*V))

[0094] V=(V w +V h ) / 2

[0095] Among them, G / S represents the number of pixels of the original image in one dimension, and V is the average number of pixels in the width and height of the image.

[0096] Execute step 1330 to extract the row, column, and height range of the L-layer grid, and calculate the intersection of the octree Tree and the bounding box range View of the viewport in the world coordinate system:

[0097] W cut =W tree ∩WView

[0098] W cut Convert the local coordinate system to obtain L cut :

[0099] L cut =W cut *T -1

[0100] Among them, W cut is the bounding box after cropping, W tree is the bounding box of the octree, W View is the bounding box of the current viewport, T is the matrix of octree to world coordinates, L cut is the range to be extracted from the octree (T x ,T y ,T z ), and then according to the level L, the row, column, and height range (R, C, H) of the L-layer grid are obtained, and the number of grids is n.

[0101] Execute step 1340, the rendering scheduler reads the grid weight data from the database, encodes the n grids according to the encoding rule, and uses the data as the key to query the binary data from the database table, and obtains the weight value (ρ, k) of each grid after deserialization.

[0102] Execute step 1350 to synthesize a new view. For the newly generated image, parallel calculation is performed by pixel through CUDA. Each calculation task accumulates the color value of the current pixel according to the direction of the current light and the grid weight (ρ, k), and finally obtains a synthesized image.

[0103] Execute step 1360. During the interactive rendering process, the rendering scheduler is responsible for maintaining the mesh list in the memory. For the requested mesh, the memory is searched first. If the mesh is not found in the memory, it is loaded from the database according to the key and the local mesh list is updated. At the same time, it is responsible for maintaining the local mesh queue and ensuring the stability of the local memory according to the first-in-first-out principle.

[0104] Embodiment 2

[0105] The purpose of the present invention is to introduce NeRF, an implicit neural radiation field technology, into city-level 3D scene modeling, especially for high-definition image data collected by drones, taking into account the technical advantages of NeRF (high fidelity, arbitrary perspective synthesis, saving labor costs, etc.), while realizing interactive browsing of city-level 3D scenes. The innovation of the present invention is to use a sparse octree structure to solve the problem of city-level 3D scene modeling and interactive rendering.

[0106] The flow chart of the present invention is as follows Figure 2 shown.

[0107] Step 1: Partitioning the training data

[0108] like Figure 3 As shown, the entire scene is partitioned, and the center point of each zone is recorded as C i ,N training images, the image width and height are (W, H) pixels, the estimated closest distance is Near, the farthest distance is Far, and each training image is processed as follows:

[0109] For W*H pixels, each pixel to C i The light samples 256 points between (Near, Far) and calculates W*H*256 points to C i The distance from pixel to C is the minimum value of MinDistance. i If the actual distance Dis / MinDistance is less than the threshold (set to 1.15), the pixel participates in C i The training of mask is recorded as 1, otherwise it is recorded as 0; after W*H pixels are processed, a Mask (C i ,Image j ). After processing each partition in turn, the total Mask is obtained.

[0110] Step 2: Parallel training of sub-regions

[0111] like Figure 4 As shown, NeRF-SHs are trained for each sub-region until convergence is reached (about 500,000 iterations).

[0112] The construction of the octree is actually a discretization process. The classic NeRF model cannot directly discretize space. The NeRF-SH network is improved so that the final inferred color and density are only related to the position, but not the observation angle. In this way, only the density and spherical harmonic function coefficients are stored in the voxel.

[0113] Step 3: Generate a sparse octree and store the weight data of each node

[0114] The entire scene range is divided into octrees. The size of each layer of voxels is consistent. From top to bottom, the octrees are divided into finer and finer layers, and a sparse octree with actual distribution is obtained, such as Figure 5 shown.

[0115] Step 3.1 Octree level calculation

[0116] The octree is divided from top to bottom and according to the rules. The division is terminated when L meets the following conditions:

[0117] G / (S*2 L ) <N 1 / 2

[0118] Where G = (G x +G y ) / 2,(G x ,G y ) is the geographical extent (length and width) of the urban scene;

[0119] S is the spatial resolution of the training image;

[0120] N is the threshold, representing the average number of pixels in the grid, and the default value is 10,000.

[0121] Step 3.2 For L l Layer voxels, using the sub-region model in step 2 to calculate the density and spherical harmonic coefficients:

[0122] 1) Evaluation. Use a 3D grid to uniformly sample points and obtain the volume density ρ of each voxel.

[0123] 2) Filtering. Use the perspective of all training images to render the alpha map, record the weight of each voxel on the ray (meaning the opacity of the i-th fragment of the ray), and delete the voxels with weights below the threshold. The voxels that are not deleted are used as nodes of the octree at this layer;

[0124] 3) Sampling. Randomly sample 256 points in each remaining voxel, and use the average weight value of the sampled points as the value of the node k.

[0125] At this point, the (ρ, k) corresponding to each node is the weight value of the voxel.

[0126] Step 3.3 stores the calculation result of step 3.2.

[0127] 1) Each node's unique identifier is represented by L (layer), R (row), C (column), and H (height). To ensure the retrieval efficiency after storage in the database, it needs to be encoded. The specific method is: L and H are represented by 8 bits, R and C are represented by 24 bits (maximum value 16,777,216), a total of 64 bits, the rules are as follows Figure 6 shown.

[0128] 2) The weight data of each node (ρ, k) is actually a float array, where ρ has only one value and the spherical harmonic weight k is an array. To improve storage efficiency, it is packaged into binary stream storage. Let the size of array k be S(k). The stream structure is as follows: Figure 7 As shown, len is the length of the entire array, that is, len=S(k)+1.

[0129] 3) According to the design of 1) and 2) above, the weight data of each node in each layer of the octree is stored in the Key-Value database. The data structure is shown in Table 1.

[0130] Table 1 Node data storage table structure

[0131]

[0132] Step 3.4: Repeat steps 3.2 and 3.3 for each node in each layer of the octree, and the octree generation is complete.

[0133] Step 4: Interactive Rendering

[0134] like Figure 8 As shown, during the rendering process, the appropriate level is adapted according to the position and angle of the current viewpoint, and the layer-row-column-height range of each level of the grid related to the current viewport is calculated. The rendering scheduler is responsible for retrieving the corresponding grid weight data and handing it over to CUDA for parallel processing, and finally inferring pictures at different angles.

[0135] The details are as follows:

[0136] Step 4.1 Calculate the position of the viewport center and the viewport boundary in the world coordinate system based on the position pos and angle dir of the camera (viewpoint) in the scene, as well as the width and height of the viewport, and then project it into the two-dimensional screen coordinate system to obtain the pixel range (V w ,V h ).

[0137] Step 4.2 Calculate the octree level L:

[0138] L = log2 (G / (S*V)) ,V=(V w +V h ) / 2

[0139] G / S represents the number of pixels of the original image in one dimension. See step 3.1. The level selected is to make the original image match the display area size after downsampling in the current display area. This can preserve the image details as much as possible and ensure a good display effect.

[0140] Step 4.3 Extract the row, column, and height range of the L layer grid

[0141] In the world coordinate system, calculate the intersection of the octree Tree and the bounding box range View of the viewport:

[0142] W cut =W tree ∩W View

[0143] W cut Convert the local coordinate system to obtain L cut :

[0144] L cut =W cut *T -1, T is the matrix of octree to world coordinates

[0145] L cut That is the range that needs to be extracted from the octree (T x ,T y ,T z ), and then according to the level L in step 4.2, get the row, column, and height range (R, C, H) of the L-layer grid, and the number of grids is n.

[0146] Step 4.3 The rendering scheduler reads the mesh weight data from the database

[0147] According to the encoding rules in step 3.3, encode n grids and use them as keys to query binary data from the database table. After deserialization, the weight value (ρ, k) of each grid is obtained.

[0148] Step 4.4 New view synthesis

[0149] For the newly generated image, CUDA is used to perform parallel calculations on a pixel-by-pixel basis. Each calculation task accumulates the color value of the current pixel based on the current light direction and grid weight (ρ, k), and finally obtains a composite image.

[0150] Step 4.5 During the interactive rendering process, the rendering scheduler is responsible for maintaining the mesh list in the memory. For the requested mesh, it first searches the memory. If it is not found in the memory, it is loaded from the database according to the key and the local mesh list is updated. It is also responsible for maintaining the local mesh queue according to the first-in-first-out principle to ensure the stability of the local memory.

[0151] The present invention uses LoD+NeRF+Octree technology to achieve interactive rendering of city-level scenes. The LoD technology of city-level scenes is combined with NeRF modeling technology, and corresponds to the sparse octree data structure. During rendering, the viewing cone generated by the current viewing angle is calculated, and the octree level and the grid of the corresponding level and position are estimated. While ensuring the rendering clarity, the additional rendering work is minimized and the rendering efficiency is improved.

[0152] The present invention adopts a key-value database to store the weight data of a sparse octree grid, thereby solving the storage problem of an octree in a very large scene.

[0153] In order to better understand the present invention, the above is described in detail in conjunction with the specific embodiments of the present invention, but it is not intended to limit the present invention. Any simple modifications made to the above embodiments based on the technical essence of the present invention still fall within the scope of the technical solution of the present invention. Each embodiment in this specification focuses on the differences from other embodiments, and the same or similar parts between the embodiments can be referenced to each other. For the system embodiment, since it basically corresponds to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

Claims

1. A NeRF-based method for efficient modeling and visualization of city-level scenes, including training data partitioning, characterized in that: The following steps are also included: Step 1: Parallel training of sub-regions; Step 2: Generate a sparse octree and store the weight data of each node, including the following sub-steps: Step 21: Calculate the octree level, including the top-down and regular division of the octree. The division is terminated when the level L meets the following conditions: G / (S*2 L )<N 1 / 2 Where G = (G x +G y ) / 2 and (G x ,G y ) is the length and width of the geographical range of the urban scene, S is the spatial resolution of the training image, and N is the threshold of the average number of pixels in the grid; Step 22: For L l Layer voxel, sub-region model calculates density and spherical harmonic coefficients; Step 23: storing the calculation result in step 22 according to the coding rule; Step 24: Repeat steps 22 and 23 for the nodes at each layer of the octree to generate the octree; Step 3: Perform interactive rendering.

2. The NeRF-based city-level scene efficient modeling and visualization method according to claim 1, characterized in that: The training data partitioning includes partitioning the entire scene, and the center point of each area is recorded as C i , N training images, the image width and height are (W, H) pixels, the estimated closest distance is Near, the farthest distance is Far, and each training image is processed as follows: For W*H pixels, each pixel to C i The light samples 256 points between (Near, Far) and calculates W*H*256 points to C i The minimum distance is MinDistance; Pixel to C i The actual distance Dis / MinDistance is less than the threshold, then the pixel participates in C i For training, mask is recorded as 1, otherwise it is recorded as 0; After processing W*H pixels, a Mask (C i ,Image j ); process each partition in turn and get the total Mask.

3. The NeRF-based city-level scene efficient modeling and visualization method according to claim 2, characterized in that: The step 2 includes dividing the entire scene range into an octree, with the voxels of each layer having the same size, and dividing the voxels step by step from top to bottom to obtain a sparse octree according to the actual distribution.

4. The NeRF-based city-level scene efficient modeling and visualization method according to claim 3, characterized in that: The step 22 includes the following sub-steps: Step 221: Use 3D grid uniform sampling points to obtain the volume density ρ of the area where each voxel is located; Step 222: Render the alpha map using the viewing angles of all training images, record the weight of each voxel on the ray, and delete the voxels with weights lower than the threshold. The voxels that are not deleted serve as nodes of the octree at this layer. Step 223: Randomly sample 256 points in each remaining voxel, and use the average weight value of the sampled points as the value k of the node. The (ρ, k) corresponding to each node is the weight value of the voxel.

5. The NeRF-based city-level scene efficient modeling and visualization method according to claim 4, characterized in that: The encoding rules include: 1) The unique identifier of each node is represented by layer L, row R, column C and height H. In order to ensure the retrieval efficiency after storage in the database, it needs to be encoded. The specific method is: L and H are represented by 8 bits, R and C are represented by 24 bits, a total of 64 bits; 2) The weight data (ρ, k) of each node is actually a float array, where ρ has only one value and the spherical harmonic weight k is an array. To improve storage efficiency, it is packaged into a binary stream storage. Let the size of array k be S(k), and len be the length of the entire array, that is, len = S(k) + 1; 3) In the Key-Value database, the weight data of each node in each layer of the octree is stored.

6. The NeRF-based city-level scene efficient modeling and visualization method according to claim 5, characterized in that: The step 3 includes the following sub-steps: Step 31: According to the position pos and angle dir of the camera in the scene, and the width and height of the viewport, calculate the position of the viewport center and the viewport boundary in the world coordinate system, and then project it into the two-dimensional screen coordinate system to obtain the number of pixels (V w ,V h ); Step 32: Calculate the octree level L; Step 33: Extract the row, column, and height range of the L-layer grid; Step 34: The rendering scheduler reads the mesh weight data from the database; Step 35: Synthesize a new view; Step 36: During interactive rendering, the rendering scheduler is responsible for maintaining the mesh list in memory. For the requested mesh, it first searches the memory. If it is not found in the memory, it is loaded from the database according to the key and the local mesh list is updated.

7. The NeRF-based city-level scene efficient modeling and visualization method according to claim 6, characterized in that: The calculation formula of the octree level L is: L log2 (G / (S*V)) V=(V w +V h ) / 2 Among them, G / S represents the number of pixels of the original image in one dimension, and V is the average number of pixels in the width and height of the image.

8. The NeRF-based city-level scene efficient modeling and visualization method according to claim 7, characterized in that: The step 33 includes calculating the intersection of the octree Tree and the bounding box range View of the viewport in the world coordinate system: IN cut =In tree ∩In View W cut Convert the local coordinate system to obtain L cut : L cut =W cut *T -1 Among them, W cut is the bounding box after cropping, W tree is the bounding box of the octree, W View is the bounding box of the current viewport, T is the matrix of octree to world coordinates, L cut is the range to be extracted from the octree (T x ,T y ,T z ), and then according to the level L, the row, column, and height range (R, C, H) of the L-layer grid are obtained, and the number of grids is n.

9. The NeRF-based city-level scene efficient modeling and visualization method according to claim 8, characterized in that: The step 34 includes encoding n grids according to the encoding rule, and using the encoding rule as a key to query binary data from a database table, and obtaining the weight value (ρ, k) of each grid after deserialization.

10. The NeRF-based city-level scene efficient modeling and visualization method according to claim 9, characterized in that: The step 35 includes performing parallel calculations on a pixel-by-pixel basis on the newly generated image through CUDA. For each calculation task, the color value of the current pixel is accumulated according to the direction of the current light and the grid weight (ρ, k), and finally a composite image is obtained.

Images