Real-time rendering method and device for 3D scenes based on spatial partitioning and hierarchical tree

Abstract

This invention provides a method and device for real-time rendering of 3D scenes based on spatial partitioning and a hierarchical tree. The method includes initializing the 3D scene and dividing it into several spatial blocks; performing parallel dense optimization of 3D Gaussian primitives within each spatial block to generate a set of 3D Gaussian primitives; constructing a hierarchical tree based on the 3D Gaussian primitive set, ensuring that the screen contribution of intermediate nodes and their child nodes is consistent when viewed from a distance; further optimizing the hierarchical tree at the target screen granularity and performing smooth interpolation of attributes between nodes at adjacent detail levels; finally, integrating the hierarchical trees of each spatial block into a global structure, and performing prefetching scheduling and rendering in conjunction with viewpoint motion trends during rendering. This invention reduces the optimization time of Gaussian primitives by spatially partitioning the 3D scene, and reduces memory usage and improves operating efficiency by constructing a hierarchical tree and performing detail level optimization. This invention can better achieve real-time rendering of 3D scenes.

Description

Technical Field

[0001] This invention relates to the field of scene rendering technology, and in particular to a method and device for real-time rendering of three-dimensional scenes based on spatial segmentation and hierarchical trees. Background Technology

[0002] With the rapid development of technologies such as virtual reality, digital twins, autonomous driving, and smart cities, the reconstruction and real-time rendering of large-scale 3D scenes in the real world has become a core research direction in the fields of computer vision and computer graphics.

[0003] Currently, most 3D scene rendering uses 3D reconstruction methods based on implicit neural representations (such as NeRF, Neural Radiance Field). This approach mainly uses neural networks to implicitly represent 3D scenes and synthesizes new perspective images by volume rendering. However, these methods heavily rely on dense spatial sampling and volume rendering integral calculations, resulting in training processes that can take tens or even hundreds of hours, and rendering speeds that are extremely slow (non-real-time), making it difficult to scale up to ultra-large-scale real street scenes at the kilometer level.

[0004] Therefore, existing technologies still need to be improved and developed. Summary of the Invention

[0005] This invention provides a method and device for real-time rendering of three-dimensional scenes based on spatial segmentation and hierarchical trees. The main purpose of this invention is to solve the technical problems mentioned in the background art of the prior art.

[0006] The first aspect of this invention provides a real-time rendering method for 3D scenes based on spatial partitioning and hierarchical trees, comprising: A full-scene coarse scan initialization is performed on the 3D scene, which is then divided into several spatial blocks, and a corresponding supervisory camera view is assigned to each spatial block. Dense optimization of 3D Gaussian elements is performed within each of the aforementioned spatial blocks to generate a set of 3D Gaussian elements for each of the aforementioned spatial blocks; For each of the 3D Gaussian meta-sets in the spatial blocks, a hierarchical tree is constructed by first dividing the space from top to bottom and then merging the attributes from bottom to top, so that each intermediate node of the generated hierarchical tree has a consistent screen contribution with all its child nodes when viewed from a distance. The hierarchical tree is further optimized at the level of detail based on the target screen granularity, and smooth interpolation of node attributes is performed between adjacent levels; The hierarchical trees optimized by each spatial block are integrated into a global hierarchical structure, and in the real-time rendering stage, the prefetching scheduling of nodes is performed in combination with the viewpoint movement trend.

[0007] In an optional embodiment of the first aspect of the present invention, the step of performing density optimization of 3D Gaussian elements within each of the spatial blocks to generate a set of 3D Gaussian elements for each of the spatial blocks includes: A densification strategy triggered by maximum gradient is adopted, and the densification criterion of 3D Gaussian primitives is set as the maximum value of the screen space gradient. The monocular depth estimation results after scale and offset alignment are used as the weakly supervised anchoring geometry. A pre-defined affine color matrix is ​​learned for each input image to perform learnable exposure compensation, reducing the interference of exposure differences between multiple cameras and at multiple times on geometric fitting.

[0008] In an optional embodiment of the first aspect of the present invention, the construction of a hierarchical tree for the 3D Gaussian element set for each of the spatial blocks, first through top-down spatial partitioning and then through bottom-up attribute merging, includes: First, perform a top-down AABB-BVH spatial partitioning. Take the axis-aligned bounding box formed by all 3D Gaussian elements in the spatial block as the root node. Recursively split the current node's bounding box in half according to the median of the longest axis until each leaf node contains only a single 3D Gaussian element, thus determining the tree's topological relationship. Then, a bottom-up attribute weighted merging process is performed to generate intermediate node parameters. Based on the screen contribution equivalence constraint, the weights determined by the opacity of the child node and its screen projection area at the reference distance are used to weight and merge the various attributes of the child node to generate the corresponding attributes of the intermediate node.

[0009] In an optional embodiment of the first aspect of the present invention, the step of performing detailed level re-optimization of the hierarchical tree based on the target screen granularity and smooth interpolation of node attributes between adjacent levels includes: During the optimization process, the leaf nodes at the bottom layer are frozen, the target screen granularity is randomly sampled and traversed from top to bottom along the hierarchical tree to find cutting nodes that meet the target screen granularity requirements as cutting lines to replace leaf nodes in the rendering calculation. When adjacent levels switch due to viewpoint movement, power interpolation smoothing is applied to the transparency of nodes between adjacent levels.

[0010] In an optional embodiment of the first aspect of the present invention, the real-time rendering method for a three-dimensional scene further includes: During the optimization process, the proportion of frames actually selected by the cutting line for each node out of all training frames is counted and recorded as the cumulative visibility. The screen contribution error change caused by deleting the node from the parent node is evaluated. When the screen contribution error change is lower than a preset cutting redundancy threshold and the cumulative visibility of the node is lower than a set threshold, the node is determined to meet the cutting redundancy condition, and a pruning deletion operation is performed on the node.

[0011] In an optional embodiment of the first aspect of the present invention, integrating the hierarchical trees optimized from each spatial block into a global hierarchical structure includes: The hierarchical trees optimized from each spatial block are combined into a global tree, and it is checked whether the same 3D Gaussian primitive is repeatedly assigned at the boundary of adjacent spatial blocks. If present, a weighted comparison is performed based on the distance between the position of the 3D Homo sapiens and the center of each adjacent spatial block. The 3D Homo sapiens on the side with the closer distance are retained, and the 3D Homo sapiens on the other side with the farther distance are deleted, in order to avoid double counting in the boundary area, which would cause the rendering to be too bright.

[0012] In an optional embodiment of the first aspect of the present invention, the step of performing prefetch scheduling of nodes in conjunction with viewpoint movement trends during the real-time rendering stage includes: During the real-time rendering phase, the camera's current position, motion speed, and orientation are obtained to predict the adjacent spatial blocks that the view frustum is about to enter. The parent nodes of adjacent spatial blocks are asynchronously moved from CPU memory to GPU memory in advance; As the viewpoint approaches the boundary of the adjacent spatial block, intermediate nodes and leaf nodes are successively loaded into the GPU memory, while leaf nodes that have exited the view frustum range in the current spatial block are unloaded from the GPU memory.

[0013] A second aspect of the present invention provides a real-time rendering system for three-dimensional scenes based on spatial segmentation and hierarchical trees, the real-time rendering system for three-dimensional scenes comprising: The scene segmentation module is used to perform a full-scene coarse scan initialization of the 3D scene, divide the 3D scene into several spatial blocks, and assign a corresponding supervisory camera view to each spatial block; The primitive optimization module is used to perform dense optimization of 3D Holonomic primitives within each of the spatial blocks, and generate a set of 3D Holonomic primitives for each of the spatial blocks. The hierarchical tree construction module is used to construct a hierarchical tree for the 3D Gaussian element set of each spatial block by first dividing the space from top to bottom and then merging the attributes from bottom to top, so that each intermediate node of the generated hierarchical tree has a consistent screen contribution with all its child nodes when viewed from a distance. The hierarchical re-optimization module is used to perform detailed hierarchical re-optimization of the hierarchical tree based on the target screen granularity, and to perform smooth interpolation of node attributes between adjacent levels. The global scheduling rendering module is used to integrate the hierarchical trees after optimization of each spatial block into a global hierarchical structure, and to perform prefetching scheduling of nodes in combination with viewpoint movement trends during the real-time rendering stage.

[0014] A third aspect of the present invention provides a real-time rendering device for a three-dimensional scene based on spatial block and hierarchical tree, the real-time rendering device for a three-dimensional scene based on spatial block and hierarchical tree includes: a memory and at least one processor, the memory stores instructions, and the memory and the at least one processor are interconnected by a line. The at least one processor invokes the instructions in the memory to cause the three-dimensional scene real-time rendering device based on spatial block and hierarchical tree to execute the three-dimensional scene real-time rendering method based on spatial block and hierarchical tree as described in any one of the first aspects of the present invention.

[0015] The fourth aspect of the present invention provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the real-time rendering method for a three-dimensional scene based on spatial segmentation and hierarchical tree as described in any one of the first aspects of the present invention.

[0016] Beneficial Effects: This invention provides a real-time rendering method and device for 3D scenes based on spatial partitioning and a hierarchical tree. The method includes initializing the 3D scene and dividing it into several spatial blocks; performing parallel dense optimization of 3D Gaussian primitives within each spatial block to generate a set of 3D Gaussian primitives; constructing a hierarchical tree based on the 3D Gaussian primitive set, ensuring that the screen contribution of intermediate nodes and their child nodes is consistent when viewed from a distance; further optimizing the hierarchical tree at the target screen granularity and performing smooth interpolation of attributes between nodes at adjacent detail levels; finally, integrating the hierarchical trees of each spatial block into a global structure, and performing prefetching scheduling and rendering in conjunction with viewpoint motion trends during rendering. This invention reduces the optimization time of Gaussian primitives by spatially partitioning the 3D scene, and reduces memory usage and improves operating efficiency by constructing a hierarchical tree and performing detail level optimization. This invention can better achieve real-time rendering of 3D scenes. Attached Figure Description

[0017] Figure 1 This is a schematic diagram of an embodiment of a real-time rendering method for 3D scenes based on spatial segmentation and hierarchical trees according to the present invention; Figure 2 This is a top-down schematic diagram of an embodiment of the present invention, which involves scene segmentation, primitive optimization, and global integration. Figure 3This is a schematic diagram of an embodiment of the present invention, which dynamically selects granularity nodes for rendering based on camera distance. Figure 4 This is a schematic diagram of an embodiment comparing the technical effects that the present invention can achieve in terms of various indicators; Figure 5 This is a schematic diagram of an embodiment of a real-time rendering system for 3D scenes based on spatial segmentation and hierarchical trees according to the present invention; Figure 6 This is a schematic diagram of an embodiment of a real-time rendering device for a 3D scene based on spatial segmentation and hierarchical tree according to the present invention. Detailed Implementation

[0018] The terms "first," "second," "third," "fourth," etc. (if present) in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" or "having" and any variations thereof are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0019] The first aspect of this invention provides a real-time rendering method for 3D scenes based on spatial partitioning and hierarchical trees. This method can be applied to real-time, high-quality rendering of ultra-large-scale 3D scenes (such as city street scenes spanning several kilometers). See [link to relevant documentation]. Figure 1 The real-time rendering method for the three-dimensional scene includes: S100. Perform a full-scene coarse scan initialization on the 3D scene, divide the 3D scene into several spatial blocks, and assign a corresponding supervisory camera viewpoint to each spatial block.

[0020] Specifically, in this invention, this step can utilize the Structure from Motion (SfM) algorithm to extract sparse 3D point clouds from multi-view images, estimate camera poses, and perform a lightweight 3D 3D Gaussian Splatting optimization using all cameras and the SfM sparse point cloud (the primitive positions need to be frozen and densification disabled during this optimization process). Furthermore, to handle distant scenes such as the sky, this invention can also add a skybox with a radius approximately 10 times the physical diameter of the scene as a unified background shared by all blocks to handle background rendering at infinity.

[0021] Subsequently, based on the scene type and scale, the scene is divided into multiple blocks on the horizontal plane. For example, the 3D scene is spatially physically divided into rectangular grids (e.g., the walking scene is divided into 50×50 meters / block, and the vehicle-mounted scene is divided into 100×100 meters / block). When allocating cameras, the allocation principle can be to select cameras located inside the spatial block, or cameras that are no more than twice the side length from the boundary of the spatial block and whose view frustum can observe a sufficient number (e.g., ≥50) of SfM points within the block, in order to prepare data for subsequent parallel supervised training.

[0022] S200. Perform density optimization of 3D Gaussian elements within each spatial block to generate a set of 3D Gaussian elements for each spatial block. In this invention, this step may include: employing a maximum gradient-triggered densification strategy, setting the densification criterion of 3D Gaussian elements to the maximum value of the screen spatial gradient; introducing the monocular depth estimation result after scale and offset alignment as a weakly supervised anchored geometry; learning a preset-sized affine color matrix for each input image to perform learnable exposure compensation and reduce the interference of exposure differences between multiple cameras and at multiple times on geometric fitting.

[0023] Specifically, this step of the invention involves independently and in parallel performing the improved 3DGS dense optimization (using a large number of 3D Gaussian primitives with color, transparency, and ellipsoidal shapes to explicitly represent the scene, enabling real-time rendering; in traditional 3DGS, the number of primitives per scene increases linearly with the scene size, leading to a rapid increase in memory usage in large scenes) within each spatial block. Addressing the issue of holes easily appearing in low-texture areas (such as roads and white walls) in sparsely acquired data from vehicles or helmets, the invention employs the following joint optimization mechanism: Maximum gradient densification: Unlike the original 3DGS which uses the average gradient in screen space, this invention uses the maximum gradient value as the criterion for triggering primitive cloning or splitting. This makes the method more sensitive to low-texture areas (such as roads and walls) that lack sufficient information in sparse viewpoints, and can more effectively fill holes and improve reconstruction integrity.

[0024] Monocular Depth Anchoring: A pre-trained monocular depth estimation algorithm is run in parallel on each input image to obtain the corresponding depth map. These depth maps are then aligned by scale and offset and added as a weakly supervised regularization term to the optimization objective of 3DGS. This can provide additional constraints for geometric convergence and effectively prevent model floating or morphological errors in regions with sparse viewpoints or geometric blur.

[0025] Learnable Exposure Compensation: To address the inconsistency in exposure and white balance caused by multi-camera and multi-time-period acquisition, this invention associates a learnable 3×4 affine color transformation matrix with each input image. Before calculating the loss function between the rendered image and the ground truth image, this matrix is ​​used to perform color correction on the rendered result. This matrix is ​​optimized along with the 3D Gaussian trigonometric parameters during training, thus effectively compensating for luminance differences without contaminating geometric learning. Through these three steps, this invention can generate a high-quality and dense set of 3D Gaussian trigonometric elements in parallel for each spatial block.

[0026] S300. For the set of 3D Gaussian elements in each spatial block, a hierarchical tree is constructed by first performing top-down spatial partitioning and then bottom-up attribute merging, so that each intermediate node of the generated hierarchical tree has a consistent screen contribution with all its child nodes when viewed from a distance. In this invention, this step may include: first performing top-down AABB-BVH (axis-aligned bounding box hierarchical tree) spatial partitioning, taking the axis-aligned bounding box (AABB) formed by all 3D Gaussian elements in the spatial block as the root node, recursively splitting the current node's bounding box in half according to the longest axis median, until each leaf node contains only a single 3D Gaussian element, determining the tree's topological relationship; then performing bottom-up attribute weighted merging to generate intermediate node parameters, based on screen contribution equivalence constraints, using weights determined by the opacity of the child node and its screen projection area at a reference distance, weighted merging of various attributes of the child node to generate the corresponding attributes of the intermediate node.

[0027] Specifically, after block optimization is completed, due to the large number of 3D Gaussian elements, this invention proposes a hierarchical tree construction method based on the "screen contribution equivalence" constraint. The method mainly consists of two stages: The first stage (top-down spatial partitioning) adopts the AABB-BVH (axis-aligned bounding box hierarchy tree) structure. With all 3D Gothic elements as leaf nodes, the current node's bounding box is recursively halved according to the median of the longest axis until the leaf node contains only a single 3D Gothic element. This stage is used to determine the tree's topology.

[0028] Phase Two (Bottom-Up Attribute Weighted Merging): Generate intermediate node parameters and set the merging weight w. _i =o _i ×S _i , (where w) _i Let o be the merge weight of the i-th child node. _i Let S be the opacity of the i-th child node. _i (where is the screen projection area of ​​the ellipsoid of the i-th child node at the reference distance), the position, covariance, scale, rotation, and spherical harmonic color coefficients of the intermediate nodes are all based on w. _i Weighted fusion is performed, and the opacity of intermediate nodes is calculated according to formula α. _parent =1 ∏(1 α _i ) Synthesis, where α _parent α represents the opacity of the intermediate node. _i Let represent the opacity of the i-th child node, and ∏ represent the product of all child nodes. By designing this formula, it can be strictly guaranteed that when viewed from a distance, the screen visual contribution of a single intermediate node is highly consistent with that of the original set of quantum nodes. That is, when viewed from a distance, the occlusion effect of the intermediate node is equivalent to the superposition effect of all its child nodes. Through the above two stages, the present invention can finally form a multi-branch tree in which each intermediate node has 2 to 8 child nodes.

[0029] S400. The hierarchical tree is further optimized at the level of detail based on the target screen granularity, and smooth interpolation of node attributes is performed between adjacent levels. In this invention, this step may include: during the optimization process, freezing the leaf nodes at the bottom layer, randomly sampling the target screen granularity and traversing the hierarchical tree from top to bottom to find cutting nodes that meet the target screen granularity requirements as cutting lines to replace leaf nodes in rendering calculations; when adjacent levels switch due to viewpoint movement, power-law interpolation smoothing is performed on the transparency of nodes between adjacent levels.

[0030] Specifically, in this invention, to eliminate image jumps and aliasing during LOD (Level of Detail, which switches models with different precision based on camera distance, using coarser models for distant objects and finer models for closer objects to save resources) level switching, this invention proposes a differentiable cutting and smooth interpolation mechanism. During optimization training, the bottom-level leaf nodes are frozen, and only intermediate nodes are optimized. The target screen granularity (unit: pixels) is randomly sampled. Based on the target screen granularity, suitable nodes are found from top to bottom along the tree as "cutting lines" to replace leaves for rendering. The loss is calculated with the real image, and the parameters of the intermediate nodes are updated through backpropagation. When switching between adjacent levels, this invention creatively uses a power interpolation formula α′=1 for transparency. (1) α _p )^(1 / K), where α _p K represents the transparency of the intermediate node, and K represents the number of child nodes. The smooth transition mechanism described above can greatly reduce the flickering phenomenon when the camera zooms in / out during primitive switching.

[0031] In this invention, to further reduce video memory usage, offline pruning can be performed after the detail-level optimization steps. The offline pruning steps may include: during the optimization process, counting the frequency at which each node is actually selected by the cutting line and recording it as cumulative visibility; evaluating the change in screen contribution error caused by deleting the node from the parent node; when the change in screen contribution error is lower than a preset cutting redundancy threshold and the cumulative visibility of the node is lower than a set threshold, determining that the node meets the cutting redundancy condition, and performing a pruning deletion operation on the node.

[0032] Specifically, in the detailed-level re-optimization process of this invention, the system records the frequency at which each node is hit by the "cutting line" (the proportion of the actual selected frames to all training frames), i.e., the cumulative visibility. Then, a visibility threshold (e.g., 0.1%) and a cutting redundancy threshold (e.g., 0.5%) are set. When the visibility of a node is lower than the threshold, and after removing the node from the child list of its parent node, the screen contribution change caused by recalculating the parent node's attributes is less than the cutting redundancy threshold, the node is considered redundant and is safely pruned and deleted. For example, if the visibility of some fine nodes (extremely distant street corner building details) is extremely low (e.g., 0.07%), and the screen contribution error caused by simulating their removal from the parent node is less than the preset cutting redundancy threshold (0.5%), then it is determined to be a redundant node and physically removed. This method of the present invention can significantly reduce the overall model size with almost no loss of visual quality.

[0033] S500: Integrate the optimized hierarchical trees of each spatial block into a global hierarchical structure, and perform node prefetching scheduling in conjunction with viewpoint movement trends during the real-time rendering stage. In this invention, step S500 includes: merging the optimized hierarchical trees of each spatial block into a global tree, and checking whether the same 3D high-order primitive is repeatedly assigned at the boundary of adjacent spatial blocks; if so, performing a weighted comparison based on the distance between the position of the 3D high-order primitive and the center of each adjacent spatial block, retaining the 3D high-order primitive on the side with the closer distance, and deleting the 3D high-order primitive on the other side with the farther distance, to avoid double counting in the boundary area leading to overly bright rendering.

[0034] Specifically, in this invention, after all spatial blocks obtain optimized hierarchical trees, the invention will merge the subtrees generated by each spatial block into a global tree (such as...). Figure 2 As shown in the figure, and in response to the problem of excessively bright rendering caused by primitive overlap at the junction boundary, this invention calculates the distance of overlapping 3D high-order primitives from the center of their respective blocks. For example, if there are duplicate 3D high-order primitives at the junction of block A and block B, and if the 3D high-order primitive is closer to the center of block A, then the 3D high-order primitive is retained as belonging to block A, and the corresponding duplicate 3D high-order primitive in block B is deleted.

[0035] Furthermore, in this invention, step S500 further includes: during the real-time rendering stage, obtaining the current position, motion speed, and orientation of the camera, and predicting the adjacent spatial blocks that the view frustum is about to enter; asynchronously moving the parent nodes of the adjacent spatial blocks from CPU memory to GPU memory in advance; as the viewpoint approaches the boundary of the adjacent spatial blocks, loading intermediate nodes and leaf nodes into the GPU memory one after another, and simultaneously unloading the leaf nodes that have exited the view frustum range in the current spatial block from the GPU memory.

[0036] Specifically, during runtime, to achieve kilometer-scale scene roaming with limited video memory, this invention can employ a speed-driven video memory prefetching strategy. For example, when the camera is currently roaming within block A, the system predicts, based on the camera's movement speed and orientation vector, that the view frustum will enter block B in the first instant (0.5s). This invention will pre-load the coarse-grained nodes (parent nodes) of block B from CPU memory to GPU memory in a background thread. As the camera approaches the boundary, medium- and fine-grained nodes (intermediate-grained nodes and leaf nodes) of block B are loaded sequentially. Simultaneously, fine-grained nodes (leaf nodes) in block A that have exited behind the camera's view frustum are immediately released from GPU memory. Through this dynamic scheduling, the number of actively rendered 3D Gaussian primitives can be dynamically controlled within the range of 3%–66% of the original 3DGS scheme, ensuring a stable rendering frame rate and no significant screen abruptness or stuttering during viewpoint switching. The schematic diagram of this invention's granular node rendering based on camera distance and the comparison diagram of the overall technical effects achieved by the scheme are shown below. Figure 3 and Figure 4 As shown.

[0037] In summary, the real-time rendering scheme for 3D scenes based on block-based and hierarchical trees in this invention includes at least the following technical points: Technical Point 1: A 3D Gaussian hierarchical tree construction and bottom-up merging method based on the "screen contribution equivalence" constraint. Leaf nodes are the original 3D Gaussian primitives, and intermediate nodes are formed by their child nodes according to weight w. _i =o _i ×S _i It was formed by merging, of which o _i For child node opacity, S _i The projection area of ​​its ellipsoid on the screen is used to ensure that the intermediate node has a consistent or approximately consistent screen contribution with its set of child nodes when viewed from a distance.

[0038] Technical Point 2: Differentiable LOD (Level of Detail) cut selection based on target screen granularity and parent-child smooth interpolation mechanism. During rendering, a "cutting line" is selected from top to bottom along the tree to replace the leaves for rendering; attributes are interpolated between adjacent LOD layers, with power interpolation α′=1 used for transparency. (1) α _p )^(1 / K), α _p The parent node's transparency is set to K, and the number of the parent node's children is set to eliminate the jumps and flickering during LOD switching.

[0039] Technical point 3: The joint optimization method of "maximum gradient densification + monocular depth anchoring + learnable exposure compensation" adapted to sparse acquisition changes the original 3DGS densification criterion from "screen space gradient mean" to "maximum value". Monocular depth estimation is used as weakly supervised anchor geometry after scale-offset alignment. A 3×4 affine color matrix is ​​learned for each image to compensate for photometric differences and reduce the interference of exposure inconsistency on the geometric fitting process.

[0040] Technical point 4: A hierarchical maintenance and prefetching scheduling method based on node visibility statistics and viewpoint motion prediction. In the hierarchical compression stage, pruning is performed based on the cumulative visibility of nodes and the relationship between cutting redundancy. In the real-time rendering stage, nodes that are about to enter the view frustum are prefetched based on the camera motion direction, so as to balance memory control and rendering stability during continuous viewpoint movement.

[0041] The beneficial effects of the real-time 3D scene rendering scheme based on block and hierarchical tree in this invention are as follows: 1. Under typical consumer-grade single-card GPU configurations (such as RTX 4080), this method can support real-time interactive rendering of street scenes spanning several kilometers and tens of thousands of images.

[0042] 2. Dynamic LOD can reduce the active Gaussian element count to 3%–66% of the original 3DGS, thereby reducing memory usage and improving performance.

[0043] 3. Block-based parallel training helps to compress the training cycle of ultra-large scenarios from long training times on multiple GPUs to a few hours on a single machine.

[0044] 4. The use of alpha power interpolation for layer transitions can reduce flickering during LOD switching and improve image quality stability under coarse-grained conditions.

[0045] 5. For low-texture areas such as roads and walls under sparse acquisition conditions, this method helps to improve reconstruction integrity and image quality indicators.

[0046] 6. By using node visibility constraints and motion direction prefetching mechanisms, the sudden changes in detail loading during viewpoint movement can be mitigated, thus improving rendering continuity.

[0047] See Figure 5 The second aspect of the present invention provides a real-time rendering system for three-dimensional scenes based on spatial segmentation and hierarchical trees, the real-time rendering system for three-dimensional scenes comprising: Scene segmentation module 10 is used to perform a full-scene coarse scan initialization of the 3D scene, divide the 3D scene into several spatial blocks, and assign a corresponding supervisory camera view to each spatial block. The primitive optimization module 20 is used to perform dense optimization of 3D Holo primitives in each of the spatial blocks to generate a set of 3D Holo primitives for each of the spatial blocks. The hierarchical tree construction module 30 is used to construct a hierarchical tree for the 3D Gaussian element set of each spatial block by first dividing the space from top to bottom and then merging the attributes from bottom to top, so that each intermediate node of the generated hierarchical tree has a consistent screen contribution with all its child nodes when viewed from a distance. The hierarchical re-optimization module 40 is used to perform detailed hierarchical re-optimization of the hierarchical tree based on the target screen granularity, and to perform smooth interpolation of node attributes between adjacent levels. The global scheduling rendering module 50 is used to integrate the hierarchical trees after spatial block optimization into a global hierarchical structure, and to perform prefetching scheduling of nodes in combination with viewpoint movement trends during the real-time rendering stage.

[0048] In an optional embodiment of the second aspect of the present invention, the primitive optimization module includes: The strategy configuration unit is used to adopt a densification strategy triggered by the maximum gradient, and set the densification criterion of 3D Gaussian primitives to the maximum value of the screen space gradient. The supervised introduction unit is used to introduce the scale- and offset-aligned monocular depth estimation results as a weakly supervised anchoring geometry. The exposure compensation unit is used to learn a preset-sized affine color matrix for each input image to perform learnable exposure compensation, reducing the interference of exposure differences between multiple cameras and at multiple times on geometric fitting.

[0049] In an optional embodiment of the second aspect of the present invention, the hierarchical tree construction module includes: The spatial partitioning unit is used to first perform a top-down AABB-BVH spatial partitioning. The axis-aligned bounding box formed by all 3D Gaussian elements in the spatial block is taken as the root node. The current node's bounding box is recursively halved according to the median of the longest axis until each leaf node contains only a single 3D Gaussian element, thus determining the tree's topological relationship. The attribute weighting unit is used to perform bottom-up attribute weighting and merging to generate intermediate node parameters. Based on the screen contribution equivalence constraint, it uses a weight determined by the opacity of the child node and its screen projection area at the reference distance to weight and merge the various attributes of the child node to generate the corresponding attributes of the intermediate node.

[0050] In an optional embodiment of the second aspect of the present invention, the hierarchical re-optimization module includes: The node cutting unit is used to freeze the leaf nodes at the bottom layer during the optimization process, randomly sample the target screen granularity and traverse the hierarchical tree from top to bottom to find cutting nodes that meet the target screen granularity requirements as cutting lines to replace leaf nodes in the rendering calculation. The interpolation smoothing unit is used to perform power interpolation smoothing on the transparency of nodes between adjacent levels when adjacent levels switch due to viewpoint movement.

[0051] In an optional embodiment of the second aspect of the present invention, the real-time rendering method for a three-dimensional scene further includes a layer compression module, the layer compression module comprising: The visibility statistics unit is used to count the proportion of frames actually selected by the cutting line for each node during the optimization process, which is recorded as the cumulative visibility. The pruning processing unit is used to evaluate the change in screen contribution error caused by deleting the node from the parent node. When the change in screen contribution error is lower than a preset cutting redundancy threshold and the cumulative visibility of the node is lower than a set threshold, it is determined that the node meets the cutting redundancy condition, and the pruning deletion operation is performed on the node.

[0052] In an optional embodiment of the second aspect of the present invention, the global scheduling rendering module includes: The primitive checking unit is used to combine the hierarchical trees optimized from each spatial block into a global tree, and to check whether the same 3D Gaussian primitive is repeatedly assigned at the boundary of adjacent spatial blocks. The primitive deduplication unit is used to perform a weighted comparison based on the distance between the position of the 3D high-order primitive and the center of each adjacent spatial block, if such a primitive exists. The 3D high-order primitive on the side with the closer distance is retained, and the 3D high-order primitive on the other side with the farther distance is deleted, so as to avoid double counting in the boundary area, which would cause the rendering to be too bright.

[0053] In an optional embodiment of the second aspect of the present invention, the global scheduling rendering module includes: The view frustum prediction unit is used to obtain the current position, motion speed and orientation of the camera during the real-time rendering stage, and predict the adjacent spatial blocks that the view frustum is about to enter. The prefetch scheduling unit is used to asynchronously move the parent nodes of adjacent spatial blocks from CPU memory to GPU memory in advance; and to load intermediate nodes and leaf nodes into GPU memory one after another as the viewpoint approaches the boundary of adjacent spatial blocks, while simultaneously unloading leaf nodes that have exited the view frustum range in the current spatial block from GPU memory.

[0054] Figure 6This is a schematic diagram of the structure of a real-time 3D scene rendering device based on spatial block and hierarchical tree provided by an embodiment of the present invention. This real-time 3D scene rendering device based on spatial block and hierarchical tree can vary considerably due to different configurations or performance. It may include one or more processors 60 (central processing units, CPUs) (e.g., one or more processors) and memory 70, and one or more storage media 80 (e.g., one or more mass storage devices) for storing applications or data. The memory and storage media can be temporary or persistent storage. The program stored in the storage media may include one or more modules (not shown in the diagram), each module may include a series of instruction operations on the real-time 3D scene rendering device based on spatial block and hierarchical tree. Furthermore, the processor may be configured to communicate with the storage media and execute the series of instruction operations in the storage media on the real-time 3D scene rendering device based on spatial block and hierarchical tree.

[0055] The real-time 3D scene rendering device based on spatial segmentation and hierarchical trees of this invention may also include one or more power supplies 90, one or more wired or wireless network interfaces 100, one or more input / output interfaces 110, and / or one or more operating systems, such as Windows Server, Mac OS X, Unix, Linux, FreeBSD, etc. Those skilled in the art will understand that... Figure 6 The illustrated structure of a real-time 3D scene rendering device based on spatial partitioning and hierarchical trees does not constitute a limitation on real-time 3D scene rendering devices based on spatial partitioning and hierarchical trees. It may include more or fewer components than illustrated, or combine certain components, or have different component arrangements.

[0056] The present invention also provides a computer-readable storage medium, which can be a non-volatile computer-readable storage medium or a volatile computer-readable storage medium, wherein the computer-readable storage medium stores instructions that, when the instructions are executed on a computer, cause the computer to perform the steps of the real-time rendering method for a three-dimensional scene based on spatial partitioning and hierarchical trees.

[0057] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working process of the system or system / unit described above can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0058] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0059] The above-described embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A real-time rendering method for 3D scenes based on spatial partitioning and hierarchical trees, characterized in that, include: A full-scene coarse scan initialization is performed on the 3D scene, which is then divided into several spatial blocks, and a corresponding supervisory camera view is assigned to each spatial block. Dense optimization of 3D Gaussian elements is performed within each of the aforementioned spatial blocks to generate a set of 3D Gaussian elements for each of the aforementioned spatial blocks; For each of the 3D Gaussian meta-sets in the spatial blocks, a hierarchical tree is constructed by first dividing the space from top to bottom and then merging the attributes from bottom to top, so that each intermediate node of the generated hierarchical tree has a consistent screen contribution with all its child nodes when viewed from a distance. The hierarchical tree is further optimized at the level of detail based on the target screen granularity, and smooth interpolation of node attributes is performed between adjacent levels; The hierarchical trees optimized by each spatial block are integrated into a global hierarchical structure, and in the real-time rendering stage, the prefetching scheduling of nodes is performed in combination with the viewpoint movement trend. The step of performing dense optimization of 3D Gaussian elements within each of the spatial blocks to generate a set of 3D Gaussian elements for each spatial block includes: A densification strategy triggered by maximum gradient is adopted, and the densification criterion of 3D Gaussian primitives is set as the maximum value of the screen space gradient. The monocular depth estimation results after scale and offset alignment are introduced as weak supervision to anchor the geometry; For each input image, a pre-defined affine color matrix is ​​learned to perform learnable exposure compensation, reducing the interference of exposure differences between multiple cameras and at multiple times on geometric fitting. The step of integrating the optimized hierarchical trees of each spatial block into a global hierarchical structure includes: The hierarchical trees optimized from each spatial block are combined into a global tree, and it is checked whether the same 3D Gaussian primitive is repeatedly assigned at the boundary of adjacent spatial blocks. If they exist, a weighted comparison is performed based on the distance between the position of the 3D Homo sapiens and the center of each adjacent spatial block. The 3D Homo sapiens on the side with the closer distance are retained, and the 3D Homo sapiens on the other side with the farther distance are deleted, in order to avoid double counting in the boundary area, which would cause the rendering to be too bright. The prefetching scheduling of execution nodes in the real-time rendering phase, combined with viewpoint movement trends, includes: During the real-time rendering phase, the camera's current position, motion speed, and orientation are obtained to predict the adjacent spatial blocks that the view frustum is about to enter. The parent nodes of adjacent spatial blocks are asynchronously moved from CPU memory to GPU memory in advance; As the viewpoint approaches the boundary of the adjacent spatial block, intermediate nodes and leaf nodes are successively loaded into the GPU memory, while leaf nodes that have exited the view frustum range in the current spatial block are unloaded from the GPU memory.

2. The real-time rendering method for 3D scenes based on spatial segmentation and hierarchical trees according to claim 1, characterized in that, The 3D Gaussian meta-sets for each of the aforementioned spatial blocks are first constructed through top-down spatial partitioning and then through bottom-up attribute merging to obtain a hierarchical tree, including: First, perform a top-down AABB-BVH spatial partitioning. Take the axis-aligned bounding box formed by all 3D Gaussian elements in the spatial block as the root node. Recursively split the current node's bounding box in half according to the median of the longest axis until each leaf node contains only a single 3D Gaussian element, thus determining the tree's topological relationship. Then, a bottom-up attribute weighted merging process is performed to generate intermediate node parameters. Based on the screen contribution equivalence constraint, the weights determined by the opacity of the child node and its screen projection area at the reference distance are used to weight and merge the various attributes of the child node to generate the corresponding attributes of the intermediate node.

3. The real-time rendering method for 3D scenes based on spatial segmentation and hierarchical trees according to claim 1, characterized in that, The step of performing detailed hierarchical re-optimization of the hierarchical tree based on the target screen granularity and smooth interpolation of node attributes between adjacent levels includes: During the optimization process, the leaf nodes at the bottom layer are frozen, the target screen granularity is randomly sampled and traversed from top to bottom along the hierarchical tree to find cutting nodes that meet the target screen granularity requirements as cutting lines to replace leaf nodes in the rendering calculation. When adjacent levels switch due to viewpoint movement, power interpolation smoothing is applied to the transparency of nodes between adjacent levels.

4. The real-time rendering method for 3D scenes based on spatial segmentation and hierarchical trees according to claim 3, characterized in that, The real-time rendering method for 3D scenes also includes: During the optimization process, the proportion of frames actually selected by the cutting line for each node out of all training frames is counted and recorded as the cumulative visibility. The screen contribution error change caused by deleting the node from the parent node is evaluated. When the screen contribution error change is lower than a preset cutting redundancy threshold and the cumulative visibility of the node is lower than a set threshold, the node is determined to meet the cutting redundancy condition, and a pruning deletion operation is performed on the node.

5. A real-time rendering system for 3D scenes based on spatial segmentation and hierarchical trees, characterized in that, The real-time rendering system for the three-dimensional scene includes: The scene segmentation module is used to perform a full-scene coarse scan initialization of the 3D scene, divide the 3D scene into several spatial blocks, and assign a corresponding supervisory camera view to each spatial block; The primitive optimization module is used to perform dense optimization of 3D Holonomic primitives within each of the spatial blocks, and generate a set of 3D Holonomic primitives for each of the spatial blocks. The hierarchical tree construction module is used to construct a hierarchical tree for the 3D Gaussian element set of each spatial block by first dividing the space from top to bottom and then merging the attributes from bottom to top, so that each intermediate node of the generated hierarchical tree has a consistent screen contribution with all its child nodes when viewed from a distance. The hierarchical re-optimization module is used to perform detailed hierarchical re-optimization of the hierarchical tree based on the target screen granularity, and to perform smooth interpolation of node attributes between adjacent levels. The global scheduling rendering module is used to integrate the hierarchical trees after optimization of each spatial block into a global hierarchical structure, and to perform prefetching scheduling of nodes in combination with viewpoint movement trends during the real-time rendering stage. The primitive optimization module includes: The strategy configuration unit is used to adopt a densification strategy triggered by the maximum gradient, and set the densification criterion of 3D Gaussian primitives to the maximum value of the screen space gradient. The supervised introduction unit is used to introduce the scale- and offset-aligned monocular depth estimation results as weak supervision to anchor the geometry; The exposure compensation unit is used to learn an affine color matrix of a preset size for each input image to perform learnable exposure compensation and reduce the interference of exposure differences between multiple cameras and at multiple times on geometric fitting. The global scheduling rendering module includes: The primitive checking unit is used to combine the hierarchical trees optimized from each spatial block into a global tree, and to check whether the same 3D Gaussian primitive is repeatedly assigned at the boundary of adjacent spatial blocks. The primitive deduplication unit is used to perform a weighted comparison based on the distance between the position of the 3D high-order primitive and the center of each adjacent spatial block if it exists. The 3D high-order primitive on the side with the closer distance is retained, and the 3D high-order primitive on the other side with the farther distance is deleted, so as to avoid double counting in the boundary area, which would cause the rendering to be too bright. The global scheduling rendering module includes: The view frustum prediction unit is used to obtain the current position, motion speed and orientation of the camera during the real-time rendering stage, and predict the adjacent spatial blocks that the view frustum is about to enter. The prefetch scheduling unit is used to asynchronously move the parent nodes of adjacent spatial blocks from CPU memory to GPU memory in advance; and to load intermediate nodes and leaf nodes into GPU memory one after another as the viewpoint approaches the boundary of adjacent spatial blocks, while simultaneously unloading leaf nodes that have exited the view frustum range in the current spatial block from GPU memory.

6. A real-time rendering device for 3D scenes based on spatial segmentation and hierarchical trees, characterized in that, The real-time rendering device for a 3D scene based on spatial segmentation and hierarchical tree includes: a memory and at least one processor, wherein the memory stores instructions, and the memory and the at least one processor are interconnected via a line; The at least one processor invokes the instructions in the memory to cause the 3D scene real-time rendering device based on spatial block and hierarchical tree to execute the 3D scene real-time rendering method based on spatial block and hierarchical tree as described in any one of claims 1-4.

7. A computer-readable storage medium storing a computer program thereon, characterized in that, When the computer program is executed by the processor, it implements the real-time rendering method for three-dimensional scenes based on spatial segmentation and hierarchical trees as described in any one of claims 1-4.

Images