A real-time rendering method and system for large-scale 3D Gaussian splash scene

Abstract

The application discloses a kind of real-time rendering method and system for large-scale 3D Gaussian splashing scene, comprising: first, the Gaussian primitive data of original 3D space is preprocessed, by constructing incremental octree index structure, 3D space is divided into multiple levels of incremental octree node;Then, according to the current camera parameter when rendering each frame, traverse incremental octree, eliminate the node outside view pyramid, and according to the distance between node and camera Distance scheduling different detail levels of Gaussian primitive data;Then determine the rendering order of Gaussian primitive between the block to be rendered and within the block;Finally, Gaussian primitive is projected to 2D screen, executes alpha blending formula for each pixel, accumulates color and opacity pixel by pixel to generate final rendering image.The beneficial effects of the application are: by the method of the application, the limitation of video memory capacity is broken through, the overall computing efficiency is optimized, high-performance real-time rendering is realized, and the rendering visual quality is guaranteed.

Description

Technical Field

[0001] This invention relates to the field of computer graphics processing technology, and more specifically, to a real-time rendering method and system for large-scale 3D Gaussian splash scenes. Background Technology

[0002] Traditional 3D Gaussian Splatting (3DGS) real-time rendering methods typically achieve high-quality real-time new perspective synthesis through explicit 3D Gaussian primitives and their alpha blending, but are only suitable for small to medium-sized scenes. When facing large-scale scenes (more than 1 billion Gaussian primitives and a spatial range of more than 20 km²) generated by applications such as city-level digital twins and wide-area SLAM, problems arise such as full loading easily leading to memory overflow, extremely low rendering frame rates that cannot meet the requirements of real-time interaction (frame rate greater than or equal to 30 FPS), and difficulty in guaranteeing sorting efficiency and consistency. Summary of the Invention

[0003] To address the aforementioned issues, the present invention aims to provide an effective scheduling and management mechanism for massive external storage data, enabling efficient and high-quality real-time rendering of ultra-large-scale 3DGS scenes on limited video memory hardware.

[0004] To achieve the objectives of this invention, a real-time rendering method for large-scale 3D Gaussian splash scenes is provided, comprising the following steps: S1: Preprocess the 3D Gaussian metadata of the original 3D space, construct the incremental octree index structure of the 3D space, divide the 3D space into multiple levels of incremental octree nodes, each incremental octree node corresponds to a block to be rendered, and each block to be rendered contains multiple Gaussian metadata. S2: At the start of each frame rendering, based on the current camera parameters, traverse the constructed incremental octree, remove incremental octree nodes and subtrees located outside the view frustum, and for incremental octree nodes located within the view frustum, schedule Gaussian metadata of different levels of detail for rendering based on the distance between the incremental octree node and the camera. S3: Determine the rendering order between blocks to be rendered and the rendering order between different high-order primitives within the blocks to be rendered; S4: Based on the rendering order obtained in S3, Gaussian pixels are projected onto the 2D screen in sequence, and the standard alpha blending formula is executed on each pixel of the obtained 2D screen to accumulate color and opacity pixel by pixel, finally generating the rendered image.

[0005] As a further improvement of the present invention, step S1 includes: S11: Calculate the Morton code for the three-dimensional coordinates of each 3D Gaussian element, mapping adjacent Gaussian elements in 3D space to a one-dimensional key-value space; S12: Starting from the root node of the incremental octree, recursively check the Gaussian elements in each node. When the number of Gaussian elements in a node exceeds a preset threshold τ and the screen space projection size of the node's bounding box in the current reference view is greater than a preset threshold ε, trigger node splitting. During splitting, the Gaussian element is assigned to the corresponding 8 child nodes according to the Morton code of the Gaussian element, and a loose axial bounding box with a relaxation factor γ is created for each child node. Repeat the above process until no nodes need to be split. After the split is complete, the parent node clears its own Gaussian metadata, and only the child nodes store the corresponding Gaussian metadata.

[0006] As a further improvement of the present invention, step S1 further includes: S13: Calculate the access popularity of the leaf nodes of the incremental octree. When the access popularity of a node is lower than the threshold, merge the Gaussian elements of all its child nodes into that node and release the storage resources of the child nodes.

[0007] As a further improvement of the present invention, in step S2, for the incremental octree nodes located in the view frustum, the projection size of the node's bounding box in the screen space is calculated, that is, the screen space error is calculated, and only nodes with a screen space error greater than a preset threshold ε are selected for rendering.

[0008] As a further improvement of the present invention, step S3 includes: S31: Calculate a representative depth for each block to be rendered, sort all blocks to be rendered according to their representative depths, and determine the rendering order between different blocks to be rendered. S32: Sort the Gaussian primitives within each block to be rendered according to their respective depth values ​​to determine the rendering order among different Gaussian primitives within the block to be rendered.

[0009] As a further improvement of the present invention, in step S31, the Radix Sort algorithm is used to sort all blocks to be rendered according to their representative depths; in step S32, the Bitonic Sort algorithm is used to sort the Gaussian elements in each block to be rendered according to their respective depth values.

[0010] As a further improvement of the present invention, in step S4, when executing the standard α blending formula, color and opacity are accumulated pixel by pixel until the opacity is greater than or equal to 1 or all primitives are processed, and finally a rendered image is generated.

[0011] This invention also provides a real-time rendering system for large-scale 3D Gaussian splash scenes, the system comprising: The incremental octree index structure construction module is used to preprocess the 3D Gaussian metadata of the original 3D space, construct the incremental octree index structure of the 3D space, divide the 3D space into multiple levels of incremental octree nodes, each incremental octree node corresponds to a block to be rendered, and each block to be rendered contains multiple Gaussian metadata. The frustum culling and LOD scheduling module is used to traverse the constructed incremental octree at the beginning of each frame rendering based on the current camera parameters, culling incremental octree nodes and subtrees located outside the frustum. For incremental octree nodes located within the frustum, different levels of detail Gaussian metadata are scheduled for rendering based on the distance between the incremental octree node and the camera. The hybrid sorting module is used to determine the rendering order between blocks to be rendered and the rendering order between different high-order primitives within a block to be rendered. The alpha blending and image compositing module projects Gaussian pixels onto a 2D screen sequentially according to the rendering order determined by the blending sorting module. It then applies the standard alpha blending formula to each pixel of the resulting 2D screen, accumulating color and opacity pixel by pixel to ultimately generate a rendered image.

[0012] As a further improvement of the present invention, the incremental octree index structure construction module includes: The Morton coding and spatial partitioning module is used to calculate the Morton code of the three-dimensional position of each Gaussian element, mapping adjacent Gaussian elements in 3D space to a one-dimensional key-value space. The node splitting module recursively checks the Gaussian elements in each node, starting from the root node of the incremental octree. When the number of Gaussian elements in a node exceeds a preset threshold τ and the screen space projection size of the node's bounding box in the current reference view is greater than a preset threshold ε, node splitting is triggered. During splitting, the Gaussian elements are assigned to the corresponding 8 child nodes according to their Morton codes, and a loose axial bounding box with a relaxation factor γ is created for each child node. The above process is repeated until no node needs to be split. After splitting, the parent node clears its own Gaussian metadata, and only the child nodes store the corresponding Gaussian metadata.

[0013] As a further improvement of the present invention, the hybrid sorting module includes: The block-level coarse sorting module is used to calculate a representative depth for each block to be rendered, sort all blocks to be rendered according to their representative depth, and determine the rendering order between blocks to be rendered. The fine sorting module within each block is used to sort the Gaussian elements in each block to be rendered according to their respective depth values, thereby determining the rendering order among different Gaussian elements within the block to be rendered.

[0014] As a further improvement of the present invention, the incremental octree index structure construction module further includes: The node merging module is used to calculate the access popularity of the leaf nodes of the incremental octree. When the access popularity of a node is lower than the threshold, the Gaussian elements of all its child nodes are merged into that node, and the storage resources of the child nodes are released.

[0015] As a further improvement of the present invention, the view frustum culling and LOD scheduling module calculates the projection size of the bounding box of the incremental octree node located in the view frustum in the screen space, that is, calculates the screen space error, and selects only the nodes with screen space error greater than a preset threshold ε for rendering.

[0016] As a further improvement of the present invention, the hybrid sorting module uses the Radix Sort algorithm to sort all blocks to be rendered according to their representative depth; and uses the Bitonic Sort algorithm to sort the Gaussian elements in each block to be rendered according to their respective depth values.

[0017] As a further improvement of the present invention, when the α-mixing and image synthesis module executes the standard α-mixing formula, it accumulates color and opacity pixel by pixel until the opacity is greater than or equal to 1 or all primitives have been processed, and finally generates a rendered image.

[0018] The beneficial effects of this invention are as follows: By employing block scheduling and a LOD (Level of Detail) mechanism, it eliminates the need to load all scene data into video memory, achieving "load-as-you-go" functionality. This makes it possible to render scenes that would otherwise require hundreds of gigabytes of storage on a consumer-grade graphics card with only 12GB of video memory, breaking through the limitations of video memory capacity. By significantly reducing the amount of data processed per frame and employing an efficient hybrid sorting algorithm, the rendering frame rate is effectively improved, achieving high-performance real-time rendering and fully meeting the needs of real-time interaction. LOD scheduling based on screen space error is a perception-driven optimization that removes details within a range imperceptible to the user, achieving visual losslessness. Simultaneously, strict cross-block sorting ensures the correctness of alpha blending, avoiding rendering artifacts and guaranteeing rendering visual quality. By decomposing computationally intensive tasks and adapting them to GPU hardware characteristics through a hybrid sorting algorithm, the sorting time is reduced from hundreds of milliseconds in traditional methods to 18ms, solving the core performance bottleneck in large-scale scenes and optimizing overall computational efficiency. Attached Figure Description

[0019] Figure 1 This is a flowchart illustrating a real-time rendering method for large-scale 3D Gaussian splash scenes according to an embodiment of the present invention; Figure 2 This is a flowchart illustrating the method for constructing an incremental octree index structure for real-time rendering of large-scale 3D Gaussian splash scenes, as described in an embodiment of the present invention. Figure 3 This is a flowchart of a hybrid sorting algorithm for a real-time rendering method for large-scale 3D Gaussian splash scenes, as described in an embodiment of the present invention. Figure 4 This is a schematic diagram of the system structure of a real-time rendering system for large-scale 3D Gaussian splash scenes, as described in an embodiment of the present invention. Detailed Implementation

[0020] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this disclosure.

[0021] It should be noted that if the embodiments of the present invention involve directional indicators (such as up, down, left, right, front, back, etc.), the directional indicators are only used to explain the relative positional relationship and movement of the components in a certain specific posture (as shown in the figure). If the specific posture changes, the directional indicators will also change accordingly.

[0022] Furthermore, the terminology used in the description of this invention is for illustrative purposes only and is not intended to limit the scope of this disclosure. The terms "comprising" and / or "including" are used to specify the presence of elements, steps, operations, and / or components, but do not exclude the presence or addition of one or more other elements, steps, operations, and / or components. The terms "first," "second," etc., may be used to describe various elements, do not represent an order, and do not limit these elements. Additionally, in the description of this invention, unless otherwise stated, "a plurality of" means two or more. These terms are used only to distinguish one element from another. These and / or other aspects become apparent in conjunction with the following drawings, and those skilled in the art will more readily understand the description of embodiments of the invention. The drawings are used for illustrative purposes only to depict embodiments of this disclosure. Those skilled in the art will readily recognize from the following description that alternative embodiments of the structures and methods shown in this invention can be employed without departing from the principles of this disclosure.

[0023] The present invention provides a real-time rendering method for large-scale 3D Gaussian splash scenes, as described in this embodiment. Figure 1 As shown, the method includes the following steps: Step S1: The original 3D Gaussian metadata dataset includes key information for each primitive, such as 3D spatial coordinates (x, y, z), color parameters (r, g, b), opacity α, and shape description parameters (e.g., covariance matrix). The original 3D Gaussian metadata is preprocessed to construct an incremental octree index structure for the 3D space. The 3D space is divided into multiple levels of incremental octree nodes, each corresponding to a rendering block, and each rendering block contains multiple Gaussian primitives. The massive 3D Gaussian metadata is adaptively hierarchically divided into blocks according to spatial distribution and visual importance, forming a "hierarchical, schedulable" incremental octree index structure, solving the problems of large-scale data storage management and fast access. Each incremental octree node corresponds to a block to be rendered. The block size is adaptively adjusted according to the scene data density: the blocks in data-dense areas (such as urban building clusters) are more refined (fewer primitives and smaller spatial range), while the blocks in data-sparse areas (such as open spaces) are coarser (more primitives and larger spatial range), ensuring that the blocks meet the rendering accuracy requirements while avoiding redundant data processing.

[0024] Step S2: At the start of each frame rendering, based on the current camera parameters, traverse the constructed incremental octree, removing incremental octree nodes and subtrees located outside the view frustum. For incremental octree nodes located within the view frustum, schedule different levels of Gaussian metadata for rendering based on the distance between the incremental octree node and the camera. Before each frame rendering, quickly filter out the blocks to be rendered in the current view and their corresponding levels of Gaussian metadata, significantly reducing the amount of data processed per frame and overcoming the bottlenecks of video memory and bandwidth. Based on the distance between the incremental octree node and the camera, schedule different levels of 3D Gaussian metadata for rendering based on the distance between the incremental octree node and the camera; nodes farther away use lower levels of detail, and nodes closer use higher levels of detail, achieving distance-based level of detail control.

[0025] Step S3: Determine the rendering order between the blocks to be rendered and the rendering order between different hexagonal primitives within the blocks to be rendered. Determine the global rendering order of 3D hexagonal primitives to ensure the correctness of alpha blending while avoiding the performance bottleneck of traditional global sorting.

[0026] Step S4: Based on the rendering order obtained in S3, Gaussian pixels are projected onto the 2D screen in sequence, and the standard alpha blending formula is applied to each pixel of the obtained 2D screen to accumulate color and opacity pixel by pixel, finally generating the rendered image.

[0027] Each Gaussian element in the globally ordered list is transformed from 3D space to 2D screen coordinates using the camera's view matrix and projection matrix. Simultaneously, based on the shape parameters of the Gaussian element, such as the covariance matrix, its projection area (ellipse or rectangle) on the 2D screen is calculated, clarifying the pixel range affected by the Gaussian element. After all Gaussian elements have been processed, the pixel data in the color buffer is read, and post-processing operations such as Gamma correction and tone mapping are performed to generate the final rendered image that conforms to the characteristics of human vision, which is then output to the display device or stored as an image file.

[0028] An alternative implementation, such as Figure 2 As shown, when preprocessing the 3D Gaussian metadata of the original 3D space, the original 3D Gaussian metadata dataset is read, and Morton encoding is performed on the three-dimensional coordinates of each Gaussian element. This maps adjacent Gaussian elements in the 3D space to a one-dimensional key-value space, making the spatially adjacent data continuously distributed in storage. Through Morton encoding, the spatially adjacent Gaussian elements that were originally scattered are centrally classified. When splitting nodes later, the primitive affiliation can be quickly determined through Morton codes, improving the block splitting efficiency.

[0029] Based on the sorted Gaussian metadata, the node splitting process of the incremental octree is initiated. The root node of the incremental octree is created, and all Gaussian elements sorted by Morton encoding are assigned to the root node. The bounding box of the root node is set as the bounding box of the entire 3D scene. Two key thresholds for node splitting are preset: the number of Gaussian elements threshold τ and the screen space projection size threshold ε. Starting from the root node, each node is recursively traversed. For each node, the following is performed: count the number of Gaussian elements N contained in the node and calculate the screen space projection size S of the node's bounding box under the current reference view (the initial reference view is set as the default view of the scene center point). If N>τ and S>ε, then node splitting is triggered.

[0030] The splitting process includes: Divide the current node's bounding box into 8 equally sized sub-bounding boxes, corresponding to 8 child nodes. Based on the last three bits of the Morton code of each Gaussian primitive (the lower bits of the Morton code correspond to the spatial subdivision level), assign the primitive to the corresponding child node (e.g., the last three bits of the Morton code are 000 for the first child node, 001 for the second child node, and so on). Create a loose axial bounding box with a relaxation factor γ (e.g., γ=1.2) for each child node (i.e., the bounding box of the child node is expanded outward by 20% from the original partition boundary) to cover possible spatial distribution deviations of the primitives and improve the efficiency of subsequent frustum culling. Repeat the above node checking and splitting process until all nodes satisfy "N ≤ τ" or "S ≤ ε", then stop splitting.

[0031] After the split, the parent node retains only the child node index and its own bounding box information, and clears its stored Gaussian metadata. All Gaussian metadata is stored only in leaf nodes or unsplit intermediate nodes. This splitting method in this embodiment only subdivides nodes in data-intensive (numerous primitives) and visually important (large screen projection size) areas, avoiding the problem of too many empty nodes caused by the global uniform partitioning of traditional octrees.

[0032] In one optional implementation, to optimize memory resource usage, a node merging operation is performed on the split incremental octree. The access frequency of the leaf nodes of the incremental octree is calculated. When the access frequency of a node is below a threshold, the Gaussian elements of all its child nodes are merged into that node, and the storage resources of the child nodes are released. Specific processes include, for example: An access heat counter is added to each leaf node, with an initial value of 0 and a counter update cycle set, for example, 100 frames. During subsequent real-time rendering, if a leaf node is scheduled for rendering in a frame, its access heat counter is incremented by 1; if it is not scheduled, the counter is decremented by 1 (minimum value is 0). Every 100 frames, the access heat of all leaf nodes is checked. If the access heat of a node is lower than a threshold (in this embodiment, the threshold is set to 20 for example), a merge operation is triggered: first, all Gaussian metadata stored in all child nodes (if they exist) of the node is migrated to the node; then, the storage resources of the child nodes (including the bounding box information of the child nodes, primitive metadata buffers, etc.) are released; and the bounding box of the node is updated to the smallest outer bounding box of all the original child node bounding boxes. The merged node participates in subsequent node split checks as a new leaf node. If its access heat increases or the split conditions are met, a split can be triggered again.

[0033] Compared to traditional octrees, which use a one-time global uniform partitioning that creates trees of the same depth for sparse and dense regions, resulting in a large number of empty nodes and low efficiency, this invention reduces the total number of nodes in the incremental octree through node merging operations. This reduces memory usage, effectively improves hardware resource utilization, and significantly reduces memory overhead and traversal costs.

[0034] An optional implementation involves performing LOD scheduling on the candidate rendering incremental octree nodes retained after view frustum culling, selecting the primitive data corresponding to the level of detail for rendering. Specifically, this includes calculating the projection size of the node's bounding box in screen space, i.e., calculating the screen space error, and selecting only nodes with a screen space error greater than a preset threshold ε for rendering. The specific implementation process includes: For each candidate node, the projection size of the node's bounding box in the current screen space is calculated, i.e., the screen space error S_error is calculated. A preset threshold ε is used, which is consistent with the node splitting threshold to ensure visual consistency. If the node's S_error > ε, then the node's child nodes (if they exist) are selected as the rendering unit, and the Gaussian metadata (higher level of detail) stored in the child nodes is loaded. If the node's S_error ≤ ε, then the node is directly selected as the rendering unit, and the Gaussian metadata (lower level of detail) stored in the node is loaded, without traversing its child nodes. For node data loaded into video memory in the previous frame but not selected in the current frame, the video memory space it occupies is released through a memory reclamation mechanism after the current frame is rendered, ensuring the recycling of video memory resources.

[0035] In this embodiment, through LOD scheduling, in a scene covering a 20km² urban area and containing 520 million GoSk units, the number of GoSk units actually loaded into the video memory per frame was reduced from 520 million to an average of 48 million, and the video memory usage was controlled at 7.3GB, which is far lower than the 12GB video memory capacity of the NVIDIA RTX 4060. While ensuring image quality, the rendering load per frame was greatly reduced.

[0036] An alternative implementation, to ensure that primitives scheduled from different incremental octree nodes can be correctly α-mixed, such as... Figure 3 As shown, this embodiment of the invention employs a hybrid parallel sorting method combining block-level coarse sorting and intra-block fine sorting to perform depth sorting on all Gaussian primitives to be rendered. The block-level coarse sorting includes: calculating a representative depth for each block to be rendered, sorting all blocks according to their representative depths, and determining the rendering order between different blocks; the intra-block fine sorting includes: sorting the Gaussian primitives within each block according to their respective depth values, and determining the rendering order between different Gaussian primitives within the block.

[0037] An optional implementation scheme, the specific implementation process of block-level coarse sorting, is as follows: For each block (node) to be rendered, take the center three-dimensional coordinates (x_c, y_c, z_c) of the node's loose axis bounding box. According to the current camera's view matrix, transform the center coordinates to camera space. The absolute value of the z-axis coordinate in camera space is the representative depth D_block of the node (the larger the value, the farther the node is from the camera); collect the representative depths D_block of all blocks to be rendered and construct a block depth list; the sorting radix is ​​set to 256 for example (i.e., sorting 8-bit data each time), and the sorting rounds are set to 4 rounds for example (adapting to 32-bit depth values); call the CUDA kernel function on the GPU to execute Radix Sort, using the block's representative depth D_block as the sorting key, and sort all blocks to be rendered in ascending order (the smaller the depth value, the closer the block is to the camera, and the higher the rendering priority); after sorting, a rendering order list of blocks is obtained, and each block will be processed in this order subsequently.

[0038] The implementation process of fine sorting within each block is as follows: For each Gaussian primitive, its 3D coordinates (x, y, z) are transformed to camera space using the camera view matrix, and the absolute value of the z-axis coordinate in camera space is taken as the depth value d_primitive of the primitive; the Gaussian primitive data (including the depth value d_primitive) within each block is allocated to the GPU's thread blocks, with each thread block processing 1024 primitives (adapting to the GPU warp size); within each thread block, the Bitonic Sort kernel function is called to perform descending sorting using the primitive depth value d_primitive as the sorting key (primitives with larger depth values ​​are processed first during alpha blending to ensure correct transparency); after the primitives within each block are sorted, they are concatenated according to the block order obtained from the block-level coarse sorting to form a globally ordered primitive rendering list.

[0039] Existing technologies mostly employ global sorting strategies, requiring the transmission of several gigabytes to tens of gigabytes of visible primitive data between the CPU and GPU each frame. This creates a significant data transmission bandwidth bottleneck, causing the rendering frame rate to plummet to single digits (e.g., 2-8 FPS), failing to meet the basic requirements of real-time interaction (30 FPS). The hybrid scheme of this invention combines the efficiency of Radix in large-scale bucketing with the branchless and low-overhead advantages of Bitonic in sorting within fixed-size blocks, achieving optimal overall sorting performance. Under the same scene and hardware, this invention can achieve a rendering frame rate of 42 FPS, more than 20 times that of the existing technology (2 FPS), fully meeting the needs of real-time interaction.

[0040] In one alternative implementation, when performing alpha blending and image composition on the ordered Gaussian primitive rendering list obtained by the above-described hybrid sorting, each Gaussian primitive is processed sequentially according to the globally ordered primitive list until the opacity is greater than or equal to 1 or all primitives have been processed to generate the final rendered image, thereby improving rendering efficiency.

[0041] Specific processes include, for example, determining the pixel regions (u_min~u_max, v_min~v_max) that the primitives affect on the screen; and applying the standard α-mixing formula to each pixel (u, v) within that region. Color accumulation: C_final = C_current × (1 - α_primitive) + C_primitive × α_primitive; Opacity accumulation: α_final = α_current + α_primitive × (1 - α_current); Where C_current is the color value of the current pixel in the color buffer, α_current is the opacity value of the current pixel in the opacity buffer, C_primitive is the color value of the current high-order pixel, and α_primitive is the opacity value of the current high-order pixel; The calculated C_final and α_final are updated to the color buffer and opacity buffer of the corresponding pixel. If the α_final of a pixel is ≥ 1, the influence of subsequent primitives on the color of the pixel can be ignored, and the subsequent blending calculation of the pixel is skipped to improve rendering efficiency.

[0042] In one optional implementation, the real-time rendering method of this invention for large-scale 3D Gaussian splash scenes is applied to a scene covering a 20km² urban area containing 520 million Gaussian primitives, using an NVIDIA RTX 4060 (12GB VRAM) hardware. Through LOD scheduling, the number of primitives actually loaded and processed per frame is reduced to tens of millions, with VRAM usage of only 7.3GB. Through hybrid parallel sorting, sorting time is reduced to 18ms. During rendering tests of this scene, traditional Gaussian splashing solutions could not achieve rendering efficiency above 10 FPS; only the solution of this invention ultimately achieved smooth real-time rendering at an average of 42 FPS.

[0043] This invention also provides a real-time rendering system for large-scale 3D Gaussian splash scenes, such as... Figure 4 As shown, the system includes: The incremental octree index structure construction module is used to preprocess the 3D Gaussian metadata of the original 3D space, constructing an incremental octree index structure for the 3D space. This divides the 3D space into multiple levels of incremental octree nodes, each corresponding to a block to be rendered, and each block containing multiple Gaussian primitives. The massive 3D Gaussian metadata is adaptively hierarchically divided into blocks according to spatial distribution and visual importance, forming a "hierarchical and schedulable" incremental octree index structure, solving the problems of large-scale data storage management and fast access. Each incremental octree node corresponds to a block to be rendered, and the block size adaptively adjusts according to the scene data density: data-dense areas (such as urban building clusters) have finer blocks (fewer primitives, smaller spatial range), while data-sparse areas (such as open spaces) have coarser blocks (more primitives, larger spatial range), ensuring that the blocks meet rendering accuracy requirements while avoiding redundant data processing.

[0044] The view frustum culling and LOD scheduling module is used at the beginning of each frame rendering to traverse the constructed incremental octree based on the current camera parameters, culling incremental octree nodes and subtrees located outside the view frustum. For incremental octree nodes located within the view frustum, different levels of detail Gaussian metadata are scheduled for rendering based on the distance between the incremental octree node and the camera. Before each frame rendering, the module quickly selects the blocks to be rendered and their corresponding levels of detail Gaussian metadata for the current viewpoint, significantly reducing the amount of data processed per frame and overcoming the bottlenecks of video memory and bandwidth. Based on the distance between the incremental octree node and the camera, different levels of detail 3D Gaussian metadata are scheduled for rendering of incremental octree nodes at different distances; nodes that are farther away use a lower level of detail, and nodes that are closer use a higher level of detail, achieving distance-based level of detail control. LOD scheduling based on screen space error is a perception-driven optimization that removes details in a range imperceptible to the user, achieving visual losslessness; at the same time, strict sorting across blocks ensures the correctness of alpha blending and avoids rendering defects.

[0045] The hybrid sorting module is used to determine the rendering order between blocks to be rendered and the rendering order between different Gaussian elements within a block to be rendered. The hybrid sorting algorithm decomposes computationally intensive tasks and adapts them to GPU hardware characteristics, reducing the sorting time from hundreds of milliseconds in traditional methods to 18ms, thus solving the core performance bottleneck in large-scale scenarios.

[0046] The alpha blending and image compositing module projects Gaussian primitives onto a 2D screen sequentially according to the rendering order determined by the blending sorting module. It then applies a standard alpha blending formula to each pixel of the resulting 2D screen, accumulating color and opacity pixel-by-pixel to ultimately generate the rendered image. Each Gaussian primitive in the globally ordered list is transformed from 3D space to 2D screen coordinates using the camera's view matrix and projection matrix. Simultaneously, based on the primitive's shape parameters, such as the covariance matrix, its projection area (ellipse or rectangle) on the 2D screen is calculated, defining the range of pixels affected by that primitive. After all Gaussian primitives have been processed, pixel data from the color buffer is read, and post-processing operations such as Gamma correction and tone mapping are performed to generate the final rendered image that conforms to human visual characteristics, which is then output to a display device or stored as an image file.

[0047] In one optional implementation, the incremental octree index structure construction module includes a Morton coding and spatial partitioning module and a node splitting module. The Morton coding and spatial partitioning module is used to calculate the Morton code for the three-dimensional position of each Gaussian primitive, mapping adjacent Gaussian primitives in 3D space to a one-dimensional key-value space; this ensures that spatially adjacent data is continuously distributed in storage. Through Morton coding, spatially adjacent Gaussian primitives that were originally scattered are grouped together, and subsequent node splitting can quickly determine the primitive affiliation through the Morton code, improving block efficiency.

[0048] The node splitting module recursively checks the Gaussian elements in each node, starting from the root node of the incremental octree. When the number of Gaussian elements in a node exceeds a preset threshold τ and the screen space projection size of the node's bounding box in the current reference view is greater than a threshold ε, node splitting is triggered. During splitting, the Gaussian elements are assigned to the corresponding 8 child nodes according to their Morton codes, and a loose axial bounding box with a relaxation factor γ is created for each child node. The above process is repeated until no node needs to be split. After splitting, the parent node clears its own Gaussian metadata, and only the child nodes store the corresponding Gaussian metadata.

[0049] The node splitting process includes, for example, the following steps: based on the sorted Gaussian metadata, initiating the node splitting process of the incremental octree, creating the root node of the incremental octree, assigning all Gaussian elements sorted by Morton encoding to the root node, and setting the bounding box of the root node as the bounding box of the entire 3D scene; presetting two key thresholds for node splitting: the Gaussian element count threshold τ and the screen space projection size threshold ε; starting from the root node, recursively traversing each node, and performing the following for each node: counting the number of Gaussian elements N contained in the node and calculating the screen space projection size S of the node's bounding box under the current reference view (the initial reference view is set as the default view of the scene center point); if N>τ and S>ε, then triggering node splitting.

[0050] The splitting process includes: Divide the current node's bounding box into 8 equally sized sub-bounding boxes, corresponding to 8 child nodes. Based on the last three bits of the Morton code of each Gaussian primitive (the lower bits of the Morton code correspond to the spatial subdivision level), assign the primitive to the corresponding child node (e.g., the last three bits of the Morton code are 000 for the first child node, 001 for the second child node, and so on). Create a loose axial bounding box with a relaxation factor γ (e.g., γ=1.2) for each child node (i.e., the bounding box of the child node is expanded outward by 20% from the original partition boundary) to cover possible spatial distribution deviations of the primitives and improve the efficiency of subsequent frustum culling. Repeat the above node checking and splitting process until all nodes satisfy "N ≤ τ" or "S ≤ ε", then stop splitting.

[0051] After the split, the parent node retains only the child node index and its own bounding box information, and clears its stored Gaussian metadata. All Gaussian metadata is stored only in leaf nodes or unsplit intermediate nodes. This splitting method in this embodiment only subdivides nodes in data-intensive (numerous primitives) and visually important (large screen projection size) areas, avoiding the problem of too many empty nodes caused by the global uniform partitioning of traditional octrees.

[0052] In one optional implementation, the incremental octree index structure construction module further includes a node merging module. This module calculates the access popularity of the leaf nodes of the incremental octree. When the access popularity of a node falls below a threshold, the Gaussian elements of all its child nodes are merged into that node, and the storage resources of the child nodes are released. Specific processes include, for example: An access heat counter is added to each leaf node, with an initial value of 0 and a counter update cycle set, for example, 100 frames. During subsequent real-time rendering, if a leaf node is scheduled for rendering in a frame, its access heat counter is incremented by 1; if it is not scheduled, the counter is decremented by 1 (minimum value is 0). Every 100 frames, the access heat of all leaf nodes is checked. If the access heat of a node is lower than a threshold (in this embodiment, the threshold is set to 20 for example), a merge operation is triggered: first, all Gaussian metadata stored in all child nodes (if they exist) of the node is migrated to the node; then, the storage resources of the child nodes (including the bounding box information of the child nodes, primitive metadata buffers, etc.) are released; and the bounding box of the node is updated to the smallest outer bounding box of all the original child node bounding boxes. The merged node participates in subsequent node split checks as a new leaf node. If its access heat increases or the split conditions are met, a split can be triggered again.

[0053] Compared to traditional octrees, which use a one-time global uniform partitioning that creates trees of the same depth for sparse and dense regions, resulting in a large number of empty nodes and low efficiency, this invention reduces the total number of nodes in the incremental octree through node merging operations. This reduces memory usage, effectively improves hardware resource utilization, and significantly reduces memory overhead and traversal costs.

[0054] In one optional implementation, the hybrid sorting module includes a block-level coarse sorting module and an intra-block fine sorting module. The block-level coarse sorting module calculates a representative depth for each block to be rendered and sorts all blocks according to their representative depths to determine the rendering order among the blocks. The intra-block fine sorting module sorts the Gaussian elements within each block according to their respective depth values ​​to determine the rendering order among different Gaussian elements within the block. This invention achieves a significant leap in rendering frame rate by employing an efficient hybrid sorting algorithm. Experimental data shows that, under the same scene and hardware, the frame rate of this invention (42 FPS) is more than 20 times that of the official implementation (2 FPS), fully meeting the requirements of real-time interaction.

[0055] An optional implementation method for block-level coarse sorting is as follows: For each block (node) to be rendered, take the center 3D coordinates (x_c, y_c, z_c) of the node's loose axis bounding box. Based on the current camera's view matrix, transform the center coordinates to camera space. The absolute value of the z-axis coordinate in camera space is the representative depth D_block of the node (the larger the value, the farther the node is from the camera). Collect the representative depths D_block of all blocks to be rendered and construct a block depth list. The sorting radix is ​​set to 256 for example (i.e., sorting 8-bit data each time), and the sorting rounds are 4 rounds for example (adapting to 32-bit depth values). On the GPU, call the CUDA kernel function to execute Radix Sort, using the block's representative depth D_block as the sorting key, and sort all blocks to be rendered in ascending order (the smaller the depth value, the closer the block is to the camera, and the higher the rendering priority). After sorting, a rendering order list of blocks is obtained, and each block will be processed sequentially according to this order.

[0056] The implementation process of fine sorting within each block is as follows: For each Gaussian primitive, its 3D coordinates (x, y, z) are transformed to camera space using the camera view matrix, and the absolute value of the z-axis coordinate in camera space is taken as the depth value d_primitive of the primitive; the Gaussian primitive data (including the depth value d_primitive) within each block is allocated to the GPU's thread blocks, with each thread block processing 1024 primitives (adapting to the GPU warp size); within each thread block, the Bitonic Sort kernel function is called to perform descending sorting using the primitive depth value d_primitive as the sorting key (primitives with larger depth values ​​are processed first during alpha blending to ensure correct transparency); after the primitives within each block are sorted, they are concatenated according to the block order obtained from the block-level coarse sorting to form a globally ordered primitive rendering list.

[0057] Existing technologies mostly employ global sorting strategies, requiring the transmission of several gigabytes to tens of gigabytes of visible primitive data between the CPU and GPU each frame. This creates a significant data transmission bandwidth bottleneck, causing the rendering frame rate to plummet to single digits (e.g., 2-8 FPS), failing to meet the basic requirements of real-time interaction (30 FPS). The hybrid scheme of this invention combines the efficiency of Radix in large-scale bucketing with the branchless and low-overhead advantages of Bitonic in sorting within fixed-size blocks, achieving optimal overall sorting performance. Under the same scene and hardware, this invention can achieve a rendering frame rate of 42 FPS, more than 20 times that of the existing technology (2 FPS), fully meeting the needs of real-time interaction.

[0058] In one optional implementation, the view frustum culling and LOD scheduling module calculates the projection size of the bounding box of the incremental octree node located within the view frustum in screen space, i.e., calculates the screen space error, and selects only nodes with a screen space error greater than a preset threshold ε for rendering. The specific implementation process includes: For each candidate node, the projection size of the node's bounding box in the current screen space is calculated, i.e., the screen space error S_error is calculated. A preset threshold ε is used, which is consistent with the node splitting threshold to ensure visual consistency. If the node's S_error > ε, then the node's child nodes (if they exist) are selected as the rendering unit, and the Gaussian metadata (higher level of detail) stored in the child nodes is loaded. If the node's S_error ≤ ε, then the node is directly selected as the rendering unit, and the Gaussian metadata (lower level of detail) stored in the node is loaded, without traversing its child nodes. For node data loaded into video memory in the previous frame but not selected in the current frame, the video memory space it occupies is released through a memory reclamation mechanism after the current frame is rendered, ensuring the recycling of video memory resources.

[0059] In this embodiment, through LOD scheduling, in a scene covering a 20km² urban area and containing 520 million GoSk units, the number of GoSk units actually loaded into the video memory per frame was reduced from 520 million to an average of 48 million, and the video memory usage was controlled at 7.3GB, which is far lower than the 12GB video memory capacity of the NVIDIA RTX 4060. While ensuring the image quality, the rendering load per frame was greatly reduced.

[0060] In one alternative implementation, the hybrid sorting module uses the Radix Sort algorithm to perform a coarse sort of all blocks to be rendered according to their representative depths at the block level; and uses the Bitonic Sort algorithm to perform a fine sort of Gaussian elements within each block to be rendered according to their respective depth values ​​at the block level.

[0061] The implementation process of fine sorting within each block is as follows: For each Gaussian primitive, its 3D coordinates (x, y, z) are transformed to camera space using the camera view matrix, and the absolute value of the z-axis coordinate in camera space is taken as the depth value d_primitive of the primitive; the Gaussian primitive data (including the depth value d_primitive) within each block is allocated to the GPU's thread blocks, with each thread block processing 1024 primitives (adapting to the GPU warp size); within each thread block, the Bitonic Sort kernel function is called to perform descending sorting using the primitive depth value d_primitive as the sorting key (primitives with larger depth values ​​are processed first during alpha blending to ensure correct transparency); after the primitives within each block are sorted, they are concatenated according to the block order obtained from the block-level coarse sorting to form a globally ordered primitive rendering list.

[0062] Existing technologies mostly employ global sorting strategies, requiring the transmission of several gigabytes to tens of gigabytes of visible primitive data between the CPU and GPU each frame. This creates a significant data transmission bandwidth bottleneck, causing the rendering frame rate to plummet to single digits (e.g., 2-8 FPS), failing to meet the basic requirements of real-time interaction (30 FPS). The hybrid scheme of this invention combines the efficiency of Radix in large-scale bucketing with the branchless and low-overhead advantages of Bitonic in sorting within fixed-size blocks, achieving optimal overall sorting performance. Under the same scene and hardware, this invention can achieve a rendering frame rate of 42 FPS, more than 20 times that of the existing technology (2 FPS), fully meeting the needs of real-time interaction.

[0063] In one optional implementation, the alpha blending and image compositing module accumulates color and opacity pixel-by-pixel when executing the standard alpha blending formula, until the opacity is greater than or equal to 1 or all primitives have been processed, generating the final rendered image, thus improving rendering efficiency. Specific processes include, for example, determining the pixel regions (u_min~u_max, v_min~v_max) that affect the primitives on the screen; and executing the standard alpha blending formula for each pixel (u, v) within that region. Color accumulation: C_final = C_current × (1 - α_primitive) + C_primitive × α_primitive; Opacity accumulation: α_final = α_current + α_primitive × (1 - α_current); Where C_current is the color value of the current pixel in the color buffer, α_current is the opacity value of the current pixel in the opacity buffer, C_primitive is the color value of the current high-order pixel, and α_primitive is the opacity value of the current high-order pixel; The calculated C_final and α_final are updated to the color buffer and opacity buffer of the corresponding pixel. If the α_final of a pixel is ≥ 1, the influence of subsequent primitives on the color of the pixel can be ignored, and the subsequent blending calculation of the pixel is skipped to improve rendering efficiency.

[0064] That is, those skilled in the art will understand that all or part of the steps in the methods of the above embodiments can be implemented by a program instructing related hardware. This program is stored in a storage medium and includes several instructions to cause a device (which may be a microcontroller, chip, etc.) or processor to execute all or part of the steps of the methods described in the embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as a USB flash drive, a portable hard drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0065] Numerous specific details are set forth in the specification provided herein. However, it will be understood that embodiments of the invention may be practiced without these specific details. In some instances, well-known methods, structures, and techniques have not been shown in detail so as not to obscure the understanding of this specification.

[0066] Furthermore, those skilled in the art will understand that although some embodiments described herein include certain features but not others included in other embodiments, combinations of features from different embodiments are intended to be within the scope of the invention and form different embodiments. For example, in the claims, any of the claimed embodiments can be used in any combination.

[0067] Those skilled in the art will understand that although the invention has been described with reference to exemplary embodiments, various changes may be made and its elements may be substituted with equivalents without departing from the scope of the invention. Furthermore, many modifications may be made to adapt particular situations or materials to the teachings of the invention without departing from the essential scope of the invention. Therefore, the invention is not limited to the specific embodiments disclosed, but rather the invention will include all embodiments falling within the scope of the appended claims.

Claims

1. A real-time rendering method for large-scale 3D Gaussian splash scenes, characterized in that, The method includes the following steps: S1: Preprocess the 3D Gaussian metadata of the original 3D space, construct the incremental octree index structure of the 3D space, divide the 3D space into multiple levels of incremental octree nodes, each incremental octree node corresponds to a block to be rendered, and each block to be rendered contains multiple Gaussian metadata. S2: At the start of each frame rendering, based on the current camera parameters, traverse the constructed incremental octree, remove incremental octree nodes and subtrees located outside the view frustum, and for incremental octree nodes located within the view frustum, schedule Gaussian metadata of different levels of detail for rendering based on the distance between the incremental octree node and the camera. S3: Determine the rendering order between blocks to be rendered and the rendering order between different high-order primitives within the blocks to be rendered; S4: Based on the rendering order obtained in S3, Gaussian pixels are projected onto the 2D screen in sequence, and the standard alpha blending formula is executed on each pixel of the obtained 2D screen to accumulate color and opacity pixel by pixel, finally generating the rendered image.

2. The method according to claim 1, characterized in that, Step S1 includes: S11: Calculate the Morton code for the three-dimensional coordinates of each 3D Gaussian element, mapping adjacent Gaussian elements in 3D space to a one-dimensional key-value space; S12: Starting from the root node of the incremental octree, recursively check the Gaussian elements in each node. When the number of Gaussian elements in a node exceeds a preset threshold τ and the screen space projection size of the node's bounding box in the current reference view is greater than a preset threshold ε, trigger node splitting. During splitting, the Gaussian element is assigned to the corresponding 8 child nodes according to the Morton code of the Gaussian element, and a loose axial bounding box with a relaxation factor γ is created for each child node. Repeat the above process until no nodes need to be split. After the split is complete, the parent node clears its own Gaussian metadata, and only the child nodes store the corresponding Gaussian metadata.

3. The method according to claim 2, characterized in that, Step S1 also includes: S13: Calculate the access popularity of the leaf nodes of the incremental octree. When the access popularity of a node is lower than the threshold, merge the Gaussian elements of all its child nodes into that node and release the storage resources of the child nodes.

4. The method according to claim 1, characterized in that, In step S2, for incremental octree nodes located within the view frustum, the projection size of the node's bounding box in screen space is calculated, i.e., the screen space error is calculated, and only nodes with a screen space error greater than a preset threshold ε are selected for rendering.

5. The method according to claim 1, characterized in that, Step S3 includes: S31: Calculate a representative depth for each block to be rendered, sort all blocks to be rendered according to their representative depths, and determine the rendering order between different blocks to be rendered. S32: Sort the Gaussian elements within each block to be rendered according to their respective depth values ​​to determine the rendering order among different Gaussian elements within the block to be rendered.

6. The method according to claim 5, characterized in that, In step S31, the Radix Sort algorithm is used to sort all blocks to be rendered according to their representative depths; in step S32, the Bitonic Sort algorithm is used to sort the Gaussian elements in each block to be rendered according to their respective depth values.

7. The method according to claim 1, characterized in that, In step S4, when executing the standard α blending formula, color and opacity are accumulated pixel by pixel until the opacity is greater than or equal to 1 or all primitives have been processed, and finally the rendered image is generated.

8. A real-time rendering system for large-scale 3D Gaussian splash scenes, characterized in that, The system includes: The incremental octree index structure construction module is used to preprocess the 3D Gaussian metadata of the original 3D space, construct the incremental octree index structure of the 3D space, divide the 3D space into multiple levels of incremental octree nodes, each incremental octree node corresponds to a block to be rendered, and each block to be rendered contains multiple Gaussian metadata. The frustum culling and LOD scheduling module is used to traverse the constructed incremental octree at the beginning of each frame rendering based on the current camera parameters, culling incremental octree nodes and subtrees located outside the frustum. For incremental octree nodes located within the frustum, different levels of detail Gaussian metadata are scheduled for rendering based on the distance between the incremental octree node and the camera. The hybrid sorting module is used to determine the rendering order between blocks to be rendered and the rendering order between different high-order primitives within a block to be rendered. The alpha blending and image compositing module projects Gaussian pixels onto a 2D screen sequentially according to the rendering order determined by the blending sorting module. It then applies the standard alpha blending formula to each pixel of the resulting 2D screen, accumulating color and opacity pixel by pixel to ultimately generate a rendered image.

9. The system according to claim 8, characterized in that, The incremental octree index structure construction module includes: The Morton coding and spatial partitioning module is used to calculate the Morton code of the three-dimensional position of each Gaussian element, mapping adjacent Gaussian elements in 3D space to a one-dimensional key-value space. The node splitting module recursively checks the Gaussian elements in each node, starting from the root node of the incremental octree. When the number of Gaussian elements in a node exceeds a preset threshold τ and the screen space projection size of the node's bounding box in the current reference view is greater than a preset threshold ε, node splitting is triggered. During splitting, the Gaussian elements are assigned to the corresponding 8 child nodes according to their Morton codes, and a loose axial bounding box with a relaxation factor γ is created for each child node. The above process is repeated until no node needs to be split. After splitting, the parent node clears its own Gaussian metadata, and only the child nodes store the corresponding Gaussian metadata.

10. The system according to claim 8, characterized in that, The hybrid sorting module includes: The block-level coarse sorting module is used to calculate a representative depth for each block to be rendered, sort all blocks to be rendered according to their representative depth, and determine the rendering order between blocks to be rendered. The fine sorting module within each block is used to sort the Gaussian elements in each block to be rendered according to their respective depth values, thereby determining the rendering order among different Gaussian elements within the block to be rendered.

Images