Scene rendering method and system based on three-dimensional Gaussian splashing

Abstract

The invention discloses a scene rendering method and system based on three-dimensional Gaussian splashing. The method comprises the following steps: organizing and constructing an original 3D Gaussian set to obtain a spatial hierarchical structure; generating corresponding level details for the primitives in the spatial hierarchical structure to obtain a spatial hierarchical structure associated with the level details; traversing the spatial hierarchical structure associated with level details, and executing hierarchical view cone cutting and shielding elimination to obtain a visible node list; traversing the visible node list, calculating a level detail selection standard, and generating a level detail activity Gaussian set; performing optimization sorting on the level detail activity Gaussian set to obtain an activity primitive list; and performing tile-based rasterization on the active primitive list, and performing adaptive processing according to level details of the primitives to obtain a final color value of each pixel. According to the method, the rendering performance can be greatly improved, the occupation of a memory and a video memory is remarkably reduced, the rendering quality is improved, visual flaws are reduced, and the expandability of the 3DGS rendering method is enhanced.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of three-dimensional scene representation and real-time rendering, in particular to a scene rendering method and system based on three-dimensional Gaussian splatting. BACKGROUND

[0002] In today's digital era, there is an increasing demand for high-fidelity, interactive three-dimensional reconstruction and rendering of real-world scenes, widely used in virtual reality (VR), augmented reality (AR), digital twinning, game entertainment, film production, cultural heritage protection, online product display, and many other fields. These application scenarios often require not only photo-realistic rendering results, but also the ability to observe from any new viewpoint (i.e., Novel View Synthesis), and the rendering speed to achieve real-time or even super-real-time levels. For example, VR applications usually require a refresh rate of 90Hz or higher to provide a smooth and immersive user experience.

[0003] For a long time, the representation and rendering of three-dimensional scenes have mainly relied on traditional methods based on explicit geometry (such as triangular meshes, point clouds) and texture mapping. These methods dominate computer graphics, with mature modeling tool chains and hardware-accelerated rendering pipelines. However, for complex structures, rich details, and real-world scenes containing non-Lambertian surfaces (such as reflective and transparent objects), it is often difficult, costly, and difficult to achieve complete photo-realism through traditional methods, especially in terms of Novel View Synthesis, which is prone to problems such as holes, distortion, or lack of lighting details. In recent years, image-based rendering (IBR) techniques have attempted to synthesize new viewpoints by capturing dense image information of the scene, but their viewpoint range and interactivity are usually limited.

[0004] To overcome the limitations of traditional methods, implicit neural representations based on neural networks, especially neural radiance fields (NeRF) and its variants, have made revolutionary progress in recent years. NeRF uses a deep neural network to learn a continuous function that maps three-dimensional space coordinates and viewing directions to the color and density of that point. By integrating camera rays through volume rendering techniques, NeRF can generate realistic and view-dependent images with stunning realism and view-dependent effects, such as reflections, refractions, and shadows. However, NeRF also has some limitations, such as the need for large amounts of training data, the difficulty of handling complex scenes with non-Lambertian surfaces, and the difficulty of achieving real-time rendering speeds. Figure OneNeRF is particularly good at handling complex geometric details, semi-transparent effects, and fine lighting variations. However, there is a fatal flaw in the NeRF method: its rendering process involves hundreds of neural network queries and integrations for each ray, which is computationally intensive, resulting in extremely slow rendering speed, often taking seconds or even minutes to render a frame of image, far from meeting the needs of real-time interaction. Although a large number of NeRF acceleration techniques have emerged subsequently (such as Instant-NGP, Plenoxels, MERF, etc. based on multi-resolution hash coding), they have improved the rendering speed to some extent, but often at the cost of sacrificing part of the rendering quality, increasing memory occupancy, or limiting the complexity of the scene, still having a gap from achieving universal high-quality real-time rendering on various devices.

[0005] In this context, 3D Gaussian Splatting (3DGS) emerged as a new scene representation and rendering technique based on explicit point primitives, quickly becoming a research hotspot in this field. The 3DGS method represents a three-dimensional scene as a collection of a large number (usually millions or more) of three-dimensional Gaussian distributions with optimized parameters. Each 3D Gaussian function is defined by its center position, shape and direction (represented by a covariance matrix, usually decomposed into rotation and scaling factors), opacity, and color (usually represented by spherical harmonics or other view-dependent functions). The rendering process of the scene uses a rasterization-based rather than volume rendering technique: first, all 3D Gaussian functions are clipped and projected onto the two-dimensional image plane according to the current camera frustum, forming a series of two-dimensional Gaussian splats; then, these two-dimensional splats are sorted by depth (usually from back to front); finally, on the pixels of the image (or more efficiently, on the image tiles), the sorted two-dimensional Gaussian splats are accumulated and blended according to their shape, color, and opacity, and the final rendered image is synthesized. Since the projection and blending calculations of Gaussian functions can be efficiently executed in parallel on modern graphics processing units (GPUs), and avoid the expensive per-point network queries and integrations in NeRF, 3DGS can achieve extremely high real-time rendering frame rates while maintaining or even surpassing the rendering quality of NeRF, for example, easily reaching tens or even hundreds of FPS on consumer-grade GPUs. In addition, the training process of 3DGS (usually initialized from sparse point clouds obtained from Structure-from-Motion, then optimized Gaussian parameters using a differentiable renderer) is also relatively fast.

[0006] The emergence of 3DGS technology has greatly promoted the development of high-quality real-time rendering and new view synthesis, showing great application potential in VR / AR, real-time preview, digital people, etc. Its development trends include: further improving rendering quality and efficiency, reducing memory occupancy, supporting larger-scale scene representation and rendering, realizing dynamic scene (scene changing over time) capture and rendering, exploring more effective editing and interaction methods, researching hybrid rendering with traditional rendering pipeline, and deployment optimization on resource-constrained devices such as mobile devices.

[0007] However, although 3DGS has made breakthroughs, as a new technology, it still faces a series of problems to be solved in practical applications. First, although the rendering speed is fast, when dealing with scenes containing a large number of Gaussian primitives (such as representing large-scale, high-detail scenes may require tens of millions or even hundreds of millions of Gaussians), the computational overhead of the rendering pipeline (especially in the sorting and rasterization stages) may still become a bottleneck, especially on high-resolution rendering or resource-limited devices. Second, storing the parameters (position, covariance, color coefficient, opacity, etc.) of a large number of Gaussian primitives requires a huge amount of memory or storage space, which puts high demands on memory bandwidth and capacity, limiting its application on memory-constrained platforms (such as mobile devices) or the ability to handle ultra-large-scale scenes. Third, although the rendering quality of 3DGS is high, it may still have visual defects in some cases, such as not sharp enough for very fine geometric structures or high-frequency texture details, blurred or "floaters" when the observation angle changes greatly, or holes or color overflow when the Gaussian distribution is not optimized. Fourth, the existing 3DGS rendering pipeline may have scalability challenges, such as how to efficiently render a small part of the screen in an ultra-large scene, or how to dynamically load / unload scene parts to support infinite world exploration. Therefore, there is a strong market demand for improved technologies that can further optimize 3DGS rendering performance, reduce resource consumption, improve rendering quality and robustness, and enhance scalability. Developing new rendering methods to fully exploit the advantages of 3DGS while overcoming its existing limitations is key to promoting the widespread application of this technology.

[0008] Although the standard 3D Gaussian Splatting (3DGS) method has made significant breakthroughs in real-time rendering quality and speed, as a relatively new technology, it still exposes some inherent technical defects and limitations in practical applications and further development, which limit its application potential in a wider range of scenarios.

[0009] (1) The rendering performance bottleneck is a major challenge faced by existing 3DGS methods, especially when dealing with complex or large-scale scenes containing a vast number of Gaussians (e.g., millions to tens of millions or even hundreds of millions). Two key steps in the rendering pipeline, Global Sorting and Rasterization & Blending, are the main sources of performance bottlenecks. The accurate depth sorting of millions of projected two-dimensional Gaussians per frame (usually using GPU radix sorting) itself requires considerable computational overhead and memory bandwidth. More seriously, during the rasterization stage, each pixel (or tile) can be covered by a large number of overlapping Gaussian blobs (high overdraw), resulting in intensive two-dimensional Gaussian function evaluation, color calculation (such as spherical harmonic decoding), and complex alpha blending operations. When the scene is very complex or has high resolution, this pixel / tile-by-pixel intensive calculation consumes a large amount of GPU computing resources and memory bandwidth, which can cause frame rate to drop, failing to meet the requirements of certain applications (such as high-resolution VR) for extreme real-time performance. This constitutes the defect of inefficiency and performance bottleneck.

[0010] (2) Existing 3DGS methods have significant memory and storage overhead problems. Each 3D Gaussian needs to store its position (3 floating-point numbers), rotation (usually 4 floating-point numbers of quaternion), scaling (3 floating-point numbers), opacity (1 floating-point number), and color representation (e.g., 3-order spherical harmonics require (3+1)² * 3 = 48 floating-point numbers, even low-order SHs require dozens of floating-point numbers). This means that representing a Gaussian may require tens to hundreds of bytes. For scenes containing millions or even hundreds of millions of Gaussians, the total data volume can reach GB or even tens of GB. This not only requires a large storage space, but more importantly, during rendering, these massive data need to be loaded into the GPU memory and frequently accessed, causing a huge pressure on GPU memory capacity and bandwidth. This limits the application of 3DGS technology on memory-constrained devices (such as mobile phones and entry-level GPUs), and hinders its ability to represent and render truly large-scale scenes (such as city-level). This is a defect of high cost (high resource consumption) and poor scalability.

[0011] (3) In terms of rendering quality, although 3DGS can usually achieve photo-realism, visual artifacts may still occur in certain specific cases. For example:

[0012] Blur and detail loss. For very fine geometric structures or high-frequency textures in the scene, using a limited number of Gaussians with certain sizes to fit may cause blurring or loss of details. The "soft" boundary characteristics of Gaussians may also make the representation of hard edges not sharp enough.

[0013] Holes and Floaters. Rendering holes may occur in areas where the training data is insufficient or the optimization is not sufficient. Sometimes there are also some translucent "floating" Gaussian artifacts that do not seem to be precisely attached to the surface.

[0014] Aliasing and Flickering. Fast viewpoint movement or the interaction of Gaussian size and pixel grid may cause edge aliasing (jaggies) or flickering over time, especially when the Gaussian projection size is close to or less than one pixel. Standard alpha blending also has difficulty in perfectly processing very thin or complex transparent / semi-transparent structures.

[0015] Limitations of View-Dependent Effects. Although using spherical harmonics can simulate certain view-dependent color changes, it is difficult to fully capture complex material reflection properties (such as specular reflection, anisotropic reflection) or fine gloss changes.

[0016] These quality issues, although may not always be obvious in all scenes, limit the performance of 3DGS in applications that pursue extreme visual fidelity or require accurate detail reproduction, and are functional imperfections.

[0017] (4) The existing 3DGS method has relatively limited editability and controllability. Although 3DGS is an explicit representation, it is theoretically easier to edit than implicit representations such as NeRF, but directly editing millions of independent Gaussian parameters (such as moving, deleting, changing properties) is still very difficult, and it is difficult to ensure that the edited scene still maintains realism and consistency. How to perform semantic-level editing (such as "remove this chair" or "change the color of the wall") or physically reasonable interactions (such as collision response) is still a challenge in research. This limits the application of 3DGS in scenarios that require content creation, interactive design, or integration of physical simulation, and is a functional limitation.

[0018] In summary, although the existing standard 3DGS rendering method has achieved great success, there are still objective technical defects in rendering performance (especially for large-scale scenes), memory / storage consumption, rendering quality in certain situations (details, flaws), and editability / controllability. These defects hinder the further popularization of the technology and its application in more extensive and demanding scenarios. SUMMARY

[0019] The purpose of the present application is to provide a three-dimensional Gaussian splash-based scene rendering method and system, which can greatly improve rendering performance, significantly reduce memory and video memory occupation, improve rendering quality, reduce visual flaws, and enhance the scalability of the 3DGS rendering method, to solve at least one of the above technical problems.

[0020] Embodiments of the present application are implemented as follows:

[0021] A scene rendering method based on three-dimensional Gaussian splatting, comprising:

[0022] S100, organizing and constructing a spatial hierarchy from an original 3D Gaussian set.

[0023] S200, generating corresponding level of detail for primitives in the spatial hierarchy, obtaining the spatial hierarchy associated with level of detail.

[0024] S300, traversing the spatial hierarchy associated with level of detail, performing hierarchical view frustum culling and occlusion culling, obtaining a visible node list.

[0025] S400, traversing the visible node list, calculating level of detail selection criteria, generating a level of detail active Gaussian set.

[0026] S500, optimizing the level of detail active Gaussian set, obtaining an active primitive list.

[0027] S600, performing tile-based rasterization on the active primitive list, performing adaptive processing according to the level of detail of the primitives, obtaining the final color value of each pixel.

[0028] In a preferred embodiment of the present application, in the above-mentioned scene rendering method based on three-dimensional Gaussian splatting, in S100, the organization and construction of the spatial hierarchy from the original 3D Gaussian set comprises:

[0029] S110, selecting an octree spatial partitioning strategy, recursively partitioning the space of the original 3D Gaussian set from the root node, if the number of Gaussians contained in the current node exceeds a predetermined threshold, or the spatial range of the current node is greater than a predetermined size, then the space of the current node is evenly divided into eight sub-nodes along three axes.

[0030] S120, assigning each 3D Gaussian primitive to the corresponding leaf node according to the center position, one 3D Gaussian primitive belongs to one corresponding leaf node.

[0031] S130, obtaining the constructed spatial hierarchy, the spatial hierarchy includes node, sub-node relationship, and node bounding box information.

[0032] The technical effect is that the spatial partitioning and hierarchy construction of all Gaussians are completed, the flat Gaussian set is structured, which facilitates spatial query and management, and provides a basic spatial index for subsequent fast culling, LOD management and rendering optimization.

[0033] In the preferred embodiment of the present application, in the above-mentioned scene rendering method based on three-dimensional Gaussian splatting, in S200, the level of detail corresponding to each primitive in the spatial hierarchy is generated, and the spatial hierarchy with associated levels of detail is obtained, which comprises:

[0034] In S210, all the 3D Gaussian primitives in a node in the spatial hierarchy are clustered, and the 3D Gaussian primitives in each cluster are Gaussian merged to obtain the merged Gaussian distribution.

[0035] In S220, Gaussian parameters of the merged Gaussian distribution are calculated, and statistical properties of Gaussians in a node are calculated, the Gaussian parameters including at least one of position, covariance, color and opacity, and the statistical properties including at least one of average position, total energy and dominant color.

[0036] In S230, according to the calculation and statistical results, a representative 3D Gaussian primitive in the current node is selected as the level of detail of the current node.

[0037] In S240, the level of detail corresponding to each level in the spatial hierarchy is generated, and the spatial hierarchy with associated levels of detail is obtained.

[0038] The technical effect is to provide a scene representation with lower calculation cost for use under different observation distances or rendering accuracy requirements, which is a key to adaptive rendering and performance optimization. A multi-resolution representation of the scene is realized, which provides a data basis for adaptive rendering.

[0039] In the preferred embodiment of the present application, in the above-mentioned scene rendering method based on three-dimensional Gaussian splatting, in S300, the spatial hierarchy with associated levels of detail is traversed, hierarchical view frustum culling and occlusion culling are performed, and a visible node list is obtained, which comprises:

[0040] In S310, the spatial hierarchy with associated levels of detail is traversed from the root node.

[0041] In S320, for each traversed node, an intersection test is performed between the bounding box of the node and the camera view frustum, and if the bounding box of the node is completely outside the view frustum, the node and all descendant nodes corresponding thereto are removed.

[0042] In S330, an occlusion query is performed, and if the bounding box of the node is completely occluded by an opaque object in the scene closer to the camera view frustum, the node and all descendant nodes corresponding thereto are removed.

[0043] Its technical effect lies in: traversal is completed, and all possible level node sets that can contribute to the current frame are determined.

[0044] In the preferred embodiment of the application, in the above-mentioned three-dimensional Gaussian splash-based scene rendering method, in S400, the step of traversing the visible node list, calculating level detail selection criteria, and generating a level detail active Gaussian set comprises:

[0045] In S410, the level detail used for rendering is calculated according to the distance of the node to the camera, the projection area of the node on the screen, and the visual importance index.

[0046] In S420, for the node, if simplified level detail is selected for representation, the corresponding child node no longer needs to be recursively processed, and a level detail primitive is obtained, and if high-level detail is selected for representation, the corresponding child node is further traversed, the selection process of the used level detail is repeated, and a 3D Gaussian primitive is obtained.

[0047] In S430, the level detail primitive and the 3D Gaussian primitive that need to be rendered are obtained to form a level detail active Gaussian set.

[0048] Its technical effect lies in: the LOD selection of all visible nodes is completed, and an active rendering set of the frame is generated. The number of primitives that need to be sorted and rasterized is further greatly reduced, the effect close to rendering all original Gaussians is maintained as much as possible in vision, adaptive adjustment of the rendering load is realized, and the number of primitives processed per frame is significantly reduced.

[0049] In the preferred embodiment of the application, in the above-mentioned three-dimensional Gaussian splash-based scene rendering method, in S500, the step of optimizing the sorting of the level detail active Gaussian set to obtain an active primitive list comprises:

[0050] In S510, hierarchical sorting is adopted to preliminarily sort the level detail primitives in the level detail active Gaussian set.

[0051] In S520, the 3D Gaussian primitives inside the leaf node that need to be expanded for rendering are locally sorted to obtain an active primitive list.

[0052] In the preferred embodiment of the application, in the above-mentioned three-dimensional Gaussian splash-based scene rendering method, in S500, the step of optimizing the sorting of the level detail active Gaussian set to obtain an active primitive list comprises:

[0053] In S530, approximate sorting is adopted to sort all primitives in the level detail active Gaussian set to obtain an active primitive list.

[0054] The technical effects are that the calculation complexity of the sorting stage is reduced, and the performance bottleneck problem of global sorting in the standard 3DGS is solved.

[0055] In the preferred embodiment of the present application, in the scene rendering method based on three-dimensional Gaussian splatting, in S600, the tile-based rasterization is performed on the active primitive list, adaptive processing is performed according to the level of detail of the primitive, and the final color value of each pixel is obtained.

[0056] In S610, a rasterizer capable of simultaneously processing the level of detail primitive and the 3D Gaussian primitive is selected.

[0057] In S620, adaptive processing is performed according to the level of detail of the primitive, a corresponding low calculation precision is used for a low-order level of detail, and a corresponding high calculation precision is used for a high-order level of detail.

[0058] In S630, tile-based rasterization is performed, and if the tile is saturated, subsequent primitive processing is terminated, and the final color value of each pixel is obtained.

[0059] The technical effects are that the hierarchical structure information helps to determine whether the tile has been saturated earlier, the alpha is close to 1, and subsequent more distant primitives are terminated in advance, and for the LOD primitive representing the entire cluster, the coverage range is larger, and the tile can be saturated faster.

[0060] In the preferred embodiment of the present application, in the scene rendering method based on three-dimensional Gaussian splatting, S600 further includes:

[0061] In S640, hardware acceleration is used, and corresponding shaders or calculation paths are used for different primitive types.

[0062] A scene rendering system based on three-dimensional Gaussian splatting includes:

[0063] A space division module is configured to organize and construct a spatial hierarchy from an original 3D Gaussian set.

[0064] A level of detail generation module is configured to generate corresponding levels of detail for primitives in the spatial hierarchy, and obtain the spatial hierarchy associated with the levels of detail.

[0065] A culling module is configured to traverse the spatial hierarchy associated with the levels of detail, perform hierarchical view frustum culling and occlusion culling, and obtain a visible node list.

[0066] A level of detail selection module is configured to traverse the visible node list, calculate a level of detail selection criterion, and generate an active Gaussian set associated with the levels of detail.

[0067] A sorting module is configured to sort the level-of-detail activity Gaussian set to obtain an active primitive list.

[0068] A rasterization and hybrid module is configured to perform tile-based rasterization on the active primitive list, and adaptively process the primitives according to the level-of-detail to obtain the final color value of each pixel.

[0069] The embodiment of the present application has the following advantages:

[0070] The present application introduces a spatial hierarchy (such as Octree) based 3DGS scene representation. First, in the preprocessing stage, the originally disorganized and large number of 3D Gauss primitives are constructed into a spatial hierarchy. This structure organizes the spatially adjacent Gauss together and provides a coarse-to-fine spatial indexing capability, which is fundamentally different from the standard 3DGS that regards all Gauss as a single set. This lays the foundation for subsequent rendering optimization, making efficient view frustum culling and occlusion culling possible, and quickly filtering out a large number of invisible Gauss, significantly reducing the amount of data entering the rendering pipeline.

[0071] The present application generates and uses multi-level-of-detail (LOD) representations for the nodes of the hierarchy. The present application generates simplified LOD representations for the nodes in the hierarchy, especially for non-leaf nodes. These LODs can approximate the visual appearance of a large number of original Gauss they represent at a lower computational cost. During rendering, the appropriate LOD level is dynamically selected according to the viewpoint, distance, or screen projection area, etc. The standard 3DGS has no LOD concept and always tries to render all visible original Gauss. The present application realizes adaptive control of rendering load, further significantly reduces the number and complexity of primitives including sorting and rasterization that need to be actually processed per frame, and significantly reduces the computational overhead and memory bandwidth requirements, on the premise of ensuring visual quality.

[0072] The present application proposes a sorting and rasterization method optimized by using the hierarchy and LOD. The active set produced by using the LOD selection is significantly reduced in size, and the sorting is performed, thereby directly reducing the sorting cost. An adaptive rasterization strategy is proposed, such as using simplified calculations like low-order SH and approximate shapes for primitives from low LOD, or using hierarchical information to optimize tile processing like early termination, which improves the two main performance bottleneck links of sorting and rasterization in the standard 3DGS. The performance bottleneck problem of the standard 3DGS in processing a large number of Gauss is directly solved. Through the optimization of sorting and rasterization calculation, the rendering frame rate is significantly improved, especially for large-scale, high-detail scenes or high-resolution rendering effects.

[0073] The present application enhances the scalability and resource efficiency of the 3DGS rendering method. The hierarchical structure and LOD mechanism naturally support efficient processing of large-scale scenes. The system can only load and process the fine details of the area close to the viewpoint or high visual importance, and use highly simplified LOD for the distant or unimportant area. This makes the rendering cost no longer proportional to the total number of Gauss in the scene, but more related to the complexity within the field of view. It greatly improves the feasibility of the 3DGS method to process super large-scale scenes. At the same time, due to the reduction of the number of primitives processed during rendering, the pressure on GPU memory bandwidth is reduced, and the possibility of deployment on memory-limited devices is provided, improving the utilization efficiency of resources. BRIEF DESCRIPTION OF DRAWINGS

[0074] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following will briefly introduce the drawings needed to be used in the embodiments. It should be understood that the following drawings only show some embodiments of the present application, and therefore should not be regarded as a limitation on the scope. For those skilled in the art, other related drawings can also be obtained without creative labor.

[0075] Figure 1 The present application is based on a three-dimensional Gaussian splash scene rendering method flowchart. DETAILED DESCRIPTION

[0076] In order to make the purpose, technical scheme and advantages of the embodiments of the present application more clear, the following will combine the drawings in the embodiments of the present application to clearly and completely describe the technical scheme in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all. The components of the embodiments of the present application described and shown in the drawings herein can be arranged and designed in various different configurations.

[0077] Please refer to Figure 1 The first embodiment of the present application provides a three-dimensional Gaussian splash-based scene rendering method, which comprises: S100, organizing and constructing an original 3D Gaussian set to obtain a spatial hierarchical structure; S200, generating corresponding level of detail for primitives in the spatial hierarchical structure to obtain the spatial hierarchical structure associated with level of detail; S300, traversing the spatial hierarchical structure associated with level of detail, performing hierarchical view frustum culling and occlusion culling to obtain a visible node list; S400, traversing the visible node list, calculating level of detail selection criteria, and generating a level of detail active Gaussian set; S500, optimizing and sorting the level of detail active Gaussian set to obtain an active primitive list; S600, performing tile-based rasterization on the active primitive list, and adaptively processing according to the level of detail of the primitives to obtain the final color value of each pixel.

[0078] In the preferred embodiment of the present application, in the above-mentioned scene rendering method based on three-dimensional Gaussian splatting, in S100, the organization and construction of the original 3D Gaussian set to obtain a spatial hierarchy structure comprises: S110, selecting an octree spatial division strategy, recursively dividing the space of the original 3D Gaussian set from the root node, and if the number of Gaussians contained in the current node exceeds a preset threshold or the spatial range of the current node is greater than a preset size, then the space of the current node is divided into eight sub-nodes along three axes; S120, each 3D Gaussian primitive is assigned to the corresponding leaf node at the bottom layer according to the center position, and one 3D Gaussian primitive belongs to one corresponding leaf node; S130, a constructed spatial hierarchy structure is obtained, and the spatial hierarchy structure comprises node, sub-node relationship, and node bounding box information.

[0079] Specifically, in addition to the octree structure, a bounding volume hierarchy BVH can also be generated. The trigger condition is the first loading of scene data or after the update of scene data. The running environment parameters include hierarchy structure types such as Octree and BVH, node splitting thresholds such as the number of Gaussians and the spatial size, and the maximum depth of the tree. The running principle is to apply a standard spatial data structure construction algorithm to organize the unordered Gaussian set into a hierarchy structure with spatial proximity. After the spatial division and hierarchy construction of all Gaussians are completed, the flat Gaussian set is structured, which facilitates spatial query and management, and provides a basic spatial index for subsequent fast clipping, LOD management, and rendering optimization.

[0080] In the preferred embodiment of the present application, in the above-mentioned scene rendering method based on three-dimensional Gaussian splatting, in S200, the generation of corresponding level of detail for primitives in the spatial hierarchy structure to obtain the spatial hierarchy structure associated with the level of detail comprises: S210, clustering all 3D Gaussian primitives in a node in the spatial hierarchy structure based on spatial position and / or color similarity, performing Gaussian merging on the 3D Gaussian primitives in each cluster to obtain a merged Gaussian distribution comprising one or a few equivalent larger Gaussians; S220, calculating Gaussian parameters for the merged Gaussian distribution, calculating statistical characteristics of Gaussians in a node, the Gaussian parameters comprising at least one of position, covariance, color, and opacity, and the statistical characteristics comprising at least one of average position, total energy approximately equal to opacity*area, and dominant color, and using one or several parameterized primitives, which are not necessarily Gaussian primitives, but can also be colored spheres or other simplified geometries, to represent; S230, selecting a representative 3D Gaussian primitive from the node as the level of detail LOD of the current node according to the calculation and statistical results; and S240, generating corresponding level of detail for each level in the spatial hierarchy structure to obtain the spatial hierarchy structure associated with the level of detail.

[0081] Specifically, the method of calculating Gaussian parameters includes weighted average, matrix matching or optimization-based method. Generally, the closer to the root node, the higher the level of the node, the higher the degree of simplification of its LOD representation, the fewer and larger the Gaussians or primitives contained. The running environment parameters include LOD generation algorithm selection, clustering parameters, merging strategy, target simplification rate or error threshold of each level. The running principle is to create a multi-resolution representation for different parts of the hierarchy by applying model simplification, clustering, statistical analysis and other techniques. Providing a scene representation with lower computational cost for use under different observation distances or rendering accuracy requirements is the key to adaptive rendering and performance optimization. The multi-resolution representation of the scene is realized, providing a data basis for adaptive rendering.

[0082] In the preferred embodiment of the present application, in the above-mentioned scene rendering method based on three-dimensional Gaussian splashing, in S300, the traversal of the spatial hierarchy associated with level of detail is performed, hierarchical view frustum culling and occlusion culling are performed to obtain a list of visible nodes, including: S310, starting from the root node, traversing the spatial hierarchy associated with the level of detail; S320, for each traversed node, performing intersection test between the bounding box of the node and the camera view frustum, if the bounding box of the node is completely outside the view frustum, then the node and all descendant nodes corresponding to the node are removed; S330, performing occlusion query, if the bounding box of the node is completely occluded by opaque objects or high-density Gaussian regions in the scene that are closer to the camera view frustum, then the node and all descendant nodes corresponding to the node are removed.

[0083] Specifically, the hierarchy helps to perform efficient occlusion query, such as hierarchical occlusion culling based on level, Hierarchical Z-Buffer Culling. The running environment parameters include view frustum culling algorithm, occlusion culling algorithm and camera parameters. The running principle is to quickly exclude a large number of invisible Gaussians by using the hierarchy, which is much more efficient than testing each Gaussian individually. After traversal, the set of all hierarchical nodes that may contribute to the current frame is determined. The number of Gaussians that need to be processed subsequently is significantly reduced, greatly reducing the load of the subsequent stages of the rendering pipeline.

[0084] In the preferred embodiment of the present application, in the above-mentioned scene rendering method based on three-dimensional Gaussian splatting, in S400, the traversing of the visible node list, the calculation of level-of-detail selection criteria, and the generation of a level-of-detail active Gaussian set comprises: S410, according to the distance of a node to a camera, the projected area of a node on a screen, and a visual importance index, the level-of-detail used for rendering is calculated, for example, nodes that are very far from the camera or have very small projected areas on the screen can select very simplified LODs, possibly only one combined Gaussian or primitive, nodes that are closer or have larger projected areas need to use more refined LODs, until the leaf nodes at the bottom use the original Gaussians contained therein; S420, for the node, if simplified level-of-detail is selected for representation, the corresponding child nodes no longer need to be recursively processed, and a level-of-detail primitive is obtained, if high-level-of-detail is selected for representation, the corresponding child nodes are further traversed, the selection process of the used level-of-detail is repeated, and a 3D Gaussian primitive is obtained; S430, the level-of-detail primitive and the 3D Gaussian primitive obtained for rendering constitute a level-of-detail active Gaussian set.

[0085] Specifically, the running environment parameters include LOD selection criteria such as distance-based threshold values and screen space error threshold values, camera parameters, and screen resolution. The running principle is to achieve adaptive level-of-detail rendering, dynamically select appropriate representations according to observation conditions, and avoid rendering too much detail in unnecessary places. After the LOD selection of all visible nodes is completed, an active rendering set for this frame is generated. Further, the number of primitives that need to be sorted and rasterized is greatly reduced, while the visual effect is as close as possible to that of rendering all original Gaussians, the adaptive adjustment of rendering load is achieved, and the average number of primitives processed per frame is significantly reduced.

[0086] In the preferred embodiment of the present application, in the above-mentioned scene rendering method based on three-dimensional Gaussian splatting, in S500, the optimization sorting of the level-of-detail active Gaussian set to obtain an active primitive list comprises: S510, hierarchical sorting is adopted for the active set containing LOD primitives representing the entire cluster, and the level-of-detail primitives in the level-of-detail active Gaussian set or the nodes to which they belong are preliminarily sorted; S520, the 3D Gaussian primitives inside the leaf nodes that need to be expanded for rendering are locally sorted to obtain an active primitive list, and hierarchical sorting is more efficient than global flattening sorting.

[0087] In the preferred embodiment of the present application, in the three-dimensional Gaussian splash-based scene rendering method, in S500, the active primitive list is obtained by optimizing and sorting the level-of-detail active Gaussian set, including: in S530, for the LOD primitives with very long distance or small visual contribution, approximate sorting is adopted, and all the primitives in the level-of-detail active Gaussian set are sorted, and the active primitive list is obtained based on the node center depth, so that the sorting speed is faster at the expense of slight mixing accuracy.

[0088] Specifically, the running environment parameters include sorting algorithms such as GPU radix sorting and sorting optimization strategies, and whether to enable hierarchical / approximate sorting. The calculation complexity of the sorting stage is reduced, and the performance bottleneck problem of global sorting in the standard 3DGS is solved.

[0089] In the preferred embodiment of the present application, in the three-dimensional Gaussian splash-based scene rendering method, in S600, the active primitive list is obtained by optimizing and sorting the level-of-detail active Gaussian set, including: in S530, for the LOD primitives with very long distance or small visual contribution, approximate sorting is adopted, and all the primitives in the level-of-detail active Gaussian set are sorted, and the active primitive list is obtained based on the node center depth, so that the sorting speed is faster at the expense of slight mixing accuracy.

[0090] Specifically, the hierarchical structure information helps to determine whether the tile is saturated earlier, that is, the alpha is close to 1, so that the subsequent primitives are terminated in advance, and the final color value of each pixel is obtained.

[0091] In the preferred embodiment of the present application, in the three-dimensional Gaussian splash-based scene rendering method, in S600, it further includes: in S640, hardware acceleration is adopted, and corresponding shaders or calculation paths are adopted for different primitive types.

[0092] Specifically, the running environment parameters include tile size, rasterization algorithm, mixed precision setting and LOD processing strategy. The running principle is to further utilize the LOD information to optimize the calculation in the rasterization stage, use cheaper calculation for the part with small visual contribution, and concentrate resources to process the high details of the nearby and important areas. The performance bottleneck of mixed rasterization in high-density areas in the standard 3DGS is solved, and the quality and performance can be balanced by adaptive precision control. The calculation load of the rasterization stage is significantly reduced, the rendering frame rate is improved, and the quality / performance trade-off capability is provided.

[0093] The second embodiment of the application provides a scene rendering system based on three-dimensional Gaussian splash, which comprises: a space division module for organizing and constructing a space hierarchy from an original 3D Gaussian set; a level of detail generation module for generating corresponding levels of detail for primitives in the space hierarchy to obtain the space hierarchy associated with the levels of detail; a culling module for traversing the space hierarchy associated with the levels of detail, performing hierarchical view frustum culling and occlusion culling to obtain a visible node list; a level of detail selection module for traversing the visible node list, calculating a level of detail selection criterion, and generating an active Gaussian set of levels of detail; an ordering module for optimizing the active Gaussian set of levels of detail to obtain an active primitive list; and a rasterization and mixing module for performing tile-based rasterization on the active primitive list, performing adaptive processing according to the levels of detail of the primitives, and obtaining the final color value of each pixel.

[0094] The embodiments of the application aim to protect a scene rendering method and system based on three-dimensional Gaussian splash, which have the following effects:

[0095] 1. The application can filter out a large number of invisible or unimportant Gaussians / nodes in the early stage of rendering through hierarchical culling and LOD selection, and replace a large number of original Gaussians in distant or unimportant detail areas with LODs with low calculation cost. This greatly reduces the number of active primitives that need to be sorted and rasterized. At the same time, adaptive rasterization with LOD further reduces the calculation load of pixels / tiles. It significantly reduces the pressure on the key bottleneck of the rendering pipeline, thereby greatly improving the rendering frame rate, making it possible to render larger and more detailed 3DGS scenes in real time.

[0096] 2. The present application significantly reduces the amount of primitive data that needs to be read from the video memory and processed per frame by using LODs. When rendering distant objects, only a small number of LOD primitives, which can be more compact in parameters, need to be read and processed, instead of hundreds or thousands of original Gaussians. Although the hierarchical structure and the LODs themselves require storage space, the reduction in data access during rendering is usually more significant. It effectively reduces the GPU memory bandwidth pressure during rendering, improves cache efficiency, and provides the possibility of running 3DGS or processing larger scenes on devices with limited memory resources.

[0097] 3. The hierarchical structure and the LOD mechanism of the present application naturally support scalable rendering. The rendering cost depends more on the complexity near the viewpoint and the screen resolution, rather than the total size of the scene. The system can efficiently cull and simplify most of the scene content far from the viewpoint. This enables the 3DGS-based method to efficiently render super-large-scale environments that were previously difficult to handle, greatly expanding its application range, such as in digital cities, large-scale scene roaming, etc.

[0098] 4. The LOD selection strategy of the present application can be dynamically adjusted according to real-time performance requirements. For example, when the GPU load is too high, a more aggressive LOD strategy can be selected, such as using coarser LODs or starting simplification at a closer distance, sacrificing part of the visual quality to ensure smooth frame rate; conversely, when the load allows, finer LODs can be used to obtain higher quality. This gives the rendering system the ability to adaptively adjust the balance between performance and quality, enabling it to provide acceptable user experience on different hardware platforms and performance targets.

[0099] 5. The present application can naturally reduce the noise or floating feeling of distant details by using optimized LOD representations, such as merged Gaussians that can have smoother characteristics when rendering distant objects. The hierarchical structure also provides the possibility of implementing more advanced, spatial relationship-considered anti-aliasing or filtering techniques. In some cases, it can improve the visual stability of the rendered image and reduce defects in distant or complex areas.

[0100] The computer program product based on the three-dimensional Gaussian splash-based scene rendering method and device provided by the embodiment of the present application includes a computer readable storage medium storing program code, the program code includes instructions for executing the method in the preceding method embodiment, and specific implementation can be referred to the method embodiment, which will not be described here.

[0101] Specifically, the storage medium can be a general storage medium such as a mobile disk, a hard disk, etc., and when the computer program on the storage medium is run, the computer program can execute the above-mentioned scene rendering method based on three-dimensional Gaussian splatting, thereby significantly improving rendering performance, reducing resource consumption, enhancing scalability and flexibility, and possibly improving visual quality stability, solving the main pain points of the prior art, and enabling the 3DGS technology to better serve application scenarios with higher requirements for real-time performance, scale, and resource efficiency.

[0102] If the functions are implemented in the form of software function units and sold or used as independent products, they can be stored in a non-volatile computer readable storage medium executable by a processor. Based on this understanding, the technical solutions of the present application or the part of the prior art that essentially contributes to the present application or the part of the technical solutions can be embodied in the form of a software product, which is stored in a storage medium and includes instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method described in the various embodiments of the present application. The aforementioned storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and various media that can store program codes.

[0103] Finally, it should be noted that the above-described embodiments are merely specific implementations of the present application, which are used to illustrate the technical solutions of the present application, but not to limit the same, and the protection scope of the present application is not limited thereto. Although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that any person skilled in the art can make modifications or easily think of changes to the technical solutions described in the foregoing embodiments within the technical scope disclosed by the present application, or make equivalent replacements to some of the technical features; and these modifications, changes or replacements do not cause the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present application, and should be covered within the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A scene rendering method based on three-dimensional Gaussian splashing, characterized in that, include: S100 organizes and constructs a spatial hierarchical structure from the original 3D Gaussian set; S200, Generate corresponding level details for the primitives in the spatial hierarchy to obtain the spatial hierarchy with associated level details; S300, traverse the spatial hierarchy of the associated level details, perform hierarchical view frustum culling and occlusion culling, and obtain a list of visible nodes; S400, traverse the list of visible nodes, calculate the level detail selection criteria, and generate a level detail activity Gaussian set; S500, optimize and sort the Gaussian set of activities at the level of detail to obtain a list of activity primitives; S600, perform tile-based rasterization on the active primitive list, and perform adaptive processing according to the level details of the primitives to obtain the final color value of each pixel.

2. The scene rendering method based on three-dimensional Gaussian splashing according to claim 1, characterized in that, In S100, organizing and constructing the spatial hierarchy from the original 3D Gaussian set includes: S110, Select the octree space partitioning strategy, and recursively partition the space of the original 3D Gaussian set starting from the root node. If the number of Gaussians contained in the current node exceeds the preset threshold, or the space range of the current node is larger than the preset size, then the space of the current node is divided into eight child nodes along the three axes. S120, each 3D Gaussian element is assigned to its lowest-level leaf node according to its center position, and one 3D Gaussian element belongs to one corresponding leaf node; S130, the constructed spatial hierarchy structure is obtained, which includes nodes, child node relationships, and node bounding box information.

3. The scene rendering method based on three-dimensional Gaussian splashing according to claim 2, characterized in that, In S200, generating corresponding level details for the primitives in the spatial hierarchy to obtain the spatial hierarchy with associated level details includes: S210, cluster all the 3D Gaussian elements in a node of the spatial hierarchy, and perform Gaussian merging on the 3D Gaussian elements in each cluster to obtain the merged Gaussian distribution. S220, calculate Gaussian parameters for the merged Gaussian distribution, and calculate the statistical properties of Gaussian within a node. The Gaussian parameters include at least one of position, covariance, color, and opacity, and the statistical properties include at least one of average position, total energy, and dominant color. S230, Based on the calculation and statistical results, select representative 3D Gaussian elements from the current node as the level details of the current node; S240, generate corresponding level details for each level in the spatial hierarchy to obtain the spatial hierarchy with associated level details.

4. The scene rendering method based on three-dimensional Gaussian splashing according to claim 1, characterized in that, In S300, the spatial hierarchy of the associated level details is traversed, hierarchical view frustum culling and occlusion removal are performed, and the resulting list of visible nodes includes: S310, Starting from the root node, traverse the spatial hierarchy of the association level details; S320, For each traversed node, perform an intersection test between the bounding box of the node and the camera view frustum. If the bounding box of the node is completely outside the view frustum, then discard the node and all its corresponding descendant nodes. S330, perform an occlusion query. If the bounding box of the node is completely occluded by an opaque object in the scene that is closer than the camera's view frustum, then remove the node and all its corresponding descendant nodes.

5. The scene rendering method based on three-dimensional Gaussian splashing according to claim 1, characterized in that, In S400, traversing the list of visible nodes, calculating the level detail selection criteria, and generating the level detail activity Gaussian set includes: S410 calculates the level of detail used for rendering based on the distance from the node to the camera, the projected area of ​​the node on the screen, and the visual importance index. S420, for the node, if the simplified level of detail is selected for representation, then it is no longer necessary to recursively process the corresponding child nodes to obtain the level of detail primitives. If the high level of detail is selected for representation, then the corresponding child nodes are traversed downwards, and the selection process of the level of detail used is repeated to obtain the 3D high level primitives. S430, obtain the level detail primitives and the 3D Gaussian primitives to be rendered, and form a level detail active Gaussian set.

6. The scene rendering method based on three-dimensional Gaussian splashing according to claim 5, characterized in that, In S500, the optimization and sorting of the Gaussian set of activities at the level of detail to obtain the list of activity primitives includes: S510, using hierarchical sorting, the level detail primitives in the level detail activity Gaussian set are initially sorted; S520, perform local sorting on the 3D Gaussian primitives inside the leaf nodes that need to be expanded and rendered to obtain a list of active primitives.

7. The scene rendering method based on three-dimensional Gaussian splashing according to claim 5, characterized in that, In S500, the optimization and sorting of the Gaussian set of activities at the level of detail to obtain the list of activity primitives includes: S530, using approximate sorting, sort all primitives in the Gaussian set of activity levels of detail to obtain a list of activity primitives.

8. The scene rendering method based on three-dimensional Gaussian splashing according to claim 5, characterized in that, In S600, the step of performing tile-based rasterization on the active primitive list and adaptively processing according to the level detail of the primitives to obtain the final color value of each pixel includes: S610, Select a rasterizer capable of simultaneously processing the level detail primitives and the 3D Gaussian primitives; S620 adaptively processes details based on the level of primitives, using lower computational precision for lower-level details and higher computational precision for higher-level details. S630 performs tile-based rasterization. If the tile is saturated, the processing of subsequent primitives is terminated to obtain the final color value of each pixel.

9. The scene rendering method based on three-dimensional Gaussian splashing according to claim 8, characterized in that, The S600 also includes: The S640 employs hardware acceleration, using corresponding shaders or computation paths for different primitive types.

10. A scene rendering system based on three-dimensional Gaussian splashing, characterized in that, include: The spatial partitioning module is used to organize and construct a spatial hierarchy from the original 3D Gaussian set. The level detail generation module is used to generate corresponding level details for the primitives in the spatial hierarchy, thereby obtaining the spatial hierarchy with associated level details. The culling module is used to traverse the spatial hierarchy of related level details, perform hierarchical view frustum culling and occlusion culling, and obtain a list of visible nodes; The level detail selection module is used to traverse the list of visible nodes, calculate the level detail selection criteria, and generate a level detail activity Gaussian set. The sorting module is used to optimize and sort the Gaussian set of activities at the level of detail to obtain a list of activity primitives; The rasterization and blending module is used to perform tile-based rasterization on the list of active primitives, and to adaptively process the primitives according to their level of detail to obtain the final color value of each pixel.

Images