3D gaussian splatting rendering optimization method in VR based on unity engine

Abstract

The application discloses a 3D Gaussian splash rendering optimization method based on a Unity engine in VR, relates to the technical field of computer graphics, and converts a PLY point cloud file into a SOG high compression format containing multi-level LOD information, removes low-transparency Gaussian points, carries out Morton coding sorting on Gaussian point center coordinates, constructs an octree space index containing a space bounding box and multi-LOD data, removes Gaussian points according to camera distance grading and selects LOD levels, carries out depth bucket sorting on visible Gaussian points, executes drawing in the unit of a bucket, combines near-to-far sorting, an improved mixing equation and screen space edge removal and magnification operation, extracts Gaussian ellipse center points to calculate motion vectors, and fuses multiple optimization results to generate frame image output. The application improves VR device loading efficiency, reduces rendering burden, adapts to Tile-Based hardware, and realizes smooth running of 3D Gaussian splash on a VR all-in-one machine through multi-dimensional optimization such as loading, space removal and rendering.

Description

Technical Field

[0001] This invention belongs to the field of computer graphics technology, specifically relating to a method for optimizing 3D Gaussian splash rendering in VR based on the Unity engine. Background Technology

[0002] Existing 3D Gaussian splash algorithm (3DGS) often faces performance bottlenecks when running on VR all-in-one devices, making it unable to run smoothly. Furthermore, the high resolution and high refresh rate requirements of VR devices place higher demands on computing resources.

[0003] Therefore, an effective algorithm optimization method is needed to improve the running efficiency of 3DGS on VR devices. Summary of the Invention

[0004] To address the problems mentioned in the background, this invention provides a 3D Gaussian splash rendering optimization method based on the Unity engine in VR, which solves the problem that existing 3D Gaussian splash algorithms often face performance bottlenecks and cannot run smoothly on VR all-in-one devices.

[0005] To achieve the above objectives, the present invention provides the following technical solution:

[0006] The optimization method for 3D Gaussian splash rendering in VR based on the Unity engine includes the following steps:

[0007] S1: Data preprocessing and format conversion; convert the original PLY point cloud file into an SOG file. The SOG file stores the center coordinates, radius, direction matrix, color, transparency, and LOD level data of the Gaussian points.

[0008] S2: Alpha Removal and Reorganization; Parse the SOG file, set the Alpha threshold, remove Gaussian points with transparency below the Alpha threshold, and rewrite the filtered Gaussian points into a new SOG file.

[0009] S3: Morton coding sorting; Calculate the Morton code for the center coordinates of the filtered Gaussian points, and sort the Gaussian points according to the coding results;

[0010] S4: Construct an octree spatial index; using sorted Gaussian points as input, construct an octree spatial structure. Each node of the octree contains the bounding box of the three-dimensional spatial range it manages, the Gaussian point data of different LOD levels contained in the node, and the child node index pointing to its child nodes.

[0011] S5: Distance-based elimination and LOD fusion; Based on the distance from the camera to the center of the octree node, determine the retention ratio of Gaussian points and select the corresponding LOD level, and output the set of visible Gaussian points;

[0012] S6: Depth Bucket Sort; Divide the depth range of visible Gaussian points into multiple buckets, and perform sorting only between buckets. If the camera movement distance is less than the preset motion threshold, skip the sorting operation between buckets.

[0013] S7: Rendering stage and screen space optimization; Draw calls are executed in units of buckets, using a sorting method from near to far and a preset blending equation, and edge culling and magnification operations are performed on visible Gaussian points in screen space based on radial distance;

[0014] S8: Spatial warp adaptation; Based on the center coordinates and direction matrix of the visible Gaussian point, determine the Gaussian ellipse formed by its screen projection, extract the center point of each Gaussian ellipse to calculate the motion vector and corresponding depth value;

[0015] S9: Output and Display; Combine the edge culling results of S7, the distance culling results of S5, and the spatial distortion adaptation results of S8 to generate the final frame image and output it to the VR display.

[0016] Preferably, in S1, the original PLY point cloud file is converted into an SOG file through a toolchain, the toolchain being the open-source command-line tool SplatTransform, and the multi-level LOD information includes three precision levels: high, medium, and low.

[0017] Preferably, in S3, the Morton encoding is calculated by the EncodeMorton3D function, which processes the x, y, and z components of the center coordinates of the Gaussian point using the Part1By2 function and then combines them to obtain the encoding result.

[0018] Preferably, in S5, the specific rules for distance-based elimination are as follows:

[0019] When the distance from the camera to the node center is less than the first threshold, the retention ratio is set to 1 and the highest precision LOD level is selected.

[0020] When the distance is between the first threshold and the second threshold, the retention ratio decreases linearly with the distance and a medium-precision LOD level is selected.

[0021] When the distance is greater than the second threshold, the retention ratio is set to the preset minimum value and the lowest precision LOD level is selected.

[0022] Preferably, in S6, the number of buckets for dividing the depth interval is 256, and the sorting operation is performed only for a single main camera.

[0023] Preferably, in S7, the preset mixing equation is Blend(OneMinusDstAlpha,One).

[0024] Preferably, in S7, when culling the screen space edge, an inner radius and an outer radius are defined. When the radial distance of the Gaussian point corresponding to the screen is less than or equal to the inner radius, the retention ratio is 1; when the radial distance is between the inner radius and the outer radius, the retention ratio gradually decreases; when the radial distance is greater than or equal to the outer radius, the retention ratio is 0; and a magnification operation based on the radial distance is performed on the retained Gaussian points.

[0025] Preferably, in S8, the motion vector is the coordinate difference of the center point of the Gaussian ellipse at adjacent time points.

[0026] Preferably, in S7, the drawing call is executed in batches with a fixed number of Gaussian points.

[0027] Compared with the prior art, the beneficial effects of the present invention are:

[0028] 1. Improved loading efficiency: The SOG high compression format is adopted, combined with Alpha culling and Morton encoding sorting, to shorten the model initialization time, improve the GPU cache hit rate, and adapt to the bandwidth limitations of VR all-in-one devices.

[0029] 2. Highly efficient spatial culling: By using octree spatial partitioning and distance-based hierarchical culling, large-scale point clouds can be quickly culled, preserving the foreground fully, preserving the midground proportionally, and preserving less of the background, thus reducing the rendering burden;

[0030] 3. Rendering performance optimization: Depth bucket sort eliminates the need for precise point-by-point sorting, and the threshold mechanism reduces redundant sorting, improving real-time performance; Batch rendering and near-to-far sorting, combined with an improved blending method, reduce redundant computations.

[0031] 4. Screen and frame prediction optimization: EdgeCulling gradually removes edge points and enlarges the retained points to maintain immersion; motion vectors are calculated using the center point of the Gaussian ellipse to avoid artifacts and reduce overhead. Overall collaborative optimization makes 3D Gaussian splashing run smoothly on VR all-in-one devices. Attached Figure Description

[0032] Figure 1 and Figure 2 This is a flowchart illustrating the process of this application. Detailed Implementation

[0033] To facilitate understanding of the technical content of this invention by those skilled in the art, the invention will be further described in detail below with reference to the accompanying drawings and specific examples. It should be understood that the specific examples described herein are merely illustrative and not intended to limit the scope of the invention.

[0034] Optimization methods for 3D Gaussian splash rendering in VR based on the Unity engine, such as Figure 1 , Figure 2 As shown, the specific steps include:

[0035] S1: Data preprocessing and format conversion;

[0036] The original PLY point cloud file is input and converted into a highly compressed SOG (Self-Organizing Gaussians) format using a toolchain (the open-source command-line tool SplatTransform). In addition to storing the center coordinates, radius, orientation matrix, color, and transparency of the Gaussian points, the SOG file also adds multi-level LOD (Level of Detail) information.

[0037] ;

[0038] These correspond to three precision levels: high, medium, and low.

[0039] The structure of each Gaussian point is as follows:

[0040] ;

[0041] in: With the center coordinates, For radius, This is the direction matrix (result of covariance decomposition). For transparency, To which level belongs;

[0042] S2: Alpha removal and reorganization;

[0043] Parse the SOG file and traverse all Gaussian splats. Set threshold ,like If so, then remove that point:

[0044] };

[0045] After filtering, the results are rewritten into a new SOG file as input for subsequent processing.

[0046] S3: Morton coding sorting (caching optimization);

[0047] For each Gaussian point Calculate Morton Code (Z-order):

[0048] ;

[0049] The pseudocode for unfolding EncodeMorton3D is:

[0050] uint EncodeMorton3D(int x,int y,int z)

[0051] {

[0052] return(Part1By2((uint)z)<<2)+(Part1By2((uint)y)<<1)+Part1By2((uint)x);

[0053] }

[0054] uint Part1By2(uint x)

[0055] {

[0056] x&=0x000003ff;

[0057] x=(x^(x<<16))&0xff0000ff;

[0058] x=(x^(x<<8))&0x0300f00f;

[0059] x=(x^(x<<4))&0x030c30c3;

[0060] x=(x^(x<<2))&0x09249249;

[0061] return x;

[0062] }

[0063] Morton sorting maintains local consistency in the storage order of adjacent spatial points, thereby improving GPU cache hit rate and reducing bandwidth waste.

[0064] In this step, after loading is complete, Gaussian points that have negligible visual contribution are first removed to reduce unnecessary rendering overhead. Then, all point data are sorted using Morton encoding, so that during GPU access, spatially adjacent points are included in the same cache line, thereby improving cache hit rate and reducing GPU bandwidth waste, which is especially suitable for large-scale datasets (more than a million points).

[0065] S4: Construct the Splattree spatial index;

[0066] Using the sorted SOG as input, an octree (Splattree) spatial structure is constructed. During the construction process, the SOG files are simultaneously categorized and stored according to their LOD level (low-precision level).

[0067] ;

[0068] Each node contains: its bounding box, point data at three LOD levels, and a child node index.

[0069] Pseudocode example:

[0070] function BuildSplattree(points,depth): if depth==MaxDepth or len(points) <MinCount: return LeafNode(points grouped by LOD) subdivide spaceinto 8 octants for each octant: child=BuildSplattree(points_in_octant,depth+1) return Node(children)

[0071] In this step, after the model is loaded, Gaussian points and Level of Detail (LOD) are spatially partitioned using an octree (Splattree). This structure not only facilitates rapid view frustum pruning and removal of invisible points, but also provides an efficient data organization foundation for subsequent optimization operations such as distance-based culling.

[0072] S5: Distance-based elimination and LOD fusion;

[0073] Before rendering, based on the distance from the camera to the node center Calculate the retention ratio and select the appropriate LOD:

[0074] ;

[0075] in:

[0076] Distance threshold The linear attenuation coefficient is... To minimize the retention ratio, the output is a set of visible Gaussian points selected by adaptive LOD based on distance culling;

[0077] In this step, the invention draws on the concept of Shadow Map Cascade, dividing the rendering space into multiple ranges based on camera distance and combining it with LOD (Local Area Definition): In the near-distance range (less than A), all Gaussian points are retained; in the mid-distance range (between A and B), the retention ratio of points is reduced according to a linear function; in the far-distance range (greater than B), only a preset minimum ratio of points is retained. This method significantly reduces the number of distant points while ensuring near-field quality, thus improving overall rendering performance.

[0078] S6: Depth Bucket Sorting.

[0079] Divide the depth range of visible splats into There are 256 buckets, and sorting is performed only among the buckets:

[0080] ;

[0081] If the camera moves a distance If so, the rearrangement operation is skipped to improve real-time performance.

[0082] In this step, to avoid the high computational overhead of precisely sorting all points, this invention proposes a bucket-based depth counting sorting method. Specifically, the camera's view frustum is divided into several depth intervals (e.g., 256 buckets), and sorting is performed only between buckets, while points within a bucket do not need to be precisely arranged. This method balances rendering quality and computational efficiency, and avoids frequent repetitive sorting caused by slight camera movements by setting a threshold. When the camera moves rapidly beyond the threshold, the system discards unfinished sorting tasks and immediately starts a new background sorting, thereby improving real-time performance. Sorting is performed only for a single main camera, without sorting the left and right eyes separately, to reduce unnecessary redundant calculations.

[0083] S7: Rendering stage and screen space optimization;

[0084] Each draw call is executed on a bucket-by-bucket basis.

[0085] ;

[0086] During the rendering phase, this invention introduces the concept of Render Bucket Size, which means rendering a fixed number of Gaussian point instances in each draw call. By controlling the scale of each batch of draws, the performance bottleneck caused by excessive draw calls is effectively reduced, making it particularly suitable for VR devices such as Pico that adopt a Tile-Based Rendering architecture, thereby improving overall rendering efficiency.

[0087] (1) Near-to-far sorting and hybrid optimization;

[0088] Using a near-to-far ordering and a special mixed equation:

[0089] ;

[0090] When the cumulative alpha of the target pixel reaches 1, the Tile-Based GPU can terminate the fragment calculation early.

[0091] Traditional Gaussian splatter algorithms often employ painter's algorithms, sorting fragments from far to near and using a conventional transparent blending method (SrcAlpha, OneMinusSrcAlpha). However, this invention adopts a near-to-far sorting strategy, combined with a blending mode of "BlendOneMinusDstAlpha, One". This scheme leverages the tile-based rendering characteristics of VR devices, allowing the hardware to prematurely terminate subsequent fragment stacking during the fragment shading stage when the alpha of the target pixel has reached complete opacity, thus significantly reducing fragment computation. Experimental results show that, compared to traditional methods, this approach effectively improves the frame rate in the same scene.

[0092] (2) Edge Culling;

[0093] Within the Eye Buffer screen space, with the center of the screen as the focal point, two radial ranges are defined:

[0094] Inner radius (the area of ​​focus);

[0095] Outer radius (completely excluded area);

[0096] Normalized radius:

[0097] ;

[0098] Retain the proportional function:

[0099] ;

[0100] in Control the steepness of the transition curve for elimination;

[0101] For the reserved splats, perform a screen space magnification operation:

[0102] ;

[0103] in, This is the edge ambiguity coefficient, which controls the intensity of the ambiguity band; typical values ​​are... ;

[0104] Central area ( Maintain original size and transparency;

[0105] Only The scope is subject to progressive elimination and expansion.

[0106] Inspired by Fixed Foveated Rendering (FFR), this invention performs differentiated processing on Gaussian points in different regions of the screen space. Specifically, within the eye buffer of the VR device, the center of the screen is used as the focal point, dividing the space into a central region and an edge region: The central focal region: The rendering characteristics of the original Gaussian points are completely preserved without any culling or scaling to ensure the detail and clarity of the visual center. The edge region: A progressive culling function is set based on the radial distance between the pixel and the screen center. The closer the distance is to the edge, the more likely the projection of the corresponding Gaussian point in the screen space is to be culled, with the culling ratio gradually increasing with radial distance. For Gaussian points that are not culled, their projection scale is appropriately enlarged in the screen space to compensate for the gaps created by culling and to create a visually blurred transition effect. By employing this screen space culling mechanism that keeps the center constant while progressively eliminating edges, the computational burden of fragment filling and blending in the edge areas is significantly reduced, thereby improving overall rendering performance. On the other hand, the high quality and clarity of the visual center are maintained perceptually, ensuring the immersiveness and viewing comfort of VR scenes.

[0107] S8: App SpaceWarp;

[0108] To support spatial warp frame interpolation (AppSW) in VR applications, the center point of each Gaussian ellipse is extracted for motion vector calculation:

[0109] ;

[0110] The corresponding depth value is output. This calculation is performed only at 1 / 4 resolution of the Eye Buffer to reduce computational burden. Utilizing the high-density characteristics of the point cloud, a stable and smooth motion field can be obtained, improving the inter-frame prediction quality in transparent scenes.

[0111] This invention optimizes the App SpaceWarp (AppSW) algorithm used in VR devices to meet the specific needs of Gaussian splash rendering. Traditional AppSW suffers from insufficient accuracy and stability when handling transparent overlapping or interwoven objects. Since Gaussian splashing is essentially composed of numerous semi-transparent Gaussian ellipses, directly applying traditional AppSW leads to artifacts and distortion. To address these issues, this invention proposes using the center point of the Gaussian ellipse as the basis for calculating the motion vector. Specifically, the motion vector and corresponding depth information of each Gaussian center point are calculated based on camera motion parameters, and this result is then used for inter-frame prediction in AppSW. Because the Gaussian point cloud is densely distributed in the scene, the motion information of the center point can more completely reflect the dynamic characteristics of the overall scene, thus significantly improving the spatial warp effect in transparent scenes. Furthermore, this invention only requires rendering at one-quarter of the EyeBuffer resolution when calculating the motion vector to meet the accuracy requirements of AppSW.

[0112] S9: Output and Display;

[0113] By combining Edge Culling, Distance Culling, and AppSW results, the final Eye Buffer frame image is generated and output to the VR display, achieving a high frame rate, low latency, and clear stereoscopic visual effect.

[0114] This application replaces the traditional PLY format with the SOG high-compression format and combines Alpha Removal and Morton Code sorting to enable Gaussian point cloud data to be loaded quickly and accessed efficiently on bandwidth-constrained devices such as VR all-in-one machines, significantly shortening model initialization time and improving GPU cache hit rate.

[0115] By using a SplatTree (octree) for spatial partitioning, large-scale point pruning can be completed quickly before rendering. At the same time, combined with the hierarchical strategy of Distance Culling (foreground fully preserved, midground proportionally preserved, and background minimally preserved), the number of distant Gaussian points is effectively reduced, significantly reducing the rendering burden while ensuring the quality of the visual center.

[0116] The Depth Bucket sorting method proposed in this invention avoids the high overhead of precise point-by-point sorting. It only needs to sort between buckets to achieve a visual effect comparable to precise sorting. Furthermore, it reduces repeated sorting caused by slight camera shake through a threshold mechanism, thereby improving real-time performance and stability.

[0117] The rendering stage introduces a batch rendering strategy using Render Bucket and a near-to-far sorting strategy, combined with an improved transparency blending method (Blend OneMinusDstAlpha,One). This allows the Tile-Based rendering architecture to terminate calculations early when fragment alpha accumulates to complete opacity, significantly reducing redundant fragment operations and improving frame rate.

[0118] The Edge Culling method of this invention gradually removes Gaussian points near the edge of the field of view within the screen space, while progressively enlarging the retained points. This reduces the fragment filling pressure in the edge area and avoids fragmentation and voids through blurring transitions, thereby improving overall performance and maintaining an immersive visual experience at the cost of sacrificing edge details.

[0119] To address the shortcomings of AppSW in transparent overlay scenarios, this invention uses the center point of a Gaussian ellipse as the basis for MotionVector calculation and performs calculations only at one-quarter resolution of the Eye Buffer. By leveraging the high density characteristics of point clouds, it achieves stable and efficient inter-frame distortion prediction, which avoids the artifact problem caused by transparency and further reduces computational overhead.

[0120] Through the synergistic effect of the above-mentioned multi-level optimization methods (storage loading, space culling, rendering strategy, screen space optimization, and frame prediction adaptation), this invention significantly improves the frame rate and stability on VR devices while ensuring high quality at the visual center, enabling the 3D Gaussian splashing algorithm to run smoothly on resource-constrained mobile VR all-in-one devices.

[0121] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.

Claims

1. A method for optimizing 3D Gaussian splash rendering in VR based on the Unity engine, characterized in that, Includes the following steps: S1: Data preprocessing and format conversion; Convert the original PLY point cloud file into an SOG file. The SOG file stores the center coordinates, radius, orientation matrix, color, transparency, and LOD level data of the Gaussian points. S2: Alpha Removal and Reorganization; Parse the SOG file, set the Alpha threshold, remove Gaussian points with transparency below the Alpha threshold, and rewrite the filtered Gaussian points into a new SOG file. S3: Morton coding sorting; Calculate the Morton code for the center coordinates of the filtered Gaussian points, and sort the Gaussian points according to the coding results; S4: Construct an octree spatial index; using sorted Gaussian points as input, construct an octree spatial structure. Each node of the octree contains the bounding box of the three-dimensional spatial range it manages, the Gaussian point data of different LOD levels contained in the node, and the child node index pointing to its child nodes. S5: Distance-based elimination and LOD fusion; Based on the distance from the camera to the center of the octree node, determine the retention ratio of Gaussian points and select the corresponding LOD level, and output the set of visible Gaussian points; S6: Depth Bucket Sort; Divide the depth range of visible Gaussian points into multiple buckets, and perform sorting only between buckets. If the camera movement distance is less than the preset motion threshold, skip the sorting operation between buckets. S7: Rendering stage and screen space optimization; Draw calls are executed in units of buckets, using a sorting method from near to far and a preset blending equation, and edge culling and magnification operations are performed on visible Gaussian points in screen space based on radial distance; S8: Spatial warp adaptation; Based on the center coordinates and direction matrix of the visible Gaussian point, determine the Gaussian ellipse formed by its screen projection, extract the center point of each Gaussian ellipse to calculate the motion vector and corresponding depth value; S9: Output and Display; By combining the edge culling results of S7, the distance culling results of S5, and the spatial warp adaptation results of S8, the final frame image is generated and output to the VR display.

2. The 3D Gaussian splash rendering optimization method in VR based on the Unity engine according to claim 1, characterized in that, In S1, the original PLY point cloud file is converted into an SOG file through a toolchain, which is the open-source command-line tool SplatTransform. The multi-level LOD information includes three precision levels: high, medium, and low.

3. The 3D Gaussian splash rendering optimization method in VR based on the Unity engine according to claim 1, characterized in that, In S3, Morton encoding is calculated using the EncodeMorton3D function, which processes the x, y, and z components of the Gaussian point center coordinates using the Part1By2 function and then combines them to obtain the encoding result.

4. The 3D Gaussian splash rendering optimization method in VR based on the Unity engine according to claim 2, characterized in that, In S5, the specific rules for distance-based elimination are as follows: When the distance from the camera to the node center is less than the first threshold, the retention ratio is set to 1 and the highest precision LOD level is selected. When the distance is between the first threshold and the second threshold, the retention ratio decreases linearly with the distance and a medium-precision LOD level is selected. When the distance is greater than the second threshold, the retention ratio is set to the preset minimum value and the lowest precision LOD level is selected.

5. The 3D Gaussian splash rendering optimization method in VR based on the Unity engine according to claim 1, characterized in that, In S6, the depth interval is divided into 256 buckets, and the sorting operation is performed only for a single main camera.

6. The 3D Gaussian splash rendering optimization method in VR based on the Unity engine according to claim 1, characterized in that, In S7, the preset mixing equation is Blend(OneMinusDstAlpha,One).

7. The 3D Gaussian splash rendering optimization method in VR based on the Unity engine according to claim 1, characterized in that, In S7, when culling the screen space edge, an inner radius and an outer radius are defined. When the radial distance of the Gaussian point to the screen is less than or equal to the inner radius, the retention ratio is 1; when the radial distance is between the inner radius and the outer radius, the retention ratio gradually decreases; when the radial distance is greater than or equal to the outer radius, the retention ratio is 0; and a magnification operation based on the radial distance is performed on the retained Gaussian points.

8. The 3D Gaussian splash rendering optimization method in VR based on the Unity engine according to claim 1, characterized in that, In S8, the motion vector is the coordinate difference of the center point of the Gaussian ellipse at adjacent time points.

9. The 3D Gaussian splash rendering optimization method in VR based on the Unity engine according to claim 1, characterized in that, In S7, the drawing call is executed in batches with a fixed number of Gaussian points.

Images