A three-dimensional gaussian point cloud rendering method based on a graphics pipeline

Abstract

The application discloses a kind of three-dimensional Gaussian point cloud rendering methods based on graphics pipeline.The present application uses computing shader for point cloud preprocessing and attribute calculation, uses graphics pipeline for hardware rasterization and color mixing, makes the most of the characteristics of GPU, more general at the same time reaches extremely fast rendering speed.The present application is composed of stage:1.point cloud preprocessing stage, calculates point cloud attribute and NDC coordinate, while conducting view volume rejection.2.depth sorting.3.point cloud coloring.4.graphics pipeline stage, including point cloud Gaussian splash rasterization, and color transparency mixing.Compared with original CUDA rendering, the present application can run in any platform supporting graphics pipeline without running in the environment of Nvidia graphics card, has stronger universality, and uses graphics pipeline for point cloud Gaussian splash rasterization, graphics pipeline can naturally use hardware acceleration for rasterization and color mixing, so that extremely fast rendering speed is realized.

Description

Technical Field

[0001] This invention belongs to the field of 3D Gaussian splash rendering, specifically referring to rendering 3D Gaussian point clouds on non-CUDA platforms. Background Technology

[0002] In recent years, with the rise of 3D reconstruction technology represented by neural radiation field (NRF) 3D, the field of 3D reconstruction has also ushered in a period of rapid development. NRF, through multiple fully connected layers, infers information such as the color and density of a voxel from an input multi-view image and camera position. However, because NRF requires forward propagation through all neural network layers for each rendering iteration, real-time rendering is not possible.

[0003] Gaussian splashing has gradually replaced neural radiance fields due to its higher rendering quality and faster rendering speed. Gaussian splashing stores information through point clouds and uses rasterization rendering, thus achieving real-time rendering speed. The information in each point includes position, size, rotation angle, covariance, and spherical harmonic parameters. The covariance determines the rasterization range of the current point, and the spherical harmonic parameters are used to simulate the shading information of the current point at different angles.

[0004] Gaussian splashing can achieve real-time rendering, mainly due to its rasterization rendering method. Unlike the pixel-by-pixel ray stepping of neural radiation fields, which requires a complete forward propagation of the entire model for each step, rasterization only needs to perform world coordinate matrix transformation, camera coordinate matrix transformation, projection matrix transformation, and cubic matrix multiplication to calculate the pixel contribution of a certain point.

[0005] CUDA (Compute Unified Device Architecture) is a parallel computing platform and programming model introduced by NVIDIA. It allows developers to leverage the powerful computing capabilities of NVIDIA GPUs to accelerate computationally intensive tasks. The Gaussian splashing primitive rendering pipeline is implemented based on CUDA and mainly includes the following stages:

[0006] 1. Preprocessing: Obtain the 2D covariance, depth, and 2D AABB bounding box of each point.

[0007] 2. Accumulate all the colored pixels and generate a 64-bit Key value for each pixel. The first 32 bits of the Key value are the screen block position of the pixel, and the last 32 bits are the depth information of the pixel.

[0008] 3. Sort based on the Key value of each pixel.

[0009] 4. Perform color blending based on transparency on the sorted pixels.

[0010] Due to CUDA's powerful parallel capabilities, the original pipeline also utilizes tile-based parallel rendering to the maximum extent. The second step uses the screen block position as the first 32 bits, enabling the third step to perform depth sorting of pixels within the block. Then, the fourth step can directly retrieve all pixels of the current block via index. CUDA lacks automatic rasterization capabilities; therefore, the first step needs to generate an AABB bounding box for a specific point, simulating the pixel generated by rasterization at that point.

[0011] The graphics rendering pipeline is a core component of modern graphics hardware and software, primarily targeting geometric primitives composed of triangle vertices. It includes vertex processing, geometry processing, rasterization, and pixel processing stages. The graphics rendering pipeline is a general-purpose hardware acceleration solution for triangle meshes. Unlike CUDA, which only runs on NVIDIA graphics cards, the graphics rendering pipeline is a fundamental capability of the GPU and can run on virtually any platform.

[0012] The graphics rendering pipeline includes vertex shaders for batch processing of vertices, pixel shaders for batch processing of pixels, and compute shaders for handling general computations. Summary of the Invention

[0013] This invention proposes a 3D Gaussian point cloud rendering method based on the graphics pipeline. Based on hardware rasterization in the graphics pipeline, it can run on all platforms that support the graphics pipeline, has stronger versatility, and maintains extremely fast rendering speed.

[0014] The technical solution steps of this invention are as follows:

[0015] Step (1) Preprocessing stage: Use computational shaders to read all point clouds, calculate covariance, NDC coordinates and depth, and perform view frustum culling at the same time.

[0016] Step (2), sorting stage: use a computation shader to sort the depths of all Gaussian point clouds.

[0017] Step (3), the coloring stage, uses a computational shader to calculate the color of the current point cloud based on the covariance of the point cloud, the camera position, and the spherical harmonic parameters.

[0018] Step (4): Construct the graphics pipeline, set the parameters to allow color and transparency mixing, and use indirect drawing to directly read the attribute values ​​of the Gaussian point cloud from the video memory space.

[0019] Step (5), the vertex shader stage of the graphics pipeline, uses a rectangle (i.e., 6 points) composed of 2 identical triangles to simulate the range of its two-dimensional splash for each Gaussian point cloud.

[0020] Step (6), Pixel Shader Stage of Graphics Pipeline: For each pixel, the pixel is shaded according to the distance information between the pixel and the Gaussian cloud position.

[0021] Step (7), the graphics pipeline blending stage, blends colors and transparency from front to back according to the sorted depths. This step uses hardware acceleration.

[0022] The beneficial effects of this invention are:

[0023] Compared to existing technologies, this technology uses a graphics pipeline to render 3D Gaussian point clouds, without the limitations of CUDA and NVIDIA graphics card environments. It can run on any platform that supports a graphics pipeline, while achieving extremely high rendering speeds.

[0024] (1) A more universal platform

[0025] Since the graphics rendering pipeline is the most basic function of a GPU, this method runs on the graphics rendering pipeline. Compared to the original three-dimensional Gaussian point cloud, which uses CUDA rendering and can only run on NVIDIA graphics cards, this method can run on any platform that supports the graphics rendering pipeline, and can run on almost any GPU.

[0026] (2) Faster rendering speed

[0027] Since the original 3D Gaussian point cloud pipeline was built using CUDA, which has powerful general-purpose computing capabilities, it lacks graphics rendering capabilities. For example, Gaussian splatter rasterization and pixel transparency blending, which are required in the pipeline, are implemented by CUDA through pixel-by-pixel generation and assignment. This method uses a graphics rendering pipeline to implement pixel rasterization and pixel color blending in hardware, achieving extremely high rendering speed. Attached Figure Description

[0028] Figure 1 This is the specific process of the three-dimensional Gaussian point cloud graphics pipeline in this invention.

[0029] Figure 2 It represents the percentage of time spent at each stage in the entire process.

[0030] Figure 3 It is a performance comparison chart, including hardware rasterization and blending using the graphics pipeline, pixel-by-pixel rasterization and blending using compute shaders, and the raw CUDA pipeline. Detailed Implementation

[0031] The technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. The embodiments described are only some embodiments of the present invention and cannot encompass all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort are within the scope of protection of the present invention.

[0032] This invention discloses a graphics rendering pipeline method for 3D Gaussian point clouds. The method employs a graphics rendering pipeline for rendering 3D Gaussian point clouds, uses computational shaders for point cloud attribute calculation and preprocessing, and utilizes the graphics rendering pipeline for rasterization and color mixing. This maximizes GPU utilization and achieves extremely high rendering speed. Furthermore, because this method is based on a graphics rendering pipeline, it can run on almost all GPU platforms.

[0033] The experimental method of this invention uses the Vulkan graphical interface.

[0034] like Figure 1 The diagram illustrates the specific workflow of a 3D Gaussian point cloud graphics pipeline. It includes three computational pipelines for attribute preprocessing and one graphics pipeline for rasterization. The programmable stage comprises three computational shaders for attribute preprocessing, depth sorting, and point cloud shading, respectively; a vertex shader for simulating the Gaussian splash range of the point cloud; and a pixel shader for color shading the pixels. The specific steps are as follows:

[0035] Step 1: Preprocess the point cloud to generate the required attributes;

[0036] Since any Gaussian can be viewed as an affine transformation of a standard Gaussian, that is...

[0037] ω=Ax+b

[0038] Where ω represents the position of the Gaussian point cloud; A and b together represent the affine transformation parameters, where A represents the rotation and scaling matrix; and b represents the translation parameter.

[0039] The covariance of any Gaussian can be viewed as a linear transformation of the standard Gaussian, i.e.:

[0040] Σ′=AΣA T

[0041] Where Σ′ represents the Gaussian point cloud after linear transformation, Σ represents the standard Gaussian point cloud, and A represents the rotation and scaling matrix.

[0042] Since the covariance of a standard Gaussian is the identity matrix, and the linear transformation is controlled by the scaling matrix S and the rotation matrix R, the final covariance is:

[0043] ∑′=RSS T RT

[0044] Then calculate the normalized coordinates w of the point cloud. ndc (NDC coordinates);

[0045] w ndc =P·V·M·w model

[0046] Where P represents the projection matrix, V represents the view matrix, M represents the world coordinate transformation matrix, and w model Represents the coordinates in the model space coordinate system;

[0047] Finally, frustum culling is performed on NDC coordinates that are less than the set minimum threshold or greater than the set maximum threshold.

[0048] Step 2: Perform depth sorting on the point cloud

[0049] This method uses GPU-based radix sorting. The depth values ​​are floating-point numbers using the IEEE 754 standard, with single-precision 32 bits, including 1 sign bit, 8 exponent bits, and 23 mantissa bits. Since the depth of all point clouds is positive, the first sign bit is not included in the calculation. Therefore, the IEEE 754 representation format satisfies the condition that the larger the value of the higher-order bits, the larger the numerical value.

[0050] The sorting process consists of four rounds, starting from the least significant digit, with each round sorting only the current eight digits.

[0051] Each round consists of three phases:

[0052] (1) Upward scanning phase: For the 8 bits in this round, construct the counting information upward and accumulate the number of times each number appears in this position.

[0053] (2) Scanning phase: Calculate the prefix sum of the count information output in the upward scanning phase to obtain the starting position of the current 8-bit output.

[0054] (3) Downward scanning stage: Based on the position information determined in the previous stage, the elements in the original data are placed in the correct sorting position to complete the final sorting of the data in this round.

[0055] Step 3: Point Cloud Shading Stage

[0056] The color of each Gaussian sphere in the 3D point cloud is approximated using the spherical harmonic function f(t):

[0057]

[0058] in, These are the coefficients of the basis functions. is a basis function, and l represents the order of the spherical harmonic function;

[0059] This method uses a third-order spherical harmonic function. The coefficients are obtained by using the current point cloud and the camera angle, as well as the 3D covariance of the point cloud.

[0060] Steps 4, 5, and 6: Graphics Pipeline Stage

[0061] In Vulkan, if no vertex rendering order is specified, the vertex indices are read from front to back in the vertex buffer by default, and then per-vertex rendering is performed. Since color and opacity blending are used, the expected rendering order is to render the point cloud with shallow depth first, and then render the point cloud with deep depth. Therefore, the sorted point cloud indices are used as the vertex index input for the graphics pipeline.

[0062] In this stage, for each Gaussian point cloud, 6 points are generated, which are rectangles composed of 2 identical triangles. The position w of each point in the rectangle is:

[0063] w = w ndc +R·S·I i

[0064] Among them, w ndx Here are the ndc coordinates of the Gaussian point cloud, R is the rotation matrix of the point cloud, S is the scaling matrix of the point cloud, and I... i It is a fixed vector in screen space, where i corresponds to the four points of the rectangle (-1,-1), (-1,1), (1,-1), and (1,1).

[0065] The graphics rendering pipeline uses the screen space coordinates output by the vertex shader and the index of the vertex buffer to perform rasterization in hardware, completing the shading of the current pixel at the pixel shader stage. The specific color value is:

[0066] color = color point *α point *e (-0.5*pos*pos)

[0067] Among them, color point The color information of the Gaussian point cloud is obtained from step 3 using spherical harmonic functions, α point It is the transparency of the Gaussian point cloud, and pos is the distance between the current pixel position and the position of the Gaussian point cloud. pos will be automatically interpolated during the rasterization stage, which is done by hardware.

[0068] The final pixel color is obtained by mixing and superimposing multiple colors and transparency levels. The formulas for each color and transparency mixing are as follows:

[0069] color final =color src +color dst*(1-α src )

[0070] Among them, color src The color information of the current pixel. dst This represents the color information to be superimposed, α. src Indicates the current pixel transparency.

[0071] like Figure 2 The figure shows the time consumption percentage of the 3D Gaussian graphics pipeline in this method. The experiments used point clouds of 160,000 points and 6.13 million points respectively, with the final graphics pipeline accounting for about 65% in both cases.

[0072] like Figure 3 The figure shows a performance comparison between this method, the original CUDA pipeline, and the ablation Gaussian graphics pipeline using only computational shaders for rasterization. The figure clearly shows that the present invention has a significant advantage in terms of speed. Therefore, the graphics pipeline for Gaussian point clouds proposed in this invention, compared to the original CUDA rendering, can run on any platform that supports the graphics pipeline without requiring an NVIDIA graphics card, offering greater versatility. Furthermore, by using the graphics pipeline for Gaussian splash rasterization of point clouds, the graphics pipeline can naturally utilize hardware acceleration for rasterization and color mixing, thereby achieving extremely fast rendering speeds.

[0073] Although embodiments of the present invention have been shown, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A method for rendering 3D Gaussian point clouds based on a graphics pipeline, characterized in that, Includes the following steps: Step (1), Preprocessing stage: Use computation shader to read all Gaussian point clouds, calculate covariance, NDC coordinates and depth, and perform view frustum culling at the same time. Step (2), sorting stage: use a computation shader to sort the depths of all Gaussian point clouds; Step (3), the coloring stage, uses a computational shader to calculate the color of the current Gaussian point cloud based on the covariance of the Gaussian point cloud, the camera position, and the spherical harmonic parameters; Step (4): Construct the graphics pipeline, set the parameters to allow color and transparency mixing, and use indirect drawing to directly read the attribute values ​​of the Gaussian point cloud from the video memory space. Step (5), the vertex shader stage of the graphics pipeline, uses a rectangle composed of 2 identical triangles, i.e., 6 points, to simulate the range of its two-dimensional splash for each Gaussian point cloud. Step (6), Pixel shader stage of graphics pipeline: For each pixel, the pixel is shaded according to the distance information between the pixel and the Gaussian point cloud position; The specific implementation is as follows: The graphics rendering pipeline uses the screen space coordinates output by the vertex shader and the index of the vertex buffer to perform rasterization in hardware, completing the shading of the current pixel at the pixel shader stage. The specific color value is: in The color information of the Gaussian point cloud is obtained from step 3 using spherical harmonic functions. It is the transparency of the Gaussian point cloud, and pos is the distance between the current pixel position and the position of the Gaussian point cloud. pos will be automatically interpolated during the rasterization stage, which is done by hardware. Step (7), Graphics Pipeline Blending Stage: Based on the sorted depths, color and transparency are blended from front to back. This step uses hardware acceleration. The specific implementation is as follows: The final pixel color is obtained by mixing and superimposing multiple colors and transparency levels. The formulas for each color and transparency mixing are as follows: in, This indicates the color information of the current pixel. This indicates the color information to be overlaid. Indicates the current pixel transparency.

2. The 3D Gaussian point cloud rendering method based on graphics pipeline according to claim 1, characterized in that, The pre-calculation and frustum culling of the Gaussian point cloud described in step (1) are specifically implemented as follows: Since any Gaussian can be viewed as an affine transformation of a standard Gaussian, that is: in, A represents the position of the Gaussian point cloud; A and b together represent the affine transformation parameters, where A represents the rotation and scaling matrix and b represents the translation parameter. The covariance of any Gaussian can be viewed as a linear transformation of a standard Gaussian, that is: in, This represents a Gaussian point cloud after linear transformation. Represents a standard Gaussian point cloud; A represents the rotation and scaling matrix; Since the covariance of a standard Gaussian is the identity matrix, and the linear transformation is controlled by the scaling matrix S and the rotation matrix R, the final covariance is: Then calculate the normalized coordinates of the Gaussian point cloud. : Where P represents the projection matrix, V represents the view matrix, and M represents the world coordinate transformation matrix. Represents the coordinates in the model space coordinate system; Finally, for the normalized coordinates Frustum rejection is performed if the threshold is less than the set minimum threshold or greater than the set maximum threshold.

3. The 3D Gaussian point cloud rendering method based on graphics pipeline according to claim 1, characterized in that, The depth sorting of the Gaussian point cloud described in step (2) is specifically implemented as follows: The sorting process consists of four rounds, starting from the least significant digit. Each round sorts only the current eight digits, and each round comprises three stages. composition: (1) Upward scanning phase: For the 8 bits in this round, construct the counting information upward, that is, accumulate the number of times each number appears in this position; (2) Scanning phase: Calculate the prefix sum of the count information output in the upward scanning phase to obtain the starting position of the current 8-bit output; (3) Downward scanning phase: Based on the output starting position determined in the scanning phase, the elements in the original data are placed in the correct sorting position to complete the final sorting of the data in this round.

4. The 3D Gaussian point cloud rendering method based on the graphics pipeline according to claim 1 or 3, characterized in that, Step (4) is implemented as follows: If no vertex rendering order is specified, the vertex indices are read from front to back in the vertex buffer by default, and then vertex-by-vertex rendering is performed. Since color and transparency blending are used, the rendering order is to render Gaussian point clouds with a depth less than the set threshold first, and then render Gaussian point clouds with a depth greater than or equal to the set threshold. Therefore, the sorted Gaussian point cloud indices are used as the vertex index inputs of the graphics pipeline.

5. The 3D Gaussian point cloud rendering method based on the graphics pipeline according to claim 4, characterized in that, Step (5) is implemented as follows: For each Gaussian point cloud, 6 points are generated, which are rectangles composed of 2 triangles. The position of each point in the rectangle is as follows: in Here are the ndc coordinates of the Gaussian point cloud, R is the rotation matrix of the Gaussian point cloud, and S is the scaling matrix of the Gaussian point cloud. It is a fixed vector in screen space, and for the four points of the rectangle, they are (-1,-1), (-1,1), (1,-1), and (1,1).

Images