Three-dimensional Gaussian point cloud rendering method based on graph pipeline

Abstract

The invention discloses a three-dimensional Gaussian point cloud rendering method based on a graphic pipeline. According to the method, the calculation shader is used for point cloud preprocessing and attribute calculation, the graphic pipeline is used for hardware rasterization and color mixing, the characteristics of the GPU are utilized to the maximum extent, and the extremely high rendering speed is achieved while the universality is higher. The method comprises the following stages: 1, a point cloud preprocessing stage: calculating point cloud attributes and NDC coordinates, and removing view cones at the same time; 2, performing depth sorting; and 3, coloring the point cloud. And 4, a graph pipeline stage, including point cloud Gaussian splashing rasterization and color transparency mixing. Compared with original CUDA rendering, the method can be operated on any platform supporting the graphic pipeline in the environment of a NVIDIA display card which does not need to be operated, higher universality is achieved, meanwhile, rasterization of point cloud Gaussian splashing is conducted through the graphic pipeline, the rasterization and color mixing can be conducted through the graphic pipeline, hardware acceleration can be naturally used, and the method has the advantages of being high in practicability and easy to popularize. Therefore, the extremely high rendering speed is realized.

Description

Technical Field

[0001] The present invention belongs to the field of three-dimensional Gaussian splash rendering, and in particular refers to rendering three-dimensional Gaussian point cloud on a non-CUDA platform. Background Art

[0002] In recent years, with the rise of 3D reconstruction technology represented by neural radiance fields, the field of 3D reconstruction has also reached a climax. Neural radiance fields use multiple fully connected layers to infer the color, density and other information of a voxel based on the input multi-view images and camera positions. However, since neural radiance fields need to propagate forward through all neural network layers each time they are rendered, real-time rendering is not possible.

[0003] Gaussian splatting has gradually replaced neural radiation fields due to its higher rendering quality and faster rendering speed. Gaussian splatting stores information through point clouds and uses rasterization rendering to achieve real-time rendering speed. The information in each point includes position, size, rotation angle, covariance and spherical harmonic parameters, where covariance determines the rasterization range of the current point, and spherical harmonic parameters are used to simulate the shading information of the current point at different angles.

[0004] Gaussian splatting can achieve real-time rendering, mainly due to its rasterized rendering method. Unlike the pixel-by-pixel ray stepping of the neural radiation field, each step requires a complete forward propagation of the entire model. Rasterization only requires world coordinate matrix transformation, camera coordinate matrix transformation, projection matrix transformation, and three matrix multiplications to calculate the pixel contribution of a certain point.

[0005] CUDA (Compute Unified Device Architecture) is a parallel computing platform and programming model launched by NVIDIA. It allows developers to use the powerful computing power of NVIDIA GPU to accelerate computing-intensive tasks. The original rendering pipeline of Gaussian splash is implemented based on CUDA, which mainly includes the following stages:

[0006] 1. Preprocessing: Get the 2D covariance, depth, and 2D AABB bounding box of each point.

[0007] 2. Accumulate all the shaded pixels and generate a 64-bit key value for each pixel. The first 32 bits of the value are the screen block position of the pixel, and the last 32 bits are the depth information of the pixel.

[0008] 3. Sort based on the Key value of each pixel.

[0009] 4. Perform transparency-based color blending on the sorted pixels.

[0010] Due to the powerful parallel capability of CUDA, the original pipeline also uses block parallel rendering to the greatest extent. In the second step, the screen block position is used as the first 32 bits, so that the depth sorting of pixels in the block can be realized in the third step, and then all pixels of the current block can be directly obtained through the index in the fourth step. CUDA does not have the ability to automatically rasterize, so in the first step, it is necessary to generate an AABB bounding box of a certain point, that is, to simulate the pixel points generated by rasterization of the point.

[0011] The graphics rendering pipeline is the core part of modern graphics hardware and graphics software. It is mainly aimed at geometric primitives composed of triangle vertices, including vertex processing stage, geometry processing stage, rasterization stage and pixel processing stage. The graphics rendering pipeline is a universal solution for hardware acceleration of triangle meshes. Unlike CUDA, which can only run on NVIDIA graphics cards, the graphics rendering pipeline is the basic capability of the GPU and can run on almost all platforms.

[0012] The graphics rendering pipeline contains vertex shaders for batch processing of vertices, pixel shaders for batch processing of pixels, and compute shaders for general calculations. Summary of the invention

[0013] The present invention proposes a three-dimensional Gaussian point cloud rendering method based on a graphics pipeline. Based on the hardware rasterization in the graphics pipeline, the method can run on all platforms that support the graphics pipeline, has stronger versatility, and maintains an extremely fast rendering speed.

[0014] The technical solution steps of the present invention are as follows:

[0015] Step (1), preprocessing stage, use the compute shader to read all point clouds, calculate covariance, NDC coordinates and depth, and perform frustum culling.

[0016] Step (2), the sorting stage, uses a compute shader to sort the depths of all Gaussian point clouds.

[0017] Step (3), the coloring stage, uses a compute shader to calculate the color of the current point cloud based on the covariance of the point cloud, the camera position, and the spherical harmonic parameters.

[0018] Step (4), construct a graphics pipeline, set parameters to allow color and transparency mixing, and use indirect rendering to directly read the attribute values ​​of the Gaussian point cloud from the video memory space.

[0019] Step (5), in the vertex shader stage of the graphics pipeline, for each Gaussian point cloud, a rectangle consisting of two identical triangles (i.e., six points) is used to simulate the range of its two-dimensional splashing.

[0020] Step (6), in the pixel shader stage of the graphics pipeline, for each pixel, the pixel is shaded according to the distance information between the pixel and the point Gaussian cloud position.

[0021] Step (7), the graphics pipeline mixing stage, performs color and transparency mixing from front to back according to the sorted depth. This step uses hardware acceleration.

[0022] The beneficial effects of the present invention are:

[0023] Compared with existing technologies, this technology uses the graphics pipeline to render three-dimensional Gaussian point clouds. It is not limited by CUDA and NVIDIA graphics card environments and can run on any platform that supports the graphics pipeline while achieving extremely high rendering speeds.

[0024] (1) A more universal platform

[0025] Since the graphics rendering pipeline is the most basic function of the GPU, this method is based on the graphics rendering pipeline. Compared with the original three-dimensional Gaussian point cloud, which is rendered using CUDA and can only run on NVIDIA graphics cards, this method can run on any platform that supports the graphics rendering pipeline and can run on almost any GPU.

[0026] (2) Faster rendering speed

[0027] Since the original three-dimensional Gaussian point cloud pipeline is built using CUDA, CUDA has powerful general computing capabilities, but CUDA does not have graphics rendering capabilities. For example, the Gaussian splash rasterization and pixel transparency blending required in the pipeline are all achieved by CUDA through pixel-by-pixel generation and pixel-by-pixel assignment. This method uses the graphics rendering pipeline to achieve pixel rasterization and pixel color blending through hardware, achieving extremely high rendering speed. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Figure 1 This is the specific process of the three-dimensional Gaussian point cloud graphics pipeline in the present invention.

[0029] Figure 2 It is the proportion of time consumed by each stage in the whole process.

[0030] Figure 3 This is a performance comparison chart, including using the graphics pipeline hardware rasterization and blending, using compute shaders for per-pixel rasterization and blending, and the original CUDA pipeline. DETAILED DESCRIPTION

[0031] The technical solution of the present invention will be clearly and completely described below with reference to the accompanying drawings in the implementation examples of the present invention. The embodiments described are only part of the embodiments of the present invention and cannot cover all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0032] The present invention discloses a graphics rendering pipeline method for three-dimensional Gaussian point clouds. The graphics rendering pipeline is used to render three-dimensional Gaussian point clouds, a computing shader is used to calculate and preprocess the attributes of the point clouds, and the graphics rendering pipeline is used to perform rasterization and color mixing, so that the GPU is used to the greatest extent, and an extremely high rendering speed is achieved. At the same time, since the method is based on the graphics rendering pipeline, it can be run on almost all GPU platforms.

[0033] The experimental method of the present invention uses the Vulkan graphics interface.

[0034] like Figure 1 The figure shows the specific process of the 3D Gaussian point cloud graphics pipeline, which includes 3 computing pipelines for attribute preprocessing and 1 graphics pipeline for rasterization. The programmable stage includes 3 computing shaders for attribute preprocessing, depth sorting, and point cloud shading, a vertex shader for Gaussian splash range simulation of the point cloud, and a pixel shader for pixel coloring. The specific steps are as follows:

[0035] Step 1: Preprocess the point cloud to generate the required attributes;

[0036] Since any Gaussian can be regarded as an affine variation of the standard Gaussian, that is,

[0037] ω=Ax+b

[0038] Among them, ω represents the position of the Gaussian point cloud; A and b together represent the affine transformation parameters, A represents the rotation and scaling matrix; b represents the translation parameter.

[0039] The covariance of any Gaussian can be regarded as a linear transformation of the standard Gaussian, that is:

[0040] Σ′=AΣA T

[0041] Among them, Σ′ represents the Gaussian point cloud after linear transformation, Σ represents the standard Gaussian point cloud; A represents the rotation and scaling matrix.

[0042] Since the covariance of the standard Gaussian is the unit matrix, and the linear transformation is controlled by the scaling matrix S and the rotation matrix R, the final covariance is:

[0043] ∑′=RSS T RT

[0044] Then calculate the normalized coordinates w of the point cloud ndc (NDC coordinates);

[0045] w ndc =P·V·M·w model

[0046] Among them, P represents the projection matrix, V represents the view matrix, M represents the world coordinate transformation matrix, and w model Represents the coordinates in the model space coordinate system;

[0047] Finally, the frustum is culled if the NDC coordinates are less than the set minimum threshold or greater than the set maximum threshold.

[0048] Step 2: Depth sorting of point cloud

[0049] This method uses GPU-based radix sorting. The depth value is a floating point number using the IEEE754 standard, single-precision 32 bits, including 1 sign bit, 8 exponent bits, and 23 mantissa bits. Since the depth of all point clouds is a positive number, the first sign bit is not included in the calculation. Therefore, the IEEE754 representation format satisfies that the larger the high-order value, the larger the value.

[0050] There are 4 rounds of sorting, starting from the low bit, and each round only sorts the current 8 bits of information.

[0051] Each round consists of three phases:

[0052] (1) Upward scanning phase: For the 8 bits of this round, the counting information is constructed upward and the number of times each digit appears in this bit is accumulated.

[0053] (2) Scanning phase: Calculate the prefix sum of the counting information output in the upward scanning phase to obtain the output starting position of the current 8 bits.

[0054] (3) Downward scanning phase: According to the position information determined in the previous phase, the elements in the original data are placed in the correct sorting position to complete the final sorting of this round of data.

[0055] Step 3: Point cloud coloring stage

[0056] For each 3D point cloud Gaussian sphere, the color is approximated using the spherical harmonic function f(t):

[0057]

[0058] in, are the coefficients of the basis functions, is the basis function, l represents the order of the spherical harmonic function;

[0059] This method uses 3rd order spherical harmonics. Each coefficient is obtained through the angle between the current point cloud and the camera and the three-dimensional covariance of the point cloud.

[0060] Steps 4, 5, 6: Graphics pipeline stages

[0061] In Vulkan, if the rendering order of vertices is not specified, the vertex index is read from the front to the back in the vertex buffer by default, and then rendered vertex by vertex. Due to the use of color and transparency blending, the expected rendering order is to render the point cloud with small depth first, and then the point cloud with large depth. Therefore, the sorted point cloud index is used as the vertex index input of the graphics pipeline.

[0062] At this stage, 6 points are generated for each Gaussian point cloud, that is, a rectangle composed of 2 identical triangles, and the position w of each point in the rectangle is:

[0063] w=w ndc +R.S.I i

[0064] Among them, w ndx is the ndc coordinate of the Gaussian point cloud, R is the rotation matrix of the point cloud, S is the scaling matrix of the point cloud, and I i is a fixed vector in screen space, i corresponds to the four points of the rectangle: (-1,-1), (-1,1), (1,-1), (1,1).

[0065] The graphics rendering pipeline will complete the rasterization by hardware based on the screen space coordinates output by the vertex shader and the index of the vertex buffer, and complete the shading of the current pixel in the pixel shader stage. The specific color value color is:

[0066] color=color point *α point *e (-0.5*pos*pos)

[0067] Among them, color point is the color information of the Gaussian point cloud, obtained by spherical harmonic function in step 3, α point is the transparency of the Gaussian point cloud, pos is the distance between the current pixel position and the Gaussian point cloud position, pos will be automatically interpolated in the rasterization stage, which is completed by hardware.

[0068] The final pixel color is obtained by mixing and superimposing multiple colors and transparency. The formula for each color and transparency mixing is as follows:

[0069] color final =color src +color dst*(1-α src )

[0070] Among them, color src Indicates the color information of the current pixel, color dst Indicates the color information to be superimposed, α src Indicates the current pixel transparency.

[0071] like Figure 2 As shown in FIG. 1 , it is the time consumption proportion of the 3D Gaussian graphics pipeline in this method. The experiments used point clouds of 160,000 points and 6.13 million points respectively, of which the final graphics pipeline accounted for about 65%.

[0072] like Figure 3 The figure shows a performance comparison of this method with the original CUDA pipeline and the method of rasterizing only with compute shaders after the Gaussian graphics pipeline is ablated. It can be clearly seen from the figure that the present invention has a very obvious advantage in terms of timeliness. Therefore, compared with the original CUDA rendering, the graphics pipeline for Gaussian point clouds proposed by the present invention does not require the running of an NVIDIA graphics card environment, and can run on any platform that supports the graphics pipeline. It has stronger versatility, and at the same time uses the graphics pipeline for rasterization of point cloud Gaussian splashing. The graphics pipeline can naturally use hardware acceleration for rasterization and color mixing, thereby achieving extremely fast rendering speeds.

[0073] Although embodiments of the present invention have been shown, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A three-dimensional Gaussian point cloud rendering method based on a graphics pipeline, characterized in that: The steps include: Step (1), preprocessing stage, use the compute shader to read all point clouds, calculate covariance, NDC coordinates and depth, and perform frustum culling; Step (2), sorting stage, using compute shader to sort the depths of all Gaussian point clouds; Step (3), coloring stage, using a compute shader, calculates the color of the current point cloud based on the covariance of the point cloud, the camera position, and the spherical harmonic parameters; Step (4), constructing a graphics pipeline, setting parameters to allow color and transparency mixing, and using indirect rendering to directly read the attribute values ​​of the Gaussian point cloud from the video memory space; Step (5), in the vertex shader stage of the graphics pipeline, for each Gaussian point cloud, a rectangle consisting of two identical triangles, i.e., six points, is used to simulate the range of its two-dimensional splashing; Step (6), in the pixel shader stage of the graphics pipeline, for each pixel, the pixel is shaded according to the distance information between the pixel and the point Gaussian cloud position; Step (7), the graphics pipeline mixing stage, performs color and transparency mixing from front to back according to the sorted depth. This step uses hardware acceleration.

2. The three-dimensional Gaussian point cloud rendering method based on the graphics pipeline according to claim 1, characterized in that: The pre-calculation of the point cloud and the frustum culling described in step (1) are specifically implemented as follows: Since any Gaussian can be regarded as an affine variation of the standard Gaussian, that is: ω=Ax+b Among them, ω represents the position of the Gaussian point cloud; A and b together represent the affine change parameters, A represents the rotation and scaling matrix; b represents the translation parameter; The covariance of any Gaussian can be viewed as a linear transformation of the standard Gaussian, namely: ∑′=A∑A T Among them, ∑′ represents the Gaussian point cloud after linear transformation, ∑ represents the standard Gaussian point cloud; A represents the rotation and scaling matrix; Since the covariance of the standard Gaussian is the unit matrix, and the linear transformation is controlled by the scaling matrix S and the rotation matrix R, the final covariance is: ∑′=RSS T R T Then calculate the normalized coordinates w of the point cloud ndc : w ndc =P·V·M·w model Among them, P represents the projection matrix, V represents the view matrix, M represents the world coordinate transformation matrix, and w model Represents the coordinates in the model space coordinate system; Finally, for the normalized coordinate w ndc If the value is less than the minimum threshold or greater than the maximum threshold, the frustum will be culled.

3. The three-dimensional Gaussian point cloud rendering method based on the graphics pipeline according to claim 1, characterized in that: The depth sorting of the point cloud described in step (2) is specifically implemented as follows: There are 4 rounds of sorting, starting from the lowest bit, and each round only sorts the current 8 bits of information, and each round consists of three stages: (1) Upward scanning stage: For the 8 bits of this round, count information is constructed upward, that is, the number of times each digit appears in this bit is accumulated; (2) Scanning stage: Calculate the prefix sum of the count information output in the upward scanning stage to obtain the output starting position of the current 8 bits; (3) Downward scanning phase: According to the output starting position determined in the scanning phase, the elements in the original data are placed in the correct sorting position to complete the final sorting of this round of data.

4. The three-dimensional Gaussian point cloud rendering method based on a graphics pipeline according to claim 1 or 3, characterized in that: Step (4) is specifically implemented as follows: If the rendering order of vertices is not specified, the vertex index is read from the front to the back in the vertex buffer by default, and then rendered vertex by vertex. Due to the use of color and transparency blending, the rendering order is to render the point cloud with a depth less than the set threshold first, and then render the point cloud with a depth greater than or equal to the set threshold. Therefore, the sorted point cloud index is used as the vertex index input of the graphics pipeline.

5. The three-dimensional Gaussian point cloud rendering method based on the graphics pipeline according to claim 4, characterized in that: Step (5) is specifically implemented as follows: For each Gaussian point cloud, 6 points will be generated, that is, a rectangle composed of 2 triangles. For the position of each point in the rectangle: w=w ndc +R·S·I i where w ndx is the ndc coordinate of the Gaussian point cloud, R is the rotation matrix of the point cloud, S is the scaling matrix of the point cloud, and I i is a fixed vector in screen space, which are (-1, -1), (-1, 1), (1, -1), and (1, 1) for the four points of the rectangle.

6. The three-dimensional Gaussian point cloud rendering method based on the graphics pipeline according to claim 5, characterized in that: Step (6) is specifically implemented as follows: The graphics rendering pipeline will complete the rasterization by hardware based on the screen space coordinates output by the vertex shader and the index of the vertex buffer, and complete the shading of the current pixel in the pixel shader stage. The specific color value is: color=color point *α point *and (-0.5*pos*pos) where color point is the color information of the Gaussian point cloud, obtained by spherical harmonic function in step 3, α point is the transparency of the Gaussian point cloud, pos is the distance between the current pixel position and the Gaussian point cloud position, pos will be automatically interpolated in the rasterization stage, which is completed by hardware.

7. The three-dimensional Gaussian point cloud rendering method based on a graphics pipeline according to claim 6, characterized in that: Step (7) is specifically implemented as follows: The final pixel color is obtained by mixing and superimposing multiple colors and transparency. The formula for each color and transparency mixing is as follows: color final =color src +color dst *(1-α src ) Among them, color src Indicates the color information of the current pixel, color dst Indicates the color information to be superimposed, α src Indicates the current pixel transparency.

Images