A 3D Gaussian rendering method based on depth denoising

Abstract

The present invention discloses a three-dimensional Gaussian rendering method based on deep denoising. First, the scene point cloud and camera pose are obtained based on a picture sequence or video, and then a neural radiation map is constructed based on the obtained point cloud; the obtained neural radiation map is subjected to deep denoising; a trained scene expression is obtained through end-to-end training; and finally, three-dimensional Gaussian rendering is realized through the trained scene expression. The present invention focuses on solving the noise problem introduced by insufficient sampling during the rasterization process. By utilizing deep denoising technology, the present invention can effectively remove "artifacts" in the rendering results, can greatly improve the clarity and realism of the rendering results, and provides a reliable technical support for high-quality graphics generation and display.

Description

Technical Field

[0001] The present invention belongs to the field of point cloud rendering, and in particular relates to a three-dimensional Gaussian rendering method based on depth denoising. Background Art

[0002] Point cloud is a collection of three-dimensional points, which is often used to represent the geometric structure of an object. However, due to the lack of volume of the point cloud itself, existing methods usually depict each point as a sphere, and render it by projecting it onto the pixel plane to finally generate an image. As an emerging three-dimensional expression method, three-dimensional Gaussian uses anisotropic spheres to represent 3D scenes, becoming a new paradigm for point cloud rendering. However, due to its anisotropic expression, the storage increases dramatically. In addition, during the rasterization process, due to the occupation of the sphere volume, noise, such as holes and jagged effects, will inevitably be introduced; existing denoising methods can alleviate the artifact problem caused by noise to a certain extent, but they cannot effectively remove artifacts in complex scenes and weak texture areas. And it will increase the storage size. To this end, the present invention proposes a new method that specifically handles the noise generated during the rasterization process while reducing storage. The method of the present invention greatly improves the rendering quality of point clouds based on three-dimensional Gaussian. Summary of the invention

[0003] The present invention proposes a three-dimensional Gaussian rendering method based on depth denoising, which improves the quality of three-dimensional Gaussian rendering and reduces its storage space.

[0004] The present invention proposes a three-dimensional Gaussian rendering method based on depth denoising, comprising the following steps:

[0005] Step 1: Obtain scene point cloud and camera pose based on image sequence or video.

[0006] Collect the video or picture sequence of the scene to be rendered, preprocess it, and use the structure from motion (SfM) algorithm to extract the point cloud data of the scene and the pose information of the camera corresponding to the image.

[0007] Step 2: Construct a neural radiation map based on the acquired point cloud.

[0008] Step 3: Perform deep denoising on the obtained neural radiation map.

[0009] Step 4: Obtain the trained scene expression through end-to-end training.

[0010] Step 5: Implement 3D Gaussian rendering through the scene expression obtained through training;

[0011] In order to obtain the rendering result of the desired perspective, the camera pose of the desired perspective is input into the trained scene expression to achieve three-dimensional Gaussian rendering.

[0012] Furthermore, the specific method of step 2 is as follows:

[0013] Reparameterize the 3D Gaussian into , , , , These five variables are represented as quaternions, splash length, spatial position, opacity and appearance model. The spatial position of the 3D Gaussian is initialized by the acquired point cloud ; other attributes are randomly initialized. For each view, the 3D Gaussian is splashed onto the 2D pixel plane by the camera pose. After that, each pixel to be rendered is occupied by multiple Gaussians. The neural radiation map of the corresponding pixel is obtained using the following formula :

[0014] (1)

[0015] Where N represents the number of Gaussians at the pixel location.

[0016] Introduction to enhance the neural radiance map, which takes into account the position of each pixel The direction of light To correct the neural radiation map, thus alleviating the impact of noise in the process of three-dimensional Gaussian projection into two-dimensional space. It consists of a multi-layer perceptron.

[0017] (2)

[0018] Furthermore, the specific method of step 3 is as follows:

[0019] Use a lightweight network The neural radiation map is processed by three standard convolutional layers. The dimension is , not . The input is , the output is a After obtaining the mask, multiply it to get Denoised neural radiance map , and finally use U-Net to Decoded into The U-Net is composed of five convolution blocks. The number of channels output by the convolution blocks of each layer is 64, 128, 256, 512, and 1024 respectively. The five convolution blocks are composed of gated convolution and 2x2 average pooling.

[0020] Furthermore, the specific method of step 4 is as follows:

[0021] For the images collected from the scene to be rendered, the properties of the three-dimensional Gaussian , , , , , Multilayer Perceptron , lightweight network The U-Net is jointly optimized during the end-to-end training process. During the training process, the three-dimensional Gaussian is splashed onto the two-dimensional pixel plane through the camera pose, and the neural radiation map is obtained using formula (1). After that, the rendered image is decoded by U-Net. The loss function between the rendered image and the true value is calculated.

[0022] The loss function is:

[0023] (3)

[0024] in, and They represent L1 loss function and SSIM loss function respectively.

[0025] After the training is completed, the trained scene expression is obtained, and the trained scene expression includes optimized three-dimensional Gaussian, multi-layer perceptron , lightweight network And U-Net.

[0026] The characteristics and beneficial effects of the present invention are as follows:

[0027] The present invention focuses on solving the noise problem introduced by insufficient sampling during the rasterization process. By using deep denoising technology, the present invention can effectively remove "artifacts" in the rendering results. These artifacts usually appear as visual distortion or unnatural texture features, which seriously affect the quality of the final image. With this method, the clarity and realism of the rendering results can be greatly improved, providing a reliable technical support for high-quality graphics generation and display. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Figure 1 This is a structural diagram of the denoising network used in the present invention.

[0029] Figure 2 This is a visualization result of an embodiment of the present invention. DETAILED DESCRIPTION

[0030] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0031] For the sake of uniformity, the symbols to be used in the present invention are explained in advance:

[0032] Rotation Matrix ,and =1.

[0033] Scaling Matrix , and the corresponding three-dimensional vector .

[0034] Mean of a 3D Gaussian , which is the spatial position of the corresponding point cloud.

[0035] Quaternions .

[0036] Opacity .

[0037] Multilayer Perceptron ,in Represents its trainable parameters.

[0038] Ray tracing direction .

[0039] 3D Gaussian appearance model .

[0040] Covariance matrix .

[0041] The parameterization of the three-dimensional Gaussian is as follows:

[0042] (1)

[0043] The covariance matrix , Represents the rotation matrix, which can be represented by the unit quaternion get. Represents the scaling matrix, the corresponding three-dimensional vector . Represents the mean of the three-dimensional Gaussian, which corresponds to the spatial position of the point cloud.

[0044] The three-dimensional Gaussian for rendering was proposed in "3D Gaussian Splatting for Real-Time Radiance Field Rendering", referred to as 3DGS, in which each three-dimensional Gaussian contains , , , , To represent quaternion, splash length, spatial position, opacity and color. The color is represented by the coefficient of the third-order spherical harmonic function. The final rendered image is obtained by the following formula:

[0045] Image= (2)

[0046] Where i represents the index of the queried three-dimensional Gaussian.

[0047] The present invention proposes a three-dimensional Gaussian rendering method based on depth denoising, comprising the following steps:

[0048] Step 1: Obtain scene point cloud and camera pose based on image sequence or video.

[0049] Collect the video or picture sequences of the scene to be rendered, preprocess them, and use the Structure from Motion (SfM) algorithm to extract the point cloud data of the scene and the pose information of the camera corresponding to the image, providing the necessary basic data support for the subsequent rendering process.

[0050] Step 2: Construct a neural radiation map based on the acquired point cloud.

[0051] Different from the original 3D Gaussian, we observe that the colors of the same area observed from different viewing angles are usually inconsistent. To this end, we reparameterize the 3D Gaussian as , , , , These five variables are represented as quaternions, splash length, spatial position, opacity and appearance model. The spatial position of the 3D Gaussian is initialized by the acquired point cloud ; other attributes are randomly initialized. For each training view, we splash the 3D Gaussian onto the 2D pixel plane using the camera pose. After that, each pixel to be rendered is occupied by multiple Gaussians. The following formula is used to get the neural radiation map of the corresponding pixel :

[0052] (3)

[0053] Where N represents the number of Gaussians at the pixel location.

[0054] Simply convert the original three-dimensional Gaussian color Replace with The approximate process of three-dimensional Gaussian projection to two-dimensional space is not considered, so we introduce to enhance the neural radiance map, which takes into account the position of each pixel The direction of light To correct the neural radiation map, thus alleviating the impact of noise in the process of three-dimensional Gaussian projection into two-dimensional space. It consists of a simple multi-layer perceptron.

[0055] (4)

[0056] Step 3: Perform deep denoising on the obtained neural radiation map.

[0057] Step 2: Obtain the neural radiation map Although the noise of the three-dimensional Gaussian mapping to the two-dimensional space is considered, the direct domain relationship of the image pixels is not considered. For this reason, we use a lightweight network , which consists of three standard convolutional layers, and its structure is as follows Figure 1 As shown. The dimension is , not . The input is , the output is a After getting the mask, we multiply it to get Denoised neural radiance map Finally, we use the U-Net proposed in "NPBG++: Accelerating Neural Point-Based Graphics" to Decoded into Color pictures. U-Net consists of five convolutional blocks. The number of channels output by the convolutional blocks of each layer is 64, 128, 256, 512, and 1024 respectively. The five convolutional blocks are composed of gated convolution and 2x2 average pooling.

[0058] Step 4: Obtain the trained scene expression through end-to-end training.

[0059] For the images collected from the scene to be rendered, an end-to-end training method is used. The properties of the three-dimensional Gaussian are , , , , , , And U-Net will be jointly optimized in the end-to-end training process. The number of training iterations is 60k, and the optimizer uses Adam. In the process of 3k-15k, the densification operation is performed. Specifically, in the densification process, the opacity If the Gaussian gradient is less than 0.005, it is removed; if the gradient of the Gaussian is greater than 0.0002 and the maximum scale is greater than 0.1, a split operation is performed; if the gradient of the Gaussian is greater than 0.0002 and the maximum scale is less than 0.1, a clone operation is performed. During the training process, we splash the 3D Gaussian onto the 2D pixel plane through the camera pose and use formula (3) to obtain the neural radiation map. After that, the rendered image is obtained by decoding through U-Net. The loss function between the rendered image and the true value is calculated.

[0060] The loss function is:

[0061] (5)

[0062] in, and They represent L1 loss function and SSIM loss function respectively.

[0063] Step 5: Free perspective rendering.

[0064] After training, we get a scene representation consisting of three-dimensional Gaussian, , And U-Net. You only need to input the camera pose of the desired rendering perspective into the scene expression to get the rendering result of the desired perspective. The camera pose can be obtained in real time through interactive methods such as mouse movement, keyboard input, VR glasses, etc.

[0065] Embodiment 1:

[0066] The technical solution of the present invention is verified on the Mip-NeRF360 public dataset, which includes indoor and outdoor scenes. The quantitative indicators are shown in Table 1 below:

[0067] Table 1. Experimental quantitative indicators

[0068]

[0069] We use PSNR (peak signal-to-noise ratio), SSIM (Structural Similarity), and LPIPS (perceptual similarity) as evaluation indicators for rendering results. PSNR is one of the commonly used indicators for measuring image rendering quality. The specific formula is as follows:

[0070] PSNR = 10

[0071] Among them, MAX refers to the possible maximum value of the image pixel, and MSE is the root mean square error between the rendered image and the true value. Since it uses the log function, it is difficult to improve the index. SSIM and LPIPS are different from PSNR, which take into account the structural information of the image and the perceptual characteristics of the human visual system.

[0072] In the embodiment, the method of the present invention is compared with 3DGS and Mip-Splatting (Mip-Splatting: Alias-free3D Gaussian Splatting). Mip-Splatting is an improved version based on 3DGS, which performs denoising on the noise introduced in the rasterization process. The results in Table 1 show that compared with the denoising method of Mip-Splatting, our proposed method achieves higher rendering quality with smaller storage.

[0073] When users collect video or image sequences, they first use the Structure from Motion (SfM) algorithm to obtain the point cloud data and image pose information of the scene to be rendered. A neural radiant map is constructed based on the acquired point cloud, and after deep denoising, a trained scene representation is obtained through end-to-end training, including three-dimensional Gaussian, multi-layer perceptron, etc. , lightweight network And U-Net. By inputting the camera pose of the required perspective into the scene representation, the rendering result of the corresponding perspective can be obtained.

[0074] Figure 2 To visualize the results.

[0075] The above contents are further detailed descriptions of the present invention in combination with specific / preferred embodiments, and it cannot be determined that the specific implementation of the present invention is limited to these descriptions. For ordinary technicians in the technical field to which the present invention belongs, they can also make several substitutions or modifications to these described embodiments without departing from the concept of the present invention, and these substitutions or modifications should be regarded as belonging to the protection scope of the present invention.

[0076] Parts of the present invention that are not described in detail belong to the well-known technologies of those skilled in the art.

Claims

1. A three-dimensional Gaussian rendering method based on depth denoising, characterized in that: The following steps are involved: Step 1: Obtain scene point cloud and camera pose based on image sequence or video; Collect the video or picture sequence of the scene to be rendered, preprocess it, and use the structure from motion (SfM) algorithm to extract the point cloud data of the scene and the pose information of the camera corresponding to the image; Step 2: construct a neural radiation map based on the acquired point cloud; Step 3: deep denoising of the obtained neural radiation map; Step 4: Obtain the trained scene expression through end-to-end training; Step 5: Implement 3D Gaussian rendering through the scene expression obtained through training; In order to obtain the rendering result of the desired perspective, the camera pose of the desired perspective is input into the trained scene expression to achieve three-dimensional Gaussian rendering.

2. A three-dimensional Gaussian rendering method based on depth denoising according to claim 1, characterized in that: Step 2: Reparameterize the 3D Gaussian into , , , , These five variables are represented as quaternion, splash length, spatial position, opacity and appearance model; the spatial position of the three-dimensional Gaussian is initialized by the acquired point cloud ; The other attributes are randomly initialized; for each view, the three-dimensional Gaussian is splashed onto the two-dimensional pixel plane through the camera pose; then each pixel to be rendered is occupied by multiple Gaussians; the neural radiation map of the corresponding pixel is obtained using the following formula : (1) Where N represents the number of Gaussians occupied at the pixel position; Introduction to enhance the neural radiance map, which takes into account the position of each pixel The direction of light To correct the neural radiation pattern; It is composed of a multi-layer perceptron; (2)。 3. The three-dimensional Gaussian rendering method based on depth denoising according to claim 2, characterized in that: Step 3: Use a lightweight network The neural radiation map is processed, which consists of three standard convolutional layers; The dimension is ; The input is , the output is a After obtaining the mask, multiply it to get Denoised neural radiance map , and finally use U-Net to Decoded into The U-Net is composed of five convolution blocks, and the number of channels output by the convolution block of each layer is 64, 128, 256, 512, and 1024 respectively. The five convolution blocks are composed of gated convolution and 2x2 average pooling.

4. The 3D Gaussian rendering method based on depth denoising according to claim 3, characterized in that: Step 4: For the images collected from the scene to be rendered, the properties of the three-dimensional Gaussian , , , , , Multilayer Perceptron , lightweight network and U-Net for joint optimization during end-to-end training; During the training process, the three-dimensional Gaussian is splashed onto the two-dimensional pixel plane through the camera pose, and the neural radiation map is obtained using formula (1); then, the rendered image is obtained through U-Net decoding; the loss function between the rendered image and the true value is calculated; The loss function is: (3) in, and Represent the L1 loss function and SSIM loss function respectively; After the training is completed, the trained scene expression is obtained, and the trained scene expression includes optimized three-dimensional Gaussian, multi-layer perceptron , lightweight network And U-Net.

Images