Neural radiance field rendering method based on monte carlo rendering and neural network denoising

Abstract

The application relates to a neural radiance field rendering method based on Monte Carlo rendering and neural network noise reduction, and comprises the following steps: training a neural radiance field by using a training data set to obtain an octree structure neural radiance field representation; performing Monte Carlo volume rendering on the neural radiance field representation based on a conventional tracking algorithm to obtain a noisy image; predicting a filter kernel required for noise reduction according to the noisy image by using a pre-trained filter kernel prediction network; and performing multi-level noise reduction on the noisy image by using multiple filter layers with different filter kernel sizes to obtain a final image. Compared with the prior art, the application greatly reduces the sampling quantity in the volume rendering process, improves the rendering speed and can guarantee the image quality.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image rendering, in particular to a neural radiance field rendering method based on Monte Carlo rendering and neural network denoising. BACKGROUND

[0002] Neural Radiance Fields (NeRF) use implicit neural representations for novel view synthesis and have become a popular method for producing photorealistic renderings of static scenes. However, despite the increasing image quality, real-time rendering of NeRF models remains a significant challenge. The inference process of NeRF models is time-consuming mainly due to two reasons: (1) the large number of floating-point operations (FLOPs) accessed by multi-layer perceptron (MLP) in each sampling point, (2) the large number of sampling points needed on each ray to compute the final pixel color value.

[0003] Recent NeRF acceleration methods mainly focus on reducing the FLOPs caused by neural networks, but still cannot effectively achieve NeRF acceleration. SUMMARY

[0004] The purpose of the present application is to overcome the defects of the prior art and provide a neural radiance field rendering method based on Monte Carlo rendering and neural network denoising to reduce the number of NeRF sampling points and improve inference speed.

[0005] The purpose of the present application can be achieved by the following technical solutions:

[0006] A neural radiance field rendering method based on Monte Carlo rendering and neural network denoising, comprising the following steps:

[0007] Training the neural radiance field using a training data set to obtain an octree structure neural radiance field representation;

[0008] Performing Monte Carlo volume rendering on the neural radiance field representation based on a conventional tracking algorithm to obtain a noisy image;

[0009] Using a pre-trained filter kernel prediction network to predict the filter kernel required for denoising according to the noisy image;

[0010] Using multiple filter layers with different filter kernel sizes to perform multi-level denoising on the noisy image to obtain a final image.

[0011] Further, the construction process of the octree structure neural radiance field comprises:

[0012] Constructing a neural radiance field based on spherical harmonics;

[0013] Training the neural radiance field based on spherical harmonics;

[0014] The trained spherical harmonic based neural radiance field is converted into an octree structure and parameter optimization is performed to obtain an octree structure neural radiance field.

[0015] Further, the process of Monte Carlo volume rendering on the neural radiance field representation comprises:

[0016] The rendering equation is estimated by Monte Carlo integration, a plurality of sampling rays are emitted at each pixel, and the pixel colors obtained by the sampling rays are weighted and averaged according to the probability density function to perform image rendering.

[0017] All boundary points where the sampling rays intersect the neural radiance field representation are found, and the positions of the sampling points are solved by an analytical method based on the found boundary points.

[0018] A batch regular tracing method is used to enable each pixel to obtain the results of a plurality of sampling rays in one light path calculation, and the color values calculated by the sampling rays are accumulated.

[0019] Further, the estimation of the rendering equation by Monte Carlo integration is as follows:

[0020] The integrand function is averaged at each sampling point and divided by its probability density function, which is proportional to the transmittance probability density function.

[0021] Further, the positions of the sampling points are solved using inverse transform sampling.

[0022] Further, the positions of the sampling points are solved by an analytical method based on the found boundary points as follows:

[0023] The transmittance at the positions of the random sampling points is obtained by sampling the probability density function according to the found boundary points, and the transmittance between the boundary points is accumulated to solve the positions of the sampling points.

[0024] Further, the batch regular tracing method comprises:

[0025] For each pixel, one GPU thread is used to perform a ray-voxel intersection process, and all sampling ray samples in the pixel are batch processed in this intersection process.

[0026] In the process of sampling N rays for each pixel, N random numbers are first generated and sorted, and then in the ray-voxel intersection process, when the ray finds the sampling point corresponding to the smallest random number, the spherical harmonic function color values of all sampling point samples contained in the node are calculated, and the color is accumulated into the final result. Repeat this process until all random samples or rays intersect with the boundary are estimated.

[0027] Further, the filter kernel prediction network is a RepVGG block provided with a plurality of convolution branches; in the training process of the filter kernel prediction network, M 1x1 and M 3x3 convolution branches are combined by element-wise addition and fed into a ReLU6 layer to form the filter kernel prediction network for training, and M is a positive integer;

[0028] After training, all branches are combined by element-wise addition by filling zero values on the parameters of the M 1x1 convolution branches, converted into a single 3x3 convolution layer, and then connected with a ReLU6 layer to form the trained filter kernel prediction network.

[0029] Further, the multi-level noise reduction comprises:

[0030] According to the output of the filter kernel prediction network, a filter kernel graph of different levels is constructed;

[0031] According to the noisy image of the filter kernel graph of different levels, image noise reduction is performed, and the image noise reduction results are weighted and averaged to obtain a final image.

[0032] Further, according to the output of the filter kernel prediction network, a filter kernel graph of different levels is constructed, which specifically comprises:

[0033] For the i-th filter layer, the corresponding filter kernel size is set as ki=2i+1, the feature map output by the filter kernel prediction network is expanded using a sliding window with a size of ki x ki, and then normalized using a softmax function to construct a filter kernel graph with a resolution of H x W x (ki x ki), wherein H is the height and W is the width.

[0034] Further, the image noise reduction results of different levels are weighted and averaged to obtain a final image, which specifically comprises:

[0035] The noisy image is filtered according to the obtained filter kernel graphs of different levels, and the weighted average value of the filtered images is calculated to fuse the filtered images to obtain a final image.

[0036] According to the final image, the parameters of the filter kernel prediction network are optimized by minimizing the symmetric mean absolute percentage error loss function.

[0037] Compared with the prior art, the present application has the following advantages:

[0038] (1) The present application uses a neural radiance field based on spherical harmonics and converts it into an octree structure, which can improve the rendering speed and the quality of the rendered picture.

[0039] The Monte Carlo volume rendering method based on the conventional tracking algorithm is adopted to improve the current ray stepping algorithm of NeRF, which can reduce the sampling number of each pixel without introducing significant deviation.

[0040] The prediction result is multi-level denoised using a light filter kernel prediction network, so that the rendering speed is improved without affecting the image quality.

[0041] (2) The present application is 1.5-2 times faster than the PlenOctree benchmark method in inference time, and several orders of magnitude faster than NeRF, and achieves a rendering speed of more than 100 frames per second on a real dataset with a resolution of 1920x1080. BRIEF DESCRIPTION OF DRAWINGS

[0042] Figure 1 A flowchart of a neural radiance field rendering method based on Monte Carlo rendering and neural network denoising provided in an embodiment of the present application is shown in the figure;

[0043] Figure 2 A comparison diagram of ray stepping and conventional tracing provided in an embodiment of the present application is shown in the figure;

[0044] Figure 3 A structure of a filter kernel prediction network and a modified RepVGG block provided in an embodiment of the present application are shown in the figure;

[0045] Figure 4 A flowchart of multi-level denoising provided in an embodiment of the present application is shown in the figure. DETAILED DESCRIPTION

[0046] In order to make the purpose, technical scheme and advantages of the embodiments of the present application clearer, the technical scheme of the embodiments of the present application will be described clearly and completely below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all the embodiments. The components of the embodiments of the present application described and shown in the drawings herein can be arranged and designed in various different configurations.

[0047] Therefore, the following detailed description of the embodiments of the present application provided in the drawings is not intended to limit the scope of the claimed present application, but only represents selected embodiments of the present application. All other embodiments obtained by those of ordinary skill in the art based on the embodiments in the present application without creative labor are within the scope of protection of the present application.

[0048] It should be noted that: similar numbers and letters represent similar items in the following drawings, so once an item is defined in one drawing, it does not need to be further defined and explained in subsequent drawings.

[0049] Embodiment 1

[0050] As Figure 1As shown, the embodiment provides a neural radiance field rendering method based on Monte Carlo rendering and neural network noise reduction, including the following steps:

[0051] S1: training the neural radiance field using the training data set to obtain an octree structure neural radiance field representation;

[0052] S2: performing Monte Carlo volume rendering on the neural radiance field representation based on a regular tracking algorithm to obtain a noisy image;

[0053] S3: predicting the filter kernel required for noise reduction according to the noisy image using a pre-trained filter kernel prediction network;

[0054] S4: performing multi-level noise reduction on the noisy image using multiple filter layers with different filter kernel sizes to obtain a final image.

[0055] The above steps will be described in detail below.

[0056] In step S1, an octree structure neural radiance field is constructed, specifically including:

[0057] A spherical harmonic function (SH) based neural radiance field (NeRF-SH) is used to improve the predicted picture quality;

[0058] The spherical harmonic function based neural radiance field is trained on the training set to obtain a neural radiance field representation of the scene;

[0059] The trained spherical harmonic function based neural radiance field is converted into an octree structure to improve the rendering speed during testing;

[0060] Further parameter optimization is performed on the converted octree to obtain better rendering picture quality.

[0061] In step S2, a Monte Carlo volume rendering method based on a regular tracking algorithm (Regular Tracking) is proposed, mainly including:

[0062] The original NeRF ray marching volume rendering algorithm is changed to a regular tracking algorithm (Regular Tracking);

[0063] The rendering equation is estimated by Monte Carlo integration, that is, the integral function is averaged at each sampling point and divided by its probability density function, the pixel value of each sampling point is calculated, and image rendering is performed;

[0064] Find all boundaries intersecting the scene representation, and solve the position of the sampling point by an analytical method;

[0065] A plurality of sampling light rays are respectively sampled for each pixel point in a batch regular tracing manner, and a color value calculated by the sampling light rays is accumulated.

[0066] The position of the sampling point is solved by an analytical method, and specifically is:

[0067] The transmittance at the position of the random sampling point is obtained by sampling the probability density function according to the found boundary points, and the position of the sampling point is solved by accumulating the transmittance between the boundary points.

[0068] Preferably, a probability density function proportional to the transmittance is selected;

[0069] Preferably, the position of the sampling point is solved by inverse transform sampling;

[0070] The current volume rendering strategy of NeRF is based on the ray marching algorithm, and if the number of sampling points is directly reduced, the result will have a significant deviation, which makes it unsuitable for the low sampling strategy we use. In addition, the ray marching needs to calculate the color of each step along the ray, which is very expensive and leads to a decrease in rendering speed.

[0071] To solve this problem, the embodiment introduces a regular tracking algorithm, which is a Monte Carlo-based volume rendering method that provides an unbiased estimate of the rendering result. Regular tracking estimates the rendering equation by Monte Carlo integration, that is, the integrand is averaged at each sampling point and divided by its probability density function (PDF), which ensures that the expected value of the estimate is equal to the true value and is suitable for subsequent neural denoising. In order to find the position of the sampling point, the regular tracking algorithm finds all the boundaries intersecting the scene representation, and the position is solved by an analytical method. Compared with ray marching, there is no need to perform FLOP operations for color calculation before reaching the sampling point, which greatly reduces the computational cost.

[0072] For regular tracking, each pixel needs to sample N rays. The common way is to create N GPU threads, and perform regular tracking on each thread. This results in a lot of redundant computation. Considering the same pixel, the ray-voxel intersection process of N rays has the same starting point and advancing direction. Therefore, the calculation of the first few intersection nodes along the way of these rays is repeated. At the same time, some nodes in the octree may have a large density value, resulting in some different sampling rays performing color calculation in the same node. That is, if these sampling rays are directly parallelized, the spherical harmonic (SH) at the same position needs to be calculated repeatedly. In addition, directly increasing the number of parallel threads by N times may lead to performance degradation due to the thread scheduling of the GPU.

[0073] To this end, the embodiment proposes a method of batched regular tracking, aiming to solve these problems and further improve real-time performance. For each pixel, we batch all the ray samples and process all the sampling rays in the same ray-voxel intersection process, so we only need to create one GPU thread for each pixel. To sample N samples per pixel, first generate N random numbers and sort them. In the ray-voxel intersection process, when the ray finds the node corresponding to the current smallest random number, calculate the SH function of all sample point samples in this node and accumulate the color to the final result. Repeat this process until all random samples are estimated or the ray intersects with the boundary. The use of batched regular tracking significantly reduces the number of accessed octree nodes and the number of SH calculations per pixel.

[0074] The comparison between ray marching and regular tracking is shown in FIG. 1. Figure 2

[0075] In step S3, an image denoising method based on a neural network is proposed, which includes:

[0076] A convolutional neural network is used to predict the filter kernel required for denoising.

[0077] RepVGG blocks are used and more 1x1 and 3x3 branches are used to improve the network's ability.

[0078] As shown in FIG. 2, the image denoising method based on the neural network includes the following steps: Figure 3 ​As shown, in the training process, M 1x1 and M 3x3 convolution branches are combined by element-wise addition and sent to the ReLU6 layer. We use ReLU6 instead of ReLU in the experiment to prevent precision overflow when training with float16, and set M to 5. After training, the multi-branch structure of the RepVGG block is merged by element-wise addition by padding zero values on the parameters of the 1x1 branch, converting it into a single 3x3 convolution layer, thereby providing an efficient single-branch fully convolutional network that performs the same calculations as before. The RepVGG block structure enhances the capacity of the fully convolutional neural network without increasing the additional time cost during training.

[0079] As shown in FIG. 1, the method comprises the following steps: Figure 4 As shown, the multi-level denoising process proposed in step S4 mainly includes two steps:

[0080] 1) Filter kernel reconstruction: according to the output of the filter kernel prediction network, construct filter kernel maps of different levels;

[0081] Specifically, the method involves using multi-level image denoising with different kernel sizes to enhance the quality of the denoised image. For the i-th layer with kernel size ki=2i+1, a filter kernel map with resolution HxWx(kixki) needs to be constructed from the feature map output by the network. This encoding technique allows each pixel to share kernel weights with its neighboring pixels, significantly reducing the number of parameters required by the neural network and improving inference speed. We calculate the Kernel map by first expanding the feature map using a sliding window of size ki x ki, and then normalizing it using the softmax function.

[0082] 2) Image filtering at each level and weighted averaging of the results of different levels: image denoising is performed on the noisy input according to the filter kernel maps of different levels, and the denoised images are weighted and averaged to obtain the final image.

[0083] Specifically, after obtaining the Kernel maps of different levels, they are applied to the noisy input for filtering, and the weighted average of these filtered images is calculated to fuse to obtain the final image. The final color is used to optimize the parameters of the filter kernel prediction network by minimizing the symmetric mean absolute percentage error (SMAPE) loss function.

[0084] The preferred embodiments of the present application have been described above in detail. It should be understood that modifications and variations to the preferred embodiments could be made by those skilled in the art in light of the teachings above without departing from the spirit of the present application. It is therefore to be understood that what is desired to be protected by letters patent is defined by the scope of the claims that follow and that on behalf of which the summary of the application is intended to serve.

Claims

1. A neural radiance field rendering method based on Monte Carlo rendering and neural network denoising, characterized in that, The method comprises the following steps: training the neural radiance field using a training data set to obtain an octree-structured neural radiance field representation; performing Monte Carlo volume rendering on the neural radiance field representation based on a conventional tracking algorithm to obtain a noisy image; predicting a filter kernel required for noise reduction using a pre-trained filter kernel prediction network according to the noisy image; performing multi-level noise reduction on the noisy image using a plurality of filter layers with different filter kernel sizes to obtain a final image; the process of performing Monte Carlo volume rendering on the neural radiance field representation comprises: estimating the rendering equation by Monte Carlo integration, emitting a plurality of sampling rays at each pixel, and performing weighted average on pixel colors obtained by the sampling rays according to a probability density function to perform image rendering; finding all boundary points where the sampling rays intersect with the neural radiance field representation, and solving the positions of the sampling points by an analytical method based on the found boundary points; adopting a batch conventional tracking mode to enable each pixel to obtain results of a plurality of sampling rays in one light path calculation, and accumulating color values obtained by the sampling ray calculation; the batch conventional tracking mode comprises: for each pixel, performing a ray-voxel intersection process using one GPU thread, and batch processing all sampling ray samples in the pixel in the intersection process; in the process of sampling N rays for each pixel, first, N random numbers are generated and sorted, and then in the ray-voxel intersection process, when the ray finds a sampling point in a node corresponding to the smallest random number, the spherical harmonic function color values of all sampling point samples contained in the node are calculated, and the color is accumulated into the final result; repeat the process until all random samples or the ray intersects with the boundary are estimated.

2. The neural radiance field rendering method based on Monte Carlo rendering and neural network denoising according to claim 1, characterized in that, the construction process of the octree-structured neural radiance field comprises: constructing a spherical harmonic function-based neural radiance field; training the spherical harmonic function-based neural radiance field; converting the trained spherical harmonic function-based neural radiance field into an octree structure, and performing parameter optimization to obtain the octree-structured neural radiance field.

3. The neural radiance field rendering method based on Monte Carlo rendering and neural network denoising according to claim 1, characterized in that, the estimation of the rendering equation by Monte Carlo integration specifically comprises: averaging the integrand at each sampling point and dividing it by its probability density function, which is proportional to the transmittance.

4. The neural radiance field rendering method based on Monte Carlo rendering and neural network denoising according to claim 3, characterized in that, the solving of the positions of the sampling points by the analytical method based on the found boundary points specifically comprises: sampling the probability density function according to the found boundary points to obtain the transmittance at the positions of the random sampling points, and solving the positions of the sampling points by accumulating the transmittance between the boundary points.

5. The neural radiance field rendering method based on Monte Carlo rendering and neural network denoising according to claim 1, characterized in that, the filter kernel prediction network is a RepVGG block with a plurality of convolution branches; in the training process of the filter kernel prediction network, M 1x1 and M 3x3 convolution branches are combined by element-wise addition and sent to a ReLU6 layer to form the filter kernel prediction network for training, and M is a positive integer; after training, all branches are combined by element-wise addition by filling zero values on the parameters of the M 1x1 convolution branches, converted into a single 3x3 convolution layer, and then connected with a ReLU6 layer to form the trained filter kernel prediction network.

6. The neural radiance field rendering method based on Monte Carlo rendering and neural network denoising according to claim 1, characterized in that, the multi-level noise reduction comprises: According to the output of the filter kernel prediction network, filter kernel maps of different levels are constructed; According to the noisy image of the filter kernel map of different levels, image denoising is respectively performed, and a final image is obtained by weighted average of the image denoising results.

7. The neural radiance field rendering method based on Monte Carlo rendering and neural network denoising according to claim 6, characterized in that, According to the output of the filter kernel prediction network, filter kernel maps of different levels are constructed, specifically as follows: For the i-th filter layer, the corresponding filter kernel size is set as ki=2i+1, the feature map output by the filter kernel prediction network is unfolded using a sliding window with a size of ki×ki, then a softmax function is used for normalization, and a filter kernel map with a resolution of H×W×(ki×ki) is constructed, H is the height of the SD, and W is the width.

8. The neural radiance field rendering method based on Monte Carlo rendering and neural network denoising according to claim 7, characterized in that, The image denoising results of different levels are weighted and averaged to obtain a final image, specifically as follows: The obtained filter kernel maps of different levels are used to filter the noisy image, and the weighted average value of the filtered image is calculated to fuse each filtered image to obtain a final image; The parameters of the filter kernel prediction network are optimized by minimizing the symmetric mean absolute percentage error loss function according to the final image.

Images