Filtering-based Monte Carlo gradient path tracking rendering image reconstruction method and device

Abstract

The invention provides a filtering-based Monte Carlo gradient path tracking rendering image reconstruction method and device. The method comprises the steps of obtaining an initial rendering image obtained by Monte Carlo gradient path tracking rendering and an auxiliary information graph in a rendering process; solving an optimized gradient through a gradient optimization objective function according to the initial rendering image and the auxiliary information graph, and generating a multi-resolution hierarchy; based on the initial rendering image, the optimized gradient and the auxiliary information graph, performing filtering calculation layer by layer from the lowest resolution hierarchy to the highest resolution hierarchy according to the multi-resolution hierarchy to obtain a target filtering result; wherein the target filtering result is the filtering result output by the highest resolution level, and the filtering result output by the highest resolution level is the reconstruction result of the initial rendering image. According to the method, gradient domain information and an auxiliary information graph in the rendering process are fully utilized, filtering kernel coefficients are solved in a resolution-by-resolution hierarchy mode, a high-quality and consistent noise reduction image (reconstruction result) is generated at a low sampling rate, the influence of noise in gradient estimation on the reconstruction result is reduced, and the reconstruction efficiency is improved. And the method also has a series of good characteristics such as consistency (progressive convergence to a reference image) and the like, and meets high-efficiency and high-quality rendered image reconstruction requirements.

Description

Technical Field

[0001] The present invention relates to the field of image rendering technology, and in particular to a Monte Carlo gradient path tracing rendering image reconstruction method and device based on filtering. Background Art

[0002] The traditional Monte Carlo rendering method generates noisy rendered images by randomly sampling the light path. Although image noise can be reduced by increasing the number of samples, it still requires a lot of computing time and resources.

[0003] To address this challenge, some studies have introduced the gradient path tracing method, which estimates the adjacent pixels through gradient information while calculating the image color. This aims to obtain lower variance results with fewer samples, while further improving the image quality with the help of reconstruction algorithms.

[0004] At present, the reconstruction methods used for Monte Carlo gradient path tracing rendering can be roughly divided into two categories. The first category is based on the Poisson equation, which constructs the Poisson equation by using the estimated values ​​of the gradient and the image, and solves it by the least squares method to jointly remove the noise in the gradient and the image. This type of method can usually generate high-quality, low-noise rendering images at a high sampling number, but if the sampling number is low, its performance will be significantly reduced, and it will show deficiencies in the robustness and consistency of the image, and it is easy to have visually unsatisfactory burrs and other discontinuities.

[0005] The second type of method is to use neural networks to learn the prior relationship between noisy images and reference images through large amounts of data training. This type of method can produce visually high-quality images at lower sampling rates and embed stronger prior knowledge, thereby showing better visual effects. However, the performance of neural network methods is highly dependent on the data distribution characteristics of the training set, and their generalization ability may be unstable, and there may be large performance fluctuations in different scenarios. In addition, due to their black-box characteristics, this type of method lacks a guarantee for the consistency of algorithm results.

[0006] Therefore, how to solve the problem that the existing Monte Carlo gradient path rendering image reconstruction method performs poorly on low-sampling number rendering images, or cannot ensure the consistency of calculation results in different scenes, is an important issue that needs to be urgently solved in the field of image rendering. Summary of the invention

[0007] The present invention provides a filtering-based Monte Carlo gradient path tracing rendering image reconstruction method and device, which are used to overcome the defects of the existing Monte Carlo gradient path rendering image reconstruction method, which performs poorly on low sampling number rendering images or cannot ensure the consistency of calculation results in different scenes. It can generate high-quality and consistent denoised images at a lower sampling rate, has good generalization performance, and meets the needs of high-efficiency and high-quality image generation.

[0008] On the one hand, the present invention provides a filtering-based Monte Carlo gradient path tracing rendering image reconstruction method, comprising: obtaining an initial rendering image obtained by Monte Carlo gradient path tracing rendering, and an auxiliary information map in the rendering process; according to the initial rendering image and the auxiliary information map, solving the optimized gradient through the gradient optimization objective function, and generating a multi-resolution level; based on the initial rendering image, the optimized gradient and the auxiliary information map, according to the multi-resolution levels from the lowest resolution level to the highest resolution level, filtering calculation is performed layer by layer to obtain a target filtering result; wherein the target filtering result is the filtering result output by the highest resolution level, and the filtering result output by the highest resolution level is the reconstruction result of the initial rendering image.

[0009] Furthermore, after obtaining the initial rendered image, the initial rendered image is subjected to denoising processing, specifically including: pre-defining multiple sets of fixed bilateral filtering parameters; using each set of bilateral filtering parameters to perform preliminary denoising on the initial rendered image to obtain multiple sets of corresponding denoising results; determining the denoised image based on the errors between the multiple sets of denoising results and the target image; accordingly, the step of generating multiple resolution levels based on the initial rendered image and the auxiliary information map specifically includes: generating multiple resolution levels based on the denoised image and the auxiliary information map.

[0010] Furthermore, based on the denoised image and the auxiliary information map, a multi-resolution level is generated, including: based on the denoised image and the auxiliary information map, solving content-related upsampling parameters and downsampling parameters; downsampling the denoised image and the auxiliary information map multiple times according to the downsampling parameters to obtain a multi-resolution level; wherein the auxiliary information map includes a normal map and an albedo map.

[0011] Furthermore, based on the initial rendered image, the optimized gradient and the auxiliary information map, filtering calculation is performed layer by layer from the low-resolution layer to the high-resolution layer according to the multi-resolution layers, including: at the lowest resolution layer, taking the denoised image, the optimized gradient and the auxiliary information map as input, filtering calculation is performed through the filtering optimization objective function to obtain an intermediate filtering result; according to the calculated upsampling parameters, upsampling is performed layer by layer to the higher resolution layers of the lowest resolution layer, taking the denoised image, the optimized gradient and the updated auxiliary information map as input, filtering calculation is performed through the filtering optimization objective function to obtain a target filtering result; wherein the updated auxiliary information map includes a normal map, an albedo map and the intermediate filtering result outputted from the lower resolution layer is sampled to an image of the upper resolution layer through the upsampling parameters.

[0012] Furthermore, the filtering optimization objective function is defined as follows: ; ; ; ; ; ; in, , , , express , , , The weight coefficient of Express expectations, Represents the pixels in the initial rendered image The pixel color, Represents the pixels in the initial rendered image Color expectations, Represents pixels Pixels in the neighborhood, Represents pixels Neighborhood of represents the filter kernel coefficient, Represents the pixels in the initial rendered image Color expectations, represents the variance of the rendering, represents the albedo map, represents the normal map, Indicates that the intermediate filtering result outputted at a lower resolution level is sampled to an image at the next higher resolution level through an upsampling parameter. Represents the pixels in the image after noise reduction The pixel color, Represents the pixels in the initial rendered image The pixel color.

[0013] Furthermore, the gradient optimization objective function is defined as follows: ; ; ; in, represents the gradient optimization objective function, Represents the parameter used to control the strength of auxiliary information. Represents pixels, , Respectively for , The pixel-by-pixel weight of the gradient in the direction, , Respectively , The optimized gradient in the direction, , Respectively , The original gradient in direction, , Respectively for , The pixel-wise weight of the gradient in the direction.

[0014] In the second aspect, the present invention also provides a filtering-based Monte Carlo gradient path tracing rendering image reconstruction device, including: a rendering information acquisition module, used to obtain the initial rendering image obtained by Monte Carlo gradient path tracing rendering, and an auxiliary information map in the rendering process; an optimized gradient solution module, used to solve the optimized gradient through the gradient optimization objective function according to the initial rendering image and the auxiliary information map, and generate a multi-resolution level; a resolution-by-resolution level filtering module, used to perform filtering calculations layer by layer from the lowest resolution level to the highest resolution level according to the multi-resolution levels based on the initial rendering image, the optimized gradient and the auxiliary information map, to obtain a target filtering result; wherein the target filtering result is the filtering result output by the highest resolution level, and the filtering result output by the highest resolution level is the reconstruction result of the initial rendering image.

[0015] In a third aspect, the present invention also provides an electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the filtering-based Monte Carlo gradient path tracing rendering image reconstruction method as described in any one of the above is implemented.

[0016] In a fourth aspect, the present invention further provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements any of the above-described filtering-based Monte Carlo gradient path tracing rendering image reconstruction methods.

[0017] In a fifth aspect, the present invention further provides a computer program product, comprising a computer program, which, when executed by a processor, implements any of the above-described filtering-based Monte Carlo gradient path tracing rendering image reconstruction methods.

[0018] The present invention provides a method for reconstructing a rendered image based on a Monte Carlo gradient path tracing rendering. The method obtains the initial rendered image obtained by Monte Carlo gradient path tracing rendering and the auxiliary information map in the rendering process. The method solves the optimized gradient through the gradient optimization objective function according to the initial rendered image and the auxiliary information map, and generates a multi-resolution level. Then, based on the initial rendered image, the optimized gradient and the auxiliary information map, the multi-resolution level performs filtering calculations layer by layer from the lowest resolution level to the highest resolution level to obtain the target filtering result. The target filtering result is the filtering result output by the highest resolution level, and the filtering result output by the highest resolution level is the reconstruction result of the initial rendered image. The method fully utilizes the gradient domain information and the auxiliary information map in the rendering process, solves the filtering kernel coefficients by resolution level, and generates a high-quality and consistent denoised image (reconstruction result) at a lower sampling rate. It not only reduces the influence of noise in the gradient estimation on the reconstruction result, but also has a series of good characteristics such as consistency (progressive convergence to the reference image), meeting the requirements of efficient and high-quality rendering image reconstruction. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0020] Figure 1 It is a flowchart of a filtering-based Monte Carlo gradient path tracing rendering image reconstruction method provided in an embodiment of the present invention.

[0021] Figure 2 It is a schematic diagram of the general framework of the filtering-based Monte Carlo gradient path tracing rendering image reconstruction method provided in an embodiment of the present invention.

[0022] Figure 3It is a complete flowchart of the filtering-based Monte Carlo gradient path tracing rendering image reconstruction method provided by an embodiment of the present invention.

[0023] Figure 4 It is a structural schematic diagram of a filtering-based Monte Carlo gradient path tracing rendering image reconstruction device provided in an embodiment of the present invention.

[0024] Figure 5 It is a schematic diagram of the physical structure of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0025] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0026] It should be noted that there are currently two main types of reconstruction methods used for Monte Carlo gradient path rendering: the first type of method is a reconstruction algorithm based on the Poisson equation, which mainly uses the estimated values ​​of the gradient and the image to construct the Poisson equation, perform the least squares solution, and jointly remove the gradient and image noise; the second type of method is to use a neural network to learn the prior knowledge about the mapping between the noisy image and the reference image by training on a large amount of data.

[0027] Among them, the first type of methods can usually only generate high-quality, low-noise rendered images at high sampling numbers. If the sampling number is low, its performance will drop significantly, and it will show deficiencies in image robustness and consistency, and it is easy to have visually unsatisfactory discontinuities such as glitches.

[0028] The performance of the second type of method is highly dependent on the data distribution characteristics of the training set, and its generalization ability may be unstable, and there may be large performance fluctuations in different scenarios. In addition, due to its black box characteristics, this type of method lacks a guarantee for the consistency of the algorithm results.

[0029] In view of this, the present invention proposes a Monte Carlo gradient path tracing rendering image reconstruction method based on filtering, specifically, Figure 1 A schematic flow chart of a filtering-based Monte Carlo gradient path tracing rendering image reconstruction method provided in an embodiment of the present invention is shown.

[0030] like Figure 1 As shown, the method includes steps S110-S130, and steps S110-S130 and related steps will be described in detail below.

[0031] S110, obtaining an initial rendering image obtained by Monte Carlo gradient path tracing rendering, and an auxiliary information graph during the rendering process.

[0032] Monte Carlo gradient path tracing is a physically based rendering technique that calculates images by simulating the propagation of light in a scene. This technique can produce very realistic lighting effects, including soft shadows, reflections, refractions, and global illumination.

[0033] It is understandable that, through Monte Carlo gradient path tracing rendering, an initial rendering image can be obtained for subsequent reconstruction process. Since Monte Carlo gradient path tracing rendering technology is relatively mature and is not the focus of the present invention, it will not be described in detail here.

[0034] It is worth mentioning that in the Monte Carlo gradient path tracing rendering process, in addition to recording the image buffer of pixel color, it is also possible to record the image buffer of pixel gradient (divided into x direction and y direction), and can also record a variety of auxiliary information buffers and statistical information to help understand and debug the rendering results. The various auxiliary information buffers here include but are not limited to normal maps, albedo maps, material color maps, sample variance maps, etc.

[0035] Among them, the normal map shows the direction of the surface normal at each point in the scene. The albedo map refers to the ratio of light reflected by the surface to the incident light, and is usually used to describe the color properties of the material without considering the influence of lighting. The material color map shows the inherent color of the objects in the scene without any lighting effects. The sample variance map can help identify areas with higher noise during the rendering process, because high variance means that more samples are needed to reduce noise.

[0036] In a specific embodiment, the auxiliary information map includes a normal map and an albedo map.

[0037] After obtaining the initial rendering image obtained by Monte Carlo gradient path tracing rendering and the auxiliary information map in the rendering process in step S110, further, step S120 is performed.

[0038] S120, according to the initial rendering image and the auxiliary information graph, solving the optimized gradient through the gradient optimization objective function, and generating a multi-resolution level.

[0039] It is easy to understand that by defining a loss function for measuring the difference between images of different resolutions, this loss function takes the initial rendered image and each auxiliary information map as input, and by solving the defined loss function using the gradient descent method or other optimization algorithms, the optimal and content-related upsampling parameters and downsampling parameters can be obtained. Among them, the downsampling parameters are used to guide the downsampling operation to generate multi-resolution levels, and the upsampling parameters are used to guide the upsampling filter to obtain the final reconstruction result of the initial rendered image.

[0040] Then, according to the optimal downsampling parameters obtained by solving, the rendered initial rendering image and the auxiliary information map are downsampled multiple times to obtain a series of images with different resolutions, that is, multi-resolution levels.

[0041] It is worth mentioning that since the initial rendering image and pixel gradient map generated by the Monte Carlo gradient path tracing rendering algorithm contain a lot of noise, in an optimized embodiment, after obtaining the initial rendering image, the initial rendering image is denoised using predefined multiple sets of fixed bilateral filtering parameters to obtain a denoised image, and the pixel gradient map is optimized to obtain an optimized gradient, that is, an optimized pixel gradient map.

[0042] After solving the optimized gradient through the gradient optimization objective function according to the initial rendering image and the auxiliary information map and generating multi-resolution levels in step S120, further, step S130 is performed.

[0043] S130, based on the initial rendered image, the optimized gradient and the auxiliary information graph, filtering calculation is performed layer by layer from the lowest resolution layer to the highest resolution layer according to the multi-resolution layers to obtain a target filtering result; wherein the target filtering result is the filtering result output by the highest resolution layer, and the filtering result output by the highest resolution layer is the reconstruction result of the initial rendered image.

[0044] It is easy to understand that a quadratic optimization problem of filtering is constructed at each resolution level, and the initial rendered image, optimized gradient, and auxiliary information maps (normal map, albedo map) and the intermediate filtering results output by the lower resolution level (except for the lowest resolution level, the inputs of other resolution levels have the intermediate filtering results) are used to calculate the filter kernel coefficients, and the downsampled image corresponding to the resolution level is filtered to obtain the intermediate filtering results. Then, the upsampling parameters corresponding to the downsampling parameters of the resolution level are used to upsample the intermediate filtering results to a higher resolution level, and the intermediate filtering results output by the lower resolution level are used as one of the auxiliary information maps to solve the quadratic optimization problem of filtering, thereby obtaining the filtering results output by the highest resolution level, which is the final reconstruction result of the initial rendered image.

[0045] In this embodiment, the initial rendering image obtained by Monte Carlo gradient path tracing rendering and the auxiliary information map in the rendering process are obtained, and a multi-resolution level is generated based on the initial rendering image and the auxiliary information map, and then based on the initial rendering image, the optimized gradient and the auxiliary information map, filtering calculation is performed layer by layer from the lowest resolution level to the highest resolution level according to the multi-resolution level to obtain the target filtering result; wherein the target filtering result is the filtering result output by the highest resolution level, and the filtering result output by the highest resolution level is the reconstruction result of the initial rendering image. This method fully utilizes the gradient domain information and the auxiliary information map in the rendering process, solves the filter kernel coefficients resolution by resolution level, and generates a high-quality and consistent denoised image (reconstruction result) at a lower sampling rate. It not only reduces the impact of noise in the gradient estimation on the reconstruction result, but also has a series of good characteristics such as consistency (progressive convergence to the reference image), meeting the requirements of efficient and high-quality rendering image reconstruction.

[0046] On the basis of the above-mentioned embodiment, the preliminary denoising process of the initial rendered image will be described in detail below.

[0047] After obtaining the initial rendered image, the initial rendered image is subjected to denoising processing, specifically including: pre-defining multiple sets of fixed bilateral filtering parameters; using each set of bilateral filtering parameters to perform preliminary denoising on the initial rendered image to obtain multiple sets of corresponding denoising results; and determining the denoised image according to the errors between the multiple sets of denoising results and the target image.

[0048] Bilateral filtering is a nonlinear filtering method that can maintain edges well while smoothing images. Bilateral filtering parameters include but are not limited to neighborhood diameter, color space standard deviation, and spatial domain standard deviation. The selection of bilateral filtering parameters has a great influence on the noise reduction effect.

[0049] Among them, the neighborhood diameter determines the size of the neighborhood considered by the filter. The color space standard deviation controls the influence of color differences between pixels. A larger color space standard deviation will make more distant color differences also considered. The spatial domain standard deviation controls the influence of the distance between pixels. A larger spatial domain standard deviation will make more distant pixels also affect the current pixel.

[0050] It is easy to understand that the present embodiment predefines multiple sets of fixed bilateral filtering parameters, and applies these multiple sets of different and fixed bilateral filtering parameters to the initial rendered image, respectively, to obtain multiple sets of corresponding denoising results. Then, a preset loss function is used to calculate the error between each set of denoising results and the target image, and the result with the smallest error is selected by comparing the error of each pixel in different denoising results. Thus, based on the denoising result with the smallest error selected pixel by pixel, the continuity is ensured by linear combination relaxation operation, and the optimal denoising result, that is, the denoised image, can be determined.

[0051] The number of groups of predefined bilateral filtering parameters can be set according to actual needs and is not specifically limited here. For example, in a specific embodiment, three groups of bilateral filtering parameters are predefined: the neighborhood diameter is 5, the color space standard deviation is 75, and the spatial domain standard deviation is 75; the neighborhood diameter is 9, the color space standard deviation is 100, and the spatial domain standard deviation is 100; the neighborhood diameter is 15, the color space standard deviation is 150, and the spatial domain standard deviation is 150.

[0052] The preset loss function can be set according to actual needs and is not specifically limited here.

[0053] For example, in a specific embodiment, the preset loss function is a SURE loss function. The SURE loss function can be used to optimize the parameters of the denoising algorithm without knowing the actual noise-free image.

[0054] After the initial rendered image is subjected to denoising, correspondingly, multiple resolution levels are generated according to the initial rendered image and the auxiliary information map, including: generating multiple resolution levels according to the denoised image and the auxiliary information map.

[0055] It should be noted that when performing preliminary denoising on the initial rendered image, auxiliary information maps including normals and albedo can also be combined to optimize the denoising process. Specifically, on the one hand, the bilateral filter parameters can be dynamically adjusted based on the information of the normal map and the albedo map. For example, the value of the bilateral filter parameters can be reduced in edge areas (where the normal changes greatly); on the other hand, when calculating the error, the information of the normal map and the albedo map can be combined to increase the weight coefficients of certain areas, making the errors in certain areas more important.

[0056] In this embodiment, multiple groups of fixed bilateral filtering parameters are predefined, and each group of bilateral filtering parameters is used to perform preliminary denoising on the initial rendered image, so as to obtain multiple groups of corresponding denoising results, and then the denoised image is determined according to the error between the multiple groups of denoising results and the target image, and the multi-resolution level is generated according to the denoised image and the auxiliary information map, so that based on the initial rendered image, the optimized gradient and the auxiliary information map, the filtering calculation is performed layer by layer from the lowest resolution level to the highest resolution level according to the multi-resolution level to obtain the target filtering result. This method fully utilizes the gradient domain information and the auxiliary information map in the rendering process, solves the filter kernel coefficients at each resolution level, and generates a high-quality and consistent denoised image (reconstruction result) at a lower sampling rate, which not only reduces the influence of noise in the gradient estimation on the reconstruction result, but also has a series of good characteristics such as consistency (progressive convergence to the reference image), meeting the requirements of efficient and high-quality rendering image reconstruction.

[0057] On the basis of the above-mentioned embodiment, the process of generating multiple resolution levels will be described in detail below.

[0058] Generate a multi-resolution level according to the denoised image and the auxiliary information map, including: solving content-related upsampling parameters and downsampling parameters according to the denoised image and the auxiliary information map; downsampling the denoised image and the auxiliary information map multiple times according to the downsampling parameters to obtain a multi-resolution level; wherein the auxiliary information map includes a normal map and an albedo map.

[0059] It is understandable that in Monte Carlo gradient path tracing rendering, downsampling can effectively increase the number of effective samples per pixel and reduce the variance, which means that the smaller variance after downsampling to a low resolution is more friendly to various noise reduction algorithms. However, simple average downsampling will also lead to loss of details, and the fine structure in the image cannot be restored after being averaged during the downsampling process.

[0060] In this regard, this embodiment adopts a content-related up-sampling and down-sampling generation method. (i.e. denoised image and auxiliary information map) and downsampled map , downsampled image Each pixel in the image is the original image The linear combination of the two pixels in , for details, please refer to the following formula (1).

[0061] (1).

[0062] In formula (1), Represents the pixels in the downsampled image The pixel color, Represents the pixels in the original image The weight of , Represents the pixels in the original image , Pixels The pixel color.

[0063] The optimization objective of the content-dependent up- and down-sampling algorithm is as follows (2).

[0064] (2).

[0065] In formula (2), represents the original image (i.e. the denoised image and the auxiliary information image), Represents the downsampled image The upsampled image is obtained by upsampling.

[0066] The linear combination parameters of upsampling and downsampling are jointly optimized by equations (1) and (2) to minimize the reconstruction error, and the upsampling parameters and downsampling parameters are obtained.

[0067] Among them, the upsampling map It can be obtained by linear interpolation from four pixels at the corresponding position in the downsampled image, or by selecting a pixel from the neighborhood of the corresponding position in the downsampled image. The specific selection strategy is obtained by jointly optimizing the downsampling parameters.

[0068] It should be noted that since the rendering result (initial rendering image) usually has more noise, using the rendering result to directly calculate the upsampling parameters and downsampling parameters can easily mistake the noise for fine structures. Therefore, the reference image actually used in the calculation is the result after preliminary denoising, that is, the denoised image.

[0069] After solving the upsampling parameters and downsampling parameters, the denoised image and auxiliary information map are downsampled according to the downsampling parameters. Each downsampling will generate a new lower resolution image version. The same downsampling operation is continuously applied to the low-resolution image obtained by the previous downsampling operation until the required minimum resolution or number of layers is reached. All images of different resolutions are combined to form a multi-resolution hierarchical structure. Each layer of this structure represents the performance of the denoised image and auxiliary information map at a specific resolution.

[0070] In this way, multiple resolution levels can be generated.

[0071] Furthermore, a quadratic optimization problem of filtering is constructed at each resolution level, and the filter kernel coefficients are calculated using the denoised image, optimized gradient, auxiliary information graph, and intermediate filtering results output by the lower resolution level (except for the lowest resolution level, the inputs of other resolution levels all have the intermediate filtering result item), and the downsampled image corresponding to the resolution level is filtered to obtain the intermediate filtering result. Then, the upsampling parameters corresponding to the downsampling parameters of the resolution level are used to upsample the intermediate filtering results to a higher resolution level, and the intermediate filtering results output by the lower resolution level are used as one of the auxiliary information graphs to solve the quadratic optimization problem of filtering, thereby obtaining the filtering result output by the highest resolution level, which is the final reconstruction result of the initial rendered image.

[0072] In this embodiment, by solving content-related upsampling parameters and downsampling parameters based on the denoised image and the auxiliary information map, and performing multiple downsampling on the denoised image and the auxiliary information map according to the downsampling parameters, a multi-resolution level is obtained, and then based on the initial rendered image, the optimized gradient, and the auxiliary information map, filtering calculations are performed layer by layer from the lowest resolution level to the highest resolution level according to the multi-resolution level to obtain the reconstruction result of the initial rendered image. This method fully utilizes the gradient domain information and the auxiliary information map in the rendering process, solves the filter kernel coefficients at each resolution level, and generates a high-quality and consistent denoised image (reconstruction result) at a lower sampling rate. It not only reduces the impact of noise in the gradient estimation on the reconstruction result, but also has a series of good characteristics such as consistency (progressive convergence to the reference image), meeting the requirements of efficient and high-quality rendering image reconstruction.

[0073] On the basis of the above-mentioned embodiment, the process of optimizing the pixel gradient map obtained by rendering will be further described in detail below.

[0074] It is understandable that the gradient (pixel gradient map) generated by the Monte Carlo gradient path tracing rendering algorithm is noisy, which makes the gradients obtained by this set of renderings ( , direction) is not the gradient of any image. This embodiment defines the "legitimacy" of the gradient here, requiring the divergence of the gradient field to be , that is, the gradient The requirement is to satisfy the following formula (3):

[0075] (3).

[0076] In formula (3), Indicates that the image after denoising is located at coordinates The pixels in The gradient in direction, Indicates that the image after denoising is located at coordinates exist The gradient in direction, Indicates that the image after denoising is located at coordinates The pixels in The gradient in direction, Indicates that the image after denoising is located at coordinates The pixels in Directional gradient.

[0077] In this embodiment, the gradient that satisfies the above formula (3) is a legal gradient, and the legal gradient can reconstruct the entire image based on any pixel.

[0078] Under the conditional constraint of satisfying the legality (Formula (1) above), a gradient optimization objective function is constructed to optimize the gradient. The optimization mainly considers the gradient itself and the auxiliary information to obtain the optimized gradient.

[0079] In terms of the gradient itself, the optimization objective function is formulated based on the gradient itself , please refer to the following formula (4) for details.

[0080] (4).

[0081] In formula (4), Represents pixels, and They are respectively for direction, The pixel-by-pixel weight of the gradient in the direction, , Respectively , The optimized gradient in the direction, , Respectively , The original gradient in direction.

[0082] and The definition of can be found in the following equations (5) and (6).

[0083] (5).

[0084] (6).

[0085] In equations (5)-(6), is a predefined constant, A decimal number used to avoid denominators being zero.

[0086] The design of the gradient itself can ensure that the optimization process tends to retain the parts with larger and smaller absolute values ​​of the gradient, which represent the boundary and the interior of the region respectively.

[0087] In terms of auxiliary information, the objective function is formulated using the boundary and internal information provided by the auxiliary information. , please refer to the following formula (7) for details.

[0088] (7).

[0089] In formula (7), we also set , Pixel-wise weight of the gradient in the direction and , controlling the weight of the gradient in the optimization according to the auxiliary information graph.

[0090] and The definition of can be found in the following equations (8)-(9).

[0091] (8).

[0092] (9).

[0093] In equations (8)-(9), , Represents the pixel gradient of each auxiliary information map. Pixel-by-pixel weight and The gradient inside the image area is required to be as close to 0 as possible, so as to remove noise in the gradient domain.

[0094] According to the above formulas (4)-(9), the gradient optimization objective function can be defined. The gradient optimization objective function can be seen in the following formula (10).

[0095] (10).

[0096] In formula (10), It is a parameter used to control the strength of the auxiliary information, and it will converge to 0 as the number of input samples increases to ensure the consistency of gradient optimization.

[0097] Based on formula (10), the optimized gradient can be obtained , This operation can make the rendering image reconstruction algorithm more robust and avoid obvious visual glitches and other defects.

[0098] In this embodiment, the gradient obtained by rendering is optimized by the gradient optimization objective function to obtain the optimized gradient, and then based on the initial rendered image, the optimized gradient and the auxiliary information map, filtering calculation is performed layer by layer from the lowest resolution layer to the highest resolution layer according to the multi-resolution layer to obtain the reconstruction result of the initial rendered image. This method fully utilizes the gradient domain information and auxiliary information map in the rendering process, solves the filter kernel coefficients at each resolution layer, and generates a high-quality and consistent denoised image (reconstruction result) at a lower sampling rate. It not only reduces the impact of noise in the gradient estimation on the reconstruction result, but also has a series of good characteristics such as consistency (progressive convergence to the reference image), meeting the requirements of efficient and high-quality rendering image reconstruction.

[0099] On the basis of the above-mentioned embodiment, the process of layer-by-layer filtering and reconstruction will be described in detail below.

[0100] Based on the initial rendered image, optimized gradients and auxiliary information maps, filtering calculations are performed layer by layer from low-resolution layers to high-resolution layers according to multiple resolution levels, including: at the lowest resolution level, using the denoised image, optimized gradients and auxiliary information maps as inputs, filtering calculations are performed through a filtering optimization objective function to obtain an intermediate filtering result; according to the calculated upsampling parameters, upsampling is performed layer by layer to a higher resolution level of the lowest resolution level, using the denoised image, optimized gradients and updated auxiliary information maps as inputs, filtering calculations are performed through a filtering optimization objective function to obtain a target filtering result; wherein the updated auxiliary information maps include normal maps, albedo maps and intermediate filtering results output from a lower resolution level, which are sampled to an image of an upper resolution level through upsampling parameters.

[0101] It is understandable that filtering is a very important and widely used method in Monte Carlo gradient path tracing rendering denoising. This method uses a linear combination of each pixel neighborhood to calculate the denoised result, that is, for pixels , using a series of filter kernel coefficients In its neighborhood The inner linear summation is used to obtain the filtering result, which can be specifically referred to in the following equation (11). The filtering can obtain a smooth output result while ensuring consistency.

[0102] (11).

[0103] This embodiment proposes a gradient-guided filter kernel calculation algorithm based on the filtering method, using the optimized gradient and auxiliary information Figure 1The filter kernel with consistent convergence can be used to robustly reconstruct the gradient path tracing rendering result. The algorithm obtains the filter kernel coefficient by filtering the optimization objective function, which is divided into three parts: gradient guidance part, auxiliary information autoregression part, and preliminary denoising regression part.

[0104] In the gradient guidance part, we start from the perspective of minimizing the mean square error of the filtering results, as shown in the following formula (12).

[0105] (12).

[0106] In formula (12), Represents the pixels in the initial rendered image The pixel color, Represents the pixels in the initial rendered image Color expectations, Represents pixels Pixels in the neighborhood, Represents pixels Neighborhood of represents the filter kernel coefficient, Represents the pixels in the initial rendered image Color expectations, Indicates the variance of the rendering.

[0107] According to formula (12), the optimal filter kernel coefficient can be obtained in a statistical sense , however, the expected color of the image pixels , It cannot be obtained, so this embodiment uses the optimized gradient to estimate In addition, the variance The sample variance estimate obtained by rendering can be used directly.

[0108] In the auxiliary information autoregression part, there is a restriction on the optimization objective, which requires that for each auxiliary information map (normal map, albedo map, and the reconstruction result sampled from a lower resolution level (i.e., the intermediate filtering result)), the difference between the filter kernel applied to it and the original input should be as small as possible. For details, please refer to the following equations (13)-(15).

[0109] (13).

[0110] (14).

[0111] (15).

[0112] In equations (13)-(15), represents the albedo map, represents the normal map, It means that the intermediate filtering result outputted at a lower resolution level is sampled to an image at a higher resolution level through an upsampling parameter.

[0113] It should be noted that if the current resolution level is the lowest resolution level, Does not exist, can take value 0.

[0114] In the initial denoising regression part, the input image (initial rendered image) is used to regress the initial denoising result (denoised image) for further constraints. For details, see the following formula (16).

[0115] (16).

[0116] In formula (16), Represents the pixels in the image after noise reduction The pixel color, Represents the pixels in the initial rendered image The pixel color.

[0117] Based on the above equations (12)-(16), the filtering optimization objective function can be defined, as shown in the following equation (17).

[0118] (17).

[0119] By solving The minimum value of can be used to obtain the filter kernel coefficient and complete the filtering.

[0120] Then, the obtained filter kernel coefficient Substituting into formula (11), we can get the filtering result , which is the intermediate filtering result of each resolution level, that is, the reconstruction result of the intermediate rendered image.

[0121] In this embodiment, the denoised image, optimized gradient and auxiliary information map are used as inputs at the lowest resolution level, and filtering calculation is performed through the filtering optimization objective function to obtain an intermediate filtering result, and upsampling is performed layer by layer to a higher resolution level of the lowest resolution level according to the calculated upsampling parameters, and the denoised image, optimized gradient and updated auxiliary information map are used as inputs, and filtering calculation is performed through the filtering optimization objective function to obtain the target filtering result. This method fully utilizes the gradient domain information and auxiliary information map in the rendering process, solves the filter kernel coefficients at each resolution level, and generates a high-quality and consistent denoised image (reconstruction result) at a lower sampling rate. It not only reduces the impact of noise in the gradient estimation on the reconstruction result, but also has a series of good characteristics such as consistency (progressive convergence to the reference image), meeting the requirements of efficient and high-quality rendering image reconstruction.

[0122] Additionally, in some embodiments, Figure 2 A schematic diagram showing the general framework of a filtering-based Monte Carlo gradient path tracing rendering image reconstruction method provided by an embodiment of the present invention is shown.

[0123] like Figure 2 As shown, the filtering-based Monte Carlo gradient path tracing rendering image reconstruction method provided in the embodiment of the present invention includes a downsampling process and an upsampling process. During downsampling, the input includes an image (initial rendering image), a gradient (pixel gradient map obtained by rendering), preliminary denoising (denoised image) and auxiliary information map (normal map and albedo map) to generate multi-resolution levels. Among them, the denoised image is obtained by performing preliminary denoising on the initial rendering image, which has been described in detail in the above embodiment and will not be repeated here.

[0124] During upsampling, the optimized gradient (optimized gradient), preliminary denoising result (denoising image), auxiliary information map (normal map, albedo map) and upsampled guidance map (intermediate filtering result / reconstruction result output by a lower resolution layer) are used as input to execute the gradient-guided filtering algorithm provided by the embodiment of the present invention, and filtering calculations are performed layer by layer from the lowest resolution layer to the highest resolution layer. The filtering result output by the highest resolution layer is the final reconstruction result of the initial rendered image.

[0125] The optimized gradient is obtained by optimizing the defined gradient optimization objective function. The specific optimization process has been described in detail in the above embodiment and will not be repeated here.

[0126] In some other embodiments, Figure 3 A complete flow chart of a filtering-based Monte Carlo gradient path tracing rendering image reconstruction method provided in an embodiment of the present invention is shown.

[0127] like Figure 3 As shown, the first thing is the noise input, which refers to the initial rendered image with noise, the pixel gradient map (including the gradients in the X and Y directions), and the auxiliary information map (normal map and albedo map).

[0128] This is followed by a preliminary denoising process of the initial rendered image and a gradient optimization process of the pixel gradient map, to obtain the denoised image and the optimized gradient respectively.

[0129] Next, the denoised image, optimized gradient, and auxiliary information map are used as inputs of the gradient-guided filtering algorithm. After filtering calculations at each resolution level, the reconstructed result of the initial rendered image is obtained by upsampling.

[0130] It should be noted that, at the lowest resolution level, the auxiliary information map here includes at least a normal map and an albedo map; and at other resolution levels, the auxiliary information map here includes at least a normal map, an albedo map and an upsampled reconstruction result (the intermediate filtering result outputted at a lower resolution level, that is, a low-resolution reconstruction result).

[0131] Corresponding to the filtering-based Monte Carlo gradient path tracing rendering image reconstruction method provided in the above embodiment, the present invention also provides a filtering-based Monte Carlo gradient path tracing rendering image reconstruction device.

[0132] Specifically, Figure 4 A schematic diagram of the structure of a Monte Carlo gradient path tracing rendering image reconstruction device based on filtering provided by an embodiment of the present invention is shown.

[0133] like Figure 4 As shown, the device includes: a rendering information acquisition module 410, which is used to obtain the initial rendering image obtained by Monte Carlo gradient path tracing rendering, and an auxiliary information map in the rendering process; an optimized gradient solution module 420, which is used to generate a multi-resolution level according to the initial rendering image and the auxiliary information map; a resolution-by-resolution level filtering module 430, which is used to perform filtering calculations layer by layer from the lowest resolution level to the highest resolution level according to the multi-resolution levels based on the initial rendering image, the optimized gradient and the auxiliary information map to obtain a target filtering result; wherein the target filtering result is the filtering result output by the highest resolution level, and the filtering result output by the highest resolution level is the reconstruction result of the initial rendering image.

[0134] In this embodiment, the rendering information acquisition module 410 obtains the initial rendering image obtained by Monte Carlo gradient path tracing rendering, and the auxiliary information map in the rendering process, the optimized gradient solution module 420 generates a multi-resolution level according to the initial rendering image and the auxiliary information map, and then the resolution-by-resolution level filtering module 430 performs filtering calculations layer by layer from the lowest resolution level to the highest resolution level according to the multi-resolution level based on the initial rendering image, the optimized gradient and the auxiliary information map, to obtain the target filtering result; wherein the target filtering result is the filtering result output by the highest resolution level, and the filtering result output by the highest resolution level is the reconstruction result of the initial rendering image. The device fully utilizes the gradient domain information and the auxiliary information map in the rendering process, solves the filter kernel coefficients by resolution level, and generates a high-quality and consistent denoised image (reconstruction result) at a lower sampling rate, which not only reduces the influence of noise in the gradient estimation on the reconstruction result, but also has a series of good characteristics such as consistency (progressive convergence to the reference image), meeting the requirements of high-efficiency and high-quality rendering image reconstruction.

[0135] It should be noted that the filtering-based Monte Carlo gradient path tracing rendering image reconstruction device provided in the embodiment of the present invention and the filtering-based Monte Carlo gradient path tracing rendering image reconstruction method provided in the above embodiments can correspond to each other and will not be repeated here.

[0136] Figure 5 An example of a physical structure diagram of an electronic device is shown in FIG. Figure 5 As shown, the electronic device may include: a processor 510, a communication interface 520, a memory 530 and a communication bus 540, wherein the processor 510, the communication interface 520 and the memory 530 communicate with each other through the communication bus 540. The processor 510 may call the logic instructions in the memory 530 to execute the Monte Carlo gradient path tracing rendering image reconstruction method based on filtering, the method comprising: obtaining the initial rendering image obtained by Monte Carlo gradient path tracing rendering, and the auxiliary information map in the rendering process; generating a multi-resolution level according to the initial rendering image and the auxiliary information map; based on the initial rendering image, the optimized gradient and the auxiliary information map, performing filtering calculation layer by layer from the lowest resolution level to the highest resolution level according to the multi-resolution level to obtain a target filtering result; wherein the target filtering result is the filtering result output by the highest resolution level, and the filtering result output by the highest resolution level is the reconstruction result of the initial rendering image.

[0137] In addition, the logic instructions in the above-mentioned memory 530 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0138] On the other hand, the present invention also provides a computer program product, which includes a computer program, which can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the filtering-based Monte Carlo gradient path tracing rendering image reconstruction method provided by the above-mentioned methods, and the method includes: obtaining an initial rendering image obtained by Monte Carlo gradient path tracing rendering, and an auxiliary information map in the rendering process; generating a multi-resolution level according to the initial rendering image and the auxiliary information map; based on the initial rendering image, the optimized gradient and the auxiliary information map, filtering calculation is performed layer by layer from the lowest resolution level to the highest resolution level according to the multi-resolution levels to obtain a target filtering result; wherein the target filtering result is the filtering result output by the highest resolution level, and the filtering result output by the highest resolution level is the reconstruction result of the initial rendering image.

[0139] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute the filtering-based Monte Carlo gradient path tracing rendering image reconstruction method provided by the above-mentioned methods, the method comprising: obtaining an initial rendering image obtained by Monte Carlo gradient path tracing rendering, and an auxiliary information map in the rendering process; generating a multi-resolution level according to the initial rendering image and the auxiliary information map; based on the initial rendering image, the optimized gradient and the auxiliary information map, performing filtering calculations layer by layer from the lowest resolution level to the highest resolution level according to the multi-resolution levels to obtain a target filtering result; wherein the target filtering result is the filtering result output by the highest resolution level, and the filtering result output by the highest resolution level is the reconstruction result of the initial rendering image.

[0140] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0141] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0142] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A Monte Carlo gradient path tracing rendering image reconstruction method based on filtering, characterized in that: include: Obtain the initial rendering image obtained by Monte Carlo gradient path tracing rendering, as well as the auxiliary information graph during the rendering process; According to the initial rendered image and the auxiliary information graph, solving the optimized gradient through the gradient optimization objective function, and generating a multi-resolution level; Based on the initial rendered image, the optimized gradient and the auxiliary information graph, filtering calculation is performed layer by layer from the lowest resolution layer to the highest resolution layer according to the multi-resolution layers to obtain a target filtering result; The target filtering result is a filtering result output at the highest resolution level, and the filtering result output at the highest resolution level is a reconstruction result of the initial rendering image.

2. The method for rendering image reconstruction based on filtering Monte Carlo gradient path tracing according to claim 1, characterized in that: After acquiring the initial rendered image, performing noise reduction processing on the initial rendered image specifically includes: Predefine multiple sets of fixed bilateral filtering parameters; Using each set of bilateral filtering parameters respectively to perform preliminary denoising on the initial rendered image to obtain multiple sets of corresponding denoising results; Determine a denoised image according to errors between multiple groups of denoising results and a target image; Accordingly, the step of generating multiple resolution levels according to the initial rendered image and the auxiliary information map specifically includes: A multi-resolution level is generated according to the denoised image and the auxiliary information map.

3. The method for rendering image reconstruction based on filtering Monte Carlo gradient path tracing according to claim 2, characterized in that: Generate multiple resolution levels according to the denoised image and the auxiliary information map, including: According to the denoised image and the auxiliary information graph, solving content-related upsampling parameters and downsampling parameters; Downsampling the denoised image and the auxiliary information map multiple times according to the downsampling parameters to obtain multiple resolution levels; Wherein, the auxiliary information map includes a normal map and an albedo map.

4. The method for rendering image reconstruction based on filtering Monte Carlo gradient path tracing according to claim 3, characterized in that: The filtering calculation is performed layer by layer from a low-resolution layer to a high-resolution layer according to the multi-resolution layers based on the initial rendered image, the optimized gradient and the auxiliary information map, including: At the lowest resolution level, the denoised image, optimized gradient, and auxiliary information map are used as inputs, and filtering calculation is performed through the filtering optimization objective function to obtain the intermediate filtering result; According to the calculated upsampling parameters, upsampling is performed layer by layer to a higher resolution layer of the lowest resolution layer, and the denoised image, the optimized gradient, and the updated auxiliary information map are used as inputs, and filtering calculation is performed through the filtering optimization objective function to obtain the target filtering result; The updated auxiliary information map includes a normal map, an albedo map, and an intermediate filtering result outputted at a lower resolution level, which is sampled to an image at an upper resolution level through an upsampling parameter.

5. The method for rendering image reconstruction based on filtering Monte Carlo gradient path tracing according to claim 4, characterized in that: The filtering optimization objective function is defined as follows: ； ； ； ； ； ； in, , , , express , , , The weight coefficient of Express expectations, Represents the pixels in the initial rendered image The pixel color, Represents the pixels in the initial rendered image Color expectations, Represents pixels Pixels in the neighborhood, Represents pixels Neighborhood of represents the filter kernel coefficient, Represents the pixels in the initial rendered image Color expectations, represents the variance of the rendering, represents the albedo map, represents the normal map, Indicates that the intermediate filtering result outputted at a lower resolution level is sampled to an image at the next higher resolution level through an upsampling parameter. Represents the pixels in the image after noise reduction The pixel color, Represents the pixels in the initial rendered image The pixel color.

6. The method for rendering an image by filtering and Monte Carlo gradient path tracing according to any one of claims 1 to 5, characterized in that: The gradient optimization objective function is defined as follows: ； ； ； in, represents the gradient optimization objective function, Represents the parameter used to control the strength of auxiliary information. Represents pixels, , Respectively for , The pixel-by-pixel weight of the gradient in the direction, , Respectively , The optimized gradient in the direction, , Respectively , The original gradient in direction, , Respectively for , The pixel-wise weight of the gradient in the direction.

7. A Monte Carlo gradient path tracing rendering image reconstruction device based on filtering, characterized in that: include: A rendering information acquisition module is used to obtain an initial rendering image obtained by Monte Carlo gradient path tracing rendering, as well as an auxiliary information map during the rendering process; An optimized gradient solving module, used to solve the optimized gradient through the gradient optimization objective function according to the initial rendering image and the auxiliary information graph, and generate a multi-resolution level; A resolution-by-resolution filtering module, for performing filtering calculations layer by layer from the lowest resolution layer to the highest resolution layer according to the multi-resolution layers based on the initial rendered image, the optimized gradient, and the auxiliary information graph, to obtain a target filtering result; The target filtering result is a filtering result output at the highest resolution level, and the filtering result output at the highest resolution level is a reconstruction result of the initial rendering image.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the filtering-based Monte Carlo gradient path tracing rendering image reconstruction method according to any one of claims 1 to 6 is implemented.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the filtering-based Monte Carlo gradient path tracing rendering image reconstruction method according to any one of claims 1 to 6 is implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the filtering-based Monte Carlo gradient path tracing rendering image reconstruction method according to any one of claims 1 to 6 is implemented.

Images