Filter-based monte carlo gradient path tracing rendering image reconstruction method and device

Abstract

The application provides a filter-based Monte Carlo gradient path tracking rendering image reconstruction method and device, wherein the method comprises the following steps: obtaining an initial rendering image obtained by Monte Carlo gradient path tracking rendering, and auxiliary information map in the rendering process; solving an optimized gradient by a gradient optimization objective function according to the initial rendering image and the auxiliary information map, and generating a multi-resolution level; performing filter calculation on the initial rendering image, the optimized gradient and the auxiliary information map according to the multi-resolution level from the lowest resolution level to the highest resolution level to obtain a target filter result; wherein the target filter result is a filter result output by the highest resolution level, and the filter result output by the highest resolution level is a reconstruction result of the initial rendering image. The method fully utilizes the gradient domain information and the auxiliary information map in the rendering process, solves the filter kernel coefficient at a resolution level, generates a high-quality and consistent denoising image (reconstruction result) at a low sampling rate, reduces the influence of noise in the gradient estimation on the reconstruction result, has a series of good characteristics such as consistency (gradual convergence to a reference image), and meets the efficient and high-quality rendering image reconstruction requirements.

Description

Technical Field

[0001] The present invention relates to the field of image rendering technology, and in particular to a method and device for reconstructing a rendered image using a Monte Carlo gradient path tracing method based on filtering. Background Art

[0002] Traditional Monte Carlo rendering methods generate noisy rendered images by randomly sampling light paths. Although image noise can be reduced by increasing the number of samples, it still requires a lot of computing time and resources.

[0003] To address this challenge, studies have introduced gradient path tracing methods. This method estimates adjacent pixels through gradient information while calculating image color, aiming to obtain lower variance results with fewer samples, while further improving image quality with the help of reconstruction algorithms.

[0004] Currently, reconstruction methods used for Monte Carlo gradient path tracing rendering can be roughly divided into two categories. The first category is based on the Poisson equation. This equation is constructed using estimated values ​​of the gradient and image, and solved using the least squares method to jointly remove noise from the gradient and image. This type of method generally produces high-quality, low-noise rendered images at high sampling rates. However, at lower sampling rates, its performance degrades significantly, and the image exhibits deficiencies in robustness and consistency, prone to visually undesirable discontinuities such as glitches.

[0005] The second category of methods utilizes neural networks, trained with large amounts of data to learn the prior relationship between noisy images and reference images. These methods can produce visually high-quality images at lower sampling rates and embed stronger prior knowledge, resulting in superior visual performance. However, the performance of neural network methods is highly dependent on the data distribution characteristics of the training set, and their generalization ability can be unstable, with significant performance fluctuations across different scenarios. Furthermore, due to their black-box nature, these methods lack guarantees for the consistency of their algorithmic results.

[0006] Therefore, how to solve the problem that the existing Monte Carlo gradient path rendering image reconstruction method performs poorly on low-sampling number rendering images, or cannot ensure the consistency of calculation results in different scenarios, is an important issue that needs to be urgently solved in the field of image rendering. Summary of the Invention

[0007] The present invention provides a filtering-based Monte Carlo gradient path tracing rendering image reconstruction method and device, which are used to overcome the defects of existing Monte Carlo gradient path rendering image reconstruction methods, such as poor performance on low-sampling number rendering images, or inability to ensure the consistency of calculation results in different scenarios. It can generate high-quality and consistent denoised images at a lower sampling rate, has good generalization performance, and meets the requirements of high-efficiency and high-quality image generation.

[0008] On the one hand, the present invention provides a filtering-based Monte Carlo gradient path tracing rendering image reconstruction method, including: obtaining an initial rendering image obtained by Monte Carlo gradient path tracing rendering, and an auxiliary information map during the rendering process; according to the initial rendering image and the auxiliary information map, solving the optimized gradient through the gradient optimization objective function, and generating a multi-resolution level; based on the initial rendering image, the optimized gradient and the auxiliary information map, performing filtering calculation layer by layer from the lowest resolution level to the highest resolution level according to the multi-resolution level to obtain a target filtering result; wherein, the target filtering result is the filtering result output by the highest resolution level, and the filtering result output by the highest resolution level is the reconstruction result of the initial rendering image.

[0009] Furthermore, after obtaining the initial rendered image, the initial rendered image is subjected to denoising processing, specifically including: pre-defining multiple sets of fixed bilateral filtering parameters; using each set of bilateral filtering parameters to perform preliminary denoising on the initial rendered image to obtain multiple sets of corresponding denoising results; determining the denoised image based on the errors between the multiple sets of denoising results and the target image; accordingly, the step of generating multiple resolution levels based on the initial rendered image and the auxiliary information map specifically includes: generating multiple resolution levels based on the denoised image and the auxiliary information map.

[0010] Furthermore, a multi-resolution hierarchy is generated based on the denoised image and the auxiliary information map, including: solving content-related upsampling parameters and downsampling parameters based on the denoised image and the auxiliary information map; downsampling the denoised image and the auxiliary information map multiple times according to the downsampling parameters to obtain a multi-resolution hierarchy; wherein the auxiliary information map includes a normal map and an albedo map.

[0011] Furthermore, based on the initial rendered image, the optimized gradient and the auxiliary information map, filtering calculation is performed layer by layer from the low-resolution layer to the high-resolution layer according to the multi-resolution layer, including: at the lowest resolution layer, the denoised image, the optimized gradient and the auxiliary information map are used as input, and filtering calculation is performed through the filtering optimization objective function to obtain an intermediate filtering result; according to the calculated upsampling parameters, upsampling is performed layer by layer to the higher resolution layer of the lowest resolution layer, and the denoised image, the optimized gradient and the updated auxiliary information map are used as input, and filtering calculation is performed through the filtering optimization objective function to obtain a target filtering result; wherein, the updated auxiliary information map includes a normal map, an albedo map and the intermediate filtering result outputted from the lower resolution layer is sampled to the image of the upper resolution layer through the upsampling parameters.

[0012] Furthermore, the filtering optimization objective function is defined as follows:

[0013] ;

[0014] ;

[0015] ;

[0016] ;

[0017] ;

[0018] ;

[0019] in, 、 、 、 express 、 、 、 The weight coefficient of Express expectations, Represents the pixels in the initial rendered image The pixel color, Represents the pixels in the initial rendered image Color expectations, Represents pixels Pixels in the neighborhood, Represents pixels Neighborhood, represents the filter kernel coefficient, Represents the pixels in the initial rendered image Color expectations, represents the variance of the rendering, represents the albedo map, represents the normal map, Indicates that the intermediate filtering result outputted by the lower resolution level is sampled to the image of the upper resolution level through the upsampling parameter. Represents the pixels in the image after noise reduction The pixel color, Represents the pixels in the initial rendered image The pixel color.

[0020] Furthermore, the gradient optimization objective function is defined as follows:

[0021] ;

[0022] ;

[0023] ;

[0024] in, represents the gradient optimization objective function, Represents the parameters used to control the strength of auxiliary information. represents pixels, 、 Respectively for 、 The pixel-by-pixel weight of the gradient in the direction, 、 Respectively 、 The optimized gradient in the direction, 、 Respectively 、 The original gradient in the direction, 、 Respectively for 、 Pixel-wise weighting of the gradient in the direction.

[0025] In a second aspect, the present application also provides a device for reconstructing a rendering image based on a filtering-based Monte Carlo gradient path tracing, comprising: a rendering information acquisition module, configured to acquire an initial rendering image obtained by a Monte Carlo gradient path tracing rendering, and auxiliary information maps in a rendering process; an optimized gradient solving module, configured to solve an optimized gradient by optimizing a gradient objective function according to the initial rendering image and the auxiliary information maps, and generate a multi-resolution level; and a resolution level-by-level filtering module, configured to perform filtering calculation according to the multi-resolution level from a lowest resolution level to a highest resolution level based on the initial rendering image, the optimized gradient and the auxiliary information maps, and obtain a target filtering result; wherein the target filtering result is a filtering result output by the highest resolution level, and the filtering result output by the highest resolution level is a reconstruction result of the initial rendering image.

[0026] In a third aspect, the present application also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the method for reconstructing a rendering image based on a filtering-based Monte Carlo gradient path tracing according to any one of the above aspects.

[0027] In a fourth aspect, the present application also provides a non-transitory computer readable storage medium, having a computer program stored thereon, wherein the computer program is executable by a processor to implement the method for reconstructing a rendering image based on a filtering-based Monte Carlo gradient path tracing according to any one of the above aspects.

[0028] In a fifth aspect, the present application also provides a computer program product, comprising a computer program, wherein the computer program is executable by a processor to implement the method for reconstructing a rendering image based on a filtering-based Monte Carlo gradient path tracing according to any one of the above aspects.

[0029] The present invention provides a filtering-based Monte Carlo gradient path tracing rendered image reconstruction method. The method obtains an initial rendered image obtained by Monte Carlo gradient path tracing rendering and an auxiliary information map during the rendering process. Based on the initial rendered image and the auxiliary information map, the optimized gradient is solved using a gradient optimization objective function, and a multi-resolution hierarchy is generated. Then, based on the initial rendered image, the optimized gradient, and the auxiliary information map, filtering calculations are performed layer by layer, from the lowest resolution layer to the highest resolution layer, to obtain a target filtering result. The target filtering result is the filtering result output by the highest resolution layer, and the filtering result output by the highest resolution layer is the reconstruction result of the initial rendered image. By fully utilizing the gradient domain information and the auxiliary information map during the rendering process, the method solves the filter kernel coefficients resolution by resolution layer, generating a high-quality, consistent denoised image (reconstructed result) at a low sampling rate. This method not only reduces the impact of noise in the gradient estimation on the reconstruction result, but also exhibits a series of favorable properties, such as consistency (progressive convergence to the reference image), meeting the requirements for efficient and high-quality rendered image reconstruction. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0031] Figure 1 It is a flowchart of a filtering-based Monte Carlo gradient path tracing rendering image reconstruction method provided by an embodiment of the present invention.

[0032] Figure 2 It is a schematic diagram of the general framework of the filtering-based Monte Carlo gradient path tracing rendering image reconstruction method provided by an embodiment of the present invention.

[0033] Figure 3 It is a complete flowchart of the filtering-based Monte Carlo gradient path tracing rendering image reconstruction method provided by an embodiment of the present invention.

[0034] Figure 4 It is a structural diagram of a Monte Carlo gradient path tracing rendering image reconstruction device based on filtering provided by an embodiment of the present invention.

[0035] Figure 5 It is a schematic diagram of the physical structure of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0036] To make the objectives, technical solutions, and advantages of the present invention more clear, the technical solutions of the present invention will be clearly and completely described below in conjunction with the accompanying drawings. Obviously, the embodiments described are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts shall fall within the scope of protection of the present invention.

[0037] It should be noted that there are currently two main types of reconstruction methods used for Monte Carlo gradient path rendering: the first type of method is a reconstruction algorithm based on the Poisson equation, which mainly uses the estimated values ​​of the gradient and image to construct the Poisson equation, perform the least squares solution, and jointly remove the gradient and image noise; the second type of method uses a neural network to learn the prior knowledge about the mapping between the noisy image and the reference image by training on a large amount of data.

[0038] Among them, the first type of methods can usually only generate high-quality, low-noise rendered images at high sampling numbers. If the sampling number is low, its performance will drop significantly, and it will show deficiencies in the robustness and consistency of the image, and it is easy to have visually unsatisfactory discontinuities such as glitches.

[0039] The performance of the second category of methods is highly dependent on the data distribution characteristics of the training set. Their generalization ability may be unstable and may experience large performance fluctuations in different scenarios. In addition, due to their black-box nature, these methods lack guarantees for the consistency of algorithmic results.

[0040] In view of this, the present invention proposes a Monte Carlo gradient path tracing rendering image reconstruction method based on filtering, specifically, Figure 1 A schematic flow chart of a filtering-based Monte Carlo gradient path tracing rendering image reconstruction method provided by an embodiment of the present invention is shown.

[0041] like Figure 1 As shown, the method includes steps S110-S130, and steps S110-S130 and related steps will be described in detail below.

[0042] S110, obtaining an initial rendered image obtained by Monte Carlo gradient path tracing rendering, and an auxiliary information graph during the rendering process.

[0043] Monte Carlo gradient path tracing is a physically based rendering technique that calculates images by simulating the propagation of light in a scene. This technique can produce very realistic lighting effects, including soft shadows, reflections, refractions, and global illumination.

[0044] It is understood that, through Monte Carlo gradient path tracing rendering, an initial rendered image can be obtained for subsequent reconstruction. Since Monte Carlo gradient path tracing rendering technology is relatively mature and is not the focus of the present invention, it will not be described in detail here.

[0045] It is worth mentioning that during the Monte Carlo gradient path tracing rendering process, in addition to recording the image buffer of pixel color, it is also possible to record the image buffer of pixel gradient (divided into x-direction and y-direction), and to record a variety of auxiliary information buffers and statistical information to help understand and debug the rendering results. The various auxiliary information buffers here include but are not limited to normal maps, albedo maps, material color maps, sample variance maps, etc.

[0046] The normal map shows the direction of the surface normal at each point in the scene. The albedo map, which is the ratio of light reflected by a surface to incident light, is often used to describe the color properties of a material without considering the effects of lighting. The material color map shows the inherent color of objects in the scene without any lighting effects. The sample variance map can help identify areas of high noise during rendering, as high variance means more samples are needed to reduce noise.

[0047] In a specific embodiment, the auxiliary information map includes a normal map and an albedo map.

[0048] After obtaining the initial rendering image obtained by Monte Carlo gradient path tracing rendering and the auxiliary information map during the rendering process in step S110, step S120 is further performed.

[0049] S120 , solving the optimized gradient through the gradient optimization objective function according to the initial rendered image and the auxiliary information graph, and generating multiple resolution levels.

[0050] It's easy to understand that by defining a loss function that measures the difference between images of different resolutions, and taking the initial rendered image and various auxiliary information maps as input, and solving the defined loss function using gradient descent or other optimization algorithms, we can obtain optimal and context-sensitive upsampling and downsampling parameters. The downsampling parameters guide the downsampling operation to generate multiple resolution layers, while the upsampling parameters guide the upsampling filtering to obtain the final reconstruction of the initial rendered image.

[0051] Then, according to the optimal downsampling parameters obtained by solving, the rendered initial rendering image and auxiliary information image are downsampled multiple times to obtain a series of images with different resolutions, that is, multi-resolution levels.

[0052] It is worth mentioning that since the initial rendered image and pixel gradient map generated by the Monte Carlo gradient path tracing rendering algorithm contain a lot of noise, in an optimized embodiment, after obtaining the initial rendered image, the initial rendered image is denoised using pre-defined multiple sets of fixed bilateral filtering parameters to obtain a denoised image, and the pixel gradient map is optimized to obtain an optimized gradient, that is, an optimized pixel gradient map.

[0053] After solving the optimized gradient by the gradient optimization objective function according to the initial rendered image and the auxiliary information map and generating the multi-resolution levels in step S120, step S130 is further performed.

[0054] S130, based on the initial rendered image, the optimized gradient and the auxiliary information map, filtering calculation is performed layer by layer from the lowest resolution layer to the highest resolution layer according to the multi-resolution layer to obtain a target filtering result; wherein, the target filtering result is the filtering result output by the highest resolution layer, and the filtering result output by the highest resolution layer is the reconstructed result of the initial rendered image.

[0055] It's easy to understand that a quadratic optimization problem for filtering is constructed at each resolution level. The filter kernel coefficients are calculated using the initial rendered image, the optimized gradient, and the auxiliary information maps (normal map, albedo map), as well as the intermediate filtering results output by the lower resolution level (except for the lowest resolution level, all other resolution levels have this intermediate filtering result as input). The downsampled image corresponding to that resolution level is then filtered to obtain the intermediate filtering result. The intermediate filtering result is then upsampled to a higher resolution level using the upsampling parameters corresponding to the downsampling parameters of that resolution level. The intermediate filtering result output by the lower resolution level is used as one of the auxiliary information maps to solve the quadratic optimization problem for filtering. This results in the filtering result output by the highest resolution level, which is the final reconstruction of the initial rendered image.

[0056] In this embodiment, an initial rendered image obtained by Monte Carlo gradient path tracing rendering and an auxiliary information map during the rendering process are obtained. Based on the initial rendered image and the auxiliary information map, a multi-resolution hierarchy is generated. Then, based on the initial rendered image, the optimized gradient, and the auxiliary information map, filtering calculations are performed layer by layer from the lowest resolution level to the highest resolution level to obtain a target filtering result. The target filtering result is the filtering result output by the highest resolution level, which is the reconstruction result of the initial rendered image. This method fully utilizes the gradient domain information and the auxiliary information map during the rendering process to solve the filter kernel coefficients resolution by resolution level, generating a high-quality, consistent denoised image (reconstruction result) at a low sampling rate. This method not only reduces the impact of noise in the gradient estimate on the reconstruction result, but also exhibits a series of favorable characteristics, such as consistency (progressive convergence to the reference image), meeting the requirements for efficient and high-quality rendered image reconstruction.

[0057] On the basis of the above embodiment, the preliminary denoising process of the initial rendered image will be described in detail below.

[0058] After obtaining the initial rendered image, the initial rendered image is subjected to denoising processing, specifically including: pre-defining multiple sets of fixed bilateral filtering parameters; using each set of bilateral filtering parameters to perform preliminary denoising on the initial rendered image to obtain multiple sets of corresponding denoising results; and determining the denoised image based on the errors between the multiple sets of denoising results and the target image.

[0059] Bilateral filtering is a nonlinear filtering method that smooths images while preserving edges. Bilateral filtering parameters include, but are not limited to, neighborhood diameter, color space standard deviation, and spatial domain standard deviation. The selection of these parameters significantly impacts the noise reduction effect.

[0060] The neighborhood diameter determines the size of the neighborhood considered by the filter. The color space standard deviation controls the influence of color differences between pixels. A larger color space standard deviation causes more distant color differences to be considered. The spatial domain standard deviation controls the influence of distance between pixels. A larger spatial domain standard deviation causes more distant pixels to also affect the current pixel.

[0061] It's easy to understand that this embodiment predefines multiple sets of fixed bilateral filtering parameters and applies these different, fixed sets of bilateral filtering parameters to the initial rendered image, resulting in multiple corresponding denoising results. A preset loss function is then used to calculate the error between each denoising result and the target image. By comparing the error of each pixel in the different denoising results, the result with the smallest error is selected. Thus, based on the pixel-by-pixel denoising result with the smallest error, a linear combination relaxation operation is performed to ensure continuity, resulting in the optimal denoising result, i.e., the denoised image.

[0062] The number of the predefined sets of bilateral filtering parameters can be set according to actual needs, which is not specifically limited herein. For example, in a specific embodiment, three sets of bilateral filtering parameters are predefined: a neighborhood diameter of 5, a color space standard deviation of 75, and a spatial domain standard deviation of 75; a neighborhood diameter of 9, a color space standard deviation of 100, and a spatial domain standard deviation of 100; and a neighborhood diameter of 15, a color space standard deviation of 150, and a spatial domain standard deviation of 150.

[0063] The preset loss function can be set according to actual needs, which is not specifically limited herein.

[0064] For example, in a specific embodiment, the preset loss function is a SURE loss function. The SURE loss function can be used to optimize the parameters of the denoising algorithm without knowing the true noise-free image.

[0065] After the initial rendered image is denoised, a multi-resolution level is generated according to the initial rendered image and the auxiliary information map, including: generating the multi-resolution level according to the denoised image and the auxiliary information map.

[0066] It should be noted that when the initial rendered image is preliminarily denoised, the auxiliary information map including the normal and albedo can also be used to optimize the denoising process. Specifically, on the one hand, the bilateral filtering parameters can be dynamically adjusted according to the information of the normal map and the albedo map, for example, the value of the bilateral filtering parameters can be reduced in the edge area (large normal change); on the other hand, when calculating the error, the weight coefficient of some areas can be increased by combining the information of the normal map and the albedo map, so that the error of some areas is more important.

[0067] In the embodiment, a plurality of sets of fixed bilateral filtering parameters are predefined, and each set of bilateral filtering parameters is used to preliminarily denoise the initial rendered image to obtain a plurality of corresponding denoising results. Then, the denoised image is determined according to the errors between the plurality of denoising results and the target image, and the multi-resolution level is generated according to the denoised image and the auxiliary information map. Thus, based on the initial rendered image, the optimized gradient, and the auxiliary information map, the filter kernel coefficients are calculated layer by layer according to the multi-resolution level from the lowest resolution level to the highest resolution level, and the target filtering result is obtained. This method fully utilizes the gradient domain information and the auxiliary information map in the rendering process to solve the filter kernel coefficients layer by layer at a low sampling rate to generate a high-quality and consistent denoised image (reconstruction result), which not only reduces the influence of noise on the reconstruction result in the gradient estimation, but also has a series of good characteristics such as consistency (gradual convergence to the reference image), thereby meeting the demand for efficient and high-quality rendering image reconstruction.

[0068] Based on the above embodiment, the process of generating multiple resolution levels will be described in detail below.

[0069] Generating a multi-resolution hierarchy based on the denoised image and the auxiliary information map includes: solving content-related upsampling parameters and downsampling parameters based on the denoised image and the auxiliary information map; downsampling the denoised image and the auxiliary information map multiple times according to the downsampling parameters to obtain the multi-resolution hierarchy; wherein the auxiliary information map includes a normal map and an albedo map.

[0070] Understandably, in Monte Carlo gradient path tracing rendering, downsampling effectively increases the number of effective samples per pixel while reducing variance. This means that the smaller variance after downsampling to a lower resolution is more friendly to various noise reduction algorithms. However, simple averaging downsampling can also lead to loss of detail, and fine structures in the image cannot be restored after averaging during the downsampling process.

[0071] In this regard, this embodiment adopts a content-related up-sampling and down-sampling generation method. (i.e., denoised image and auxiliary information map) and downsampled map , downsampled graph Each pixel in the image is the original image The linear combination of the two pixels in , for details, please refer to the following formula (1).

[0072] (1).

[0073] In formula (1), Represents pixels in the downsampled image The pixel color, Represents pixels in the original image The weight of 、 Represents the pixels in the original image , pixels The pixel color.

[0074] The optimization objective of the content-dependent up- and down-sampling algorithm is as follows (2).

[0075] (2).

[0076] In formula (2), represents the original image (i.e. the denoised image and auxiliary information image), Represents the downsampled graph The upsampled image obtained by upsampling.

[0077] The reconstruction error is minimized by jointly optimizing the linear combination parameters of upsampling and downsampling through equations (1) and (2), and the upsampling parameters and downsampling parameters are obtained.

[0078] Among them, the upsampling graph It can be obtained by linear interpolation from four pixels at the corresponding position in the downsampled image, or by selecting a pixel from the neighborhood of the corresponding position in the downsampled image. The specific selection strategy is obtained by jointly optimizing with the downsampling parameters.

[0079] It should be noted that since the rendering results (initial rendered images) usually have a lot of noise, using the rendering results to directly calculate the upsampling and downsampling parameters can easily mistake the noise for fine structures. Therefore, the reference image actually used in the calculation is the result after preliminary denoising, that is, the denoised image.

[0080] After solving the upsampling and downsampling parameters, the denoised image and auxiliary information map are downsampled according to the downsampling parameters. Each downsampling will generate a new lower resolution image version. The same downsampling operation is continuously applied to the low-resolution image obtained by the previous downsampling operation until the required minimum resolution or number of layers is reached. All images of different resolutions are combined to form a multi-resolution hierarchical structure. Each layer of this structure represents the performance of the denoised image and auxiliary information map at a specific resolution.

[0081] In this way, multiple resolution levels can be generated.

[0082] Furthermore, a quadratic filtering optimization problem is constructed at each resolution level. The filter kernel coefficients are calculated using the denoised image, the optimized gradient, the auxiliary information map, and the intermediate filtering result output from the next lower resolution level (except for the lowest resolution level, all resolution levels have this intermediate filtering result as input). The downsampled image corresponding to that resolution level is filtered to obtain the intermediate filtering result. The intermediate filtering result is then upsampled to a higher resolution level using the upsampling parameters corresponding to the downsampling parameters of that resolution level. The intermediate filtering result output from the next lower resolution level is used as one of the auxiliary information maps, and the quadratic filtering optimization problem is solved. The result is the filtering result output at the highest resolution level, which is the final reconstruction of the initial rendered image.

[0083] In this embodiment, content-dependent upsampling and downsampling parameters are calculated based on the denoised image and auxiliary information map. The denoised image and auxiliary information map are then downsampled multiple times according to the downsampling parameters to obtain multiple resolution levels. Then, based on the initial rendered image, the optimized gradient, and the auxiliary information map, filtering calculations are performed layer by layer, from the lowest resolution level to the highest resolution level, to obtain a reconstruction of the initial rendered image. This method fully utilizes the gradient domain information and auxiliary information map during the rendering process to calculate the filter kernel coefficients at each resolution level, generating a high-quality, consistent denoised image (reconstruction result) at a low sampling rate. This method not only mitigates the impact of noise in the gradient estimate on the reconstruction result, but also exhibits a series of favorable properties, such as consistency (progressive convergence to the reference image), meeting the requirements for efficient and high-quality rendered image reconstruction.

[0084] On the basis of the above embodiment, the process of optimizing the pixel gradient map obtained by rendering will be further described in detail below.

[0085] It is understandable that the gradient (pixel gradient map) generated by the Monte Carlo gradient path tracing rendering algorithm is noisy, which makes the gradient obtained by this set of renderings ( 、 direction) is not the gradient of any image. This embodiment defines the "legitimacy" of the gradient here, requiring the divergence of the gradient field to be , that is, the gradient The requirement is to satisfy the following formula (3):

[0086] (3).

[0087] In formula (3), Indicates that the image after denoising is located at coordinates The pixels in The gradient in direction, Indicates that the image after denoising is located at coordinates exist The gradient in direction, Indicates that the image after denoising is located at coordinates The pixels in The gradient in direction, Indicates that the image after denoising is located at coordinates The pixels in Gradient in direction.

[0088] In this embodiment, the gradient that satisfies the above formula (3) is a legal gradient, and the legal gradient can reconstruct the entire image based on any pixel.

[0089] Under the conditional constraints of legality (Formula (1) above), a gradient optimization objective function is constructed to optimize the gradient. The optimization mainly considers the gradient itself and the auxiliary information to obtain the optimized gradient.

[0090] In terms of the gradient itself, the optimization objective function is formulated based on the gradient itself , please refer to the following formula (4) for details.

[0091] (4).

[0092] In formula (4), represents pixels, and They are respectively for direction, The pixel-by-pixel weight of the gradient in the direction, 、 Respectively 、 The optimized gradient in the direction, 、 Respectively 、 The original gradient in the direction.

[0093] and The definition of can be found in the following formulas (5) and (6).

[0094] (5).

[0095] (6).

[0096] In formulas (5)-(6), is a predefined constant, A decimal number used to avoid denominators being zero.

[0097] The design of the gradient itself can ensure that the optimization process tends to retain the parts with larger and smaller absolute values ​​of the gradient, which represent the boundary and the interior of the region respectively.

[0098] In terms of auxiliary information, the boundary and internal information provided by the auxiliary information are used to formulate the objective function , please refer to the following formula (7) for details.

[0099] (7).

[0100] In formula (7), we also set 、 Pixel-wise weight of the gradient in the direction and According to the auxiliary information map, the weight of the gradient in optimization is controlled.

[0101] And The definitions can be seen in the following formulas (8)-(9).

[0102] (8).

[0103] (9).

[0104] In the formulas (8)-(9), , Indicate the pixel gradient of each auxiliary information map. The pixel-by-pixel weight And Requires the gradient inside the image region to be as close to 0 as possible, so as to achieve the removal of noise in the gradient domain.

[0105] According to the above formulas (4)-(9), the gradient optimization objective function can be defined, which can be seen in the following formula (10).

[0106] (10).

[0107] In the formula (10), Is a parameter for controlling the degree of auxiliary information force, and will converge to 0 as the number of input samples increases, to ensure the consistency of gradient optimization.

[0108] Based on formula (10), the optimized gradient , This operation can make the rendering image reconstruction algorithm more robust and avoid producing obvious visual glitches.

[0109] In this embodiment, the gradient obtained by rendering is optimized by the gradient optimization objective function to obtain the optimized gradient, and then based on the initial rendering image, the optimized gradient and the auxiliary information map, the filtering calculation is performed layer by layer from the lowest resolution level to the highest resolution level according to the multi-resolution level, to obtain the reconstruction result of the initial rendering image. This method fully utilizes the gradient domain information and auxiliary information map in the rendering process, and solves the filtering kernel coefficient at each resolution level, to generate a high-quality and consistent denoising image (reconstruction result) at a lower sampling rate, which not only reduces the influence of noise in gradient estimation on the reconstruction result, but also has a series of good characteristics such as consistency (gradual convergence to the reference image), to meet the efficient and high-quality rendering image reconstruction requirements.

[0110] On the basis of the above embodiment, further, the process of layer-by-layer filtering reconstruction will be described in detail.

[0111] Based on the initial rendered image, optimized gradient and auxiliary information map, filtering calculation is performed layer by layer from low resolution layer to high resolution layer according to multiple resolution layers, including: at the lowest resolution layer, using the denoised image, optimized gradient and auxiliary information map as input, filtering calculation is performed through the filtering optimization objective function to obtain an intermediate filtering result; according to the calculated upsampling parameters, upsampling is performed layer by layer to the higher resolution layer of the lowest resolution layer, using the denoised image, optimized gradient and updated auxiliary information map as input, filtering calculation is performed through the filtering optimization objective function to obtain the target filtering result; wherein, the updated auxiliary information map includes the normal map, albedo map and the intermediate filtering result outputted from the lower resolution layer, which is sampled to the image of the upper resolution layer through the upsampling parameters.

[0112] It is understandable that filtering is a very important and widely used method in Monte Carlo gradient path tracing rendering noise reduction. This method uses a linear combination of each pixel neighborhood to calculate the denoised result, that is, for pixels , using a series of filter kernel coefficients In its neighborhood The inner linear summation is used to obtain the filtering result, which can be specifically referred to in the following formula (11). The filtering can obtain a smooth output result while ensuring consistency.

[0113] (11).

[0114] This embodiment proposes a gradient-guided filter kernel calculation algorithm based on the filtering method, using the optimized gradient and auxiliary information Figure 1 By calculating the filter kernels that converge consistently, we can robustly reconstruct the gradient path tracing rendering results. The algorithm obtains the filter kernel coefficients by optimizing the filter objective function. The filter optimization objective function is divided into three parts: the gradient guidance part, the auxiliary information autoregressive part, and the preliminary noise reduction regression part.

[0115] In the gradient guidance part, we start from the perspective of minimizing the mean square error of the filtering results, as shown in the following formula (12).

[0116] (12).

[0117] In formula (12), Represents the pixels in the initial rendered image The pixel color, Represents the pixels in the initial rendered image Color expectations, Represents pixels Pixels in the neighborhood, Represents pixels Neighborhood, represents the filter kernel coefficient, Represents the pixels in the initial rendered image Color expectations, Indicates the variance of the rendering.

[0118] According to formula (12), the optimal filter kernel coefficient can be obtained in a statistical sense , however, the expected image pixel color 、 It cannot be obtained, so this embodiment uses the optimized gradient to estimate In addition, the variance The sample variance estimate obtained by rendering can be used directly.

[0119] In the auxiliary information autoregressive part, there is a restriction on the optimization target, which requires that for each auxiliary information map (normal map, albedo map and reconstruction result sampled from a lower resolution level (i.e., intermediate filtering result)), the difference between the filter kernel and the original input should be as small as possible after applying the filter kernel. For details, please refer to the following equations (13)-(15).

[0120] (13).

[0121] (14).

[0122] (15).

[0123] In formulas (13)-(15), represents the albedo map, represents the normal map, Indicates that the intermediate filtering result outputted by the lower resolution layer is sampled to the image of the upper resolution layer through the upsampling parameters.

[0124] It should be noted that if the current resolution level is the lowest resolution level, Does not exist, can take value 0.

[0125] In the initial denoising regression part, the input image (initial rendered image) is used to regress the initial denoising result (denoised image) for further constraints. For details, see the following formula (16).

[0126] (16).

[0127] In formula (16), Represents the pixels in the image after noise reduction The pixel color, Represents the pixels in the initial rendered image The pixel color.

[0128] Based on the above equations (12)-(16), the filtering optimization objective function can be defined, as shown in the following equation (17).

[0129] (17).

[0130] By solving The minimum value of can be used to obtain the filter kernel coefficient and complete the filtering.

[0131] Then, the obtained filter kernel coefficient Substituting into formula (11), we can get the filtering result , which is the intermediate filtering result of each resolution level, that is, the reconstruction result of the intermediate rendered image.

[0132] In this embodiment, a filtering calculation is performed at the lowest resolution level using the denoised image, optimized gradients, and an auxiliary information map as inputs, and a filtering optimization objective function is used to obtain an intermediate filtering result. Upsampling is then performed layer by layer to higher resolution levels below the lowest resolution level according to the calculated upsampling parameters. Using the denoised image, optimized gradients, and updated auxiliary information map as inputs, a filtering calculation is performed using the filtering optimization objective function to obtain a target filtering result. This method fully utilizes the gradient domain information and auxiliary information map during the rendering process, solving the filter kernel coefficients at each resolution level, and generating a high-quality, consistent denoised image (reconstruction result) at a low sampling rate. This method not only mitigates the impact of noise in the gradient estimate on the reconstruction result, but also exhibits a series of favorable properties, such as consistency (progressive convergence to the reference image), meeting the requirements for efficient and high-quality rendered image reconstruction.

[0133] Additionally, in some embodiments, Figure 2 The schematic diagram shows the general framework of the filtering-based Monte Carlo gradient path tracing rendering image reconstruction method provided by an embodiment of the present invention.

[0134] like Figure 2 As shown, the filtering-based Monte Carlo gradient path tracing rendered image reconstruction method provided in an embodiment of the present invention includes a downsampling process and an upsampling process. The downsampling input includes an image (the initial rendered image), a gradient (the rendered pixel gradient map), preliminary denoising (the denoised image), and auxiliary information maps (normal map and albedo map), generating multiple resolution levels. The denoised image is obtained by performing preliminary denoising on the initial rendered image, which has been described in detail in the above embodiments and will not be repeated here.

[0135] During upsampling, the optimized gradient (optimized gradient), preliminary denoising result (denoising image), auxiliary information map (normal map, albedo map) and upsampled guidance map (intermediate filtering result / reconstruction result output by the lower resolution layer) are used as input to execute the gradient-guided filtering algorithm provided by the embodiment of the present invention. Filtering calculations are performed layer by layer from the lowest resolution layer to the highest resolution layer. The filtering result output by the highest resolution layer is the final reconstruction result of the initial rendered image.

[0136] The optimized gradient is obtained by optimizing the defined gradient optimization objective function. The specific optimization process has been described in detail in the above embodiment and will not be repeated here.

[0137] In some other embodiments, Figure 3 A complete flow chart of a filtering-based Monte Carlo gradient path tracing rendering image reconstruction method provided by an embodiment of the present invention is shown.

[0138] like Figure 3 As shown in the figure, the first thing is the noise input, which refers to the initial rendered image with noise, the pixel gradient map (including the gradient in the X and Y directions), and the auxiliary information map (normal map and albedo map).

[0139] This is followed by a preliminary denoising process of the initial rendered image and a gradient optimization process of the pixel gradient map, to obtain the denoised image and the optimized gradient respectively.

[0140] Next, the denoised image, optimized gradient, and auxiliary information map are used as inputs to the gradient-guided filtering algorithm. After filtering calculations at each resolution level, the upsampled image is used to obtain the reconstruction result of the initial rendered image.

[0141] It should be noted that at the lowest resolution level, the auxiliary information map here includes at least the normal map and the albedo map; and at other resolution levels, the auxiliary information map here includes at least the normal map, the albedo map and the upsampled reconstruction result (the intermediate filtering result output by the lower resolution level, that is, the low-resolution reconstruction result).

[0142] Corresponding to the filtering-based Monte Carlo gradient path tracing rendering image reconstruction method provided in the above embodiment, the present invention also provides a filtering-based Monte Carlo gradient path tracing rendering image reconstruction device.

[0143] Specifically, Figure 4 A schematic structural diagram of a Monte Carlo gradient path tracing rendering image reconstruction device based on filtering provided by an embodiment of the present invention is shown.

[0144] like Figure 4As shown, the device includes: a rendering information acquisition module 410, which is used to obtain the initial rendering image obtained by Monte Carlo gradient path tracing rendering, and an auxiliary information map in the rendering process; an optimized gradient solution module 420, which is used to generate a multi-resolution level according to the initial rendering image and the auxiliary information map; a resolution-by-resolution level filtering module 430, which is used to perform filtering calculations layer by layer from the lowest resolution level to the highest resolution level according to the multi-resolution levels based on the initial rendering image, the optimized gradient and the auxiliary information map to obtain a target filtering result; wherein, the target filtering result is the filtering result output by the highest resolution level, and the filtering result output by the highest resolution level is the reconstruction result of the initial rendering image.

[0145] In this embodiment, the rendering information acquisition module 410 obtains an initial rendered image obtained by Monte Carlo gradient path tracing rendering, as well as an auxiliary information map used during the rendering process. The optimized gradient solution module 420 generates multiple resolution levels based on the initial rendered image and the auxiliary information map. The resolution-by-resolution filtering module 430 then performs filtering calculations layer by layer, from the lowest resolution level to the highest resolution level, based on the initial rendered image, the optimized gradient, and the auxiliary information map, to obtain a target filtering result. The target filtering result is the filtering result output by the highest resolution level, which is the reconstruction result of the initial rendered image. By fully utilizing the gradient domain information and the auxiliary information map used during the rendering process, this device solves the filter kernel coefficients resolution by resolution level, generating a high-quality, consistent denoised image (reconstructed result) at a low sampling rate. This not only mitigates the impact of noise in the gradient estimation on the reconstruction result, but also exhibits a series of favorable properties, such as consistency (progressive convergence to the reference image), meeting the requirements for efficient and high-quality rendered image reconstruction.

[0146] It should be noted that the filtering-based Monte Carlo gradient path tracing rendering image reconstruction device provided in the embodiment of the present invention and the filtering-based Monte Carlo gradient path tracing rendering image reconstruction method provided in the above embodiments can be referred to each other, and will not be repeated here.

[0147] Figure 5 An example of a physical structure diagram of an electronic device is shown below. Figure 5As shown, the electronic device may include: a processor 510, a communication interface 520, a memory 530, and a communication bus 540, wherein the processor 510, the communication interface 520, and the memory 530 communicate with each other via the communication bus 540. The processor 510 may call logic instructions in the memory 530 to execute a filtering-based Monte Carlo gradient path tracing rendering image reconstruction method, which includes: obtaining an initial rendered image obtained by Monte Carlo gradient path tracing rendering and an auxiliary information map during the rendering process; generating multiple resolution levels based on the initial rendered image and the auxiliary information map; and performing filtering calculations layer by layer from the lowest resolution level to the highest resolution level according to the multiple resolution levels based on the initial rendered image, the optimized gradient, and the auxiliary information map to obtain a target filtering result; wherein the target filtering result is the filtering result output by the highest resolution level, and the filtering result output by the highest resolution level is the reconstruction result of the initial rendered image.

[0148] Furthermore, the logic instructions in the aforementioned memory 530 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the portion that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions for causing a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as a USB flash drive, a mobile hard drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0149] On the other hand, the present invention also provides a computer program product, which includes a computer program, which can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the filtering-based Monte Carlo gradient path tracing rendering image reconstruction method provided by the above methods, the method including: obtaining an initial rendering image obtained by Monte Carlo gradient path tracing rendering, and an auxiliary information map during the rendering process; generating a multi-resolution level based on the initial rendering image and the auxiliary information map; based on the initial rendering image, the optimized gradient and the auxiliary information map, performing filtering calculations layer by layer from the lowest resolution level to the highest resolution level according to the multi-resolution levels to obtain a target filtering result; wherein, the target filtering result is the filtering result output by the highest resolution level, and the filtering result output by the highest resolution level is the reconstruction result of the initial rendering image.

[0150] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute the filtering-based Monte Carlo gradient path tracing rendering image reconstruction method provided by the above-mentioned methods, the method comprising: obtaining an initial rendered image obtained by Monte Carlo gradient path tracing rendering, and an auxiliary information map during the rendering process; generating a multi-resolution level based on the initial rendered image and the auxiliary information map; performing filtering calculations layer by layer from the lowest resolution level to the highest resolution level according to the multi-resolution levels based on the initial rendered image, the optimized gradient and the auxiliary information map to obtain a target filtering result; wherein the target filtering result is the filtering result output by the highest resolution level, and the filtering result output by the highest resolution level is the reconstruction result of the initial rendered image.

[0151] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, i.e., they may be located in one location or distributed across multiple network units. Some or all of the modules may be selected based on actual needs to achieve the objectives of the present embodiment. Persons of ordinary skill in the art will be able to understand and implement the present invention without inventive effort.

[0152] Through the above description of the embodiments, those skilled in the art will clearly understand that each embodiment can be implemented using software plus a necessary general-purpose hardware platform, or of course, hardware. Based on this understanding, the essence of the above technical solution, or the portion that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a magnetic disk, or an optical disk, and includes a number of instructions for causing a computer device (such as a personal computer, server, or network device) to execute the methods described in each embodiment or certain portions of the embodiments.

[0153] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present invention.

Claims

1. A Monte Carlo gradient path tracing rendering image reconstruction method based on filtering, characterized in that: include: Obtain the initial rendered image obtained by Monte Carlo gradient path tracing rendering, as well as the auxiliary information graph during the rendering process; According to the initial rendered image and the auxiliary information map, solving the optimized gradient through the gradient optimization objective function and generating a multi-resolution level; Based on the initial rendered image, the optimized gradient, and the auxiliary information map, filtering calculation is performed layer by layer from the lowest resolution layer to the highest resolution layer according to the multi-resolution layer to obtain a target filtering result; The target filtering result is the filtering result outputted at the highest resolution level, and the filtering result outputted at the highest resolution level is the reconstruction result of the initial rendered image; The filtering calculation is performed layer by layer from a low-resolution layer to a high-resolution layer according to the multi-resolution layers based on the initial rendered image, the optimized gradient, and the auxiliary information map, including: At the lowest resolution level, the denoised image, optimized gradient, and auxiliary information map are used as input, and filtering calculation is performed through the filtering optimization objective function to obtain the intermediate filtering result; According to the calculated upsampling parameters, upsampling is performed layer by layer to the higher resolution layer of the lowest resolution layer. The denoised image, optimized gradient, and updated auxiliary information map are used as inputs, and filtering calculation is performed through the filtering optimization objective function to obtain the target filtering result. The denoised image is the denoised image of the initial rendered image, and the updated auxiliary information map includes a normal map, an albedo map, and an intermediate filtering result output at a lower resolution level, which is sampled to an image at an upper resolution level through an upsampling parameter; The filtering optimization objective function is defined as follows: Among them, λ a ,λ n ,λ up ,λ d express The weight coefficient of represents the expected color, I(p) represents the pixel color of pixel p in the initial rendered image, μ p represents the expected color of pixel p in the initial rendered image, q represents the pixels in the neighborhood of pixel p, and N p represents the neighborhood of pixel p, w q Represents the filter kernel coefficient, μ q represents the desired color of pixel q in the initial rendered image, v q represents the variance of rendering, F albedo (p) represents the albedo map, F normal (p) represents the normal map, F up (p) represents the intermediate filtering result outputted by the lower resolution layer, which is sampled to the image of the next higher resolution layer through the upsampling parameter. d I(p) represents the pixel color of pixel p in the denoised image, and I(q) represents the pixel color of pixel q in the initial rendered image; The gradient optimization objective function is defined as follows: in, represents the gradient optimization objective function, λ f Represents the parameter used to control the strength of auxiliary information, p represents pixels, m x (p), m y (p) represents the pixel-by-pixel weights for the gradient in the x and y directions, Represent the optimized gradients in the x and y directions, I dx (p), I dy (p) represents the original gradient in the x and y directions, respectively, a x (p), a y (p) represents the pixel-by-pixel weights for the gradient in the x and y directions, respectively.

2. The filtering-based Monte Carlo gradient path tracing rendering image reconstruction method according to claim 1, characterized in that: After obtaining the initial rendered image, performing noise reduction processing on the initial rendered image specifically includes: Predefine multiple sets of fixed bilateral filtering parameters; Performing preliminary denoising on the initial rendered image using each set of bilateral filtering parameters respectively, to obtain multiple sets of corresponding denoising results; Determine a denoised image based on errors between multiple sets of denoising results and the target image; Accordingly, the step of generating multiple resolution levels according to the initial rendered image and the auxiliary information map specifically includes: A multi-resolution hierarchy is generated based on the denoised image and the auxiliary information map.

3. The filtering-based Monte Carlo gradient path tracing rendering image reconstruction method according to claim 2, characterized in that: Generate multiple resolution levels based on the denoised image and the auxiliary information map, including: Determining content-related upsampling parameters and downsampling parameters based on the denoised image and the auxiliary information graph; Downsampling the denoised image and the auxiliary information map multiple times according to the downsampling parameters to obtain multiple resolution levels; Wherein, the auxiliary information map includes a normal map and an albedo map.

4. A filtering-based Monte Carlo gradient path tracing rendering image reconstruction device, applying the filtering-based Monte Carlo gradient path tracing rendering image reconstruction method according to any one of claims 1 to 3, characterized in that: include: A rendering information acquisition module is used to obtain the initial rendering image obtained by Monte Carlo gradient path tracing rendering, as well as auxiliary information images during the rendering process; An optimized gradient solving module is used to solve the optimized gradient through the gradient optimization objective function according to the initial rendered image and the auxiliary information map, and generate a multi-resolution layer; A resolution-by-resolution filtering module is configured to perform filtering calculations layer by layer from the lowest resolution layer to the highest resolution layer based on the initial rendered image, the optimized gradient, and the auxiliary information map to obtain a target filtering result; The target filtering result is the filtering result outputted at the highest resolution level, and the filtering result outputted at the highest resolution level is the reconstruction result of the initial rendering image.

5. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the filtering-based Monte Carlo gradient path tracing rendering image reconstruction method according to any one of claims 1 to 3 is implemented.

6. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the filtering-based Monte Carlo gradient path tracing rendering image reconstruction method according to any one of claims 1 to 3 is implemented.

7. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the filtering-based Monte Carlo gradient path tracing rendering image reconstruction method according to any one of claims 1 to 3 is implemented.

Images