A Monte Carlo rendering denoising method and system based on temporal multi-component hierarchical features
By decomposing Monte Carlo rendered images into multiple components and combining temporal features with filtering, the problems of severe noise and ghosting in Monte Carlo rendering are solved, achieving efficient real-time denoising.
Patent Information
- Application Number
- CN202410286566.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-03-13
- Publication Date
- 2026-11-17
- Estimated Expiration
- 2044-03-13
AI Technical Summary
Existing Monte Carlo rendering methods suffer from severe noise in real-time rendering due to low sampling rates. Traditional methods also exhibit ghosting or excessive blurring when handling glossy and specular reflections, and neural networks are computationally expensive and difficult to apply in real time.
A Monte Carlo rendering denoising method based on temporal multi-component hierarchical features is adopted. By dividing the image into diffuse reflection, glossy reflection and specular reflection components, a hierarchical feature extraction network and a temporal information fusion module are used for denoising. Spatial filtering and temporal filtering are combined to process the noise of each component.
It achieves high-quality denoising at low sampling rates, avoids ghosting issues, and effectively utilizes inter-frame information, improving the quality and efficiency of rendered images.
Smart Images

Figure CN118052734B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of 3D rendering post-processing denoising technology, and in particular to a Monte Carlo rendering denoising method and system based on temporal multi-component hierarchical features. Background Technology
[0002] The statements in this section merely refer to the background art related to this invention and do not necessarily constitute prior art.
[0003] Monte Carlo integration, thanks to its flexibility and universality, is widely used in fields such as realistic image rendering, computer animation generation, and video game production. However, this method samples the scene using random ray paths, and obtaining a convergent result often requires a large number of samples, which can take several hours for complex scenes. In real-time rendering, at least 30 frames per second are typically required, necessitating Monte Carlo rendering to emit ray samples at an extremely low sampling rate per pixel, usually 1 spp (samples per pixel). While this approach ensures real-time efficiency, the excessively low sample count prevents Monte Carlo integration from converging, introducing significant noise into the rendered image and resulting in a poor visual experience. Therefore, the key to successful real-time Monte Carlo rendering lies in combining it with efficient post-processing denoising methods.
[0004] Denoising real-time Monte Carlo rendered images with extremely low sampling rates requires satisfying both image quality and processing speed. The inventors discovered that current methods typically compute motion vectors to construct a temporal mapping between two consecutive frames to compensate for the insufficient sampling rate of a single frame, while simultaneously using neural networks to generate pixel-by-pixel filtering kernels to achieve high-quality denoising. However, traditional motion vectors map the motion relationship of the scene model surface between two consecutive frames, failing to properly handle virtual images caused by glossy or specular reflections. This results in ghosting or excessive blurring in specular reflection areas when objects or the camera move rapidly. Furthermore, while conventional neural networks can deliver high-quality denoising, their high computational cost makes them unsuitable for real-time Monte Carlo rendering denoising applications. Summary of the Invention
[0005] To address the shortcomings of existing technologies, this invention provides a Monte Carlo rendering denoising method and system based on temporal multi-component hierarchical features. The image rendered by Monte Carlo at 1spp is divided into multiple components according to the different behaviors of the light samples when they first hit the scene. At the same time, the corresponding temporal reliable motion vectors are introduced, and a hierarchical feature extraction network and a temporal information fusion module are designed to achieve high-quality denoising while ensuring real-time performance.
[0006] On the one hand, a Monte Carlo rendering denoising method based on temporal multi-component hierarchical features is provided;
[0007] Monte Carlo rendering denoising methods based on temporal multi-component hierarchical features include:
[0008] Obtain the Monte Carlo render image to be denoised;
[0009] The Monte Carlo rendering image to be denoised is divided into diffuse reflection components, glossy reflection components, and specular reflection components.
[0010] The diffuse reflection component, gloss reflection component, and specular reflection component are input into the trained denoising model, and the final denoised image is output.
[0011] The trained denoising model is used to extract spatial features for each component and perform spatial filtering on the extracted spatial features. If the current image to be denoised belongs to the first frame, the spatial filtering result is used as the denoised image of the current component. If the current image to be denoised does not belong to the first frame, the current image to be denoised is subjected to temporal filtering based on the spatial filtering result and temporal features of the previous frame, and the temporal filtering result is used as the denoised image of the current component. The denoised images of the three components are fused to obtain the final denoised image.
[0012] On the other hand, a Monte Carlo rendering denoising system based on temporal multi-component hierarchical features is provided;
[0013] A Monte Carlo rendering denoising system based on temporal multi-component hierarchical features includes:
[0014] The acquisition module is configured to acquire the Monte Carlo rendered image to be denoised;
[0015] The partitioning module is configured to partition the Monte Carlo rendered image to be denoised into diffuse reflection components, glossy reflection components, and specular reflection components.
[0016] The denoising module is configured to input the diffuse reflection component, gloss reflection component, and specular reflection component into the trained denoising model and output the final denoised image.
[0017] The trained denoising model is used to extract spatial features for each component and perform spatial filtering on the extracted spatial features. If the current image to be denoised belongs to the first frame, the spatial filtering result is used as the denoised image of the current component. If the current image to be denoised does not belong to the first frame, the current image to be denoised is subjected to temporal filtering based on the spatial filtering result and temporal features of the previous frame, and the temporal filtering result is used as the denoised image of the current component. The denoised images of the three components are fused to obtain the final denoised image.
[0018] Furthermore, an electronic device is also provided, including:
[0019] Memory, used for non-transitory storage of computer-readable instructions; and
[0020] Processor, for executing the computer-readable instructions,
[0021] When the computer-readable instructions are executed by the processor, they perform the method described in the first aspect above.
[0022] In another aspect, a storage medium is also provided for non-transitory storage of computer-readable instructions, wherein when the non-transitory computer-readable instructions are executed by a computer, the instructions of the method described in the first aspect are executed.
[0023] In another aspect, a computer program product is also provided, including a computer program that, when run on one or more processors, is used to implement the method described in the first aspect above.
[0024] The above technical solution has the following advantages or beneficial effects:
[0025] (1) The scheme described in this disclosure provides a Monte Carlo rendering denoising method based on temporal multi-component hierarchical features. Its hierarchical feature extraction network extracts feature vectors for diffuse reflection component, gloss reflection component and specular reflection component generated by 1spp Monte Carlo rendering respectively. The spatial filtering module performs spatial denoising on the noisy image of each component at multiple resolution scales. The temporal filtering module fuses the current frame and the previous frame image of each component for denoising. Finally, the denoising results of the multiple components are fused together to obtain a high-quality denoised image.
[0026] (2) The hierarchical feature extraction network used in the above scheme can extract multi-component data features more effectively, thereby generating better pixel-by-pixel filtering kernels and linear fusion parameters, making efficient use of the limited samples in noisy images and auxiliary information, and achieving better denoising effect.
[0027] (3) The proposed scheme is based on a time-domain multi-component denoising structure, which introduces a time-domain reliable motion vector for the corresponding component, making more accurate use of the reusable information between frames, making up for the lack of single-frame samples, and avoiding the ghosting problem caused by the time-domain information mismatch caused by traditional motion vectors. Attached Figure Description
[0028] The accompanying drawings, which form part of this invention, are used to provide a further understanding of the invention. The illustrative embodiments of the invention and their descriptions are used to explain the invention and do not constitute an improper limitation of the invention.
[0029] Figure 1 This is a flowchart of the Monte Carlo rendering denoising method based on temporal multi-component hierarchical features described in Embodiment 1 of this disclosure.
[0030] Figure 2 This is a visualization of the multi-component data described in Embodiment 1 of this disclosure;
[0031] Figure 3 This is a schematic diagram illustrating the calculation of the reflection motion vector as described in Embodiment 1 of this disclosure;
[0032] Figure 4 This is a structural diagram of the hierarchical feature extraction network described in Embodiment 1 of this disclosure;
[0033] Figure 5 This is a schematic diagram of the operation of the feature compression encoding module of the hierarchical feature extraction network encoder described in Embodiment 1 of this disclosure;
[0034] Figure 6 This is a schematic diagram of the operation of the hierarchical feature extraction network encoder partial feature recovery decoding module described in Embodiment 1 of this disclosure;
[0035] Figure 7 A schematic diagram of the operation of the spatial filtering module described in Embodiment 1 of this disclosure;
[0036] Figure 8 This is a schematic diagram of the operation of the kernel prediction module described in Embodiment 1 of this disclosure;
[0037] Figure 9 This is a flowchart illustrating the operation of the main modules described in Embodiment 1 of this disclosure;
[0038] Figure 10 This is a flowchart of the operation of the time filtering module described in Embodiment 1 of this disclosure;
[0039] Figures 11(a) to 11(c) This is a comparison chart of the results of the Monte Carlo rendering denoising method based on temporal multi-component hierarchical features described in Embodiment 1 of this disclosure. Detailed Implementation
[0040] It should be noted that the following detailed descriptions are exemplary and intended to provide further illustration of the invention. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains.
[0041] It should be noted that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the exemplary embodiments of the invention. The terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion, for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.
[0042] Where there is no conflict, the embodiments and features in the embodiments of the present invention can be combined with each other.
[0043] All data acquisition in this embodiment is carried out in accordance with laws and regulations and with user consent, and the data is used legally.
[0044] Example 1
[0045] This embodiment provides a Monte Carlo rendering denoising method based on temporal multi-component hierarchical features;
[0046] like Figure 1 As shown, the Monte Carlo rendering denoising method based on temporal multi-component hierarchical features includes:
[0047] S101: Obtain the Monte Carlo rendering image to be denoised;
[0048] S102: Divide the Monte Carlo rendering image to be denoised into diffuse reflection component, gloss reflection component and specular reflection component;
[0049] S103: Input the diffuse reflection component, gloss reflection component, and specular reflection component into the trained denoising model and output the final denoised image;
[0050] The trained denoising model is used to extract spatial features for each component and perform spatial filtering on the extracted spatial features. If the current image to be denoised belongs to the first frame, the spatial filtering result is used as the denoised image of the current component. If the current image to be denoised does not belong to the first frame, the current image to be denoised is subjected to temporal filtering based on the spatial filtering result and temporal features of the previous frame, and the temporal filtering result is used as the denoised image of the current component. The denoised images of the three components are fused to obtain the final denoised image.
[0051] Further, in S101: the Monte Carlo rendering image to be denoised is obtained; the noise image of the current frame 1spp and its accompanying auxiliary information are obtained in the real-time Monte Carlo rendering pipeline, wherein the auxiliary information includes: depth, normal, roughness and motion vector.
[0052] Further, in S102: the Monte Carlo rendering image to be denoised is divided into diffuse reflection component, glossy reflection component and specular reflection component. This is based on the different performance of the light sample when it first hits the scene, and the noise map and its accompanying auxiliary information are divided into three components: diffuse reflection component, glossy reflection component and specular reflection component.
[0053] The diffuse reflection component includes: a diffuse reflection noise map and diffuse reflection auxiliary information; the gloss reflection component includes: a gloss reflection noise map and gloss reflection auxiliary information; the specular reflection component includes: a specular reflection noise map and specular reflection auxiliary information.
[0054] Furthermore, such as Figure 9 As shown, in step S103: the diffuse reflection component, the gloss reflection component, and the specular reflection component are input into the trained denoising model, and the final denoised image is output. The trained denoising model includes three parallel branches, namely the first branch, the second branch, and the third branch.
[0055] The first branch includes: a first-level feature extraction network, a first spatial filtering module, a first judgment module, and a first temporal filtering module connected in sequence; the first branch is used to denoise the diffuse reflection noise map to obtain a diffuse reflection denoised map;
[0056] The second branch includes: a second-level feature extraction network, a second spatial filtering module, a second judgment module, and a second temporal filtering module connected in sequence; the second branch is used to denoise the gloss reflection noise map to obtain a gloss reflection denoised map;
[0057] The third branch includes: a third-level feature extraction network, a third spatial filtering module, a third judgment module, and a third temporal filtering module connected in sequence; the third branch is used to denoise the specular reflection noise map to obtain a denoised specular reflection map.
[0058] The outputs of the first time filtering module, the second time filtering module, and the third time filtering module are all connected to the input of the synthesis module, and the output of the synthesis module outputs the final denoised image; the outputs of the first judgment module, the second judgment module, and the third judgment module are also connected to the input of the synthesis module.
[0059] Further, the first branch includes: a first-level feature extraction network, which performs spatial feature extraction on the diffuse noise map and diffuse auxiliary information of the Monte Carlo rendering image to be denoised; a first spatial filtering module performs spatial filtering on the diffuse noise map based on the spatial features to obtain a first diffuse denoised map; a first judgment module determines whether the Monte Carlo rendering image to be denoised in the current frame has a previous frame image; if not, the first diffuse denoised map is directly used as the final diffuse denoised map; if it does, the diffuse denoised map of the previous frame and the temporal features of the previous frame image are obtained, and the first diffuse denoised map is temporally filtered based on the diffuse denoised map of the previous frame and the temporal features of the previous frame image to obtain a second diffuse denoised map, and the second diffuse denoised map is used as the final diffuse denoised map.
[0060] Further, the second branch includes: a second-level feature extraction network, which extracts spatial features from the gloss reflection noise map and gloss reflection auxiliary information of the Monte Carlo rendering image to be denoised; a second spatial filtering module performs spatial filtering on the gloss reflection noise map based on the spatial features to obtain a first gloss reflection denoised map; a second judgment module determines whether the Monte Carlo rendering image to be denoised in the current frame has a previous frame image; if not, the first gloss reflection denoised map is directly used as the final gloss reflection denoised map; if it does, the gloss reflection denoised map of the previous frame image and the temporal features of the previous frame image are obtained, and the first gloss reflection denoised map is temporally filtered based on the gloss reflection denoised map of the previous frame image and the temporal features of the previous frame image to obtain a second gloss reflection denoised map, and the second gloss reflection denoised map is used as the final gloss reflection denoised map.
[0061] Further, the third branch includes: a third-level feature extraction network, which extracts spatial features from the specular noise map and specular auxiliary information of the Monte Carlo rendering image to be denoised; the third spatial filtering module performs spatial filtering on the specular noise map based on the spatial features to obtain a first specular denoised map; the third judgment module determines whether the Monte Carlo rendering image to be denoised in the current frame has a previous frame image; if not, the first specular denoised map is directly used as the final specular denoised map; if it does, the specular denoised map of the previous frame and the temporal features of the previous frame image are obtained, and the first specular denoised map is temporally filtered based on the specular denoised map of the previous frame and the temporal features of the previous frame image to obtain a second specular denoised map, which is used as the final specular denoised map.
[0062] Furthermore, the internal structures of the first-level feature extraction network, the second-level feature extraction network, and the third-level feature extraction network are identical; the first-level feature extraction network includes:
[0063] The encoder and decoder are connected in sequence;
[0064] The encoder includes: a first convolution module, a second convolution module, a third convolution module, and a fourth convolution module connected in sequence;
[0065] The first convolutional module includes: a first convolutional layer, a second convolutional layer, and a first linear rectifier unit connected in sequence;
[0066] The second convolutional module includes: a first feature compression coding module, a third convolutional layer, a fourth convolutional layer, and a second linear rectifier unit connected in sequence;
[0067] The third convolutional module includes: a second feature compression coding module, a fifth convolutional layer, a sixth convolutional layer, and a third linear rectifier unit connected in sequence;
[0068] The fourth convolutional module includes: a third feature compression coding module, a seventh convolutional layer, an eighth convolutional layer, and a fourth linear rectifier unit connected in sequence;
[0069] The output of the first linear rectifier unit is connected to the input of the first feature compression coding module, the output of the second linear rectifier unit is connected to the input of the second feature compression coding module, and the output of the third linear rectifier unit is connected to the input of the third feature compression coding module.
[0070] The decoder includes: a fifth convolutional module, a sixth convolutional module, and a seventh convolutional module connected in sequence;
[0071] The fifth convolutional module includes: a first feature recovery decoding module, a ninth convolutional layer, a tenth convolutional layer, and a fifth linear rectifier unit connected in sequence;
[0072] The sixth convolutional module includes: a second feature recovery decoding module, an eleventh convolutional layer, a twelfth convolutional layer, and a sixth linear rectifier unit connected in sequence;
[0073] The seventh convolutional module includes: a third feature recovery decoding module, a thirteenth convolutional layer, a fourteenth convolutional layer, and a seventh linear rectifier unit connected in sequence;
[0074] The input of the first feature recovery decoding module is connected to the output of the fourth linear rectifier unit, the input of the second feature recovery decoding module is connected to the output of the fifth linear rectifier unit, and the input of the third feature recovery decoding module is connected to the output of the sixth linear rectifier unit.
[0075] The input of the first feature recovery decoding module is also connected to the output of the third linear rectifier unit, the input of the second feature recovery decoding module is also connected to the output of the second linear rectifier unit, and the input of the third feature recovery decoding module is also connected to the output of the first linear rectifier unit.
[0076] It should be understood that the architecture of a hierarchical feature extraction network is as follows: Figure 4 As shown, it contains cascaded encoders and decoders.
[0077] Figure 4 The area within box (a) shows the encoder part of the hierarchical feature extraction network. Its main function is to continuously reduce the resolution of the original input and extract high-dimensional features from it. The encoder consists of four convolutional blocks, each containing two convolutional layers and one linear rectified unit. The last three convolutional blocks are each preceded by a feature compression coding module that performs downsampling on the input from the previous layer.
[0078] Figure 4 The area within box (b) shows the decoder part of the hierarchical feature extraction network. Its main function is to continuously restore the high-dimensional feature resolution from the encoder output to the original input size and compress the feature dimension. The decoder consists of three convolutional blocks, each preceded by a feature recovery decoding module that upsamples the input from the previous layer. Each convolutional block has two convolutional layers and one linear rectified unit.
[0079] The linear rectifier unit is implemented using a linear rectifier function.
[0080] Furthermore, the internal structures of the first feature compression coding module, the second feature compression coding module, and the third feature compression coding module are the same; the first feature compression coding module includes: an aggregation weight generation layer and a downsampling layer; the aggregation weight generation layer includes: a fifteenth convolutional layer and a first softmax function layer connected in sequence.
[0081] Furthermore, the first feature compression coding module generates pixel-by-pixel block aggregation weights through embedded convolutional layers and a flexible maximum transfer function. The convolutional layer has a 2×2 kernel size and two convolution steps, generating four weights for each 2×2 pixel region. The formula for this process is as follows:
[0082] w g =Softmax(Conv(f in (4)
[0083] Among them, w g This represents the aggregated weights for each pixel block, and Softmax represents the flexible maximum transfer function used for weight normalization. Conv represents the convolution operation, f in This indicates the feature input from the previous layer.
[0084] Next, as follows Figure 5 As shown, for f in In the middle, each 2×2 pixel area uses the w of the corresponding pixel position. g The four feature vectors are merged into one, thus completing the processing of the input feature f. in Downsampling reduces the resolution to one-quarter of its original value. The formula for this process is as follows:
[0085]
[0086] in f out The eigenvector represented by the coordinates (x, y) in the middle is... f in The eigenvector represented by the coordinates (u,v) in the middle is... Indicates w g The second (u-2x)+(v-2y) weight value in the aggregate weight vector at the coordinate (x,y) position.
[0087] The feature compression coding module is a learnable neural network that generates aggregate weights for pixel-by-pixel blocks that perform downsampling operations on the input of the previous layer through embedded convolutional layers and a flexible maximum transfer function.
[0088] Furthermore, the internal structures of the first feature recovery decoding module, the second feature recovery decoding module, and the third feature recovery decoding module are the same. The first feature recovery decoding module includes a pixel-wise sputtering weight generation layer and an upsampling layer. The pixel-wise sputtering weight generation layer includes a sixteenth convolutional layer and a second softmax function layer connected in sequence.
[0089] Furthermore, the first feature recovery decoding module generates pixel-by-pixel sputtering weights that perform upsampling operations on the input of the previous layer through an embedded convolutional layer and a flexible maximum transfer function. The convolutional layer has a 3×3 kernel size, generating four groups of four weights per pixel, with each group corresponding to a 2×2 region. The formula for this process is shown below:
[0090]
[0091] Where w s This represents the pixel-by-pixel sputtering weight. Each contains 4 weight values, corresponding to four 2×2 regions: the top left corner, the top right corner, the bottom left corner, and the bottom right corner of the center pixel. Because the regions overlap, the four regions cover a total of 3×3 pixel blocks.
[0092] Next, as Figure 6 As shown, for f respectively in The four regions of each pixel are weighted accordingly, and the four feature vectors contained in each region are fused into one, thus generating a total of four feature vectors for each pixel, completing the processing of the input feature f. in Upsampling increases the resolution to four times its original value. The formula for calculating a single region is as follows:
[0093]
[0094] Where, ab∈{00,01,10,11}, f out The eigenvector represented by the coordinates (2x+a, 2y+b) in the middle. f in The eigenvector represented by the coordinates (x+1-a+i, y+1-b+j) in the middle is... express The 2i+jth weight value in the sputtering weight vector at the (x,y) position.
[0095] The feature recovery decoding module is a learnable neural network that generates pixel-by-pixel sputtering weights that perform upsampling operations on the input of the previous layer through embedded convolutional layers and a flexible maximum transfer function.
[0096] Furthermore, such as Figure 7 As shown, the internal structures of the first spatial filtering module, the second spatial filtering module, and the third spatial filtering module are the same; the first spatial filtering module includes:
[0097] The first core prediction module, the first downsampling module, the second core prediction module, the second downsampling module, the third core prediction module, the first filtering submodule, the first upsampling module, the first level combination module, the second upsampling module, and the second level combination module are connected in sequence.
[0098] The output of the first core prediction module is connected to the input of the second filtering submodule, and the output of the second filtering submodule is connected to the input of the second-level combination module.
[0099] The output of the second filtering submodule is connected to the input of the third downsampling module, the output of the third downsampling module is connected to the input of the third filtering submodule, the output of the third filtering submodule is connected to the input of the fourth downsampling module, and the output of the fourth downsampling module is connected to the input of the first filtering submodule.
[0100] The output of the second core prediction module is connected to the input of the third filtering submodule, and the output of the third filtering submodule is connected to the input of the first-level combination module.
[0101] Furthermore, the first spatial filtering module processes the extracted spatial features to generate a filtering kernel and linear fusion parameters for each pixel at different resolutions. The noise map in each component is denoised using the filtering kernel at different resolutions, and the denoised images at multiple resolutions are merged into a single image using the linear fusion parameters to obtain the spatial filtering result for that multi-component in the current frame.
[0102] Specifically, the spatial filtering module mainly includes four operation processes: downsampling, kernel prediction, hierarchical filtering, and hierarchical combination.
[0103] Downsampling is mainly used to divide the image and spatial features into different levels. The feature compression coding module of the encoder part of the feature extraction network at these levels (the structure and method are the same, but they are independent of each other and do not share internal learnable weights) outputs the spatial features and the denoised image to different resolutions for denoising at different levels.
[0104]
[0105]
[0106] Where the superscript l∈{1,2} indicates downsampling to 1 / 2 2l Resolution size, 0 represents the original resolution size, Down represents the feature compression coding module, f s l Indicates 1 / 2 2l Spatial features at resolution Indicates 1 / 2 2l The spatially denoised irradiance image at this resolution will be explained later when introducing the filtering module.
[0107] Furthermore, the nuclear prediction process is as follows: Figure 8 As shown. The first kernel prediction module, the second kernel prediction module, and the third kernel prediction module have the same internal functions. The first kernel prediction module is used to generate a pixel-by-pixel filtering kernel. For spatial feature f s l The two convolutional layers map the data to two sub-features with a feature dimension of 4. and Then, 25 filter weights are generated for any pixel location, representing the filter kernel within the 5×5 neighborhood of that pixel. For pixel location (x, y), the filter weight at the neighboring pixel location (u, v) is obtained by taking the dot product of the feature vectors corresponding to the sub-features.
[0108]
[0109] in This represents the filtering weights between pixel (x, y) and its neighboring pixels (u, v) at resolution level l. Sub-features The feature vector at pixel position (x,y) Sub-features The feature vector at pixel position (u,v);
[0110] The weights are normalized using the Softmax transfer function to ensure energy conservation.
[0111]
[0112] in This represents the normalized weight value, which takes values in the range [0,1], and guarantees that... Represents the set of neighboring pixel coordinates of the pixel position (x, y).
[0113] Furthermore, the internal functions of the first filtering submodule, the second filtering submodule, and the third filtering submodule are identical. The first filtering submodule is used to denoise the image input at the corresponding resolution using the pixel-by-pixel filtering kernel generated by the kernel prediction module.
[0114] First, the irradiance noise map at the original resolution... use Denoising is performed to obtain the spatially denoised irradiance image at the original resolution. The specific process formula is as follows:
[0115]
[0116] in, Represents spatially denoised irradiance images The value at pixel position (x, y), Represents the set of neighboring pixel coordinates of the pixel position (x, y). Represents the original resolution noise map The value at pixel position (u,v).
[0117] Next, the remaining levels are denoised from high to low resolution to obtain spatially denoised irradiance images at each resolution:
[0118]
[0119] Where the superscript l∈{1,2} indicates downsampling to 1 / 2 2l Resolution size, 0 indicates the original resolution size. Represents the set of neighboring pixel coordinates of the pixel position (x, y).
[0120] Furthermore, the internal functions of the first-level combination module and the second-level combination module are identical. The first-level combination module is used to fuse spatially denoised irradiance images from multiple resolution levels together, thereby preserving low-frequency information of the image at the low-resolution level and preserving high-frequency details of the image at the high-resolution level.
[0121]
[0122]
[0123] Where the superscript l∈{0,1}, This represents the result of linearly mixing the spatially denoised irradiance image at the l-th level resolution with the spatially denoised irradiance image at the (l+1)-th level resolution. Up indicates that the operation is applied, using the feature recovery decoding module (with the same structure but independent of each other, and not sharing internal learnable weights) of the decoder part of the hierarchical feature extraction network. represents the pixel-wise linear fusion parameter at the l-th level resolution. The value of each element is in the range of [0,1]. It is obtained by passing the spatial features of the corresponding level through one convolutional layer and one Sigmoid activation function.
[0124] The final irradiance image obtained is the spatially multi-level fused and denoised image at the original resolution. in
[0125] Correspondingly, spatial filtering ultimately outputs a denoised result with three components: a spatially denoised diffuse irradiance image. Spatially denoised glossy reflectance irradiance images Spatially denoised specular reflection irradiance image
[0126] If the current i-th frame does not have a consecutive preceding frame, then the temporal feature f of the current frame is... t i equal to the spatial feature f of the current frame s The temporal denoised image of the current frame Equal to the spatially denoised irradiance image of the current frame
[0127] It should be understood that the first spatial filtering module generates pixel-by-pixel filtering kernels and linear fusion parameters at different resolutions from the pixel-by-pixel spatial feature vectors. The filtering kernels are used to denoise the noisy image at different resolutions, and the linear fusion parameters are used to merge the denoised images at multiple resolutions into one image to obtain the multi-component spatial filtering result of the current frame.
[0128] Filter kernels and linear fusion parameters for each pixel at different resolutions are generated using spatial feature vectors. Noise in each component is denoised using the aforementioned filter kernels at different resolutions, and the denoised images from multiple resolutions are merged into a single image using the linear fusion parameters to obtain the spatial filtering result for that multi-component in the current frame. The multi-component spatial filtering result corresponds to: diffuse reflection spatial filtering result, gloss spatial filtering result, and specular reflection spatial filtering result.
[0129] Furthermore, the internal structures of the first time filtering module, the second time filtering module, and the third time filtering module are identical; the first time filtering module is as follows: Figure 10 As shown, it is used for:
[0130] If the current i-th frame of the image to be denoised contains consecutive (i-1)-th frames of the image to be denoised, then obtain the temporal feature f of each component in the (i-1)-th frame. t i-1 The denoised image of the current component in frame i-1. Using the motion vectors corresponding to the components to f t i-1 and Projected onto the corresponding pixel in the current frame:
[0131]
[0132]
[0133] in, Represents the temporal features projected onto the current frame. The value at pixel position (x, y). This represents the denoised image projected onto the current frame. The value at pixel position (x, y), M xyu and M xyvThese represent the values in the horizontal direction and the vertical direction of the motion vector M at pixel position (x, y), respectively. The motion vector corresponding to the component refers to the surface motion vector for the diffuse reflection component, the reflection motion vector for the gloss reflection component, and the reflection motion vector for the specular reflection component.
[0134] Next, the temporal features of the (i-1)th frame after projection are used. Spatial features f of the current frame s Generate temporal fusion parameter α t :
[0135]
[0136] Where Conv represents the convolutional layer Conv, Sigmoid represents the Sigmoid activation function, and the resulting pixel-wise temporal fusion parameter α t The range of values for is [0,1]. Indicates will with f s They are pieced together along the feature dimension.
[0137] Then use α t The projected temporal features of the (i-1)th frame Spatial features f s By fusing, the temporal features f of the current i-th frame are obtained. t i :
[0138]
[0139] Using α t The projected temporally denoised image of the previous frame Denoising image in the current frame space By fusing, the temporal fused image of the current component in the i-th frame is obtained.
[0140]
[0141] Using time-domain features f t i Pixel-wise filtering kernels are generated in the same manner as the first kernel prediction module, and the filtering kernels are used to fuse the temporal image in the same manner as the first filtering submodule. Denoising is performed to obtain the time-domain filtering result.
[0142] Correspondingly, the temporal filtering ultimately outputs a denoised result with three components: a temporally denoised diffuse irradiance image. Temporal denoising gloss reflectance irradiance image Temporal domain denoised specular reflection irradiance image
[0143] It should be understood that the first temporal filtering module is used to project the feature vector of the previous frame and the denoised image onto the current frame through the motion vector corresponding to each component when a previous frame exists, and fuse them with the spatial feature vector and spatial filtering result of the current frame, and denoise the mixed filtering result by using the pixel-by-pixel filter generated by the fused feature vector, so as to obtain the temporal filtering result of each component.
[0144] If the current frame has a continuous output from the previous frame, the feature vector of each component in the previous frame and the denoised image of the previous frame are obtained. The feature vector of the previous frame and the denoised image are projected onto the current frame using the temporal reliable motion vector of the current frame. The projected feature vector of the previous frame and the feature vector of the corresponding component in the current frame are used to generate temporal fusion parameters through a temporal fusion neural network. These parameters are then used to fuse the projected feature vector of the previous frame with the spatial feature vector and spatial filtering result of the denoised image, respectively. The fused feature vector is used to generate a filter kernel for each pixel, and the fused filtering result is denoised to obtain a multi-component temporal filtering result. The multi-component temporal filtering result corresponds to: diffuse reflection temporal filtering result, gloss temporal filtering result, and specular reflection temporal filtering result.
[0145] Applying the inverse transform of the decomposed components to the denoising result of each component yields the denoised image of the current frame. The inverse transform of the decomposed components method is then used to synthesize the multi-component temporal filtering results into the final denoised image. Simultaneously, a Monte Carlo renderer is used to render the scene, outputting a reference image at 8192 spp that is virtually noise-free and can be considered convergent. Then, based on a predefined minimum loss function, the difference between the denoised image generated by this method and the reference image is calculated, iterating through the learnable parameter weights in all modules.
[0146] Furthermore, the fusion of the denoised images of the three components to obtain the final denoised image includes:
[0147] The denoised image of the i-th frame is obtained by multiplying the three components of the final output from the temporal filtering by their corresponding reflectance values, and then summing the three products.
[0148]
[0149] Further, in step S103: the diffuse reflection component, gloss reflection component, and specular reflection component are input into the trained denoising model, and the final denoised image is output. The training process of the trained denoising model includes:
[0150] Construct a training set, which consists of Monte Carlo rendered images of known denoised images;
[0151] The training set is input into the denoising model to train it. Training is stopped when the loss function value of the denoising model no longer decreases, and the trained denoising model is obtained.
[0152] Furthermore, the loss function of the denoising model is specifically expressed as:
[0153] If the current i-th frame does not have a consecutive preceding frame, the spatial loss is calculated using the following formula:
[0154]
[0155] For the i-th frame, the reference image includes: a diffuse irradiance reference map. Reference chart of glossy reflectance irradiance Specular Reflection Irradiance Reference Chart Composite reference figure E i .
[0156] If the current i-th frame has a consecutive previous frame i-1, the temporal loss is calculated using the following formula:
[0157]
[0158] Then, by weighted summing the spatial loss and temporal loss, the complete loss function can be obtained:
[0159] L = 0.64·L s +0.36·L t (twenty four)
[0160] The above loss function is used to backpropagate and optimize all learnable weights in this method.
[0161] The training set uses the Amazon tavern scene from the Open Research Content Archive. The roughness of some objects in the scene was edited to make the scene have more gloss and specular reflection areas.
[0162] The scene was rendered using the Falcor renderer in 1spp. During the process, the camera viewpoint and some objects in the scene were moved quickly, generating a total of 33 consecutive frame data segments. Each segment included 32 consecutive frames, for a total of 1056 frames. Each frame included a noise map and corresponding auxiliary information.
[0163] Based on the reflection behavior of the light emitted by Monte Carlo rendering after it first hits the scene, each frame of data is divided into diffuse reflection component, glossy reflection component and specular reflection component.
[0164] The diffuse component data includes: 1spp diffuse irradiance noise map Diffuse reflectance diagram A diff Normal N, depth Q, roughness R, surface motion vector M.
[0165] The data for the gloss reflectance component includes: a 1spp gloss reflectance irradiance noise map. Gloss Reflectance A spec Normal N, Depth Q, Roughness R, Reflection Motion Vector M refl .
[0166] The data for the specular reflection component includes: a 1spp specular reflection irradiance noise map. Specular Reflectance A refl Reflection normal N refl Depth Q, Reflection roughness R refl , Reflection motion vector M refl .
[0167] In this process, all irradiance noise maps undergo a logarithmic transformation before being input into the hierarchical feature extraction network. The reason for this is that the magnitude of irradiance often far exceeds 1. By using a logarithmic function, the range of values of this number is compressed to a suitable interval for network training. However, the original irradiance noise image is still used in subsequent filtering and denoising. The reflection normal and reflection roughness represent the surface normal and material roughness of the scene after the light first hits the scene and produces specular reflection, respectively. Figure 2 The visualization results of the above multi-component data are shown. The calculation of motion vectors is described in detail below.
[0168] Given two consecutive frames, where i represents the current frame and i-1 represents the previous frame, the motion vector represents the pixel-to-pixel positional correspondence in the image space between these two frames. Specifically, for the pixel position p in the current frame... i and its corresponding pixel position p in the previous frame i-1 p i The motion vector M(p) of the position i Then it is represented as p i-1 With p i The distance difference between the two in the image space is:
[0169] M(p i ) = p i-1 -p i (1)
[0170] Therefore, find the pixel position p i The corresponding pixel position p in the previous frame i-1 This is crucial for calculating motion vectors, which is typically done using back projection.
[0171] For surface-based motion vectors, first, the pixel position p in the image space is... i The model is projected into the world coordinate system corresponding to the current frame, then transformed back to the world coordinate system of the previous frame based on the geometric movement of the scene model. Finally, the transformed world coordinate system is projected back into the image space in the previous frame, and the position in that image space is p. i The corresponding pixel position p in the previous frame i-1 The formula for this process is expressed as:
[0172] p i-1 =P i-1 [T i ] -1 [P i ] -1 p i (2)
[0173] Where P represents MVP (Model-View-Projection) transformation, and T represents the geometric transformation of the scene model between two frames.
[0174] However, surface-based motion vectors cannot accurately represent the correspondence between glossy or specular reflection areas, leading to incorrect temporal accumulation and resulting in ghosting or excessive blurring after temporal filtering and fusion. For example, when the viewpoint remains constant, the plane is stationary, and the object moves, its reflection will be reflected on the plane. Since surface motion vectors track the geometric changes of the model's surface in the scene, and the plane is stationary, the motion vector at the reflection's position will have a magnitude of 0, failing to correspond to the actual movement. Therefore, for areas with glossy or specular reflection, the motion vector should change as the reflected reflection moves.
[0175] To address the aforementioned issues, the calculation of the reflection motion vector should trace the object that produces the reflected image. Figure 3 This demonstrates the calculation process of the reflection motion vector. First, for pixel p of the current frame image... i Its corresponding reflection shading point in the scene is s. i The point is located in a reflective area on the plane, and the reflected light from this colored point hits point h on the object. i Based on the geometric transformation of the scene model, the reflection hit point h of the current frame can be obtained. i The corresponding point h in the previous frame i-1 Similarly, we can determine the current frame viewpoint v. i In the previous frame, it was at position v i-1 Here it is assumed that reflections between the two frames both occur through the shading point s. iOn the plane with n as the normal vector, the viewpoint v can be easily derived from the law of mirror reflection. i-1 and the reflection hit point h i-1 The reflection shading point on the plane is s. i-1 Then s i-1 The pixel position p in the image space after MVP transformation i-1 It is the pixel p of the current frame image. i The corresponding pixel in the previous frame. The above process discusses the case of perfect specular reflection, but actual reflection may occur within a range near the path of perfect reflection. Therefore, based on the roughness of the material of the reflecting object, in s i-1 A point is randomly sampled from the vicinity of the plane according to a Gaussian distribution. As an approximation, the subsequent process remains unchanged. By obtaining the pixel correspondence between two frames using the above method, the reflection motion vector M, which can accurately respond to changes in the reflection area in the time domain, can be obtained using formula (1). r .
[0176] The generated multi-component data is input into a hierarchical feature extraction network to obtain pixel-by-pixel spatial feature vectors. These spatial feature vectors are then used to generate pixel-by-pixel filtering kernels and linear fusion parameters at different resolutions. The noisy images are then denoised using these filtering kernels at different resolutions. Finally, the linear fusion parameters are used to merge the denoised images from multiple resolutions into a single image, yielding the multi-component spatial filtering result for the current frame. If a consecutive previous frame exists, the feature vectors from the previous frame and the denoised image are projected onto the current frame using the motion vectors corresponding to each component. This is then fused with the spatial feature vectors and spatial filtering result of the current frame. Finally, the pixel-by-pixel filtering kernels generated from the fused feature vectors are used to denoise the mixed filtering result, resulting in the temporal filtering result for each component.
[0177] The generated multi-component data are then input into the corresponding hierarchical feature extraction networks for feature extraction.
[0178]
[0179] Among them, f s This represents the pixel-wise spatial features proposed from the current component data, with a feature dimension of 8. `Net` represents the hierarchical feature extraction network. It's worth noting that the hierarchical feature extraction network corresponding to each component has the exact same architecture, but their internal learnable weights are independent of each other; therefore, the same logic applies to the other components.
[0180] In this embodiment, the PyTorch framework is used, the optimizer is Adam, the initial learning rate is set to 0.001, a total of 128 training rounds are performed, and the graphics processor is NVIDIA RTX 3090.
[0181] Compared with existing technologies, this embodiment provides better denoising quality, with a peak signal-to-noise ratio (PSNR) of 27.51 dB and a temporal peak signal-to-noise ratio (tPSNR) of 24.54 dB. Furthermore, the model trained using this method utilizes TensorRT to accelerate runtime inference, processing a 1280×720 resolution input frame in approximately 13.2 ms. This is suitable for real-time rendering tasks that generate at least 30 frames per second (i.e., a single frame takes no more than 33 ms). Figures 11(a) to 11(c) This is a comparison chart of the results of the Monte Carlo rendering denoising method based on temporal multi-component hierarchical features described in Embodiment 1 of this disclosure.
[0182] This disclosure provides a Monte Carlo rendering denoising method based on temporal multi-component hierarchical features. Its hierarchical feature extraction network extracts feature vectors for the diffuse reflection component, gloss reflection component, and specular reflection component generated by 1spp Monte Carlo rendering, respectively. A spatial filtering module is used to perform spatial denoising on the noisy image of each component at multiple resolution scales. A temporal filtering module is used to fuse the current frame and the previous frame image of each component for denoising. Finally, the denoising results of the multiple components are fused together to obtain a high-quality denoised image.
[0183] A pre-trained hierarchical feature extraction network extracts features from each component of the current frame, obtaining spatial feature vectors for each component. These spatial feature vectors are then used to generate filtering kernels and linear fusion parameters for each pixel at different resolutions. Noise in each component is denoised using these filtering kernels at different resolutions, and the denoised images from multiple resolutions are merged into a single image using the linear fusion parameters, yielding the spatial filtering result for that multi-component in the current frame. Specifically, the multi-component spatial filtering result corresponds to: diffuse reflection spatial filtering result, gloss reflection spatial filtering result, and specular reflection spatial filtering result.
[0184] If there is a consecutive previous frame output before the current frame, the feature vector of each component in the previous frame and the denoised image of the previous frame are obtained. The feature vector of the previous frame and the denoised image are projected onto the current frame using the temporal reliable motion vector of the current frame. The projected feature vector of the previous frame and the feature vector of the corresponding component of the current frame are used to generate temporal fusion parameters through a temporal fusion neural network. These parameters are then used to fuse the projected feature vector of the previous frame with the spatial feature vector and spatial filtering result of the denoised image, respectively. The fused feature vector is used to generate a filter kernel corresponding to each pixel, and the fused filtering result is denoised to obtain a multi-component temporal filtering result. The multi-component temporal filtering result corresponds to: diffuse reflection temporal filtering result, gloss reflection temporal filtering result, and specular reflection temporal filtering result.
[0185] By applying the inverse transform of the decomposed components to the denoising result of each component, the denoised image of the current frame can be obtained.
[0186] The hierarchical feature extraction network, the temporal fusion network, and various filtering, transformation, and fusion operations are trained in a unified manner. Based on the predefined minimum loss function, the difference between the denoised image generated by this method and the reference image is calculated, and the module is iteratively optimized.
[0187] This disclosure provides a Monte Carlo rendering denoising method and system based on temporal multi-component hierarchical features. The scheme belongs to the field of 3D rendering post-processing denoising. The scheme includes: acquiring a 1spp noise image of the current frame in the Monte Carlo rendering pipeline and its accompanying auxiliary information, and dividing it into three components: diffuse reflection, gloss reflection, and specular reflection; inputting the multi-component noise image and auxiliary information into a pre-trained hierarchical feature extraction network to obtain pixel-by-pixel spatial feature vectors; and generating pixel-by-pixel filtering kernels and linear fusion parameters at different resolutions from the pixel-by-pixel spatial feature vectors. The noisy image is denoised using the aforementioned filtering kernel, and the denoised images at multiple resolutions are merged into a single image using linear fusion parameters to obtain the multi-component spatial filtering result of the current frame. When a previous frame exists, the feature vector of the previous frame and the denoised image are projected onto the current frame through the motion vector corresponding to each component, and fused with the spatial feature vector and spatial filtering result of the current frame. The pixel-by-pixel filtering kernel generated by the fused feature vector is used to denoise the mixed filtering result to obtain the temporal filtering result of each component. The filtering results of the multiple components are then fused together again in a predetermined manner to obtain the complete denoised image.
[0188] Example 2
[0189] This embodiment provides a Monte Carlo rendering denoising system based on temporal multi-component hierarchical features;
[0190] A Monte Carlo rendering denoising system based on temporal multi-component hierarchical features includes:
[0191] The acquisition module is configured to acquire the Monte Carlo rendered image to be denoised;
[0192] The partitioning module is configured to partition the Monte Carlo rendered image to be denoised into diffuse reflection components, glossy reflection components, and specular reflection components.
[0193] The denoising module is configured to input the diffuse reflection component, gloss reflection component, and specular reflection component into the trained denoising model and output the final denoised image.
[0194] The trained denoising model is used to extract spatial features for each component and perform spatial filtering on the extracted spatial features. If the current image to be denoised belongs to the first frame, the spatial filtering result is used as the denoised image of the current component. If the current image to be denoised does not belong to the first frame, the current image to be denoised is subjected to temporal filtering based on the spatial filtering result and temporal features of the previous frame, and the temporal filtering result is used as the denoised image of the current component. The denoised images of the three components are fused to obtain the final denoised image.
[0195] It should be noted that the acquisition module, segmentation module, and noise reduction module described above correspond to steps S101 to S103 in Embodiment 1. The examples and application scenarios implemented by these modules and their corresponding steps are the same, but they are not limited to the content disclosed in Embodiment 1. It should also be noted that these modules, as part of the system, can be executed in a computer system, such as a set of computer-executable instructions.
[0196] The descriptions of each embodiment in the above embodiments have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions in other embodiments.
[0197] The proposed system can be implemented in other ways. For example, the system embodiments described above are merely illustrative, and the division of modules described above is only a logical functional division. In actual implementation, there may be other division methods. For example, multiple modules may be combined or integrated into another system, or some features may be ignored or not executed.
[0198] Example 3
[0199] This embodiment also provides an electronic device, including: one or more processors, one or more memories, and one or more computer programs; wherein, the processor is connected to the memory, and the one or more computer programs are stored in the memory. When the electronic device is running, the processor executes the one or more computer programs stored in the memory to cause the electronic device to perform the method described in Embodiment 1.
[0200] It should be understood that in this embodiment, the processor can be a central processing unit (CPU), or it can be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor, etc.
[0201] Memory may include read-only memory and random access memory, and provides instructions and data to the processor. A portion of memory may also include non-volatile random access memory. For example, memory may also store information about the device type.
[0202] In the implementation process, each step of the above method can be completed by the integrated logic circuits in the processor hardware or by software instructions.
[0203] The method in Embodiment 1 can be directly implemented by a hardware processor, or implemented by a combination of hardware and software modules within the processor. The software modules can reside in readily available storage media in the art, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, or registers. This storage medium is located in memory; the processor reads information from the memory and, in conjunction with its hardware, completes the steps of the above method. To avoid repetition, a detailed description is not provided here.
[0204] Those skilled in the art will recognize that the units and algorithm steps described in connection with the various examples of this embodiment can be implemented in electronic hardware or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this invention.
[0205] Example 4
[0206] This embodiment also provides a computer-readable storage medium for storing computer instructions, which, when executed by a processor, complete the method described in Embodiment 1.
[0207] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.
Claims
1. A Monte Carlo rendering denoising method based on time-domain multi-component hierarchical features, characterized in that, include: Obtain the Monte Carlo render image to be denoised; The Monte Carlo rendering image to be denoised is divided into diffuse reflection components, glossy reflection components, and specular reflection components. The diffuse reflection component, gloss reflection component, and specular reflection component are input into the trained denoising model, and the final denoised image is output. The trained denoising model is used to extract spatial features for each component and perform spatial filtering on the extracted spatial features. If the current image to be denoised belongs to the first frame, the spatial filtering result is used as the denoised image of the current component. If the current image to be denoised does not belong to the first frame, the current image to be denoised is subjected to temporal filtering based on the spatial filtering result and temporal features of the previous frame, using the motion vector corresponding to the component. The temporal filtering result is used as the denoised image of the current component. The denoised images of the three components are fused to obtain the final denoised image. The motion vector corresponding to the component refers to the surface motion vector for the diffuse reflection component, the reflection motion vector for the glossy reflection component, and the reflection motion vector for the specular reflection component. The reflection motion vector is calculated by tracking the object that produces the reflected image: first, for the pixels of the current frame image... Its corresponding reflection shading point in the scene is The point is located in a reflective area on the plane, and the reflected light from this colored point hits a point on the object. Based on the geometric transformation of the scene model, the reflection hit point of the current frame is obtained. The corresponding point in the previous frame Similarly, the current frame viewpoint is obtained. Position in the previous frame Assume that reflections between the two frames both occur through the shading point. and with On the plane with normal vector, the viewpoint is obtained by the law of specular reflection. and reflection hit point The reflection shading point on the plane is Next Pixel positions in image space after MVP transformation It is the number of pixels in the current frame image. The corresponding pixel in the previous frame; calculate the reflection motion vector based on the corresponding pixel; The trained denoising model includes three parallel branches: a first branch, a second branch, and a third branch. The first branch includes a first-level feature extraction network, a first spatial filtering module, a first judgment module, and a first temporal filtering module connected in sequence. The second branch includes a second-level feature extraction network, a second spatial filtering module, a second judgment module, and a second temporal filtering module connected in sequence. The third branch includes a third-level feature extraction network, a third spatial filtering module, a third judgment module, and a third temporal filtering module connected in sequence. The internal structures of the first, second, and third spatial filtering modules are identical. The first spatial filtering module includes, in sequence, a first kernel prediction module, a first downsampling module, a second kernel prediction module, a second downsampling module, a third kernel prediction module, a first filtering submodule, a first upsampling module, a first-level combination module, a second upsampling module, and a second-level combination module. The internal functions of the first, second, and third kernel prediction modules are the same. The first kernel prediction module is used to generate pixel-by-pixel filtering kernels for spatial features. The two convolutional layers map the data to two sub-features with a feature dimension of 4. and Then, 25 filter weights are generated for any pixel location, representing the filter kernel within a 5×5 neighborhood of that pixel; for pixel location... By performing a dot product of the feature vectors corresponding to the sub-features, we can obtain the position of the sub-feature in the neighboring pixels. Filter weights at the point : ; in Indicates in At layered resolution, pixels With neighboring pixels The filter weights; Use flexible maximum transfer function Normalize the weight values to ensure energy conservation: ; in This represents the normalized weight values, and its range is... and guarantee ; Indicates pixel position The set of neighboring pixel coordinates.
2. The Monte Carlo rendering denoising method based on temporal multi-component hierarchical features as described in claim 1, characterized in that, The first branch is used to denoise the diffuse reflection noise map to obtain a diffuse reflection denoised map; the second branch is used to denoise the gloss reflection noise map to obtain a gloss reflection denoised map; and the third branch is used to denoise the specular reflection noise map to obtain a specular reflection denoised map. The outputs of the first time filtering module, the second time filtering module, and the third time filtering module are all connected to the input of the synthesis module, and the output of the synthesis module outputs the final denoised image; the outputs of the first judgment module, the second judgment module, and the third judgment module are also connected to the input of the synthesis module.
3. The Monte Carlo rendering denoising method based on temporal multi-component hierarchical features as described in claim 1, characterized in that, The first branch includes: a first-level feature extraction network, which extracts spatial features from the diffuse noise map and diffuse auxiliary information of the Monte Carlo rendering image to be denoised; a first spatial filtering module performs spatial filtering on the diffuse noise map based on the spatial features to obtain a first diffuse denoised map; a first judgment module determines whether the Monte Carlo rendering image to be denoised in the current frame has a previous frame image; if not, the first diffuse denoised map is directly used as the final diffuse denoised map; if it does, the diffuse denoised map of the previous frame and the temporal features of the previous frame image are obtained, and the first diffuse denoised map is temporally filtered based on the diffuse denoised map of the previous frame and the temporal features of the previous frame image to obtain a second diffuse denoised map, which is used as the final diffuse denoised map.
4. The Monte Carlo rendering denoising method based on temporal multi-component hierarchical features as described in claim 1, characterized in that, The first-level feature extraction network includes: a first feature compression coding module; the first feature compression coding module generates pixel-by-pixel block aggregation weights through embedded convolutional layers and a flexible maximum transfer function, wherein the convolutional kernel size of the convolutional layer is 2×2, the convolution step number is 2, and four corresponding weights are generated for each 2×2 pixel region; the formula for this process is as follows: (4) in, This represents the aggregation weight for each pixel block. This represents the flexible maximum transfer function, used for weight normalization, i.e. , This represents the convolution operation. This indicates feature input from the previous layer; right Each 2×2 pixel area uses the corresponding pixel position This merges four feature vectors into one, completing the processing of the input features. Downsampling reduces the resolution to the original value. The formula for this process is as follows: (5) in, express The median coordinate is The feature vector represented by the position, express The median coordinate is The feature vector represented by the position, express The median coordinate is The first position in the aggregate weight vector at the position Each weight value.
5. The Monte Carlo rendering denoising method based on temporal multi-component hierarchical features as described in claim 1, characterized in that, The first-level feature extraction network includes: a first feature recovery and decoding module; the first feature recovery and decoding module generates pixel-by-pixel sputtering weights that perform upsampling operations on the input of the previous layer through embedded convolutional layers and a flexible maximum transfer function; wherein the convolutional kernel size of the convolutional layer is 3×3, generating 4 groups of 4 weights for each pixel, and each group of weights corresponds to a 2×2 region; the formula for this process is as follows: (6) in This represents the pixel-by-pixel sputtering weight. Each contains 4 weight values, corresponding to four 2×2 regions: the top left corner of the center pixel, the top right corner of the center pixel, the bottom left corner of the center pixel, and the bottom right corner of the center pixel. To each The four regions of each pixel are weighted accordingly, and the four feature vectors contained in each region are fused into one, thus generating a total of four feature vectors for each pixel, completing the processing of the input features. Upsampling increases the resolution to four times its original value; the formula for the calculation process of a single region is expressed as follows: (7) in, , express The median coordinate is The feature vector represented by the position, express The median coordinate is The eigenvectors they represent express The median coordinate is The first position in the sputtering weight vector Each weight value.
6. The Monte Carlo rendering denoising method based on temporal multi-component hierarchical features as described in claim 1, characterized in that, The first time filtering module is used for: If the current number is The frame of the image to be denoised has consecutive first frames. If the frame is an image to be denoised, then obtain the first frame. Temporal features of each component in the frame and the Denoising image of a frame Using the reliable motion vectors in the time domain corresponding to the components, and Projected onto the current frame of the image to be denoised: (16) (17) in, Represents the temporal features projected onto the current frame. At pixel position The value at that location, This represents the denoised image projected onto the current frame. At pixel position The value at that location, and Representing motion vectors respectively At pixel position In a vector of length 2, represents the value in the horizontal direction and represents the value in the vertical direction; Next, using the projected [number]th [item] Frame temporal features Spatial features of the current frame Temporal fusion parameters are generated using a temporal fusion neural network. : (18) in, Represents a convolutional layer , express Activation function, generating pixel-by-pixel temporal fusion parameters The range of values is , Indicates will and They are joined together along the feature dimension; Then use The projected first Frame temporal features Spatial features Merge to obtain the current number Temporal features of frames : (19) use The projected temporally denoised image of the previous frame Denoising image in the current frame space Merge to obtain the current number Temporal fusion image of frames : (20) Using time-domain features Generate pixel-wise filter kernels and use the filter kernels to fuse the temporal domain images. Denoising is performed to obtain the time-domain filtering result. .
7. A Monte Carlo rendering denoising system based on temporal multi-component hierarchical features, characterized in that... include: The acquisition module is configured to acquire the Monte Carlo rendered image to be denoised; The partitioning module is configured to partition the Monte Carlo rendered image to be denoised into diffuse reflection components, gloss reflection components, and specular reflection components. The denoising module is configured to input the diffuse reflection component, gloss reflection component, and specular reflection component into the trained denoising model and output the final denoised image. The trained denoising model is used to extract spatial features for each component and perform spatial filtering on the extracted spatial features. If the current image to be denoised belongs to the first frame, the spatial filtering result is used as the denoised image of the current component. If the current image to be denoised does not belong to the first frame, the current image to be denoised is subjected to temporal filtering based on the spatial filtering result and temporal features of the previous frame, using the motion vector corresponding to the component. The temporal filtering result is used as the denoised image of the current component. The denoised images of the three components are fused to obtain the final denoised image. The motion vector corresponding to the component refers to the surface motion vector for the diffuse reflection component, the reflection motion vector for the glossy reflection component, and the reflection motion vector for the specular reflection component. The reflection motion vector is calculated by tracking the object that produces the reflected image: first, for the pixels of the current frame image... Its corresponding reflection shading point in the scene is The point is located in a reflective area on the plane, and the reflected light from this colored point hits a point on the object. Based on the geometric transformation of the scene model, the reflection hit point of the current frame is obtained. The corresponding point in the previous frame Similarly, the current frame viewpoint is obtained. Position in the previous frame Assume that reflections between the two frames both occur through the shading point. and with On the plane with normal vector, the viewpoint is obtained by the law of specular reflection. and reflection hit point The reflection shading point on the plane is Next Pixel positions in image space after MVP transformation It is the number of pixels in the current frame image. The corresponding pixel in the previous frame; calculate the reflection motion vector based on the corresponding pixel; The trained denoising model includes three parallel branches: a first branch, a second branch, and a third branch. The first branch includes a first-level feature extraction network, a first spatial filtering module, a first judgment module, and a first temporal filtering module connected in sequence. The second branch includes a second-level feature extraction network, a second spatial filtering module, a second judgment module, and a second temporal filtering module connected in sequence. The third branch includes a third-level feature extraction network, a third spatial filtering module, a third judgment module, and a third temporal filtering module connected in sequence. The internal structures of the first, second, and third spatial filtering modules are identical. The first spatial filtering module includes, in sequence, a first kernel prediction module, a first downsampling module, a second kernel prediction module, a second downsampling module, a third kernel prediction module, a first filtering submodule, a first upsampling module, a first-level combination module, a second upsampling module, and a second-level combination module. The internal functions of the first, second, and third kernel prediction modules are the same. The first kernel prediction module is used to generate pixel-by-pixel filtering kernels for spatial features. The two convolutional layers map the data to two sub-features with a feature dimension of 4. and Then, 25 filter weights are generated for any pixel location, representing the filter kernel within a 5×5 neighborhood of that pixel; for pixel location... By performing a dot product of the feature vectors corresponding to the sub-features, we can obtain the position of the sub-feature in the neighboring pixels. Filter weights at the point : ; in Indicates in At layered resolution, pixels With neighboring pixels The filter weights; Use flexible maximum transfer function Normalize the weight values to ensure energy conservation: ; in This represents the normalized weight values, and its range is... and guarantee ; Indicates pixel position The set of neighboring pixel coordinates.
8. An electronic device, characterized in that it comprises: Memory is used to store computer-readable instructions in a non-transitory manner. as well as Processor, for executing the computer-readable instructions, When the computer-readable instructions are executed by the processor, they perform the method described in any one of claims 1-6.
9. A storage medium characterized by being non-transitory. The system stores computer-readable instructions, wherein, when the non-transitory computer-readable instructions are executed by a computer, the instructions of the method according to any one of claims 1-6 are executed.
