Denoising an image rendered using Monte Carlo rendering

Abstract

Embodiments of the present disclosure relate to denoising an image rendered using Monte Carlo rendering. A plurality of pixel-based sampling points are identified within an image, wherein the sampling points of the pixels are distributed within the pixels. For individual sampling points of individual pixels, corresponding radiation vectors are estimated. The radiation vector includes one or more radiation values ​​characterizing the light received at the sampling point. A first machine learning module generates a corresponding intermediate radiation feature vector for each pixel based on the radiation vector associated with the sampling point within the pixel. A second machine learning module generates a corresponding final radiation feature vector for each pixel based on the intermediate radiation feature vector for the pixel and one or more other intermediate radiation feature vectors for one or more other pixels adjacent to the pixel. One or more kernels are generated based on the final radiation feature vector and applied to corresponding pixels of the image to generate a lower noise image.

Description

Technical Field

[0001] The present disclosure relates generally to denoising of images and, more particularly, to techniques for denoising images rendered using Monte Carlo methods. Background Art

[0002] With advances in computer graphics over the past few years, synthetic images can be generated using computers. For example, image processing applications can be used to digitally generate synthetic images from synthetic or virtual scenes, where the scenes include various digitally generated objects. Often, such synthetic images may appear unrealistic due to a lack of lighting effects in the synthetic scenes. A scene file describes the virtual scene, including information about its geometry, lighting, shadows, viewpoint, and / or other properties. The process of adding lighting effects to the synthetic scene to make the resulting synthetic image appear realistic is generally referred to as realistic rendering of the synthetic scene, or, for the purposes of this disclosure, as rendering the scene. In any case, the scene can be rendered by an image processing application based on the scene file. Rendering techniques include methods that attempt to simulate the interaction of light with various objects within the image, known as light transport modeling. The depiction of light and its interaction with various scene elements within the image enhances the realism of the content depicted within the image. Existing techniques for this type of realistic rendering of synthetic scenes include, for example, Monte Carlo methods. However, Monte Carlo methods can generate noisy images, particularly when the number of sampling points used for the Monte Carlo method is relatively low. Compounding this problem is the relatively large number of sampling points that imposes a significant computational burden. This increased computational cost inhibits real-time processing. Summary of the Invention

[0003] Techniques for denoising an image are disclosed. For example, a method for denoising a first image includes identifying a plurality of corresponding sampling points within individual pixels of a plurality of pixels in the first image. For each sampling point in the individual pixels, a corresponding radiance vector is estimated. The radiance vector of the sampling point includes one or more radiance values ​​representing light received at the sampling point. Based on the radiance vectors associated with the plurality of corresponding sampling points within the corresponding pixel, a first machine learning module generates a corresponding intermediate radiance eigenvector for the individual pixel. A second machine learning module generates a corresponding final radiance eigenvector for the individual pixel. More specifically, a first final radiance eigenvector for the first pixel is generated based at least in part on: (a) the intermediate radiance eigenvector for the first pixel, and (b) one or more other intermediate radiance eigenvectors for one or more other pixels adjacent to the first pixel. Based at least in part on the final radiance eigenvector, one or more filter weight subsets, or kernel subsets, are generated. More specifically, a kernel is a two-dimensional convolution matrix and can be considered a mask or filter that comprises a relatively small matrix (e.g., small relative to the image to which it is applied). Each subset can include one or more kernels. The one or more kernel subsets are applied to one or more corresponding pixels in the first image to generate the second image. In some instances, the second image has less noise and is more realistic than the first image.

[0004] In another example, a system for rendering a realistic image from an original image comprising a plurality of pixels is provided. The system includes one or more processors; and an image denoising subsystem executable by the one or more processors. In some examples, the subsystem is configured to generate a corresponding radiation eigenvector and a corresponding auxiliary eigenvector for each of a plurality of pixels of the original image. The subsystem is further configured to generate at least a first kernel subset and a second kernel subset based on the radiation eigenvectors and the auxiliary eigenvectors of the plurality of pixels. The subsystem is configured to convolve the first kernel subset with the original image to generate a first image. In some instances, the subsystem is configured to convolve the second kernel subset with the first image to generate a second image that is realistic relative to the original image.

[0005] In another example, a computer program product comprising one or more non-transitory machine-readable media encoded with instructions, which, when executed by one or more processors, cause a process to be performed for reducing noise within a digital image. In some examples, the process includes identifying a first plurality of sampling points within a first pixel of a plurality of pixels included in a first image and identifying a second plurality of sampling points within a second pixel of the plurality of pixels. For each sampling point in the first plurality of sampling points and the second plurality of sampling points, a corresponding radiance vector is estimated, wherein the radiance vector of the sampling point includes one or more radiance values ​​associated with light received at the sampling point. The process also includes: (i) generating a first intermediate radiance eigenvector for the first pixel based at least in part on the radiance vector associated with each sampling point in the first plurality of sampling points, and (ii) generating a second intermediate radiance eigenvector for the second pixel based at least in part on the radiance vector associated with each sampling point in the second plurality of sampling points. The process also includes generating a first final radiance eigenvector for the first pixel and a second final radiance eigenvector for the second pixel based at least in part on the first intermediate radiance eigenvector and the second intermediate radiance eigenvector. The process also includes generating one or more kernels based at least in part on the first final radiance eigenvector and the second final radiance eigenvector, and applying the one or more kernels to one or more corresponding pixels of the first image to generate a second image, wherein the second image has less noise than the first image and the second image can be perceived as more realistic than the first image. BRIEF DESCRIPTION OF THE DRAWINGS

[0006] Figure 1 is a block diagram schematically illustrating selected components of an example computing device configured to generate a noisy image (or otherwise access a noisy image) using a Monte Carlo method and to denoise or at least reduce noise in the noisy image, according to some embodiments of the present disclosure.

[0007] Figure 2 is a block diagram schematically illustrating selected components of an example system according to some embodiments of the present disclosure, the example system including a server device in communication with a server device(s) Figure 1 A computing device, wherein the combination of the computing device and (multiple) server devices is configured to generate a noisy image (or otherwise access the noisy image) using a Monte Carlo method, and to denoise or at least reduce noise in the noisy image.

[0008] Figure 3A An example scenario is illustrated according to some embodiments of the present disclosure, where an image is to be generated and denoised by the techniques discussed herein.

[0009] Figure 3BIllustrated is a scene, a relatively noisy image generated for the scene using Monte Carlo rendering with 8 samples per pixel (8SPP), and a relatively noise-free image generated by denoising the noisy image, according to some embodiments of the present disclosure.

[0010] Figure 3C Illustrated are a plurality of adjacent pixels of a scene to be rendered and example sampling points of some example pixels according to some embodiments of the present disclosure.

[0011] Figure 4 A radiation feature vector generation module according to some embodiments of the present disclosure is illustrated, which is configured to receive a corresponding radiation vector for each of a plurality of sampling points of a plurality of pixels of a scene, and generate a corresponding final radiation vector for each of the plurality of pixels, wherein the final radiation vector can be used to denoise a noisy image of the scene.

[0012] Figure 5 An auxiliary feature vector generation module according to some embodiments of the present disclosure is illustrated, which is configured to receive a corresponding auxiliary vector for each of a plurality of sampling points of a plurality of pixels of a scene, and generate a corresponding final auxiliary vector for each of the plurality of pixels, wherein the final auxiliary vector can be used to denoise a noisy image of the scene.

[0013] Figure 6A 、 Figure 6B 、 Figure 6C Illustrated is the generation of corresponding example kernels, which are used as filters to denoise a noisy image, according to some embodiments of the present disclosure.

[0014] Figure 7A Denoising a noisy image using multiple denoising stages, where each stage uses a corresponding subset of kernels, is illustrated according to some embodiments of the present disclosure.

[0015] Figure 7B Illustrated is an example convolution operation between a kernel subset and a noisy image according to some embodiments of the present disclosure.

[0016] Figure 7C The diagram shows a summary of some embodiments of the present disclosure. Figure 7A and Figure 7B Example pseudocode for the illustrated operations.

[0017] Figure 8A and Figure 8B are flow charts that collectively illustrate example methods for denoising a noisy image generated using Monte Carlo rendering so that the denoised image is less noisy and more realistic than the noisy image, according to some embodiments of the present disclosure. DETAILED DESCRIPTION

[0018] Techniques for rendering realistic images are disclosed. These techniques are particularly useful in the context of Monte Carlo rendering of noisy images from synthetic scenes using a relatively small number of sampling points, where the techniques can be applied to denoise the noisy images. Because the initial noisy image is generated using a relatively low number of sampling points, the computational burden and time consumption of executing the Monte Carlo method are relatively low. In any case, a scene file containing information describing various properties of the synthetic scene is used to generate radiances and auxiliary vectors. The denoising technique uses deep learning to learn a feature vector for each pixel of the noisy image based on the radiances and auxiliary vectors. Pairwise relationships between the feature vectors, such as L2 distances, are then calculated to determine how much radiance a pixel should contribute to its neighboring pixels. Specifically, the L2 distances between pairs of feature vectors are used to generate weights for various kernels, which are then used in the corresponding denoising stages. For example, in the first denoising stage, a first subset of kernels is convolved with the noisy image to generate a first intermediate image. In the second denoising stage, a second subset of kernels is convolved with the first intermediate image to generate a second intermediate image. This process continues for a given number of stages. The final denoising stage outputs a final, low-noise, realistic image.

[0019] The realistic images generated using the techniques provided herein are comparable in quality and realism to images generated using time-consuming and resource-intensive Monte Carlo rendering with higher sampling points. The techniques discussed herein are fast (e.g., because low-sampling Monte Carlo methods were initially used to generate noisy images) and can be implemented in real time or near real time, making them suitable for rendering at interactive frame rates. Many embodiments and variations will be appreciated in light of this disclosure.

[0020] General Overview

[0021] As discussed above, Monte Carlo methods are often used to render synthetic scenes, generating realistic images from the synthetic scenes using scene files. For example, by assigning expected radiance or light levels to individual pixels of the synthetic scene, Monte Carlo methods aim to generate realistic versions of the scene. Monte Carlo involves a computational technique that uses repeated random sampling to numerically estimate unknown information. The number of samples used during Monte Carlo determines the quality of the estimate. For example, because the sampling process is random, the results are subject to variance, which manifests as noise, and, for example, the number of sample points must be increased fourfold to halve the error. Therefore, convergence requires a relatively large number of samples. Therefore, when Monte Carlo methods are used for image rendering, lower sampling can produce low-quality (e.g., noisy) renderings, while higher sampling can produce higher-quality (e.g., relatively noise-free) renderings. To generate high-quality and realistic images, the number of samples must be relatively large, which increases the computational burden and / or computation time of the Monte Carlo method. Even with today's high computing power, rendering complex scenes using Monte Carlo can take a relatively long time (e.g., measured in hours) to produce noise-free (or noise-less) results.

[0022] Therefore, this paper provides a technique for denoising noisy images drawn by Monte Carlo methods. Since the initial noisy image can be generated using a relatively low number of sampling points, the computational burden and time consumption of executing the Monte Carlo method are relatively low. The denoising technique uses deep learning to learn feature vectors from each sampling point in each of the multiple pixels of the noisy image. This denoising technique then uses the pairwise affinity of these deep features to calculate the distance between feature vectors (such as L2 distance) to determine how much radiation a pixel should contribute to its neighboring pixels. This pairwise relationship between feature vectors results in the rendering of low-noise realistic images. The realistic images generated using this technique are comparable in quality and realism to images generated using the time-consuming and resource-intensive Monte Carlo rendering with a higher number of sampling points. The techniques discussed herein are relatively fast (for example, because the low-sampling Monte Carlo method is initially used to generate the noisy image) and can be implemented at interactive frame rates, as will be discussed in further detail.

[0023] In more detail, and according to some example embodiments, the noise image generator module of the image denoising system generates a Monte Carlo rendering of the synthetic scene using a relatively small number of samples per pixel (SPP). By way of example only, 8 SPP, 16 SPP, or 24 SPP may be used for the Monte Carlo rendering, but a higher number of samples may be used if warranted by the specifics of a given application. For example, by attributing desired radiance or light levels to individual pixels of the synthetic scene, the Monte Carlo method aims to generate a realistic version of the scene. Due to the relatively low number of SPPs used by the Monte Carlo rendering, the resulting image is relatively noisy and far from being perceived as realistic.

[0024] During or after the Monte Carlo rendering, the image denoising system identifies a plurality of corresponding sampling points within each pixel. The sampling points of a pixel can be randomly distributed within the pixel. Various examples discussed herein assume an 8SPP scenario, however, as will be appreciated, another suitable SPP can also be used in other examples.

[0025] The feature identification module of the image denoising system is used for each sampling point S i Identify one or more radiation features. For example, the radiation feature is the sampling point S i For example, in some such example embodiments, the incoming light at the sampling point is classified into different categories, such as (i) direct illumination and diffuse reflection, (ii) direct illumination and specular reflection, (iii) indirect illumination and diffuse reflection, and (iv) indirect illumination and specular reflection, and the radiation signature is identified based on such classification. The image denoising system (also referred to simply as the "system") then generates a corresponding radiance vector L for each sampling point of each pixel of the noisy image. si (where S i is the index of the sampling point), as discussed in further detail herein.

[0026] The feature identification module of the image denoising system also targets each sampling point S i Identify one or more auxiliary features. Sampling point S i The auxiliary features include uniquely defining or characterizing the sampling point S i Different features or attributes of . Auxiliary features include sampling points S i The intrinsic properties of (such as roughness, emissivity, albedo) and the sampling point S i The system then generates a corresponding auxiliary vector for each sampling point of each pixel. For example, the auxiliary vector A si For sampling point S i generated, as further discussed in detail herein.

[0027] Subsequently, a radiation feature vector generation module of the image denoising system receives, for each sampling point, a corresponding radiation vector for the corresponding sampling point. A machine learning module (such as a neural network module) of the radiation feature vector generation module generates, for example, a corresponding intermediate radiation feature vector for each pixel. Thus, the machine learning module receives the radiation vector for each sampling point and generates an intermediate radiation feature vector for each pixel. For example, the machine learning module is trained to identify how multiple radiation vectors corresponding to the various sampling points of a particular pixel are processed and averaged to generate a corresponding intermediate radiation feature vector for the particular pixel. By way of example only, the machine learning module is a neural network that is trained to detect patterns in the radiation vectors and determine how the radiation vectors are to be combined or averaged to form the intermediate radiation feature vector, as will be discussed in detail further on.

[0028] Subsequently, a radiation feature vector generation module of the image denoising system generates a corresponding final radiation feature vector and corresponding one or more scaling factors for each pixel. For example, the radiation feature vector generation module includes another neural network module that receives the intermediate radiation feature vector and generates a final radiation feature vector. In some such example embodiments, the neural network module is trained to correlate the radiation features of adjacent pixels to generate the final radiation feature vector. For example, the neural network module determines how the radiation value associated with a pixel can contribute to adjacent pixels. The neural network module spatially propagates the radiation property of the first pixel to one or more adjacent pixels (for example, if the first pixel and the one or more adjacent pixels have some similar radiation properties and / or are located on the same or similar segment of the image) and correspondingly generates a final radiation feature vector for the pixel, as will be discussed in further detail.

[0029] In some embodiments, each final radiance eigenvector has multiple segments that match multiple denoising stages. Various examples discussed herein assume three denoising stages, and therefore, each final radiance eigenvector has three corresponding segments. For example, the final radiance eigenvector F_Ri for example pixel i has three segments F_Ri_1, F_Ri_2, and F_Ri_3. In some such embodiments, for individual pixel i, three corresponding scaling factors a i 1 、a i 2 and a i 3 is also generated, as further discussed herein. Variations will be understood.

[0030] In addition, the auxiliary feature vector generation module of the image denoising system receives, for each sampling point, a corresponding auxiliary vector for the corresponding sampling point. A machine learning module (such as a neural network module) of the auxiliary feature vector generation module generates, for example, a corresponding intermediate auxiliary feature vector for a given pixel. Thus, the machine learning module receives the auxiliary vector for each sampling point and generates an intermediate auxiliary feature vector for each pixel. For example, the machine learning module is trained to identify how multiple auxiliary vectors corresponding to the various sampling points of a particular pixel are processed and averaged to generate the corresponding intermediate auxiliary feature vector for the particular pixel. By way of example only, the machine learning module is a neural network that is trained to detect patterns in the auxiliary vectors and determine how the auxiliary vectors are to be combined or averaged to form the intermediate auxiliary feature vector, as will be discussed in detail further on.

[0031] Subsequently, the auxiliary feature vector generation module of the image denoising system generates a corresponding final auxiliary feature vector for each pixel. For example, the radiation feature vector generation module includes another neural network module that receives the intermediate auxiliary feature vector and generates a final auxiliary feature vector. In some examples, the neural network module is trained to correlate the auxiliary features of adjacent pixels to generate the final auxiliary feature vector. For example, the neural network module determines how the auxiliary value associated with the pixel can contribute to the adjacent pixels. The neural network module propagates the auxiliary attribute space of the first pixel to one or more adjacent pixels (for example, if the first pixel and the one or more adjacent pixels have some similar auxiliary attributes and / or are located on the same or similar segment of the image) and correspondingly generates a final auxiliary feature vector for the pixel, as will be discussed in further detail.

[0032] Subsequently, the kernel generation module of the image denoising system generates K subsets of kernels corresponding to the number of K denoising stages. Each subset may include one or more kernels. The various examples discussed herein have three denoising stages, and therefore, three subsets of kernels are generated. However, it is to be noted that, depending on a given application, and as will be appreciated in light of this disclosure, any number of denoising stages may be used. As will be further appreciated in light of this disclosure, a kernel is a two-dimensional convolution matrix and may be considered a mask or filter that comprises a relatively small matrix (e.g., small relative to the image to which it is applied). To this end, the kernels may be used to process images in various image processing applications. In an example embodiment, each subset has a stride of 2 (k-1)where k=1, ..., K, as will be discussed below in this document. By way of example only, each kernel in the first subset of kernels is a 3x3 matrix corresponding to a window of 3x3 pixels (although in other examples, larger matrix sizes may be used, such as a 13x13 matrix). The center pixel in a particular kernel is the target pixel for that kernel, and the kernel is applied to the noisy image to correct the radiance value for that particular target pixel. The stride of each kernel in the first subset is set to 2 (1-1) =2 0 =1. Therefore, when generating the kernel, each pixel that is 1 pixel away from the target pixel is assigned a non-zero weight. Each kernel in the second subset of kernels is a 5x5 matrix corresponding to a window of 5x5 pixels. The stride of each kernel in the second subset is set to 2 (2-1) =2 1 =2. Therefore, when generating the kernel, each pixel that is 2 pixels away from the target pixel is assigned a non-zero weight. For example, other pixels within the kernel that are 1 pixel away are assigned a zero weight. Each kernel in the third kernel subset is a 9x9 matrix, corresponding to a window of 9x9 pixels. The stride of each kernel in the third subset is set to 2 (3-1) =2 2 = 4. Therefore, when generating the kernel, every pixel that is 4 pixels away from the target pixel is assigned a non-zero weight. For example, other pixels within the kernel that are 2 or 3 pixels away are assigned a zero weight.

[0033] Thus, each kernel has weights assigned to at least some of the pixels. For example, a kernel with a stride of 1 has 8 pixels adjacent to the target pixel, and each of these 8 adjacent pixels is assigned a corresponding non-zero weight. A kernel with a stride of 2 has 24 pixels adjacent to the target pixel, and 8 of the 20 adjacent pixels are assigned a corresponding non-zero weight. A kernel with a stride of 4 has 80 pixels adjacent to the target pixel, and 8 of the 80 adjacent pixels are assigned a corresponding non-zero weight.

[0034] Assume that the kernel has pixel i as the target pixel, and assume that the neighboring pixel j in the kernel is assigned a corresponding non-zero weight. The weight assigned to the neighboring pixel j in the kernel is denoted as wk(i,j), where k is the subset count to which the kernel belongs. For example, if the kernel belongs to the first kernel subset, the weight is w1(i,j); if the kernel belongs to the second kernel subset, the weight is w2(i,j), and so on. In any case, to generate the weight wk(i,j), the first L2 norm between (i) the kth segment of the final radiation feature of pixel i (e.g., F_Ri_k) and (ii) the kth segment of the final radiation feature of pixel j (e.g., F_Rj_k) is calculated. In addition, the second L2 norm between (i) the final auxiliary feature of pixel i and (ii) the final auxiliary feature of pixel j is also calculated. The weight wk(i,j) is based on the first L2 norm, the second L2 norm, and the scaling factor a j k For example, when calculating the weight wk(i,j), the negative exponent of the first L2 norm and the negative exponent of the second L2 norm are taken into account. In addition, the weight wk(i,j) is proportional to the scaling factor a. j k Scaling factor a j k Make the weights wk(i,j) and wk(j,i) asymmetric. For example, if there is no scaling factor, the weights wk(i,j) and wk(j,i) will be the same because the first L2 norm and the second L2 norm discussed above are the same for both weights. However, if the scaling factor a j k and a i k If the weights wk(i,j) and wk(j,i) are different, the resulting weights wk(i,j) and wk(j,i) may also be different. This ensures that the amount of radiation a first pixel contributes to a second neighboring pixel is different from the amount of radiation a second pixel contributes to the first pixel. Thus, the scaling factor ensures that an outlier pixel can receive radiation contributions from neighboring pixels while contributing substantially no radiation to neighboring pixels. For example, if a pixel is a light source, such as one that emits a significant amount of radiation (e.g., is a light source) compared to one or more neighboring pixels, then the pixel is an outlier pixel.

[0035] Once the kernels for each subset are calculated, they are applied to the noisy image (initially generated using a Monte Carlo method using a relatively small number of sampling points) to produce a noise-free or noise-reduced output image. For example, the kernels from the first subset are convolved with the noisy image to produce a first intermediate image. The kernels from the second subset are convolved with the first intermediate image to produce a second intermediate image. Finally, the kernels from the third subset are convolved with the second intermediate image to produce a noise-free, or at least noise-reduced, output image that is more realistic than the initial noisy image. Thus, according to this example embodiment, three denoising stages are used to produce a final realistic output image.

[0036] The techniques discussed herein have several advantages over traditional denoising methods. For example, as discussed, in some embodiments, to provide a sufficiently large spatial context for the kernel, rather than using a single stage with a large, single, fixed-size kernel, multiple stages with multiple kernels having corresponding strides can be used as provided herein. It should be noted that, according to an embodiment, the kernels of the third subset have a size of 9x9 pixels, the kernels of the second subset have a size of 5x5 pixels, and the kernels of the first subset have a size of 3x3 pixels. However, in each kernel in the kernel, only the eight pixels adjacent to the target pixel are weighted with non-zero values. Therefore, although the spatial context of the kernel increases with stage, the computation does not increase proportionally, and each kernel computes the same number of weights regardless of the stage in which the kernel is used. Therefore, in some embodiments, denoising can be performed at a faster rate, such as at interactive frame rate. For example, denoising is fast because not all pixels within each kernel are weighted.

[0037] In addition, the radiation and auxiliary feature attributes of multiple sampling points are intelligently averaged by the corresponding neural network modules to generate an average value for each pixel. Therefore, as soon as the attributes of new sampling points are available, the average value is updated to obtain a refined image. In the example, this progressive rendering of the image makes the denoising process fast and achieves an interactive frame rate for the denoising process.

[0038] As discussed, the scaling factor contributes to the asymmetry of the radiance value contributions while determining the weights wk(i,j). Thus, an outlier pixel can receive radiance contributions from neighboring pixels while contributing substantially no radiance to neighboring pixels.

[0039] Furthermore, this system leverages deep learning to cast denoising as a supervised learning problem. Given a noisy input image, a neural network predicts a feature vector for each pixel by analyzing its spatial proximity. When determining how much radiance a pixel should contribute to another pixel, the system explicitly calculates the distance between these pixels' feature vectors. This system thus enables the use of low-capacity networks with lower computational cost, allowing the denoising operations discussed in this paper to run at interactive frame rates. Consequently, this system can generate a large number of noise-free or low-noise images, even though it uses only a fraction of the parameters used by traditional denoising systems.

[0040] System Architecture

[0041] Figure 1 1 is a block diagram schematically illustrating selected components of an example computing device 100 (also referred to as device 100) configured to generate a noisy image using a Monte Carlo method and to denoise or at least reduce noise in the noisy image, according to some embodiments of the present disclosure. As can be seen, device 100 includes an image denoising system 102 (also referred to as system 102) that allows device 100 to access a noisy image generated according to the Monte Carlo method and to denoise or at least reduce noise in the noisy image.

[0042] As will be appreciated, the configuration of device 100 can vary from one embodiment to the next. For this reason, the discussion herein will focus more on various aspects of device 100 related to reducing noise in noisy images, and less on standard component parts and functionality typical of computing devices. Device 100 includes, for example, a desktop computer, a laptop computer, a workstation, an enterprise-class server computer, a handheld computer, a tablet computer, a smartphone, a set-top box, a game console, and / or any other computing device that can reduce noise in a digital image.

[0043] In the illustrated embodiment, the device 100 includes one or more software modules configured to implement certain functionality disclosed herein and hardware configured to implement such implementations. These hardware and software components may include, among other things, a processor 132, a memory 134, an operating system 136, an input / output (I / O) component 138, a communications adapter 140, a data storage module 146, and the image denoising system 102. A digital content database 148 (e.g., comprising non-transitory computer memory) stores noisy images and / or noise-free (or noise-reduced) images and is coupled to the data storage module 146. A bus and / or interconnect 144 is also provided to allow for inter-device and intra-device communication using, for example, the communications adapter 140. In some embodiments, the system 100 includes a display screen 142 (referred to as display 142), although in some other embodiments, the display 142 may be external to the system 100 and communicatively coupled to the system 100. It is noted that, in an example, components such as the operating system 136 and the image denoising system 102 may be software modules stored in the memory 132 and executable by the processor 132. In an example, at least sections of the image denoising system 102 may be at least partially implemented by hardware, such as by an application specific integrated circuit (ASIC) or a microcontroller having one or more embedded routines. The bus and / or interconnect 144 is representative of all standard and proprietary technologies that allow the various functional components shown within the device 100 to interact, whether that interaction actually occurs through a physical bus structure or via software calls, request / response constructs, or any other such inter-component and intra-component interface technology, as will be appreciated.

[0044] The processor 132 may be implemented using any suitable processor and may include one or more co-processors or controllers, such as an audio processor or a graphics processing unit, to assist in the processing operations of the device 100. Likewise, the memory 134 may be implemented using any suitable type of digital storage device, such as one or more of a disk drive, a solid-state drive, a universal serial bus (USB) drive, flash memory, random access memory (RAM), or any suitable combination of the foregoing. The operating system 136 may include any suitable operating system, such as Google Android, Microsoft Windows, or Apple OS X. In view of this disclosure, it will be appreciated that the techniques provided herein may be implemented without regard to the particular operating system provided in conjunction with the device 100 and, therefore, may be implemented using any suitable existing or subsequently developed platform. The communication adapter 140 may be implemented using any suitable network chip or chipset that allows wired or wireless connection to a network and / or other computing devices and / or resources. The device 100 also includes one or more I / O components 138, such as one or more of a tactile keyboard, a display 142, a mouse, a touch-sensitive or touch screen display (e.g., the display 142), a touchpad, a microphone, a camera, a scanner, and a location service. Typically, not provided herein. Figure 1 Other standard component parts and functionalities reflected in the schematic block diagram will be apparent, and it will also be understood that the present disclosure is not intended to be limited to any specific hardware configuration. Therefore, other configurations and subassemblies can be used in other embodiments.

[0045] exist Figure 1 Also illustrated in the figure is an image denoising system 102 implemented on the device 100. In an example embodiment, the system 102 includes a feature identification module 103, a radiation feature vector generation module 104, an auxiliary feature vector generation module 108, a kernel generation module 112, a noise image generation module 116, and a noise reduction image generation module 120, each of which will be discussed in detail below. In the example, the components of the system 102 communicate with each other or other components of the device 102 using a bus and / or interconnect 144, as will be discussed in further detail below. The components of the system 102 can communicate with one or more other devices, including other computing devices of a user, a server device (e.g., a cloud storage device), a licensing server, or other devices / systems. Although the components of the system 102 are in Figure 1 Although shown separately in the drawings, any subcomponents may be combined into fewer components, such as into a single component, or divided into more components as may serve a particular implementation.

[0046] In an example, the components of system 102 that perform the functions discussed herein with respect to system 102 can be implemented as part of a standalone application, a module of an application, a plug-in to an application, a library function, or a function that can be called by other applications and / or a cloud computing model. Thus, the components of system 102 can be implemented as part of a standalone application on a personal computing device or a mobile device. Alternatively or additionally, the components of image denoising system 102 can be implemented in any application that allows for the reduction of noise in images generated and / or rendered using Monte Carlo methods.

[0047] Figure 2 is a block diagram schematically illustrating selected components of an example system 200 according to some embodiments of the present disclosure, including a server device 201 in communication with a server device 201. Figure 1 A computing device 100 is provided, wherein the combination of the device 100 and (a plurality of) server devices 201 (hereinafter also generally referred to as servers 201) is configured to generate a noisy image (or otherwise access the noisy image) using a Monte Carlo method, and to denoise or at least reduce the noise in the noisy image. In an example, the communication adapter 140 of the device 100 can be implemented using any suitable network chip or chipset that allows wired or wireless connection to a network 205 and / or other computing devices and / or resources. To this end, the device 100 is coupled to the network 205 via the adapter 140 to allow communication with other computing devices and resources such as the server 201. The network 205 is any suitable network through which computing devices communicate. For example, the network 205 can be a local area network (such as a home or office-based network), a wide area network (such as the Internet), or a combination of such networks, whether public, private, or both. In some cases, accessing resources on a given network or computing system may require credentials, such as a username, password, or any other suitable security mechanism.

[0048] In one embodiment, server 201 comprises one or more enterprise-class devices configured to provide a series of services that are invoked to provide noise reduction services for digital images, as variously described herein. In some embodiments, server 201 comprises an image denoising system 202 that provides such services, as variously described herein. Although one server implementation of the image denoising system may be Figure 2 , but it will be appreciated that typically tens, hundreds, thousands, or more such servers may be used to manage an even greater number of image denoising functions.

[0049] In the illustrated embodiment, server 201 includes one or more software modules configured to implement certain functionality disclosed herein and hardware configured to implement such implementations. These hardware and software components may include, among others, processor 232, memory 234, operating system 236, image denoising system 202 (also referred to as system 202), data storage module 246, and communications adapter 240. Digital content database 248 (e.g., comprising non-transitory computer memory) includes noisy images and denoised images and is coupled to data storage module 246. Buses and / or interconnects 244 are also provided to allow for inter-device and intra-device communication using, for example, communications adapter 240 and / or network 205. It should be noted that components such as operating system 236 and image denoising system 202 may be software modules stored in memory 234 and executable by processor 232. As will be appreciated, the previous discussion regarding the symbolic nature of buses and / or interconnects 144 applies equally to buses and / or interconnects 244.

[0050] Processor 232 is implemented using any suitable processor and may include one or more co-processors or controllers, such as an audio processor or a graphics processing unit, to assist in the processing operations of server 201. Likewise, memory 234 may be implemented using any suitable type of digital storage device, such as one or more of a disk drive, a universal serial bus (USB) drive, flash memory, random access memory (RAM), or any suitable combination of the foregoing. Operating system 236 may include any suitable operating system, and the particular operating system used is not particularly relevant, as previously mentioned. Communications adapter 240 may be implemented using any suitable network chip or chipset that allows wired or wireless connection to network 205 and / or other computing devices and / or resources. Server 201 is coupled to network 205 to allow communication with other computing devices and resources, such as device 100. Typically, not in Figure 2 Other component parts and functionalities reflected in the schematic block diagram will be apparent in light of this disclosure, and it will also be understood that this disclosure is not intended to be limited to any specific hardware configuration. In short, any suitable hardware configuration can be used.

[0051] Server 201 can generate, store, receive and transmit any type of data, including digital content and / or codes associated with such digital content. As shown, server 201 includes an image denoising system 202 in communication with system 102 on client device 100. In the example, with respect to Figure 1 The image denoising features discussed can be found in Figure 2The image denoising system 102 may be implemented only by the image denoising system 102, implemented only by the image denoising system 202, and / or may be shared between the image denoising system 102 and the image denoising system 202. Thus, in examples, none, some, or all of the image denoising features are implemented by the image denoising system 202.

[0052] For example, when located on server 201, image denoising system 202 comprises an application running on server 201 or part of a software application that can be downloaded to device 100. For example, system 102 may include a web-hosted application that allows device 100 to interact with content from image denoising system 202 hosted on server 201. In this way, server 201 reduces the noise level of digital images. Therefore, the location of some functional modules in system 202 may vary from one embodiment to the next. For example, while kernel generation module 112 is shown on the client side in this example, in other embodiments it may be on the server side (e.g., within system 202). Any number of client-server configurations will be apparent in light of this disclosure. In other embodiments, these techniques may be implemented entirely on the user's computer, for example, simply as a stand-alone image modification application. Similarly, while digital content database 148 is shown on the client side in this example, in other embodiments it may be on the server side, such as a cloud-based image database. Thus, the database of digital content and / or codes may be local or remote to device 102 , as long as it is accessible by the system 102 or by a module implemented by system 202 .

[0053] Example Operation

[0054] Figure 3A An example scene according to some embodiments of the present disclosure is illustrated, in which images are generated and denoised using the techniques discussed herein. The example scene illustrates a room near a window, wherein the room includes a table, a chair, and two lamps. Shadows from the lamps are incident on the walls of the room. An example location of a camera 304 for capturing scene 300 is also illustrated. It will be noted that in some examples, scene 300 is synthetically generated. Furthermore, in some such examples, the camera is a virtual camera. If the location of the virtual camera changes, the perspective of scene 300 will also change.

[0055] Scene 300 is initially synthetically generated without any lighting or shadows. This scene does not look very realistic. To make scene 300 look more realistic, scene 300 is rendered using an appropriate digital image rendering method (such as the Monte Carlo method).

[0056] An image generated from scene 300 (e.g., generated using a relatively large number of sampling points via a Monte Carlo method) may be a ground truth image. Thus, scene 300 is relatively noise-free and has a relatively high resolution. A rectangular region 308 is illustrated within the scene, which will be discussed later herein with respect to Figure 3B discuss.

[0057] Also illustrated is the path 312 of an example ray of light entering scene 300, reflecting from an example sample point 316 of scene 300 (which happens to be a wall), and reaching the camera. During realistic rendering of scene 300 (e.g., using a Monte Carlo method), properties of sample point 316 (also known as "auxiliary features") are taken into account. For example, if sample point 316 is a shiny object such as glass or a mirror, the incoming light will reflect from sample point 316 in a specific direction, e.g., specular reflection, and sample point 316 will be rendered in a specific manner. On the other hand, if sample point 316 is a dark object such as a wall, the light will reflect more evenly in many (e.g., all) different directions, e.g., diffuse reflection, and sample point 316 will be rendered in a completely different manner. Thus, examples of such properties or auxiliary features include the surface characteristics of sample point 316, the distance of sample point 316 from the camera, and several other auxiliary features, as will be discussed in further detail. Furthermore, the rendering of sample point 316 is also based on the amount and / or type of light input from various directions onto sample point 316. For example, the mapping of the sampling point 316 is also based on, for example, whether the sampling point receives direct illumination, indirect illumination, diffuse illumination, specular illumination, etc., which are collectively referred to as the radiation signature at the sampling point 316. Therefore, the mapping of the sampling point 316 is based on the auxiliary signature of the sampling point 316 as well as the radiation signature of the sampling point 316, which will be discussed in further detail.

[0058] Figure 3B The present invention illustrates some embodiments of the present invention. Figure 3A A relatively noisy image 349 rendered for the scene using Monte Carlo rendering with 8 samples per pixel (8SPP) and a relatively noise-free image 344 generated by denoising the image 349. Figure 3A discussed, Figures 3A to 3B The scene 300 is a ground truth scene, for example, which is rendered by a Monte Carlo method using a relatively large number of sampling points.

[0059] Figure 3B Also illustrated is an image 340, which represents a rectangular region 308 of the scene 300. The image 340 is a noisy image rendered by a Monte Carlo method, for example using a relatively small number of sampling points, such as 8SPP as will be discussed.

[0060] Figure 3BAlso illustrated is image 344, which represents a denoised version of image 344, as will be discussed in further detail herein. For example, noise in image 340 is removed or reduced to generate image 344. Generating image 344 from image 340 will be discussed in further detail.

[0061] Figure 3C 1 and 2. A plurality of adjacent pixels of a scene to be drawn according to some embodiments of the present disclosure and example sampling points of two example pixels are illustrated. For example, pixels Y11, Y12, ..., Y46 are illustrated as being arranged in rows and columns, such as in a matrix. As an example, pixel Y24 has adjacent pixels Y13, Y14, Y15, Y23, Y25, Y33, Y34, and Y35. Sampling points S_1_24, S_2_24, ..., S_8_24 within the example pixel Y24 are also illustrated. Sampling points S_1_42, S_2_42, ..., S_8_42 within another example pixel Y42 are also illustrated.

[0062] Therefore, 8SPP or 8 samples per pixel is Figure 3C , although individual pixels may have any number of sampling points therein. Although sampling points are illustrated for example pixels Y24 and Y42, other pixels also include such sampling points, although they are not illustrated in the figures for clarity of illustration. The sampling points of a pixel are randomly located within the pixel. For example, sampling points S_1_24, S_2_24, ..., S_8_24 are randomly distributed within pixel Y24, and sampling points S_1_42, S_2_42, ..., S_8_42 are randomly distributed within pixel Y42.

[0063] Equation 1 below is the rendering equation that quantifies the distribution of light energy in a scene at any given sampling point in space and forms the basis of modern photorealistic image rendering:

[0064] L(x,ω0)=L e (x,ω0)+∫ Ω+ f(x,ω i ,ω0).L(x,ψ i ).(n.ω i )dω i Equation 1 In Equation 1, L(x, ω0) is the outgoing radiation at the sampling point x in the output direction ω0. The term L e represents the radiant emissivity at the sampling point x in the output direction ω0. If the sampling point x is a light source that emits light or radiation, then the term L ewill have a non-zero value. The integral term is the reflected radiation at the sampling point x in the output direction ω0, for example, representing the light reflected from the sampling point x in the output direction ω0. Therefore, the total outgoing radiation L(x, ω0) at the sampling point x in the output direction ω0 is the sum of the emitted radiation and the reflected radiation at the sampling point x.

[0065] As can be seen in Equation 1, the reflected radiation is calculated by integrating over all possible input directions (e.g., integrating over all input directions represented by the Ω+ symbol). i ,ω0) is the bidirectional reflectance distribution function (BRDF), which defines the i The BRDF is a function of surface properties, referred to herein as auxiliary features, such as emissivity, reflection, roughness, albedo, and / or other properties at the sampling point x. i ) is from the input direction ω i The radiation value of the incident light, and the integral is represented by Ω + ω i Executed in all possible directions. i ) is a cosine factor that models the reduction in light energy due to light being spread over a larger area at grazing angles. Here, n is the normal or direction perpendicular to the surface at the sample point x.

[0066] Ideally, Equation 1 can be used to render synthetic images to make the images appear more realistic. However, a prominent challenge in solving Equation 1 is that the solution requires computing a recursive integral. For example, since the integrand of Equation 1 is often high-dimensional and discontinuous, it is often intractable or difficult to use numerical integration methods or closed-form integrals. Therefore, Monte Carlo methods and ray tracing are often used to solve Equation 1. Ray tracing involves tracing rays from a light source after being reflected by one or more sampling points and reaching a camera, such as Figure 3A Illustrated light path 312. In path tracing, the path is reversed so that light received by the camera can be traced back to the light source, including possible reflections at different sampling points.

[0067] In some examples, the Monte Carlo method randomly samples the integrand of Equation 1. The Monte Carlo estimator of Equation 1 for individual pixel i can be expressed as follows:

[0068]

[0069] In Equation 2, N is the number of samples per pixel. For example, image 340 is generated using 8 samples per pixel or 8SPP, also as Figure 3B As shown. Therefore, in this case, N=8. Figure 3A 、 Figure 3B The ground truth scene 300 is generated using a large number of SPPs (such as 4096 SPPs) and can be used as ground truth, for example, to train various machine learning modules discussed herein.

[0070] In Equation 2, L i is the final radiation output of each pixel, F(x) represents the integrand of Equation 1, X j is a random variable drawn from the probability density function p(x), and p(Xj) is a random variable selected from j The probability of X. j The domain can be the upper hemisphere of direction Ω+, but can also be the set of all surface points, depending on the parameterization of the plotting equations. For example, refer to Figure 3C , where pixel Y24 is sampled with 8 sampling points, the factor F(X1) / p(X1) is calculated for sample point S_1_24, the factor F(X2) / p(X2) is calculated for sample point S_2_24, and so on. These factors are then summed and divided by 8 to provide the average of the samples according to Equation 2 to generate the final radiant output L for pixel Y24 Y24 Ω+ represents a set of large numbers with nearly infinite dimensions, which makes it difficult to solve Equation 1 using limited computing power. In contrast, the number of sampling points N in Equation 2 is finite, allowing Equation 2 to be calculated.

[0071] Therefore, Equation 2 can be used to calculate Equation 1 as follows:

[0072] L(x,ω0)=L e (x,ω0)+L i (x, ω0) Equation 1a

[0073] where L in Equation 1a i (x, ω0) is calculated using the Monte Carlo method expressed by Equation 2.

[0074] The sample size N of Equation 2 indicates the quality or amount of noise within the image drawn according to Equation 2 and Equation 1a. For example, Figure 3A and Figure 3B The ground truth image 300 can be generated using relatively high values ​​of N (such as hundreds or thousands). However, solving Equation 2 for each pixel of the scene (such as with very high values ​​of N) is computationally intensive and time consuming. In contrast, Figure 3B The image 340 is generated using a relatively small value of N in Equation 2 (such as 8).

[0075] As further discussed in detail herein, Equation 2 and Equation 1a are used to render a noisy image, for example, by using a relatively low value for the sample size N. For example, 8SPP is used in Equation 2 and the Monte Carlo method to render a noisy image. Subsequently, various techniques discussed herein are used to reduce noise in the noisy image and generate a noise-free (or denoised) image such as Figure 3B A segment of image 344 is shown.

[0076] In the context of image denoising, a “sample” is defined by Each sampling point (such as Figure 3C The illustrated sampling points S_1_24, S_2_24, etc. have corresponding samples represented by the above. Each sampling point is associated with a corresponding plurality of radiation features and a corresponding plurality of auxiliary features. For example, the sampling point S i (i) The corresponding radiation vector L including multiple radiation features si , and (ii) a corresponding auxiliary vector A including multiple auxiliary features si associated.

[0077] For example, the radiation vector L si Including sampling point S i The radiation characteristics of the sampling point S i The properties or characteristics of the incoming light radiating at the surface. In some examples, the incoming light radiance at a sample point is classified into different categories, such as (i) direct lighting and diffuse reflection, (ii) direct lighting and specular reflection, (iii) indirect lighting and diffuse reflection, and (iv) indirect lighting and specular reflection. These categories are generated by defining a Cartesian product between (direct lighting, indirect lighting) and (diffuse reflection, specular reflection), which leads to the example categories discussed above.

[0078] Sampling point S i The radiation vector L si The direct illumination component includes the direct illumination component from the light source and irradiating the sampling point S i In contrast, the sampling point S i The radiation characteristics L si The indirect illumination consists of reaching the sampling point S after being reflected from one or more surfaces. i The light on.

[0079] Reflection from a smooth, flat surface (such as a mirror) is called specular reflection. For a single incoming beam, the reflected beam is specularly reflected in a specific single direction. In some examples, the sampling point S i The radiation vector L si Including sampling point S i The specular component of the light received at .

[0080] In contrast to specular reflection, if the surface is rough (such as paper, wall), diffuse reflection occurs. In diffuse reflection, the reflected light is scattered in multiple directions. In some examples, the sampling point S i The radiation vector L si Also included is the sampling point S i The diffuse reflection component of the light received at .

[0081] Therefore, considering the use case discussed above, where the sampling point S i The incoming light radiation on is classified into these categories, and the radiation vector Ls i It includes the red (R), blue (B), and green (G) components of (i) direct lighting and diffuse reflection, (ii) direct lighting and specular reflection, (iii) indirect lighting and diffuse reflection, and (iv) indirect lighting and specular reflection. i , radiation vector L si is defined as:

[0082] L si =[R direct diffuse , G direct diffuse , B direct diffuse , R direct specular , G direct specular , B direct spcular , R indirect diffuse , G indirect diffuse , B indirect diffuse , R indirect specular , G indirect specular , B indirect specnlar Equation 3

[0083] where R direct diffuse is the R or red component of direct illumination and diffuse reflection, G direct diffuse is the G or green component of direct illumination and diffuse reflection, etc. Therefore, the radiation vector L si includes 12 floating point numbers or 12 values, such as 4 RGB values ​​representing 4 corresponding types of illumination and reflection (e.g., (i) direct illumination and diffuse reflection, (ii) direct illumination and specular reflection, (iii) indirect illumination and diffuse reflection, and (iv) indirect illumination and specular reflection), although in some other examples, the radiation vector includes one or more other types of radiation characteristics. Therefore, in some examples, the sampling point S i The radiation vector L si is the characterization sampling point S i In some embodiments, Figure 1 and Figure 2 The illustrated feature identification module 103 estimates the radiation vector of each sampling point. Figure 3C The sampling points S_1_24 and S_2_24, the radiation vectors are represented as L_1_24 and L_2_24 respectively.

[0084] Therefore, during rendering, incoming radiation is divided into direct and indirect illumination, as well as diffuse and specular reflections. Since direct illumination and specular reflections typically have higher frequency content (e.g., relative to indirect illumination and / or diffuse reflections), this classification provides the machine learning module (which is discussed later in this article) with a better understanding of the spatial frequency components of the input radiant light.

[0085] In some examples, the auxiliary vector A si Including sampling point S i Auxiliary features of Sampling point S i The auxiliary features include uniquely defining or characterizing the sampling point S i Different features or attributes of . Auxiliary features include sampling points S i The inherent characteristics of (such as roughness) and the sampling point S i Characteristics relative to the camera's position (such as distance or "depth" from the camera).

[0086] The auxiliary feature defines that the incoming light will be from the sampling point S i Some example assist features are discussed herein, however, this list is not exhaustive and any other suitable assist features may be used by the system 102 discussed herein. In some examples, the assist features for a given model or scene may be included in a corresponding scene file, which may be stored in Figure 1 and Figure 2 in database 148 and / or 248.

[0087] A first example of an auxiliary feature includes a "Normal" feature, which indicates the direction normal or perpendicular to the surface of the sample point. Based on the direction of the incoming light, the direction specified by the Normal feature can specify the direction in which the light will be reflected from the sample point. The "Normal" feature includes three corresponding values, such as X, Y, and Z directions or coordinates, which in combination define the direction normal to the sample point S. i For example, database 148 includes scene files that store "normal maps" that specify the normal directions of various pixels and sample points within the scene.

[0088] A second example of an auxiliary feature includes an "albedo" feature. Albedo is a measure of diffuse reflection from a surface and is measured on a scale of 0 to 1. For example, 0 corresponds to black, which absorbs all or a large amount of incident radiation, and 1 corresponds to white, which reflects all or a large amount of incident radiation. In other words, the sample point S i The albedo is what is the light from the sampling point S iA measure of absorption or reflection. In some examples, albedo can have a three-dimensional value, having R, G, and B (red, green, and blue) components. For example, a scene file stores an "albedo map" that specifies albedo values ​​for various pixels and sample points within the scene.

[0089] A third example of an auxiliary feature includes a "depth" feature. i The depth is the distance from the virtual camera to the sampling point S i It is important to note that when the camera position changes, the scene is captured from a different position, which will also change the depth features.

[0090] A fourth example of an auxiliary feature includes a "camera space position" feature. For example, a scene is generated using camera space coordinates comprising X, Y, and Z axes, where the camera forms the origin of the coordinate system. Therefore, the sampling point S i With X, Y and Z coordinates, which specify the sampling point S i relative to the camera's position. It can be noted that depth features can be derived from camera space position features. Therefore, in some examples, both depth and camera space position features are used to denoise the scene, while in some other examples, only one of the depth and camera space position features is used.

[0091] Another example of auxiliary features includes as sampling point S i For example, if the sampling point S i is the light source, then the sampling point S i will have a high value for this feature.In some examples, the emission feature is a binary indicator indicating whether the sampling point is an emission or not.

[0092] Other examples of auxiliary features include metallic features and transmissive features. A metallic feature is a binary indicator that indicates whether a surface is metallic. For example, a metallic surface would be indicated using a first value of the metallic feature, while a plastic or glass surface would be indicated using a second value of the metallic feature. A transmissive feature is a binary indicator that indicates whether a surface is transmissive (e.g., how much light passes through the surface). In some examples, the transmissive and metallic features allow the machine learning module to distinguish between reflection and refraction, and between dielectrics and conductors. Another example of an auxiliary feature includes a roughness feature that varies between 0 and 1 to indicate that the sample point S i roughness.

[0093] Thus, some example auxiliary features discussed above include normal features, albedo features, depth features, camera space position features, emissive features, metallic features, transmittance features, and roughness features. i , auxiliary vector A si is formed (e.g. by Figure 1 and Figure 2 The feature identification module 103) is:

[0094] A si =[X normal feature , Y normal feature , Z normal feature , R albedo , G albedo , B albedo , Depth, X camera-space position , Y camera-space position , Z camera-space position ,Emissive,Metallic,Transmissive,Roughness]…Equation 4

[0095] Therefore, with the sampling point S i The corresponding auxiliary vector A si There are fourteen example values: X, Y, Z coordinates of the normal feature; R, G, B values ​​of the albedo feature; depth feature; X, Y, Z coordinates of the camera space position feature; emissive feature; metallic feature; transmissive feature; and roughness feature. Therefore, in this example embodiment, the sample point corresponding to S i Auxiliary vector A si is a fourteen-dimensional vector.

[0096] Each sampling point Si has a corresponding auxiliary vector A si For example, for Figure 3C The sampling point S_1_24 of pixel Y24 corresponds to the auxiliary vector A_1_24. Similarly, for Figure 3C The sampling point S_2_24 of pixel Y24 corresponds to the auxiliary vector A_2_24, and so on.

[0097] As discussed, in some embodiments, the sampling point S i The radiation vector L si Is the sampling point S i In some such embodiments, the twelve-dimensional vector corresponding to the characteristics of the incoming light at S i Auxiliary vector A si Is a fourteen-dimensional vector representing the characteristics of the sampling point. Therefore, with the sampling point S i The corresponding radiation vector L si and auxiliary vector A si The combination can be used to calculate the sampling point S i The amount of light radiated. Therefore, the radiation vector L si and auxiliary vector A si The combination of can be used to generate realistic scenes, where objects in the image have lighting effects similar to real life. System 102 and / or 103 uses the radiation vector L siand auxiliary vector A si The noisy image 340 is denoised, where the noisy image 340 is initially rendered using a low SPP (eg, 8 SPP) Monte Carlo method, as will be discussed further.

[0098] Figure 4 The radiation eigenvector generation module 104 (also referred to herein as module 104) according to some embodiments of the present disclosure is illustrated. Figure 1 and Figure 2 ), which is configured to receive a radiance vector for each of a plurality of sampling points for a plurality of pixels of a scene, and generate a final radiance vector for each of the plurality of pixels, wherein the final radiance vector can be used to denoise a noisy image of the scene.

[0099] The input to module 104 is the radiance vector 414 for each of a plurality of sampling points of a plurality of pixels of the scene to be rendered. For example, as previously discussed with respect to Equation 3, the sampling point S i The radiation vector L si Is the sampling point S i The twelve-dimensional vector of characteristics of the incoming light on . Figure 3C and Figure 4 For example, for pixel Y24, the sampling points are S_1_24, S_2_24, ..., S_8_24; and for pixel Y42, the sampling points are S_1_42, S_2_42, ..., S_8_42. Module 104 receives radiation vector 414. For example, for pixel Y24 and for sampling points S_1_24, S_2_24, ..., S_8_24, module 104 receives radiation vectors L_1_24, L_2_24, ..., L_8_24, respectively. Similarly, for example, for pixel Y42 and for sampling points S_1_42, S_2_42, ..., S_8_42, module 104 receives radiation vectors L_1_42, L_2_42, ..., L_8_42, respectively.

[0100] In some embodiments, module 104 outputs a final radiance feature vector 422 (also referred to simply as a "radiance feature vector 422") for each pixel of the scene. For example, module 104 outputs a final radiance feature vector F_R24 for pixel Y24, a final radiance feature vector F_R42 for pixel Y42, and generally a final radiance feature vector F_Ri for pixel Yi. The letter "F" in the final radiance feature vector F_Ri indicates that this is a "final" radiance feature vector. The letter "R" in the final radiance feature vector F_Ri indicates that this is a final "radiance" feature vector.

[0101] Module 104 includes a neural network module 410 (also referred to herein as "module 410"), which is also referred to as the sample embedding stage of module 104. The neural network module 410, including the sample embedding stage, encodes each noise input data (e.g., radiance vector 414) for each sample for sample embedding. Specifically, for a given pixel, module 410 receives the radiance vector 414 associated with the sample point. Based on the radiance vector 414 associated with the given pixel, module 104 generates a corresponding intermediate radiance feature vector 418 for the pixel.

[0102] The neural network module 410 is a machine learning module with adjustable or learnable weights that are tuned during the training phase of the module 410. The training phase may use ground truth images such as Figure 3A 300 for training the weights of module 410.

[0103] For example, for the eight sampling points of pixel Y24, module 410 receives the corresponding radiation vectors L_1_24, L_2_24, ..., L_8_24, respectively. Module 410 generates an intermediate radiation eigenvector I_R24 for pixel Y24. Module 410 essentially performs some kind of averaging on the radiation vectors L_1_24, L_2_24, ..., L_8_24 to generate the corresponding intermediate radiation eigenvector F_R24—however, this is not a simple averaging function. Instead, module 410 is trained to perform an intelligent weighted averaging to generate the intermediate radiation eigenvector I_R24 from the eight radiation vectors L_1_24, L_2_24, ..., L_8_24.

[0104] It can be noted that the intermediate radiation eigenvector I_R24 of pixel Y24 is generated by the radiation vectors L_1_24, L_2_24, ..., L_8_24 associated with the sample point of pixel Y24. The intermediate radiation eigenvector I_R24 of pixel Y24 is not affected by the radiation vectors of the sample points of any neighboring pixels, such as neighboring pixel Y33. Therefore, the radiation vector of a pixel sample point cannot affect the intermediate radiation eigenvectors of neighboring pixels. In other words, for pixel i, the intermediate radiation eigenvector I_Ri is generated based on the radiation vectors of the sample points within pixel i and is not affected by any other radiation vectors of any other sample points outside pixel i.

[0105] Figure 4Also illustrated is an example implementation of module 410. In some embodiments, module 410 includes a three-stage neural network, including neural network stages 430a, 430b, and 430c. A leaky rectified linear unit (ReLU) 434 is between stages 430a and 430b, and another leaky ReLU 434 is between stages 430b and 430c. For example, ReLU 434 introduces nonlinearity into module 410.

[0106] Stages 430a, 430b, and 430c encode the corresponding radiation vector for each sample point. For example, stages 430a, 430b, and 430c detect patterns in the data and determine how the radiation vectors for the sample points within a particular pixel should be combined or averaged to output the intermediate radiation feature vector 418 for the particular pixel. In some embodiments, stages 430a, 430b, and 430c include adjustable or learnable weights that are tuned during a training phase of module 410. For example, module 410 is trained to identify how multiple radiation vectors 414 for various sample points of a particular pixel should be processed and averaged to generate the intermediate radiation feature vector 418 for the particular pixel. By way of example only, module 410 detects patterns in the radiation vectors and determines how the radiation vectors should be combined or averaged to form the intermediate radiation feature vector 418.

[0107] exist Figure 4 In an example implementation of , each intermediate radiation eigenvector 418 is a 64-dimensional vector. The first stage 430a receives the 12-dimensional radiation vectors 414 and converts each vector into a corresponding 64-dimensional vector. Stages 430b, 430c also operate in a 64-dimensional space and output a 64-dimensional intermediate radiation eigenvector 418 for each pixel. Thus, stages 430a, 430b, 430c transform the 12-dimensional radiation vectors 414 into a 64-dimensional output space. As discussed, stages 430a, 430b, 430c consider the sampling points within each pixel, and there is no cross-influence between pixels (e.g., a sampling point of a first pixel affects other sampling points of the first pixel, but does not affect sampling points of other pixels). Thus, there is no spatial propagation of radiation characteristics between adjacent or neighboring pixels.

[0108] The output 432 of stage 430c is a 64-dimensional vector E for each sample point si Therefore, each pixel will have 8 different outputs 432. For example, for pixel Y24, the outputs 432 will be E_1_24, E_2_24, ..., E_8_24, corresponding to the 8 underlying sampling points (e.g., assuming 8SPP, as compared to Figure 3C discussed).

[0109] Thus, as discussed, the sample embedding component Ψ (e.g., stages 430a, 430b, 430c) receives the radiance vector L for each sampling point s in each pixel i. si , and encoded as sample code E si , making Ψ(L si )=E si , where E si is the output 432 of stage 430c. Here, as an example, Figure 4 In the example, d is chosen to be 64.

[0110] The averaging module 434 then averages the corresponding 8 outputs E for a particular pixel. si , to generate the corresponding intermediate radiance eigenvector 418 for the pixel. For example, for pixel i, where the sampling point is generally denoted by s, the output of the averaging module 434 is:

[0111]

[0112] In Equation 5, the value used for 8SPP is Figure 3C In an example implementation of , N is 8. Thus, according to Equation 5, for pixel Y24, the averaging module 434 averages the outputs E_1_24, E_2_24, ..., E_8_24 to generate the corresponding intermediate radiance eigenvector I_R24 for pixel Y24. In some examples, averaging forces the sample order in the pixel to not change the output, and permutation invariance is preserved.

[0113] Intermediate radiance eigenvector 418 indicates the radiance vector of Equation 3. For example, for a particular pixel, intermediate radiance eigenvector 418 is the average of the encoded versions of the corresponding radiance vectors of the corresponding sampling points, where the encoding is performed by stages 430a, 430b, and 430c using the trainable weights of these neural network stages. Therefore, the intermediate radiance eigenvector 418 for a pixel is an intelligent average of the radiance vectors of the corresponding sampling points.

[0114] Refer again Figure 4 Module 104 includes another neural network module 411 (also referred to herein as module 411), which is also referred to herein as the spatial propagation stage. For each pixel of the scene, module 411 receives the corresponding intermediate radiance feature vector 418 from module 410. For example, module 411 receives the intermediate radiance feature vector I_R24 for pixel Y24, the intermediate radiance feature vector I_R42 for pixel Y42, and generally the intermediate radiance feature vector I_Ri for pixel i.

[0115] In some embodiments, module 411 generates for each pixel a corresponding final radiance eigenvector 422. For example, module 411 generates a final radiance eigenvector F_R24 for pixel Y24, a final radiance eigenvector F_R42 for pixel Y42, and generally a final radiance eigenvector F_Ri for pixel i.

[0116] As will be discussed in further detail, each final radiation eigenvector comprises a plurality of segments. Figure 4 and Figure 7A In the example implementation of Figure 7A A three-stage denoising operation is illustrated), each final radiance eigenvector is divided into 3 segments. That is, the number of segments of the final radiance eigenvector corresponds to the number of denoising stages to be used.

[0117] For example, in Figure 4 In the example of , each radiation eigenvector is 15-dimensional (e.g., includes 15 values), and the first five values ​​form a first segment of the radiation eigenvector, the next five values ​​form a second segment of the radiation eigenvector, and the last five values ​​form a third segment of the radiation eigenvector. Figure 4 As shown, the radiation eigenvector F_R24 is divided into [F_R24_1, F_R24_2, F_R24_3], and the radiation eigenvector F_R42 is divided into [F_R42_1, F_R42_2, F_R42_3]. Each segment of the radiation eigenvector is used to generate a corresponding kernel, as will be described herein with respect to Figure 6A 、 Figure 6B and Figure 6C discussed in further detail.

[0118] In some embodiments, module 411 also generates three corresponding scaling factors a for each pixel i: i 1 、a i 2 and a i 3 The number of scaling factors generated for each pixel corresponds to the number of denoising stages to be used (e.g. Figure 7A As illustrated in the example of , 3 stages are used). For example, module 411 generates a scaling factor a for pixel Y24 24 1 、a 24 2 and a 24 3 These scaling factors introduce asymmetries during kernel generation, as discussed in further detail herein.

[0119] Thus, as discussed, the spatially propagated component Φ (eg, module 411) receives the median radiation vector I_R for pixel i i , and output three different final radiation eigenvectors F_Ri_k (where k is 1, 2 and 3) for pixel i, such that Φ(I_R i )=[F_Ri_1,F_Ri_2,F_Ri_3,a i 1 ,a i 2 ,a i 3 ]. Here, F_Ri_1 for pixel Y24 is F_R24_1, F_Ri_2 for pixel Y24 is F_R24_2, and so on.

[0120] In some embodiments, module 411 cross-correlates the radiance signatures of adjacent pixels to generate a final radiance signature vector. For example, module 411 determines how the radiance value associated with a pixel may contribute to the contributions of adjacent pixels. For example, assuming Figure 3B 344 ) and an adjacent illuminated segment 346 of the wall of scene 300. Adjacent pixels within segment 346 should have somewhat similar radiance properties (e.g., similar radiance vectors), and thus, module 411 can learn the radiance properties of pixels within segment 346 from the radiance properties of neighboring pixels within the same segment 346. Similarly, adjacent pixels within segment 348 should have somewhat similar radiance properties (e.g., similar radiance vectors), and thus, module 411 can learn the radiance properties of pixels within segment 348 from the radiance properties of neighboring pixels within the same segment 348. However, module 411 may not be able to learn the radiance properties of pixels within segment 346 from the radiance properties of neighboring pixels within adjacent segments 348. Similarly, module 411 may not be able to learn the radiance properties of pixels within segment 348 from the radiance properties of neighboring pixels within adjacent segments 346.

[0121] Thus, in some embodiments, module 411 spatially propagates the radiance property of a first pixel to one or more neighboring pixels, for example, if the first pixel and the one or more neighboring pixels have similar radiance properties and / or are located in the same or similar segments of the image. However, if the first pixel is along or near an edge (e.g., near the edge of segments 346 and 348), then propagation of the radiance property may not occur across the edge of the two segments. Thus, the radiance eigenvectors of neighboring pixels within segment 346 may affect each other but may not substantially affect pixels in segment 348. Similarly, the radiance eigenvectors of neighboring pixels within segment 348 may affect each other but may not substantially affect pixels in segment 346.

[0122] exist Figure 4 In the example implementation of , module 411 includes stages 440a, 440b, ..., 440i, for example nine different stages. Figure 4 Each stage 440 in includes a number that represents the capacity or number of layers within the corresponding stage. By way of example only, stage 440a includes 32 layers, stage 440b includes 48 layers, stage 440c includes 64 layers, stage 440d includes 80 layers, stage 440e includes 96 layers, stage 440f includes 80 layers, stage 440g includes 64 layers, stage 440h includes 48 layers, and stage 440i includes 15 layers. In some embodiments, two (2x) max pooling is performed after one or more (e.g., each) of stages 440a, 440b, 440c, and 440d. In some embodiments, two (2x) linear upsampling is performed after one or more (e.g., each) of stages 440e, 440f, 440g, and 440h.

[0123] Stage 440 includes trainable weights that were tuned during the training stage of module 411. Thus, once trained, module 411 can cross-correlate the radiance feature vectors of adjacent pixels to obtain a better estimate of the radiance properties of the various pixels. As discussed, cross-correlation takes into account the location of the pixel—if a pixel is on or near the edge of a segment, the radiance properties of that pixel are unaffected (or unaffected) by the radiance properties of the pixels on the other side of the edge.

[0124] exist Figure 4 In an example implementation of , module 411 transforms a 64-dimensional intermediate radiance eigenvector 418 of a pixel into a 15-dimensional final radiance eigenvector 422 of the pixel. During the transformation, the radiance eigenvector may be affected by the radiance properties of one or more neighboring pixels with similar light conditions.

[0125] For example, assume that the intermediate radiation eigenvectors for pixels Y24, Y33, and Y45 are I_R24, I_R33, and I_R45, respectively. Module 411 generates final radiation eigenvectors F_R24, F_R33, and F_R45 for pixels Y24, Y33, and Y45, respectively. Thus, module 411 transforms the 64-dimensional intermediate radiation eigenvector I_R24 for pixel Y24 into a 15-dimensional final radiation eigenvector F_R24 for pixel Y24. During the transformation process, the final radiation eigenvector F_R24 may be affected by the radiation properties of one, both, or neither of the intermediate radiation eigenvectors I_R33 and I_R45 for neighboring pixels Y33 and Y45, respectively.

[0126] Figure 5The auxiliary feature vector generation module 108 (also referred to herein as module 108, in FIG. 1 ) of the system 102 and / or 202 according to some embodiments of the present disclosure is illustrated. Figure 1 and Figure 2 ), which is configured to receive a corresponding auxiliary vector for each of a plurality of sampling points for a plurality of pixels of a scene, and generate a corresponding final auxiliary vector for each of the plurality of pixels, wherein the final auxiliary vector can be used to denoise a noisy image of the scene.

[0127] Recall that Equation 3 discussed earlier in this paper involves the radiation vector L si , while Equation 4 involves the auxiliary vector A si . Radiation vector L si Entered Figure 4 Module 104, and the auxiliary vector A si Entered Figure 5 Module 108. Therefore, Figure 4 Module 104 will process the radiation related information, and Figure 5 Module 108 will process the auxiliary feature related information. Apart from this basic difference, there are some structural similarities between module 104 and module 108, as will be discussed further.

[0128] As discussed, Figure 5 The input to the module 108 is the auxiliary vector 514 for each of the plurality of sampling points of the plurality of pixels of the scene to be drawn. For example, as previously discussed, the sampling point S i Auxiliary vector A si For example, it represents the sampling point S i Surface characteristics and sampling points S i The fourteen-dimensional vector of the location relative to the camera position, as discussed with respect to Equation 4. Figure 3C and Figure 5 , for example, for pixel Y24, the sampling points are S_1_24, S_2_24, ..., S_8_24; and for pixel Y42, the sampling points are S_1_42, S_2_42, ..., S_8_42. Module 108 receives auxiliary vectors 514. For example, for pixel Y24 and for sampling points S_1_24, S_2_24, ..., S_8_24, module 108 receives auxiliary vectors A_1_24, A_2_24, ..., A_8_24, respectively. Similarly, for pixel Y42 and for sampling points S_1_42, S_2_42, ..., S_8_42, module 104 receives auxiliary vectors A_1_42, A_2_42, ..., A_8_42, respectively. The generation of auxiliary vectors has been discussed with respect to Equation 4.

[0129] In some embodiments, module 108 outputs a corresponding final auxiliary feature vector 522 for each pixel of the scene. For example, module 108 outputs a final auxiliary feature vector F_A24 for pixel Y24, a final auxiliary feature vector F_A42 for pixel Y42, and generally a final auxiliary feature vector F_Ai for pixel i. The letter "F" in the final auxiliary feature vector F_Ai indicates that this is a "final" auxiliary feature vector. The letter "A" in the final auxiliary feature vector F_Ai indicates that this is a final "auxiliary" feature vector.

[0130] Module 108 includes a neural network module 510 (also referred to herein as "module 510"), which is also referred to as the sample embedding stage of module 108. The neural network module 510, including the sample embedding stage, encodes each noise input data (e.g., auxiliary vector 514) for each sample for sample embedding. Specifically, for a given pixel, module 510 receives the auxiliary vector 514 of the associated sample points. Based on the auxiliary vector 514 of the associated sample points of the given pixel, module 108 generates a corresponding intermediate auxiliary feature vector 518 for the pixel.

[0131] For example, for the eight sampling points of pixel Y24, module 510 receives the corresponding auxiliary vectors A_1_24, A_2_24, ..., A_8_24, respectively. Module 510 generates an intermediate auxiliary feature vector I_A24 for pixel Y24. Module 510 essentially performs a sort of averaging on the auxiliary vectors A_1_24, A_2_24, ..., A_8_24 to generate the corresponding intermediate auxiliary feature vector F_A24—however, this is not a simple averaging function. Instead, module 510 performs an intelligent weighted averaging to generate the intermediate auxiliary feature vector I_A24 from the eight auxiliary vectors A_1_24, A_2_24, ..., A_8_24.

[0132] Note that the intermediate auxiliary feature vector I_A24 for pixel Y24 is generated using the auxiliary vectors A_1_24, A_2_24, ..., A_8_24 associated with the sample point of pixel Y24. The intermediate auxiliary feature vector I_A24 for pixel Y24 is not affected by the auxiliary vectors of any neighboring pixel sample points, such as neighboring pixel Y33. Therefore, the auxiliary vectors of a pixel sample point cannot affect the intermediate auxiliary feature vectors of neighboring pixels. In other words, for pixel i, the intermediate auxiliary feature vector I_Ai is generated based on the auxiliary vectors of the sample points within pixel i and is not affected by any other auxiliary vectors of any other sample points outside pixel i.

[0133] Figure 5Also illustrated is an example implementation of module 510 comprising a three-stage neural network, including neural network stages 530a, 530b, and 530c. A leaky ReLU 534 is located between stages 530a and 530b, and another leaky ReLU 534 is located between stages 530b and 530c. For example, ReLU 534 introduces nonlinearity in module 510.

[0134] Stages 530a, 530b, 530c encode the corresponding auxiliary vector for each sample. For example, stages 530a, 530b, 530c detect patterns in the data and determine how the auxiliary vectors should be combined or averaged to output the intermediate auxiliary feature vector 518. In some embodiments, stages 530a, 530b, 530c include adjustable or learnable weights that are tuned during the training phase of module 510. For example, module 510 is trained to identify how multiple auxiliary vectors 514 for various sample points of a particular pixel should be processed and averaged to generate the intermediate auxiliary feature vector 518 for that particular pixel.

[0135] exist Figure 5 In the example implementation of , each intermediate auxiliary feature vector 518 is a 64-dimensional vector. The first stage 530a receives the 12-dimensional auxiliary vectors 514 and converts each vector into a 64-dimensional vector. Stages 530b and 530c also operate in a 64-dimensional space and output a 64-dimensional intermediate auxiliary feature vector 518 for each pixel. Thus, stages 530a, 530b, and 530c transform the 12-dimensional auxiliary vectors 514 into a 64-dimensional output space. As discussed, stages 530a, 530b, and 530c consider the sampling points within each pixel, and there is no cross-influence between pixels (e.g., the sampling points of the first pixel affect the sampling points of the first pixel, but do not affect the sampling points of other pixels). Therefore, no auxiliary attributes are propagated across the space of pixels.

[0136] The output 532 of stage 530c is the i The 64-dimensional vector E' si Therefore, each pixel will have 8 different outputs 532 (assuming 8SPP, as compared to Figure 3C For example, for pixel Y24, the output 532 would be E"_1_24, E"_2_24, ..., E"_8_24, corresponding to 8 underlying sample points.

[0137] Thus, as discussed, the sample embedding component Ψ (e.g., stages 530a, 530b, 530c) receives the auxiliary vector s for each sample point s in each pixel i. And encoded as sample code E" si , making Ψ”(Asi )=E” si , where E' si is the output 532 of stage 530c. Here, as an example, Figure 5 In the example, d is chosen to be 64.

[0138] The averaging module 534 then averages the corresponding 8 outputs for a particular pixel to generate the pixel's corresponding intermediate auxiliary feature vector 518. For example, for pixel i, where the sampling point is generally represented by s, the output of the averaging module 534 is:

[0139]

[0140] In Equation 6, the value used for 8SPP is Figure 3C In an example implementation of , N is 8. Therefore, according to Equation 6, for pixel Y24, the averaging module 534 averages the outputs E"_1_24, E"_224, ..., E"_8_24 to generate the corresponding intermediate auxiliary feature vector I_A24 for pixel Y24. In some examples, averaging forces the sample order in the pixel to not change the output, and permutation invariance is preserved.

[0141] The intermediate auxiliary feature vector 518 indicates the auxiliary vector of Equation 4. For example, for a particular pixel, the intermediate auxiliary feature vector 518 is the average of the encoded versions of the corresponding auxiliary vectors for the corresponding sampling points, where the encoding is done by stages 530a, 530b, 530c using the trainable weights of these neural network stages.

[0142] In some embodiments, module 108 further includes another neural network module 511 (also referred to herein as module 511), which is also referred to herein as the spatial propagation stage. For each pixel of the scene, module 511 receives the corresponding intermediate auxiliary feature vector 518 from module 510. For example, module 511 receives the intermediate auxiliary feature vector I_A24 for pixel Y24 and the intermediate auxiliary feature vector I_A42 for pixel Y42.

[0143] In some embodiments, module 511 generates a corresponding final auxiliary feature vector 522 (usually represented by F_A si For example, module 511 generates a final auxiliary feature vector F_A24 for pixel Y24, a final auxiliary feature vector F_A42 for pixel Y42, and generally a final auxiliary feature vector F_Ai for pixel i.

[0144] Thus, as discussed, the spatial propagation component Φ" (eg, module 511) receives the intermediate auxiliary vector I_A for each pixel i i , and output the corresponding final radiation feature vector F_Ai of pixel i, so that Φ(I_A i )=F_Ai.

[0145] It can be noted that Figure 4 Each final radiation eigenvector of module 104 is divided into a plurality of segments (e.g., the final radiation eigenvector is the concatenation of the three underlying segments of the final radiation eigenvector). In contrast, in some embodiments, the subdivision of the final auxiliary eigenvector is not provided by module 108. However, in some other embodiments, the final auxiliary eigenvector may also be subdivided into three corresponding segments, each segment being used to generate a corresponding kernel. It may be noted that Figure 4 Module 411 generates the scaling factor aj. In contrast, in some embodiments, Figure 5 The module 511 may not generate any such scaling factors.

[0146] In some embodiments, module 511 cross-correlates the auxiliary features of adjacent pixels, similar to the Figure 4 For example, module 511 determines how the auxiliary feature value associated with a pixel may contribute to the neighboring pixels. For example, assuming Figure 3B 346 and 348. The shadow segment 348 of the wall of the scene 300 (marked in image 344) and the adjacent illuminated segment 348 of the wall. Segments 346 and 348 are both part of the wall and can have similar auxiliary features. However, the auxiliary features of the wall are different from the pictures hung on the wall.

[0147] Therefore, in some instances, adjacent pixels within segment 346 and segment 348 should have somewhat similar auxiliary feature attributes (e.g., similar auxiliary vectors), and thus, module 511 may learn the auxiliary feature attributes of the pixels within segment 346 from the auxiliary feature attributes of one or more adjacent pixels within segment 346 and / or segment 348. However, module 511 may not be able to learn the auxiliary feature attributes of the pixels within segment 346 from the auxiliary feature attributes of adjacent pixels of a picture hanging on the wall.

[0148] Thus, in some embodiments, module 511 spatially propagates the auxiliary feature attribute of a first pixel to one or more neighboring pixels, for example, if the first pixel and the one or more neighboring pixels have similar auxiliary feature attributes (e.g., when each of these pixels is part of a wall). However, if the first pixel is on a wall and the second pixel is on a picture hanging on the wall, then propagation of the auxiliary feature attribute may not occur between the first pixel and the second pixel. Thus, the auxiliary feature vectors of neighboring pixels within segments 346 and / or 348 may affect each other but may not substantially affect pixels of the picture hanging on the wall.

[0149] exist Figure 5 In the example implementation of , module 511 includes stage 540a and stage 540b, eg, two different stages. Figure 5 Each stage 540 in comprises a number representing the capacity or number of layers within the stage. For example, stage 540a comprises 64 layers, and stage 540b comprises 4 layers. Stage 540 comprises trainable weights that are tuned during the training phase of module 511. Thus, once trained, module 511 can cross-correlate the auxiliary feature vectors of one or more adjacent pixels to obtain a better estimate of the auxiliary properties of various pixels.

[0150] Thus, module 511 transforms the 64-dimensional intermediate auxiliary feature vector 518 of the pixel into a 4-dimensional final auxiliary feature vector 522 of the pixel. During the transformation, the auxiliary feature vector may be affected by auxiliary properties of one or more neighboring pixels.

[0151] For example, assume that the intermediate auxiliary feature vectors for pixels Y24, Y33, and Y45 are I_A24, I_A33, and I_A45, respectively. Module 511 generates final auxiliary feature vectors F_A24, F_A33, and F_A45 for pixels Y24, Y33, and Y45, respectively. Thus, module 511 transforms the 64-dimensional intermediate auxiliary feature vector I_A24 for pixel Y24 into a 4-dimensional final auxiliary feature vector F_A24 for pixel Y24. During the transformation process, the final auxiliary feature vector F_A24 may be affected by the radiometric properties of one, both, or neither of the intermediate auxiliary feature vectors I_A33 and I_A45 for neighboring pixels Y33 and Y45, respectively.

[0152] It can be noted that the number of stages in module 511 is two, which is less than Figure 4 The nine stages included in module 104 and module 411 are as follows: Figure 5 The spatial propagation module 511 of the module 108 has a Figure 4 104 of the spatial propagation module 411. In some instances, the variation in radiance data between adjacent pixels may be greater than the variation in auxiliary feature attributes between adjacent pixels. For example, pixels throughout a wall (such as the wall illustrated in scene 300) may have somewhat similar auxiliary feature attributes. However, different sections of the same wall may have different radiance attributes (e.g., some sections of the wall are in shadow while some other sections of the wall are illuminated). Therefore, in some instances, detecting radiance attribute data patterns in pixels may be more challenging than detecting auxiliary feature data patterns. Therefore, in some embodiments, Figure 5 The spatial propagation module 108 is larger than the spatial propagation module 511. Figure 4 The spatial propagation module 411 of the module 104 has a smaller number of stages.

[0153] Figure 6A 、 Figure 6B 、 Figure 6C Illustrated is the generation of corresponding kernels according to some embodiments of the present disclosure (e.g., by Figure 1 and Figure 2 The kernel generation module 112 is used as a filter to denoise the noisy image. As previously explained, a kernel is a two-dimensional convolution matrix, or a mask or filter, which includes a small matrix (e.g., small relative to the image to which it is applied). Such kernels can be used in image processing applications, such as for blurring, sharpening, embossing, edge detection, and / or other image processing applications, such as denoising in the present disclosure.

[0154] For example, refer to Figure 6A , which illustrates the Figure 3C The kernel 620a of the target pixel Y24 is generated. For example, the stride of the kernel 620a is 2 0 =1. Thus, when generating kernel 620a, every pixel that is 1 pixel away from the target pixel Y24 is taken into account. For example, assume that the kernel includes a pixel matrix with a stride X and the target pixel at its center. Neighboring pixels within the kernel that are X pixels away from the target pixel are assigned a non-zero weight, while neighboring pixels within the kernel that are not X pixels away from the target pixel are assigned a zero weight. For example, Figure 6A The kernel 620a of φ includes a 3x3 window or matrix with the target pixel Y24 in the middle and 8 pixels adjacent to the target pixel. For example, the adjacent pixels Y33, Y25 and Y15 are in Figure 6A Since the kernel stride is 1, all pixels in kernel 620 that are 1 pixel away from the target pixel Y24 are marked in Figure 6A are all shaded (by contrast, Figure 6B and Figure 6C Only some pixels in the kernel are shaded), which means that all neighboring pixels are taken into account when generating kernel 620a.

[0155] To generate kernel 620a, a final radiance eigenvector F_Ri_1 (eg, a first segment of the final radiance eigenvector F_Ri) is received from module 104 for each shadow pixel of kernel 620a. Figure 4 As discussed, the final radiation eigenvector F_Ri is divided into three segments: F_Ri_1, F_Ri_2, and F_Ri_3. The first segment F_Ri_1 of the final radiation eigenvector F_Ri for the shaded pixel i (e.g., segment F_R24_1 for pixel Y24) is used to generate the kernel 620a. In addition, the final auxiliary eigenvectors F_Ai of the pixels shaded within the kernel 620a are also used to generate the kernel 620a. Scaling factor aj 1 (Depend on Figure 4 Module 104 output) is also used to generate kernel 620a.

[0156] Each shadow pixel within the kernel (except the center target pixel) is assigned a corresponding weight. For example, if the center target pixel is i and another pixel within the kernel is j, then the weight of pixel j is represented by w1(i, j). For example, for pixel Y15, the corresponding weight is w1(24, 15); for pixel Y33, the corresponding weight is w1(24, 33), and so on. Therefore, kernel 620a includes multiple weights w1(i, j) corresponding to multiple pixels. The number "1" after the "w" in the weight means that these are the weights for the first denoising stage (or count k=1), as will be discussed further.

[0157] In some embodiments, system 102 and / or system 104 ( Figure 1 and Figure 2 The kernel generation module 112 of FIG. 1 (shown) uses the per-pixel final radiance and auxiliary eigenvector pairs to generate kernel weights w1(i, j) as follows:

[0158]

[0159] In Equation 7, the term e is Euler's number, a mathematical constant approximately equal to 2.71828, and the operator ||...|| 2 Outputs the L2 norm or distance. For example, the term in the exponential ||(F_Ri_1-F_Rj_1)|| 2 is the L2 norm or L2 distance between the first segment of the final radiance eigenvector F_Ri for pixel i and the first segment of the final radiance eigenvector F_Rj for pixel j. For example, for weight w1(24, 33) corresponding to pixel Y33, the term would be ||(F_R24_1-F_R33_1)|| 2 .

[0160] Similarly, in Equation 7, the term within the exponential ||(F_Ai-F_Aj)|| 2 is the L2 norm or L2 distance between the final auxiliary feature vector F_Ai of pixel i and the final auxiliary feature vector F_Aj of pixel j. For example, for the weight w1(24, 33) corresponding to pixel Y33, the term will be ||(F_A24-F_A33)|| 2 .

[0161] Note that in Equation 7, the terms within the exponent are symmetric, i.e., the contribution of pixel i to pixel j and the contribution of pixel j to pixel i will be the same. Therefore, in the absence of a scaling factor In this case, the weights wl(i, j) and w1(j, i) will be the same.

[0162] For example, suppose pixel j is a bright light source, while pixel i is not. In this example, the first segments of the corresponding final radiance eigenvectors F_Ri_1 and F_Rj_1 will be very different. Therefore, in some such instances, pixel j is an outlier, for example, where pixel j with high-energy radiation creates spike noise. As a result, the final radiance eigenvector for this pixel tends to be far away from the final radiance eigenvectors of regular neighboring pixels in the learned feature space. This results in the exponential term ||(F_Ai-F_Aj)|| 2 , which results in the weights w1(i,j) and w1(j,i) being almost zero (assuming the scaling factor is not used). In other words, if the scaling factor If not used, the outlier pixels will have no chance to gain radiance contribution from their neighbors since the corresponding weights will be very close to zero.

[0163] Therefore, in some embodiments, the scaling factor Used to break the symmetry between weights w1(i,j) and w1(j,i). Scaling factor It can be a measure of the reliability of pixel aj to its neighboring pixels. For abnormal pixels, such as pixels that are light sources or high-radiance pixels (e.g., have a radiation value above a threshold) and whose neighboring pixels are low-radiance pixels, the scaling factor For example, assume that pixel Y33 is an abnormal pixel with high radiation value (e.g., radiation value higher than the threshold), while the adjacent pixel Y24 is a low radiation value (e.g., radiation value lower than the threshold). Assigned by module 411 a very low value, while the scaling factor The slightly higher value of the rule is assigned by module 411. Now, due to the scaling factor Very low, using a scaling factor in its calculations The weight w1(24,33) is very low or almost zero. However, due to the scaling factor is not low, so a scaling factor is used in its calculation The weight w1(33,24) is not too low. That is, due to the scaling factor and The difference in weight w1(33,24) is higher than the weight w1(24,33) (e.g., where w1(24,33) is almost zero). Therefore, the abnormal pixel Y33 receives a non-zero contribution from the regular pixel Y24, but the abnormal pixel Y33 basically cannot contribute to the neighboring regular pixel Y24. Without the scaling factor In this case, the weights w1(33,24) and w1(24,33) will be symmetric.

[0164] Thus, as discussed, referring again to Equation 7, the weight w1(i,j) can be thought of as a factor or weight that specifies how much the radiance of pixel j will contribute to the radiance of pixel i while denoising the noisy image. Since the noisy image has a small number of sample points (e.g., 8 SPP), this weighting and cross-influence of the radiance properties between adjacent pixels increases the effective number of sample points, thereby reducing the noise in the image. The contribution of pixel j to pixel i will be convolved with the noisy image via kernel 620a, as further discussed herein with respect to Figure 7A discussed.

[0165] Thus, as discussed, kernel 620a is generated by calculating weights w1(i,j) for each shadow pixel j within the kernel, where i is the center pixel of kernel 620a, which is the target pixel. For the center pixel, no weight is calculated because the value of weight w1(i,i) is simply a i .

[0166] Note that kernel 620a is calculated for target pixel i, which in this example is pixel Y24. Kernel 620a is repeatedly calculated with other pixels acting as target pixels, e.g. Figure 7B As shown in the figure. For example, another kernel is generated in a similar manner with pixel Y25 as the target pixel at the center of the kernel. Furthermore, yet another kernel is generated in a similar manner with pixel Y33 as the target pixel at the center of the kernel. This process continues until 3x3 kernels have been generated for all target pixels of the image to be drawn. Thus, the stride is 2 0 A subset of kernels of 1 or 2 is generated, where the target pixel of each kernel is the corresponding pixel in the image. Figure 7B The figure shows an example of 3 such kernels from a subset of kernels with stride 1. These kernels are used for convolution operations, as discussed later in this paper with respect to Figure 7A discussed.

[0167] Now refer to Figure 6B , which illustrates the Figure 3C The kernel 620b of the target pixel Y24 is generated. For example, the stride of the kernel 620b is 2 1 = 2. Thus, when generating kernel 620a, every pixel that is 2 pixels away from the target pixel Y24 is taken into account (e.g., shaded and has a non-zero weight). Pixels that are not 2 pixels away from the target pixel are not shaded, which means that these pixels are assigned a zero weight.

[0168] Figure 6BThe kernel 620b includes a 5x5 window or matrix with the target pixel Y24 in the middle and 24 pixels adjacent to the target pixel. For example, adjacent pixels Y33 and Y26 are in Figure 6B The pixel two pixels away from the target pixel Y24 is marked in Figure 6B is shaded, which means that only the shadow pixels contribute to the target pixel P24 located at the center of the kernel.

[0169] Each shadow pixel within the kernel (except the center target pixel) has a corresponding weight. For example, if the center target pixel is i and another pixel within the kernel is j, then the weight of pixel j is represented by w2(i,j). For example, for shadow pixel Y26, the corresponding weight is w2(24,26); while for non-shadow pixel Y33, no corresponding weight is calculated. Therefore, kernel 620b includes multiple weights w2(i,j) corresponding to multiple shadow pixels. The number "2" after the "w" in the weight means that these are the weights for the second denoising stage (or count k=2), as will be discussed further.

[0170] In some embodiments, system 102 and / or 104 ( Figure 1 and Figure 2 The kernel generation module 112 of FIG. 1 (shown) uses the per-pixel final radiance and auxiliary eigenvector pairs to generate kernel weights w2(i,j) as follows:

[0171]

[0172] It can be seen that Equation 8 is similar to Equation 7. However, unlike Equation 7 which uses the first segment of the final radiation eigenvector (e.g., F_Ri_1), Equation 8 uses Figure 4 The second segment of the final radiation eigenvector (e.g., F_Ri_2) generated by module 104 is shown in FIG. 1 . In addition, the scaling factor Unlike Equation 7, Equation 8 uses a scaling factor In some embodiments, the scaling factor for pixel j is and have the same value, while in some other embodiments these scaling factors are different. As discussed, the scaling factors and Depend on Figure 4 The trained neural network module 411 generates . As will be appreciated, the discussion with respect to Equation 7 applies equally to Equation 8.

[0173] Thus, kernel 620b is generated by calculating weights w2(i,j) for each shaded pixel j within the kernel, where i is the center pixel of kernel 620b, which is the target pixel. Note that kernel 620b is calculated for target pixel i, which in this example is specifically pixel Y24. Kernel 620b is repeatedly calculated with other pixels acting as target pixels. For example, another kernel is generated in a similar manner with pixel Y25 as the target pixel at the center of the kernel. Additionally, yet another kernel is generated in a similar manner with pixel Y33 as the target pixel at the center of the kernel. This process continues until 5x5 kernels have been generated for all pixels of the image to be drawn. Thus, a stride of 2 is used. 1 A subset of kernels of size 2 or 2 is generated, where the target pixel of each kernel is the corresponding pixel in the image. These kernels are used in convolution operations, as described later in this paper with respect to Figure 7A discussed.

[0174] Now refer to Figure 6C , which illustrates the Figure 3C The kernel 620c of the target pixel Y24 is generated. For example, the stride of the kernel 620c is 2 2 =4. Therefore, when generating kernel 620c, every pixel that is 4 pixels away from the target pixel Y24 is taken into account. Figures 6A to 6C Each of the kernels 620a, 620b, and 620c is generated for the same target pixel Y24.

[0175] Figure 6B The kernel 620c includes a 9x9 window or matrix with the target pixel in the middle and 80 pixels adjacent to the target pixel. For example, adjacent pixels Y20 and Y26 are in Figure 6C The pixel four pixels away from the target pixel Y24 is marked in Figure 6C 620c is shaded, which means that only the shaded pixels contribute to the destination pixel P24 located at the center of kernel 620c.

[0176] Each shadow pixel within kernel 620c (except the center target pixel) has a corresponding non-zero weight. For example, if the center target pixel is i and another pixel within the kernel is j, then the weight of pixel j is represented by w3(i,j). For example, for shadow pixel Y20, the corresponding weight is w3(24,20); while for non-shadow pixel Y26, no corresponding weight is calculated. Therefore, kernel 620c includes multiple weights w3(i,j) corresponding to multiple shadow pixels. The number "3" after the "w" in the weight means that these are the weights for the third denoising stage (or count k=3), as will be discussed further.

[0177] In some embodiments, system 102 and / or 104 ( Figure 1 and Figure 2 The kernel generation module 112 of FIG. 1 (shown) uses the per-pixel final radiance and auxiliary eigenvector pairs to generate kernel weights w3(i,j) as follows:

[0178]

[0179] It can be seen that Equation 9 is similar to Equation 7. However, unlike Equation 7 which uses the first segment of the final radiation eigenvector (e.g., F_Ri_1), Equation 9 uses Figure 4 The third segment (e.g., F_Ri_3) of the final radiation eigenvector generated by module 104 is shown in FIG. Unlike Equation 7, Equation 9 uses the scaling factor for pixel j In some embodiments, the scaling factor for pixel j is 、 and have the same value, while in some other embodiments, these scaling factors are different. As discussed, these scaling factors are determined by Figure 4 The trained neural network module 411 generates . As will be appreciated, the discussion with respect to Equations 7 and 8 applies equally to Equation 9.

[0180] Thus, kernel 620c is generated by calculating weights w3(i,j) for each shaded pixel j within kernel 620c, where i is the center pixel of kernel 620c, which is the target pixel. Note that kernel 620c is calculated for target pixel i, which in this example is specifically pixel Y24. Kernel 620c is repeatedly calculated with other pixels acting as target pixels. For example, another kernel is generated in a similar manner with pixel Y25 as the target pixel at the center of the kernel. Additionally, yet another kernel is generated in a similar manner with pixel Y33 as the target pixel at the center of the kernel. This process continues until 9x9 kernels have been generated for all pixels of the image to be drawn. Thus, a stride of 2 is used. 2 A subset of kernels of 4 or 5 are generated, where the target pixel of each kernel is the corresponding pixel in the image. These kernels with stride 4 are used for convolution operations, as described later in this paper with respect to Figure 7A discussed.

[0181] Figure 7A The diagram illustrates denoising a noisy image 340 using multiple denoising stages, where each stage uses a corresponding subset of kernels, according to some embodiments of the present disclosure. Figure 7A The various operations discussed are Figure 1 and Figure 2 For example, the noise image 340 is formed by the noise reduction image generation module 120. Figure 7A The input image in which the segment of the noise image 340 is also Figure 3B and Figure 7A As shown in the figure. Figure 3B As discussed in Equation 2 and Equation 1a, the image 340 is a noise image generated using Monte Carlo rendering, for example, 8 samples per pixel (8SPP). For example, the noise image generator module 116 of the system 102 (e.g., Figure 1 and Figure 2 ) generates a noise image 340. In an example, using 8SPP, the noise image 340 is generated according to Equation 2 and Equation 1a discussed previously herein.

[0182] like Figure 7A As shown, kernel 620a is convolved with the noisy image 340, for example, by the noise reduction image generation module 120. According to some embodiments of the present disclosure, Figure 7B The convolution operation between a subset of kernels (e.g., kernel 620a) with a stride of 1 and the noise image 340 is shown. Figure 7A and Figure 7B , individual pixels of the noise image 340 are processed using the corresponding kernel. Figure 7B As shown, pixel Y24 of image 340 is processed using kernel 620a_24, wherein kernel 620a_24 has pixel Y24 as the center pixel or target pixel. Similarly, pixel Y25 of image 340 is processed using kernel 620a_25, wherein kernel 620a_25 has pixel Y25 as the center pixel or target pixel. Similarly, pixel Y26 of image 340 is processed using kernel 620a_26, wherein kernel 620a_26 has pixel Y26 as the center pixel or target pixel. In general, pixels of noisy image 340 are processed using corresponding kernel 620a_i, wherein kernel 620a_i has pixel Yi as the center pixel or target pixel. The generation of kernels 620a_24, 620a_25, 620a_26, and generally speaking, 620a_i are similar to Figure 6A The generation of kernel 620a (e.g. Figure 6A Kernel 620a discusses Figure 7B 10). Noisy image 340 is processed (e.g., convolved, as further discussed below with respect to Equation 10) by multiple kernels 620a to generate a first intermediate image 708. Image 708 has less noise than image 340.

[0183] like Figure 7A As shown, image 708 is similarly convolved with kernel 620b to generate a second intermediate image 712. Note that although the single representative kernel 620b Figure 7A In the figure, as compared to Figure 7B As discussed, for each target pixel of image 708, there will be a corresponding kernel with a stride of 2 and having the target pixel at the center of the kernel.

[0184] Furthermore, image 712 is similarly convolved by kernel 620c to generate the final noise-free image 344. Note that although the single representative kernel 620c Figure 7A In the figure, as compared to Figure 7B As discussed, for each target pixel of image 712, there will be a corresponding kernel with a stride of 4 and having the target pixel at the center.

[0185] Thus, the noisy image 340 is iteratively processed by kernels 620a, 620b, 620c to ultimately generate a noise-free image 344 (also in FIG. Figure 3B (see figure in the middle). Figure 7A The output image of each stage in has less noise than the input image of that stage. For example, image 708 has less noise than image 304, image 712 has less noise than image 708, and finally, image 344 has less noise than image 712. Image 344 is called "noise-free" because it has relatively less noise (or no noise at all) compared to images 340, 708, and 712, although image 344 may have some residual noise. Image 340 is the noisy version, and image 344 is a realistic denoised (or noise-free) version of the corresponding scene.

[0186] As discussed, each kernel is used to perform a convolution operation on the image output by the previous stage. For example, kernel 620a is used to perform a convolution operation on image 340 output by noise image generator module 116 to generate a first intermediate image 708. Kernel 620b is used to perform a convolution operation on first intermediate image 708 to generate a second intermediate image 712. Finally, kernel 620c is used to perform a convolution operation on second intermediate image 712 to generate the final noise-free (or noise-reduced) image 344.

[0187] Although the three stages of the denoising operation are Figure 7A In the example of FIG, however, in some other examples, the system 102 can be implemented with any other suitable number of stages, such as two, four, or more. Of course, as the number of stages changes, the system 102 will also undergo other corresponding changes, such as generating a different number of kernel subsets, dividing the final radiation eigenvector into a different number of segments, etc., as will be understood in light of this disclosure.

[0188] The following equation discusses processing the target pixel from the image of the previous stage using the corresponding kernel to generate the output image of this stage:

[0189]

[0190] In Equation 10, the term Lj is the radiance value of pixel j in the image from the previous stage, and the term is the radiance value of the target pixel i in the image output by the current stage. Therefore, Equation 10 describes the radiance value relative to Figure 7A The convolution operation discussed. The term wk(i,j) is the weight that specifies the contribution of pixel j to pixel i, where k refers to the number of stages considered.

[0191] For example, for core 620a, Figure 7A In the first stage, the term Lj refers to the radiance value of pixel j in the noise image 340, and the term refers to the radiance value of pixel i in the image 708 output by the first stage. In the first stage, Equation 10 uses the weights w1(i,j) calculated according to Equation 7. In the first stage, denoising is performed at the i-th pixel using the radiance values ​​Lj of the neighboring pixels from the set Ni. For a specific target pixel i, the set Ni is the set of shadow pixels within the corresponding kernel. Therefore, for pixel Y24, the set Ni in the first stage refers to Figure 6A That is, all eight pixels adjacent to pixel Y24 contribute to the radiance value of target pixel Y24 in output image 708. Have contributed.

[0192] Similarly, for core 620b, Figure 7A In the second stage, the term Lj refers to the radiance value of pixel j in image 708, and the term Refers to the radiance value of pixel i in the image 712 output by the second stage. In the second stage, Equation 10 uses the weights w2(i, j) calculated according to Equation 8. In the second stage, denoising is performed at the i-th pixel using the radiance values ​​Lj of the neighboring pixels from the set Ni. For example, for a specific target pixel, such as pixel Y24, the set Ni for the second stage refers to Figure 6B For example, for the second stage and for the target pixel Y24, the set Ni includes the shadow pixel Y26 but does not include the non-shadow pixel Y33, as shown in FIG. Figure 6B As shown in the figure. That is, Figure 6B The radiance values ​​of the eight shadow pixels in the kernel 620b (two pixels away from the pixel Y24) to the target pixel Y24 in the output image 712 are Therefore, for example, the radiance value of the shadow pixel Y26 to the target pixel Y24 in the output image 712 is Contributes to the radiance value of the target pixel Y24 in the output image 712, while the non-shadow pixel Y33 No contribution.

[0193] Similarly, for core 620c, Figure 7A In the third or final stage, the term Lj refers to the radiance value of pixel j in image 712, and the term Refers to the radiance value of pixel i in the image 344 output by the third stage. In the third stage, Equation 10 uses the weights w3(i,j) calculated according to Equation 9. In addition, for a specific target pixel, such as pixel Y24, the set Ni of the third stage refers to Figure 6C For example, for the second stage and for the target pixel Y24, the set Ni includes the shadow pixel Y20 but does not include the non-shadow pixel Y26, as shown in FIG. Figure 6C Thus, for example, the radiance value of the shadow pixel Y20 to the target pixel Y24 in the output image 344 is Contributes to the radiance value of the target pixel Y24 in the output image 344, while the non-shadow pixel Y26 No contribution.

[0194] In Equation 10, the term is used for normalization. In the example, Wi is calculated as follows:

[0195] Wi=∑ j∈Ni wk(i, j) Equation 11

[0196] Therefore, for a given target pixel i, the term Wi is the sum of all weights wk(i,j) used in Equation 10.

[0197] Figure 7C The diagram shows a summary of some embodiments of the present disclosure. Figure 7A Pseudocode 790 of the illustrated operation. For example, Figure 7C middle, is initialized to L, which is the light or radiance value of a pixel of the noise image 340. The stage count k=1. Pseudo-code 790 then proceeds to step 2, where the weights wk(i,j) are calculated. During the first iteration, wk(i,j) is w1(i,j) calculated according to Equation 7. In step 3, Equation 10 uses a stride of 2. k-1 (For example, for the first iteration of pseudocode 790, 2 0 or 1) the kernel is applied to , to generate ,in yes Figure 7A In the fourth step, if the count k is less than the total number of stages K (where Figure 7AIn the example above, K is 3), then the count k is incremented and the operation loops back to step 2. Once all three stages of operation are completed, the final output is , i.e., the final noise-free image 344 .

[0198] Figure 8A and Figure 8B are flowcharts collectively illustrating an example method 800 for denoising a noisy image generated using Monte Carlo rendering, such that the denoised image has less noise and is more realistic than the noisy image, according to some embodiments of the present disclosure. The method 800 may be performed, for example, using Figure 1 and / or 2 are implemented and described herein. However, as will be apparent in light of this disclosure, other system architectures may be used in other embodiments. To this end, Figure 3A The various functions shown are Figure 1 and Figure 2 The illustrated correlation of specific components and functions is not intended to imply any architectural and / or usage limitations. Rather, other embodiments may include, for example, varying degrees of integration, wherein multiple functionalities are effectively performed by a single system. In another example, multiple functionalities may be effectively performed by more than one system. Although various operations of method 800 are discussed herein as being performed by image denoising system 102 of computing device 100, one or more of these operations may also be performed by image denoising system 202 of server 201.

[0199] 8, method 800 includes: at 804, for example, by the noise image generator module 116 (eg, Figure 1 and Figure 2 In some examples, the first image is drawn using a Monte Carlo method (e.g., using Equation 2 and Equation 1a discussed above). An example of the first image is Figure 3B Noise image 340. In some examples, the first image is a digitally generated synthetic model or scene.

[0200] Monte Carlo methods are intended to render synthetic scenes to generate realistic images from the synthetic scenes using scene files. For example, by attributing desired radiance or light levels to individual pixels of the synthetic scene, Monte Carlo methods are intended to generate realistic images of the scene. However, Monte Carlo rendering uses a lower number of SPPs, such as 4, 8, 12, or another suitable number less than, for example, 24, 48, 96, 128, or 256. Monte Carlo rendering uses a lower number of SPPs, for example, due to a limited computational budget and / or to save time in such computations. For example, Figure 3BThe noisy image 340 is generated using an 8SPP Monte Carlo rendering. Therefore, since the Monte Carlo rendering uses a lower number of SPPs, the resulting image is noisy and far from realistic. The various operations of method 800 are intended to reduce the noise in this noisy image generated using a lower SPP Monte Carlo rendering.

[0201] Method 800 then proceeds from 804 to 806. At 806, within each pixel, a plurality of corresponding sampling points are identified. For example, Figure 3C The example sampling points S_1_24, S_2_24, ..., S_8_24 in the example pixel Y24 and the example sampling points S_1_42, S_2_42, ..., S_8_42 in the example pixel Y42 are shown. Figure 3C In the example of 8 SPP, 8 samples per pixel or 8 SPP are considered, but any other number of samples may be considered. In some examples, the sample points identified in 806 may be the same sample points used to generate the noisy first image using Monte Carlo rendering in 804. Thus, in some such examples, the operations in block 806 may be a subset of the operations in block 804, and the operations of blocks 804 and 806 may be combined.

[0202] The method 800 then proceeds to two different branches. For example, after the first branch, the method 800 proceeds from 806 to 808, where for each sampling point of each pixel, the corresponding radiation vector is estimated (e.g., by Figure 1 and Figure 2 The feature identification module 103 is shown. For example, the radiation vector is provided as Figure 4 The input of the radiation feature vector generation module 104 is as follows. The estimation of the radiation vector for an individual sampling point is discussed above in Equation 3 herein.

[0203] Method 800 then proceeds from 808 to 816. At 816, for each pixel, the corresponding intermediate radiance eigenvector is, for example, Figure 4 4. For example, the neural network module 410 (also known as the sample embedding stage) receives the radiation vector for each sample point and generates an intermediate radiation feature vector for each pixel. For example, the module 410 is trained to identify how the multiple radiation vectors 414 corresponding to the various sample points of a particular pixel are processed and averaged to generate the corresponding intermediate radiation feature vector 418 for the particular pixel. By way of example only, the module 410 detects patterns in the radiation vectors and determines how the radiation vectors are combined or averaged to form the intermediate radiation feature vector 418, such as with respect to Figure 4 discussed.

[0204] Method 800 then proceeds from 816 to 824. At 824, for each pixel, the corresponding final radiance eigenvector and the corresponding one or more scaling factors are calculated, for example, by Figure 4 The radiation characteristic vector generation module 104 generates. In some embodiments and as relative to Figure 4 As discussed, each final radiance eigenvector has multiple segments that match multiple denoising stages. For example, in Figure 7A In the example of , three stages are used and therefore, for this example, each final radiance eigenvector has 3 segments. For example, the final radiance eigenvector F_R24 of the example pixel Y24 has three segments F_R24_1 , F_R24_2 and F_R24_3.

[0205] In some embodiments, a single scaling factor aj is generated for each pixel. In some other embodiments, for individual pixels i, three corresponding scaling factors aj are generated. i 1 、a i 2 and a i 3 is generated, such as Figure 4 For example, the neural network module 411, also known as the spatial propagation stage, receives the intermediate radiation eigenvectors and generates the final radiation eigenvectors and scaling factors, as previously described with respect to Figure 4 discussed. Figure 4 As discussed, module 411 spatially propagates the radiation properties of a first pixel to one or more neighboring pixels, for example if the first pixel and the one or more neighboring pixels have some similar radiation properties and / or are located on the same or similar region of the image, and correspondingly generates a final radiation feature vector for the pixel.

[0206] Refer again Figure 8A , the method 800 also has a second branch from block 806, and in this second branch, the method 800 proceeds from 806 to 812. In 812, for each sampling point of each pixel, the corresponding auxiliary vector is estimated (e.g., by Figure 1 and Figure 2 The feature identification module 103 is shown. For example, the auxiliary vector is provided as Figure 5 The input of the auxiliary feature vector generation module 108. The estimation of the auxiliary vector for individual sampling points is discussed above in Equation 4 in this document.

[0207] Method 800 then proceeds from 812 to 820, where for each pixel, the corresponding intermediate radiance eigenvector is, for example, given by Figure 5The auxiliary feature vector generation module 108 generates the auxiliary feature vectors. For example, the neural network module 510 (also known as the sample embedding stage) receives the auxiliary vectors for each sample point and generates an intermediate auxiliary feature vector 518 for each pixel. For example, the module 510 is trained to identify how the multiple auxiliary vectors 514 for the various sample points of a particular pixel are processed and averaged to generate the corresponding intermediate auxiliary feature vector 518 for the particular pixel. By way of example only, the module 510 detects patterns in the auxiliary vectors and determines how the auxiliary vectors are combined or averaged to form the intermediate auxiliary feature vector 518.

[0208] The method 800 then proceeds from 820 to 828, where for each pixel, the corresponding final auxiliary feature vector is, for example, Figure 5 The auxiliary feature vector generation module 108 generates. In some embodiments and as relative to Figure 5 As discussed above, the neural network module 511 (also referred to as the spatial propagation stage) receives the intermediate auxiliary feature vectors and generates the final auxiliary feature vectors. Figure 5 As discussed, module 511 spatially propagates the auxiliary attribute of the first pixel to one or more neighboring pixels (e.g., if the first pixel and the one or more neighboring pixels have some similar auxiliary attributes and / or are located on the same or similar region of the image), and correspondingly generates a final auxiliary feature vector for the pixel.

[0209] Method 800 then proceeds from 824 and 828 to 832. At 832, system 102 (e.g., Figure 1 and Figure 2 The illustrated kernel generation module 112) generates K kernel subsets corresponding to the number of K denoising stages. In the example, each subset has a stride of 2 (k-1) kernel, where k=1, ..., K.

[0210] Therefore, in Figure 6A 、 Figure 6B 、 Figure 6C and Figure 7A In the example, the number of stages is 3 (e.g., K=3), so 3 kernel subsets are generated. For example, the first kernel subset is generated with k=1 and stride 2 (1-1) (i.e., 2 0 or 1) generated. Figure 6A illustrates example kernels from the first subset of kernels, and Figure 7B FIG shows another example kernel in the first kernel subset. For example, for each target pixel of the noise image, a corresponding kernel in the first kernel subset is generated, such as with respect to Figure 7BEquation 7 discusses the generation of the weights of the kernels in the first kernel subset. The generation of the first kernel subset involves the generation of the underlying weights w1(i,j), which specify the radiance contribution from pixel j to the target pixel i.

[0211] The second kernel subset is set with k=2 and stride 2 (2-1) (i.e., 2 1 or 2) generated. Figure 6B An example kernel from the second kernel subset is shown, where pixels two pixels away from the target pixel are shaded and weighted (white pixels in kernel 620b have zero weight). For example, for each pixel of the noise image, a corresponding kernel from the second kernel subset is generated. Equation 8 discusses the generation of weights for the kernels in the second kernel subset. The generation of the kernels involves the generation of underlying weights w2(i, j) that specify the radiance contribution from pixel j to pixel i.

[0212] The third kernel subset is set with k=3 and stride 2 (3-1) (i.e., 2 2 or 4) generated. Figure 6C An example kernel from the third kernel subset is shown, where pixels four pixels away from the target pixel are shaded and weighted (white pixels in kernel 620c have zero weight). For example, for each pixel of the noise image, a corresponding kernel from the third kernel subset is generated. Equation 9 discusses the generation of weights for the kernels in the third kernel subset. The generation of the kernels involves the generation of underlying weights w3(i, j) that specify the radiance contribution from pixel j to pixel i.

[0213] Method 800 then proceeds from Figure 8A 832 of them are Figure 8B 836 of them. Figure 8B The operations discussed in blocks 836 through 852 of method 800 are also Figure 7A Figure 2, and relative to Figure 7C Specifically, in 836, the count k is initialized to 1, and the first noise image of block 804 is initialized to the input image (e.g., by Figure 1 and Figure 2 denoised image generation module 120).

[0214] Method 800 then proceeds from 836 to 840. For example, in 840, the stride is 2 (k-1) The kernel of is convolved with the input image, for example according to Equation 10, to generate the output image. The first iteration of method 800 is for Figure 7A The first stage of the denoising pipeline is shown. For example, for k=1, the stride is 2 0 = 1 is convolved. An example of a kernel with stride 1 is Figure 6A 、 Figure 7A and Figure 7B Illustrated are cores 620a, 620a_24, 620a_25, and 620a_26. Figure 7A The convolution of a kernel with a stride of 1 with an input image (which is the noise image 340) is illustrated to generate a first intermediate image 708. The convolution operation is discussed with respect to Equation 10, where weights w1(i, j) are used for the first stage, where the number "1" after "w" means that these are the weights for the first stage (or count k=1).

[0215] The method then proceeds to 844 where the denoised image generation module 120 checks to see if the count k is equal to K, where K is Figure 7A The number of stages in the denoising pipeline.

[0216] If "No" in 844, this means that there are subsequent stages of the denoising pipeline to be executed. Therefore, method 800 proceeds to 848, from which it loops back to 840. In 848, the count k is incremented by 1, and the output image from 840 is now set as the input image. Figure 7A In the context of , the first intermediate image 708 output by the first stage is now set as the input image. The method then loops back to 840. During the second iteration of 840, the count k=2. Therefore, the stride is 2 1 or 2 kernels are used, such as Figure 6B and Figure 7A Example kernel 620b. Figure 7A The convolution of a kernel 620b with a stride of 2 is illustrated with the input image (which is the first intermediate image 708) to generate a second intermediate image 712. The convolution operation is discussed with respect to Equation 10, where weights w2(i,j) are used, where the number "2" after "w" means that these are the weights of the second stage (or count k=2).

[0217] Method 800 then proceeds from 840 to 844. Figure 7A In the example of , there are three stages (eg, K is 3), so the operations in 848 and 840 are repeated again for the third stage. During the third stage, the count k=3. Therefore, the stride is 2 2 or 4 cores are used, such as Figure 6C and Figure 7A Example kernel 620c. Figure 7AThe convolution of a kernel 620c with a stride of 4 is illustrated with the input image (which is the second intermediate image 712) to generate a noise-free image 716. The convolution operation is discussed with respect to Equation 10, where weights w3(i,j) are used, where the number "3" after "w" means that these are the weights of the third stage (or count k=3).

[0218] Method 800 then proceeds to 844 for the third time. During this iteration, the count k is 3, which in some instances is equal to the number of stages K, e.g. Figure 7A Therefore, decision block 844 outputs "yes," and method 800 proceeds to 852, where system 102 (or system 202) outputs the final output image (e.g., image 344) as a noise-free (or noise-reduced) and realistic image. At 856, the final output image is displayed on a display screen, such as Figure 1 and Figure 2 Display 142.

[0219] Thus, the output image of each stage in method 800 has less noise than the output image of the previous stage. For example, referring to FIG7 , image 708 has less noise than image 304, image 712 has less noise than image 708, and finally, image 344 has less noise than image 712. Image 344 is referred to as "noise-free" because it has relatively less noise (or no noise at all) compared to images 340, 708, and 712, although image 344 may have some residual noise. Image 340 is a noisy version, and image 344 is a realistic denoised (or noise-free) version of the scene.

[0220] As discussed, in some embodiments, in order to allow the kernel to have a sufficiently large spatial context, rather than using a single stage with a single fixed-size kernel, multiple stages with corresponding strides of multiple kernels are used in method 800. Note that, although Figure 6C The core 620c is larger than Figure 6B The kernel 620b is larger than Figure 6A 620a, but in each of kernels 620a, 620b, and 620c, only the eight pixels adjacent to the target pixel are shaded. Thus, while the spatial context of the kernel increases with each stage, the computation does not increase proportionally, and each kernel computes the same number of weights regardless of the stage in which the kernel is used. Thus, in some embodiments, relative to Figure 7A The denoising discussed can be performed at interactive frame rates. For example, denoising is fast because not all pixels within each kernel are taken into account. For example, as compared to Figure 6B and Figure 6C As discussed, only the radiance values ​​of shadow pixels in the kernel contribute to the corresponding target pixels.

[0221] In addition, the radiometric and auxiliary feature attributes of multiple sampling points are intelligently averaged by averaging modules 434 and 534, respectively, to generate an average value for each pixel. Therefore, whenever the attributes of a new sampling point are available, the average value is updated to obtain a refined image, thereby speeding up the denoising process and achieving an interactive frame rate for the denoising process.

[0222] As discussed, the scaling factors aj in Equations 7, 8, and 9 contribute to the asymmetry of the radiance value contributions. Thus, an outlier pixel may receive radiance contributions from neighboring pixels while contributing substantially no radiation to neighboring pixels. A pixel is an outlier if it is a light source, e.g., if it emits a significant amount of radiation (e.g., is a light source) compared to one or more neighboring pixels.

[0223] Furthermore, the system 102 utilizes deep learning to cast denoising as a supervised learning problem. Given a noisy input image 340, Figure 4 and Figure 5 The illustrated neural network predicts a feature vector for each pixel by analyzing each pixel with respect to its spatial proximity. When determining how much radiation a pixel should contribute to another pixel, the system 102 explicitly calculates the distance between the feature vectors of these pixels, as discussed with respect to Equations 7, 8, and 9. Thus, the system 102 enables the use of a low-capacity network with a lower computational cost, which results in Figure 7A The illustrated denoising operation runs at interactive frame rates.Thus, system 102 can generate a large number of noise-free images (such as image 344) even though system 102 uses only a small fraction of the parameters used by conventional denoising systems.

[0224] Many variations and configurations will be apparent in view of this disclosure and the following examples.

[0225] Example 1. A method for denoising an image, the method comprising: identifying multiple corresponding sampling points within individual pixels in a set of pixels of a first image; estimating, for the individual sampling points, corresponding radiance vectors characterizing light received at the corresponding sampling points; generating, by a first machine learning module, corresponding intermediate radiance feature vectors for the individual pixels in the set based at least in part on the radiance vectors associated with the multiple corresponding sampling points within the corresponding pixels; generating, by a second machine learning module, corresponding final radiance feature vectors for the individual pixels in the set based at least in part on the intermediate radiance feature vectors of the corresponding pixel and one or more neighboring pixels; and generating a second image based at least in part on the final radiance feature vectors, wherein the second image has less noise and is more realistic than the first image.

[0226] Example 2. A method according to Example 1, wherein: generating a final radiation feature vector includes: generating a corresponding final radiation feature vector having at least a corresponding first segment and a corresponding second segment for each pixel in the set; and the method further includes: generating a first kernel subset having a first stride based at least in part on the first segment of the final radiation feature vector, and generating a second kernel subset having a second stride based at least in part on the second segment of the final radiation feature vector.

[0227] Example 3. The method of Example 2, wherein generating a first kernel subset includes: generating a first kernel in the first kernel subset, the first kernel including a first pixel as a target pixel and a plurality of pixels adjacent to the target pixel, wherein the first kernel has a stride X indicating that pixels within the first kernel that are X pixels away from the target first pixel are assigned a non-zero weight, X is a positive integer, a second pixel within the first kernel is X pixels away from the target first pixel, and wherein a first weight specifying a contribution of a radiance value from the second pixel to the first pixel is calculated at least in part based on a distance between (i) a first segment of a first final radiance eigenvector for the first pixel and (ii) a first segment of a second final radiance eigenvector for the second pixel.

[0228] Example 4. The method according to Example 3 also includes: generating corresponding scaling factors for individual pixels in the set by a second machine learning module, wherein generating the first kernel includes generating a first weight based at least in part on the scaling factor assigned to the second pixel; and generating a second weight for the second kernel based at least in part on (i) the distance and (ii) the scaling factor assigned to the first pixel, the second weight specifying a contribution of the radiation value from the first pixel to the second pixel; wherein the first weight is different from the second weight at least in part based on the scaling factor assigned to the first pixel being different from the scaling factor assigned to the second pixel.

[0229] Example 5. The method according to Example 4 further includes: in response to detecting that one of the first pixel or the second pixel has a radiation value higher than a threshold and the other pixel of the first pixel or the second pixel has a radiation value lower than the threshold, assigning different values ​​to the scaling factors of the first pixel and the second pixel by a second machine learning module.

[0230] Example 6. A method according to any one of Examples 2 to 5, wherein generating the second image includes: convolving the first kernel subset with the first image to generate a first intermediate image; convolving the second kernel subset with the first intermediate image to generate a second intermediate image; and convolving the third kernel subset with the second intermediate image to generate the second image.

[0231] Example 7. A method according to Example 6, wherein: each kernel in the first kernel subset has a first stride having a value of 1, so that multiple neighboring pixels that are 1 pixel away from the target pixel in the kernel of the first subset are assigned corresponding non-zero weights; each kernel in the second kernel subset has a second stride having a value of 2, so that multiple neighboring pixels that are 2 pixels away from the target pixel in the kernel of the second subset are assigned corresponding non-zero weights; and each kernel in the third kernel subset has a third stride having a value of 4, so that multiple neighboring pixels that are 4 pixels away from the target pixel in the kernel of the third subset are assigned corresponding non-zero weights.

[0232] Example 8. The method of any one of Examples 1 to 7, wherein the one or more radiance values ​​characterizing the light received at the first sampling point include one or more of: red, green, and blue (R, G, B) color values ​​representing direct illumination and diffuse reflection light at the first sampling point; R, G, B color values ​​representing direct illumination and specular reflection light at the first sampling point; R, G, B color values ​​representing indirect illumination and diffuse reflection light at the first sampling point; and / or R, G, B color values ​​representing indirect illumination and specular reflection light at the first sampling point.

[0233] Example 9. The method of any one of Examples 1 to 8, wherein before estimating the radiance vector, the method further comprises: rendering the first image using a Monte Carlo method using multiple sampling points per pixel.

[0234] Example 10. The method according to any one of Examples 1 to 9 further includes: estimating, for individual sampling points of individual pixels, corresponding auxiliary vectors including one or more auxiliary features, the one or more auxiliary features characterizing the surface of the corresponding sampling points; generating, by a third machine learning module, corresponding intermediate auxiliary feature vectors for the individual pixels based at least in part on the auxiliary vectors associated with multiple corresponding sampling points within the corresponding pixels; and generating, by a fourth machine learning module, corresponding final auxiliary feature vectors for the individual pixels, wherein a first final auxiliary feature vector for a first pixel is generated at least in part based on: (a) a first intermediate auxiliary feature vector for the first pixel, and (b) one or more other intermediate auxiliary feature vectors for one or more other pixels adjacent to the first pixel.

[0235] Example 11. The method of Example 10, further comprising: generating one or more kernel subsets based at least in part on the final auxiliary feature vector, wherein the one or more kernel subsets can be used to generate the second image.

[0236] Example 12. The method of Example 11, wherein generating one or more kernel subsets includes: identifying a second pixel adjacent to the first pixel; and calculating a first weight based at least in part on: (i) a first L2 norm between a first final radiant eigenvector for the first pixel and a second final radiant eigenvector for the second pixel, and (ii) a second L2 norm between a first final auxiliary eigenvector for the first pixel and a second final auxiliary eigenvector for the second pixel.

[0237] Example 13. The method of Example 12, further comprising: generating a kernel having a stride of X and having a first pixel as a center pixel, such that a plurality of adjacent pixels that are X pixels away from the center pixel in the kernel are assigned corresponding non-zero weights, where X is a positive integer, where a second pixel is X pixels away from the center pixel, and where the second pixel is assigned the first weight.

[0238] Example 14. The method of any one of Examples 10 to 13, wherein the one or more auxiliary features of the first sampling point include one or more of: coordinates representing a direction normal to a surface at the first sampling point; red, green, and blue (R, G, B) color values ​​representing an albedo characteristic of the first sampling point; a depth of the first sampling point relative to a virtual camera that captured the first image; coordinates representing a position of the first sampling point relative to a coordinate system of the virtual camera; and / or a value representing an emissive property, a metallic property, a transmissive property, and / or a roughness property of the first sampling point.

[0239] Example 15. A system for drawing a lower noise image from an original image comprising multiple pixels, the system comprising: one or more processors; a denoising system executable by the one or more processors, configured to: generate corresponding radiation feature vectors and corresponding auxiliary feature vectors for individual pixels in a set of pixels of the original image; generate at least a first kernel subset and a second kernel subset based on a set of radiation feature vectors and a set of auxiliary feature vectors of the set of pixels; convolve the first kernel subset with the original image to generate a first image; and convolve the second kernel subset with the first image to generate a second image having less noise relative to the original image.

[0240] Example 16. A system according to Example 15, wherein to generate a first kernel in the first kernel subset, the system is used to: calculate a first weight, the first weight being based on (i) a first L2 norm between a first radiation eigenvector of the first pixel and a second radiation eigenvector of the second pixel, and (ii) a second L2 norm between a first auxiliary eigenvector of the first pixel and a second auxiliary eigenvector of the second pixel; and generate a first kernel having a stride, the stride having a first number, wherein the first kernel includes the first pixel and the second pixel separated by the first number of pixels, and wherein the first kernel includes the first weight.

[0241] Example 17. A system according to any of Examples 15 to 16, wherein the radiation characteristic vector of a pixel represents one or more of: red, green, and blue (R, G, B) color values ​​representing direct illumination and diffuse reflection light at multiple sampling points within the pixel; R, G, B color values ​​representing direct illumination and specular reflection light at multiple sampling points within the pixel; R, G, B color values ​​representing indirect illumination and diffuse reflection light at multiple sampling points within the pixel; and / or R, G, B color values ​​representing indirect illumination and specular reflection light at multiple sampling points within the pixel.

[0242] Example 18. A computer program product comprising one or more non-transitory machine-readable media encoded with instructions, which, when executed by one or more processors, causes a process to be performed for reducing noise within a digital image, the process comprising: generating corresponding radiation eigenvectors and corresponding auxiliary eigenvectors for individual pixels in a set of pixels of an original image; generating at least a first kernel and a second kernel based on the one or more radiation eigenvectors and the one or more auxiliary eigenvectors; convolving the first kernel with the original image to generate a first image; and convolving the second kernel with the first image to generate a second image having less noise relative to the original image.

[0243] Example 19. A computer program product according to Example 18, wherein generating a first kernel comprises: calculating a first weight based on (i) a first L2 norm between a first radiation eigenvector of a first pixel and a second radiation eigenvector of a second pixel, and (ii) a second L2 norm between a first auxiliary eigenvector of the first pixel and a second auxiliary eigenvector of the second pixel; and generating the first kernel having a stride, the stride having a first number, wherein the first kernel comprises the first pixel and the second pixel separated by the first number of pixels, and wherein the first kernel comprises the first weight.

[0244] Example 20. A computer program product according to Example 19, wherein the radiation characteristic vector of the pixel represents one or more of: an indication of a direction perpendicular to the sampling point of the pixel; an albedo characteristic of the sampling point; a depth of the sampling point relative to a virtual camera that obtained an image of the sampling point; an emission characteristic of the sampling point; a metallic characteristic of the sampling point; a transmission characteristic of the sampling point; and / or a roughness characteristic of the sampling point.

[0245] The foregoing detailed description has been presented for illustrative purposes. It is not intended to be exhaustive or to limit the disclosure to the precise forms disclosed. Many modifications and variations are possible in light of this disclosure. Therefore, it is intended that the scope of the present application be limited not by this detailed description, but rather by the appended claims. Future applications claiming priority from the present application may claim the disclosed subject matter in various ways and may generally include any combination of one or more limitations variously disclosed or otherwise presented herein.

Claims

1. A method for denoising an image, the method comprising: identifying a plurality of corresponding sampling points within a first pixel of the first image; For each sampling point, estimating a corresponding radiance vector representing the light received at the corresponding sampling point; generating, by a first machine learning module, an intermediate radiation feature vector for the first pixel based at least in part on the radiation vectors corresponding to the plurality of corresponding sampling points within the first pixel, wherein generating the intermediate radiation feature vector comprises: averaging sample encoding values ​​generated by a last layer of the first machine learning module; generating, by a second machine learning module, a corresponding final radiance eigenvector for the first pixel based at least in part on the intermediate radiance eigenvectors for the first pixel and one or more neighboring pixels; and A second image is generated based at least in part on the final radiation eigenvector, wherein the second image is less noisy and more realistic than the first image.

2. The method according to claim 1, wherein: Generating the final radiation feature vector includes: generating, for each pixel in the set, a corresponding final radiation feature vector having at least a corresponding first segment and a corresponding second segment; and The method further comprises: generating a first kernel subset having a first step based at least in part on the first segment of the final radiance eigenvector, and A second kernel subset having a second stride is generated based at least in part on the second segment of the final radiance eigenvector.

3. The method of claim 2, wherein generating the first kernel subset comprises: generating a first kernel in the first kernel subset, wherein the first kernel includes a first pixel as a target pixel and a plurality of pixels adjacent to the target pixel, wherein the first kernel has a stride X, indicating that pixels within the first kernel that are X pixels away from the target first pixel are assigned non-zero weights, X is a positive integer, a second pixel within the first kernel is X pixels away from the target first pixel, and wherein a first weight specifying a contribution of the radiance value from the second pixel to the first pixel is calculated based at least in part on: a distance between (i) the first segment of a first final radiance eigenvector for the first pixel and (ii) the first segment of a second final radiance eigenvector for the second pixel.

4. The method according to claim 3, further comprising: generating, by the second machine learning module, a corresponding scaling factor for the first pixel, wherein generating the first kernel comprises generating the first weight based at least in part on the scaling factor assigned to the second pixel; as well as generating a second weight for a second kernel based at least in part on (i) the distance, and (ii) a scaling factor assigned to the first pixel, the second weight specifying a contribution of radiance values ​​from the first pixel to the second pixel; Wherein the first weight is different from the second weight based at least in part on the scaling factor assigned to the first pixel being different from the scaling factor assigned to the second pixel.

5. The method according to claim 4, further comprising: In response to detecting that one of the first pixel or the second pixel has a radiation value above a threshold and the other of the first pixel or the second pixel has a radiation value below the threshold, the second machine learning module assigns different values ​​to the scaling factors of the first pixel and the second pixel.

6. The method of claim 2, wherein generating the second image comprises: convolving the first subset of kernels with the first image to generate a first intermediate image; convolving the second kernel subset with the first intermediate image to generate a second intermediate image; as well as Convolving the third kernel subset with the second intermediate image to generate the second image.

7. The method according to claim 6, wherein: Each kernel in the first subset of kernels has a first step having a value of 1, so that a plurality of neighboring pixels that are 1 pixel away from a target pixel in a kernel of the first subset of kernels are assigned corresponding non-zero weights; Each kernel in the second subset of kernels has a second stride of 2, such that a plurality of neighboring pixels that are 2 pixels away from a target pixel in a kernel of the second subset of kernels are assigned corresponding non-zero weights; and Each kernel in the third subset of kernels has a third stride of 4, such that a plurality of neighboring pixels that are 4 pixels away from a target pixel in a kernel of the third subset of kernels are assigned corresponding non-zero weights.

8. The method of claim 1 , wherein the one or more radiance values ​​characterizing the light received at the first sampling point include one or more of the following: Representing the red, green and blue R, G, B color values ​​of the direct illumination and diffuse reflection light at the first sampling point; Represents the R, G, and B color values ​​of the direct illumination and specular reflection light at the first sampling point; representing the R, G, and B color values ​​of the indirect illumination and diffuse reflection light at the first sampling point; and / or The R, G, and B color values ​​of the indirect illumination and specular reflection light at the first sampling point are represented.

9. The method according to claim 1 , wherein before estimating the radiation vector, the method further comprises: The first image is rendered using a Monte Carlo method using multiple sampling points per pixel.

10. The method according to claim 1, further comprising: estimating, for each sampling point of the first pixel, a corresponding auxiliary vector comprising one or more auxiliary features, the one or more auxiliary features characterizing a surface of the corresponding sampling point; generating, by a third machine learning module, an intermediate auxiliary feature vector for the first pixel based at least in part on the auxiliary vectors corresponding to the plurality of corresponding sampling points within the first pixel; as well as A corresponding final auxiliary feature vector is generated for the first pixel by a fourth machine learning module, wherein the first final auxiliary feature vector for the first pixel is generated based at least in part on: (a) a first intermediate auxiliary feature vector for the first pixel, and (b) one or more other intermediate auxiliary feature vectors for one or more other pixels adjacent to the first pixel.

11. The method according to claim 10, further comprising: One or more kernel subsets are generated based at least in part on the final auxiliary feature vector, wherein the one or more kernel subsets can be used to generate the second image.

12. The method of claim 11 , wherein generating the one or more kernel subsets comprises: identifying a second pixel adjacent to the first pixel; as well as A first weight is calculated based at least in part on: (i) a first L2 norm between a first final radiance eigenvector for the first pixel and a second final radiance eigenvector for the second pixel, and (ii) a second L2 norm between the first final auxiliary eigenvector for the first pixel and the second final auxiliary eigenvector for the second pixel.

13. The method according to claim 12, further comprising: generating a kernel having a stride of X and having the first pixel as a center pixel, such that a plurality of neighboring pixels that are X pixels away from the center pixel in the kernel are assigned corresponding non-zero weights, where X is a positive integer, where the second pixel is X pixels away from the center pixel, and where the second pixel is assigned the first weight.

14. The method of claim 10, wherein the one or more auxiliary features of the first sampling point include one or more of the following: a coordinate representing a direction perpendicular to the surface of the first sampling point; Red, green and blue R, G, B color values ​​representing the albedo characteristics of the first sampling point; a depth of the first sampling point relative to a virtual camera that captured the first image; coordinates representing the position of the first sampling point relative to the coordinate system of the virtual camera; and / or The value representing the emission characteristic, metal characteristic, transmission characteristic and / or roughness characteristic of the first sampling point.

15. A system for rendering a lower noise image from an original image comprising a plurality of pixels, the system comprising: one or more processors; A denoising system executable by the one or more processors for: For each pixel in the pixel set of the original image, generating a corresponding radiation feature vector and a corresponding auxiliary feature vector; generating at least a first kernel subset and a second kernel subset based on the set of radiation feature vectors and the set of auxiliary feature vectors of the set of pixels; Convolving the first kernel subset with the original image to generate a first image; as well as The second subset of kernels is convolved with the first image to generate a second image having less noise than the original image.

16. The system of claim 15, wherein to generate the first kernel in the first subset of kernels, the system is configured to: calculating a first weight based on (i) a first L2 norm between a first radiance eigenvector of a first pixel and a second radiance eigenvector of a second pixel, and (ii) a second L2 norm between a first auxiliary eigenvector of the first pixel and a second auxiliary eigenvector of the second pixel; and The first kernel is generated with a stride having a first number, wherein the first kernel includes the first pixel and the second pixel separated by the first number of pixels, and wherein the first kernel includes the first weight.

17. The system of claim 15, wherein the radiance eigenvector of a pixel represents one or more of: Representing the red, green and blue R, G, B color values ​​of direct illumination and diffuse reflection light at multiple sampling points within the pixel; representing R, G, and B color values ​​of direct illumination and specular reflection light at the plurality of sampling points within the pixel; representing R, G, and B color values ​​of the indirect illumination and diffuse reflection light at the plurality of sampling points within the pixel; and / or The R, G, and B color values ​​of the indirect illumination and specular reflection light at the plurality of sampling points within the pixel are represented.

18. A computer program product comprising one or more non-transitory machine-readable media encoded with instructions that, when executed by one or more processors, cause a process to be performed for reducing noise within a digital image, the process comprising: For each pixel in the pixel set of the original image, a corresponding radiation feature vector and a corresponding auxiliary feature vector are generated; generating at least a first kernel and a second kernel based on the one or more radiation eigenvectors and the one or more auxiliary eigenvectors; Convolving the first kernel with the original image to generate a first image; as well as The second kernel is convolved with the first image to generate a second image having less noise than the original image.

19. The computer program product of claim 18, wherein generating the first kernel comprises: calculating a first weight based on (i) a first L2 norm between a first radiance eigenvector of a first pixel and a second radiance eigenvector of a second pixel, and (ii) a second L2 norm between a first auxiliary eigenvector of the first pixel and a second auxiliary eigenvector of the second pixel; and The first kernel is generated with a stride having a first number, wherein the first kernel includes the first pixel and the second pixel separated by the first number of pixels, and wherein the first kernel includes the first weight.

20. The computer program product of claim 19, wherein the radiance eigenvector of a pixel represents one or more of: an indication of a direction perpendicular to the sampling point of the pixel; albedo characteristics of the sampling points; the depth of the sampling point relative to the virtual camera that obtains the image of the sampling point; Emission characteristics of the sampling points; Metal characteristics of the sampling point; Transmission characteristics of the sampling points; and / or The roughness characteristics of the sampling points.

Images