Method for rendering image of 3D scene, graphics processing unit and storage medium

Abstract

The present invention relates to a method for rendering an image of a 3D scene, a graphics processing unit and a storage medium. A method for rendering an image of a 3D scene is provided. The method comprises: rendering (110) a noisy image; and obtaining (112) one or more guiding channels. For each of a plurality of local neighborhoods, the method comprises: calculating (120) parameters of a model that approximates the noisy image as a function of the one or more guiding channels, and applying (130, 150) the calculated parameters to produce a denoised image. At least one of (i) the noisy image, (ii) the one or more guiding channels and (iii) the denoised image is stored in a quantized low bit depth format.

Description

Technical Field

[0001] The present invention relates to rendering images of 3D scenes. Background Art

[0002] Path tracing is a Monte Carlo method for approximating the transport of light in a scene. The quality of the result depends on the number of samples per pixel - the more samples, the closer the result is to actual light transport.

[0003] However, increasing the number of samples is computationally expensive, not least because the standard deviation of the noise depends on the number of samples N by a factor of This means that four times as many samples are needed to achieve a 50% noise reduction. Therefore, increasing the number of samples quickly becomes impractical as a way to reduce noise in path traced images.

[0004] It is well known that applying denoising algorithms can reduce noise without increasing the number of samples. "Guided filters" have been found to work well in this task. Originally proposed by He et al., guided filters model each neighborhood of a noisy image as an affine transformation of the corresponding neighborhood of a guidance image. The guidance image should be noise-free and should contain scene structures (e.g., object edges, occlusion boundaries, or shadow edges) that correspond to the noisy image. Such guidance images are useful in the context of path tracing because the scene is synthesized and its various "auxiliary" images can be rendered by other means. A guidance image with several guidance channels can be used in a guided filter, and each guidance channel can contain different kinds of information (e.g., a depth channel and a surface normal channel) that can be used to reconstruct the noise-free image. Different combinations of guidance channels may be useful in different parts of the image, and for this reason the approach is called a locally linear (or, more correctly but less commonly, a locally affine) model.

[0005] Because it is guided by information about the structural content of the scene, the guided filter can denoise noisy path-tracing images of the scene without causing significant blurring on the edges of objects within the image, as long as suitable structural information is available in one or more of the guiding channels.

[0006] In order to better support path tracing, in particular to allow path tracing to be performed at higher frame rates and / or better quality on devices with limited computational resources and power (such as mobile devices), it is desirable to improve the quality of denoising and to implement denoising more efficiently. Summary of the invention

[0007] This Summary is provided to introduce a selection of concepts in a simplified form that are further described below in the Detailed Description. This Summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used to limit the scope of the claimed subject matter.

[0008] A method for rendering an image of a 3D scene is provided. The method includes: rendering a noisy image; and obtaining one or more guiding channels. For each of a plurality of local neighborhoods, the method includes: calculating parameters of a model that approximates the noisy image as a function of the one or more guiding channels, and applying the calculated parameters to produce a denoised image. At least one of (i) the noisy image, (ii) the one or more guiding channels, and (iii) the denoised image is stored in a quantized low bit depth format.

[0009] According to one aspect, there is provided a method for rendering an image of a 3D scene, the method comprising:

[0010] Render noisy images;

[0011] obtaining one or more guided channels; and

[0012] For each of multiple local neighborhoods:

[0013] computing parameters of a model that approximates the noisy image as a function of the one or more guiding channels; and

[0014] Apply the calculated parameters to produce a denoised image,

[0015] At least one of the noisy image, the one or more guiding channels, and the denoised image is stored in a quantized low bit depth format.

[0016] According to some examples, the method may provide a combined rendering and denoising pipeline suitable for efficient rendering of images generated at least in part using path tracing. The method of approximating a noisy image by such a function of one or more guiding channels corresponds to denoising by guided filtering. The function may include or consist of a linear combination of guiding channels and scalar offsets. Thus, the model may include or may be an affine model.

[0017] In some examples, a scalar offset can be incorporated into the model by including a guide channel with a uniform value (e.g., an array of ones) at each pixel. Optionally, the guide channel is not stored in memory—it can be instantiated on the fly in software or hardware during fitting of the model (e.g., by a model fitting unit).

[0018] The guide pass may contain information about scene structure in the 3D scene, including but not limited to object boundaries, occlusion boundaries, and shadow edges. The guide pass may be substantially noise-free. They may be rendered by deterministic calculations (e.g., by rasterization), whereas noisy images may be rendered by random sampling.

[0019] Quantization can reduce the amount of data to be stored, and thus can reduce memory bandwidth requirements. Quantization converts data from a high bit depth format (e.g., 32-bit floating point) to a low bit depth format (e.g., 8-bit integer). The inventors have recognized that at least one of (i) a noisy image, (ii) one or more guide channels, or (iii) a denoised image can be stored (and transferred to or from a memory) in a quantized format without significantly reducing the image quality of the denoised image.

[0020] The steps of calculating parameters and applying the calculated parameters may be performed using floating point calculations.

[0021] The calculated parameters may be applied to one or more guiding channels (i.e., the same guiding channels used to calculate the parameters). Alternatively, in some examples, the calculated parameters may be applied to a second set of one or more guiding channels, corresponding to those guiding channels used to calculate the parameters. For example, the second set of guiding channels may be corresponding guiding channels at a different resolution, e.g., they may be upsampled or downsampled versions of the guiding channels used in the parameter calculation.

[0022] In particular, the noisy image may be a low-resolution image, and the method may include calculating parameters of a model that approximates the noisy image as a function of one or more low-resolution guiding channels. The method may then include applying the calculated parameters to one or more full-resolution guiding channels to produce a denoised image. The full-resolution guiding channels correspond to the low-resolution guiding channels in that they contain substantially the same content.

[0023] Where more than one of the noisy image, one or more guiding channels and the denoised image are stored in a quantised low bit depth format, they may be stored in different quantised low bit depth formats, ie they need not all be stored in the same format.

[0024] The relative resolutions of the "low-resolution" and "full-resolution" images can be selected according to the needs of the application. Generally, the "full-resolution" image has a resolution greater than or equal to the resolution of the "low-resolution" image (and therefore, the "full-resolution" image has a greater number of pixels than the "low-resolution" image or the same number of pixels as the "low-resolution" image). However, in most examples, the "full-resolution" image has a higher resolution than the "low-resolution" image.

[0025] The method enables the generation of a denoised full-resolution image from a low-resolution noisy image. This is computationally more efficient and / or more efficient in terms of memory access bandwidth than performing the denoising operation on the full-resolution noisy image. When denoising based on a low-resolution image, the amount of data that must be retrieved from memory and processed can be reduced. This is because the low-resolution guide channel and the local neighborhood of the low-resolution noisy image are typically smaller (i.e., will contain fewer pixels) than the corresponding neighborhood at full resolution.

[0026] Each local neighborhood may be a local neighborhood centered at a corresponding pixel of the noisy low-resolution image, thereby computing a different set of model parameters for each pixel.

[0027] Applying the calculated parameters to the one or more full-resolution guide channels may include applying the parameters calculated for the local neighborhood of the low-resolution guide channel to the corresponding local neighborhood of the full-resolution guide channel. Applying the calculated parameters to the one or more full-resolution guide channels may include upsampling the calculated parameters and applying the upsampled calculated parameters to the one or more full-resolution guide channels.

[0028] Rendering the noisy image optionally includes rendering by path tracing. In this case, the method can be viewed as denoising the path traced image by guided filtering. Path tracing is computationally intensive because multiple rays need to be cast per pixel, and each ray may have multiple "bounces". When the noisy image is a low resolution image, examples of the present method can avoid the need to render a full resolution path traced image. The inventors have discovered that comparable results can be achieved more efficiently by using a low resolution image and investing computational work in the number of rays per pixel and / or the number of bounces per ray, rather than rendering a greater number of pixels. In other words, computational work is better invested in producing a low resolution image with less noise and / or a closer approximation of light transport, rather than producing a noisier or more approximate full resolution image.

[0029] The noisy image may (i) include indirect lighting in the scene, or (ii) include an ambient occlusion image.

[0030] Optionally, the noisy image consists only of indirect illumination. Here, "direct" illumination refers to rays that interact (intersect) with a single object before reaching the virtual camera / viewer. This means that the light travels directly from the light source to the object (or, equivalently, is traced from the object to the light source), and then travels directly from the object to the virtual camera. Therefore, the object is "directly" illuminated by the light source. In contrast, "indirect" illumination refers to rays that have interacted (intersected) with at least two objects between the light source and the virtual camera. For example, the light can be reflected by the first object toward the second object, and can be reflected by the second object toward the virtual camera. The directly illuminated image does not contain any information about the surface reflections of the objects in the scene. The indirect illuminated image does not contain any information about the surface reflections of the object "closest" to the virtual camera, which means the final surface that the light interacts with on its path from the light source to the camera. However, in general, the indirectly illuminated image does contain information about the color of the surfaces "closer" to the light source, because the interaction of the light with these colored surfaces will affect the color of the indirect illumination that falls on the "closest" object. Direct and indirect illumination can be combined before or after denoising. For example, ray tracing can be used to model the directly illuminated image. It is usually low noise or no noise. Indirect lighting is usually noisier than direct lighting.

[0031] In some examples, the noisy image may include a ray traced ambient occlusion image. This can provide an approximation of indirect lighting (ignoring the position and intensity of the light source).

[0032] The denoised image may be a denoised indirect illumination image or a denoised ambient occlusion image, and the method may further include: obtaining a direct illumination image; and combining the denoised image with the direct illumination image to generate a global illumination image.

[0033] The combination may include summing the denoised image and the direct illumination image. In this example, the direct illumination image is combined with the indirect illumination image (or ambient occlusion image) after denoising. It should be understood that if the noisy image includes an ambient occlusion image, the global illumination image will contain an approximation of the global illumination.

[0034] Obtaining the directly lit image may include rendering it by ray tracing, where each ray may be cast along a path with exactly one bounce, or rendering it by rasterization, where each ray may be cast along a path with exactly one bounce. Rendering the directly lit image by rasterization may include rendering with shadow mapping.

[0035] The noisy image may be a noisy global illumination image, including direct and indirect illumination in the scene, whereby the denoised image is a denoised global illumination image.

[0036] Rendering a noisy global illumination image may include combining (e.g., summing) a noisy indirect illumination image and a direct illumination image. In this example, the direct illumination image is combined with the indirect illumination image before denoising. Alternatively, the noisy global illumination image may be rendered directly by path tracing that simulates direct and indirect illumination.

[0037] The method may also include combining the global illumination image or the denoised global illumination image with the surface reflection image to produce a rendered image of the 3D scene. The combination may include multiplying the global illumination by the surface reflection. The surface reflection image may include or consist of an albedo, including a diffuse albedo or a specular albedo. The surface reflection image may be rendered by rasterization.

[0038] Guide passes can be rendered via ray casting or rasterization (in any combination).

[0039] Obtaining one or more guide passes may include rendering by rasterization. For example, a low-resolution guide pass may be rendered by rasterization, while a high-resolution guide pass may be rendered by ray casting or rasterization. Alternatively, a high-resolution guide pass may be rendered by rasterization, while a low-resolution guide pass may be rendered by ray casting or rasterization.

[0040] The low-resolution guide pass may be obtained by rendering at low resolution through a first rasterization pass; and the full-resolution guide pass may be obtained by rendering at full resolution through a second rasterization pass.

[0041] That is, the low-resolution guide pass and the full-resolution guide pass can be rendered separately. Alternatively, the low-resolution guide pass can be generated from the full-resolution guide pass by downsampling. However, the inventors have found that it may be more efficient to render the guide pass twice at different resolutions, rather than rendering the guide pass once at full resolution and downsampling it. This is because the memory access bandwidth can be reduced by rendering the guide pass twice. Instead of writing / reading the guide pass to / from memory, the guide pass can be rendered at the desired resolution as required by the algorithm.

[0042] A single rasterization pass may have several outputs. Thus, multiple guide passes (and optionally all guide passes) may be generated by a single rasterization pass.

[0043] The guidance channel may include any one or any combination of two or more of the following: depth information of objects in the 3D scene; information identifying the material of objects in the 3D scene; surface reflections of objects in the 3D scene; shadows in the 3D scene; and surface normals of objects in the 3D scene.

[0044] The method may include: defining a first tile, defining corresponding first adjacent portions of a noisy image and one or more guide channels, each first adjacent portion including a first plurality of pixels; defining a second tile; defining corresponding second adjacent portions of the noisy image and one or more guide channels, each second adjacent portion including a second plurality of pixels; calculating a first outer product between each pixel in the one or more guide channels and itself; and calculating a second outer product between each pixel in the one or more guide channels and a corresponding pixel in the noisy image, wherein for a pixel in the first tile, the first outer product and the second outer product are calculated (i) before the second tile or (ii) in parallel with the second tile.

[0045] In the presence of multiple guidance channels, these channels can be considered to form a guidance image, and a first outer product can be calculated between each pixel in the guidance image and itself, while a second outer product can be calculated between each pixel in the guidance image and the corresponding pixel in the noisy low-resolution image. Calculating the outer product of the first tile before the second tile means that the calculation of the first tile is completed before starting to calculate the outer product of the second tile. In this way, the tiles can be processed individually and continuously, for example by a single processor or a single core in hardware. Calculating them in parallel means calculating them individually at the same time. This allows parallel processing, especially on different processors or cores in hardware. The first tile and the second tile can be non-overlapping.

[0046] Organizing processing in this way can allow for greater data locality. This helps with memory bandwidth efficiency—data in a tile can be cached locally to the processor or core performing the computation, meaning fewer accesses to external memory may be required.

[0047] The method may also include, after rendering the noisy image, quantizing it in a quantized low-bit depth format using non-linear quantization, so that darker areas of the image are quantized to a relatively larger quantization level density and brighter areas of the image are quantized to a relatively smaller quantization level density; and storing the quantized low-bit depth format in a memory, wherein the method optionally also includes, before calculating the parameters of the model, retrieving the quantized low-bit depth values ​​from the memory and performing inverse quantization.

[0048] Here, the quantization step size in dark areas of the image is smaller than the quantization step size in bright areas of the image. This allows dark (e.g., dim) areas of a scene to be accurately represented. In one example of nonlinear quantization, quantization includes applying a square root function followed by uniform quantization of the output of the square root function.

[0049] Calculating parameters of the model optionally includes: calculating a first outer product between each pixel in one or more guiding channels and itself; calculating a second outer product between each pixel in one or more guiding channels and a corresponding pixel in the noisy image; blurring the first outer product to calculate a first moment matrix for each local neighborhood; blurring the second outer product to calculate a second moment matrix for each local neighborhood; and calculating parameters (A) of the model for each local neighborhood, including calculating an inverse matrix of the first moment matrix, and calculating the product of the inverse matrix and the second moment matrix.

[0050] Here, it should be understood that each pixel is represented by a row vector. Each pixel in one or more guiding channels is represented as a row vector x; each pixel in the noisy image is represented by a row vector y.

[0051] "Fuzzy" refers to spatial averaging, e.g., summing over a local neighborhood, optionally using a weighted summation, optionally where the center of the local neighborhood is given more weight in the summation than the periphery of the local neighborhood.

[0052] Optionally, calculating the parameters of the model includes, before calculating the inverse matrix, adding a regularization matrix to the first moment matrix. The regularization matrix may include a diagonal matrix. The regularization matrix can help avoid numerical instabilities in matrix inversion.

[0053] Fuzzifying the first outer product optionally includes calculating a first multiscale pyramid from the first outer product and calculating a first moment matrix based on the first multiscale pyramid; and / or fuzzifying the second outer product optionally includes calculating a second multiscale pyramid from the second outer product and calculating a second moment matrix based on the second multiscale pyramid.

[0054] The multiscale pyramid has multiple levels, where successive levels describe the outer product at successively different levels of detail. For example, the multiscale pyramid may include or consist of a mipmap pyramid. Mipmaps are suitable for efficient implementation in fixed-function hardware, such as a graphics processing unit (GPU).

[0055] The fuzzy outer product (i.e., the moment matrix) can be directly calculated from a predetermined level of the pyramid. In this case, the calculation of the pyramid may stop at this level. In other examples, the moment matrix can be calculated by interpolation using the pyramid. The interpolation can include bilinear or trilinear interpolation, or other sampling, such as bicubic sampling.

[0056] Blurring may include separable filtering in horizontal and vertical directions.

[0057] The filtering may use a center-weighted filter function, such as a Gaussian function. Optionally, separable filtering may be applied to a predetermined level of the multi-scale pyramid. This may help to efficiently implement center-weighted filtering with reduced computational complexity (compared to filtering the outer product directly using a center-weighted filter).

[0058] This type of blurring may be applied to one or both of the first outer product and the second outer product.

[0059] Blurring can include filtering using an anisotropic 2D filter. By anisotropic, it is meant that the filter has a major axis and a minor axis perpendicular to the major axis and extends further along the major axis than the minor axis. The axes can be aligned with the horizontal and vertical directions, or the axes can be independent of the horizontal and vertical directions.

[0060] The one or more guiding passes may include surface normals of objects in the 3D scene, and the blurring may include: for each local neighborhood, determining a major axis and a minor axis of a 2D filter based on the surface normal of the object at the center of the neighborhood; selecting a level of a multi-scale pyramid based on the length of the minor axis; and sampling the selected level of the multi-scale pyramid along the major axis.

[0061] This may provide a computationally efficient way to adapt the blur to the scene content, in particular by adjusting the blur according to the orientation of the surface being sampled.

[0062] In some examples, blurring may include IIR filtering.

[0063] In some examples, blurring can include filtering with a running box filter.

[0064] The method may include: defining a first outer product tile, defining a first adjacent part of the first outer product and a corresponding first adjacent part of the second outer product, each first adjacent part including a first plurality of pixels; and defining a second outer product tile, defining a second adjacent part of the first outer product and a corresponding second adjacent part of the second outer product, each second adjacent part including a second plurality of pixels, wherein for the first tile, the first moment matrix and the second moment matrix are calculated (i) before the second tile or (ii) in parallel with the second tile.

[0065] The first outer product tile and the second outer product tile may be overlapping tiles. This is because, generally speaking, the blur of the outer product will not be divided into non-overlapping tiles. The degree of overlap may depend on the size of the filter used for blurring.

[0066] The method may also include normalizing and / or regularizing one or both of the first moment matrix and the second moment matrix. This can improve the stability of the matrix inversion. In this regard, normalization can supplement the use of the regularization matrix. Normalization can include using the mean and standard deviation information in the moment matrix to normalize the moment before the inverse calculation. This can improve the numerical stability of the matrix inversion, thereby improving the visual quality. In some examples, one or both matrices can be normalized to have a zero mean and a constant standard deviation (such as a standard deviation equal to one).

[0067] Also provided is a graphics processing unit configured to render an image of a 3D scene, the graphics processing unit comprising:

[0068] a first rendering block, the first rendering block being configured to render a noisy image;

[0069] a second rendering block, the second rendering block being configured to render one or more guide channels;

[0070] a model fitting block configured to compute, for each of a plurality of local neighborhoods, parameters of a model that approximates the noisy image as a function of the one or more guiding channels; and

[0071] a model application block configured to apply the calculated parameters for each of the plurality of local neighborhoods to produce a denoised image,

[0072] Wherein at least one of the noisy image and the one or more guiding channels is stored in a quantized low bit depth format.

[0073] The model application block may be configured to apply the calculated parameters to one or more guiding channels (i.e., the same guiding channels used to calculate the parameters). Alternatively, in some examples, the model application block may be configured to apply the calculated parameters to a second set of one or more guiding channels, corresponding to those guiding channels used to calculate the parameters.

[0074] The noisy image and / or one or more guide channels may be stored in a memory in a quantized low bit depth format.

[0075] The first rendering block may be configured to render a noisy image by path tracing.

[0076] The second rendering tile may be configured to render one or more guide passes by rasterization or ray tracing.

[0077] The noisy image may include indirect lighting in the scene or may include an ambient occlusion image, wherein the denoised image includes the indirect lighting image or the denoised ambient occlusion image, and wherein the graphics processing unit optionally further includes: a fourth rendering block configured to render a direct lighting image; and a combination block configured to combine the denoised image with the direct lighting image to produce a global illumination image.

[0078] The fourth rendering block may be configured to render the directly illuminated image by ray tracing.

[0079] The noisy image may be a noisy global illumination image, including direct and indirect illumination in the scene, whereby the denoised image is a denoised global illumination image.

[0080] The second rendering block may be configured to render a surface reflection image, and the graphics processing unit may further include a product block configured to calculate a product of the global illumination image or the denoised global illumination image and the surface reflection image to generate a rendered image of the 3D scene.

[0081] Surface reflection images can be rendered using rasterization.

[0082] Also disclosed is a method for rendering an image of a 3D scene, the method comprising:

[0083] Render noisy low-resolution images;

[0084] obtaining one or more low-resolution guidance channels and obtaining one or more corresponding full-resolution guidance channels; and

[0085] For each of multiple local neighborhoods:

[0086] computing parameters of a model that approximates the noisy low-resolution image as a function of one or more low-resolution guidance channels; and

[0087] The calculated parameters are applied to one or more full-resolution guidance channels to produce a full-resolution denoised image.

[0088] Also disclosed is a graphics processing unit configured to render an image of a 3D scene, the graphics processing unit comprising:

[0089] A first rendering block, the first rendering block being configured to render a noisy low-resolution image;

[0090] a second rendering block configured to render one or more low-resolution guide passes;

[0091] a third rendering block configured to render one or more full-resolution guide passes, each full-resolution guide pass corresponding to a corresponding low-resolution guide pass;

[0092] a model fitting block configured to compute, for each of a plurality of local neighborhoods, parameters of a model that approximates a noisy low-resolution image as a function of one or more low-resolution guiding channels; and

[0093] A model application block is configured to apply the calculated parameters to one or more full-resolution guidance channels for each of the plurality of local neighborhoods to produce a full-resolution denoised image.

[0094] Also disclosed is a method for rendering an image of a 3D scene, the method comprising:

[0095] Render noisy images;

[0096] obtaining one or more guided channels; and

[0097] For each of multiple local neighborhoods:

[0098] computing parameters of a model that approximates the noisy image as a function of the one or more guiding channels; and

[0099] Apply the calculated parameters to produce a denoised image,

[0100] The parameters of the calculation model include:

[0101] defining a first tile, defining respective first contiguous portions of the noisy image and the one or more guide channels, each first contiguous portion comprising a first plurality of pixels;

[0102] defining a second tile defining respective second contiguous portions of the noisy image and the one or more guide channels, each second contiguous portion comprising a second plurality of pixels;

[0103] computing a first outer product between each pixel in the one or more guide channels and itself; and

[0104] computing a second outer product between each pixel in the one or more guiding channels and the corresponding pixel in the noisy image,

[0105] For pixels in the first image block, the first outer product and the second outer product are calculated (i) before the second image block or (ii) in parallel with the second image block.

[0106] Also disclosed is a method for rendering an image of a 3D scene, the method comprising:

[0107] Render noisy images;

[0108] obtaining one or more guided channels; and

[0109] For each of multiple local neighborhoods:

[0110] computing parameters of a model that approximates the noisy image as a function of the one or more guiding channels; and

[0111] Apply the calculated parameters to produce a denoised image,

[0112] The parameters of the calculation model include:

[0113] computing a first outer product between each pixel in the one or more guide channels and itself;

[0114] computing a second outer product between each pixel in the one or more guiding channels and a corresponding pixel in the noisy image;

[0115] Fuzzy first outer product to calculate the first moment matrix of each local neighborhood;

[0116] Fuzzy the second outer product to calculate the second moment matrix of each local neighborhood; and

[0117] The parameters of the calculation model for each local neighborhood include calculating the inverse matrix of the first moment matrix and calculating the product of the inverse matrix and the second moment matrix.

[0118] The method includes, when blurring the first outer product and the second outer product:

[0119] defining a first outer product patch, defining first contiguous portions of the first outer product and corresponding first contiguous portions of the second outer product, each first contiguous portion comprising a first plurality of pixels; and

[0120] defining a second outer product patch, defining second contiguous portions of the first outer product and corresponding second contiguous portions of the second outer product, each second contiguous portion comprising a first plurality of pixels,

[0121] For the first image block, the first moment matrix and the second moment matrix are calculated (i) before the second image block or (ii) in parallel with the second image block.

[0122] Also disclosed is a graphics processing unit configured to render an image of a 3D scene, the graphics processing unit comprising:

[0123] a first rendering block, the first rendering block being configured to render a noisy image;

[0124] a second rendering block, the second rendering block being configured to render one or more guide channels;

[0125] a model fitting block configured to compute, for each of a plurality of local neighborhoods, parameters of a model that approximates the noisy image as a function of the one or more guiding channels; and

[0126] a model application block configured to apply the calculated parameters for each of the plurality of local neighborhoods to produce a denoised image,

[0127] The model fitting block is configured to calculate the parameters:

[0128] defining a first tile, defining respective first contiguous portions of the noisy image and the one or more guide channels, each first contiguous portion comprising a first plurality of pixels;

[0129] defining a second tile defining respective second contiguous portions of the noisy image and the one or more guide channels, each second contiguous portion comprising a second plurality of pixels;

[0130] computing a first outer product between each pixel in the one or more guide channels and itself; and

[0131] computing a second outer product between each pixel in the one or more guiding channels and the corresponding pixel in the noisy image,

[0132] For pixels in the first image block, the first outer product and the second outer product are calculated (i) before the second image block or (ii) in parallel with the second image block.

[0133] Also disclosed is a graphics processing unit configured to render an image of a 3D scene, the graphics processing unit comprising:

[0134] a first rendering block, the first rendering block being configured to render a noisy image;

[0135] a second rendering block, the second rendering block being configured to render one or more guide channels;

[0136] a model fitting block configured to compute, for each of a plurality of local neighborhoods, parameters of a model that approximates the noisy image as a function of the one or more guiding channels; and

[0137] a model application block configured to apply the calculated parameters for each of the plurality of local neighborhoods to produce a denoised image,

[0138] The model fitting block is configured to calculate the parameters:

[0139] computing a first outer product between each pixel in the one or more guide channels and itself;

[0140] computing a second outer product between each pixel in the one or more guiding channels and a corresponding pixel in the noisy image;

[0141] Fuzzy first outer product to calculate the first moment matrix of each local neighborhood;

[0142] Fuzzy the second outer product to compute the second moment matrix of each local neighborhood; and

[0143] The parameters of the calculation model for each local neighborhood include calculating the inverse matrix of the first moment matrix and calculating the product of the inverse matrix and the second moment matrix.

[0144] The model fitting block is configured to blur the first and second outer products:

[0145] defining a first outer product patch, defining first contiguous portions of the first outer product and corresponding first contiguous portions of the second outer product, each first contiguous portion comprising a first plurality of pixels; and

[0146] defining a second outer product patch, defining second contiguous portions of the first outer product and corresponding second contiguous portions of the second outer product, each second contiguous portion comprising a first plurality of pixels,

[0147] For the first image block, the first moment matrix and the second moment matrix are calculated (i) before the second image block or (ii) in parallel with the second image block.

[0148] There is also provided a graphics processing system configured to perform the method as outlined above and / or comprising a graphics processing unit as outlined above. The graphics processing system may be embodied in hardware on an integrated circuit.

[0149] There is also provided a method of manufacturing a graphics processing system as outlined above using an integrated circuit manufacturing system.

[0150] A method of manufacturing a graphics processing system as outlined above using an integrated circuit manufacturing system is also provided, the method comprising: processing a computer-readable description of the graphics processing system using a layout processing system to generate a circuit layout description of an integrated circuit embodying the graphics processing system; and manufacturing the graphics processing system based on the circuit layout description using the integrated circuit generation system.

[0151] There is further provided a computer readable code configured to cause the method as outlined above to be performed when the code is run; and optionally a computer readable storage medium having the computer readable code encoded thereon.

[0152] An integrated circuit definition data set is also provided which, when processed in an integrated circuit manufacturing system, configures the integrated circuit manufacturing system to manufacture a graphics processing system as outlined above.

[0153] Still further provided is a computer-readable storage medium (optionally non-transitory) having stored thereon a computer-readable description of a graphics processing system as outlined above, which, when processed in an integrated circuit manufacturing system, causes the integrated circuit manufacturing system to manufacture an integrated circuit embodying the data processing system.

[0154] Also provided is a computer-readable storage medium (optionally non-transitory) having stored thereon a computer-readable description of a graphics processing system as outlined above, which, when processed in an integrated circuit manufacturing system, causes the integrated circuit manufacturing system to perform the following operations: processing the computer-readable description of the graphics processing system using a layout processing system to generate a circuit layout description of an integrated circuit embodying the graphics processing system; and manufacturing the graphics processing system based on the circuit layout description using the integrated circuit generation system.

[0155] There is also provided an integrated circuit manufacturing system configured to manufacture a graphics processing system as outlined above.

[0156] An integrated circuit manufacturing system is also provided, comprising: a computer-readable storage medium (optionally non-transitory) having stored thereon a computer-readable description of a graphics processing system as outlined above; a layout processing system configured to process the computer-readable description to generate a circuit layout description of an integrated circuit embodying the graphics processing system; and an integrated circuit generation system configured to manufacture the graphics processing system based on the circuit layout description.

[0157] The layout processing system may be configured to determine location information of logic components of the circuit derived from the integrated circuit description to generate a circuit layout description of the integrated circuit incorporating the graphics processing system.

[0158] As will be apparent to those skilled in the art, the above-described features may be combined as appropriate and may be combined with any of the aspects of the examples described herein. BRIEF DESCRIPTION OF THE DRAWINGS

[0159] Examples will now be described in detail with reference to the accompanying drawings, in which:

[0160] Figure 1 A flowchart illustrating a rendering method according to an example is shown;

[0161] Figure 2 is a block diagram of a graphics processing unit for rendering according to an example;

[0162] Figure 3 Shown in more detail Figure 1 The model fitting step;

[0163] Figure 4 Shows Figure 1 A variation of a portion of the illustrated method that uses tile-based processing;

[0164] Figure 5 shows the use of tile-based processing Figure 1 Another variation of a portion of the method shown;

[0165] Fig. 6A Show Figure 3 Variations of the model fitting method shown;

[0166] Figure 6B and Figure 6C shows the creation of a mipmap pyramid according to an example;

[0167] Figure 7 Shows Fig. 6A Example of the fuzzy step in ;

[0168] Fig. 8A The basic principle of anisotropic blur is shown;

[0169] Figure 8B Shows Fig. 6A Another example of the fuzzy step in ;

[0170] Fig. 9 Quantization and inverse quantization of data according to an example are shown;

[0171] Fig.10 is a diagram showing a method according to another example Figure 1 Flowchart of alternatives to the method;

[0172] Fig.11 A graphics processing system is shown in which a graphics processing unit is implemented; and

[0173] Fig.12 A manufacturing system for producing an integrated circuit embodying a graphics processing system is shown.

[0174] The accompanying drawings show various examples. It will be appreciated by the skilled person that the element boundaries (e.g., boxes, groups of boxes, or other shapes) shown in the accompanying drawings represent an example of boundaries. In some examples, it may be the case that an element can be designed as multiple elements, or multiple elements can be designed as one element. Where appropriate, common reference numerals are used throughout the accompanying drawings to indicate similar features. DETAILED DESCRIPTION

[0175] The following description is presented by way of example to enable one skilled in the art to make and use the invention.The present invention is not limited to the embodiments described herein, and various modifications to the disclosed embodiments will be apparent to those skilled in the art.

[0176] Embodiments will now be described by way of example only.

[0177] Various techniques for rendering are known in the field of computer graphics. "Rendering", also known as image synthesis, refers to the process of generating an image of a virtual 3D scene as if a virtual camera were looking at it at a specific position and orientation. The geometry of the objects in the scene is modeled in three dimensions, and the rendering process approximates / simulates how light interacts from a light source with the modeled geometry to reach the virtual camera.

[0178] Path tracing, ray tracing, and rasterization are three rendering techniques. The terms "ray tracing" and "path tracing" are sometimes used interchangeably in the literature. For greater clarity, in this context, we will first define what we mean by these terms.

[0179] As used herein, the term "path tracing" refers to a random sampling method that simulates the transport of light in a 3D scene. In some examples according to the present disclosure, path tracing is able to approximate global illumination by modeling all types of interactions between light and objects in the scene, potentially including higher-order reflections (second bounces and third bounces, etc.). However, due to its random nature, path tracing is inherently noisy. Note that path tracing is not limited to approximating global illumination. "Ray traced ambient occlusion" is a type of path tracing that calculates the degree to which each point in a 3D scene is exposed to ambient lighting (rather than calculating global illumination as such). Ray traced ambient occlusion can be considered a rough approximation of indirect illumination.

[0180] As used herein, the term "ray tracing" refers to a deterministic, noise-free method for simulating light transport in a 3D scene by modeling certain interactions between light rays and objects. Ray tracing typically only determines direct light, reflections, refractions, etc., and does not rely on complex light paths that interact with multiple objects.

[0181] The term "ray casting" refers to a specific kind of ray tracing in which rays are cast along a single direct path, without reflection or refraction.

[0182] Rendering via path tracing and ray tracing differs from rendering via "rasterization," in which 3D scene geometry is mapped to pixels in 2D screen space without regard for the physical transport of light in the 3D scene.

[0183] Rendering pipeline

[0184] Now refer to Figure 1 and Figure 2 A first example of rendering an image is described. Figure 1 is a flowchart illustrating a method of rendering an image of a 3D scene according to an example. Figure 2 is shown as being configured to execute Figure 1 A block diagram of functional blocks of a graphics processing unit of the method. A graphics processing unit (GPU) includes a first rendering block, a second rendering block, a third rendering block, and a fourth rendering block. In this example, the first rendering block is a path tracing block 210. The second rendering block and the third rendering block are both provided by a rasterization block 240. The fourth rendering block is a ray tracing block 260. The GPU also includes a model fitting block 220, an upsampler 230, a model combination block 250, a summation block 270, and a product block 280. The upsampler 230 and the model combination block 250 together constitute a model application block, which is responsible for applying the model. The summation block 270 operates as a combination block to combine the denoised image generated by the application of the model with the image generated by the fourth rendering block 260.

[0185] In step 110, the path tracing block 210 generates a low-resolution indirect illumination image by path tracing. In step 112, the rasterization block 240 generates a low-resolution guide channel in a first rasterization pass. The indirect illumination image and the guide channel correspond to each other because they are both rendered for the same viewpoint in the 3D scene. They are also rendered at the same (low) resolution. Due to the stochastic nature of path tracing, the indirect illumination image is noisy. Because rasterization is deterministic, the guide channel is noise-free. In this example, the low-resolution guide channel set includes a depth image, a surface reflection image, and a surface normal image. However, the low-resolution guide channel set does not need to include all of these images (for example, the surface reflection image may not be included). Optionally, other guide channels may be included, such as images that identify the material of objects in the scene and / or images that show shadows in the scene.

[0186] The low-resolution images are typically a fraction of the size of the desired full-resolution image. For example, they may be one-half smaller in each dimension (i.e., in height and width) so that they are one-quarter the area and resolution of the full-resolution image. In some examples, they may still be a smaller fraction of the full-resolution image. For example, the low-resolution images may be one-quarter smaller in each dimension so that they are one-sixteenth the area and resolution of the full-resolution image.

[0187] The rendering method will produce a full-resolution rendered image based in part on a guidance image (including multiple guidance channels) and a noisy indirect illumination image. The low-resolution image (i.e., the guidance image and the indirect illumination image) is passed to a model fitting block 220. In step 120, the model fitting block 220 fits the parameters of an affine model for each pixel of the low-resolution image so that it can be used to map the low-resolution guidance image onto the low-resolution indirect illumination image. For each pixel, the parameter estimation examines a local neighborhood centered on the pixel and approximates the noisy indirect illumination image within the local neighborhood as a linear combination of the guidance channel and a scalar offset. This generates a set of model parameters for each of the local neighborhoods (i.e., for each pixel in the pixel).

[0188] In step 130, the upsampler 230 upsamples the model parameters to full resolution. In step 140, the rasterization block 240 renders a full-resolution guide channel set in a second rasterization pass. The full-resolution guide channels correspond one-to-one to the low-resolution guide channels. In fact, in some examples, the low-resolution guide channels can be generated by subsampling the full-resolution guide channels. However, the inventors have found that by rendering the low-resolution guide channels and the full-resolution guide channels separately in two rasterization passes, memory access bandwidth can be saved. This can avoid the need to store the full-resolution multi-channel guide image and then retrieve it from the memory for downsampling. In some hardware architectures and applications, memory bandwidth may be a limiting factor; therefore, reducing memory bandwidth is very desirable. Another advantage of using a low-resolution guide channel is that the number of operations required to generate a (low-resolution) moment image is reduced, because fewer matrices will need to be summed when generating the moment image.

[0189] The full-resolution guidance channel is provided to the model combination block 250. As described above, the guidance channel in this example includes a surface reflection image. The full-resolution surface reflection image is additionally provided to the product block 280. In other examples where the surface reflection image is not used as a guidance channel, the surface reflection image is still generated (i.e., as well as the guidance channel) so that it can be provided to the product block 280.

[0190] In step 150, the model combination block combines the upsampled model parameters from the upsampler 230 with the full-resolution guidance channels produced by the rasterization block 240. This involves generating each pixel as a linear combination of the full-resolution guidance channels, where the linear combination is determined by the model parameters for that pixel. In this example, since the image to be denoised is a noisy indirect illumination image, the output of the model combination block 250 is a full-resolution denoised indirect illumination image.

[0191] In step 160, the ray tracing block 260 renders the directly illuminated image by ray tracing. As explained above, ray tracing is deterministic. As such, ray tracing is inherently noise-free and less computationally intensive than rendering the indirectly illuminated image by path tracing 110. As a result of this lower complexity, rendering the directly illuminated image at full resolution does not impose a large computational burden.

[0192] In step 170, summation block 270 combines the denoised indirect illumination image provided by model combination block 250 with the direct illumination image provided by ray tracing block 260 by summing them. The result of the summation is a global illumination image at full resolution. In step 180, product block 280 multiplies the full resolution surface reflection image with the full resolution global illumination image (pixel by pixel). This will produce the final rendered image of the scene under global illumination conditions.

[0193] The affine model created in step 120 is based on He et al. (Kaiming He, Jian Sun, and Xiaoou Tang, "Guided Image Filtering", Daniilidis K., Maragos P., Paragios N. (eds.) Computer Vision–ECCV 2010. Lecture Notes in Computer Science, Vol. 6311, Springer. https: / / doi.org / 10.1007 / 978-3-642-15549-9_1 ). For each local neighborhood Y of the noisy input image, the method attempts to find an affine transformation A that best maps the local neighborhood X of the guidance image to the corresponding local neighborhood Y of the noisy input image. These local neighborhoods can be represented as matrices, and the "best" transformation is defined as the least squares best fit. That is, find a solution A such that the matrix equation Minimize, where and and N is the number of pixels in the local neighborhood, G is the number of guide channels (including the offset term), and M is the number of output channels. It should be understood that the labeling of the dimensions is essentially arbitrary.

[0194] As in this example, there may be multiple guide channels in the guide image. In this case, the model A is an affine transform with multiple input signals. According to this formulation (and unlike the formulation of He et al.), the offset term is directly included in the matrix multiplication by adding an additional guide channel consisting of an array of ones with corresponding values ​​in the A matrix. This additional guide channel of ones is constant and therefore can be implicit in block 220 so as not to waste bandwidth transmitting it. Thus, the matrix X representing the guide image channels is constructed as follows

[0195]

[0196] Here, each row represents pixels in a local neighborhood centered on the pixel of interest. Each column represents one channel of the entire multi-channel guidance image. The least squares optimization problem is modified to include regularization, which yields smoother results and reduces sensitivity to numerical instabilities:

[0197]

[0198] Here, N is a scalar introduced for consistency with the formula of He et al. U is a diagonal matrix constructed so that the entries corresponding to the channels of ones in X are zero:

[0199] U=diag(0,∈0,∈1,…).

[0200] The number is ∈ positive. A suitable exemplary value is about 0.0001. Minimizing E with respect to A has a closed form solution:

[0201] A=(X T X+NU) -1 X T Y.

[0202] The denoised pixel is given by the vector-matrix product xA, where x is the structure vector of the pixel (a row of the X matrix) and A is the model built from the neighborhood of the target pixel. The denoised output is obtained by multiplying the guidance image with the corresponding local model parameters A at each pixel. Figure 3 As shown, the calculation of the model parameter A can be divided into several stages. The first stage involves calculating the term x T X and X T Y, which are sometimes referred to as "moment matrices" or simply "moments". This can then be divided into two stages. First, a per-pixel outer product calculation is performed. Second, the outer products are summed over a local neighborhood ("blurred") to produce a moment matrix. Although the equation above (which explains how the matrix A is calculated for a given local neighborhood) does not indicate this, it can be shown that spatially blurring the outer product terms after they have been calculated is equivalent to explicitly constructing the X and Y matrices and performing the matrix multiplications required to produce the moment matrix. The calculation of the outer product by the model fitting block 220 is given by Figure 3 We refer to the output of step 122 as the first outer product x. T x and the second outer product x T y. In step 124, the model fitting block 220 fuzzifies the outer product to produce a moment matrix. We refer to the output of the fuzzification step as the first moment matrix X T X and the second moment matrix X TY. Blurring can be performed in a variety of different ways, and will be described in further detail below.

[0203] After blurring the moments in step 124, the model fitting block 220 adds regularization and solves a least squares problem. This involves inverting the regularized first moment matrix and then multiplying the result of the matrix inversion by the second moment matrix, as indicated by the equation above. The output of step 126 is the model matrix for each pixel of the low-resolution image. In the method of He et al., the model parameters are further blurred spatially. However, the inventors have found that this is unnecessary and actually results in lower image quality in many cases. Therefore, the method proceeds without blurring the model parameters.

[0204] As explained above, the model parameters are upsampled by the upsampler 230 in step 130. This converts them to full resolution. They are then combined (in step 150) with the full resolution guidance image produced by the rasterizer. As discussed previously, this operation implements the formula Y=XA.

[0205] Tiling

[0206] Figure 4 and Figure 5 Some optional additional steps that can be introduced into model fitting 120 are shown in . The calculations involved in model fitting are somewhat computationally intensive and data intensive. The inventors have recognized that it may be advantageous to perform model fitting in a tile-based manner. This approach involves decomposing the data into tiles (generally speaking, they may be overlapping or non-overlapping) and processing each tile separately. This has several potential advantages. First, the size of the tile can be selected so that all the data required to perform the calculation of one tile can be stored in the on-chip memory in the GPU. This can avoid the need to read data from external memory and write data to external memory, which is typically a slower operation than accessing on-chip memory. Secondly, when multiple processors or cores are available in the GPU, the ability to divide processing into tiles facilitates parallel implementation. However, it should be noted that tiling is useful independently of parallelization due to the first advantage. Even if tiles are processed continuously by a single core or a single processor, the effects of greater data locality and reduced memory access bandwidth can still promote increased processing speed and / or reduced power consumption.

[0207] Tiling can be applied in at least two ways. Figure 4As shown, tiling can be used in the calculation of the outer product. In step 115, the GPU divides the low-resolution noisy indirect illumination image and the guidance image into tiles. Then, in step 122a, as a variation of step 122, the outer product of each tile is calculated independently of the other tiles. The tiles created in step 115 are non-overlapping because the data required for the outer product calculation is different from each tile to the next tile. This is because the calculation of the outer product is inherently a pixel-by-pixel operation.

[0208] Alternatively or additionally, tiling can be used in the blur, which computes the moment matrix from the outer product. Figure 5 . The outer product generated in step 122 (or step 122a) is divided into overlapping tiles in step 125. In step 124a, blurring is applied to each tile independently. It should be noted that due to the spatial extent of the local neighborhoods used to create the model, the tiles used for blurring must overlap to some extent. These local neighborhoods are reflected in the size of the filter kernel used for blurring.

[0209] Vague

[0210] Blurring (step 124 or step 124a) can be performed in a variety of ways. The purpose of blurring is to transform the outer product x over a local neighborhood centered on the pixel of interest. T x and x T y are summed to generate the moment matrix X T X and X T Y. Optionally, the sum can be a weighted sum, typically emphasizing the contribution of pixels in the center of each neighborhood and de-emphasizing the contribution of pixels in the periphery of the neighborhood. There are a number of ways to do this efficiently. In one example, the computation of the moment matrix is ​​accelerated by reusing the computation between overlapping regions and utilizing fast image filtering techniques. Let each pixel in X be represented by a row vector x, and each pixel in Y be represented by a row vector y. For each output position, compute x T x and x T y, i.e. the outer product. Considering the entire image, this gives a 4D tensor with exemplary dimensions [H,W,N,N] and [H,W,N,M], where H and W are the image height and width, and N and M are the number of channels in X and Y, respectively. A 2D box filter is convolved over the first two dimensions of these tensors. For efficiency, this box filter can be implemented using integral images or separable horizontal and vertical kernels. This can be shown to be equivalent to constructing the moment matrix:

[0211]

[0212]

[0213] Here, {X} i,jis the element in the i-th row and j-th column of the matrix X. p,: is the row vector of the pth row of X. :,i is the column vector of the i-th column of X. In this way, the moment matrix X T X can be written as the sum of the outer products of the rows of X. Constructing it this way is efficient due to the overlap between local neighborhoods and the fact that linear filters can be computed cheaply. A similar formula applies to the second moment matrix X T Y.

[0214] The above filter implements a simple box filter. This can be accelerated in a number of ways. One option is to use an integral image (also called a summed area table) to precompute the sum of the outer products up to and including each pixel in the low-resolution image. The sum of the outer products in any desired area can then be derived from the integral image based on the integral values ​​at the four corners of the rectangle.

[0215] Another option is to use a so-called "running box filter". This works on the following principle: by adding the values ​​that have entered the window and subtracting the values ​​that have fallen out of the window, the continuous sum of the outer product can be calculated incrementally for each shift of the window that defines the local neighborhood. This can be better understood by referring to a simple one-dimensional example. Suppose it is desired to calculate a three-pixel box filter along a row of pixels a, b, c, d, e. The first sum is a+b+c. The second sum is b+c+d. This can be calculated by taking the previous value (a+b+c), adding d (it enters the window) and subtracting a (it falls out of the window). This results in the calculation (a+b+c)+da=b+c+d. The process is repeated in the next step: (b+c+d)+eb=c+d+e. It should be noted that although this does not change the number of calculations at each shift of the window, for this simplified three-pixel example, it has considerable benefits for larger box filters. Regardless of the filter size, at each shift, the required sum can be computed with a single subtraction and a single addition. The possibility of separable 1-D horizontal and vertical box filtering discussed above can be combined with the use of a running box filter. That is, the running box filter can be implemented separately in two dimensions.

[0216] Although the box filter is computationally efficient, it gives equal weight to all pixels in a local neighborhood. Better results can be obtained by giving more influence to pixels near the center of the neighborhood than to pixels at the edge. The challenge is to find efficient ways to implement this more complex kind of filtering in a way that achieves a good balance between computational efficiency and image quality.

[0217] One approach that offers good potential to reconcile these competing demands is to implement blur using mipmaps. Such mipmaps are known in the computer graphics literature, where they have been applied in the context of texture sampling. It uses a scaled space pyramid where each level of the pyramid (sometimes referred to in the art as a "chain") is generated by downsampling the previous level. In some examples, each level of the pyramid is generated from the previous level through a 2×2 box filter. This can be computed recursively by bilinearly resampling the previous level. The inventors have recognized that a mipmap pyramid can be computed for each channel of the outer product, and these mipmaps can then be used to support an efficient implementation of blur. Sometimes, it is not necessary to compute all levels of the mipmap pyramid (as discussed in more detail below). This can save time, power, memory, and bandwidth.

[0218] Fig. 6A Shows Figure 3 A variation of the method of incorporating the use of mipmaps. In step 122, the model fitting block 220 computes the outer product (as described above); and in step 322, the model fitting block creates a mipmap from the outer product as part of the blurring process. This process is Figure 6B . At the bottom level of the pyramid is the original outer product tensor 10 (shown as a 2D array for simplicity, although it is actually a higher dimensional tensor). This is spatially (e.g., bilinearly) downsampled to produce the second level of the pyramid 12. The second level is downsampled to produce the third level 14, and this in turn is downsampled to produce the fourth and final level 16. Figure 6C is an illustration of bilinear downsampling when generating the second level from the bottom level. Four pixels 10-1 to 10-4 of the original outer product array are averaged to generate one pixel 12-1 of the second level of the pyramid. This is repeated for each adjacent 2×2 pixel block of the original outer product array. Thus, each level of the pyramid is approximately one quarter the size of the previous level. In general, this process can continue to iterate until one or both spatial dimensions reach 1, or until a sufficiently low resolution has been reached.

[0219] Once the mipmap has been calculated for each outer product, a blur operation is implemented by selecting the relevant level of the pyramid and sampling appropriately to obtain a blurred version of the outer product (step 324). In some examples, the blurring only includes sampling operations at the appropriate level of the mipmap pyramid. But in other examples, the blurring may optionally include applying a filter to samples retrieved from the mipmap. Due to the reduced resolution of the higher pyramid levels, the mipmap can be filtered with a smaller number of calculations than if a filter of the same size was applied directly to the outer product. In step 126, the blur moments output from the blur step 324 are processed as previously described.

[0220] Some examples of mipmap-based implementations will now be described. Typically, one or more mipmap levels are selected based on the size of the blur filter kernel we wish to construct. In some examples, as described above, individual samples from the mipmap can be used directly to generate blur moments. This will implement a relatively simple square blur filter whose effective kernel size corresponds to the level of the selected pyramid. It should be noted that if the kernel size is fixed in advance, then only a single level of the mipmap pyramid is actually required. The construction of the mipmap can terminate at this level. Alternatively, the required single level can be generated directly (i.e., no recursion is required). In its simplest form, blurring can be performed based on nearest neighbor sampling, although this may have disadvantages in the form of potential aliasing artifacts.

[0221] In other examples, the mipmap pyramid can be sampled bilinearly at a single level of detail. In still other examples, the mipmap can be sampled between levels using trilinear sampling. This involves bilinear sampling at two consecutive levels of detail and linear interpolation between the results. In still other examples, the mipmap can be sampled using bicubic sampling. Bilinear sampling, trilinear sampling, and bicubic sampling are commonly used for texture sampling applications of mipmaps, and therefore, these operations may be hardware accelerated on some GPUs. This can be used to further speed up blur calculations.

[0222] If necessary, the samples extracted from the mipmap can then be (further) filtered (by any chosen sampling scheme) to produce blur moments, typically using a small low-pass filter kernel. Figure 7An example of blurring 324 in this manner is shown. Samples are extracted from a predetermined level of the mipmap pyramid and filtered first with a horizontal Gaussian filter 402 and second with a vertical Gaussian filter 404. Separable filtering using two consecutive, orthogonal one-dimensional filters in this manner can further reduce computational complexity and increase speed. The use of a Gaussian kernel is an example of giving greater weight to samples at the center of a neighborhood than to samples at the edge. Other examples are also possible. For example, an infinite impulse response (IIR) filter can be used instead of a Gaussian filter.

[0223] Fig. 8A and Figure 8B Another example of mipmap-assisted blurring is shown. The inventors have discovered that it may be advantageous to consider the scene content when performing blurring. In particular, it may be advantageous to consider the orientation of the surface relative to the virtual camera. The reason for this is Fig. 8A . When the plane in the scene is parallel to the image plane, a circular area 25 on the image plane maps to a circular area 20a on the plane in the scene. However, when the plane in the scene is not parallel to the image plane, the same circular area 25 on the image plane maps to an elliptical area 20b on the plane in the scene. It is advantageous to take this into account for blurring. Instead of using a blur kernel that is isotropic (height=width) throughout the image, the shape of the kernel can be anisotropic (height≠width) and can be locally adapted to the orientation of the surface on which the blurring is being performed. The goal is to ensure that the projection of the kernel onto the scene is approximately constant and isotropic in scene space.

[0224] Figure 8BA method for applying this principle with the help of the mipmap created in step 322 is shown. In step 502, the model fitting block 220 determines the size and orientation of the appropriate major and minor axes of the filter footprint on the image plane based on the scene content at each pixel location. This is obtained from a combination of the screen space surface normal (e.g., available from the associated guidance channel) and the filter kernel size. The major and minor axes do not need to be axis-aligned (i.e., horizontal and vertical). Anisotropic filtering will be performed based on the determined major and minor axes, as described in the following example. In step 504, the model fitting block 220 selects an appropriate level of detail in the mipmap pyramid, for example by calculating log2 (the length of the minor axis in pixels). In general, this is not necessarily an integer value. It can be rounded to the nearest integer, or trilinear sampling can be used to obtain the level of detail between the pyramid levels. In step 506, the model fitting block 220 identifies points along the length of the major axis at the level of detail selected in step 504. It extracts outer product samples from the mipmap at the specified level of detail through appropriate sampling. This can be, for example, point sampling, bilinear sampling, or bicubic sampling. The output of step 506 is the average of the sampled points along the major axis. This process is repeated for each pixel. In some examples, the average can be a weighted average with a larger weight near the center of the kernel. The weights of the weighted average do not need to sum to one.

[0225] Of course, it should be understood that using mipmap / multi-scale pyramids is only one way to effectively implement blurring. Other effective filtering methods are also possible. For example, an IIR filter suitable for effectively implementing blurring is described in GB 2541179A.

[0226] Normalization

[0227] As explained above, the regularization is added to the moment matrix X before matrix inversion. T X, in order to increase numerical stability. However, the inventors have found that it is advantageous not to rely entirely on regularization to avoid numerical instability. Better quality rendering results can be achieved by (alternatively or additionally) normalizing the moment matrices. In this example, both moment matrices are normalized so that they appear to be generated from data with zero mean and unit standard deviation. This is achieved by extracting mean and standard deviation statistics from the original moment matrices. The first moment matrix X T X can be divided into:

[0228]

[0229] The normalized version can be extracted by the following calculation:

[0230]

[0231] in

[0232]

[0233] Here, μ is the mean and σ is the standard deviation of the data X. Similarly, the second moment matrix X T Y can be rewritten as:

[0234]

[0235] in is a row vector of 1's of compatible length.

[0236] From this the normalized version can be extracted as:

[0237]

[0238] The model is constructed from normalized matrices rather than raw moment matrices. When the model is later applied in the model assembly block 250, the transformation used for normalization should be inverted. In particular, the model assembly block “normalizes” the model parameters by applying To compute the product XA:

[0239]

[0240] Normalization has been found to help reduce unwanted blurring at the boundaries of structures in the scene (e.g., helps avoid "glow" around objects), as well as reducing noise from poorly conditioned matrices.

[0241] Quantification

[0242] Memory bandwidth and storage can be further reduced by quantizing the data for transmission to and from the GPU. Although it may be beneficial to perform calculations in floating point format, it is also possible to store and transmit data in a low bit depth integer format. This can reduce the bandwidth required to transfer data between the GPU and the memory. Memory access bandwidth is a particularly precious resource when transferring data to off-chip memory. Experiments have shown that the guide image can be stored in an 8-bit integer format (signed or unsigned, depending on the numerical range of each guide channel). The image quality of the resulting denoised image is quite close to that achieved using a floating point guide channel. Experiments have shown that more care may be needed with the illumination image. In particular, it has been found that direct (linear) quantization produces suboptimal results in terms of the quality of the denoised output image. The inventors have found that the use of nonlinear quantization can give better results in this context while still achieving a reduction in memory access bandwidth. Fig. 9A way of encoding 610 and decoding 620 illumination images for use in the rendering method of this example is shown. This can be particularly applied to noisy indirect illumination images generated by the path tracing block 210. However, it can also be applied to other illumination images, such as direct illumination images generated by the ray tracing block 260, and the final rendered output image. In step 612, nonlinearity is applied to the floating point illumination image. That is, the input floating point values ​​are transformed by mapping them to second floating point values ​​using a nonlinear function. Then, in step 614, the transformed input values ​​are uniformly quantized. The combination of nonlinearity and uniform quantization provides a larger density of quantization levels for darker areas of the image, while brighter areas of the image are relatively sparsely quantized. Suitable nonlinearities for achieving this effect include square root functions and logarithmic functions. After quantization in step 614, the quantized floating point numbers are converted into low bit depth integers in step 616. The low bit integers are then transferred from the GPU to memory (e.g., on-chip or external memory). Quantization reduces the amount of data to be stored in memory, and more importantly, reduces the bandwidth required to transfer data to and from memory. In this specific implementation, the low bit depth integers are stored with a bit depth of 8 bits. However, different low bit depth formats can be used for different guide channels and / or noisy images. That is, during any given process of producing a denoised image, different guide channels and noisy images do not need to use the same low bit depth format, and they can be encoded and decoded independently of the different formats. For example, this may be advantageous if it is found that using more bits for depth than for the surface normal component is beneficial to the overall result.

[0243] When the data is needed (e.g., when it is retrieved from memory for computing an inner product), it is decoded 620 to reverse the nonlinear quantization. In step 622, the low bit depth integers are converted back to floating point numbers. Then, in step 624, an inverse nonlinearity is applied to reverse the effect of the nonlinear function in step 612. For example, if a square root function was applied in step 612, a quadratic (square) function would be applied in step 624. If a logarithmic function was applied in step 612, the function applied in step 624 would involve an exponent.

[0244] The denoised output image may also be quantized after being produced. This provides similar benefits to those described above by outputting the denoised image at a lower bit depth (compared to without the quantization step), i.e., bandwidth is saved when outputting the denoised image. This may be particularly beneficial because the denoised output is typically significantly larger than the noisy low-resolution input, and therefore would represent a significantly greater bandwidth load if output in an unquantized manner.

[0245] Alternative pipeline

[0246] Figure 1 The pipeline shown is not the only way to render an image using low-resolution denoising via guided filtering. Fig.10 Another example is shown. Fig.10 and Figure 1 The difference is that the noisy low resolution indirect illumination image is combined with the direct illumination image prior to the denoising procedure. In steps 110 and 112, the low resolution noisy indirect illumination image is rendered by path tracing, and the low resolution guide pass is rendered by rasterization, as previously described. However, instead of rendering the full resolution direct illumination image in step 160, the ray tracing block renders a low resolution direct illumination image in step 760. In step 770, the low resolution direct illumination image and the low resolution indirect illumination image are summed. The resulting global illumination image is noisy because it includes indirect illumination. In step 720, this noisy image is passed to the model fitting block, and model parameters are estimated in the same manner as described above for step 120. The resulting model parameters produced at a low resolution are upsampled in step 730 and combined with the full resolution guide pass in step 750. This produces a full resolution denoised global illumination image. As Figure 1 As shown, in step 180, the global illumination image is combined with the surface reflection image to produce a final rendered output image. Fig.10 For best results of the pipeline, the low-resolution image is preferably the same size (same resolution) as the full-resolution image. Otherwise, there is a risk of suboptimal reconstruction of shadow edges, introducing undesirable blurring on these edges.

[0247] In another variation of the pipeline, ray tracing 160 or 760 may be replaced by rasterization, optionally using shadow mapping, for example.

[0248] Reference again Figure 1 , it is explained above that the resolution of the "full resolution" image is greater than or equal to the resolution of the "low resolution" image. When the "low resolution" image has the same resolution as the "full resolution" image, it should be understood that the rasterization in steps 112 and 140 can be performed in a single step because the same set of guidance channels is used when calculating the model parameters and applying them. Alternatively, they can still be performed as two steps (such as Figure 1 As shown specifically, step 140 repeats the rasterization of the guide channel performed in step 112.

[0249] Alternative lighting modes

[0250] While the above example combines a direct lighting image and an indirect lighting image to provide global illumination, this is not the only possible implementation. For example, in other examples, instead of an indirect lighting image, a low-resolution path tracing step similar to step 110 can implement ray-traced ambient occlusion. This is a simplified approximation of indirect lighting, but it still relies on random sampling; therefore, the resulting path-traced image is still noisy and can benefit from a combined rendering and denoising pipeline. Figure 1 The pipeline or Fig.10 Any of the pipelines can be used with ray traced ambient occlusion.

[0251] In some examples using ray traced ambient occlusion, the direct lighting image is combined with an image produced by ray traced ambient occlusion at some stage in the pipeline. Figure 1 and Fig.10 However, this is not necessary. The method can be part of a larger rendering pipeline that uses, for example, more complex rasterization techniques than simply combined with direct lighting.

[0252] Alternative model fitting

[0253] Although the above example models the noisy input image as a linear combination of the guiding channels in an affine model, the scope of the present disclosure is not limited thereto. Other models may be used, including, for example, nonlinear / higher order models. An example of such a higher order model would be a quadratic fit. A quadratic model may be implemented by including the square of any channel in the guiding image.

[0254] system

[0255] Fig.11 906. A graphics processing system is shown in which a graphics processing unit described herein may be implemented. The computer system includes a CPU 902, a GPU 904, a memory 906, and other devices 914, such as a display 916, a speaker 918, and a camera 919. Processing block 910 (corresponding to processing blocks 210-280) is implemented on GPU 904. In other examples, processing block 910 may be implemented on CPU 902. The components of the computer system may communicate with each other via a communication bus 920. A storage repository 912 (corresponding to storage repository 112) is implemented as part of the memory 906.

[0256] Although Fig. 9One specific implementation of a graphics processing system is shown, but it will be understood that a similar block diagram can be drawn for an artificial intelligence accelerator system—for example, by replacing the CPU 902 or GPU 904 with a neural network accelerator (NNA), or by adding an NNA as an additional unit. In such cases, processing block 910 can be implemented in the NNA.

[0257] Figure 2 GPU and Figure 1 and Fig.10 The method / pipeline is shown as comprising a plurality of functional blocks. This is merely illustrative and is not intended to define a strict division between different logical elements of such entities. Each functional block may be provided in any suitable manner. It should be understood that intermediate values ​​described herein as being formed by the GPU need not be physically generated by the GPU at any point, and may merely represent logical values ​​that conveniently describe the processing performed by the GPU between its input and output.

[0258] The GPU and graphics processing system described herein can be embodied in hardware on an integrated circuit. The GPU and graphics processing system described herein can be configured to perform any method described herein. In general, any of the functions, methods, techniques or components described above can be implemented in software, firmware, hardware (e.g., fixed logic circuitry), or any combination thereof. The terms "module", "functionality", "component", "element", "unit", "block" and "logic" can be used herein to generally represent software, firmware, hardware, or any combination thereof. In the case of software specific implementation, modules, functionalities, components, elements, units, blocks or logic represent program code, which performs specified tasks when executed on a processor. The algorithms and methods described herein can be performed by one or more processors that execute code, which prompts the processor to execute the algorithm / method. Examples of computer-readable storage media include random access memory (RAM), read-only memory (ROM), optical disks, flash memory, hard disk storage, and other memory devices that can use magnetic, optical and other technologies to store instructions or other data and can be accessed by a machine.

[0259] As used herein, the terms computer program code and computer readable instructions refer to any kind of executable code for execution by a processor, including code expressed in machine language, interpreted language, or scripting language. Executable code includes binary code, machine code, byte code, code that defines an integrated circuit (such as a hardware description language or netlist), and code written in languages ​​such as C, Executable code can be, for example, any kind of software, firmware, script, module or library that, when properly executed, processed, interpreted, compiled, or run in a virtual machine or other software environment, enables the processor of a computer system supporting the executable code to perform the tasks specified by the code.

[0260] A processor, computer or computer system may be any kind of device, machine or special purpose circuit, or a collection or portion thereof having processing capabilities so that it can execute instructions. A processor may be any kind of general or special purpose processor, such as a CPU, GPU, NNA, system on chip, state machine, media processor, application specific integrated circuit (ASIC), programmable logic array, field programmable gate array (FPGA), etc. A computer or computer system may include one or more processors.

[0261] The present invention is also intended to cover software that defines the configuration of hardware as described herein, such as hardware description language (HDL) software, for designing integrated circuits or for configuring programmable chips to perform desired functions. That is, a computer-readable storage medium having computer-readable program code encoded thereon in the form of an integrated circuit definition data set may be provided, which, when processed (i.e., run) in an integrated circuit manufacturing system, configures the system to manufacture a graphics processing system configured to perform any of the methods described herein, or to manufacture a graphics processing system including any of the apparatus described herein. The integrated circuit definition data set may be, for example, an integrated circuit description.

[0262] Thus, a method of manufacturing a graphics processing system as described herein at an integrated circuit manufacturing system may be provided. Furthermore, an integrated circuit definition data set may be provided which, when processed in an integrated circuit manufacturing system, enables the method of manufacturing a graphics processing system to be performed.

[0263] The integrated circuit definition data set may be in the form of computer code, for example as a netlist, code for configuring a programmable chip, as a hardware description language that defines hardware suitable for fabrication at any level in an integrated circuit, including as register transfer level (RTL) code, as a high-level circuit representation such as Verilog or VHDL, and as a low-level circuit representation such as OASIS (RTM) and GDSII. Higher-level representations that logically define hardware suitable for fabrication in an integrated circuit, such as RTL, may be processed at a computer system configured to generate a manufacturing definition of the integrated circuit in the context of a software environment that includes definitions of circuit elements and rules for combining those elements to generate a manufacturing definition of the integrated circuit so defined by the representation. As is typically the case with software executed at a computer system to define a machine, one or more intermediate user steps (e.g., providing commands, variables, etc.) may be required to configure the computer system to generate a manufacturing definition of the integrated circuit to execute code that defines the integrated circuit to generate the manufacturing definition of the integrated circuit.

[0264] Now about Fig.12 An example of processing an integrated circuit definition data set at an integrated circuit manufacturing system to configure the system to manufacture a graphics processing system is described.

[0265] Fig.12 An example of an integrated circuit (IC) manufacturing system 1002 is shown, which is configured to manufacture a graphics processing system as described in any example herein. In particular, the IC manufacturing system 1002 includes a layout processing system 1004 and an integrated circuit generation system 1006. The IC manufacturing system 1002 is configured to receive an IC definition data set (e.g., defining a graphics processing system as described in any example herein), process the IC definition data set, and generate an IC (e.g., which includes a graphics processing system as described in any example herein) based on the IC definition data set. By processing the IC definition data set, the IC manufacturing system 1002 is configured to manufacture an integrated circuit including a graphics processing system as described in any example herein.

[0266] The layout processing system 1004 is configured to receive and process an IC definition data set to determine a circuit layout. Methods for determining a circuit layout based on an IC definition data set are known in the art and may, for example, involve synthesizing RTL code to determine a gate-level representation of a circuit to be generated, for example, in terms of logic components (e.g., NAND, NOR, AND, OR, MUX, and FLIP-FLOP components). By determining location information for the logic components, the circuit layout may be determined based on the gate-level representation of the circuit. This may be done automatically or with user involvement in order to optimize the circuit layout. When the layout processing system 1004 has determined the circuit layout, it may output the circuit layout definition to the IC generation system 1006. The circuit layout definition may be, for example, a circuit layout description.

[0267] As is known in the art, IC production system 1006 produces ICs according to the circuit layout definition. For example, IC production system 1006 may implement a semiconductor device manufacturing process to produce ICs, which may involve a multi-step sequence of photolithography and chemical processing steps during which electronic circuits are gradually formed on a wafer made of semiconductor material. The circuit layout definition may be in the form of a mask that may be used in a photolithography process to produce the IC according to the circuit definition. Alternatively, the circuit layout definition provided to IC production system 1006 may be in the form of a computer readable code that IC production system 1006 may use to form a suitable mask for producing the IC.

[0268] The different processes performed by the IC manufacturing system 1002 may all be implemented at one location, such as by one party. Alternatively, the IC manufacturing system 1002 may be a distributed system such that some processes may be performed at different locations and may be performed by different parties. For example, some of the following stages may be performed at different locations and / or by different parties: (i) synthesizing RTL code representing an IC definition data set to form a gate-level representation of a circuit to be generated; (ii) generating a circuit layout based on the gate-level representation; (iii) forming a mask based on the circuit layout; and (iv) using the mask to manufacture the integrated circuit.

[0269] In other examples, processing of an integrated circuit definition data set in an integrated circuit manufacturing system may configure the system to manufacture a graphics processing system without processing the IC definition data set to determine a circuit layout. For example, the integrated circuit definition data set may define a configuration of a reconfigurable processor such as an FPGA, and processing of the data set may configure the IC manufacturing system to generate a reconfigurable processor having the defined configuration (e.g., by loading the configuration data into the FPGA).

[0270] In some embodiments, when processed in an integrated circuit manufacturing system, the integrated circuit manufacturing definition data set can cause the integrated circuit manufacturing system to generate an apparatus as described herein. For example, by using the integrated circuit manufacturing definition data set as described above with reference to Fig.12 The integrated circuit manufacturing system can be configured in the described manner to manufacture the device described herein.

[0271] In some examples, the integrated circuit definition data set may include software that runs on, or in combination with, the hardware defined at the data set. Fig.12 In the example shown, the IC production system can be further configured by the integrated circuit definition dataset to load firmware onto the integrated circuit according to the program code defined at the integrated circuit definition dataset when manufacturing the integrated circuit, or otherwise provide the integrated circuit with program code for use with the integrated circuit.

[0272] Compared with known implementations, the implementation of the concepts set forth in this application in devices, equipment, modules and / or systems (and in the methods implemented herein) can improve performance. Performance improvements may include one or more of improved computing performance, reduced latency, increased throughput and / or reduced power consumption. During the manufacture of such devices, devices, modules and systems (e.g., in integrated circuits), a trade-off may be made between performance improvements and physical implementations to improve manufacturing methods. For example, a trade-off may be made between performance improvements and layout area to match the performance of known implementations, but using less silicon. For example, this can be accomplished by reusing functional blocks in a serial manner or sharing functional blocks between elements of devices, devices, modules and / or systems. In contrast, the concepts set forth in this application that cause improvements (such as reduced silicon area) in the physical implementation of devices, devices, modules and systems can be weighed against performance improvements. This can be accomplished, for example, by manufacturing multiple instances of a module within a predefined area budget.

[0273] Applicants hereby independently disclose each individual feature described herein and any combination of two or more such features, to the extent that such feature or combination can be implemented according to the common general knowledge of a person skilled in the art based on the specification as a whole, regardless of whether such feature or combination of features solves any problem disclosed herein. In view of the foregoing description, it will be clear to a person skilled in the art that various modifications can be made within the scope of the invention.

Claims

1. A method for rendering an image of a 3D scene, the method comprising: Rendering (110) a noisy image; Obtaining (112) one or more guide channels; as well as For each of multiple local neighborhoods: Calculate (120) the parameters of a model that approximates the noisy image denoted by Y as a function of the one or more guiding channels denoted by X by finding an affine transformation A, wherein the affine transformation A is found such that the matrix equation Minimize, where and And N is the number of pixels in the local neighborhood, G is the number of guide channels, and M is the number of output channels; and Applying the parameters calculated in (130, 150) to generate a denoised image; The method further comprises: quantizing at least one of the noisy image, the one or more guide channels, and the denoised image into a quantized lower bit depth format; and The at least one of the noisy image, the one or more guide channels, and the denoised image is stored in the quantized lower bit depth format.

2. The method of claim 1, wherein rendering (110) the noisy image comprises: Rendering via path tracing.

3. A method as claimed in claim 1 or claim 2, wherein the noisy image (i) comprises indirect lighting in the scene, or (ii) comprises an ambient occlusion image.

4. The method of claim 3, wherein the denoised image is a denoised indirect illumination image or a denoised ambient occlusion image, and wherein the method further comprises: obtaining (160) a directly illuminated image; as well as The denoised image is combined (170) with the direct illumination image to produce a global illumination image.

5. The method of claim 4, wherein obtaining (160) the direct illumination image comprises: The directly illuminated image is rendered by ray tracing or the directly illuminated image is rendered by rasterization.

6. The method of claim 3, wherein the noisy image is a noisy global illumination image, including direct and indirect illumination in the scene, whereby the denoised image is a denoised global illumination image.

7. The method of claim 6, further comprising combining (180) the global illumination image or the denoised global illumination image with a surface reflectance image to produce a rendered image of the 3D scene.

8. The method of claim 1 or claim 2, wherein obtaining (112) the one or more guide passes comprises rendering by rasterization.

9. The method of claim 1 or claim 2, wherein the guide channel comprises any one or any combination of two or more of the following: Depth information of objects in the 3D scene; Identify information about materials of objects in the 3D scene; Surface reflections of objects in the 3D scene; Shadows in the 3D scene; and The surface normals of objects in the 3D scene.

10. The method of claim 1 or claim 2, comprising: defining (115) a first tile, the first tile defining respective first contiguous portions of the noisy image and the one or more guide channels, each first contiguous portion comprising a first plurality of pixels; defining (115) a second patch, the second patch defining respective second contiguous portions of the noisy image and the one or more guide channels, each second contiguous portion comprising a second plurality of pixels; Calculate (122a) a first outer product (x) between each pixel (x) in the one or more guide channels and itself. T x); and Calculate (122a) a second outer product (x) between each pixel (x) in the one or more guiding channels and a corresponding pixel (y) in the noisy image. T y), For pixels in the first image block, the first outer product and the second outer product are calculated (i) before the second image block or (ii) in parallel with the second image block.

11. The method of claim 1 or claim 2, further comprising, after rendering (110) the noisy image, quantizing it in a quantized low bit depth format using non-linear quantization (610) such that darker areas of the image are quantized to a relatively larger quantization level density and lighter areas of the image are quantized to a relatively smaller quantization level density; and storing the quantized low bit depth format in a memory, The method further includes, prior to calculating the parameters of the model, retrieving the quantized low bit depth values ​​from the memory and performing inverse quantization (620).

12. The method of claim 1, wherein calculating the parameters of the model comprises: Calculate (122, 122a) a first outer product (x) between each pixel (x) in the one or more guide channels and itself. T x); Calculate (122, 122a) a second outer product (x) between each pixel (x) in the one or more guiding channels and a corresponding pixel (y) in the noisy image. T y); The first outer product of blur (124, 124a) is used to calculate the first moment matrix (X) of each local neighborhood. T X); The second outer product of blur (124, 124a) is used to calculate the second moment matrix (X) of each local neighborhood. T Y); and Calculating (126) the parameters (A) of the model for each local neighborhood comprises calculating the inverse matrix of the matrix of the first moments and calculating the product of the inverse matrix and the matrix of the second moments.

13. The method of claim 12, wherein blurring (124, 124a) the first outer product comprises computing (322) a first multiscale pyramid from the first outer product and computing (324) a matrix of the first moments based on the first multiscale pyramid; and / or Wherein blurring (124, 124a) the second outer product comprises computing (322) a second multiscale pyramid from the second outer product and computing (324) a matrix of the second moments based on the second multiscale pyramid.

14. A method as claimed in claim 12 or claim 13, wherein the blurring comprises separable filtering (402, 404) in horizontal and vertical directions.

15. The method of claim 12 or claim 13, wherein the blurring comprises filtering using an anisotropic 2D filter (502, 504, 506).

16. The method of claim 13, wherein the one or more guide channels include surface normals of objects in the 3D scene, and wherein the blurring comprises: For each local neighborhood, determining (502) a major axis and a minor axis of a 2D filter based on the surface normal of the object at the center of the neighborhood; selecting (504) a level of the multi-scale pyramid based on the length of the minor axis; and The selected level of the multi-scale pyramid is sampled along the major axis (506).

17. The method of claim 12 or claim 13, wherein the blurring comprises IIR filtering.

18. The method of claim 12 or claim 13, wherein blurring comprises filtering with a running box filter.

19. The method of claim 12 or claim 13, comprising: defining (125) a first outer product patch, defining first contiguous portions of the first outer product and corresponding first contiguous portions of the second outer product, each first contiguous portion comprising a first plurality of pixels; as well as defining (125) a second outer product patch, defining second contiguous portions of the first outer product and corresponding second contiguous portions of the second outer product, each second contiguous portion comprising a first plurality of pixels, Wherein, for the first outer product block, the matrix of the first moments and the matrix of the second moments are calculated (124a) (i) before the second outer product block or (ii) in parallel with the second outer product block.

20. The method of claim 12 or claim 13, further comprising normalizing and / or regularizing one or both of the first matrix of moments and the second matrix of moments.

21. A graphics processing unit, the graphics processing unit being configured to render an image of a 3D scene, the graphics processing unit comprising: a first rendering block (210), the first rendering block being configured to render (110) a noisy image; a second rendering block (240) configured to render (112) one or more guide channels; a model fitting block (220) configured to compute (120), for each of a plurality of local neighborhoods, parameters of a model that approximates the noisy image denoted by Y as a function of the one or more guiding channels denoted by X by finding an affine transformation A, wherein the affine transformation A is found such that the matrix equation Minimize, where and And N is the number of pixels in the local neighborhood, G is the number of guide channels, and M is the number of output channels; and a model application block (230, 250) configured to apply (130, 150) the calculated parameters for each of the plurality of local neighborhoods to produce a denoised image, Wherein at least one of the noisy image and the one or more guiding channels is stored in a quantized low bit depth format.

22. The graphics processing unit of claim 21, wherein the first rendering block (210) is configured to render (110) the noisy image by path tracing.

23. A graphics processing unit as claimed in claim 21 or claim 22, wherein the second rendering block (240) is configured to render (112) the one or more guide passes by rasterization or ray tracing.

24. The graphics processing unit of claim 21 or claim 22, wherein the noisy image comprises indirect lighting in the scene or comprises an ambient occlusion image, wherein the denoised image comprises an indirect lighting image or a denoised ambient occlusion image, and The graphics processing unit further comprises: a fourth rendering block (260), the fourth rendering block being configured to render (160) a directly illuminated image; as well as A combination block (270) is configured to combine the denoised image with the direct illumination image (170) to produce a global illumination image.

25. A graphics processing unit as claimed in claim 21 or claim 22, wherein the noisy image is a noisy global illumination image, the noisy image comprising direct and indirect illumination in the scene, whereby the denoised image is a denoised global illumination image.

26. The graphics processing unit of claim 25, wherein the second rendering block (240) is configured to render (140) a surface reflection image, The graphics processing unit also includes a product block (280) configured to calculate (180) a product of the global illumination image or the denoised global illumination image and the surface reflection image to generate a rendered image of the 3D scene.

27. A computer-readable storage medium having a computer-readable code stored thereon, the computer-readable code being configured to cause the method of any one of claims 1 to 20 to be executed when the computer-readable code is run.

Images