Rendering images of 3D scene

Abstract

A method of rendering an image of a 3D scene is provided. The method comprises: rendering a noisy image at a first resolution; one or more guide channels are obtained at a first resolution and one or more corresponding guide channels are obtained at a second resolution. The second resolution may be the same resolution as the first resolution or a higher resolution than the first resolution. When the two resolutions are the same, one or more guide channels at the first resolution and one or more corresponding guide channels at the second resolution may be provided by a single set of guide channels. For each of the plurality of local neighborhoods, the method includes calculating parameters of a model that approximates the noisy image as a function of the one or more guide channels (at the first resolution), applying the calculated parameters to the one or more guide channels (at the second resolution) to produce a denoised image at the second resolution.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to UK patent application 2315251.5 filed on October 4, 2023, which is incorporated herein by reference in its entirety. Technical Field

[0003] The present disclosure relates to rendering images of 3D scenes. Background Art

[0004] Path tracing is a Monte Carlo method for approximating the transport of light in a scene. The quality of the result depends on the number of samples per pixel - the more samples, the closer the result is to actual light transport.

[0005] However, increasing the number of samples is computationally expensive, not least because the standard deviation of the noise depends on the number of samples N by a factor of This means that four times as many samples are needed to achieve a 50% noise reduction. Therefore, increasing the number of samples quickly becomes impractical as a way to reduce noise in path traced images.

[0006] It is well known that applying denoising algorithms can reduce noise without increasing the number of samples. "Guided filters" have been found to work well in this task. Originally proposed by He et al., guided filters model each neighborhood of a noisy image as an affine transformation of the corresponding neighborhood of a guidance image. The guidance image should be noise-free and should contain scene structures (e.g., object edges, occlusion boundaries, or shadow edges) that correspond to the noisy image. Such guidance images are useful in the context of path tracing because the scene is synthesized and its various "auxiliary" images can be rendered by other means. A guidance image with several guidance channels can be used in a guided filter, and each guidance channel can contain different kinds of information (e.g., a depth channel and a surface normal channel) that can be used to reconstruct the noise-free image. Different combinations of guidance channels may be useful in different parts of the image, and for this reason the approach is called a locally linear (or, more correctly but less commonly, a locally affine) model.

[0007] Because it is guided by information about the structural content of the scene, the guided filter can denoise noisy path-tracing images of the scene without causing significant blurring on the edges of objects within the image, as long as suitable structural information is available in one or more of the guiding channels.

[0008] GB2605472A discloses a method for rendering an image of a 3D scene, in which a noisy image (e.g., obtained by path tracing) is denoised by guided filtering. The guided channel may include: depth information of objects in the 3D scene; information identifying the material of the objects in the 3D scene; surface reflections of objects in the 3D scene; shadows in the 3D scene; and / or surface normals of objects in the 3D scene. In some examples, the noisy image may include indirect lighting in the scene. In some examples, the noisy image may include an ambient occlusion image. The ambient occlusion image may provide an approximation of the indirect lighting.

[0009] To better support path tracing, and in particular to allow path tracing to be performed at higher frame rates and / or better quality on devices with limited computational resources and power (eg, mobile devices), it is desirable to improve the quality of denoising and to implement it more efficiently. Summary of the invention

[0010] This Summary is provided to introduce a selection of concepts in a simplified form that are further described below in the Detailed Description. This Summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used to limit the scope of the claimed subject matter.

[0011] A method for rendering an image of a 3D scene is provided. The method includes: rendering a noisy image at a first resolution; obtaining one or more guide channels at the first resolution, and obtaining one or more corresponding guide channels at a second resolution. The second resolution can be the same resolution as the first resolution or a higher resolution than the first resolution. When the two resolutions are the same, the one or more guide channels at the first resolution and the one or more corresponding guide channels at the second resolution can be provided by a single set of guide channels. For each of a plurality of local neighborhoods, the method includes: calculating parameters of a model that approximates the noisy image as a function of the one or more guide channels (at the first resolution), and applying the calculated parameters to the one or more guide channels (at the second resolution) to produce a denoised image at the second resolution. The one or more guide channels include at least one guide channel that characterizes the spatial dependence of incident light on global illumination on the surface of one or more 3D models in the scene.

[0012] According to a first aspect, there is provided a method for rendering an image of a 3D scene, the method comprising:

[0013] Render noisy images;

[0014] obtaining one or more guided channels; and

[0015] For each of multiple local neighborhoods:

[0016] computing parameters of a model that approximates the noisy image as a function of one or more guiding channels, and

[0017] applying the calculated parameters to the one or more guiding channels to produce a denoised image,

[0018] The one or more guiding channels include at least one guiding channel characterizing a spatial dependency of incident light on global illumination on surfaces of one or more 3D models in the scene.

[0019] According to a second aspect, there is provided a method for rendering an image of a 3D scene, the method comprising:

[0020] Render low-resolution noisy images;

[0021] obtaining one or more low-resolution guidance channels and obtaining one or more corresponding full-resolution guidance channels; and

[0022] For each of multiple local neighborhoods:

[0023] calculating parameters of a model that approximates the low resolution noisy image as a function of the one or more low resolution guiding channels, and

[0024] applying the calculated parameters to the one or more full-resolution guiding channels to produce a full-resolution denoised image,

[0025] Wherein, the one or more low-resolution guide channels and the one or more corresponding full-resolution guide channels each include at least one guide channel, and the at least one guide channel characterizes the spatial dependence of incident light on the global illumination on the surface of one or more 3D models in the scene.

[0026] According to some examples, the method may provide a combined rendering and denoising pipeline suitable for efficient rendering of images generated at least in part using path tracing. The method of approximating a noisy image by such a function of one or more guiding channels corresponds to denoising by guided filtering. The function may include or consist of a linear combination of guiding channels and scalar offsets. Thus, the model may include or may be an affine model.

[0027] In some examples, a scalar offset can be incorporated into the model by including a guide channel with a uniform value (e.g., an array of ones) at each pixel. Optionally, the guide channel is not stored in memory—it can be instantiated on the fly in software or hardware during fitting of the model (e.g., by a model fitting unit).

[0028] It should be noted that the use of at least one guiding channel (e.g., ambient occlusion guides, as defined below) that characterizes the spatial dependence of incident light on global illumination is contrary to the way in which ambient occlusion information might be conventionally expected to be used. A more conventional approach might include ambient occlusion in a noisy image. Alternatively, ambient occlusion data can be combined with a denoised image at the end of a rendering pipeline. It will be noted that in examples of the present method, the noisy image does not include ambient occlusion information. And the denoised image is not combined with any ambient occlusion information. Ambient occlusion information is only introduced into the pipeline by the at least one guiding channel.

[0029] In some examples, guide passes can contain information about scene structure in a 3D scene, including but not limited to object boundaries, occlusion boundaries, and shadow edges. Guide passes can be substantially noise-free. They can be rendered by deterministic calculations (e.g., by rasterization), while noisy images can be rendered by random sampling.

[0030] The relative resolutions of the "low-resolution" and "full-resolution" images can be selected according to the needs of the application. Generally, the "full-resolution" image has a resolution greater than or equal to the resolution of the "low-resolution" image (and therefore, the "full-resolution" image has a greater number of pixels than the "low-resolution" image or the same number of pixels as the "low-resolution" image). However, in most examples, the "full-resolution" image has a higher resolution than the "low-resolution" image.

[0031] The method enables the generation of a denoised full-resolution image from a low-resolution noisy image. This is computationally more efficient and / or more efficient in terms of memory access bandwidth than performing the denoising operation on the full-resolution noisy image. When denoising based on a low-resolution image, the amount of data that must be retrieved from memory and processed can be reduced. This is because the low-resolution guide channel and the local neighborhood of the low-resolution noisy image will generally be smaller (i.e., will contain fewer pixels) than the corresponding neighborhood at the full resolution.

[0032] Each local neighborhood can be a local neighborhood centered on a corresponding pixel of the noisy low-resolution image, whereby a different set of model parameters is calculated for each pixel. Within a given local neighborhood, the contribution of each pixel to the calculation of the model parameters can be weighted so that pixels closer to the central pixel of the local neighborhood have a relatively greater impact than pixels further away from the central pixel. For example, each pixel can be weighted inversely proportional to its distance from the center. In one example, a Gaussian function can be used to weight the contribution.

[0033] Applying the calculated parameters to the one or more full-resolution guide channels may include applying the parameters calculated for the local neighborhood of the low-resolution guide channel to the corresponding local neighborhood of the full-resolution guide channel. Applying the calculated parameters to the one or more full-resolution guide channels may include upsampling the calculated parameters and applying the upsampled calculated parameters to the one or more full-resolution guide channels.

[0034] Obtaining the at least one guiding pass may include: obtaining precomputed texture data containing information about shadows; and projecting the precomputed texture data into screen space to generate the at least one guiding pass. The precomputed texture data may be provided in texture space. During a rendering process, the precomputed texture data is projected into screen space (or rendered in screen space).

[0035] The at least one guide pass may include or consist of ambient occlusion data. In some examples, the ambient occlusion data is contained in a single guide pass.

[0036] In some examples, the at least one guiding channel includes multiple guiding channels, each guiding channel characterizing directional illumination in a different direction. Note that this is different from the use of shadows in the 3D scene in the guiding channel of GB 2605472 A. In GB 2605472 A, shadows in the guiding channel are implicitly based on direct illumination. Indirect / global illumination is never used in the guiding channel - it is only used in noisy images. In contrast, in examples according to the present disclosure, the guiding channel characterizing directional illumination characterizes the spatial dependence of incident light on global illumination above the surface of one or more 3D models in the scene. In other words, the directional illumination characterized in at least one guiding channel is global / indirect illumination, rather than shadows due to direct illumination, as in GB 2605472 A.

[0037] Obtaining the at least one guide channel may include: obtaining one or more ambient occlusion maps of corresponding one or more models in the scene; and projecting the one or more ambient occlusion maps into screen space to generate ambient occlusion guides in screen space. That is, the ambient occlusion guides form the at least one guide channel.

[0038] The ambient occlusion map for each model can be pre-computed in advance, for example when designing the model. Each ambient occlusion map can be represented in texture space. During the rendering process, each individual ambient occlusion map is retrieved and projected into screen space (or rendered in that screen space). This is a relatively computationally inexpensive process.

[0039] The projected ambient occlusion maps may be combined into a single ambient occlusion guide. For example, combining the projected ambient occlusion maps may combine and sum them.

[0040] Rendering the noisy image may include rendering by path tracing. (For the avoidance of doubt: references to "noisy image" are intended to cover "low-resolution noisy image"; references to "denoised image" cover "full-resolution denoised image".)

[0041] In this case, the method can be viewed as denoising a path traced image via guided filtering. Path tracing is computationally intensive because multiple rays need to be cast per pixel, each ray possibly having multiple "bounces". Some examples of the present method may avoid the need to render a full resolution path traced image. The inventors have discovered that comparable results may be achieved more efficiently by using a low resolution image and investing computational work in terms of the number of rays per pixel and / or the number of bounces per ray, rather than rendering a greater number of pixels. In other words, computational work is better invested in producing a less noisy low resolution image and / or a closer approximation of light transport, rather than producing a noisier or more approximate full resolution image.

[0042] Optionally, the noisy image includes indirect lighting in the scene.

[0043] Optionally, the noisy image consists only of indirect illumination. Here, "direct" illumination refers to rays that interact (intersect) with a single object before reaching the virtual camera / viewer. This means that the light travels directly from the light source to the object (or, equivalently, is traced from the object to the light source), and then travels directly from the object to the virtual camera. Therefore, the object is "directly" illuminated by the light source. In contrast, "indirect" illumination refers to rays that have interacted (intersected) with at least two objects between the light source and the virtual camera. For example, the light can be reflected by the first object toward the second object, and can be reflected by the second object toward the virtual camera. The directly illuminated image does not contain any information about the surface reflections of the objects in the scene. The indirect illuminated image does not contain any information about the surface reflections of the object "closest" to the virtual camera, which means the final surface that the light interacts with on its path from the light source to the camera. However, in general, the indirectly illuminated image does contain information about the color of the surfaces "closer" to the light source, because the interaction of the light with these colored surfaces will affect the color of the indirect illumination that falls on the "closest" object. Direct and indirect illumination can be combined before or after denoising. For example, ray tracing can be used to model the directly illuminated image. It is usually low noise or no noise. Indirect lighting is usually noisier than direct lighting.

[0044] Typically, the noisy low-resolution image does not include a ray-traced ambient occlusion image. Although such an image can provide an approximation of indirect lighting (ignoring the position and intensity of the light source), the inventors have found that better results can be achieved by using the ambient occlusion image as one of the guiding passes instead.

[0045] The denoised image may be a denoised indirect illumination image, and the method may further comprise: obtaining a direct illumination image; and combining the denoised image with the direct illumination image to produce a global illumination image.

[0046] The combination may include summing the denoised image and the direct illumination image. In this example, the direct illumination image is combined with the indirect illumination image after denoising.

[0047] Obtaining the directly illuminated image optionally includes rendering it by ray tracing or rendering it by rasterization.

[0048] In such ray tracing, each ray may be cast along a path with exactly one bounce. Rendering a directly lit image by rasterization may include rendering with shadow mapping.

[0049] The noisy image may be a noisy global illumination image, including direct and indirect illumination in the scene, whereby the denoised image is a denoised global illumination image.

[0050] Rendering a noisy global illumination image may include combining (e.g., summing) a noisy indirect illumination image and a direct illumination image. In this example, the direct illumination image is combined with the indirect illumination image before denoising. Alternatively, the noisy global illumination image may be rendered directly by path tracing that simulates direct and indirect illumination.

[0051] The method may further include combining the global illumination image or the denoised global illumination image with the surface reflectance image to produce a rendered image of the 3D scene.

[0052] The combination may include multiplying the global illumination by the surface reflectance. The surface reflectance image may include or be composed of an albedo, including a diffuse albedo or a specular albedo. The surface reflectance image may be rendered by rasterization.

[0053] Guide passes can be rendered via ray casting or rasterization (in any combination).

[0054] Obtaining one or more guide passes (including obtaining a low-resolution guide pass and a full-resolution guide pass) optionally includes rendering by rasterization.

[0055] For example, a low-resolution guide pass may be rendered by rasterization, while a high-resolution guide pass may be rendered by ray casting or rasterization. Alternatively, a high-resolution guide pass may be rendered by rasterization, while a low-resolution guide pass may be rendered by ray casting or rasterization.

[0056] Optionally: the low-resolution guide pass may be obtained by rendering at low resolution by a first rasterization pass; and the full-resolution guide pass may be obtained by rendering at full resolution by a second rasterization pass.

[0057] That is, the low-resolution guide pass and the full-resolution guide pass can be rendered separately. Alternatively, the low-resolution guide pass can be generated from the full-resolution guide pass by downsampling. However, the inventors have found that it may be more efficient to render the guide pass twice at different resolutions, rather than rendering the guide pass once at full resolution and downsampling it. This is because the memory access bandwidth can be reduced by rendering the guide pass twice. Instead of writing / reading the guide pass to / from memory, the guide pass can be rendered at the desired resolution as required by the algorithm.

[0058] A single rasterization pass may have several outputs. Thus, multiple guide passes (and optionally all guide passes) may be generated by a single rasterization pass.

[0059] The low-resolution guided channel and the full-resolution guided channel may include any one or any combination of two or more of the following: depth information of objects in the 3D scene; information identifying the materials of objects in the 3D scene; surface reflections of objects in the 3D scene; shadows in the 3D scene; and surface normals of objects in the 3D scene.

[0060] The method may include: defining a first tile, defining corresponding first adjacent portions of a noisy image and one or more guide channels, each first adjacent portion including a first plurality of pixels; defining a second tile, defining corresponding second adjacent portions of the noisy image and one or more guide channels, each second adjacent portion including a second plurality of pixels; calculating a first outer product between each pixel in the one or more guide channels and itself; and calculating a second outer product between each pixel in the one or more guide channels and a corresponding pixel in the noisy image, wherein for a pixel in the first tile, the first outer product and the second outer product are calculated (i) before the second tile or (ii) in parallel with the second tile.

[0061] In the presence of multiple guidance channels, these channels can be considered to form a guidance image, and a first outer product can be calculated between each pixel in the guidance image and itself, while a second outer product can be calculated between each pixel in the guidance image and the corresponding pixel in the noisy image. Calculating the outer product of the first tile before the second tile means that the calculation of the first tile is completed before starting to calculate the outer product of the second tile. In this way, the tiles can be processed individually and continuously, for example by a single processor or a single core in hardware. Calculating them in parallel means calculating them individually at the same time. This allows parallel processing, especially on different processors or cores in hardware. The first tile and the second tile can be non-overlapping.

[0062] Organizing processing in this way can allow for greater data locality. This helps with memory bandwidth efficiency—data in a tile can be cached locally to the processor or core performing the computation, meaning fewer accesses to external memory may be required.

[0063] At least one of the noisy image, the one or more guiding channels, and the denoised image may be stored in a quantized low bit depth format.

[0064] Quantization can reduce the amount of data to be stored and, thereby, can reduce memory bandwidth requirements.Quantization converts data from a high bit depth format (eg, 32-bit floating point) to a low bit depth format (eg, 8-bit integer).

[0065] The method may also include: after rendering the noisy image, quantizing it in a quantized low-bit depth format using non-linear quantization, so that darker areas of the image are quantized with a relatively larger quantization level density and brighter areas of the image are quantized with a relatively smaller quantization level density; and storing the quantized low-bit depth format in a memory, wherein the method optionally also includes, before calculating the parameters of the model, retrieving the quantized low-bit depth values ​​from the memory and performing inverse quantization.

[0066] Here, the quantization step size in dark areas of the image is smaller than the quantization step size in bright areas of the image. This allows dark (e.g., dim) areas of a scene to be accurately represented. In one example of nonlinear quantization, quantization includes applying a square root function followed by uniform quantization of the output of the square root function.

[0067] Calculating parameters of the model may include: calculating a first outer product between each pixel in one or more guiding channels and itself; calculating a second outer product between each pixel in one or more guiding channels and a corresponding pixel in the noisy image; blurring the first outer product to calculate a first moment matrix for each local neighborhood; blurring the second outer product to calculate a second moment matrix for each local neighborhood; and calculating parameters of the model for each local neighborhood, including calculating an inverse matrix of the first moment matrix, and calculating the product of the inverse matrix and the second moment matrix.

[0068] Here, it should be understood that each pixel is represented by a row vector. Each pixel in one or more guiding channels is represented as a row vector x; each pixel in the noisy image is represented by a row vector y.

[0069] "Fuzzy" refers to spatial averaging, e.g., summing over a local neighborhood, optionally using a weighted summation, optionally where the center of the local neighborhood is given more weight in the summation than the periphery of the local neighborhood.

[0070] Optionally, calculating the parameters of the model includes, before calculating the inverse matrix, adding a regularization matrix to the first moment matrix. The regularization matrix may include a diagonal matrix. The regularization matrix can help avoid numerical instabilities in matrix inversion.

[0071] Optionally, blurring the first outer product may include calculating a first multiscale pyramid from the first outer product and calculating a first moment matrix based on the first multiscale pyramid; and / or blurring the second outer product may include calculating a second multiscale pyramid from the second outer product and calculating a second moment matrix based on the second multiscale pyramid.

[0072] The multiscale pyramid has multiple levels, where successive levels describe the outer product at successively different levels of detail. For example, the multiscale pyramid may include or consist of a mipmap pyramid. Mipmaps are suitable for efficient implementation in fixed-function hardware, such as a graphics processing unit (GPU).

[0073] The fuzzy outer product (i.e., the moment matrix) can be directly calculated from a predetermined level of the pyramid. In this case, the calculation of the pyramid may stop at this level. In other examples, the moment matrix can be calculated by interpolation using the pyramid. The interpolation can include bilinear or trilinear interpolation, or other sampling, such as bicubic sampling.

[0074] Blurring may include separable filtering in the horizontal and vertical directions. Filtering may use a center-weighted filter function, such as a Gaussian function. Optionally, separable filtering may be applied to a predetermined level of a multi-scale pyramid. This may help to efficiently implement center-weighted filtering with reduced computational complexity (compared to filtering the outer product directly using a center-weighted filter). This type of blurring may be applied to one or both of the first outer product and the second outer product.

[0075] In some examples, blurring can include filtering using an anisotropic 2D filter. By anisotropic, it is meant that the filter has a major axis and a minor axis perpendicular to the major axis and extends further along the major axis than the minor axis. The axis can be aligned with the horizontal and vertical directions, or the axis can be independent of the horizontal and vertical directions.

[0076] The one or more guiding passes may include surface normals of objects in the 3D scene, and the blurring may include: for each local neighborhood, determining a major axis and a minor axis of a 2D filter based on the surface normal of the object at the center of the neighborhood; selecting a level of a multi-scale pyramid based on the length of the minor axis; and sampling the selected level of the multi-scale pyramid along the major axis.

[0077] This may provide a computationally efficient way to adapt the blur to the scene content, in particular by adjusting the blur according to the orientation of the surface being sampled.

[0078] In some examples, blurring may include IIR filtering.

[0079] In some examples, blurring can include filtering with a running box filter.

[0080] The method may include: defining a first outer product tile, defining a first adjacent part of the first outer product and a corresponding first adjacent part of the second outer product, each first adjacent part including a first plurality of pixels; defining a second outer product tile, defining a second adjacent part of the first outer product and a corresponding second adjacent part of the second outer product, each second adjacent part including a second plurality of pixels, wherein for the first tile, the first moment matrix and the second moment matrix are calculated (i) before the second tile or (ii) in parallel with the second tile.

[0081] The first outer product tile and the second outer product tile may be overlapping tiles. This is because, generally speaking, the blur of the outer product will not be divided into non-overlapping tiles. The degree of overlap may depend on the size of the filter used for blurring.

[0082] The method may also include normalizing and / or regularizing one or both of the first moment matrix and the second moment matrix.

[0083] This can improve the stability of the matrix inversion. In this regard, normalization can complement the use of a regularization matrix. Normalization can include normalizing the moments using the mean and standard deviation information in the moment matrix before the inverse calculation. This can improve the numerical stability of the matrix inversion, thereby improving the visual quality. In some examples, one or both matrices can be normalized to have a zero mean and a constant standard deviation (such as a standard deviation equal to one).

[0084] There is also provided a graphics processing unit configured to render an image of a 3D scene, the graphics processing unit comprising:

[0085] a first rendering block, the first rendering block being configured to render a noisy image;

[0086] a second rendering block, the second rendering block being configured to render one or more guide channels;

[0087] a model fitting block configured to compute, for each of the plurality of local neighborhoods, parameters of a model that approximates the noisy image as a function of the one or more guiding channels; and

[0088] a model application block configured to apply the calculated parameters to the one or more guidance channels for each of the plurality of local neighborhoods to produce a denoised image,

[0089] The one or more guiding channels include at least one guiding channel characterizing a spatial dependency of incident light on global illumination on surfaces of one or more 3D models in the scene.

[0090] There is also provided a graphics processing unit configured to render an image of a 3D scene, the graphics processing unit comprising:

[0091] A first rendering block, the first rendering block being configured to render a noisy low-resolution image;

[0092] a second rendering block configured to render one or more low-resolution guide passes;

[0093] a third rendering block configured to render one or more full-resolution guide passes, each full-resolution guide pass corresponding to a corresponding low-resolution guide pass;

[0094] a model fitting block configured to compute, for each of a plurality of local neighborhoods, parameters of a model that approximates the noisy low-resolution image as a function of the one or more low-resolution guiding channels; and

[0095] a model application block configured to apply the calculated parameters to one or more full-resolution guidance channels for each of the plurality of local neighborhoods to produce a full-resolution denoised image,

[0096] Wherein, the one or more low-resolution guide channels and the one or more corresponding full-resolution guide channels each include at least one guide channel, and the at least one guide channel characterizes the spatial dependence of incident light on the global illumination on the surface of one or more 3D models in the scene.

[0097] The second rendering tile and the third rendering tile may be separate, or may be provided by a single rendering tile.

[0098] The first rendering tile may be configured to render a noisy (optionally, low resolution) image by path tracing.

[0099] Optionally, to render the at least one guide channel, the second rendering block is configured to: obtain one or more ambient occlusion maps of the corresponding one or more models in the scene; and project the one or more ambient occlusion maps into screen space to generate ambient occlusion guides in the screen space.

[0100] Optionally, the second rendering block may be configured to render one or more of the one or more (optionally, low resolution) guide passes by rasterization or ray tracing. Optionally, the third rendering block may be configured to render one or more full resolution guide passes by rasterization or ray tracing.

[0101] The one or more guide passes rendered by rasterization or ray tracing may be in addition to the guide pass(s) characterizing the spatial dependency of the incident light on the global illumination, in particular in addition to the ambient occlusion guide. Thus, overall, there are at least two guide passes (at each respective resolution).

[0102] The noisy (optionally, low resolution) image may include indirect lighting in the scene, wherein the (optionally, full resolution) denoised image includes the indirect lighting image, and the graphics processing unit may further include: a fourth rendering block configured to render a directly lit image; and a combination block configured to combine the denoised image with the directly lit image to produce a global illumination image. The fourth rendering block may be configured to render the directly lit image by ray tracing.

[0103] The noisy image may be a noisy global illumination image, including direct and indirect illumination in the scene, whereby the denoised image is a denoised global illumination image.

[0104] The third rendering block may be configured to render a surface reflection image. The graphics processing unit may further include a product block configured to calculate a product of a global illumination image or a denoised global illumination image and a surface reflection image, thereby generating a rendered image of the 3D scene. In the case of using different resolutions, the surface reflection image may be rendered at full resolution, in particular by rasterization.

[0105] There is also provided a graphics processing system configured to perform the method as outlined above and / or comprising a graphics processing unit as outlined above.The graphics processing system may be embodied in hardware on an integrated circuit.

[0106] There is also provided a method of manufacturing a graphics processing system as outlined above using an integrated circuit manufacturing system.

[0107] A method of manufacturing a graphics processing system as outlined above using an integrated circuit manufacturing system is also provided, the method comprising: processing a computer-readable description of the graphics processing system using a layout processing system to generate a circuit layout description of an integrated circuit including the graphics processing system; and manufacturing the graphics processing system based on the circuit layout description using the integrated circuit generation system.

[0108] Further provided are computer readable code configured to cause the method as outlined above to be performed when the code is executed; and optionally a computer readable storage medium having the computer readable code encoded thereon.

[0109] An integrated circuit definition data set is also provided which, when processed in an integrated circuit manufacturing system, configures the integrated circuit manufacturing system to manufacture a graphics processing system as outlined above.

[0110] Still further provided is a computer-readable storage medium (optionally non-transitory) having stored thereon a computer-readable description of a graphics processing system as outlined above, which, when processed in an integrated circuit manufacturing system, causes the integrated circuit manufacturing system to manufacture an integrated circuit containing the graphics processing system.

[0111] Also provided is a computer-readable storage medium (optionally non-transitory) having stored thereon a computer-readable description of a graphics processing system as outlined above, which, when processed in an integrated circuit manufacturing system, causes the integrated circuit manufacturing system to perform the following operations: processing the computer-readable description of the graphics processing system using a layout processing system to generate a circuit layout description of an integrated circuit including the graphics processing system; and manufacturing the graphics processing system based on the circuit layout description using the integrated circuit generation system.

[0112] There is also provided an integrated circuit manufacturing system configured to manufacture a graphics processing system as outlined above.

[0113] An integrated circuit manufacturing system is also provided, comprising: a computer-readable storage medium (optionally non-transitory) having stored thereon a computer-readable description of a graphics processing system as outlined above; a layout processing system configured to process the computer-readable description to generate a circuit layout description of an integrated circuit including the graphics processing system; and an integrated circuit generation system configured to manufacture the graphics processing system based on the circuit layout description.

[0114] The layout processing system may be configured to determine location information of logic components of the circuit derived from the integrated circuit description to generate a circuit layout description of the integrated circuit incorporating the graphics processing system.

[0115] A method for selecting one or more guiding channels for image denoising is also provided, the method comprising:

[0116] Obtain multiple noisy training images;

[0117] Obtaining a corresponding plurality of real images, wherein each real image depicts the same 3D scene as the corresponding noisy training image, and each real image has less noise than the corresponding noisy training image;

[0118] Obtain multiple candidate guide channels;

[0119] Conduct multiple assessments, each including:

[0120] Selecting one or more candidate guiding channels to be evaluated from the candidate guiding channels;

[0121] Denoising each of the noisy training images using the selected candidate guiding channel to generate a corresponding denoised training image; and

[0122] comparing each denoised training image to the corresponding true image to determine the error therebetween; and

[0123] One or more guiding channels for image denoising are selected based on the errors determined in the plurality of evaluations.

[0124] The candidate guide channels may include any one or any combination of two or more of the following: depth information of objects in the 3D scene; information identifying the material of objects in the 3D scene; surface reflections of objects in the 3D scene; shadows in the 3D scene; and surface normals of objects in the 3D scene.

[0125] Denoising optionally includes, for each noisy training image: for each of a plurality of local neighborhoods of the noisy training image: calculating parameters of a model that approximates the noisy training image as a function of one or more guiding channels, and applying the calculated parameters to the one or more guiding channels to produce a corresponding denoised training image.

[0126] The method of approximating a noisy training image by such a function of one or more guiding channels corresponds to denoising by guided filtering. The function may include or consist of a linear combination of guiding channels and scalar offsets. Thus, the model may include or may be an affine model.

[0127] In some examples, a scalar offset can be incorporated into the model by including a guide channel with a uniform value (e.g., an array of ones) at each pixel. Optionally, the guide channel is not stored in memory—it can be instantiated on the fly in software or hardware during fitting of the model (e.g., by a model fitting unit).

[0128] Each local neighborhood may be a local neighborhood centered at a corresponding pixel of the noisy training image, thereby computing a different set of model parameters for each pixel.

[0129] The plurality of candidate guiding channels optionally includes at least one guiding channel characterizing a spatial dependency of incident light on global illumination on surfaces of one or more 3D models in the scene.

[0130] It should be noted that the use of at least one guiding channel (e.g., ambient occlusion guides, as defined below) that characterizes the spatial dependence of incident light on global illumination is contrary to the way in which ambient occlusion information might be conventionally expected to be used. A more conventional approach might include ambient occlusion in a noisy image. Alternatively, ambient occlusion data can be combined with a denoised image at the end of a rendering pipeline. It will be noted that in examples of the present method, the noisy image does not include ambient occlusion information. And the denoised image is not combined with any ambient occlusion information. Ambient occlusion information is only introduced into the pipeline by the at least one guiding channel.

[0131] At least one guide channel optionally includes or consists of ambient occlusion data. In some examples, the ambient occlusion data is contained in a single guide channel. In some examples, the at least one guide channel includes a plurality of guide channels, each guide channel characterizing directional illumination in a different direction.

[0132] Obtaining the at least one guiding pass may include: obtaining pre-computed texture data containing information about shadows; and projecting the pre-computed texture data into screen space to generate the at least one guiding pass.

[0133] Precomputed texture data may be provided in texture space. During the rendering process, the precomputed texture data is projected into screen space (or rendered in screen space).

[0134] Obtaining the at least one guide channel optionally includes: obtaining one or more ambient occlusion maps of corresponding one or more models in the scene; and projecting the one or more ambient occlusion maps into screen space to generate ambient occlusion guides in the screen space.

[0135] That is, the ambient shielding guide forms the at least one guide channel.

[0136] The ambient occlusion map for each model can be pre-computed in advance, for example when designing the model. Each ambient occlusion map can be represented in texture space. During the selection of the guide pass, each individual ambient occlusion map is retrieved and projected into screen space (or rendered in that screen space). This is a relatively computationally inexpensive process.

[0137] The projected ambient occlusion maps may be combined into a single ambient occlusion guide. For example, combining the projected ambient occlusion maps may combine and sum them.

[0138] In some examples, the guide channels can contain information about scene structure in the 3D scene, including but not limited to object boundaries, occlusion boundaries, and shadow edges. The guide channels can be substantially noise-free. They can be rendered by deterministic calculations (e.g., by rasterization), while noisy training images can be rendered by random sampling.

[0139] Obtaining each noisy training image and corresponding real image optionally includes rendering by path tracing. The guidance pass can be rendered by ray casting or rasterization (in any combination).

[0140] Each evaluation optionally includes selecting one individual guiding channel to be evaluated, and wherein selecting one or more guiding channels for image denoising includes selecting in a greedy manner the one or more guiding channels associated with a lowest determination error among all individual evaluations.

[0141] Although the set of guiding channels selected in this way may not necessarily be globally optimal, a greedy selection strategy like this can help reduce the number of evaluations required (since the number of evaluations to be performed is equal to the number of candidate guiding channels).

[0142] The selection of one or more guiding channels is optionally performed iteratively, including selecting one or more first candidate guiding channels associated with the lowest determination error in one iteration, and wherein, in a subsequent iteration, each evaluation includes selecting the one or more first candidate guiding channels that have been selected, and also selecting one or more additional candidate guiding channels to be evaluated, and wherein the subsequent iteration includes selecting one or more additional candidate guiding channels associated with the lowest determination error among the one or more additional candidate guiding channels in each evaluation.

[0143] This approach defines an iterative or incremental greedy algorithm - at each stage (i.e., iteration) one or more additional guiding channels are selected that provide the lowest error in combination with the previously selected guiding channel. This may be computationally more demanding than just individually evaluating a single guiding channel; however, when used in certain combinations, it may produce more representative guiding channel performance results. This may help it identify, for example, guiding channels that complement each other.

[0144] Each evaluation optionally includes selecting a subset of candidate guiding channels to be evaluated together, and wherein selecting one or more guiding channels includes selecting the subset of candidate guiding channels associated with a lowest determination error.

[0145] Each subset includes a different combination of two or more of the candidate guiding channels.

[0146] There is also provided a computer readable code configured to cause the method as outlined above to be performed when the code is executed.

[0147] A method of constructing one or more guiding channels for image denoising is also provided, the method comprising:

[0148] Obtain multiple noisy training images;

[0149] Obtaining a corresponding plurality of real images, wherein each real image depicts the same 3D scene as the corresponding noisy training image, and each real image has less noise than the corresponding noisy training image;

[0150] The one or more guiding channels are constructed based on an optimization process, wherein the optimization process constructs the one or more guiding channels so as to minimize an error between each real image and a corresponding denoised training image, wherein the denoised training image is generated by denoising the corresponding noisy training image using the one or more guiding channels.

[0151] A plurality of noisy training images may be provided for a given object or scene, each noisy training image being associated with a corresponding real image. The plurality of noisy training images may include noisy training images containing the same object or scene under different illumination conditions.

[0152] In some examples, the optimization process can be performed directly on one or more guide channels. That is, one or more guide channels can be the direct product of the optimization process. Alternatively, in some examples, the optimization process can be performed on a precursor of one or more guide channels. That is, the direct product of the optimization process can be the precursor of one or more guide channels. The precursor can be further processed to construct the guide channel.

[0153] The optimization process can be iterative. The iterative optimization process optionally includes, for each iteration of a plurality of iterations: denoising the noisy training image using one or more guiding channels to produce a denoised training image; comparing the denoised training image with a corresponding true image to determine an error therebetween; and updating the one or more guiding channels to reduce the error.

[0154] The iterative optimization process can be based on a gradient descent algorithm.

[0155] The iterative optimization process optionally includes computing derivatives of the denoised training image with respect to one or more guidance channels.

[0156] The iterative optimization process may include updating one or more guide channels or their precursors based on the calculated derivatives in order to reduce the error.

[0157] One or more guide channels are optionally constructed in texture space. In this case, optimization can be performed on the precursors of the guide channels in texture space.

[0158] Prior to denoising the noisy training images, one or more guidance channels are optionally projected from texture space to screen space.

[0159] Denoising optionally includes, for each noisy training image: for each of a plurality of local neighborhoods of the noisy training image: calculating parameters of a model that approximates the noisy training image as a function of one or more guiding channels, and applying the calculated parameters to the one or more guiding channels to produce a corresponding denoised training image.

[0160] The method of approximating a noisy training image by such a function of one or more guiding channels corresponds to denoising by guided filtering. The function may include or consist of a linear combination of guiding channels and scalar offsets. Thus, the model may include or may be an affine model.

[0161] In some examples, a scalar offset can be incorporated into the model by including a guide channel with a uniform value (e.g., an array of ones) at each pixel. Optionally, the guide channel is not stored in memory—it can be instantiated on the fly in software or hardware during fitting of the model (e.g., by a model fitting unit).

[0162] Each local neighborhood may be a local neighborhood centered around a corresponding pixel of the noisy training image, thereby computing a different set of model parameters for each pixel.

[0163] Obtaining each noisy training image and corresponding real image optionally includes rendering by path tracing.

[0164] Also provided is a computer readable code configured to cause the method as outlined above to be performed when the code is executed.

[0165] As will be apparent to the skilled person, the above features may be combined as appropriate and may be combined with any aspect of the examples described herein. BRIEF DESCRIPTION OF THE DRAWINGS

[0166] Examples will now be described in detail with reference to the accompanying drawings, in which:

[0167] Figure 1 A flowchart illustrating a rendering method according to an example is shown;

[0168] Figure 2 is a block diagram of a graphics processing unit for rendering according to an example;

[0169] Figure 3 Shown in more detail Figure 1 The model fitting step;

[0170] Figure 4 Shows Figure 1 A variation of a portion of the illustrated method that uses tile-based processing;

[0171] Figure 5 shows the use of tile-based processing Figure 1 Another variation of a portion of the method shown;

[0172] Fig. 6A Show Figure 3 Variations of the model fitting method shown;

[0173] Figure 6B and Figure 6C shows the creation of a mipmap pyramid according to an example;

[0174] Figure 7 Shows Fig. 6A Example of the fuzzy step in ;

[0175] Fig. 8A The basic principle of anisotropic blur is shown;

[0176] Figure 8B Shows Fig. 6A Another example of the fuzzy step in ;

[0177] Fig. 9 Quantization and inverse quantization of data according to an example are shown;

[0178] Fig.10 is a diagram showing a method according to another example Figure 1 Flowchart of alternatives to the method;

[0179] Fig.11 is a diagram showing a method according to another example Figure 1 A flowchart of another alternative method of;

[0180] Fig.12 is a flow chart showing one way of obtaining an environmental guidance channel according to an example;

[0181] Fig.13 A method of selecting a guide channel for a denoising or rendering method according to an example is shown;

[0182] Fig.14 A graphics processing system is shown in which a graphics processing unit is implemented; and

[0183] Fig.15 An integrated circuit manufacturing system for producing an integrated circuit including a graphics processing system is shown.

[0184] The accompanying drawings show various examples. The skilled person will appreciate that the element boundaries (e.g., boxes, groups of boxes, or other shapes) shown in the accompanying drawings represent an example of boundaries. In some examples, it may be the case that an element can be designed as multiple elements, or multiple elements can be designed as one element. Where appropriate, common reference numerals are used throughout the accompanying drawings to indicate similar features. DETAILED DESCRIPTION

[0185] The following description is presented by way of example to enable one skilled in the art to make and use the invention.The present invention is not limited to the embodiments described herein, and various modifications to the disclosed embodiments will be apparent to those skilled in the art.

[0186] Embodiments will now be described by way of example only.

[0187] Various techniques for rendering are known in the field of computer graphics. "Rendering", also known as image synthesis, refers to the process of generating an image of a virtual 3D scene as if a virtual camera were looking at it at a specific position and orientation. The geometry of the objects in the scene is modeled in three dimensions, and the rendering process approximates / simulates how light interacts from a light source with the modeled geometry to reach the virtual camera.

[0188] Path tracing, ray tracing, and rasterization are three rendering techniques. The terms "ray tracing" and "path tracing" are sometimes used interchangeably in the literature. For greater clarity, in this context, we will first define what we mean by these terms.

[0189] As used herein, the term "path tracing" refers to a random sampling method that simulates the transport of light in a 3D scene. In some examples according to the present disclosure, path tracing is able to approximate global illumination by modeling all types of interactions between light and objects in the scene, potentially including higher-order reflections (second bounces and third bounces, etc.). However, due to its random nature, path tracing is inherently noisy. Note that path tracing is not limited to approximating global illumination. "Ray traced ambient occlusion" is a type of path tracing that calculates the degree to which each point in a 3D scene is exposed to ambient lighting (rather than calculating global illumination as such). Ray traced ambient occlusion can be considered a rough approximation of indirect illumination.

[0190] As used herein, the term "ray tracing" refers to a deterministic, noise-free method for simulating light transport in a 3D scene by modeling certain interactions between light rays and objects. Ray tracing typically only determines direct light, reflections, refractions, etc., and does not rely on complex light paths that interact with multiple objects.

[0191] The term "ray casting" refers to a specific kind of ray tracing in which rays are cast along a single direct path, without reflection or refraction.

[0192] Rendering via path tracing and ray tracing differs from rendering via "rasterization," in which 3D scene geometry is mapped to pixels in 2D screen space without regard for the physical transport of light in the 3D scene.

[0193] Rendering pipeline

[0194] Now refer to Figure 1 and Figure 2 A first example of rendering an image is described. Figure 1 is a flowchart illustrating a method of rendering an image of a 3D scene according to an example. Figure 2 is shown as being configured to execute Figure 1A block diagram of functional blocks of a graphics processing unit of the method. A graphics processing unit (GPU) includes a first rendering block, a second rendering block, a third rendering block, and a fourth rendering block. In this example, the first rendering block is a path tracing block 210. The second rendering block and the third rendering block are both provided by a rendering / rasterization block 240. The fourth rendering block is a ray tracing block 260. The GPU also includes a model fitting block 220, an upsampler 230, a model combination block 250, a summation block 270, and a product block 280. The upsampler 230 and the model combination block 250 together constitute a model application block, which is responsible for applying the model. The summation block 270 operates as a combination block to combine the denoised image generated by the application of the model with the image generated by the fourth rendering block 260.

[0195] In step 110, the path tracing block 210 generates a low-resolution indirect illumination image by path tracing. In step 112, the rendering / rasterization block 240 generates a low-resolution guide channel in the first rasterization round. The indirect illumination image and the guide channel correspond to each other because they are all rendered for the same viewpoint in the 3D scene. They are also rendered at the same (low) resolution. Due to the random nature of path tracing, the indirect illumination image is noisy. Because rasterization is deterministic, the guide channel is noise-free. In this example, the low-resolution guide channel set includes a depth image, a surface reflection image, and a surface normal image. However, the low-resolution guide channel set does not need to include all of these images (for example, the surface reflection image may not be included). Optionally, other guide channels may be included, such as images that identify the material of objects in the scene and / or images that show shadows in the scene.

[0196] In step 113, an ambient occlusion guide is obtained. In this example, this is used by the rendering / rasterization block 240, which will be referred to later. Fig.12 The process described herein is produced. An ambient occlusion guide is a guiding channel that contains ambient occlusion data. It therefore characterizes the spatial dependence of the incident light on the global illumination on the surface of (partial or full) 3D models in the scene. The inventors have found that, surprisingly, the use of such ambient occlusion data in a guiding channel can significantly improve the quality of the denoised image produced according to the method. Without wishing to be bound by theory, it is believed that this is because the ambient occlusion data does not contain information about self-shielding as well as the structure of objects in the scene. Therefore, it can complement purely structural guiding channels (e.g., based on depth or surface orientation) because such guiding channels do not contain any shadow information. When none of the available guiding channels includes shadow edges, there is a risk that the denoising process will inappropriately blur such edges, thereby reducing image quality. The method of this example can help solve this problem.

[0197] In this example, two ambient occlusion guides are generated in step 113 , namely a low-resolution version used as input to the model fitting in step 120 and a full-resolution version used as input to the combining step 150 .

[0198] The low-resolution images are typically a fraction of the size of the desired full-resolution image. For example, they may be one-half smaller in each dimension (i.e., in height and width) so that they are one-quarter the area and resolution of the full-resolution image. In some examples, they may still be a smaller fraction of the full-resolution image. For example, the low-resolution images may be one-quarter smaller in each dimension so that they are one-sixteenth the area and resolution of the full-resolution image.

[0199] The rendering method will produce a full-resolution rendered image based in part on a guidance image (including multiple guidance channels, including ambient occlusion guides) and a noisy indirect illumination image. The low-resolution image (i.e., the guidance image and the indirect illumination image) is passed to a model fitting block 220. In step 120, the model fitting block 220 fits the parameters of an affine model for each pixel of the low-resolution image so that it can be used to map the low-resolution guidance image onto the low-resolution indirect illumination image. For each pixel, the parameter estimation examines a local neighborhood centered on the pixel and approximates the noisy indirect illumination image within this local neighborhood as a linear combination of guidance channels and a scalar offset. This produces a set of model parameters for each of the local neighborhoods (i.e., for each pixel in the pixel).

[0200] In step 130, the upsampler 230 upsamples the model parameters to full resolution. In step 140, the rendering / rasterization block 240 renders a full-resolution guide channel set in a second rasterization pass. The full-resolution guide channels correspond one-to-one to the low-resolution guide channels. In fact, in some examples, the low-resolution guide channels can be generated by subsampling the full-resolution guide channels. However, the inventors have found that by rendering the low-resolution guide channels and the full-resolution guide channels separately in two rasterization passes, memory access bandwidth can be saved. This can avoid the need to store a full-resolution multi-channel guide image and then retrieve it from memory for downsampling. In some hardware architectures and applications, memory bandwidth may be a limiting factor; therefore, reducing memory bandwidth is very desirable. Another advantage of using a low-resolution guide channel is to reduce the number of operations required to generate a (low-resolution) moment image, because fewer matrices will need to be summed when generating the moment image.

[0201] The full-resolution guide pass (including the full-resolution ambient occlusion guide) is provided to the model combination block 250. As described above, the guide pass in this example includes a surface reflection image. The full-resolution surface reflection image is additionally provided to the product block 280. In other examples where the surface reflection image is not used as a guide pass, the surface reflection image is still generated (i.e., as well as the guide pass) so that it can be provided to the product block 280.

[0202] In step 150, the model combination block combines the upsampled model parameters from the upsampler 230 with the full-resolution guidance channels produced by the rendering / rasterization block 240 in steps 113 and 140. This involves generating each pixel as a linear combination of the full-resolution guidance channels, where the linear combination is determined by the model parameters for that pixel. In this example, since the image to be denoised is a noisy indirect illumination image, the output of the model combination block 250 is a full-resolution denoised indirect illumination image.

[0203] In step 160, the ray tracing block 260 renders the direct lighting image by ray tracing. As explained above, ray tracing is deterministic. As such, ray tracing is inherently noise-free and less computationally intensive than rendering the indirect lighting image by path tracing 110. As a result of this lower complexity, rendering the direct lighting image at full resolution does not incur a large computational burden.

[0204] In step 170, summation block 270 combines the denoised indirect illumination image provided by model combination block 250 and the direct illumination image provided by ray tracing block 260 by summing them. The result of the summation is a global illumination image at full resolution. In step 180, product block 280 multiplies the full resolution surface reflection image with the full resolution global illumination image (pixel by pixel). This produces the final rendered image of the scene under global illumination conditions.

[0205] The affine model created in step 120 is based on the guided filtering method of He et al. (Kaiming He, Jian Sun, and Xiaoou Tang, "Guided Image Filtering", Daniilidis K., Maragos P., Paragios N. (eds.) Computer Vision–ECCV 2010. Lecture Notes in Computer Science, Vol. 6311, Springer. https: / / doi.org / 10.1007 / 978-3-642-15549-9_1). For each local neighborhood Y of the noisy input image, the method attempts to find an affine transformation A that best maps the local neighborhood X of the guided image to the corresponding local neighborhood Y of the noisy input image. These local neighborhoods can be represented as matrices, and the "best" transformation is defined as the least squares best fit. That is, find a solution A such that the matrix equation Minimize, where and And N is the number of pixels in the local neighborhood, G is the number of guide channels (including the offset term), and M is the number of output channels. It should be understood that the labeling of the dimensions is essentially arbitrary.

[0206] As in this example, there may be multiple guide channels in the guide image. In this case, the model A is an affine transform with multiple input signals. According to this formulation (and unlike the formulation of He et al.), the offset term is directly included in the matrix multiplication by adding an additional guide channel consisting of an array of ones with corresponding values ​​in the A matrix. This additional guide channel of ones is constant and therefore can be implicit in block 220 so that bandwidth is not wasted in transmitting it. Thus, the matrix X representing the guide image channels is constructed as follows

[0207]

[0208] Here, each row represents pixels in a local neighborhood centered on the pixel of interest. Each column represents one channel of the entire multi-channel guidance image. The least squares optimization problem is modified to include regularization, which yields smoother results and reduces sensitivity to numerical instabilities:

[0209]

[0210] Here, N is a scalar introduced for consistency with the formula of He et al. U is a diagonal matrix constructed so that the entries corresponding to the channels of ones in X are zero:

[0211] U=diag(0,∈ 0,∈ 1 ,…).

[0212] The number is ∈ positive. A suitable exemplary value is about 0.0001. Minimizing E with respect to A has a closed form solution:

[0213] A=(X T X+NU) -1 X T Y.

[0214] The denoised pixel is given by the vector-matrix product xA, where x is the structure vector of the pixel (a row of the X matrix) and A is the model built from the neighborhood of the target pixel. The denoised output is obtained by multiplying the guidance image with the corresponding local model parameters A at each pixel. Figure 3 As shown, the calculation of the model parameter A can be divided into several stages. The first stage involves calculating the term X T X and X T Y, which are sometimes referred to as "moment matrices" or simply "moments". This, in turn, can be divided into two stages. First, a per-pixel outer product calculation is performed. Second, the outer products are summed ("blurred") over a local neighborhood to produce a moment matrix. Although the equation above (which explains how the matrix A is calculated for a given local neighborhood) does not indicate this, it can be shown that spatially blurring the outer product terms after they have been calculated is equivalent to explicitly constructing the X and Y matrices and performing the matrix multiplications required to produce the moment matrix. The calculation of the outer product by the model fitting block 220 is given by Figure 3 We refer to the output of step 122 as the first outer product x. T x and the second outer product x T y. In step 124, the model fitting block 220 fuzzifies the outer product to produce a moment matrix. We refer to the output of the fuzzification step as the first moment matrix X T X and the second moment matrix X T Y. Blurring can be performed in a variety of different ways, and will be described in further detail below.

[0215] After blurring the moments in step 124, the model fitting block 220 adds regularization and solves a least squares problem. This involves inverting the regularized first moment matrix and then multiplying the result of the matrix inversion by the second moment matrix, as indicated by the equation above. The output of step 126 is the model matrix for each pixel of the low-resolution image. In the method of He et al., the model parameters are further blurred spatially. However, the inventors have found that this is unnecessary and actually results in lower image quality in many cases. Therefore, the method proceeds without blurring the model parameters.

[0216] As explained above, the model parameters are upsampled by upsampler 230 in step 130. This converts them to full resolution. They are then combined (in step 150) with the full resolution guidance image produced by rendering / rasterization unit 240. As previously discussed, this operation implements the formula Y=XA.

[0217] Tiling

[0218] Figure 4 and Figure 5 Some optional additional steps that can be introduced into model fitting 120 are shown in . The calculations involved in model fitting are somewhat computationally intensive and data intensive. The inventors have recognized that it may be advantageous to perform model fitting in a tile-based manner. This approach involves decomposing the data into tiles (generally speaking, they may be overlapping or non-overlapping) and processing each tile separately. This has several potential advantages. First, the size of the tile can be selected so that all the data required to perform the calculation of one tile can be stored in the on-chip memory in the GPU. This can avoid the need to read data from external memory and write data to external memory, which is typically a slower operation than accessing on-chip memory. Secondly, when multiple processors or cores are available in the GPU, the ability to divide processing into tiles facilitates parallel implementation. However, it should be noted that tiling is useful independently of parallelization due to the first advantage. Even if tiles are processed continuously by a single core or a single processor, the effects of greater data locality and reduced memory access bandwidth can still promote increased processing speed and / or reduced power consumption.

[0219] Tiling can be applied in at least two ways. Figure 4 As shown, tiling can be used in the calculation of the outer product. In step 115, the GPU divides the low-resolution noisy indirect illumination image and the guidance image into tiles. Then, in step 122a, as a variation of step 122, the outer product of each tile is calculated independently of the other tiles. The tiles created in step 115 are non-overlapping because the data required for the outer product calculation is different from each tile to the next tile. This is because the calculation of the outer product is inherently a pixel-by-pixel operation.

[0220] Alternatively or additionally, tiling can be used in the blur, which computes the moment matrix from the outer product. Figure 5 . The outer product generated in step 122 (or step 122a) is divided into overlapping tiles in step 125. In step 124a, blurring is applied to each tile independently. It should be noted that due to the spatial extent of the local neighborhoods used to create the model, the tiles used for blurring must overlap to some extent. These local neighborhoods are reflected in the size of the filter kernel used for blurring.

[0221] Vague

[0222] Blurring (step 124 or step 124a) can be performed in a variety of ways. The purpose of blurring is to transform the outer product x over a local neighborhood centered on the pixel of interest. T x and x T y are summed to generate the moment matrix X T X and X T Y. Optionally, the sum can be a weighted sum, typically emphasizing the contribution of pixels in the center of each neighborhood and de-emphasizing the contribution of pixels in the periphery of the neighborhood. There are a number of ways to do this efficiently. In one example, the computation of the moment matrix is ​​accelerated by reusing the computation between overlapping regions and utilizing fast image filtering techniques. Let each pixel in X be represented by a row vector x, and each pixel in Y be represented by a row vector y. For each output position, compute x T x and x T y, i.e. the outer product. Considering the entire image, this gives a 4D tensor with exemplary dimensions [H,W,N,N] and [H,W,N,M], where H and W are the image height and width, and N and M are the number of channels in X and Y, respectively. A 2D box filter is convolved over the first two dimensions of these tensors. For efficiency, this box filter can be implemented using integral images or separable horizontal and vertical kernels. This can be shown to be equivalent to constructing the moment matrix:

[0223]

[0224] Here, {X} i,j is the element in the i-th row and j-th column of the matrix X. p,: is the row vector of the pth row of X. :,i is the column vector of the i-th column of X. In this way, the moment matrix X T X can be written as the sum of the outer products of the rows of X. Constructing it this way is efficient due to the overlap between local neighborhoods and the fact that linear filters can be computed cheaply. A similar formula applies to the second moment matrix X T Y.

[0225] The above filter implements a simple box filter. This can be accelerated in a number of ways. One option is to use an integral image (also called a summed area table) to precompute the sum of the outer products up to and including each pixel in the low-resolution image. The sum of the outer products in any desired area can then be derived from the integral image based on the integral values ​​at the four corners of the rectangle.

[0226] Another option is to use a so-called "running box filter". This works on the following principle: by adding the values ​​that have entered the window and subtracting the values ​​that have fallen out of the window, the continuous sum of the outer product can be calculated incrementally for each shift of the window that defines the local neighborhood. This can be better understood by referring to a simple one-dimensional example. Suppose it is desired to calculate a three-pixel box filter along a row of pixels a, b, c, d, e. The first sum is a+b+c. The second sum is b+c+d. This can be calculated by taking the previous value (a+b+c), adding d (it enters the window) and subtracting a (it falls out of the window). This results in the calculation (a+b+c)+da=b+c+d. The process is repeated in the next step: (b+c+d)+eb=c+d+e. It should be noted that although this does not change the number of calculations at each shift of the window, for this simplified three-pixel example, it has considerable benefits for larger box filters. Regardless of the filter size, at each shift, the required sum can be computed with a single subtraction and a single addition. The possibility of separable 1-D horizontal and vertical box filtering discussed above can be combined with the use of a running box filter. That is, the running box filter can be implemented separately in two dimensions.

[0227] Although the box filter is computationally efficient, it gives equal weight to all pixels in a local neighborhood. Better results can be obtained by giving more influence to pixels near the center of the neighborhood than to pixels at the edge. The challenge is to find efficient ways to implement this more complex kind of filtering in a way that achieves a good balance between computational efficiency and image quality.

[0228] One approach that offers good potential to reconcile these competing demands is to implement blur using mipmaps. Such mipmaps are known in the computer graphics literature, where they have been applied in the context of texture sampling. It uses a scaled space pyramid where each level of the pyramid (sometimes referred to in the art as a "chain") is generated by downsampling the previous level. In some examples, each level of the pyramid is generated from the previous level through a 2×2 box filter. This can be computed recursively by bilinearly resampling the previous level. The inventors have recognized that a mipmap pyramid can be computed for each channel of the outer product, and these mipmaps can then be used to support an efficient implementation of blur. Sometimes, it is not necessary to compute all levels of the mipmap pyramid (as discussed in more detail below). This can save time, power, memory, and bandwidth.

[0229] Fig. 6A Shows Figure 3A variation of the method of incorporating the use of mipmaps. In step 122, the model fitting block 220 computes the outer product (as described above); and in step 322, the model fitting block creates a mipmap from the outer product as part of the blurring process. This process is Figure 6B . At the bottom level of the pyramid is the original outer product tensor 10 (shown as a 2D array for simplicity, although it is actually a higher dimensional tensor). This is spatially (e.g., bilinearly) downsampled to produce the second level of the pyramid 12. The second level is downsampled to produce the third level 14, and this in turn is downsampled to produce the fourth and final level 16. Figure 6C is an illustration of bilinear downsampling when generating the second level from the bottom level. Four pixels 10-1 to 10-4 of the original outer product array are averaged to generate one pixel 12-1 of the second level of the pyramid. This is repeated for each adjacent 2×2 pixel block of the original outer product array. Thus, each level of the pyramid is approximately one quarter the size of the previous level. In general, this process can continue to iterate until one or both spatial dimensions reach 1, or until a sufficiently low resolution has been reached.

[0230] Once the mipmap has been calculated for each outer product, a blur operation is implemented by selecting the relevant level of the pyramid and sampling appropriately to obtain a blurred version of the outer product (step 324). In some examples, the blurring only includes sampling operations at the appropriate level of the mipmap pyramid. But in other examples, the blurring may optionally include applying a filter to samples retrieved from the mipmap. Due to the reduced resolution of the higher pyramid levels, the mipmap can be filtered with a smaller number of calculations than if a filter of the same size was applied directly to the outer product. In step 126, the blur moments output from the blur step 324 are processed as previously described.

[0231] Some examples of mipmap-based implementations will now be described. Typically, one or more mipmap levels are selected based on the size of the blur filter kernel we wish to construct. In some examples, as described above, individual samples from the mipmap can be used directly to generate blur moments. This will implement a relatively simple square blur filter whose effective kernel size corresponds to the level of the selected pyramid. It should be noted that if the kernel size is fixed in advance, then only a single level of the mipmap pyramid is actually required. The construction of the mipmap can terminate at this level. Alternatively, the required single level can be generated directly (i.e., no recursion is required). In its simplest form, blurring can be performed based on nearest neighbor sampling, although this may have disadvantages in the form of potential aliasing artifacts.

[0232] In other examples, the mipmap pyramid can be sampled bilinearly at a single level of detail. In still other examples, the mipmap can be sampled between levels using trilinear sampling. This involves bilinear sampling at two consecutive levels of detail and linear interpolation between the results. In still other examples, the mipmap can be sampled using bicubic sampling. Bilinear sampling, trilinear sampling, and bicubic sampling are commonly used for texture sampling applications of mipmaps, and therefore, these operations may be hardware accelerated on some GPUs. This can be used to further speed up blur calculations.

[0233] If necessary, the samples extracted from the mipmap can then be (further) filtered (by any chosen sampling scheme) to produce blur moments, typically using a small low-pass filter kernel. Figure 7 An example of blurring 324 in this manner is shown. Samples are extracted from a predetermined level of the mipmap pyramid and filtered first with a horizontal Gaussian filter 402 and second with a vertical Gaussian filter 404. Separable filtering using two consecutive, orthogonal one-dimensional filters in this manner can further reduce computational complexity and increase speed. The use of a Gaussian kernel is an example of giving greater weight to samples at the center of a neighborhood than to samples at the edge. Other examples are also possible. For example, an infinite impulse response (IIR) filter can be used instead of a Gaussian filter.

[0234] Fig. 8A and Figure 8B Another example of mipmap-assisted blurring is shown. The inventors have discovered that it may be advantageous to consider the scene content when performing blurring. In particular, it may be advantageous to consider the orientation of the surface relative to the virtual camera. The reason for this is Fig. 8A . When the plane in the scene is parallel to the image plane, a circular area 25 on the image plane maps to a circular area 20a on the plane in the scene. However, when the plane in the scene is not parallel to the image plane, the same circular area 25 on the image plane maps to an elliptical area 20b on the plane in the scene. It is advantageous to take this into account for blurring. Instead of using a blur kernel that is isotropic (height=width) throughout the image, the shape of the kernel can be anisotropic (height≠width) and can be locally adapted to the orientation of the surface on which the blurring is being performed. The goal is to ensure that the projection of the kernel onto the scene is approximately constant and isotropic in scene space.

[0235] Figure 8BA method for applying this principle with the help of the mipmap created in step 322 is shown. In step 502, the model fitting block 220 determines the appropriate major and minor axis sizes and orientations of the filter footprint on the image plane based on the scene content at each pixel location. This is obtained from a combination of the screen space surface normal (e.g., available from the associated guidance channel) and the filter kernel size. The major and minor axes do not need to be axis aligned (i.e., horizontal and vertical). Anisotropic filtering will be performed based on the determined major and minor axes, as described in the example below. In step 504, the model fitting block 220, for example, by calculating the log 2 (the length of the minor axis in pixels) to select the appropriate level of detail in the mipmap pyramid. In general, this is not necessarily an integer value. It can be rounded to the nearest integer, or trilinear sampling can be used to obtain the level of detail between pyramid levels. In step 506, the model fitting block 220 identifies points along the length of the major axis at the level of detail selected in step 504. It extracts outer product samples from the mipmap at the specified level of detail through appropriate sampling. This can be, for example, point sampling, bilinear sampling, or bicubic sampling. The output of step 506 is the average of the sampled points along the major axis. This process is repeated for each pixel. In some examples, the average can be a weighted average with larger weights near the center of the kernel. The weights of the weighted average do not need to sum to one.

[0236] Of course, it should be understood that using mipmap / multi-scale pyramids is only one way to effectively implement blurring. Other effective filtering methods are also possible. For example, an IIR filter suitable for effectively implementing blurring is described in GB 2541179A.

[0237] Normalization

[0238] As explained above, the regularization is added to the moment matrix X before matrix inversion. T X, in order to increase numerical stability. However, the inventors have found that it is advantageous not to rely entirely on regularization to avoid numerical instability. Better quality rendering results can be achieved by (alternatively or additionally) normalizing the moment matrices. In this example, both moment matrices are normalized so that they appear to be generated from data with zero mean and unit standard deviation. This is achieved by extracting mean and standard deviation statistics from the original moment matrices. The first moment matrix X T X can be divided into:

[0239]

[0240] The normalized version can be extracted by the following calculation:

[0241]

[0242] in

[0243]

[0244] Here, μ is the mean and σ is the standard deviation of the data X. Similarly, the second moment matrix X T Y can be rewritten as:

[0245]

[0246] in is a row vector of 1's of compatible length.

[0247] From this the normalized version can be extracted as:

[0248]

[0249] The model is constructed from normalized matrices rather than raw moment matrices. When the model is later applied in the model assembly block 250, the transformation used for normalization should be inverted. In particular, the model assembly block “normalizes” the model parameters by applying To compute the product XA:

[0250]

[0251] Normalization has been found to help reduce unwanted blurring at the boundaries of structures in the scene (e.g., helps avoid "glow" around objects), as well as reducing noise from poorly conditioned matrices.

[0252] Quantification

[0253] Memory bandwidth and storage can be further reduced by quantizing the data for transmission to and from the GPU. Although it may be beneficial to perform calculations in floating point format, it is also possible to store and transmit data in a low bit depth integer format. This can reduce the bandwidth required to transfer data between the GPU and the memory. Memory access bandwidth is a particularly precious resource when transferring data to off-chip memory. Experiments have shown that the guide image can be stored in an 8-bit integer format (signed or unsigned, depending on the numerical range of each guide channel). The image quality of the resulting denoised image is quite close to that achieved using a floating point guide channel. Experiments have shown that more care may be needed with the illumination image. In particular, it has been found that direct (linear) quantization produces suboptimal results in terms of the quality of the denoised output image. The inventors have found that the use of nonlinear quantization can give better results in this context while still achieving a reduction in memory access bandwidth. Fig. 9A way of encoding 610 and decoding 620 illumination images for use in the rendering method of this example is shown. This can be particularly applied to noisy indirect illumination images generated by the path tracing block 210. However, it can also be applied to other illumination images, such as direct illumination images generated by the ray tracing block 260, and the final rendered output image. In step 612, nonlinearity is applied to the floating point illumination image. That is, the input floating point values ​​are transformed by mapping them to second floating point values ​​using a nonlinear function. Then, in step 614, the transformed input values ​​are uniformly quantized. The combination of nonlinearity and uniform quantization provides a larger density of quantization levels for darker areas of the image, while brighter areas of the image are relatively sparsely quantized. Suitable nonlinearities for achieving this effect include square root functions and logarithmic functions. After quantization in step 614, the quantized floating point numbers are converted into low bit depth integers in step 616. The low bit integers are then transferred from the GPU to memory (e.g., on-chip or external memory). Quantization reduces the amount of data to be stored in memory, and more importantly, reduces the bandwidth required to transfer data to and from memory. In this specific implementation, the low bit depth integers are stored with a bit depth of 8 bits. However, different low bit depth formats can be used for different guide channels and / or noisy images. That is, during any given process of producing a denoised image, different guide channels and noisy images do not need to use the same low bit depth format, and they can be encoded and decoded independently of the different formats. For example, this may be advantageous if it is found that using more bits for depth than for the surface normal component is beneficial to the overall result.

[0254] When the data is needed (e.g., when it is retrieved from memory for computing an inner product), it is decoded 620 to reverse the nonlinear quantization. In step 622, the low bit depth integers are converted back to floating point numbers. Then, in step 624, an inverse nonlinearity is applied to reverse the effect of the nonlinear function in step 612. For example, if a square root function was applied in step 612, a quadratic (square) function would be applied in step 624. If a logarithmic function was applied in step 612, the function applied in step 624 would involve an exponent.

[0255] The denoised output image may also be quantized after being produced. This provides similar benefits to those described above by outputting the denoised image at a lower bit depth (compared to without the quantization step), i.e., bandwidth is saved when outputting the denoised image. This may be particularly beneficial because the denoised output is typically significantly larger than the noisy low-resolution input, and therefore would represent a significantly greater bandwidth load if output in an unquantized manner.

[0256] Alternative pipeline

[0257] Figure 1 The pipeline shown is not the only way to render an image using low-resolution denoising via guided filtering. Fig.10 Another example is shown. Fig.10 and Figure 1 The difference is that the noisy low resolution indirect illumination image is combined with the direct illumination image prior to the denoising procedure. In steps 110 and 112, the low resolution noisy indirect illumination image is rendered by path tracing, as previously described, and the low resolution guide channels (except for ambient occlusion guides) are rendered by rasterization. In step 113, the ambient occlusion guides are generated in a low resolution version and a full resolution version. However, instead of rendering the full resolution direct illumination image in step 160, the ray tracing block renders the low resolution direct illumination image in step 760. In step 770, the low resolution direct illumination image and the low resolution indirect illumination image are summed. The resulting global illumination image is noisy because it includes indirect illumination. In step 720, this noisy image is passed to the model fitting block, and model parameters are estimated in the same manner as described above for step 120. The resulting model parameters produced at a low resolution are upsampled in step 730 and combined with the full resolution guide channels in step 750. This produces a full resolution denoised global illumination image. As Figure 1 As shown, in step 180, the global illumination image is combined with the surface reflection image to produce a final rendered output image. Fig.10 For best results of the pipeline, the low-resolution image is preferably the same size (same resolution) as the full-resolution image. Otherwise, there is a risk of suboptimal reconstruction of shadow edges, introducing undesirable blurring on these edges.

[0258] In another variation of the pipeline, ray tracing 160 or 760 may be replaced by rasterization, optionally using shadow mapping, for example.

[0259] Reference again Figure 1 , it is explained above that the resolution of the "full resolution" image is greater than or equal to the resolution of the "low resolution" image. When the "low resolution" image has the same resolution as the "full resolution" image, it should be understood that the rasterization in steps 112 and 140 can be performed in a single step because the same set of guidance channels is used when calculating the model parameters and applying them. Alternatively, they can still be performed as two steps (such as Figure 1 ) wherein step 140 repeats the rasterization of the guide pass performed in step 112. Likewise, in step 113, a single ambient occlusion guide may be generated at full resolution. The upsampling step 130 may be eliminated.

[0260] Fig.11A simplified pipeline is shown that can be employed when the entire pipeline operates at the same resolution. Here, the same guide pass (generated in the rasterization step 140) and the same ambient occlusion guide (generated in step 113) are used both for model fitting in step 120 and for application of the model in step 150. As described above, the upsampling step 130 is eliminated. The low-resolution path tracing of step 110 is replaced by full-resolution path tracing in the corresponding step 110'.

[0261] Fig.12 A process 113 of obtaining (particularly generating) an ambient occlusion guide according to the present example is shown. In step 114, the rendering / rasterization block 240 retrieves a plurality of ambient occlusion maps from a memory. Each ambient occlusion map relates to an object model in a 3D scene. Each ambient occlusion map describes the interaction of ambient lighting with a corresponding object. As each model is designed, the ambient occlusion map is generated offline in texture space. In step 116, the rendering / rasterization block 240 projects each ambient occlusion map into screen space based on the pose (i.e., position and orientation) of the corresponding object in the 3D scene. In step 118, the rendering / rasterization block 240 combines the projected ambient occlusion maps in screen space to generate an ambient occlusion guide. According to the present example, the combination consists of summing the ambient occlusion maps. As previously described, the ambient occlusion guide can be generated at two resolutions to support pipelines that perform model fitting at low resolutions (e.g., Figure 1 and Fig.10 of those).

[0262] Alternative lighting modes

[0263] While the above example combines a direct and indirect lighting image to provide global illumination, this is not the only possible implementation.

[0264] Alternative model fitting

[0265] Although the above example models the noisy input image as a linear combination of the guide channels in an affine model, the scope of the present disclosure is not limited in this regard. Other models may be used, including, for example, nonlinear / higher order models. An example of such a higher order model would be a quadratic fit. A quadratic model may be implemented by including the square of any guide channel (including ambient occlusion guides) in the guide image.

[0266] Alternative boot channel

[0267] In the above example, the guiding channels include a single ambient occlusion guide. This is not required. Other forms of guiding channels that characterize the spatial dependence of incident light on global illumination can be used. The guiding channels used for denoising can include one or more of these guiding channels. For example, each 3D model can be associated with multiple maps describing the spatial dependence of incident light on (directional) global illumination, each map being associated with a different dominant illumination direction relative to the model. It may be useful to include such diversity in the guiding channels used for denoising, because the illumination in a 3D scene can have specific dominant directions. Object edges / contours that produce prominent shadows when illuminated in one direction may be less prominent when illuminated in another direction. By including guiding channels that characterize different illumination directions, the model fitting process can weight more heavily the guiding channel(s) that are closest to the appearance of objects in the 3D scene under given illumination conditions.

[0268] Similar to Fig.12 , the method may include projecting each different occlusion map into screen space based on the pose of the object-model, and combining the resulting projected maps of different object-models having corresponding principal illumination directions. For the avoidance of doubt, each combination involves one map per object-model associated with a particular illumination direction (rather than multiple maps for the same object-model). Thus, each combination produces a guidance channel associated with a different illumination direction in screen space.

[0269] Selection of guide channel

[0270] As previously mentioned, various types of guiding channels can be used. In theory, the more guiding channels are available, the better the model fit is likely to be. However, the computational burden of model fitting increases with the number of guiding channels. Therefore, it is desirable to select a small number of guiding channels that are most informative for the model. Note that in this sense, the "best" set of guiding channels may be different for different types of 3D scenes, different viewing angles, and different lighting conditions. The inventors have recognized that it is desirable to automatically select guiding channels in a principled manner.

[0271] Fig.13 A method of selecting a guide channel according to an example is shown. This method can be performed by a GPU or a CPU. In this example, these steps are implemented by the CPU and GPU working together. Specifically, denoising (step 840) is performed by the GPU, for example, implementing one of the pipelines described above. It should be understood that in other embodiments, the method can be performed entirely by the CPU, entirely by the GPU, or with some other division of labor between the CPU and the GPU.

[0272] In step 810, the CPU obtains a plurality of noisy training images. It also obtains a plurality of corresponding "real" images, which will be used as gold standards for the denoising operation. According to this example, all of these images are obtained from the GPU, which generates them by path tracing (same as steps 110 and 110' of the aforementioned method above). The noisy training images are generated using a relatively small number of samples per pixel. The real images are generated using a higher number of samples per pixel, and therefore they are much less noisy.

[0273] In step 820, the CPU obtains candidate guiding channels to be evaluated. In step 830, it starts the evaluation by selecting one or more candidate guiding channels to be evaluated in the first iteration. One or more guiding channels of other groups will be evaluated in subsequent iterations. In step 840, the GPU performs denoising on each noisy training image using the one or more candidate guiding channels selected in step 830. This produces a set of denoised training images—one denoised training image for each noisy training image. In step 850, each such denoised training image is compared with its corresponding real image. This enables the CPU to determine (in step 860) the error between each pair of images. In this example, the error metric is the sum of the squared pixel differences between the two images; however, it should be understood that other error metrics can be used. The CPU sums the errors generated by the error metric over the entire training image set (and the corresponding real images). This summed error is a measure of how much the denoised image differs from its corresponding real image in the entire training set. Low values ​​of the summed error indicate that the denoising is able to accurately approximate the low-noise true image, in other words, the denoising algorithm has been able to successfully compensate for the reduced number of samples per pixel in the noisy training image.

[0274] In several iterations, different guide channels or combinations of guide channels are evaluated and the CPU determines the associated error metrics. This enables the CPU to select the guide channels that it considers to be the best (i.e., the most informative) for denoising in step 870. These channels are the candidate guide channels associated with the lowest determination error. The selection in step 870 can be embedded in the iteration, thereby performing the selection incrementally. Alternatively, the selection in step 870 can be a subsequent step, which is performed after all desired guide channels are combinations of the guide channels that have been evaluated.

[0275] In some cases, guiding channels can be added in a greedy manner. The guiding channels can be evaluated one by one, and the guiding channel that produces the highest quality can be selected as the first guiding channel. Next, each of the remaining guiding channels can be evaluated in conjunction with the first guiding channel. The guiding channel that produces the highest quality can be selected as the second guiding channel to supplement the first guiding channel. This process can continue until a suitable number of guiding channels have been selected.

[0276] A simpler greedy algorithm can also be implemented by evaluating the guiding channels one by one and only selecting the N guiding channels that provide the best individual performance. However, a disadvantage of this simplified greedy algorithm is that it does not consider the potential synergy between complementary guiding channels (because they are only evaluated one by one).

[0277] More sophisticated strategies are also possible. For example, the CPU can evaluate all possible combinations of N guiding channels selected from M candidate guiding channels (where M>N). A subset of the N guiding channels that achieve the lowest error in the training set is selected for denoising.

[0278] In each of the above examples, the selection of the “learned” guiding channels is done by intelligently choosing by minimizing the difference between the high-sampled-rate target (ie, the true image) and the denoised image denoised with one or more “learned” guiding channels.

[0279] Another possibility is to construct one or more guiding channels by performing optimization within the guiding channels themselves. In this case, the guiding channels do not need to be provided in advance from an external source. Instead, they can be learned through an iterative optimization process. The optimization process uses a training dataset similar to the above, containing noisy training images and corresponding real images. The learning is guided by comparing the denoising results using the current guiding channel with the relevant real image. At each iteration, the pixels of the current guiding channel are updated in such a way that the effect is to reduce the error between the denoised training image and the real image. After enough iterations, the guiding channels converge to a set that minimizes the error.

[0280] Iterative optimization can use a gradient descent algorithm. By computing the derivatives of the denoised training image with respect to one or more guide channels, the gradient descent algorithm can incrementally adjust the pixels of one or more guide channels or their precursors in each iteration in a deterministic manner that can be expected to reduce the error. In order to apply an optimization algorithm based on gradient descent, the rendering pipeline should be differentiable.

[0281] The training data may include several noisy training images and corresponding real images of a given object or scene. These noisy training images may represent the object or scene under different illumination conditions.

[0282] When constructed for individual objects, guidance images can be generated in texture space. They can then be projected into screen space for use in denoising methods. When constructed for the entire scene, guidance images can be generated directly in screen space.

[0283] Other Examples

[0284] The following examples are exposed:

[0285] A1. A method for selecting one or more guiding channels for image denoising, the method comprising:

[0286] Obtaining (810) a plurality of noisy training images;

[0287] Obtaining (810) a corresponding plurality of real images, wherein each real image depicts the same 3D scene as a corresponding noisy training image, and each real image has less noise than the corresponding noisy training image;

[0288] Obtaining (820) a plurality of candidate guide channels;

[0289] Conduct multiple assessments, each including:

[0290] Selecting (830) one or more candidate guiding channels to be evaluated from the candidate guiding channels;

[0291] Denoising each of the noisy training images using the selected candidate guiding channels (840) to generate a corresponding denoised training image; and

[0292] comparing (850) each denoised training image with the corresponding true image to determine (860) the error therebetween; and

[0293] One or more guiding channels for image denoising are selected (870) based on the errors determined in the plurality of evaluations.

[0294] A2. The method of example A1, wherein the denoising comprises, for each noisy training image:

[0295] For each of the multiple local neighborhoods of the noisy training image:

[0296] calculating (120) parameters of a model that approximates the noisy training image as a function of the one or more guidance channels, and

[0297] The calculated parameters are applied (150) to the one or more guidance channels to generate corresponding denoised training images.

[0298] A3. A method according to Example A1 or Example A2, wherein the multiple candidate guiding channels include at least one guiding channel, and the at least one guiding channel characterizes the spatial dependence of incident light on the global illumination on the surface of one or more 3D models in the scene.

[0299] A4. A method according to example A3, wherein the at least one guide channel includes or consists of ambient occlusion data.

[0300] A5. A method according to any of the preceding examples, wherein obtaining the at least one guide channel comprises:

[0301] obtaining (114) one or more ambient occlusion maps for corresponding one or more models in the scene; and

[0302] The one or more ambient occlusion maps are projected (116) into screen space to generate ambient occlusion guides in screen space.

[0303] A6. A method according to any of the preceding examples, wherein obtaining (810) each noisy training image and the corresponding real image comprises rendering by path tracing.

[0304] A7. A method according to any of the preceding examples, wherein each evaluation includes (830) selecting an individual guiding channel to be evaluated, and wherein selecting (870) one or more guiding channels for image denoising includes greedily selecting one or more guiding channels associated with the lowest determination error among all individual evaluations.

[0305] A8. A method according to any of the preceding examples, wherein selecting the one or more guiding channels is performed iteratively, comprising selecting (870) one or more first candidate guiding channels associated with the lowest determination error in one iteration, and

[0306] wherein, in subsequent iterations, each evaluation includes selecting the one or more first candidate guiding channels that have been selected, and also selecting one or more additional candidate guiding channels to be evaluated, and

[0307] Wherein the subsequent iterations include selecting, in each evaluation, one or more additional candidate guiding paths among the one or more additional candidate guiding paths that are associated with the lowest determination error.

[0308] A9. A method according to any of the preceding examples, wherein each evaluation includes selecting a subset of candidate guiding channels to be evaluated together, and wherein selecting the one or more guiding channels includes selecting the subset of candidate guiding channels associated with the lowest determination error.

[0309] A10. Computer readable code, the computer readable code being configured to cause the method according to any of the preceding examples to be performed when the code is executed.

[0310] The following additional examples are also exposed:

[0311] B1. A method for constructing one or more guiding channels for image denoising, the method comprising:

[0312] Obtain multiple noisy training images;

[0313] Obtaining a corresponding plurality of real images, wherein each real image depicts the same 3D scene as the corresponding noisy training image, and each real image has less noise than the corresponding noisy training image;

[0314] The one or more guiding channels are constructed based on an optimization process, wherein the optimization process constructs the one or more guiding channels so as to minimize an error between each real image and a corresponding denoised training image, wherein the denoised training image is generated by denoising the corresponding noisy training image using the one or more guiding channels.

[0315] B2. A method according to example B1, wherein the optimization process is iterative.

[0316] B3. The method of example B2, wherein the iterative optimization process comprises, for each iteration of a plurality of iterations:

[0317] denoising the noisy training image using the one or more guiding channels to produce a denoised training image;

[0318] comparing the denoised training image with the corresponding true image to determine an error therebetween; and

[0319] The one or more guidance channels are updated to reduce the error.

[0320] B4. A method according to Example B2 or Example B3, wherein the iterative optimization process includes calculating the derivative of the denoised training image relative to the one or more guide channels.

[0321] B5. A method according to any of the preceding examples, wherein the one or more guide channels are constructed in a texture space.

[0322] B6. A method according to example B5, wherein the one or more guide channels are projected from texture space to screen space before denoising the noisy training image.

[0323] B7. A method according to any of the preceding examples, wherein the denoising comprises, for each noisy training image:

[0324] For each of the multiple local neighborhoods of the noisy training image:

[0325] computing parameters of a model that approximates the noisy training image as a function of the one or more guidance channels, and

[0326] The calculated parameters are applied to the one or more guidance channels to generate corresponding denoised training images.

[0327] B8. A method according to any of the preceding examples, wherein obtaining each noisy training image and the corresponding real image comprises rendering by path tracing.

[0328] B9. Computer readable code, which is configured to enable the method according to any one of the aforementioned examples B1 to B8 to be executed when the code is executed.

[0329] system

[0330] Fig.14 A graphics processing system in which a graphics processing unit as described herein may be implemented is shown. The computer system includes a CPU 902, a GPU 904, a memory 906, and other devices 914, such as a display 916, a speaker 918, and a camera 919. Processing block 910 (corresponding to processing blocks 210-280) is implemented on GPU 904. In other examples, processing block 910 may be implemented on CPU 902. The components of the computer system may communicate with each other via a communication bus 920. A storage repository 912 (corresponding to storage repository 112) is implemented as part of the memory 906.

[0331] Although Fig.14 One implementation of a graphics processing system is shown, but it should be understood that a similar block diagram can be drawn for an artificial intelligence accelerator system—for example, by replacing the CPU 902 or GPU 904 with a neural network accelerator (NNA), or by adding an NNA as an additional unit. In such cases, processing block 910 can be implemented in the NNA.

[0332] Figure 2 GPU and Figure 1 and Fig.10 The method / pipeline is shown as comprising a plurality of functional blocks. This is merely illustrative and is not intended to define a strict division between different logical elements of such entities. Each functional block may be provided in any suitable manner. It should be understood that intermediate values ​​described herein as being formed by the GPU need not be physically generated by the GPU at any point, and may merely represent logical values ​​that conveniently describe the processing performed by the GPU between its input and output.

[0333] The GPU and graphics processing system described herein may be included in hardware on an integrated circuit. The GPU and graphics processing system described herein may be configured to perform any of the methods described herein. In general, any of the functions, methods, techniques, or components described above may be implemented in software, firmware, hardware (e.g., fixed logic circuitry), or any combination thereof. The terms "module," "function," "component," "element," "unit," "block," and "logic" may be used herein to generally represent software, firmware, hardware, or any combination thereof. In the case of software implementation, a module, function, component, element, unit, block, or logic represents a program code that, when executed on a processor, performs a specified task. The algorithms and methods described herein may be performed by one or more processors that execute code, which causes (one or more) processors to perform the algorithm / method. Examples of computer-readable storage media include random access memory (RAM), read-only memory (ROM), optical disks, flash memory, hard disk storage, and other memory devices that can use magnetic, optical, and other technologies to store instructions or other data and can be accessed by a machine.

[0334] As used herein, the terms computer program code and computer readable instructions refer to any kind of executable code for execution by a processor, including code expressed in machine language, interpreted language, or scripting language. Executable code includes binary code, machine code, byte code, code that defines an integrated circuit (such as a hardware description language or netlist), and code written in languages ​​such as C, Executable code can be, for example, any kind of software, firmware, script, module or library that, when properly executed, processed, interpreted, compiled, or run in a virtual machine or other software environment, enables the processor of a computer system supporting the executable code to perform the tasks specified by the code.

[0335] A processor, computer or computer system may be any kind of device, machine or special purpose circuit, or a collection or part thereof having processing capabilities so that it can execute instructions. A processor may be any kind of general or special purpose processor, for example, a CPU, GPU, NNA, system on chip, state machine, media processor, application specific integrated circuit (ASIC), programmable logic array, field programmable gate array (FPGA), etc. A computer or computer system may include one or more processors.

[0336] The present invention is also intended to cover software that defines the configuration of hardware as described herein, such as HDL (Hardware Description Language) software, such as for designing integrated circuits, or for configuring programmable chips to perform desired functions. That is, a computer-readable storage medium may be provided on which is encoded a computer-readable program code in the form of an integrated circuit definition data set, which, when processed (i.e., run) in an integrated circuit manufacturing system, configures the system to manufacture a graphics processing system configured to perform any of the methods described herein, or to manufacture a graphics processing system including any of the devices described herein. The integrated circuit definition data set may be, for example, an integrated circuit description.

[0337] Thus, a method of manufacturing a graphics processing system as described herein at an integrated circuit manufacturing system may be provided. Furthermore, an integrated circuit definition data set may be provided which, when processed in an integrated circuit manufacturing system, enables the method of manufacturing a graphics processing system to be performed.

[0338] The integrated circuit definition data set may be in the form of computer code, for example as a netlist, code for configuring a programmable chip, as a hardware description language that defines hardware suitable for fabrication at any level in an integrated circuit, including as register transfer level (RTL) code, as a high-level circuit representation (e.g., Verilog or VHDL), and as a low-level circuit representation (e.g., OASIS (RTM) and GDSII). The higher-level representation (e.g., RTL) that logically defines hardware suitable for fabrication in an integrated circuit may be processed at a computer system configured to generate a manufacturing definition of the integrated circuit in the context of a software environment that includes definitions of circuit elements and rules for combining those elements to generate a manufacturing definition of the integrated circuit so defined by the representation. As is typically the case with software executed at a computer system to define a machine, one or more intermediate user steps (e.g., providing commands, variables, etc.) may be required to configure the computer system for generating a manufacturing definition of the integrated circuit to execute code that defines the integrated circuit to generate the manufacturing definition of the integrated circuit.

[0339] Now about Fig.15 An example of processing an integrated circuit definition data set at an integrated circuit manufacturing system to configure the system to manufacture a graphics processing system is described.

[0340] Fig.15An example of an integrated circuit (IC) manufacturing system 1002 is shown, which is configured to manufacture a graphics processing system as described in any example herein. In particular, the IC manufacturing system 1002 includes a layout processing system 1004 and an integrated circuit generation system 1006. The IC manufacturing system 1002 is configured to receive an IC definition data set (e.g., defining a graphics processing system as described in any example herein), process the IC definition data set, and generate an IC (e.g., which includes a graphics processing system as described in any example herein) based on the IC definition data set. By processing the IC definition data set, the IC manufacturing system 1002 is configured to manufacture an integrated circuit including a graphics processing system as described in any example herein.

[0341] The layout processing system 1004 is configured to receive and process an IC definition data set to determine a circuit layout. Methods for determining a circuit layout based on an IC definition data set are known in the art and may, for example, involve synthesizing RTL code to determine a gate-level representation of a circuit to be generated, for example, in terms of logic components (e.g., NAND, NOR, AND, OR, MUX, and FLIP-FLOP components). By determining location information for the logic components, the circuit layout may be determined by the gate-level representation of the circuit. This may be done automatically or with user involvement in order to optimize the circuit layout. When the layout processing system 1004 has determined the circuit layout, it may output the circuit layout definition to the IC generation system 1006. The circuit layout definition may be, for example, a circuit layout description.

[0342] As is known in the art, the IC generation system 1006 generates the IC according to the circuit layout definition. For example, the IC generation system 1006 may implement a semiconductor device manufacturing process to generate the IC, which may involve a multi-step sequence of photolithography and chemical processing steps during which electronic circuits are gradually formed on a wafer made of semiconductor material. The circuit layout definition may be in the form of a mask that can be used in a photolithography process to generate the IC according to the circuit definition. Alternatively, the circuit layout definition provided to the IC generation system 1006 may be in the form of a computer readable code that the IC generation system 1006 may use to form a suitable mask for generating the IC.

[0343] The different processes performed by the IC manufacturing system 1002 may all be implemented at one location, such as by one party. Alternatively, the IC manufacturing system 1002 may be a distributed system such that some processes may be performed at different locations and may be performed by different parties. For example, some of the following stages may be performed at different locations and / or by different parties: (i) synthesizing RTL code representing an IC definition data set to form a gate-level representation of a circuit to be generated; (ii) generating a circuit layout based on the gate-level representation; (iii) forming a mask based on the circuit layout; and (iv) using the mask to manufacture the integrated circuit.

[0344] In other examples, processing of an integrated circuit definition data set in an integrated circuit manufacturing system may configure the system to manufacture a graphics processing system without processing the IC definition data set to determine a circuit layout. For example, the integrated circuit definition data set may define a configuration of a reconfigurable processor such as an FPGA, and processing of the data set may configure the IC manufacturing system to generate a reconfigurable processor having the defined configuration (e.g., by loading the configuration data into the FPGA).

[0345] In some embodiments, when processed in an integrated circuit manufacturing system, the integrated circuit manufacturing definition data set can enable the integrated circuit manufacturing system to generate an apparatus as described herein. For example, by using the integrated circuit manufacturing definition data set as described above with reference to Fig.15 The integrated circuit manufacturing system can be configured in the described manner to manufacture the device described herein.

[0346] In some examples, the integrated circuit definition data set may include software that runs on, or in combination with, hardware defined at the data set. Fig.15 In the example shown, the IC generation system may be further configured by the integrated circuit definition dataset to load firmware onto the integrated circuit according to the program code defined in the integrated circuit definition dataset when manufacturing the integrated circuit, or to otherwise provide the integrated circuit with program code for use with the integrated circuit.

[0347] Compared with known implementations, the implementation of the concepts set forth in this application in devices, equipment, modules and / or systems (and in the methods implemented herein) can improve performance. Performance improvements can include one or more of improved computing performance, reduced latency, increased throughput and / or reduced power consumption. During the manufacture of such devices, devices, modules and systems (e.g., in integrated circuits), a trade-off can be made between performance improvements and physical implementations to improve manufacturing methods. For example, a trade-off can be made between performance improvements and layout area to match the performance of known implementations, but using less silicon. For example, this can be accomplished by reusing functional blocks in a serial manner or sharing functional blocks between elements of devices, devices, modules and / or systems. In contrast, the concepts set forth in this application that cause improvements in the physical implementations of devices, devices, modules and systems (e.g., reduced silicon area) can be weighed against performance improvements. This can be accomplished, for example, by manufacturing multiple instances of a module within a predefined area budget.

[0348] Applicants hereby independently disclose each individual feature described herein and any combination of two or more such features, to the extent that such feature or combination can be implemented based on the specification as a whole in view of the common general knowledge of a person skilled in the art, regardless of whether such feature or combination of features solves any problem disclosed herein. In view of the foregoing description, it will be clear to a person skilled in the art that various modifications can be made within the scope of the invention.

Claims

1. A method for rendering an image of a 3D scene, the method comprising: Rendering (110') a noisy image; obtaining (140, 113) one or more guide channels; as well as For each of multiple local neighborhoods: calculating (120) parameters of a model that approximates the noisy image as a function of the one or more guiding channels, and applying (150) the calculated parameters to the one or more guiding channels to produce a denoised image, The one or more guiding channels include at least one guiding channel characterizing a spatial dependency of incident light on global illumination on a surface of one or more 3D models in the scene.

2. A method for rendering an image of a 3D scene, the method comprising: Rendering (110) a low-resolution noisy image; obtaining (112, 113) one or more low-resolution guidance channels and obtaining (140, 113) one or more corresponding full-resolution guidance channels; as well as For each of multiple local neighborhoods: calculating (120) parameters of a model that approximates the low resolution noisy image as a function of the one or more low resolution guiding channels, and applying (130, 150) the calculated parameters to the one or more full-resolution guiding channels to produce a full-resolution denoised image, Wherein, the one or more low-resolution guide channels and the one or more corresponding full-resolution guide channels each include at least one guide channel, and the at least one guide channel characterizes the spatial dependence of incident light on the global illumination on the surface of one or more 3D models in the scene.

3. The method according to claim 1 or claim 2, wherein: The at least one guide channel comprises or consists of ambient occlusion data.

4. A method according to any one of the preceding claims, wherein: Obtaining the at least one guiding channel comprises: obtaining (114) one or more ambient occlusion maps for corresponding one or more models in the scene; and The one or more ambient occlusion maps are projected (116) into screen space to generate ambient occlusion guides in screen space.

5. A method according to any one of the preceding claims, wherein: Rendering (110, 110') the noisy image includes rendering by path tracing.

6. A method according to any one of the preceding claims, wherein: The noisy image includes indirect illumination in the scene.

7. A method according to any one of the preceding claims, comprising: defining (115) a first tile defining respective first contiguous portions of the noisy image and the one or more guide channels, each first contiguous portion comprising a first plurality of pixels; defining (115) a second tile defining respective second contiguous portions of the noisy image and the one or more guide channels, each second contiguous portion comprising a second plurality of pixels; Calculate (122a) a first outer product (x) between each pixel (x) in the one or more guide channels and itself. T x); and Calculate (122a) a second outer product (x) between each pixel (x) in the one or more guiding channels and a corresponding pixel (y) in the noisy image. T y), For pixels in the first image block, the first outer product and the second outer product are calculated (i) before the second image block or (ii) in parallel with the second image block.

8. A method according to any one of the preceding claims, wherein: At least one of the noisy image, the one or more guide channels, and the denoised image is stored in a quantized low bit depth format.

9. The method according to claim 8, further comprising: After rendering (110, 110') the noisy image, quantizing it in a quantized low bit depth format using non-linear quantization (610) such that darker areas of the image are quantized to a relatively greater quantization level density and lighter areas of the image are quantized to a relatively lesser quantization level density; and storing the quantized low bit depth format in memory, The method further includes, before calculating the parameters of the model, retrieving the quantized low bit depth values ​​from the memory and performing inverse quantization (620).

10. A method according to any one of the preceding claims, wherein: Calculating the parameters of the model includes: Calculate (122; 122a) a first outer product (x) between each pixel (x) in the one or more guide channels and itself. T x); Calculate (122; 122a) a second outer product (x) between each pixel (x) in the one or more guide channels and a corresponding pixel (y) in the noisy image. T y); Fuzzy (124, 124a) the first outer product to calculate the first moment matrix (X T X); Fuzzy (124, 124a) the second outer product to calculate the second moment matrix (X T Y); and For each local neighborhood, parameters (A) of the model are calculated (126), including calculating the inverse of the first moment matrix and calculating the product of the inverse matrix and the second moment matrix.

11. The method according to claim 10, comprising: defining (125) a first outer product patch, defining first contiguous portions of the first outer product and corresponding first contiguous portions of the second outer product, each first contiguous portion comprising a first plurality of pixels; as well as defining (125) a second outer product patch, defining second contiguous portions of the first outer product and corresponding second contiguous portions of the second outer product, each second contiguous portion comprising a second plurality of pixels, Wherein, for the first image block, the first moment matrix and the second moment matrix are calculated (124a) (i) before the second image block or (ii) in parallel with the second image block.

12. A graphics processing unit configured to render an image of a 3D scene, the graphics processing unit comprising: A first rendering block (210), the first rendering block being configured to render (110') a noisy image; a second rendering block (240), the second rendering block being configured to render (140) one or more guide channels; a model fitting block (220) configured to compute (120), for each of a plurality of local neighborhoods, parameters of a model that approximates the noisy image as a function of the one or more guiding channels; as well as a model application block (250) configured to apply (150) the calculated parameters to the one or more guidance channels for each of the plurality of local neighborhoods to produce a denoised image, The one or more guiding channels include at least one guiding channel characterizing a spatial dependency of incident light on global illumination on a surface of one or more 3D models in the scene.

13. A graphics processing unit configured to render an image of a 3D scene, the graphics processing unit comprising: A first rendering block (210) configured to render (110) a noisy low-resolution image; a second rendering block (240) configured to render (112) one or more low-resolution guide passes; a third rendering block (240) configured to render (140) one or more full-resolution guide passes, each full-resolution guide pass corresponding to a corresponding low-resolution guide pass; a model fitting block (220) configured to compute (120), for each of a plurality of local neighborhoods, parameters of a model that approximates the noisy low resolution image as a function of the one or more low resolution guiding channels; as well as a model application block (230, 250) configured to apply (130, 150) the calculated parameters to the one or more full-resolution guidance channels for each of the plurality of local neighborhoods to produce a full-resolution denoised image, Wherein, the one or more low-resolution guide channels and the one or more corresponding full-resolution guide channels each include at least one guide channel, and the at least one guide channel characterizes the spatial dependence of incident light on the global illumination on the surface of one or more 3D models in the scene.

14. A graphics processing unit according to any one of claims 12 to 13, wherein: The second rendering block (240) is configured to render (112; 140) one or more of the one or more guide passes by rasterization or ray tracing.

15. A computer-readable storage medium having computer-readable code encoded thereon, the computer-readable code being configured to cause the method according to any one of claims 1 to 11 to be performed when the code is executed.

Images