Per-illumination component correction

Abstract

Devices and / or processes are disclosed for processing image frames partially expressed by different illumination components, such as illumination components generated using ray tracing. In one embodiment, different illumination components of a previous image frame can be warped individually and combined with similar illumination components in a current image frame.

Description

Technical Field

[0001] The field generally relates to correction of illumination components in image frames rendered using ray tracing. Background Art

[0002] The use of computers to render images has evolved from the low-resolution, simple line drawings with limited colors familiar from arcade games decades ago to complex, photo-realistic images that are rendered to provide content such as immersive games, virtual reality, and high-definition CGI (computer-generated imagery) movies. While some image rendering applications such as rendering computer-generated movies can be completed over the course of many days, other applications such as video games and virtual reality or augmented reality may require real-time rendering of the relevant image content. Because the computational complexity can increase with the desired degree of realism, effectively rendering real-time content while providing acceptable image quality is an ongoing technical challenge.

[0003] Producing realistic computer-generated images typically involves a variety of image rendering techniques, including correctly rendering the viewer's perspective, rendering different surface textures, and providing realistic lighting. Rendering lighting can be particularly challenging and computationally expensive because it typically involves tracing a large number of light beams between the light source and the viewer that reflect off the various surfaces that the light may hit. Adding complexity (such as secondary reflections where the light beam bounces two or more times between the light source and the viewer, or diffuse reflections from irregular surfaces that may absorb some portion of the light) further complicates the calculations, but increases the realism of the resulting rendered image.

[0004] Several techniques can be used to reduce the number of rays traced to produce a high-quality rendered image, such as tracing only those rays that are visible to the viewer by starting from the viewer's perspective and tracing backward to the light source. Even with such techniques, the number of rays that can be traced per image while maintaining a reasonable frame rate for applications such as games or virtual / augmented reality can be several orders of magnitude lower than the number of rays captured by a camera taking the same image in real life. This can result in slightly noisy ray traced images, in which some pixels may appear too dark or too bright due to the relatively small number of rays traced when ray tracing was applied to the rendered image. For reasons such as these, it is desirable to manage noise in ray traced rendered images, such as during real-time rendering of an image stream. BRIEF DESCRIPTION OF THE DRAWINGS

[0005] The claims provided in this application are not limited by the examples provided in the specification or drawings, but rather their organization and / or method of operation as well as features and / or advantages may be best understood by reference to the following detailed description and examples provided in the drawings, wherein:

[0006] Figure 1 is a block diagram of a ray tracing pipeline consistent with an example embodiment.

[0007] Figure 2 Ray tracing of rendered objects is shown consistent with an example embodiment.

[0008] Figure 3 A three-level mip-map or image pyramid is shown, consistent with an example embodiment.

[0009] Figure 4A and Figure 4B is a block diagram of a more complex system for implementing ray tracing denoising, consistent with an example embodiment.

[0010] Figure 5 is a flow chart of a process for correcting re-projected previous image frame data, consistent with an example embodiment.

[0011] Figure 6 is a flow chart of a process for using a double buffer for an accumulated rendered image history, consistent with an example embodiment.

[0012] Figure 7 is a flow chart of a method for denoising a rendered image using progressively downsampled texture maps and trilinear filtering, consistent with an example embodiment.

[0013] Figure 8 is a schematic diagram of a neural network consistent with an example embodiment.

[0014] Fig. 9 One or more image processing and / or filtering architectures (e.g., image processing stages, Figure 1 ) computing environment.

[0015] Fig.10 A block diagram of a general computerized system consistent with an example embodiment is shown.

[0016] Reference is made to the accompanying drawings in the following detailed description, which form a part of the description, and throughout the description, similar numbers may represent similar parts, which are corresponding and / or similar. The drawings are not necessarily drawn to scale, such as for simplicity and / or clarity. For example, the dimensions of some aspects may be enlarged relative to other aspects. Other embodiments may be utilized, and structural and / or other changes may be made without departing from the content claimed. Directions and / or references (e.g., such as up, down, top, bottom, etc.) may be used to facilitate discussion of the drawings and are not intended to limit the application of the claimed subject matter. Therefore, the following detailed description does not limit the claimed subject matter and / or equivalents. DETAILED DESCRIPTION

[0017] In the following detailed description of example embodiments, reference is made to specific example embodiments through the accompanying drawings and illustrations. These examples are described in sufficient detail to enable those skilled in the art to practice what is described, and are used to illustrate how the elements of these examples can be applied to various purposes or embodiments. Other embodiments exist, and logical, mechanical, electrical, and other changes may be made.

[0018] The features or limitations of the various embodiments described herein, no matter how important to the example embodiments into which they are incorporated, do not limit other embodiments, and any reference to the elements, operations, and applications of the examples is only used to help understand these example embodiments. The features or elements shown in the various examples described herein can be combined in a manner other than that shown in the examples, and any such combination is explicitly expected to be within the scope of the examples presented here. Therefore, the detailed description below does not limit the scope of the claimed protection.

[0019] Ray tracing light in a computer rendered image can significantly improve the apparent realism of the rendered image, showing features such as directly illuminated surfaces and shadows that are part of our normal environment, showing different surface textures through direct reflected and scattered light, and showing diffraction through transparent materials and other such physical phenomena. For example, some surfaces may reflect light directly at the same angle as the light's angle of incidence, like a mirror, which is called specular reflection. Other surfaces may reflect light more diffusely, scattering the reflected light in a variety of different directions centered on the surface's normal, called diffuse reflection. Many objects in real life have both specular and diffuse components, and are modeled as such when ray tracing a rendered image to provide realistic lighting effects.

[0020] However, the number of rays traced in large or complex images to produce realistic images with acceptably low levels of illumination noise is very high, and often causes the rendering process for high-quality sequential images (such as rendering a movie) to take days or weeks instead of the one or two hour runtime length of rendering a movie. While such extended rendering and ray tracing times can produce high-quality rendered movies that are more pleasing to watch, the same approach cannot be effectively used for gaming, augmented reality or virtual reality, or other applications where it is important to render images in real time with acceptable image quality.

[0021] Certain techniques can be used to reduce ray tracing calculations for an image frame, such as tracing rays from the camera to the light source rather than from the light source to the camera, thereby ensuring that only rays that actually reach the camera (or viewer) position will be traced. Another solution for reducing ray tracing calculation time is to apply ray tracing to such images using far fewer traced rays, again resulting in far fewer ray tracing calculations per rendered image frame. But tracing fewer rays per pixel results in less accurate and noisier illumination, particularly including more lighting noise in areas that receive relatively few rays. Monte Carlo (randomized) selection or sampling of the rays to be traced can improve the approximation of real-life ray tracing, but hundreds or thousands of ray tracing samples per pixel are still expected to produce photo-realistic images with acceptably low noise using traditional techniques.

[0022] Thus, some examples presented herein provide various methods and apparatus for reducing the level of apparent ray-traced light noise in an image sequence, such as an image sequence rendered in real time or an image sequence subject to other computational or power constraints. In one such example, machine learning, such as a neural network, may be employed to predict or estimate the effect imparted to a rendered image to reduce ray-traced illumination noise.

[0023] In one example, the effect imparted includes sampling a texture using a selective downsampled version of a rendered image frame, wherein the downsampled version may be selected using a trained neural network. In a more detailed example, the downsampled version may be created using a kernel prediction network that denoises the image as it is downsampled, and in another example, at a pyramid level of the downsampled level selected by the trained neural network, pixel values ​​in the selected downsampled version are selected or interpolated between pixel values ​​of two downsampled images, such as by using trilinear filtering hardware of a graphics processing unit (GPU).

[0024] In another example, an effect imparted to a rendered image uses a first accumulated history buffer for past rendering instances prior to at least some filtering and a second accumulated history buffer for past rendering instances after filtering (such as by blurring, denoising, etc.). For example, the buffers may accumulate pixel values ​​used to illuminate the same point on an image in past image rendering instances. The pixel values ​​in the first accumulated history buffer and the second accumulated history buffer are combined with pixel values ​​from a current image frame to produce a displayed image frame, and a trained neural network may be used to select a proportion of the pixel values ​​in the first accumulated history buffer and the second accumulated history buffer to produce a displayed image frame with acceptable detail and noise levels. Because the first unfiltered image buffer accumulates unfiltered pixel data, it may initially have more noise than the second filtered (e.g., blurred or denoised) image buffer, but over time may accumulate more high-frequency detail than the second filtered image buffer and may be used in greater proportion when rendering the displayed image frame.

[0025] In another example, processes such as invalidating the accumulated history buffer when a pixel becomes newly unoccluded (or visible to the viewer) in the current image frame may be managed separately for diffuse illumination and specular illumination. This process may be referred to as a "correction," and in a more specific example may be performed using motion vectors applied to the illumination components of the previous image frame to reproject or provide a warp history aligned with the current image frame. This reprojection history or warp history may be combined with the current image frame separately for the specular illumination component and the diffuse illumination component, such as by using illumination coefficients derived by machine learning (such as by a trained neural network). In another example, separate illumination component history buffers may be maintained for the specular illumination component and the diffuse illumination component, such that each illumination correction may combine the reprojection history or warp history for each illumination component with the same illumination component of the current image frame.

[0026] Figure 1 is a block diagram of a ray tracing pipeline consistent with an example embodiment. Figure 1 The generation of image frame N-1 of a previous rendering instance (including image frames from trilinear filter illumination instance 102) is generally shown on the left. Figure 1 The central portion shows components that generate an image frame FRAME N for the current rendering instance. Figure 1 The right portion shows the output of the current rendering instance as the reprojection history block 108, which will be processed in a subsequent (eg, future) rendering instance to produce an image frame FRAME N+1.

[0027] The rendering module 106 may provide an image frame with reference to the current rendering instance N by rendering and ray tracing, for example, using a jitter offset and other such inputs to generate a current image frame FRAME N. The rendered image frame at instance N may be provided to a reprojection history (resample) block 108 along with a motion vector (for an object or texture in image frame FRAME N relative to rendering instance N-1). The reprojection history block 108 may receive both the image frame FRAME N with reference to the current rendering instance from the rendering block 106 and the trilinear filtered image output at 102 with reference to the previous rendering instance. The reprojection history block 108 may reproject or warp the trilinear filtered image output onto the current rendering image, such as by using a rendered or estimated motion vector. In one specific implementation, such an estimated motion vector may be generated by the rendering block 106. In other specific implementations, such an estimated motion vector may be generated by one or more other sources. This may be used to align the pixel values ​​of the output image frame from the previous rendering instance with the pixel values ​​of the image frame rendered in the current rendering instance. Based on the warping pattern or pixel alignment pattern of past image frames, the correction history block 110 may mitigate the effect of image content that was not visible in the previous image frame FRAME N-1 being unoccluded or made visible in the current image frame FRAME N. In another example, correction or history invalidation may also be performed for changes in view-dependent illumination, such as when the current viewing direction of the camera has changed sufficiently from the viewpoint of the previous image to invalidate specular highlights and / or reflection details between image frames.

[0028] In a specific example, a predicted mixing coefficient theta, such as from a trained neural network 112, may be received in the correction history block 110 to predict or estimate the extent to which the warp history should be added to the current slightly noisy rendered frame. This may be achieved according to the following expression [1]:

[0029] rectified(x,y)=reset(x,y)×(1-theta(x,y))+warped(x,y)×theta(x,y), [1]

[0030] in:

[0031] rectified(x, y) is the pixel value at pixel position x, y in the output image frame;

[0032] reset(x, y) is the pixel value at pixel position x, y in the current noise-rendered image frame;

[0033] theta(x, y) is the blending coefficient or per-pixel correction value in the range [0, 1] received from the trained neural network 112; and

[0034] warped(x, y) is the pixel value at pixel location x, y in the warp history (eg, received from the reprojection history block 108).

[0035] As can be observed from expression [1], the resulting corrected output may include a combination of separately accumulated specular illumination histories and diffuse illumination histories as well as some "reset" or noisy pixels from the current rendered image (such as at locations where pixels become unoccluded in image frame N). For visually perfect noise-free ray tracing rendering, each frame may accumulate samples for at least thousands of cast rays per pixel. However, casting such a large number of rays per frame may be computationally prohibitive and may therefore hinder use in real-time applications such as gaming and virtual reality or augmented reality. Therefore, in some examples, accumulated history block 114 may be used to combine the accumulated history of rendered pixel values ​​over time with the current noisy rendered image frame, thereby selectively combining the re-projected correction history received from correction history 110 with the noisy current frame received from rendering and ray tracing unit 106. The accumulation process at the accumulation history block 114 may use the parameter alpha received from the trained neural network 112 to determine the proportion of the current noisy image frame to be mixed with the reprojected, corrected history, and in another example may use separate accumulated histories and alpha values ​​for corresponding separate illumination components (such as specular illumination components and diffuse illumination components).

[0036] In another example, a double buffer may be maintained for each illumination component in the accumulation history block 114: an unbiased buffer that may store a history of past image frames lacking at least one step in filtering or processing; and a biased buffer that may store a history of past image frames to which such filtering or processing has been applied. The filtering or processing applied to the stored frames of the biased buffer may include blurring, denoising, kernel prediction filtering, or any other such processing in various examples. In a more specific example, the filtering or processing applied to the stored frames of the biased buffer may include filtering or processing that removes noise or high frequency content from the stored image, or may impart a blurring or softening effect to past image frames stored in the biased buffer. The unbiased buffer may accumulate a history of noisy frames received from the rendering and ray tracing unit 106, and the biased buffer may accumulate a history of image frames provided to a display device or a history of image frames at any other filtering stage between being filtered after rendering and being displayed (such as reprojected, corrected, and combined images).

[0037] In some examples, the unbiased buffer can accumulate samples by combining new image frame samples with the accumulated history and / or averaging the new image frame samples with the accumulated history (such as based on the number of previously accumulated samples stored in the unbiased buffer). By tracking the number of unbiased samples for pixel positions represented in the accumulated unbiased buffer, the degree to which the unbiased buffer is blended into the current image frame can be increased. Once a sufficient number of past image frame pixel values ​​have been accumulated for the unbiased buffer pixel values ​​to represent more high frequency spatial content and less noise than in the case of relatively few accumulated past unbiased or unfiltered frames, the unbiased buffer can be more strongly combined. Such accumulation in the unbiased accumulation buffer can be implemented according to the following expression [2]:

[0038]

[0039] in:

[0040] Unbiased(x,y) is the pixel value at pixel position x,y in the calculated unbiased cumulative history;

[0041] Unbiased(x,y) t-1 is the pixel value at pixel position x, y in the previous unbiased cumulative history;

[0042] count is the number of samples or image frames represented in the previous unbiased cumulative history; and

[0043] newSample(x,y) is the pixel value at position x,y in the new image frame that is added to the accumulated history.

[0044] For each successive accumulation of pixel values ​​in a new unbiased sample or image frame, the value of count may be incremented by 1. The resulting Unbiased(x,y) values ​​may be blended and / or combined into a buffer Biased(x,y) for the corresponding pixel location (x,y) using a method such as an exponential moving average formula, wherein a parameter alpha(x,y) (provided by the trained neural network 112) is applied to determine the extent to which the unbiased accumulation history in Unbiased(x,y) will be blended into and / or combined with the biased accumulation history Biased(x,y). In another example, blending the unbiased accumulation history Unbiased(x,y) into the biased accumulation history Biased(x,y) may be performed according to the following expression [3]:

[0045] Biased(x,y)=(Biased(x,y) t-1 *(1-alpha(x,y)))+(Unbiased(x,y)*alpha(x,y)) [3]

[0046] in:

[0047] Biased(x,y) is the pixel value at pixel position x,y in the calculated biased cumulative history;

[0048] Biased(x,y) t-1 is the pixel value at pixel position x, y in the biased cumulative history calculated for the past image frame instance t-1;

[0049] Unbiased(x,y) is the pixel value at pixel location x,y in the unbiased cumulative history; and

[0050] alpha(x,y) is the per-pixel blending value in the range [0,1] predicted by the trained neural network 112.

[0051] In one specific example, the value of Biased(x,y) may be calculated at the accumulated history block 114 according to expression [3]. As alpha(x,y) becomes larger, the ratio of Unbiased(x,y) relative to Biased(x,y) may grow, and the high frequency details accumulated in Unbiased(x,y) may gradually replace and / or incorporate into the blurred, denoised, or otherwise filtered accumulated history in Biased(x,y). This process may be repeated separately for each illumination component (such as for a specular illumination component and a diffuse illumination component), and the trained neural network 112 may provide separate parameters alpha(x,y) for the corresponding different illumination components.

[0052] The output from the accumulated history block 114 may be provided to a progressive downsampling (denoising) block 116, which may generate an image pyramid (or mip chain) of a progressively downsampled image. For example, such a progressively downsampled image may be aliased to a texture, such as by using a graphics processing unit (GPU) trilinear filtering hardware adapted for trilinear interpolation. The progressive downsampling at block 116 to generate the image pyramid may be performed separately for each illumination component, such as a specular illumination component and a diffuse illumination component, and in another example, kernel coefficients may be applied separately to the illumination components. Such kernel coefficients may be determined by one or more kernel prediction networks, such as may include a portion of a trained neural network 112, for each illumination component as shown at the connection between the neural network 112 and the progressive downsampling block 116. Each downsampling level may be derived from the previous downsampling level, such as by downsampling a 2048x2048 image to 1024x1024, a 1024x1024 image to 512x512, a 512x512 image to 256x256, and a 256x256 image to 128x128. In one specific example, progressive downsampling may be performed according to the following expression [4]:

[0053]

[0054] in:

[0055] I D+1 (x,y) is the pixel value of pixel position (x,y) at downsampled level D+1 in the image pyramid (computed by downsampling from level D);

[0056] I D (i, j) is the pixel value of pixel position (i, j) at downsampling level D in the image pyramid; and

[0057] k(x,y,i,j) is the kernel coefficient applied to the pixel value at pixel location (i,j) to produce the output pixel value at pixel location (x,y).

[0058] The downsampling according to expression [4] may be repeated to generate a desired number N of pyramid levels, where N may be an optional or adjustable parameter of the downsampled image pyramid. Figure 3 , and will be discussed in more detail with respect to this image. Because each successive downsampling level of the pyramid has a lower resolution than the previous downsampling level, but includes pixels averaged over a larger area, each successive level of the pyramid has less noise than the previous level, but also has a lower resolution and less high frequency detail or sharpness.

[0059] The image pyramid established at the progressive downsampling block 116 may be employed by the trilinear filter illumination block 118, which, in a more detailed example, may use the predicted x, y, and z parameters from the trained neural network 112 to select image locations within the downsampling pyramid for sampling. In a more detailed example, to filter a pixel at a given x and y location on the rendered image being processed, the trilinear filter block may receive an x ​​offset and a y offset that shift the location of the sample, such as to make the sample edge-aware, and a z parameter that may indicate the level of the progressive downsampling pyramid from which to sample. In some examples, the x, y, and z coordinates may be non-integer, such as where interpolation between pixel locations or between levels of the downsampling pyramid is required.

[0060] In some examples, the trilinear filtering in block 118 may be performed using a GPU hardware implementation of trilinear interpolation typically found in modern high-end GPUs, such as via a GLSL textureLod function. Such a GLSL textureLod function may, for example, take three parameters: a pointer to an image pyramid; a normalized non-integer texture coordinate (x, y) sampled within the pyramid; and a non-integer depth or level of detail (z) coordinate corresponding to the pyramid level at which the texture may be sampled. In some examples, for each lighting component, the degree of trilinear filtering may be controlled by the prediction or estimation of two separate parameters uv_offset (x and y) and level of detail (z) by the trained neural network 112. In a more specific example, uv_offset may form a bias to the sampled texture coordinate (a second argument), and the depth or level of detail directly predicts or estimates the sampled image pyramid level (a third argument). The trained neural network can therefore control the degree of blur of the resulting image via the level of detail or z-parameter, where larger levels of detail correspond to sampling from higher levels of the pyramid, which have been successively downsampled more times than previous lower levels and are therefore likely to have more blur and less ray tracing illumination noise.

[0061] While sampling from higher levels of the pyramid (larger levels of detail or z) can result in ray tracing illumination with less noise, sampling from higher levels results in the unfortunate disadvantage of losing high-frequency edge information due to the lower image resolution of the higher pyramid levels. To compensate for this, the trained neural network 112 can predict or estimate uv_offset or x and y parameters that can offset the horizontal and vertical sampling coordinate locations to allow edge-aware trilinear sampling, much like a bilateral filter. When the trilinear filter samples from the downsampled pyramid using the parameters provided by the trained neural network, the trilinear filter can sample the downsampled pyramid from any spatial location within the effective receptive field to preserve high-frequency details.

[0062] The resulting filtered specular image samples and diffuse image samples, which may be provided as output from the trilinear filter 118, may be resolved or combined at block 120 to form an output image, which may then be post-processed as desired, such as to render the image at a desired display resolution prior to display at 122. The filtered specular image samples and diffuse image samples from the trilinear filter 118 may also be stored or provided for processing the next image frame, as reflected by the reprojection history block 104 (corresponding to the reprojection history block 108 for the current frame, which similarly receives the previous frame trilinear filtering results for specular and diffuse illumination from block 102). Figure 1 Examples of may provide significant advantages over current prior art methods of denoising rendered images in a sequence, such as greatly reducing latency and power consumption while providing similar or improved image appearance due to reduced ray tracing illumination noise.

[0063] Figure 2 Ray tracing of rendered objects is shown consistent with an example embodiment. Ray tracing in this example may be performed by tracing rays that see light source 204 from the perspective of camera 202, so that areas illuminated by light source 204 that are not visible to the camera are not ray traced. This avoids wasting computing resources on ray tracing light from light source 204 that will not be seen from the perspective of a viewer (i.e., camera) of the rendered and illuminated frame, thereby speeding up the ray tracing process.

[0064] Figure 2The dashed lines in the simplified example of 200 show how individual light rays are traced from the camera toward the object 206 and shadow 208, reflecting off one or more surfaces toward the light source 204. The reflections in this example include two different lighting components: specular reflections, which are relatively narrow in how well their reflection angle matches the angle of incidence, and diffuse reflections, where the angle of reflection can vary significantly more from the angle of incidence due to variations in surface texture. An example of a probability function for diffuse reflection is shown at 210, which shows that diffuse light hitting the surface of object 206 can be reflected at various angles, with the probability roughly centered around the normal to the surface of the object. The specular reflection shown in the example at 212 differs in that the light is reflected at a narrower range of angles, where the range of angles is roughly centered around the angle of incidence of the light to the surface.

[0065] Specular reflections from some surfaces may be limited to very narrow reflection angles, such as for metallic, mirrored, wet, etc. surfaces, sometimes resulting in being able to see a clear reflected image off of the object 206. Diffuse reflections are more of a property of dielectric object materials, such as cloth, skin, or other such surfaces, and tend to scatter and absorb light rather than reflect clear images of other objects or light sources. Many objects, such as wood or stone, have both specular and diffuse lighting components in their reflections, and if both the specular and diffuse lighting components are ray traced to and from such objects, the objects will appear more realistic in the rendered image.

[0066] Ray-traced light reflected from a surface can use a parameter called albedo to represent the percentage of light that hits the surface that is reflected rather than absorbed. A black object that absorbs all incident light has an albedo of 0, while a white object that reflects all light has an albedo of 1. Albedo can be tracked separately for specular and diffuse lighting, but because non-metallic materials typically have very low specular albedo, specular albedo is often ignored for non-metallic objects. Therefore, reflected light from a non-metallic surface can be characterized as specular reflection plus albedo multiplied by diffuse reflection, where the albedo controls the amount of diffuse light component that is rendered as reflection rather than absorption. Metallic presentation is a special case that can have a specular albedo but a very low diffuse response, and although it is an example of a material that renders differently, it typically has relatively low illumination noise and so can often be ignored in ray tracing denoising.

[0067] exist Figure 2In a simplified example of , rays traced from camera 202 to light source 204 can be captured by calculating where the rays cross image planes of an image grid 214, which forms a ray-traced or illuminated rendered image that is processed and displayed. The image grid is shown here as a two-dimensional 12x12 array of pixels, with a different ray being traced from camera 202 through each pixel to light source 204. In some examples, the rays selected for tracing can be random so that they have a roughly equal chance of passing through any point on the grid. Rays that do not pass through grid 214, which may hit an object and reach light source 204, will not be displayed in the rendered image, and therefore are not traced. Although Figure 2 The simplified example of shows a 12x12 grid of pixels, but actual implementations may include significantly more pixels, such as to match the display resolution of a user device such as a computer, smart phone, or television.

[0068] In a simplified example, the ray tracing process may shoot at least one ray through each pixel or square in the image grid 214 and observe whether the ray hits any object primitive (typically a triangle of the rendered object grid) such as object 206 in the scene. If the ray hits an object, the distance along the ray from the camera to the object is calculated, and the color data of the object contributes to the final color of the pixel. The ray may bounce off the primitive specularly and / or diffusely and acquire color and illumination from other objects, and may acquire light and color from light sources such as light source 204. In applications with many objects in the rendered scene (such as video games), the number of rays cast per pixel may be expected to be high, such as hundreds or thousands of rays per pixel per rendered image frame. If computing resources, power, frame rate, and other such considerations may limit the number of rays cast per pixel to fewer rays, such as single digits or tens of rays per pixel, the resulting rendered and illuminated image may have visually perceptible illumination noise, and therefore may benefit from ray tracing denoising methods and systems, such as those described herein.

[0069] Figure 3 A three-level downsampling pyramid consistent with an example embodiment is shown. Here, pixel grid 314 shows a 12x12 pixel image or texture from Figure 2 214, from which the other levels of the downsampling pyramid are derived. In this example, each successive level of the downsampling pyramid has half the resolution of the previous level in each dimension, advancing to a 6x6 image at 316 and a 3x3 image at 318. In actual applications, the number of pixels of base image 314 may be much larger, such as to match the display resolution of a computerized device. Figure 3 The downsampling pyramid can be obtained by Figure 1 A simplified version of the image pyramid constructed by the progressive downsampling block 116 is shown in FIG. 1 , and some examples are discussed in more detail with respect to that figure.

[0070] As shown here, the original image 314 may be progressively downsampled to produce an image with lower resolution and lower noise, which lower resolution and lower noise image may be aliased into a rendered image object, for example using trilinear filtering hardware of a GPU typically used for trilinear interpolation. In some examples, the downsampling is performed using a denoising filter, such as a filter generated by a neural network-based kernel prediction network, which in some examples includes a convolutional neural network that generates a kernel of scalar weights or coefficients applied to neighboring pixels to calculate each denoised pixel of the image. In a more detailed example, the kernel prediction network may predict a matrix of kernel coefficients for each pixel, the matrix including coefficients or weights for neighboring pixels, which may be used to determine a denoised pixel value by calculating the denoised pixel value from the original value of the pixel and its neighboring pixels. The filter coefficients generated by the kernel prediction network may be used to construct Figure 3 The successive layers of the pyramid may be or include neural networks such as Figure 1 A portion of a neural network 112.

[0071] exist Figure 3 In an example of , each level of the downsampling pyramid may be derived from the lowest level, such as by using filter coefficients generated by a kernel prediction network, other filtering, averaging, or another suitable method, and may produce an image that is lower in both resolution and noise than the previous lower pyramid level image from which the level was derived. The progressive downsampling may be performed separately for each of the specular illumination component and the diffuse illumination component, which may be combined in another example to generate a displayed image.

[0072] although Figure 3 The number of pyramid levels in the example of is three, but the process can be repeatedly performed to generate any desired number N of pyramid levels, where N can be an optional or adjustable parameter of the downsampled image pyramid. In other examples, the number of levels N can be determined by neural network prediction or estimation, by the desired output resolution of the rendered image, by the expected noise level of the rendered image, and / or by other such parameters.

[0073] The GPU’s trilinear filter lighting hardware can use the predicted x, y, and z parameters from the neural network to adopt Figure 3to select an image texture location within the pyramid to sample. In a more detailed example, to filter a pixel at a given x and y location on a rendered image being processed, the trilinear filter block may receive an x ​​offset and a y offset that shift the location of the sample, such as to make the sample edge-aware, and a z parameter that indicates the level of a progressive downsampling pyramid from which to sample. The x, y, and z coordinates may be non-integers, such as where interpolation between pixel locations or downsampling pyramid levels is desired. Specifically, the z coordinate may correspond to a level of detail (lod), which may be an integer corresponding to a downsampling pyramid level, or may be a non-integer corresponding to an interpolated position between downsampling pyramid levels to control the degree of blur in the sampled texture for each filtered pixel.

[0074] Once the desired x and y offsets and level of detail z have been specified for each pixel and appropriate texture filtering has been applied, the filtered images for specular and diffuse lighting can be combined and / or post-filtered, such as to change the resolution to the desired display resolution for display. In another example, the combined image is stored or buffered for use in rendering a ray traced texture (e.g., Figure 1 Filtering is performed as shown at N+1 on the right side of .

[0075] Figure 4A and Figure 4B is a block diagram of a system for implementing ray tracing denoising in accordance with an example embodiment. The diagram again generally illustrates a rendering ( Figure 4A ), reprojecting or warping the buffer history, and accumulating images into the history buffer ( Figure 4A ), and downsampling to generate a six-level downsampling pyramid ( Figure 4B ), which are combined and filtered to produce an output frame that is provided to the display. These components roughly correspond to Figure 1 , where the neural network shown at 402 (an example of a detailed level neural network implementation for training) corresponds to Figure 1 112 of the trained neural network. Although six downsampling pyramid levels are used here, other numbers of pyramid or mip chain levels may be used in other examples, and the number may be an adjustable parameter such as depending on the rendered image resolution and / or other parameters. In another example, the six downsampling pyramid levels of the example of FIG. 4 may provide a wide range of detail levels for mapping a denoised texture onto a rendered image, such that a full 4k image at 3840x2160 is successively downsampled to 1920x1080, 960x540, 480x270, 240x135, and 120x67 using a kernel prediction filter.

[0076] In some examples, the downsampling pyramid level selected may be based at least in part on the number of temporal image frame samples accumulated in the history buffer. In another example, a parameter alpha for accumulating image history in the buffer may be determined or constrained based on the number of temporal image frame samples accumulated. In a more detailed example, a sample count constraint may be imposed on alpha and the pyramid level z or level of detail (lod) to at least partially accelerate training convergence and ensure the desired behavior of the trained neural network or LODNet 402. Alpha represents a linear mixture between an unbiased buffer and a biased buffer and may be predicted by the trained neural network 112, which may be constrained using a formula such as according to the following expression [5] to ensure that at least MinAlpha percentage of the first unbiased buffer is adopted when the sample count is large enough:

[0077]

[0078] in:

[0079] OutAlpha is the output alpha level selected for sampling from the downsampling pyramid

[0080] MinAlpha is the minimum percentage of the unbiased buffer that the algorithm is allowed to take once count reaches the threshold;

[0081] MaxCount is the threshold number of temporal image samples at which MinAlpha becomes the minimum allowable alpha; and

[0082] count is the number of temporal image samples accumulated in the history buffer.

[0083] Therefore, MinAlpha becomes the minimum allowable alpha only when count reaches MaxCount, and the minimum allowable alpha gradually increases as count increases toward MaxCount. The MinAlpha and MaxCount values ​​are adjustable to ensure that after a sufficient count of temporal image samples have been accumulated, the network exploits slower converging high spatial frequency lighting features, such as high frequency soft shadows, that are most likely to converge only inside the unbiased buffer. In another example, the detail level z or pyramid level is scaled according to the count according to the following expressions [6], [7], and [8]:

[0084]

[0085]

[0086] LodOut = LodIn × LodScale + LodBias [8]

[0087] in:

[0088] PyramidMin is a constraint that defines the minimum level of downsampling in the downsampled image pyramid;

[0089] PyramidMax is a constraint that defines the maximum level of downsampling of the downsampled image pyramid; and

[0090] LodMin and LodMax are adjustable values ​​that define the minimum and maximum Lod (level of detail) values ​​when the sample count is below the adjustable parameter MaxCount.

[0091] One purpose of the constraints according to expressions [6], [7] and / or [8] is to encourage the neural network to use the full breadth of the progressive downsampling pyramid over time, while preventing overblurring once a sufficient number of previous image samples have been accumulated in the unbiased buffer. Thus, the maximum and minimum amounts of blur can be constrained as a function of the number of samples accumulated over time in the above formulas.

[0092] Figure 5 is a flow chart of a process for correcting reprojected previous image frame data consistent with an example embodiment. Reprojection may include warping or mapping pixels of a previous image or texture onto a current rendered image so that the previous image data or history may be used to determine current pixel texture values ​​for a current instance or current rendered image. Correction may include selectively blending previous image history of portions of the rendered image that are unoccluded or where view-dependent lighting effects (such as specular highlights and reflections) change due to a changed view angle of a camera. For example, such correction may be performed by correction history block 110 and / or according to expression [1].

[0093] At 502, a rendering frame and one or more previous image frames may be received. A plurality of illumination components may be generated for the rendered image frame at 504, which may include a specular illumination component and a diffuse illumination component in another example. For example, the illumination components in some examples may be generated via ray tracing, such as by the rendering module 106. At 506, one or more motion vectors may be applied to the illumination components from the previous image frame, which may be used to warp or map the previous image frame texture data to corresponding pixel locations in the current rendered image instance. In another example, the motion vectors may include different motion vectors for different illumination components, such as diffuse and specular.

[0094] At 508, first illumination component coefficients and second illumination component coefficients may be generated, such as using a trained neural network (e.g., neural network 112). Such illumination component coefficients may determine how portions of one or more previously warped image frames will be blended with an image frame rendered in a current rendering instance to account for effects such as unocclusion or changes in illumination components. At 510, a first set of illumination component coefficients may be applied to combine pixels of a first illumination component (such as specular illumination) in a currently rendered image frame with pixels of the first illumination component in the one or more previously warped image frames. At 512, a set of second illumination component coefficients may be applied to combine pixels of a second illumination component (such as diffuse illumination) in a currently rendered image frame with pixels of the second illumination component in one or more previously warped image frames. For example, operations at blocks 510 and 512 may apply illumination component coefficients according to expression [1]. In a more detailed example, illumination component coefficients may be determined per pixel, thereby determining, on a pixel-by-pixel basis, the extent to which one or more warped historical rendered image textures are included in a combined image.

[0095] An output image frame including the first illumination component and the second illumination component may be generated at 514, wherein the historical warped rendered image is combined with the current rendered image using a term referred to as albedo (used to determine at least a portion of the diffuse illumination that is reflected by the texture rather than absorbed). In various embodiments, the albedo may also be determined on a per-pixel, per-texture or object basis, or based on other such rendered image characteristics.

[0096] Figure 5 The flowchart shows how the use of per-lighting correction can improve the noise level of a ray-traced rendered image. The process uses a low percentage of relatively noisy newly rendered and newly illuminated image textures to support past warp texture data to provide reduced noise, except in cases where newly unoccluded or newly illuminated pixels in image objects invalidate such past warp texture data and support the newly rendered image texture.

[0097] Figure 6 1 is a flow chart of a process for using a double buffer for accumulated rendered image history consistent with an example embodiment. By using different buffers to accumulate pixel values ​​for filtered and unfiltered versions of illumination components of images in past rendering instances, the amount of ray tracing or illumination noise in the combined image can be reduced while improving the preservation of high frequency details. In another example, the blended values ​​for each illumination component used to blend the filtered and unfiltered image histories to produce a combined rendered image can be determined by a trained neural network or other such machine learning system.

[0098] At 602, a rendered image frame may be illuminated using ray tracing, such as by rendering a texture on the image and using one or more lighting components (such as a specular lighting component and a diffuse lighting component) to render ray traced light within the image. At 604, a first buffer may accumulate pixel values ​​for the lighting component for past rendering instances, thereby storing a rendered image with its ray traced lighting component before one or more additional image processing stages such as correction, spatial filtering, and / or denoising occur. In a more detailed example, separate first buffers may be maintained for the one or more lighting components, such as for diffuse lighting and specular lighting.

[0099] At 606, the second buffer may similarly accumulate pixel values ​​for the lighting component for past rendering instances, thereby storing a rendered image with its ray traced lighting component after one or more additional image processing stages such as correction, spatial filtering, and / or denoising have occurred. The second buffer may thus maintain a denoised, softened, blurred, or filtered version of the rendered and ray traced image stored in the first buffer, and in some examples may be referred to as a biased buffer, while the first buffer may be referred to as an unbiased buffer. Although the image stored in the second buffer may have less ray traced illumination noise and be smoother in appearance than the image stored in the first buffer, the image stored in the first buffer will accumulate greater high frequency detail or high resolution detail over time and have less noise because more rays per pixel are accumulated in the buffer.

[0100] At 608, a blend value, such as a value alpha, is determined for each illumination component using a trained neural network, such as trained neural network 112, which blend value may be used to combine pixel values ​​from the first buffer and the second buffer to create an image frame. The blend value may generally change from being biased toward including more filtered image pixel values ​​from the second buffer when the accumulated history of the pixel is low (such as when the pixel was recently unoccluded during correction) to being biased toward including more unfiltered image pixel values ​​from the first buffer when the accumulated history of the pixel increases to a level at which ray tracing illumination noise is reduced. These blend values ​​may thus be employed at 610 to combine pixel values ​​in the first buffer with pixel values ​​in the second buffer to provide pixel values ​​for one or more illumination components in the accumulated image frame, such as according to expression [3]. By favoring pixel values ​​from the second buffer when the buffer history of the pixel is low, an acceptable noise level may be maintained, and by increasing the weight of pixel values ​​from the first buffer when the buffer history of the pixel has accumulated a large number of samples, high frequency details from the first buffer may be introduced.

[0101] Figure 7Flowchart of a method for denoising a rendered image using progressively downsampled texture maps and trilinear filtering consistent with an example embodiment. At 702, a rendered image frame may be received, which may have been previously processed, such as by using a history buffer, image correction and combination, and other methods such as those described in the examples herein. A desired number N of consecutive spatial downsampled images may be determined at 704, which may be determined, for example, using a function or method according to expression [4] in another example. In one embodiment, the desired number N of consecutive spatial downsampled images may be based at least in part on the overall resolution of the image, the amount of noise in the rendered image, the hardware capabilities of a GPU for performing trilinear filtering using the downsampled image, and other such characteristics of the rendered image and / or image processing pipeline.

[0102] The coefficients generated by the kernel prediction network can be applied to the rendered input image received from 702 to generate a first spatial downsampled image having a lower resolution but lower noise than the received rendered input image. In this example, the reduction in noise is due in part to averaging pixels to a lower resolution version of the original image and in part to downsampling the rendered image using the coefficients generated by the kernel prediction filter. The new downsampled image can then be used to create the next spatial downsampled image, and such a process is repeated until a desired number of consecutive spatial downsampled images N are generated.

[0103] At 708, the neural network selects one or more versions of the rendered image from the rendered image initially received at 702 and the successively rendered downsampled images (such as by specifying a level z in a downsampled image pyramid). For example, such level z may be selected according to expressions [6], [7], and / or [8]. In some examples, the selected level z may be an integer specifying which of the N images to use alone, while in other examples, the level z is a non-integer in the range of 1 to N, such that a value that is not an integer specifies a level or degree of interpolation between two consecutive or adjacent images in the image pyramid.

[0104] In some examples, as shown at 710, in addition to the z value or level of detail for each pixel in the rendered image, an x, y spatial offset for sampling the texture can be provided by the trained neural network, and the spatial offset can represent an offset for sampling the texture from the pixel location of the original image. Such an offset can compensate for various motion effects, improve the perception of edges, and perform other such compensations to improve the visual accuracy of the selected texture. The offset coordinates in various examples can also be integer values, or can be non-integer values ​​that can result in interpolation between textures at two different pixel locations.

[0105] Then, at 712, the texture may be sampled from the x, y offset position and z or pyramid detail level, interpolating in the case where the x, y, or z parameters are non-integer. In some examples, the specular lighting component and the diffuse lighting component may be filtered separately using separate rendered image pyramids, and may be combined at 514, for example, after the above steps in the flowchart have been performed for each of the specular lighting component and the diffuse lighting component. In another example, the sample texture is combined at 714 with sampled textures from one or more previous image frames, and may be further processed (such as adjusting resolution, gamma, or other parameters) as needed for display to ensure that the rendered image looks as expected on the display hardware.

[0106] Selecting pixel textures from a pyramid of continuously spatially downsampled images can help reduce ray tracing rendering noise in each pixel, especially when significant noise remains in at least some pixels after previous processing steps (such as correcting for unoccluded pixels in the current rendered image). A trained neural network can be employed to determine the pyramid depth or level of detail and x,y offset for texture sampling to improve the appearance of textures in a rendered image, for example by reducing apparent ray tracing noise.

[0107] In some examples, a neural network may include a graph including nodes for modeling neurons in the brain. In this context, "neural network" means an architecture of a processing device defined and / or represented by a graph, the graph including: nodes for representing neurons that process input signals to generate output signals; and edges that connect these nodes to represent input and / or output signal paths between and / or among neurons represented by the graph. In a specific implementation, a neural network may include a biological neural network composed of real biological neurons or an artificial neural network composed of artificial neurons, for example, to solve artificial intelligence (AI) problems. In a specific implementation, such an artificial neural network may be implemented by one or more computing devices, such as a computing device including a central processing unit (CPU), a graphics processing unit (GPU), a digital signal processing (DSP) unit, and / or a neural processing unit (NPU) (to provide only a few examples). In a specific implementation, the neural network weights associated with the edges for representing the input and / or output paths may reflect the gain to be applied and / or whether the associated connection between the connecting nodes is an excitatory connection (e.g., a weight with a positive value) or an inhibitory connection (e.g., a weight with a negative value). In an example implementation, a neuron may apply neural network weights to input signals and sum the weighted input signals to generate a linear combination.

[0108] In one example embodiment, an edge in a neural network connecting nodes may model a synapse that is capable of transmitting a signal (e.g., represented by a real value) between neurons. In response to receiving such a signal, the node / neuron may perform some calculations to generate an output signal (e.g., to be provided to another node in the neural network connected by the edge). Such an output signal may be based at least in part on one or more weights and / or numerical coefficients associated with the node and / or edge providing the output signal. For example, such a weight may increase or decrease the strength of the output signal. In a particular implementation, such weights and / or numerical coefficients may be adjusted and / or updated as the machine learning process proceeds. In one implementation, if the strength of the output signal does not exceed a threshold, the transmission of the output signal from a node in the neural network may be prohibited.

[0109] Figure 8 800 is a schematic diagram of a neural network 800 formed in "layers," where an initial layer is formed by nodes 802 and a final layer is formed by nodes 806. All or part of the features of neural network 800 may be implemented in various embodiments of the systems described herein. Neural network 800 may include one or more intermediate layers, illustrated here by an intermediate layer of node 804. The edge shown between nodes 802 and 804 illustrates the flow of signals from the initial layer to the intermediate layer. Similarly, the edge shown between nodes 804 and 806 illustrates the flow of signals from the intermediate layer to the final layer. Although Figure 8 Each node in a layer is shown to be connected to each node in the previous layer or subsequent layer to which the layer is connected, that is, the nodes are fully connected, but other neural networks will not be fully connected, but will adopt different node connection structures. Although neural network 800 shows a single intermediate layer formed by nodes 804, it should be understood that other specific implementations of neural networks may include multiple intermediate layers formed between the initial layer and the final layer.

[0110] According to an embodiment, nodes 802, 804, and / or 806 may process input signals (e.g., received on one or more incoming edges) to provide output signals (e.g., on one or more outgoing edges) according to an activation function. As referred to herein, an "activation function" means a set of one or more operations associated with a node of a neural network to map one or more input signals to one or more output signals. In a particular implementation, such an activation function may be defined based at least in part on weights associated with the nodes of the neural network. The operations of the activation function that map one or more input signals to one or more output signals may include, for example, identity, binary step, logic (e.g., sigmoid and / or soft step), hyperbolic tangent, corrected linear unit, Gaussian error linear unit, Softplus, exponential linear unit, scaled exponential linear unit, leakage corrected linear unit, parameter corrected linear unit, sigmoid linear unit, Swish, Mish, Gaussian, and / or grown cosine unit operations. However, it should be understood that these are merely examples of operations that may be applied in an activation function to map an input signal of a node to an output signal, and the claimed subject matter is not limited in this regard.

[0111] In addition, as referred to herein, "activation input value" means a value provided as an input parameter and / or signal to an activation function defined and / or represented by a node in a neural network. In this context, as referred to herein, "activation output value" means an output value provided by an activation function defined and / or represented by a node in a neural network. In a specific implementation, the activation output value may be calculated and / or generated according to the activation function based on and / or in response to one or more activation input values ​​received at the node. In a specific implementation, the activation input value and / or the activation output value may be constructed, sized and / or formatted as a "tensor". Therefore, in this context, as referred to herein, "activation input tensor" means an expression of one or more activation input values ​​according to a specific structure, size and / or format. Similarly, in this context, as referred to herein, "activation output tensor" means an expression of one or more activation output values ​​according to a specific structure, size and / or format.

[0112] In certain implementations, neural networks can achieve improved results in a wide range of tasks, including image recognition, speech recognition, to provide just a few example applications. To be able to perform such tasks, features of a neural network (e.g., nodes, edges, weights, layers of nodes and edges) can be structured and / or configured to form a "filter" that can have a measurable / numerical state such as a value of an output signal. Such a filter can include nodes and / or edges arranged in a "path" and that will respond to sensor observations provided as input signals. In one implementation, the state and / or output signal of such a filter can indicate and / or infer the detection of the presence or absence of a feature in the input signal.

[0113] In certain implementations, intelligent computing devices for performing functions supported by a neural network may include a variety of stationary and / or mobile devices, such as, for example, automotive sensors, biochip transponders, cardiac monitoring implants, Internet of Things (IoT) devices, kitchen appliances, locks or similar fastening devices, solar panel arrays, home gateways, smart meters, robots, financial trading platforms, smartphones, cellular phones, security cameras, wearable devices, thermostats, global positioning system (GPS) transceivers, personal digital assistants (PDAs), virtual assistants, laptops, personal entertainment systems, tablet personal computers (PCs), PCs, personal audio or video devices, personal navigation devices, to provide just a few examples.

[0114] According to one embodiment, the neural network may be hierarchically structured so that nodes in a particular neural network layer may receive output signals from one or more nodes located in an upstream layer in the neural network and provide output signals to one or more nodes located in a downstream layer in the neural network. A particular classification of hierarchical neural networks may include a convolutional neural network (CNN) or a spatially invariant artificial neural network (SIANN) that enables deep learning. Such CNNs and / or SIANNs may be based at least in part on a shared weight architecture of convolutional kernels that can transform input features and provide translation-equivariant responses. Such CNNs and / or SIANNs may be applied to image and / or video recognition, recommendation systems, image classification, image segmentation, medical image analysis, natural language processing, brain-computer interfaces, financial time series, to provide just a few examples.

[0115] Another class of hierarchical neural networks may include a recurrent neural network (RNN), which is a class of neural networks in which the connections between nodes form a directed cyclic graph along a time sequence. Such a time sequence can achieve modeling of temporal dynamic behavior. In a specific implementation, the RNN may use an internal state (e.g., a memory) to process input sequences of variable length. This can be applied to tasks such as unsegmented, connected handwriting recognition or speech recognition, for example, to provide only a few examples. In a specific implementation, the RNN may use a finite impulse response (FIR) or infinite impulse response (IIR) structure to mimic temporal behavior. The RNN may include additional structures to control the aging of the storage state of such FIR and IIR structures. The structure for controlling such storage state may include a network or graph containing a time delay and / or having a feedback loop, such as, in a long short-term memory network (LSTM) and a gated recurrent unit.

[0116] According to an embodiment, the output signals of one or more neural networks (e.g., obtained individually or in combination) may at least partially define a "predictor" to generate a predicted value associated with some observable and / or measurable phenomenon and / or state. In a specific implementation, the neural network may be "trained" to provide a predictor capable of generating such a predicted value based on input values ​​(e.g., measurements and / or observations) optimized according to a loss function. For example, the training process may employ backpropagation techniques to iteratively update the neural network weights to be associated with the nodes and / or edges of the neural network based at least in part on a "training set". Such a training set may include training measurements and / or observations to be provided as input values ​​paired with "ground truth" observations or expected outputs. Based on a comparison of such ground truth observations with the associated predicted values ​​generated according to such input values ​​in the training process, backpropagation may be used to update the weights according to the loss function. The neural network employed in various examples may be any known or future neural network architecture, including a traditional feedforward neural network, a convolutional neural network, or other such networks.

[0117] Fig. 9 One or more image processing and / or filtering architectures (e.g., image processing stages, Figure 1 ) computing environment. Here, the cloud server 902 includes: a processor 904, which is operable to process stored computer instructions; a memory 906, which is operable to store computer instructions, values, symbols, parameters, etc. for processing on the cloud server; and input / output 908, such as a network connection, a wireless connection, and a connection to accessories such as a keyboard. The storage device 910 can be non-volatile and can store values, parameters, symbols, content, code, etc., such as code for an operating system 912 and code for software such as an image processing module 914. The image processing module 914 may include multiple signal processing and / or filtering architectures 916 and 918, which are operable to render and / or process images. The signal processing and / or filtering architectures can be used to process images or other content stored on the server, or to provide remote services or "cloud" services to remote computers such as computers 930 connected via a public network 922 such as the Internet.

[0118] Smartphone 924 can also be coupled to Fig. 9, and may include applications 926, such as video games, virtual reality applications, or other applications 926, that utilize image processing and / or filtering architectures 928 to process rendered images. Image processing and / or filtering architectures 916, 918, and 928 may provide faster and more efficient computation of effects (such as denoising ray-traced rendered images in environments such as smartphones), and may provide longer battery life due to the reduction in power required to render desired effects and / or computational results. In some examples, a device such as a smartphone 924 may use a dedicated signal processing and / or filtering architecture 928 for some tasks, such as relatively simple image rendering that does not require a lot of computing resources or power, and offload other processing tasks to the signal processing and / or filtering architectures 916 or 918 of the cloud server 902 for more complex tasks.

[0119] In some examples, Fig. 9 The signal processing and / or filtering architecture 916, 918, and 928 may be implemented in software, where the processing stages (e.g., Figure 1 Various nodes, tensors, and other elements of the processing blocks in the signal processing and / or filtering architectures 916, 918, and 928 may be stored in data structures in a memory such as 906 or a storage device 910. In other examples, the signal processing and / or filtering architectures 916, 918, and 928 may be implemented in hardware, such as a convolutional neural network structure embodied in transistors, resistors, and other elements of an integrated circuit. In an alternative example, the signal processing and / or filtering architectures 916, 918, and 928 may be implemented in a combination of hardware and software, such as a neural processing unit (NPU) with software configurable weights, network size and / or structure, and other such configuration parameters.

[0120] Trained neural network 112( Figure 1) and other neural networks as described herein in certain examples may be formed in whole or in part by and / or expressed in processes (e.g., front end-of-line processes and / or back-end-of-line processes) such as transistors and / or underlying metal interconnects (not shown) in processes for forming complementary metal oxide semiconductor (CMOS) circuits. The various blocks, neural networks, and other elements disclosed herein may be described using computer-aided design tools and expressed (or represented) as data and / or instructions embodied in various computer-readable media in terms of their behavior, register transfers, logic components, transistors, layout geometry, and / or other characteristics. The formats of files and other objects in which such circuit expressions may be implemented include, but are not limited to, formats supporting behavioral languages ​​such as C, Verilog, and VHDL, formats supporting register-level description languages ​​such as RTL, and formats supporting geometric description languages ​​such as GDSII, GDSIII, GDSIV, CIF, MEBES, and any other suitable formats and languages. Storage media that can embody such formatted data and / or instructions include, but are not limited to, various forms of non-volatile storage media (e.g., optical, magnetic, or semiconductor storage media) and carrier waves that can be used to transmit such formatted data and / or instructions via wireless, optical, or wired signaling media, or any combination thereof. Examples of transmitting such formatted data and / or instructions via carrier waves include, but are not limited to, transmission (upload, download, email, etc.) via one or more data transmission protocols (e.g., HTTP, FTP, SMTP, etc.) over the Internet and / or other computer networks.

[0121] Computing devices such as cloud server 902, smart phone 924, and other such devices that may employ signal processing and / or filtering architectures may take many forms and may include many features or functions, including those described and those not described herein.

[0122] Fig.10 A block diagram of a general computerized system consistent with an example embodiment is shown. Fig.10 Only one particular example of a computing device 1000 is shown, and other computing devices 1000 may be used in other embodiments. Although the computing device 1000 is shown as a stand-alone computing device, the computing device 1000 may be any component or system including one or more processors or another suitable computing environment for executing software instructions in other examples, and need not include all of the elements shown here.

[0123] like Fig.10As shown in the specific example of FIG, the computing device 1000 includes one or more processors 1002, a memory 1004, one or more input devices 1006, one or more output devices 1008, one or more communication modules 1010, and one or more storage devices 1012. In one example, the computing device 1000 also includes an operating system 1016 that can be executed by the computing device 1000. The operating system includes services such as network services 1018 and virtual machine services 1020 (such as virtual servers) in various examples. One or more applications such as an image processor 1022 are also stored on the storage device 1012 and can be executed by the computing device 1000.

[0124] Each of components 1002, 1004, 1006, 1008, 1010, and 1012 may be interconnected (physically, communicatively, and / or operationally) for inter-component communication, such as via one or more communication channels 1014. In some examples, communication channels 1014 include a system bus, a network connection, an inter-processor communication network, or any other channel for transmitting data. Applications such as graphics processor 1022 and operating system 1016 may also transmit information between each other and with other components in computing device 1000.

[0125] In one example, the processor 1002 is configured to implement functions and / or process instructions for execution within the computing device 1000. For example, the processor 1002 may be capable of processing instructions stored in the storage device 1012 or the memory 1004. Examples of the processor 1002 include any one or more of a microprocessor, a controller, a central processing unit (CPU), a graphics processing unit (GPU), a neural processing unit (NPU), an image signal processor (ISP), a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), or similar discrete or integrated logic circuits.

[0126] One or more storage devices 1012 may be configured to store information within the computing device 1000 during operation. In some examples, the storage device 1012 is referred to as a computer-readable storage medium. In some examples, the storage device 1012 includes temporary storage, which means that the primary purpose of the storage device 1012 is not long-term storage. In some examples, the storage device 1012 is a volatile memory, which means that when the computing device 1000 is turned off, the storage device 1012 does not retain the stored content. In other examples, data is loaded from the storage device 1012 into the memory 1004 during operation. Examples of volatile memory include random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), and other forms of volatile memory known in the art. In some examples, the storage device 1012 is used to store program instructions executed by the processor 1002. In various examples, the storage device 1012 and the memory 1004 are used by software or applications (such as the image processor 1022) running on the computing device 1000 to temporarily store information during program execution.

[0127] In some examples, storage device 1012 includes one or more computer-readable storage media that can be configured to store a larger amount of information than volatile memory. Storage device 1012 can be further configured to store information for a long time. In some examples, storage device 1012 includes a non-volatile storage element. Examples of such non-volatile storage elements include a magnetic hard disk, an optical disk, a floppy disk, a flash memory, or a form of electrically programmable memory (EPROM) or electrically erasable programmable (EEPROM) memory.

[0128] In some examples, computing device 1000 also includes one or more communication modules 1010. Computing device 1000 in one example uses communication module 1010 to communicate with external devices via one or more networks, such as one or more wireless networks. Communication module 1010 can be a network interface card such as an Ethernet card, an optical transceiver, a radio frequency transceiver, or any other type of device capable of sending and / or receiving information. Other examples of such network interfaces include Bluetooth, 4G, LTE or 5G, WiFi radio, and near field communication (NFC) and universal serial bus (USB). In some examples, computing device 1000 uses communication module 1010 to communicate with external devices such as via Fig. 9 The public network 922 communicates wirelessly with external devices.

[0129] In one example, computing device 1000 also includes one or more input devices 1006. In some examples, input device 1006 is configured to receive input from a user through tactile, audio, or video input. Examples of input device 1006 include a touch screen display, a mouse, a keyboard, a voice response system, a video camera, a microphone, or any other type of device for detecting input from a user.

[0130] One or more output devices 1008 may also be included in computing device 1000. In some examples, output device 1008 is configured to provide output to a user using tactile, audio, or video stimulation. In one example, output device 1008 includes a display, a sound card, a video graphics adapter card, or any other type of device for converting signals into an appropriate form understandable to a human or machine. Additional examples of output device 1008 include a speaker, a light emitting diode (LED) display, a liquid crystal display (LCD or OLED), or any other type of device that can generate output to a user.

[0131] The computing device 1000 may include an operating system 1016. In some examples, the operating system 816 controls the operation of the components of the computing device 1000 and provides interfaces from various applications such as the image processor 1022 to the components of the computing device 1000. For example, in one example, the operating system 1016 facilitates the communication of various applications such as the image processor 1022 with the processor 1002, the communication unit 1010, the storage device 1012, the input device 1006, and the output device 1008. Applications such as the image processor 1022 may include program instructions and / or data that can be executed by the computing device 1000. As an example, the image processor 1022 may implement a signal processing and / or filtering architecture 1024 to perform image processing tasks such as those described above or rendered image denoising tasks, which in further examples include using signal processing and / or filtering hardware elements such as those described in the above examples. These and other program instructions or modules may include instructions that cause the computing device 1000 to perform one or more of the other operations and actions described in the examples given herein.

[0132] In one embodiment, Fig. 9 and Fig.10The features of the example devices in may include, for example, features of a client computing device and / or a server computing device. It should also be noted that the term computing device generally refers to at least a processor and a memory connected by a communication bus, whether used as a client and / or a server, or in addition thereto. For example, "processor" and / or "processing circuit" is understood to mean a specific structure of a computing device that may include a control unit and an execution unit, such as a central processing unit (CPU), a digital signal processor (DSP), a graphics processing unit (GPU), an image signal processor (ISP) and / or a neural processing unit (NPU) or a combination thereof. In one aspect, a processor and / or a processing circuit may include a device that acquires, interprets and executes instructions to process input signals to provide output signals. Therefore, at least in the context of the present patent application, this is understood to refer to a sufficient structure within the meaning of 35 USC § 112 (f) so that it is specifically intended that 35 USC § 112 (f) is not implied by the use of the terms "computing device", "processor", "processing unit", "processing circuit" and / or similar terms. However, if, for some obscure reason, it is determined that the foregoing understanding does not hold and that 35 USC § 112(f) is therefore necessarily implied by the use of the term "computing device" and / or similar terms, then, under that statutory section, the corresponding structure, materials, and / or acts for performing one or more functions are to be understood and construed as at least Figure 1 and described in the text associated with the aforementioned figures of this patent application.

[0133] As used herein, the term "electronic file" and / or the term "electronic document" refers to a collection of stored memory states and / or a collection of physical signals that are associated in some manner so as to at least logically form a file (e.g., an electronic file) and / or an electronic document. That is, this is not intended to implicitly reference a specific syntax, format, and / or method used, for example, with respect to a collection of associated memory states and / or a collection of associated physical signals. If, for example, a specific type of file storage format and / or syntax is expected, then the file storage format and / or syntax is explicitly referenced. It should also be noted that the association of memory states can be, for example, in a logical sense and not necessarily in a tangible physical sense. Therefore, although the signal and / or state components of a file and / or electronic document will, for example, be logically associated, in one embodiment, the storage of the signal and / or state components may, for example, reside in one or more different locations in a tangible physical memory.

[0134] In the context of this patent application, the terms "entry", "electronic entry", "document", "electronic document", "content", "digital content", "item" and / or similar terms are intended to refer to signals and / or states in a physical format (such as a digital signal and / or digital state format) that may be perceptible to a user if displayed, played, tactilely generated, etc. and / or otherwise executed by a device such as a digital device (including, for example, a computing device), but may not necessarily be easily perceptible to a human (for example, if in a digital format).

[0135] In addition, for one or more embodiments, the electronic document and / or electronic file may include multiple components. As previously mentioned, in the context of the present patent application, the component is physical, but not necessarily tangible. For example, in one or more embodiments, the component of the reference electronic document and / or electronic file may include text in the form of, for example, physical signals and / or physical states (e.g., capable of being physically displayed). Typically, the memory state, for example, includes tangible components, and the physical signal is not necessarily tangible, although the signal may become (e.g., become) tangible, such as, if it appears on a tangible display, this is not uncommon. In addition, for one or more embodiments, the component of the reference electronic document and / or electronic file may include a graphic object and / or a sub-object, the graphic object, for example, an image such as a digital image, the sub-object includes its attributes, and the attribute again includes a physical signal and / or a physical state (e.g., capable of being tangibly displayed). In one embodiment, digital content may include, for example, text, images, audio, video and / or other types of electronic documents and / or electronic files, including, for example, parts of electronic documents and / or electronic files.

[0136] Additionally, in the context of the present patent application, the terms "parameter" (e.g., one or more parameters), "value" (e.g., one or more values), "symbol" (e.g., one or more symbols), "bit" (e.g., one or more bits), "element" (e.g., one or more elements), "character" (e.g., one or more characters), "number" (e.g., one or more numbers), "number" (e.g., one or more numbers), or "measurement" (e.g., one or more measurements) refer to material that describes a collection of signals, such as in one or more electronic documents and / or electronic files, and exists in the form of physical signals and / or physical states, such as memory states. For example, one or more parameters, values, symbols, bits, elements, characters, numbers, quantities, or measurements (such as reference to one or more aspects of an electronic document and / or electronic file including an image) may include, for example, the time of day of the captured image, the latitude and longitude of an image capture device such as a camera, etc. In another example, for example, one or more parameters, values, symbols, bits, elements, characters, numbers, quantities, or measurements associated with digital content (such as digital content including a technical paper) may include, for example, one or more authors. The claimed subject matter is intended to cover any meaningful descriptive parameter, value, symbol, bit, element, character, number, quantity, or measurement in any format, so long as the one or more parameters, values, symbols, bits, elements, characters, numbers, quantities, or measurements include physical signals and / or states, which may include, as examples of parameters, collective names (e.g., electronic file and / or electronic document identifier names), creation techniques, creation purposes, creation times and dates, logical paths if stored, encoding formats (e.g., types of computer instructions such as markup languages), and / or standards and / or specifications used to enable protocol compatibility for one or more purposes (e.g., meanings of substantially compatible and / or essentially compatible), etc.

[0137] Although specific embodiments have been illustrated and described herein, any arrangement that achieves the same purpose, structure, or function may replace the specific embodiments shown. This application is intended to cover any modifications or variations of the exemplary embodiments of the invention described herein. These and other embodiments are within the scope of the following claims and their equivalents.

Claims

1. A method, comprising: Obtaining a plurality of illumination components of a rendered image frame at a current time instance, the plurality of illumination components of the rendered image frame being associated with a plurality of illumination coefficients; warping an illumination component of a previous image frame to provide a warped image frame referenced to the current time instance; applying a first illumination coefficient to combine pixel values ​​of a first illumination component in the rendered image frame with pixel values ​​of the first illumination component in the warped image frame to provide pixel values ​​of the first illumination component in an output image frame; as well as applying a second illumination coefficient to combine pixel values ​​of a second illumination component in the rendered image frame with pixel values ​​of the second illumination component in the warped image frame to provide pixel values ​​of the second illumination component in the output image frame; Wherein the first lighting coefficient and the second lighting coefficient are derived from an output tensor of a neural network.

2. The method according to claim 1, wherein: Applying the first illumination coefficient to combine pixel values ​​of the first illumination component in the warped image frame includes interpolating pixel values ​​of the first illumination component; and Applying the second illumination coefficient to combine pixel values ​​of the second illumination component in the warped image frame includes interpolating pixel values ​​of the second illumination component. The method of claim 1 , wherein the first illumination coefficient and the second illumination coefficient are per-pixel coefficients. The method according to claim 1 , wherein the plurality of illumination components comprises at least a specular illumination component and a diffuse illumination component. The method of claim 1 , wherein the plurality of lighting components are rendered using ray tracing.

6. The method of claim 1 , wherein warping the illumination component of the previous image frame to provide the warped image frame referenced to the current time instance further comprises: applying one or more first motion vectors to pixel values ​​of the first illumination component to provide pixel values ​​in the warped image frame for the first illumination component; as well as One or more second motion vectors are applied to pixel values ​​of the second illumination component to provide pixel values ​​in the warped image frame for the second illumination component.

7. The method of claim 1, further comprising combining the first illumination component and the second illumination component to generate an output image frame.

8. The method of claim 7, further comprising using albedo to determine a ratio of the first illumination component and the second illumination component used to generate the output image frame.

9. The method of claim 7, further comprising filtering the combined first illumination component and second illumination component to reduce noise in generating the output image frame.

10. A method according to claim 1, wherein the first illumination coefficient and the second illumination coefficient are derived from the unocclusion of one or more pixels in the rendered image frame in the current time instance, or are derived from the change of perspective-dependent illumination of one or more pixels in the rendered image frame in the current time instance, or are derived from a combination thereof.

11. A computing device, the computing device comprising: a memory, the memory comprising one or more storage devices; and one or more processors, the one or more processors coupled to the memory, the one or more processors operable to: obtaining from the memory a plurality of illumination components of a rendered image frame at a current time instance, the plurality of illumination components of the rendered image frame being associated with a plurality of illumination coefficients; warping an illumination component of a previous image frame to provide a warped image frame referenced to the current time instance; applying a first illumination coefficient to combine pixel values ​​of a first illumination component in the rendered image frame with pixel values ​​of the first illumination component in the warped image frame to provide pixel values ​​of the first illumination component in an output image frame; as well as applying a second illumination coefficient to combine pixel values ​​of a second illumination component in the rendered image frame with pixel values ​​of the second illumination component in the warped image frame to provide pixel values ​​of the second illumination component in the output image frame; Wherein the first lighting coefficient and the second lighting coefficient are derived from an output tensor of a neural network.

12. The computing device of claim 11, wherein: Applying the first illumination coefficient to combine pixel values ​​of the first illumination component in the warped image frame is based at least in part on interpolating pixel values ​​of the first illumination component; and Applying the second illumination coefficient to combine pixel values ​​of the second illumination component in the warped image frame is based at least in part on interpolation of pixel values ​​of the second illumination component.

13. The computing device of claim 11, wherein the first illumination coefficient and the second illumination coefficient are per-pixel coefficients.

14. The computing device of claim 11, wherein the plurality of illumination components comprises at least a specular illumination component and a diffuse illumination component.

15. The computing device of claim 11, wherein the plurality of lighting components are to be rendered using ray tracing.

16. The computing device of claim 11, wherein applying one or more motion vectors to the illumination component of the previous image frame to provide the warped image frame referenced to the current time instance is based at least in part on: applying one or more first motion vectors to pixel values ​​of the first illumination component to provide pixel values ​​in the warped image frame for the first illumination component; and One or more second motion vectors are applied to pixel values ​​of the second illumination component to provide pixel values ​​in the warped image frame for the second illumination component.

17. The computing device of claim 11, the one or more processors further operable to combine the first illumination component and the second illumination component to generate an output image frame.

18. The computing device of claim 17, the one or more processors further operable to apply albedo to determine a ratio of the first illumination component and the second illumination component for generating the output image frame.

19. A computing device according to claim 11, wherein the first lighting coefficient and the second lighting coefficient are derived from unocclusion of pixels in the rendered image frame in the current time instance, or are derived from changes in perspective-dependent lighting in the rendered image frame in the current time instance, or are derived from a combination thereof.

20. A method for training a neural network, the method comprising: Receiving an input tensor in an input layer of the neural network, the input tensor representing one or more characteristics of an image frame; providing an output tensor to an output layer of the neural network, the output tensor representing a first illumination coefficient and a second illumination coefficient, the first illumination coefficient being used to combine pixel values ​​of a first illumination component in a rendered image frame with pixel values ​​of the first illumination component in a warped image frame to provide pixel values ​​of the first illumination component in an output image frame, and the second illumination coefficient being used to combine pixel values ​​of a second illumination component in the rendered image frame with pixel values ​​of the second illumination component in the warped image frame to provide pixel values ​​of the second illumination component in the output image frame, the output layer of the neural network being connected by one or more intermediate layers of the neural network; as well as The neural network is trained by adjusting weights of one or more activation functions linking one or more nodes of one or more layers of the neural network using back-propagation to predict a provided output tensor when provided with a received input tensor.

Images