Method and apparatus for real-time ray tracing image rendering

Abstract

For real-time ray tracing image rendering, a sampling recommendation for a frame to be rendered in real time is obtained and pixel values are sampled in the frame with a ray-tracing renderer according to the sampling recommendation. A learned latent representation of the frame is obtained with a second neural network configured as a latent state encoder. Further, the latent state encoder is fed with the sampled pixel values and a latent representation of a previous rendered frame. Then the learned latent representation of the frame is processed with a third neural network configured as a decoder to obtain a real-time rendering output. Feasible and visually appealing real-time ray-traced rendering is provided within the computational constraints imposed by contemporary consumer hardware.

Description

[0001] METHOD AND APPARATUS FOR REAL-TIME RAY TRACING IMAGE RENDERING

[0002] TECHNICAL FIELD

[0003] The present disclosure relates generally to the field of computer graphics and, more specifically, to a method and an apparatus for real-time ray tracing image rendering.

[0004] BACKGROUND

[0005] Typically, in the field of computer graphics, real-time ray tracing (RT) is a cutting-edge rendering technique renowned for an ability to faithfully replicate the intricate behavior of light in the physical world. The RT’s unparalleled quality is most pronounced when a substantial number of samples are meticulously drawn for each pixel in an image. However, the practical application of RT encounters significant challenges in real-time scenarios, primarily due to computational constraints. The computational constraints of amassing a multitude of samples per pixel prove prohibitive, particularly in a context of contemporary consumer hardware, including smartphones, consoles, and PCs, where real-time rendering is a staple.

[0006] Currently, certain attempts have been made to solve the problem of amassing the multitude of samples per pixel by employing a multifaceted approach in real-time rendering with the RT. In an attempt, the number of samples required for each pixel is strategically reduced, striking a balance between computational cost and visual quality. In another attempt, post-processing techniques, such as denoising, are employed to refine an output of Monte Carlo (MC) ray tracing process for mitigating noise. In yet another attempt, non-uniform sample allocation across pixels within a frame, coupled with heuristic methods, is employed to optimize computational resources. However, the problem associated with real-time RT is that the realtime RT lacks adaptive sampling, relies on a probabilistic latent space representation, uses averaged sampled pixel values, and employs temporal feedback that relies on averaged denoised frames, which is undesirable. Additionally, the latent space representation of the realtime RT is limited to capturing only the first and second moments, which is again undesirable. Thus, there exists a technical problem of how to make real-time ray-traced rendering feasible and visually appealing within the computational constraints imposed by contemporary consumer hardware, a challenge that encompasses mobile gaming on smartphones, consoles, and PCs alike.

[0007] Therefore, in light of the foregoing discussion, there exists a need to overcome the aforementioned drawbacks associated with conventional real-time rendering approaches.

[0008] SUMMARY

[0009] The present disclosure provides a method of real-time ray tracing image rendering and an apparatus for real-time ray tracing image rendering. The present disclosure provides a solution to the existing problem of how to make real-time ray-traced rendering feasible and visually appealing within the computational constraints imposed by contemporary consumer hardware, a challenge that encompasses mobile gaming on smartphones, consoles, and PCs alike. An aim of the present disclosure is to provide a solution that overcomes at least partially the problems encountered in the prior art and provides an improved method of real-time ray tracing image rendering and an improved apparatus for real-time ray tracing image rendering, such as latent space encoding for path tracing adaptive sampling and denoising.

[0010] One or more objectives of the present disclosure are achieved by the solutions provided in the enclosed independent claims. Advantageous implementations of the present disclosure are further defined in the dependent claims.

[0011] In one aspect, the present disclosure provides a method of real-time ray tracing image rendering, the method comprising: obtaining a sampling recommendation for a frame to be rendered in real time, sampling pixel values in the frame with a ray-tracing Tenderer according to the sampling recommendation, obtaining a learned latent representation of the frame with a second neural network, wherein the second neural network is configured as a latent state encoder and wherein the latent state encoder is fed with the sampled pixel values and a latent representation of a previous rendered frame, and processing the learned latent representation of the frame with a third neural network configured as a decoder to obtain a real-time rendering output.

[0012] The method of real-time ray tracing image rendering delivers superior visual quality in less time, achieving higher frame-rate at equal Peak Signal-to-Noise Ratio (PSNR) when compared to conventional real-time rendering approaches. The method allows users to balance quality and rendering speed by drawing fewer samples. The method further introduces adaptiveness to the rendering process, enabling the drawing of zero samples at certain pixels, a groundbreaking capability in real-time ray tracing. The adaptiveness is further enhanced by effective pre- and post-processing techniques, allowing for the drawing of significantly fewer samples, including less than one sample per pixel (spp), thus enabling real-time ray tracing even on resource- constrained hardware.

[0013] In an implementation form, the sampling recommendation is obtained with a first neural network trained using a reinforcement learning technique or a deep learning technique.

[0014] The first neural network is trained using the reinforcement learning technique or the deep learning technique in order to allow a sampling size of less than 1 sample per pixel (< 1 spp) on average, i.e. zero samples in some pixels.

[0015] In further implementation form, the method further comprises feeding additional data as additional argument to one or more of the first neural network, the second neural network and the third neural network, wherein the additional data comprises one or more of non-av eraged sampled pixel values, motion vectors, albedos, normals and depths from one or more previously rendered frames.

[0016] The one or more of the first neural network, the second neural network and the third neural network are fed with the additional data or addition arguments in order to enhance the performance and quality of the one or more of the first neural network, the second neural network and the third neural network operations, ultimately leading to more realistic and detailed image rendering. The feeding of additional data facilitates these networks to make better-informed decisions when processing frames, resulting in improved real-time rendering outputs.

[0017] In another implementation form, the method further comprises warping a latent representation of one or more previously rendered frames, and feeding the latent representation of the one or more previously rendered frames to the first neural network and / or the second neural network.

[0018] The feeding of the latent representations of the one or more previously rendered frames to the first neural network and / or the second neural network can lead to more accurate and context- aware processing, which ultimately contributes to improved quality of the real-time rendering outputs. In further implementation form, the method further comprises upsampling the real-time rendering output with a supersampling neural network.

[0019] The upsampling of the real-time rendering output provides an additional enhancement step that involves increasing a resolution or size of the frame, making the frame more detailed and visually refined.

[0020] In another implementation form, the obtaining of the sampling recommendation and the sampling of the pixel values are iteratively repeated for two or more times during rendering of the same frame.

[0021] By virtue of iterative repetitions of obtaining the sampling recommendation and the sampling of the pixel values, the method updates the sampling recommendation and sampling of the pixel values based on a previous iteration. Such an iteration assists in optimizing allocation of resources (sampling budget) for each of the frame, contributing to enhanced quality of the realtime rendering output.

[0022] In another aspect, the present disclosure provides an apparatus configured for real-time ray tracing image rendering. The apparatus comprising: a sampling recommendation module configured for providing a sampling recommendation for a frame to be rendered in real time, a ray-tracing Tenderer configured for sampling pixel values in the frame according to the sampling recommendation, a latent state encoder configured for obtaining a learned latent representation of the frame with a second neural network that is fed with the sampled pixel values and a latent representation of a previous rendered frame, and a decoder configured for processing the learned latent representation of the frame with a third neural network to obtain a real-time rendering output.

[0023] The apparatus achieves all the advantages and technical effects of the method of the present disclosure.

[0024] It is to be appreciated that all the aforementioned implementation forms can be combined. It is be noted that all devices, elements, circuitry, units, and means described in the present application could be implemented in the software or hardware elements or any kind of combination thereof. All steps which are performed by the various entities described in the present application, as well as the functionalities described to be performed by the various entities, are intended to mean that the respective entity is adapted to or configured to perform the respective steps and functionalities. Even if, in the following description of specific embodiments, a specific functionality or step to be performed by external entities is not reflected in the description of a specific detailed element of that entity that performs that specific step or functionality, it should be clear for a skilled person that these methods and functionalities can be implemented in respective software or hardware elements, or any kind of combination thereof. It will be appreciated that features of the present disclosure are susceptible to being combined in various combinations without departing from the scope of the present disclosure as defined by the appended claims.

[0025] Additional aspects, advantages, features, and objects of the present disclosure would be made apparent from the drawings and the detailed description of the illustrative implementations construed in conjunction with the appended claims that follow.

[0026] BRIEF DESCRIPTION OF THE DRAWINGS

[0027] The summary above, as well as the following detailed description of illustrative embodiments, is better understood when read in conjunction with the appended drawings. For the purpose of illustrating the present disclosure, exemplary constructions of the disclosure are shown in the drawings. However, the present disclosure is not limited to specific methods and instrumentalities disclosed herein. Moreover, those skilled in the art will understand that the drawings are not to scale. Wherever possible, like elements have been indicated by identical numbers.

[0028] Embodiments of the present disclosure will now be described, by way of example only, with reference to the following diagrams wherein:

[0029] FIG. 1 is a flow chart of a method of real-time ray tracing image rendering, in accordance with an embodiment of the present disclosure;

[0030] FIG. 2 is a block diagram that depicts an apparatus configured to real-time ray tracing image render, in accordance with different embodiments of the present disclosure;

[0031] FIG. 3 A is a block diagram that depicts a computing device configured to carry out the method of real-time ray tracing image rendering; FIG. 3B is a block diagram that depicts a computer chip configured to carry out the method of real-time ray tracing image rendering; and

[0032] FIG. 4 is a diagram that depicts an exemplary implementation of the method of realtime ray tracing image rendering, in accordance with embodiments of the present disclosure.

[0033] In the accompanying drawings, an underlined number is employed to represent an item over which the underlined number is positioned or an item to which the underlined number is adjacent. A non-underlined number relates to an item identified by a line linking the nonunderlined number to the item. When a number is non-underlined and accompanied by an associated arrow, the non-underlined number is used to identify a general item at which the arrow is pointing.

[0034] DETAILED DESCRIPTION OF EMBODIMENTS

[0035] The following detailed description illustrates embodiments of the present disclosure and ways in which they can be implemented. Although some modes of carrying out the present disclosure have been disclosed, those skilled in the art would recognize that other embodiments for carrying out or practising the present disclosure are also possible.

[0036] FIG. l is a flow chart of a method of real-time ray tracing image rendering, in accordance with an embodiment of the present disclosure. With reference to FIG. 1, there is shown a flow chart of a method 100 of real-time ray tracing image rendering. The method 100 includes steps 102 and 108 of real-time ray tracing image rendering that can be used in any latent space encoder for path tracing adaptive sampling and denoising.

[0037] The method 100 provides an efficient solution for real-time ray tracing image rendering. At step 102, the method 100 comprises obtaining a sampling recommendation for a frame to be rendered in real time. The frame refers to a single complete image that is displayed on a screen or rendered in a video game or movie. In an implementation, the frame corresponds to a static picture that, when displayed in rapid succession with the other frames, creates the illusion of motion. For example, in a video game, each frame may represent a single snapshot of the video game at a particular moment. In such an example, when each of the frames is shown quickly one after another (typically at 30 or more frames per second), it creates an appearance of fluid motion. In another implementation, the frame corresponds to a single complete image composed of a collection of individual pixels, where each pixel within the frame represents a tiny dot that holds specific colour or brightness values. In such an implementation, the frame is a composite image created by arranging and displaying numerous pixels in a grid, forming a coherent visual scene.

[0038] The sampling recommendation refers to a set of instructions or guidance generated by a first neural network that suggests a process to collect or sample pixel values from the frame in realtime rendering. The pixel values refer to individual colour or brightness values assigned to each of the pixels, within the frame. The pixel values determine the specific colour and intensity of each of the pixels, combining to create the overall appearance and details of the frame. In the single complete image, the pixel values consist of three colour channels (red, green, and blue), with each channel's pixel value contributing to a final colour of the pixel.

[0039] In accordance with an embodiment, the sampling recommendation is obtained with the first neural network trained using a reinforcement learning technique or a deep learning technique. The reinforcement learning technique refers to a machine learning technique where an Al agent learns to make decisions by providing the sampling recommendation in an environment and receiving feedback based on the provided sampling recommendation, aiming to maximize a cumulative reward over time, ultimately improving the reinforcement learning technique’s decision-making abilities. In such an embodiment, the first neural network is trained using the reinforcement learning technique or the deep learning technique in order to allow a sampling size of less than 1 sample per pixel (< 1 spp) on average, i.e. zero samples in some pixels. In an implementation, the sampling recommendation may be uniform, sparsely uniform (for upsampling), or adaptive. The sampling recommendation is obtained to improve quality and realism of an image.

[0040] The first neural network provides the sampling recommendation (r) as r=S(x_i), where: i represents a variable that denotes a specific frame number or index, i indicates a reference to the frame in a sequence. x_i represents a latent representation that stands for some data or information associated with a specific frame i. x_i refers to the latent representation from ithframe. S represents a function or operation that takes the latent representation x_i as input and produces the sampling recommendation r as output. The S function determines how to collect pixel values from the frame to improve image rendering.

[0041] At step 104, the method 100 comprises sampling the pixel values in the frame with a ray-tracing renderer according to the sampling recommendation. The ray-tracing Tenderer (R) refers to a computer program or component that is responsible for creating highly realistic images by simulating a way that rays of light (via the pixel values) interact with objects in a virtual 3D environment. The ray-tracing renderer (R) is configured to generate visually accurate lighting, shadows, and reflections by sampling the pixel values in the frame according to the sampling recommendation. Thus, the ray-tracing renderer (R) is configured to sample the pixel values p_{i+l } following the sampling recommendation r.

[0042] At step 106, the method 100 comprises obtaining a learned latent representation of the frame with a second neural network, wherein the second neural network is configured as a latent state encoder and wherein the latent state encoder is fed with the sampled pixel values and a latent representation of a previous rendered frame. The second neural network is configured as the latent state encoder (T) such that x_{i+l } = T(x_i, p_{i+l }) represents a transformation of the sampled pixel values, where, x_{i+l } represents the learned latent representation for the (i+ 1 )thframe. x_i represents the latent representation of the previous rendered frame (i). p_{i+l } represents the sampled pixel values of the frame (i+1).

[0043] T represents the latent state encoder that takes two inputs: x_i and p_{i+l }.

[0044] The learned latent representation of the frame refers to a transformed latent representation of the previous rendered frame where the latent representation of the previous rendered frame is transformed to the learned latent representation of the frame by a way of processing and refining through the second neural network. The learned latent representation of the frame is processed and improved by the second neural network to be more informative and useful for tasks like image rendering by using information from the previous rendered frame. The second neural network plays a crucial role in the transformation of the latent representation of the previous rendered frame to the learned latent representation of the frame, aiding to capture relevant features and details from the previous rendered frame and applying relevant features and details to the frame. The learned latent representation of the frame yields higher image quality for a number of ray-tracing samples.

[0045] At step 108, the method 100 comprises processing the learned latent representation of the frame with a third neural network configured as a decoder to obtain a real-time rendering output. The third neural network is configured as the decoder (Q) such that y_{i+ 1 } = Q(x_{i+1 }) in order to obtain the real-time rendering output of the frame as y_{i+l } . The method 100 comprises computing the real-time rendering output of the frame y_{i+l } as the output of Q(x_{i+1 }) where, x_{i+l } represents the learned latent representation for the (i+l)thframe.

[0046] Q represents the decoder that takes the learned latent representation for the (i+l)thframe as input and produces the real-time rendering output of the (i+l)thframe.

[0047] In an implementation, the method 100 comprises computing the real-time rendering output of the frame y_{i+ 1 } as the output of Q(p_{i+ 1 } ,y_{i }) where, p_{i+l } represents the sampled pixel values of the (i+l)thframe. y_{i } represents the real-time rendering output for the previous rendered frame i.

[0048] Q represents the decoder that takes two inputs: the sampled pixel values of the (i+l)thframe and the real-time rendering output for the previous rendered frame i to produce the real-time rendering output of the (i+l)thframe.

[0049] In accordance with an embodiment, the method 100 comprises feeding additional data as additional argument to one or more of the first neural network, the second neural network and the third neural network, wherein the additional data comprises one or more of non-av eraged sampled pixel values, motion vectors, albedos, normals and depths from one or more previously rendered frames. The non-averaged sampled pixel values refer to individual pixel values within the frame that haven't been averaged or combined with other pixel values of the frame. The non-averaged sampled pixel values provide detailed information about the colour and brightness of the pixels in the frame. The motion vectors represent how elements in the scene are moving from one frame to another frame. The motion vectors aids in capturing dynamics and movement in the image. The albedos refer to reflective properties of surface of a screen. The albedos indicate how much light is reflected by different materials in the scene. The normals refer to vectors that describe a direction in which the surfaces are facing. The normals aids in calculating lighting and shading effects. The depths refer to information that indicates how far objects are placed from the camera. The depths are crucial for creating realistic spatial effects and depth perception. In an example, the additional data of the frame is fed to the third neural network in order to get the real-time image rendering output, such as y_{i+l } = Q(P_{i+l },y_{i} ,add_{i+l }) where, p_{i+l } represents the sampled pixel values of the (i+l)thframe. y_{i } represents the real-time rendering output for the previous rendered frame i. add_{i+l } represents additional data of the (i+l)thframe.

[0050] Q represents the decoder that takes three inputs: the sampled pixel values of the (i+ 1 )thframe, the real-time rendering output for the previous rendered frame i, and the additional data of the (i+l)thframe to produce the real-time rendering output of the (i+l)thframe.

[0051] The one or more of the first neural network, the second neural network and the third neural network are fed with the additional data or additional arguments in order to enhance the performance and quality of the one or more of the first neural network, the second neural network and the third neural network operations, ultimately leading to more realistic and detailed image rendering. The feeding of additional data facilitates these networks to make better-informed decisions when processing frames, resulting in improved real-time rendering outputs.

[0052] In accordance with an embodiment, the method 100 comprises warping a latent representation of one or more previously rendered frames, and feeding the latent representation of the one or more previously rendered frames to the first neural network and / or the second neural network. The warping of the latent representation of one or more previously rendered frames corresponds to considering the latent representations of the one or more frames rendered before the frame and adjusting the latent representation of one or more previously rendered frames based on how objects or elements have moved in the scene. A warper W is configured to warp the latent representation of one or more previously rendered frames (with index i) to generate adjusted latent representations of the one or more previously rendered frames. The adjusted latent representations of the one or more previously rendered frames are fed into the first neural network and / or the second neural network. By virtue of feeding the adjusted latent representations of the one or more previously rendered frames, the first neural network and / or the second neural network can take into account history of the scene, how the scene has evolved, and adjustments made to the one or more previously rendered frames information. The feeding of the adjusted latent representations of the one or more previously rendered frames to the first neural network and / or the second neural network can lead to more accurate and context-aware processing, which ultimately contributes to improved quality of the real-time rendering outputs.

[0053] In an example, the additional data (motion vectors) of the frame is fed to the warper before feeding the same additional data to the third neural network in order to get the real-time image rendering output, such as y_{i+l } = Q("p_{i+l }",W("y_{i},add_{i+l })" ,"add_{i+l }" ) where, p_{i+l } represents the sampled pixel values of the (i+l)thframe. y_{i} represents the real-time rendering output for the previous rendered frame i. add_{i+l } represents additional data of the (i+l)thframe.

[0054] W represents the warper that takes two inputs: y_{i} and add_{i+l } to produce a warping output of the (i+ 1 )thframe.

[0055] Q represents the decoder that takes three inputs: the sampled pixel values of the (i+ 1 )thframe, the warping output of the (i+l)thframe, and the additional data of the (i+l)thframe to produce the real-time rendering output of the (i+l)thframe.

[0056] In accordance with an embodiment, the method 100 comprises upsampling the real-time rendering output with a supersampling neural network. The upsampling of the real-time rendering output refers to an operation in the method 100 in which the real-time rendering output undergoes an additional enhancement step that involves increasing a resolution or size of the frame, making the frame more detailed and visually refined. The supersampling neural network (U) is configured to upsample the real-time rendering output. In an example, the real- time rendering output of the frame is fed to the supersampling neural network (U) in order to get an improved real-time image rendering output, such as y’_{i+l }=U(y_{i+l }) where, y_{i+l } represents the real-time rendering output of the (i+l)thframe.

[0057] U represents the supersampling neural network that takes y_{i+l } as in input to produce the improved real-time rendering output y ’_{i+l } of the (i+l)thframe.

[0058] In accordance with an embodiment, the obtaining of the sampling recommendation and the sampling of the pixel values are iteratively repeated for two or more times during rendering of the same frame. Thereby, the method 100 comprises iteratively repeating the step 102 of obtaining the sampling recommendation and the step 104 of sampling the pixel values within the rendering of the frame. The iterative repetition of the steps 102 and 104 is carried out to refine and improve the rendering quality of the frame. Each iteration involves updating the sampling recommendation and sampling the pixel values based on a previous iteration. Such an iteration facilitates to optimize allocation of resources (sampling budget) for each of the frame, contributing to enhanced quality of the real-time rendering output. In an example, the obtaining of the sampling recommendation and the sampling of the pixel values is repeated, such as, r_t=S(x_i, p_{i+l,t-l }) and p_{i+l,t} = R(r_t), where, r_t represents an initial sampling recommendation. The initial sampling recommendation r_t is generated based on a current frame's latent representation ("x_i") and potential information from previous iterations. p_{i+l,t-l (represents an initial sampling of pixel values. The initial sampling of pixel values is used to create a preliminary rendering of the frame.

[0059] S represents a function or operation that takes the latent representation, x_i and the sampled pixel values from the previous iteration (t-1), p_{i+l,t-l (as inputs and produces the sampling recommendation r_t as output. p_{i+l ,t( represents the pixel values for the (i+l)thframe in a current iteration t.

[0060] R represents a ray-tracing rendered operation that calculates the pixel values by taking the initial sampling recommendation r_t as in input. The method 100 in all the embodiments and implementation forms described above provides an efficient solution for real-time ray tracing image rendering. The method 100, instead of averaging output of the ray tracing rendered across sample recommendation of each of the pixel, retains the full distribution of the sampled pixel values, providing valuable additional information to subsequent stages in ray-tracing image rendering. The method 100 further introduces a latent state encoder (LSE) neural network that maintains stateful information about the rendering output in screen-space coordinates. The LSE network is fed with the sampled pixel values and the latent representation of the previous rendered frame, and the LSE network utilizes optical flow information to warp the latent state of the previous frame for each new frame, enhancing the quality of the final rendering. The method 100 also employs reinforcement learning for training the adaptive sampling network, allowing for sampling with spp<l, including the ability to use zero samples in some pixels. The method 100 revolutionizes realtime ray tracing by simultaneously improving visual quality and rendering speed, making it suitable for a wide range of hardware and applications.

[0061] The steps 102 to 108 are only illustrative, and other alternatives can also be provided where one or more steps are added, one or more steps are removed, or one or more steps are provided in a different sequence without departing from the scope of the claims herein.

[0062] There is further provided a computer program comprising instructions which, when the computer program is executed by a computing device 302. The computer program causes the computing device 302 to carry out the steps of the method 100. There is further provided a computer-readable medium comprising instructions which, when executed by the computing device 302, cause the computing device 302 to carry out the steps of the method 100. The computer program is implemented as an algorithm, embedded in a software stored in a non- transitory computer-readable storage medium. The non-transitory computer-readable storage means may include but are not limited to, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. Examples of implementation of computer- readable storage medium, but are not limited to, Electrically Erasable Programmable Read- Only Memory (EEPROM), Random Access Memory (RAM), Read Only Memory (ROM), Hard Disk Drive (HDD), Flash memory, a Secure Digital (SD) card, Solid-State Drive (SSD), a computer-readable storage medium, and / or CPU cache memory. FIG. 2 is a block diagram that depicts an apparatus configured to real-time ray tracing image render. FIG. 2 is described in conjunction with elements from FIG. 1. With reference to FIG. 2 there is shown a block diagram 200 of an apparatus 202 that is configured to real-time ray tracing image render.

[0063] The apparatus 202 comprises the sampling recommendation module 204, the ray-tracing renderer 206, the latent state encoder 208, the decoder 210, the data feeding module 212, and the warper 214.

[0064] The sampling recommendation module 204 refers to a component of the apparatus 202 that suggests how to gather data about the pixels in the frame effectively. In other words, the sampling recommendation module 204 recommends the apparatus 202 where to look for the pixel data. The sampling recommendation module 204 is associated with the first neural network that plays a crucial role in making decisions about how to gather the pixel data during real-time rendering. In other words, the first neural network 216 uses advanced techniques to provide guidance on gathering pixel data.

[0065] The ray-tracing renderer 206 refers to a core component that takes the gathered pixel data as an input and enhances the gathered pixel data to create a more realistic image. In other words, the ray-tracing renderer 206 adds details, colours, and shading to make the image look excellent.

[0066] The latent state encoder 208 refers to a part of the apparatus 202 that learns from the rendered frames and creates a memory of how to improve images based on past experiences. The latent state encoder 208 is associated with the second neural network that is configured to serve as a learner by understanding patterns in the pixel data and previous frames to improve image rendering. The second neural network observes the pixel data and uses that knowledge to make the images look better.

[0067] The decoder 210 corresponds to a part of the apparatus 202 that takes the learned information from the latent state encoder 208 and uses the learned information to refine the image even further. The decoder 210 is associated with the third neural network 220 that is responsible for refining the learned information to produce the final, high-quality image output. The third neural network 220 perfects the image. The data feeding module 212 refers to a part of the apparatus 202 that provides extra information, like how objects move or how light behaves, to aid in improving the image quality. The warper 214 refers to a part of the apparatus 202 that is responsible for adjusting or transforming the latent representation of previously rendered frames to make it useful for a set of neural networks 224.

[0068] In operation, the apparatus 202 is configured for real time ray-tracing image rendering in accordance with any embodiment or implementation form of the method 100 described above with reference to FIG. 1. The sampling recommendation module 204 is configured to provide the sampling recommendation for the frame to be rendered in real time. In accordance with an embodiment, the sampling recommendation module 204 comprises the first neural network trained for providing the sampling recommendation using the reinforcement learning technique or the deep learning technique. In such an embodiment, the first neural network is trained using the reinforcement learning technique or the deep learning technique in order to allow a sampling size of less than 1 sample per pixel (< 1 spp) on average, i.e. zero samples in some pixels. In an implementation, the sampling recommendation may be uniform, sparsely uniform (for upsampling), or adaptive. The sampling recommendation is obtained to improve quality and realism of an image.

[0069] The ray-tracing Tenderer 206 is configured to sample the pixel values in the frame according to the sampling recommendation. The ray-tracing Tenderer 206 supports in generating visually accurate lighting, shadows, and reflections by sampling the pixel values in the frame according to the sampling recommendation.

[0070] The latent state encoder 208 is configured to obtain the learned latent representation of the frame with the second neural network that is fed with the sampled pixel values and the latent representation of the previous rendered frame. The learned latent representation of the frame refers to the transformed latent representation of the previous rendered frame where the latent representation of the previous rendered frame is transformed to the learned latent representation of the frame by the way of processing and refining through the second neural network. The learned latent representation of the frame is processed and improved by the second neural network to be more informative and useful for tasks like image rendering by using information from the previous rendered frame. The second neural network plays a crucial role in the transformation of the latent representation of the previous rendered frame to the learned latent representation of the frame, supporting to capture relevant features and details from the previous rendered frame and applying relevant features and details to the frame. The learned latent representation of the frame in the 3D virtual environment yields higher image quality for a number of ray-tracing samples.

[0071] The decoder 210 is configured to process the learned latent representation of the frame with the third neural network to obtain the real-time rendering output. The third neural network is configured as the decoder (Q) in order to obtain the real-time rendering output of the frame.

[0072] In accordance with an embodiment, the data feeding module 212 is configured to feed additional data as additional argument to one or more of the first neural network, the second neural network and the third neural network, wherein the additional data comprises one or more of non-averaged sampled pixel values, motion vectors, albedos, normals and depths from one or more previously rendered frames. The non-averaged sampled pixel values refer to individual pixel values within the frame that haven't been averaged or combined with other pixel values of the frame. The non-averaged sampled pixel values provide detailed information about the colour and brightness of the pixels in the frame. The motion vectors represent how elements in the scene are moving from one frame to another frame. The motion vectors aid in capturing dynamics and movement in the image. The albedos refer to reflective properties of surface of a screen. The albedos indicate how much light is reflected by different materials in the scene. The normals refer to vectors that describe a direction in which the surfaces are facing. The normals aid in calculating lighting and shading effects. The depths refer to information that indicates how far objects are placed from the camera. The depths are crucial for creating realistic spatial effects and depth perception. The one or more of the first neural network, the second neural network and the third neural network are fed with the additional data or addition arguments in order to enhance the performance and quality of the one or more of the first neural network, the second neural network and the third neural network operations, ultimately leading to more realistic and detailed image rendering. The feeding of additional data facilitates these networks to make better-informed decisions when processing frames, resulting in improved real-time rendering outputs.

[0073] In accordance with an embodiment, the warper 214 is configured to warp the latent representation of one or more previously rendered frames and feeding the latent representation of the one or more previously rendered frames to the first neural network and / or the second neural network. The warping of the latent representation of one or more previously rendered frames corresponds to considering the latent representations of the one or more frames rendered before the frame and adjusting the latent representation of one or more previously rendered frames based on how objects or elements have moved in the scene. The warper 214 is configured to warp the latent representation of one or more previously rendered frames to generate adjusted latent representations of the one or more previously rendered frames. The adjusted latent representations of the one or more previously rendered frames are fed into the first neural network and / or the second neural network. By virtue of feeding the adjusted latent representations of the one or more previously rendered frames, the first neural network and / or the second neural network can take into account history of the scene, how the scene has evolved, and adjustments made to the one or more previously rendered frames information. The feeding the adjusted latent representations of the one or more previously rendered frames to the first neural network and / or the second neural network can lead to more accurate and context-aware processing, which ultimately contributes to improved quality of the ray-tracing image rendering output.

[0074] In accordance with an embodiment, the supersampling neural network is configured to upsample the real-time rendering output. The supersampling neural network corresponds to a specialized neural network that increases the resolution and quality of the final image output. The supersampling neural network enhances the overall visual quality. The upsampling of the real-time rendering output refers to an operation in which the real-time rendering undergoes an additional enhancement step that involves increasing a resolution or size of the frame, making the frame more detailed and visually refined.

[0075] In accordance with an embodiment, the sampling recommendation module 204 and the raytracing Tenderer 206 are configured to iteratively repeat the obtaining of the sampling recommendation and the sampling of the pixel values for two or more times during rendering of the same frame. The iterative repetition is carried out to refine and improve the rendering quality of the frame. Each iteration involves updating the sampling recommendation and sampling the pixel values based on a previous iteration. Such an iteration facilitates to optimize allocation of resources (sampling budget) for each of the frame, contributing to enhanced quality of the real-time rendering output. The apparatus 202 provides an efficient solution for real-time ray tracing image rendering and is characterized by all the benefits and advantages of the method 100 described above.

[0076] FIG. 3 A is a block diagram that depicts a computing device configured to carry out the method of real-time ray tracing image rendering. FIG. 3A is described in conjunction with elements from FIGs. 1 and 2. With reference to FIG. 3 A there is shown a block diagram 300A that depicts the computing device 302 configured to carry out the method 100 of real-time ray tracing image rendering.

[0077] The computing device 302 comprises one or more of a graphics processing unit, GPU, 304, a neural processing unit, NPU, 306, and a central processing unit, CPU, 308, configured to carry out the method 100 of real-time ray tracing image rendering.

[0078] The GPU 304 refers to a specialized hardware component designed to accelerate the processing of graphics and parallel computing tasks. The GPU 304 is essential for rendering complex images and performing calculations in real-time graphics applications. The GPU 304 is configured to rapidly calculate the complex interactions of the sampled pixel values with the objects in the scene, enabling realistic image rendering output. The GPU 304 excel in parallel processing, making it ideal for rendering tasks. The GPU 304 significantly speed up graphics rendering and simulations, resulting in faster and more immersive visuals.

[0079] The NPU 306 refers to a hardware component optimized for running artificial intelligence (Al) based enhancements in rendering and deep learning algorithms. In real-time ray tracing image rendering, the NPU 306 is configured to accelerate Al-based denoising techniques, improving the quality of rendered images by removing noise. The NPU 306 is highly efficient at running Al workloads, making the NPU 306 valuable for enhancing rendering quality through denoising and other Al-driven processes.

[0080] The CPU 308 refers to a central processor of the computing device 302, responsible for executing general-purpose tasks and managing the overall operation of the computing device 302. The CPU 308 is configured to manage an overall rendering process, coordinate tasks, and handle non-parallelized calculations in real-time ray tracing. The CPU 308 provide general- purpose computing power and control, ensuring the overall rendering process runs smoothly and efficiently. FIG. 3B is a block diagram that depicts a computer chip configured to carry out the method of real-time ray tracing image rendering. FIG. 3B is described in conjunction with elements from FIGs. 1 and 2. With reference to FIG. 3B there is shown a block diagram 300B that depicts the computer chip 310 configured to carry out the method 100 of real-time ray tracing image rendering.

[0081] The computer chip 310 refers to a small electronic component made of semiconductor material. The computer chip 310 comprises a processor 312 configured to carry out the steps of the method 100 of real-time ray tracing image rendering. Examples of implementation of the processor 312 may include but are not limited to a central data processing device, a microprocessor, a microcontroller, a complex instruction set computing (CISC) processor, an application-specific integrated circuit (ASIC) processor, a reduced instruction set (RISC) processor, a very long instruction word (VLIW) processor, a central processing unit (CPU), a state machine, a data processing unit, and other processors or circuitry.

[0082] FIG. 4 is a diagram that depicts an exemplary implementation of the method of real-time ray tracing image rendering. FIG. 4 is described in conjunction with elements from FIGs. 1 and 2. With reference to FIG. 4 there is shown a diagram 400 that schematically depicts the method 100 of real-time ray tracing image rendering, which for example can be implemented by the apparatus 202 of FIG. 2

[0083] With reference to FIG. 4, there are shown operations and states of the frame that can be used for real-time ray tracing image rendering, for example, the sampling recommendation module 204 provides the sampling recommendation for the frame to be rendered in real time. The sampling recommendation module 204 comprises the first neural network trained for providing the sampling recommendation using the reinforcement learning technique or the deep learning technique. Thus, the sampling recommendation module 204 generates a sample map 402 of the frame. Further, in accordance with the sample map 402 of the frame, the ray-tracing Tenderer 206 provides the sampling of the pixel values in the frame according to the sampling recommendation. The ray-tracing Tenderer 206 supports in generating visually accurate lighting, shadows, and reflections by sampling the pixel values in the frame according to the sampling recommendation. Thus, the ray-tracing Tenderer 206 generates a new sample map 404 of the frame providing the sampling of the pixel values in the frame. Further, the latent state encoder 208 obtains the learned latent representation of the frame with the second neural network that is fed with the sampled pixel values and the latent representation of the previous rendered frame. The learned latent representation of the frame is processed and improved by the second neural network to be more informative and useful for tasks like image rendering by using information from the previous rendered frame. The second neural network plays a crucial role in the transformation of the latent representation of the previous rendered frame to the learned latent representation of the frame, aiding to capture relevant features and details from the previous rendered frame and applying relevant features and details to the frame. The learned latent representation of the frame in the 3D virtual environment yields higher image quality for a number of ray-tracing samples. Optionally, the additional data 410 is also fed to the latent state encoder 208, wherein the additional data comprises one or more of non-averaged sampled pixel values, motion vectors, albedos, normals and depths from one or more previously rendered frames. The latent state encoder 208 is fed with the additional data 410 in order to enhance the performance and quality of the latent state encoder 208, ultimately leading to a more realistic and detailed image rendering state 406 of the frame. Optionally, the detailed image rendering state 406 of the frame undergoes a temporal delay 412 before inputting the detailed image rendering state 406 of the frame to the warper 214 and a warped state of the frame is fed to the latent state encoder 208. Further, the decoder 210 is configured to process the learned latent representation of the frame (that is, detailed image rendering state 406 of the frame) with the third neural network to obtain the real-time rendering output 408 of the frame. The third neural network is configured as the decoder 210 in order to obtain the real-time rendering output 408 of the frame.

[0084] Compared to existing methods, the method 100 excels in providing superior visual quality while maintaining lower latency, making the method 100 an ideal solution for real-time ray-traced rendering. The method 100 can be implemented in software post-deployment and further accelerated when paired with specialized hardware. The application scenarios for the method 100 are diverse and forward-thinking as the application may extend to the usability of ray tracing image rendering to emerging domains like Metaverse and virtual reality, where immersive and realistic visuals are paramount. Additionally, the method 100 may cater to the needs of more complex video games, even on hardware with cost-effective specifications, ensuring that a broader range of users can enjoy the benefits of ray tracing technology. Modifications to embodiments of the present disclosure described in the foregoing are possible without departing from the scope of the present disclosure as defined by the accompanying claims. Expressions such as "including", "comprising", "incorporating", "have", "is" used to describe and claim the present disclosure are intended to be construed in a non-exclusive manner, namely allowing for items, components or elements not explicitly described also to be present. Reference to the singular is also to be construed to relate to the plural. The word "exemplary" is used herein to mean "serving as an example, instance or illustration". Any embodiment described as “exemplary” is not necessarily to be construed as preferred or advantageous over other embodiments or to exclude the incorporation of features from other embodiments. The word "optionally" is used herein to mean "is provided in some embodiments and not provided in other embodiments". It is appreciated that certain features of the present disclosure, which are, for clarity, described in the context of separate embodiments, may also be provided in combination in a single embodiment. Conversely, various features of the invention, which are, for brevity, described in the context of a single embodiment, may also be provided separately or in any suitable combination or as suitable in any other described embodiment of the disclosure.

Claims

CLAIMS1. A method (100) of real-time ray tracing image rendering, the method comprising: obtaining a sampling recommendation for a frame to be rendered in real time, sampling pixel values in the frame with a ray-tracing Tenderer (206) according to the sampling recommendation, obtaining a learned latent representation of the frame with a second neural network, wherein the second neural network is configured as a latent state encoder (208) and wherein the latent state encoder (208) is fed with the sampled pixel values and a latent representation of a previous rendered frame, and processing the learned latent representation of the frame with a third neural network configured as a decoder (210) to obtain a real-time rendering output.

2. The method (100) of claim 1, wherein the sampling recommendation is obtained with a first neural network trained using a reinforcement learning technique or a deep learning technique.

3. The method (100) of claim 1 or 2, further comprising: feeding additional data as additional argument to one or more of the first neural network, the second neural network and the third neural network, wherein the additional data comprises one or more of: non-averaged sampled pixel values, motion vectors, albedos, normals and depths from one or more previously rendered frames.

4. The method (100) of any of claims 1 to 3, further comprising: warping a latent representation of one or more previously rendered frames, and feeding the latent representation of the one or more previously rendered frames to the first neural network and / or the second neural network.

5. The method (100) of any of claims 1 to 4, further comprising: upsampling the real-time rendering output with a supersampling neural network.

6. The method (100) of any of claims 1 to 5, wherein the obtaining of the sampling recommendation and the sampling of the pixel values are iteratively repeated for two or more times during rendering of the same frame.

7. An apparatus (202) for real-time ray tracing image rendering, the apparatus comprising: a sampling recommendation module (204) configured for providing a sampling recommendation for a frame to be rendered in real time, a ray-tracing Tenderer (206) configured for sampling pixel values in the frame according to the sampling recommendation, a latent state encoder (208) configured for obtaining a learned latent representation of the frame with a second neural network that is fed with the sampled pixel values and a latent representation of a previous rendered frame, and a decoder (210) configured for processing the learned latent representation of the frame with a third neural network to obtain a real-time rendering output.

8. The apparatus (202) of claim 7, wherein the sampling recommendation module (204) comprises a first neural network trained for providing the sampling recommendation using a reinforcement learning technique or a deep learning technique.

9. The apparatus (202) of claim 7 or 8, further comprising a data feeding module (212) configured for feeding additional data as additional argument to one or more of the first neural network, the second neural network and the third neural network, wherein the additional data comprises one or more of non-averaged sampled pixel values, motion vectors, albedos, normals and depths from one or more previously rendered frames.

10. The apparatus (202) of any of claims 7 to 9, further comprising a warper (214) configured for warping a latent representation of one or more previously rendered frames and feeding the latent representation of the one or more previously rendered frames to the first neural network and / or the second neural network.

11. The apparatus (202) of any of claims 7 to 10, further comprising a supersampling neural network configured for upsampling the real-time rendering output.

12. The apparatus (202) of any of claims 7 to 11, wherein the sampling recommendation module (204) and the ray-tracing Tenderer (206) are configured for iteratively repeating the obtaining of the sampling recommendation and the sampling of the pixel values for two or more times during rendering of the same frame.

13. A computing device (302) comprising one or more of a graphics processing unit, GPU (304), a neural processing unit, NPU (306), and a central processing unit, CPU (308), configured for carrying out the method (100) of real-time ray tracing image rendering of any of claims 1 to 6.

14. A computer chip (310) comprising a processer (312) configured for carrying out the method (100) of real-time ray tracing image rendering of any of claims 1 to 6.

15. A computer program comprising instructions which, when the computer program is executed by a computing device (302), cause the computing device (302) to carry out the method (100) of real-time ray tracing image rendering of any of claims 1 to 6.

16. A computer-readable medium comprising instructions which, when executed by a computing device (302), cause the computing device (302) to carry out the method (100) of any of real-time ray tracing image rendering of any of claims 1 to 6.

Images