View angle independent ray tracing by joint rendering

Abstract

There is described a method (1500) for ray tracing on a device (801) wherein the device (801) comprises a device neural network (1205), the method (1500) comprising: at the device (801): (1501) acquiring a virtual scene (204); emitting (1502) one or more rays of light (206, 207) from a single camera (201) of a single viewpoint (202) of the virtual scene (204); determining (1503) a location (1101) of a first rebound (206) of each of the one or more rays of light (206, 207); calculating (1504) a direct illumination of each of the one or more light rays (206, 207) at the location of the first rebound (206); obtaining (1505) a weight of a neural network of a server (802), where the neural network of the server (802) is trained to infer indirect illumination of each of the one or more rays (206, 207), the weight being obtained from the server (802); predicting (1506) indirect illumination of each of the one or more rays (206, 207) at the location of the first rebound (206) by inputting the weights into the device neural network (1205); a frame (1402) of the virtual scene is rendered (1507) according to the direct illumination and the indirect illumination of each of the one or more rays (206, 207). Using the weights of the neural network (1205) to predict the indirect illumination can reduce computational load on the device.

Description

Technical Field

[0001] This invention relates to an apparatus and method for ray tracing. Background Technology

[0002] With technological advancements, the demand for high-quality graphics in video games continues to rise. One way to achieve exceptional graphics is through ray tracing to obtain global illumination. The differences between standard direct lighting and global illumination can be found in [link to relevant documentation]. Figure 1 . Figure 1 The comparison between direct lighting and global lighting is shown. Global lighting image 102 shows reflections from the walls on the sides of the cube, while direct lighting image 101 does not have this effect.

[0003] Figure 2 An example of path tracing in the prior art is illustrated schematically. Ray tracing is a rendering technique in computer graphics that generates realistic global illumination images by simulating the propagation of light rays in a virtual environment 204. To run the algorithm, a virtual camera 201 is placed in the scene, and rays 206 and 207 are emitted from the camera into the scene. These rays are traced: they collide and bounce with objects 204, interact with the environment, until they reach a light source 203, are absorbed by a surface, or are terminated according to specific rules. At each bounce, a shadow ray 205 is projected from the surface intersection to the light source 203 to measure direct lighting. After the path terminates, all the radiance of each bounce is traced backward and accumulated until it reaches the origin of the camera 201. This iterative process effectively converts the global illumination of a single bounce into the indirect lighting of previous bounces. To ensure the accuracy of the accumulated results, the radiance is attenuated at each bounce based on the surface material of the intersection and the direction of the incident ray relative to the surface normal. The process approximates the light transport equation using a finite number of ray samples at each bounce. Path tracing uses only one ray per bounce, while ray tracing uses multiple rays per bounce.

[0004] Figure 2 The path tracing example in the diagram shows how rays 206 and 207 bounce several times before hitting light source 203. Solid arrows represent path trajectories 206 and 207, while dashed lines represent shadow rays 205 used to calculate the direct light's shadow for each bounce.

[0005] To accurately approximate global illumination, path tracing may require enormous computing power, which edge devices (i.e., mobile devices) do not always have.

[0006] There are many techniques that can approximate and bake global illumination to render more aesthetically pleasing images in real time, such as lightmaps and light probes. While these techniques are fast, they have limitations in handling dynamic content, may require a large amount of memory to store high-resolution textures, and can produce artifacts such as light leaks due to improper automatic probe placement.

[0007] Lightmaps are pre-computed textures used to store the light rays illuminating the scene's surface, primarily for static content. During the baking process, the 3D scene is rendered from multiple angles, storing the lighting information in the lightmap texture. This texture is then applied to the scene's surface, providing realistic lighting without requiring real-time calculations.

[0008] Light probes work by placing small spheres or cubes at various locations throughout the scene and in open spaces, capturing lighting information at each location. This information is then used to create texture maps, which can be interpolated at any location in world space to simulate the effect of global illumination. These probes can be updated dynamically, but are mostly for diffuse content because storing specular reflection information requires significant memory and path tracing calculations. Furthermore, probe placement can be critical, as errors at this stage can lead to visible light leaks. Light leaks occur when light probes from different lighting locations are mixed together, resulting in unrealistic artifacts.

[0009] A new technique for storing radiance data in neural networks is called Neural Radiance Cache (NRC). Figure 3 The processing flow of an existing NRC system is illustrated schematically. This technique effectively manages dynamic content while fundamentally eliminating interpolation problems such as light leakage. It supports both diffuse and specular lighting, but its ability to handle specular lighting may be limited. NRC approximates the emitted radiance of a scene surface by training a small neural network (NN). Within each frame, cache updates can be implemented by focusing only on the viewpoint of a single user and using a small number of data samples. One of the main differences between NRC and the previously mentioned traditional methods is that NRC may require path tracing during rendering. The primary goal of NRC is likely to improve path tracing efficiency by utilizing neural caches to prematurely terminate certain paths during secondary bounces. To terminate a path and query the NRC, a specific termination heuristic needs to be satisfied to avoid significant inaccuracies in the display cache. By querying the NRC, the algorithm can reuse previous spatiotemporal data to approximate the surface radiance at that location, thereby reducing the variance of the path tracker's estimates. A visual example of this path termination process can be found in [link to relevant documentation]. Figure 4 Path 207 in the middle.

[0010] Figure 4 An example of path tracing in an existing NRC system is illustrated. Figure 4 The diagram illustrates the case where path 207 terminates after several bounces in the NRC. The two top paths 207 are regular long paths, terminating at window 203 or light source 203 in this case. The two bottom paths 207 are shorter, terminating at termination position 401 based on the termination heuristic of NRC 402, and approximated by NRC 402.

[0011] The NRC process 300 includes the following steps: Step 1 301: A set of rays 206, 207 are emitted from the viewpoint of the user camera 201. Step 2: After several bounces, a portion of the rays 206, 207 are terminated at specific 3D locations 401. Step 3 302: NRC 402 is applied at these specific termination 3D locations 401 to predict their corresponding radiance. After terminating all remaining paths through path tracing, for each bounce, the algorithm backtracks all approximate radiances, including radiances from path tracing and radiances from NRC, to generate a noisy rendered image. Step 4 303: A small portion of the terminated rays are extended to train the cache, such as... Figure 5 As shown. Figure 5 An example of extended path tracing in a prior art NRC system is illustrated. This extension is achieved by reactivating a subset of the 3D positions 401 at the point of termination and performing more bounces until the extended 3D position 501 is reached. The NRC 402 is queried at the 3D position 501 at the point of termination to better approximate the radiance at the surface location. This process of training using neural network predictions is called self-training. Finally, the NRC makes final predictions at these extended 3D positions 501 and processes this data to generate training radiances, which are used to update the NRC for the next frame. All these steps are repeated frame-by-frame during the rendering runtime.

[0012] Figure 6 The architecture of an existing NRC system is illustrated schematically. The neural network architecture in NRC 402 includes a multilayer perceptron (MLP) 604, which comprises five hidden layers activated by rectified linear units (ReLU) between layers, with 64 channels per layer. The input to the neural network can include: • 3D position x: The precise 3D position on the surface, expressed in world space coordinates.

[0013] • Scattering direction vector w: The outgoing direction angle at the 3D position x in world space coordinates.

[0014] • Normal n: The normal at each 3D location in world space coordinates.

[0015] • Roughness r: The roughness of the material at each 3D location.

[0016] • Diffuse reflectance α: The primary color of the diffuse reflectance component at each 3D location.

[0017] • Specular reflectance β: The primary color of the specular reflection component at each 3D location.

[0018] A key aspect of the NRC scheme is likely how to represent the input data to establish a stronger linear relationship between it and the output radiance. This allows the neural network to find correlations more easily and quickly. To this end, the authors propose using the well-known Fourier feature 601 as the positional encoding of x, and as a one-blob encoding 602 for the orientation w, normal n, and roughness r. Diffuse reflectance and specular reflectance are considered sufficiently linear and are therefore directly input into the NN.

[0019] To further reduce the complexity of the problem, the NRC authors also proposed a reflectance decomposition technique. This method involves adding the diffuse reflectance α and the specular reflectance β, and multiplying the result by 605 with the NRC output 606. The goal is to avoid learning the primary color texture, thus simplifying the task of the neural network. In summary, the quadratic bounce data is first sent to the NRC. Secondly, the inputs x, w, n, and r are encoded. Finally, α and β are directly input and used for reflectance decomposition.

[0020] Existing technologies may have the following drawbacks: Rendering requires a path tracer and a denoiser: To enable fast, dynamic updates and avoid artifacts, NRC may only query the NN on the second bounce, thus necessitating the use of a path tracer and a denoiser during rendering. Therefore, this approach cannot be applied to edge devices due to their limited ray tracing capabilities.

[0021] Self-training: Since path-tracking rays may not terminate synchronously, and querying the neural network can be costly, self-training a large number of rays can increase latency and computational overhead. This issue may not be significant in the current NRC configuration, but its impact will become very prominent if more training samples are needed.

[0022] Limited applicability: NRC may be trained using single-view data, resulting in a high error rate in radiance estimates when queried from different angles. This can occur in areas not properly sampled at a single viewpoint, such as specular content, highly detailed diffuse content, and occluded areas relative to that viewpoint. However, this issue can be exaggerated, as Fourier feature location encoding tends to over-amplify interpolation errors in out-of-distribution data. Furthermore, even with multi-view training data, the cache quality for specular content may still be suboptimal because the NN encoding may not be designed for view-dependent materials.

[0023] Position Encoding: Unless a large amount of data is used to train the cache, using Fourier features as position encoding will generate a lot of axis alignment artifacts when querying NRC on the first bounce. This is because the network may be biased towards learning better axis alignment frequencies. The image queried on the first non-mirror bounce in the original paper is shown. Figure 7 An example of axis-aligned artifact 700 is shown. More pronounced perceptual artifacts can be observed in areas where more samples are needed due to drastic changes in orientation and spatial content (such as specular reflection vertices or detail shadows).

[0024] Monolithic encoding of specular reflection content: Monolithic encoding discretizes the scattered direction vector w into several bins, thus limiting its ability to capture specular reflection content. Glossy specular reflection content varies with angle, so using discrete values ​​to represent it would limit the capabilities of the neural network.

[0025] The outgoing direction *w* serves as input to the specular reflection content: this content contains a wealth of information because the radiance varies with the outgoing direction *w*. This means that even a small change in the outgoing direction *w* can lead to a significant change in the output radiance. Therefore, finding an input variable that is highly correlated with the output radiance is crucial for achieving smooth interpolation between the input and output values. Directly inputting the outgoing direction *w* into the neural network may not be the optimal approach for handling specular reflection content.

[0026] Difficulty in capturing subtle shadows at the first bounce vertex: Due to the extremely sparse path-tracking data used to train NRC (typically only 2% to 3% of the total paths), the network may struggle to approximate small details (such as subtle shadows) at the first bounce vertex. These shadows are crucial for approximating GI and achieving good perceptual quality in the final 3D scene.

[0027] There is an urgent need to develop a device and method that can solve the above problems. Summary of the Invention

[0028] According to a first aspect, a method for performing ray tracing on a device is provided. The device includes a device neural network, and the method includes: at the device: acquiring a virtual scene; emitting one or more rays from a single camera at a single viewpoint of the virtual scene; determining the position of the first bounce of each of the one or more rays; calculating the direct illumination of each of the one or more rays at the position of the first bounce; acquiring weights of a server neural network, wherein the server neural network is trained to infer the indirect illumination of each of the one or more rays, the weights being acquired from the server; predicting the indirect illumination of each of the one or more rays at the position of the first bounce by inputting the weights into the device neural network; and rendering frames of the virtual scene based on the direct illumination and the indirect illumination of each of the one or more rays. In this way, the computational load on the device can be reduced.

[0029] In some implementations, the position of the first bounce of each of the one or more light rays can be determined by rasterization. This method allows for more efficient position determination.

[0030] In some implementations, predicting the indirect lighting for each of the one or more light rays may include: inputting one or more ambient occlusion maps at the first bounce point into the device neural network and using the device neural network to predict the indirect lighting. This method allows for more accurate determination of indirect lighting.

[0031] In some implementations, predicting the indirect illumination for each of the one or more light rays may include: inputting the position of the first bounce into the device neural network and using the device neural network to predict the indirect illumination. In this way, indirect illumination can be determined more accurately.

[0032] In some implementations, rendering a frame of the virtual scene based on the direct and indirect lighting of each of the one or more light rays may include merging the direct and indirect lighting of each of the one or more light rays. This method enables high-precision rendering.

[0033] According to a second aspect, a method for generating weights for a server neural network is provided. The weights are used by a device in ray tracing. The method includes: at the server, training the server neural network to infer indirect illumination of one or more rays; and outputting the weights of the server neural network to the device, wherein the device uses the weights of the server neural network in ray tracing. This reduces the computational load on the device.

[0034] In some implementations, the server neural network can be generated and trained offline based on data from a bounded scene or voxels, using data from a training camera layout comprising multiple training cameras arranged at different viewpoints. In this way, the server neural network can be trained to predict indirect lighting from multiple viewpoints.

[0035] In some implementations, the server neural network can be further trained online using data from a virtual scene as input, based on a single camera positioned at a single viewpoint, the virtual scene and the single camera positioned at the single viewpoint being the same as those used on the device. In this way, the server neural network can be further trained to predict indirect lighting adapted to the virtual scene.

[0036] In some implementations, the server neural network can be further trained online based on data from the bounded scene or from the voxels, using data from the training camera layout, which includes multiple training cameras arranged at different viewpoints. In this way, the server neural network can be further trained to predict indirect lighting from multiple viewpoints.

[0037] In some implementations, the optimal training camera layout is determined based on an optimization algorithm that optimizes the training camera layout based on the path tracing ray budget. This approach generates an improved training layout, thereby maximizing training quality while reducing computational load.

[0038] In some implementations, the server neural network can be generated and trained by: emitting one or more light rays from each of the training cameras at the corresponding viewpoint; determining the position of the first bounce of each of the one or more light rays emitted from each of the training cameras; and predicting the indirect illumination of each of the one or more light rays emitted from each of the training cameras. In this way, the server neural network can be trained by predicting indirect illumination.

[0039] In some implementations, the server neural network can be further trained by predicting the indirect lighting of the virtual scene and updating the weights of the server neural network for the indirect lighting based on the indirect lighting of the virtual scene. In this way, the device can use the updated weights.

[0040] In some implementations, when the difference between one or more values ​​of the indirect lighting in the virtual scene and the corresponding value of the indirect lighting in the server neural network is greater than a threshold, the weights of the server neural network used for the indirect lighting can be updated. In this way, the weights used by the device can be adapted to the virtual scene.

[0041] In some implementations, the indirect lighting of the virtual scene is predicted using random Fourier features as the position encoding input to the server neural network. In some implementations, the indirect lighting of the virtual scene is predicted using spherical harmonic functions as the direction encoding and reflection view input to the server neural network. In some implementations, the indirect lighting of the virtual scene is predicted using an ambient occlusion decomposition map as the input to the server neural network. In some implementations, the indirect lighting of the virtual scene is predicted by multiplying the ambient occlusion decomposition map by the output of the server neural network. These methods can improve the prediction performance of the server neural network.

[0042] According to a third aspect, an apparatus for ray tracing is provided. The apparatus includes a device neural network, one or more processors, and a memory storing program code, wherein the program code is executable by the one or more processors, such that the apparatus is configured to: acquire a virtual scene; emit one or more rays from a single camera at a single viewpoint of the virtual scene; determine the position of the first bounce of each of the one or more rays; calculate the direct illumination of each of the one or more rays at the position of the first bounce; acquire weights of a server neural network, wherein the server neural network is trained to infer the indirect illumination of each of the one or more rays, the weights being acquired from a server; predict the indirect illumination of each of the one or more rays at the position of the first bounce by inputting the weights into the device neural network; and render frames of the virtual scene based on the direct illumination and the indirect illumination of each of the one or more rays.

[0043] According to a fourth aspect, a server is provided for generating weights for a server neural network. The weights are used by a device in ray tracing. The server includes one or more processors and a memory storing program code, wherein the program code is executable by the one or more processors, such that the server is configured to: train the server neural network to infer indirect illumination of one or more rays; and output the weights of the server neural network to the device, wherein the device is configured to use the weights of the server neural network in ray tracing.

[0044] According to a fifth aspect, a computing device is provided, including the aforementioned equipment and the aforementioned server. In this way, the aforementioned equipment and the aforementioned server can form a computing system. Attached Figure Description

[0045] The invention will now be described by way of example with reference to the accompanying drawings. In the drawings: Figure 1 The comparison results between direct lighting and global illumination are shown.

[0046] Figure 2 An example of path tracing in the prior art is illustrated schematically.

[0047] Figure 3 The schematic diagram illustrates the processing flow of an NRC system in the prior art.

[0048] Figure 4 An example of path tracing in an existing NRC system is illustrated.

[0049] Figure 5 An example of extended path tracing in an existing NRC system is illustrated.

[0050] Figure 6 The architecture of an NRC system in the prior art is illustrated schematically.

[0051] Figure 7 An example of axis-aligned artifacts is shown.

[0052] Figure 8 The processing flow of the system in this application is illustrated schematically.

[0053] Figure 9 The layout of the training camera in this application is illustrated schematically.

[0054] Figure 10 An example of training path tracking in this application is illustrated schematically.

[0055] Figure 11 An example of rendering path tracing in this application is illustrated schematically.

[0056] Figure 12 The architecture in this application is illustrated schematically.

[0057] Figure 13 An example of a reflection direction vector is shown.

[0058] Figure 14 The results show a comparison of the rendering frames of the NRC's existing technology system and those of this application.

[0059] Figure 15 An example of a computer-implemented method for ray tracing is shown.

[0060] Figure 16A and Figure 16B An example of an apparatus for performing the methods described herein is shown. Detailed Implementation

[0061] The apparatus and methods described in this paper relate to ray tracing.

[0062] Implementations of this system can address one or more of the previously mentioned problems by obtaining weights from a server for a neural network trained to infer indirect illumination for each of one or more light rays; and using the weights of the neural network to predict indirect illumination. This approach reduces the computational load on the device.

[0063] To overcome the problems of existing technologies, this system can pre-compute and compress a portion of global illumination on the server and then send it to the edge device, thus forming a practical joint rendering scheme. This system can bake indirect lighting onto a set of neural network weights on the server, send the neural network weights to the device, calculate direct lighting on the device side, obtain indirect lighting from the neural network weights on the device side, and combine the results on the device side to obtain the final image.

[0064] This system approximates radiance at the surface level, similar to lightmaps, but through a neural network. This allows for higher compression ratios, enabling the system to bake indirect lighting at the first bounce and efficiently approximate indirect lighting from every angle and surface location. This can potentially generate a Multiview Reflected Radiance Cache (MRRC). This approach avoids light leakage, is suitable for dynamic content, and can be shared by all users within the scene to support both diffuse and specular content.

[0065] MRRC employs a different neural network architecture than NRC. The two methods may be suitable for different scenarios because NRC focuses on single-view content, storing global illumination, querying the neural network during the second bounce, employing a self-training approach, and relying on path tracers and denoisers during rendering. However, MRRC may focus on multi-view caching, storing indirect illumination, querying the neural network during the first bounce, and generating the final image without self-training or relying on path tracers or denoisers. All these characteristics may make MRRC suitable for real-time baking of indirect illumination radiance in rendering server-device joint solutions.

[0066] This system may have the following advantages: Use the Neural Network (NN) as a view-independent cache to avoid using a path tracer during rendering.

[0067] By querying the neural network during the first bounce, this system avoids the edge device's reliance on the path tracker. This is because the system can efficiently acquire master vertex information on the edge device using computationally inexpensive techniques (such as rasterization) while avoiding any neural network image quality artifacts that may occur during the first bounce.

[0068] Avoid artifacts during the first rebound by using rapid, dynamic updates.

[0069] This can be achieved through the following two approaches: Random Fourier feature encoding is used instead of standard Fourier features for position encoding. Standard Fourier features can be described by Equation 1: v), v), …](1) Here, j ranges from 0 to m–1, v is the input vector, and σ is a hyperparameter. This method uses log-linear interval frequencies to distinguish each dimension, but all frequencies may be centered on the same axis, which could cause the neural network to be biased towards learning these frequencies better, potentially leading to problems such as… Figure 7 The visible artifacts are shown. This can be addressed by replacing these codes with the random Fourier features shown in Equation 2: Bv), (2) Here, v is an input vector of dimension d, and B is a random matrix of dimension m×d, which is sampled from an isotropic Gaussian distribution N(0, σ), where σ is a hyperparameter. Since B is a random matrix and there is a linear combination between B and v, the frequency distribution in the spectrum is more uniform, reducing the bias towards certain frequencies, thus potentially eliminating the axis-aligned visible artifacts generated by the neural network.

[0070] By assuming multiple users train and query the same neural network, the model is trained using multi-view data instead of single-view data. This data better covers the world space, reduces model uncertainty, and thus generates fewer artifacts.

[0071] Using NN as a view-independent cache, high-quality specular reflection content is achieved.

[0072] In multiplayer games, several players share the same 3D content from different perspectives. By storing diffuse and specular content in a single neural network, the system described above can define a view-independent cache. This cache can store all views for any material and can be queried from every location and viewpoint in the scene.

[0073] NRC can use block coding, which discretizes the scattering direction vector w, thus limiting its ability to capture specular reflections. In this method, the system encodes this direction vector using spherical harmonics. These spherical harmonics are a set of orthogonal basis functions defined on the surface of a sphere. Since the direction vector w is a 3D vector, these codes can create a stronger linear relationship between the data and the output radiance, making the interpolation task of the NN easier.

[0074] Furthermore, the aforementioned system can convert the outgoing direction vector w into a reflection direction vector w_r instead of providing the outgoing direction vector w. This conversion can be achieved using the normal vector n at that point. n)n–w. The result can be found in the following... Figure 13 Visualization in Chinese. Figure 13 An example of a reflection direction vector is shown. This variation makes interpolation of the specular reflection content easier, because a small change in w_r may be directly related to the region where the incident light energy is most concentrated.

[0075] These two changes may enable the NN to accurately and easily approximate mirrored content, thus achieving high-quality caching.

[0076] By using the Neural Network (NN) as a view-independent cache, fast NN adaptation to dynamic content can be achieved.

[0077] Updating specular reflections frame-by-frame can require a significant amount of ray tracing calculations, which are typically not available during rendering. This is why previous view-independent caching only applied to diffuse content: by definition, diffuse content remains constant across different viewpoints and only requires sparse path tracing sampling of the 3D scene.

[0078] In this system, the scene may differ from a conventional single-view frame because it assumes multiple users are querying and training the same neural network. Therefore, more path tracing resources are allocated by default to cover the 3D scene from multiple angles, making it easier to capture specular reflections. Furthermore, the system can employ intelligent camera placement algorithms, a high-quality multi-view pre-training stage, and adaptive scene sampling techniques to efficiently reduce the real-time path tracing ray budget.

[0079] These methods, combined with random Fourier features, reflection direction vector w_r, and spherical harmonic function encoding, enable MRRC to interpolate specular reflection radiance content with far fewer samples than previous methods.

[0080] Improve neural approximation by storing only indirect illumination.

[0081] By separating direct and indirect lighting, the above system can significantly reduce the amount of information that the neural network needs to learn. Figure 2 In the diagram, the dashed line 205 represents the ray that strikes the light source for direct illumination, also known as a shadow ray. Indirect illumination originates from the solid line formed after the first bounce.

[0082] Because indirect lighting involves more bounces, it typically includes far fewer high-frequency components than direct lighting. This lower frequency content allows the aforementioned system to update MRRC frame-by-frame much faster. This is because the neural network initially converges to low-frequency content and requires more time to learn the high-frequency components.

[0083] Direct illumination from point and directional light sources can be computed at low cost in edge devices, so this system does not need to store direct illumination in each device.

[0084] Use the Neural Network (NN) as a view-independent cache to avoid using a denoiser during rendering.

[0085] Thanks to the neural network (NN) encoding and multi-view training of this system, the quality level of approximate radiance in MRRC can be relatively high. This allows the NN to be queried at the first bounce to generate indirect lighting without displaying any artifacts. Queries at the first bounce avoid rendering noisy images. Other neural methods, such as NRC, may not be able to directly generate indirect lighting at the first bounce. These methods might query global illumination at the second bounce and then use a path tracer to accumulate global illumination at the first bounce. Due to the inherent randomness of path tracers in emitting rays, the final image may be noisy, requiring a denoiser during rendering.

[0086] Improve neural approximation using ambient occlusion mapping.

[0087] Accurate shading of 3D models is crucial for reconstructing object depth perception. To this end, this system can use ambient occlusion (AO) mapping to guide MRRC in the following two ways: The AO map is used as input to the NN. This input enables MRRC to easily identify shadow regions, especially small shadow regions that are difficult to sample sufficiently.

[0088] By performing AO decomposition, the MRRC output is multiplied by the AO map. Although AO is a coarse approximation of GI, this decomposition significantly improves perceptual quality and reduces reconstruction error.

[0089] The term "radiance" can be defined as the total amount of light energy emitted, reflected, or transmitted by a surface in a specific direction. It can be used to measure the brightness of light emitted or reflected by a surface, and is usually measured in watts per square meter per steradian (W / m² / sr).

[0090] The term "path tracing" can be defined as a computer graphics rendering technique used to simulate the behavior of light in a virtual scene. It works by tracing the path of light rays as they bounce around the scene, interacting with objects and surfaces, until the light reaches a light source, is absorbed by a surface, or is terminated heuristically. Path tracing can be approximated by ray tracing, where only one ray is emitted for each bounce.

[0091] The term "light transport equation" can be defined as a mathematical equation used to describe how light propagates through a medium such as air, water, or a solid object. The light transport equation takes into account the interaction of light with the medium, including absorption, scattering, and emission. This equation may be the one that path tracing aims to approximate.

[0092] The term "diffuse light" can be defined as light that is scattered in many different directions, rather than light focused in a single beam or direction. This light may be independent of the viewing angle because, at a given surface location, it reflects the same radiance in every outgoing observation direction.

[0093] The term "spectral reflection" can be defined as light that is reflected from a smooth surface, such as a mirror or a polished metal object. This reflection can produce bright, focused highlights that appear as a mirror image of the light source. This light may be independent of the viewing angle because, at a given location on the surface, it can reflect different levels of radiance in every direction of observation.

[0094] The term "direct lighting" can be defined as light that shines directly onto a surface from a light source. This is likely the only lighting component that most mobile video games can support for real-time rendering on edge devices.

[0095] The term "indirect lighting" can be defined as light emitted from a light source, which bounces at least once before illuminating a surface. By definition, the source of these rays can be any object other than a light source. Indirect lighting is computationally very expensive, therefore approximate and pre-calculated calculations are required in most video games.

[0096] The term "global illumination" can be defined as all light rays that come from both direct and indirect illumination.

[0097] The term "neural network" can be defined as a machine learning algorithm that uses a specific differentiable architecture to approximate a continuous function. The network can be trained on a dataset with the goal of updating the weights of the architecture to minimize the error between the predicted output and the ground truth output.

[0098] The term "view-independent cache" can be defined as "view-independent" because it can store the radiance of all surface locations in the scene under all viewpoints and uses the same data structure.

[0099] The term "first bounce" can be defined by the light rays emitted by the camera; the first bounce refers to the initial point of intersection of the surfaces with these light rays.

[0100] The term "double bounce" can be defined by the light rays emitted from the camera; a double bounce refers to all the intersections of surfaces that intersect after the first bounce.

[0101] The term "self-training" can be defined as using neural network predictions as ground truth values ​​for training the neural network. This approach is particularly convenient when a single ground truth value can be generated by mixing real ground truth values ​​with a small number of neural network predictions.

[0102] Multiview Reflected Radiance Cache (MRRC) can be a type of neural network that focuses on caching the indirect radiance of diffuse and specular content across all viewpoints. MRRC can approximate radiance values ​​at the surface level, similar to lightmaps, and can efficiently handle dynamic content. Furthermore, in co-rendering scenarios, MRRC is trained on a server and its neural network weights are sent to the device. MRRC may not require a path tracker or denoising unit on the device side because the neural network is likely to be invoked only on the first bounce.

[0103] The goal of MRRC is to become a view-independent cache, allowing a single neural network (NN) to be shared by multiple users from each viewpoint, whether it's a single bounded scene or voxels within a large scene. Previous dynamic view-independent caches could only handle diffuse content because the real-time computation and memory costs required to incorporate specular content could be very high. Nevertheless, MRRC significantly reduces these costs thanks to a series of technical measures: defaulting to multi-view data, employing intelligent camera placement algorithms, starting from a pre-trained initial checkpoint, applying adaptive scene sampling, using NN encoding specifically designed for specular content, and employing AO decomposition in its architecture.

[0104] Figure 8 The processing flow 800 of the system in this application is illustrated schematically. The processing flow 800 may include three stages.

[0105] The first phase may include offline MRRC initialization. This phase can be executed on a server (802). The offline MRRC initialization phase can take a bounded scene or voxels as input, and the complexity of its indirect lighting can be appropriately approximated by the capacity of the MRRC neural network. In this 3D environment, a set of cameras can be strategically positioned using an intelligent camera placement algorithm to efficiently capture the radiance field. Subsequently, a high-quality path tracker is applied to each camera, and the MRRC model is pre-trained using data generated at the first bounce until convergence.

[0106] The second phase can include online MRRC fine-tuning. This phase can be executed on a server 802. During game execution, the online MRRC fine-tuning phase can run continuously, adapting the neural network to lighting changes for each frame. The adaptive scene sampling process allocates a ray budget to each of several predefined cameras while keeping the total real-time path tracing ray budget fixed and finite. Each camera's task might be to generate view-dependent indirect lighting at the first bounce. Depending on the scene complexity, multiple GPUs can be used simultaneously to run independent cameras. Because cameras emitting a predetermined number of rays are used, the first bounce can be efficiently computed via rasterization. Subsequently, secondary bounces and their corresponding radiance values ​​can be path-traced based on their allocated ray budgets and optionally approximated using any pre-existing global illumination estimation technique. Once the training data is ready, the MRRC weights can be updated through a finite number of training steps.

[0107] These two phases, namely MRRC offline initialization and MRRC online fine-tuning, can be performed on server 802 to generate MRRC NN weights to be sent to multiple devices 801. Once the NN weights are received on the devices, rendering of frames with global illumination can begin.

[0108] The third stage may include rendering frames using MRRC. The third stage may be performed on device 801. MRRC may include a neural network. The neural network may be trained in the first stage. The neural network may be further trained in the second stage. The neural network may be used to render frames in the third stage. Server 802 may generate and / or train the neural network to infer the indirect illumination of each of one or more rays emitted by a single camera 201 (as described herein).

[0109] The first stage may include the steps of placing the smart camera 803. Figure 9 The training camera layout 900 of this application is schematically illustrated. Before initiating real-time rendering, each bounded scene 204 or voxel can be thoroughly scanned in offline mode to determine the optimal placement of the camera 201, thereby intelligently covering the areas most important in terms of indirect lighting. This process is defined as a resource allocation optimization algorithm that considers the finite real-time path tracing ray budget allocated to each scene, as well as the amount of radiance detail estimated from each surface. The output of the algorithm is a set of camera 201 positions with different resolutions that effectively minimize the expected average error between two algorithms: a brute-force camera placement algorithm using an extremely high path tracing ray budget and an algorithm using a manually set finite real-time path tracing ray budget.

[0110] Server 802 can retrieve bounded scenes or voxels. For example... Figure 9 As shown, multiple training cameras 201a to 201e can be arranged at different viewpoints, i.e., viewpoints 1 to N. The training cameras 201a to 201e can be distributed among multiple GPUs 901a to 901c. For example, as... Figure 9As shown, each GPU 901a to 901c may include one or more training cameras 201a to 201e. Optionally, a single GPU 901 may include all training cameras 201a to 201e. Server 802 can determine the optimal training camera 201a to 201e layout. Server 802 can determine the optimal training camera 201a to 201e layout based on an optimization algorithm. The optimization algorithm can be used to optimize the training camera layout based on a path tracing ray budget. The optimization algorithm may attempt to minimize the expected average error between two algorithms: a brute-force camera placement algorithm using an extremely high path tracing ray budget and an algorithm using a manually set limited real-time path tracing ray budget. The path tracing ray budget can be predetermined. The path tracing ray budget can be set according to the computational requirements on the server. The path tracing ray budget can depend on the scene content. When the path tracing ray budget is high, the cameras can be placed more evenly, and more light can be allocated to each camera. This scene may resemble a brute-force approach or an approach that does not prioritize specific domains. When the path tracing budget is low, the number of cameras may be significantly reduced. A few cameras can be placed in the center to capture the entire scene in a globally low-resolution manner; at the same time, a sparsely arranged group of cameras can be specifically aimed at particular high-frequency areas (i.e., areas rich in detail) or specular reflection areas (i.e., areas with view-dependent content). Depending on the amount of detail on each surface, each camera may emit more or less light.

[0111] The first stage can include the steps of high-quality path tracing 804. After defining cameras 201a to 201e and still in offline mode, a pre-trained MRRC model can be generated using a standard path tracker with an ultra-high number of samples per pixel. This approach can provide robust initial checkpoints for the neural network while minimizing the adaptation time for dynamic regions, thus generating high-quality details for static regions.

[0112] Server 802 can emit one or more rays 206 from each training camera 201a to 201e, each with its own viewpoint (viewpoint 1 to N). Server 802 can determine the position of the first bounce of each of the one or more rays 206 emitted from each training camera 901a to 901e. The position of the first bounce can be a 3D position. The position of the first bounce can be located on an object 204 in a bounded scene or voxel.

[0113] During this training phase, server 802 can predict the indirect lighting of each ray of one or more rays 206 emitted from each of the training cameras 201a to 201e. Server 802 can predict the indirect lighting of each ray of one or more rays 206 using any suitable method described in the prior art. Server 802 can be used to predict indirect lighting using random Fourier features as the position encoding input to a neural network. This avoids visual artifacts when adjusting the neural network with fewer path-tracking samples. Server 802 can be used to predict indirect lighting using spherical harmonic functions as the orientation encoding and reflection view input to a neural network. This may make it easier for the neural network to approximate and interpolate specular reflection content. Server 802 can be used to predict indirect lighting using an ambient occlusion decomposition map as the input to a neural network. Server 802 can be used to predict indirect lighting by multiplying the ambient occlusion decomposition map with the output of the neural network. This can reduce the complexity of the problem, improve perception quality, and reduce reconstruction errors.

[0114] Server 802 can generate and train a neural network using data from a bounded scene or data from voxels as input. Server 802 can train the neural network based on data from training camera layouts 201a to 201e. Server 802 can generate and train the neural network using predicted indirect lighting for each of one or more rays 206 emitted from training cameras 201a to 201e. By training the neural network based on indirect lighting from a series of viewpoints (viewpoints 1 to N), the neural network can later be used to infer indirect lighting from a series of viewpoints. In this way, the neural network can be viewpoint-independent. Server 802 can train the neural network until convergence. Server 802 can generate the weights of the neural network during training.

[0115] The second phase may include the adaptive scene sampling step 806. Once real-time rendering begins, the MRRC needs to be updated frame-by-frame to adapt to changes in radiance. Multiple training cameras 201a to 201e can simultaneously perform path tracing with several GPUs 901a to 901c, each with a limited real-time ray budget. For each timestamp, radiance can be pre-stored and then accumulated in the spatiotemporal dimension using an explicit voxel hash grid in the world space coordinate system. Due to this spatiotemporal hash grid caching, it may be impossible to measure the largest differences in radiance values ​​across the temporal and spatial dimensions within the scene. The adaptive scene sampling algorithm can use this difference information for a dual purpose: The path tracing algorithm is guided in the time dimension by dynamically allocating the light budget to each camera 201a to 201e.

[0116] Only the radiance value with the largest difference is selected to update the MRRC, thereby shortening its convergence time.

[0117] The second phase may include the step of efficient path tracing 807. Server 802 can emit rays from each of the training cameras 201a to 201e. Server 802 can perform path tracing for each of the allocated ray budgets. In this further training phase, server 802 can predict the indirect lighting of each of the one or more rays 206 emitted from each of the training cameras 201a to 201e. Server 802 can predict the indirect lighting of each of the one or more rays 206 using any suitable method described in the prior art. Server 802 can be used to predict indirect lighting using random Fourier features as the position encoding input to a neural network. This avoids visual artifacts when adjusting the neural network with fewer path tracing samples. Server 802 can be used to predict indirect lighting using spherical harmonic functions as the orientation encoding and reflection view input to a neural network. This may make it easier for the neural network to approximate and interpolate specular reflection content. Server 802 can be used to predict indirect lighting using an ambient occlusion decomposition map as the input to a neural network. Server 802 can be used to predict indirect lighting by multiplying the ambient occlusion decomposition map with the output of a neural network. This can reduce the complexity of the problem, improve perception quality, and reduce reconstruction errors.

[0118] The second phase may include the step of MRRC fine-tuning 808. Server 802 may also train the neural network based on data received by the device from the virtual scene (as described herein). Server 802 may also train the neural network based on data from a single camera 201 positioned at a single viewpoint (as described herein). Server 802 may use predicted indirect lighting (as described herein) for each of one or more rays 206 emitted from the single camera 201 at the single viewpoint. By further training the neural network based on the indirect lighting of the single camera viewpoint, the neural network can be used for a specific user's camera 201 at the single viewpoint. In this way, the neural network can be adapted to a specific user's camera 201 at the single viewpoint. Server 802 may further train the neural network until convergence. Server 802 may generate updated weights for the neural network during further training.

[0119] Server 802 can further train the neural network by predicting indirect lighting in the virtual scene (as described herein) and update the weights of the neural network for indirect lighting based on the indirect lighting of the virtual scene. In other words, whenever the neural network predicts indirect lighting in the virtual scene, it is trained based on information in the virtual scene. In this way, the neural network's inference of indirect lighting in the virtual scene can be improved. Server 802 can update the weights of the neural network for indirect lighting when the difference between one or more values ​​of indirect lighting in the virtual scene and the corresponding values ​​of indirect lighting in the neural network is greater than a threshold. In other words, weights may only be updated when there is a significant change in indirect lighting between frames. In this way, the required computation can be reduced.

[0120] Server 802 can further train the neural network using data from the bounded scene or data from voxels as input. Server 802 can further train the neural network based on data from training camera layouts 201a to 201e. Server 802 can further train the neural network using predicted indirect lighting for each of one or more rays 206 emitted from training cameras 201a to 201e. By further training the neural network based on indirect lighting from a series of viewpoints (viewpoints 1 to N), the neural network can later be used to infer indirect lighting from the series of viewpoints. In this way, the neural network can be viewpoint-independent. Server 802 can further train the neural network until convergence. Server 802 can generate updated weights for the neural network during further training.

[0121] Figure 10 A training path tracking example 1000 of this application is illustrated schematically. The training path tracking example 1000 can be used in one or both steps of the generation and training, and further training, as described herein. Each time one of the training cameras 201a to 201e is selected, one or more light rays 206, 207 can be emitted from camera 201. The first bounce 206 can be determined as described herein. The second bounce 207 can be determined as described herein. Indirect illumination can be determined based on the second bounce 207 as described herein.

[0122] The process of generating a frame on device 801 can include several steps. First, rasterization can be used to obtain the 3D location of the first bounce. Then, a required set of graphics data (normals, roughness, and reflectivity) can be retrieved based on these 3D locations, and direct lighting and ambient occlusion (AO) maps can be computed on the device. Simultaneously, MRRC can be performed at these 3D locations to predict indirect lighting. Finally, a rendered frame can be generated by merging the direct and indirect lighting.

[0123] Figure 11 An example 1100 of rendering path tracing in this application is illustrated schematically.

[0124] The third stage may include the device-side rendering step 809. Device 801 can acquire virtual scene 204. Virtual scene 204 can be acquired from server 802. For example, if device 801 is a user device 801 participating in a game, device 801 can receive the game's virtual scene from server 802. Virtual scene 204 may include one or more objects 204 (such as...). Figure 2 (As shown).

[0125] Device 801 can emit one or more light rays 206, 207 from a single camera 201. Camera 201 can be located at a single viewpoint 202. Light rays 206, 207 can contact one or more objects 204. Light rays 206, 207 can contact object 204 at the location 1101 of the first bounce 206. Device 801 can determine the location 1101 of the first bounce 206. Device 801 can determine the location 1101 of the first bounce 206 for all emitted light rays 206, 207. The device can determine the location 1101 of the first bounce 206 by rasterization. In this way, the computational load on device 101 can be reduced. The location 1101 of the first bounce 206 can be a 3D location.

[0126] Device 801 can calculate the direct illumination of each of one or more rays 206, 207 at the position 1101 of the first bounce 206. Device 801 can use any suitable method in the prior art to calculate the direct illumination.

[0127] Device 801 can calculate one or more ambient occlusion (AO) maps at the first bounce. Device 801 can use any suitable method in the prior art to calculate the ambient occlusion (AO) maps.

[0128] like Figure 11 As shown, device 801 may include MRRC 1102. Device 801 may input the position 1101 of the first bounce 206 into MRRC 1102.

[0129] The third stage may include the steps of MRRC inference 810. Device 801 can acquire the weights of a neural network, wherein the neural network is trained to infer the indirect illumination of each ray in one or more rays. Device 801 can acquire the weights from server 802. As described herein, the weights can be determined by server 802. Device 801 can receive weights from server 802 in each frame. Device 801 can receive weights and predefined intervals from server 802. Device 801 can receive weights from server 802 based on whether the weights have been updated since the previous frame. Server 802 can output the weights. Server 802 can output the weights to device 801. Server 802 can output the weights to multiple devices 801. Server 802 can output the weights to multiple devices 801, wherein each device shares the same bounded scene or voxels. In this way, multiple users using the bounded scene or voxels can share these weights.

[0130] Device 801 can predict the indirect illumination of each of one or more rays 206, 207. Device 801 can predict indirect illumination via MRRC 1102.

[0131] Figure 12 The MRRC architecture 1102 of this application is illustrated schematically. The MRRC architecture 1102 is built upon the NRC architecture. The MLP components are modified to include only two hidden layers, each with 128 channels. This adjustment was made to improve the accuracy of the optical transport approximation while maintaining comparable training and inference latency. The input variables of the NN are largely consistent with those of the NRC, with two differences: the type of training data is limited to first bounce data, and ambient occlusion maps are used as input variables and output guidance. Furthermore, completely different NN encoding can be employed to facilitate the learning of specular reflection content from multi-view data.

[0132] MRRC 1102 can acquire the initial bounce data 1102. The initial bounce data 1102 can be input into the reflection view 1201. Subsequently, the output of the reflection view 1201 can be input into the spherical encoder 1203. The initial bounce data can be input into the random Fourier feature encoder 1202. Finally, the inputs x, w, n, and r complete the encoding.

[0133] MRRC may include a neural network 1205. The neural network 1205 can be used to predict the indirect illumination of each of one or more rays 206, 207. The neural network 1205 can predict the indirect illumination of each of one or more rays 206, 207 at the location of the first bounce 206. Device 801 can use the neural network 1205 to predict indirect illumination.

[0134] Input 1204 can be fed into neural network 1205. Inputs 1204 x, w, n, r, inputs α and β, and AO can be fed into neural network 1205. The position 1101 of the first bounce 206 can be fed into neural network 1205. Device 801 can predict indirect lighting by feeding the position 1101 of the first bounce 206 into neural network 1205 and then using neural network 1205 to predict indirect lighting. An ambient occlusion (AO) map can be fed into neural network 1205. Device 801 can predict indirect lighting by feeding the ambient occlusion (AO) map at the first bounce 206 into neural network 1205 and then using neural network 1205 to predict indirect lighting. Neural network 1205 can be used to predict indirect lighting by using spherical harmonic functions as the directional encoding and reflection view input of the neural network. This may make it easier for the neural network to approximate and interpolate specular reflection content. The neural network 1205 can be used to predict indirect lighting by using the ambient occlusion decomposition map as input to the neural network 1205. The neural network 1205 can also be used to predict indirect lighting by multiplying the ambient occlusion decomposition map by the output of the neural network 1205. This can reduce the complexity of the problem, improve perception quality, and reduce reconstruction errors.

[0135] To recover the indirect illumination L_i, MRRC can be trained using a differential end-to-end method, as shown in Equation 3: (3) The third stage may include the step of global illumination image 811. Device 801 can render frames of virtual scene 204 based on the direct and indirect illumination of each ray in one or more rays 206, 207. Device 801 can be used to render frame 1402 based on the direct and indirect illumination of each ray in one or more rays 206, 207 by merging the direct and indirect illumination of each ray in one or more rays 206, 207. Merging direct and indirect illumination can be represented by the optical transport equation. The optical transport equation has been linearly decomposed into direct components. and indirect components As shown in Equation 4: (4) Figure 14 The comparison results of the rendering frames of NRC's prior art system 1401 and this application 1402 are shown. Figure 14 Quantitative and qualitative results are shown between multi-view NRC and MRRC. The same training data and protocol were used for both scenarios.

[0136] To directly compare the advantages of architectural changes between MRRC and multi-view NRC (querying at the main vertex), various experiments were conducted based on different common bounded scenarios. The same training protocol and amount of training data were used in all scenarios. The experimental results were generated by randomly querying the neural network from different viewpoints during the first bounce and comparing the predicted indirect lighting with the denoised ground truth. The results showed that the average SSIM score difference was +0.11 points, with SSIM scores of 0.82 for NRC and 0.93 for MRRC. Since the behavior of SSIM is non-linear, this translates into a significant improvement in perceptual quality for the MRRC architecture.

[0137] The visual comparison results of multi-view NRC and MRRC in an open Japanese classroom scene can be found in [reference]. Figure 14 The diagram only depicts indirect lighting.

[0138] Figure 15 An example of a method 1500 for ray tracing is outlined. In step 1501, method 1500 includes: acquiring a virtual scene. In step 1502, method 1500 includes: emitting one or more rays from a single camera at a single viewpoint of the virtual scene. In step 1503, method 1500 includes: determining the position of the first bounce of each of the one or more rays. In step 1504, method 1500 includes: calculating the direct illumination of each of the one or more rays at the position of the first bounce. In step 1505, method 1500 includes: acquiring weights of a server neural network, wherein the server neural network is trained to infer the indirect illumination of each of the one or more rays, the weights being acquired from a server. In step 1506, method 1500 includes: predicting the indirect illumination of each of the one or more rays by inputting the weights into a device neural network. In step 1507, method 1500 includes rendering frames of a virtual scene based on the direct and indirect lighting of each of one or more rays.

[0139] Figure 16A An example of an apparatus 1600 for implementing method 1500 is illustrated schematically. Apparatus 1600 can implement any of the methods described herein. Apparatus 1600 can be implemented on device 801. Apparatus 1600 can be implemented on server 802. Figure 16B The interaction between device 801 and server 802 is illustrated. Device 801 and server 802 can be connected via wireless network 1604. Wireless network 1604 may include the Internet. Device 801 may be a mobile device 801. Mobile device 801 may be a mobile phone, laptop, computer, or TV. Device 801 may be implemented on a chip.

[0140] There can be multiple devices 801. Each device 801 can be used to perform any of the steps described herein. Each device 801 can belong to a different user. Each device may belong to a different user participating in the game. Each device 801 can be used to emit one or more rays of light from different viewpoints 202 of a single camera 201. A combination of one or more devices 801 and server 802 can provide a computing system 1603. Devices 801 can be connected to server 802 via wireless network 1604.

[0141] Apparatus 1600 includes a processor 1601 for processing a dataset in the manner described herein. For example, processor 1601 may be implemented as a computer program running on a programmable device such as a central processing unit (CPU). Apparatus 1600 includes a memory 1602 for communicating with processor 1601. Memory 1602 may be non-volatile memory. Processor 1601 may also include a cache (not shown in FIG. 16) for temporarily storing data in memory 1602. The above-described apparatus may include more than one processor and more than one memory. The memory may store data executable by the processor. The processor may be configured to operate according to a computer program stored in a machine-readable storage medium in a non-transitory manner. The computer program may store instructions to cause the processor to perform its methods in the manner described herein. The method steps described herein may be executed by a computer-readable storage medium. The method steps described herein may be executed by a computer program product.

[0142] The applicant hereby discloses each individual feature described herein, as well as any combination of two or more such features. With ordinary knowledge of those skilled in the art, such features or combinations can be implemented as a whole according to this specification, regardless of whether such features or combinations of features solve any problem disclosed herein; and without limiting the scope of the claims. The applicant notes that aspects of the invention may include any such individual feature or combination of features. In view of the foregoing description, those skilled in the art will appreciate that various modifications can be made within the scope of the invention.

Claims

1. A method (1500) for performing ray tracing on a device (801), characterized in that, The device includes a device neural network (1205), and the method (1500) includes: At the device (801): (1501) Obtain the virtual scene (204); (1502) One or more rays (206, 207) are emitted from a single camera (201) of a single viewpoint (202) of the virtual scene (204). (1503) Determine the position (1101) of the first bounce (206) of each of the one or more rays (206, 207); (1504) Calculate the direct illumination of each of the one or more rays (206, 207) at the position of the first bounce (206); (1505) Obtain the weights of the server (802) neural network, wherein the server (802) neural network is trained to infer the indirect illumination of each of the one or more light rays (206, 207), and the weights are obtained from the server (802). (1506) By inputting the weights into the device neural network (1205), predict the indirect illumination of each of the one or more rays (206, 207) at the location of the first bounce (206); (1507) Render a frame (1402) of the virtual scene based on the direct lighting and the indirect lighting of each of the one or more rays (206, 207).

2. The method according to claim 1, characterized in that, The position of the first bounce (206) of each of the one or more rays (206, 207) is determined by rasterization.

3. The method according to claim 1 or 2, characterized in that, Predicting the indirect lighting of each of the one or more rays (206, 207) includes: inputting one or more ambient occlusion (AO) maps at the first bounce (206) into the device neural network (1205) and using the device neural network (1205) to predict the indirect lighting; and / or predicting the indirect lighting of each of the one or more rays (206, 207) includes: inputting the position (1101) of the first bounce (206) into the device neural network (1205) and using the device neural network (1205) to predict the indirect lighting.

4. The method according to any one of the preceding claims, characterized in that, Rendering the frame (1402) of the virtual scene based on the direct lighting and the indirect lighting of each of the one or more rays (206, 207) includes: merging the direct lighting and the indirect lighting of each of the one or more rays (206, 207).

5. A method for generating weights for a server (802) neural network, characterized in that, The weights are supplied to the device (801) for use in ray tracing, and the method includes: At the server (802): The server (802) neural network is trained to infer indirect illumination from one or more light rays (206, 207); The weights of the server (802) neural network are output to the device (801), wherein the device (801) is used to use the weights of the server (802) neural network in ray tracing.

6. The method according to claim 5, characterized in that, The server (802) neural network takes data from a bounded scene or data from voxels as input (1204) and performs offline generation and training based on data from a training camera layout (201a to 201e), wherein the training camera layout includes multiple training cameras arranged at different viewpoints (viewpoint 1 to N).

7. The method according to claim 5 or 6, characterized in that, The server (802) neural network is further trained online based on a single camera arranged at a single viewpoint (viewpoint 1 to N), with data from the virtual scene as input (1204), the virtual scene (204) and the single camera (201) arranged at the single viewpoint (202) being the same as those used on the device (801); and / or When subordinate to claim 6, the server (802) neural network further takes data from the bounded scene or data from the voxels as input (1204) and performs online training based on data from the training camera layout (201a to 201e), the training camera layout including the plurality of training cameras arranged at different viewpoints (viewpoint 1 to N).

8. The method according to claim 6 or 7, characterized in that, The optimal training camera layout (201a to 201e) is determined based on an optimization algorithm, which is used to optimize the training camera layout (201a to 201e) based on the path tracing ray budget.

9. The method according to any one of claims 6 to 8, characterized in that, The server (802) neural network is generated and trained in the following manner: One or more light rays (206, 207) are emitted from each of the training cameras at the corresponding viewpoints (viewpoints 1 to N). Determine the position (1101) of the first bounce (206) of each of the one or more rays (206, 207) emitted from each of the training cameras (201a to 201e). Predict the indirect illumination of each of the one or more rays (206, 207) emitted from each of the training cameras (201a to 201e).

10. The method according to any one of claims 7 to 9, characterized in that, The server (802) neural network is further trained by predicting the indirect lighting of the virtual scene (204) and updating the weights of the server (802) neural network for the indirect lighting based on the indirect lighting of the virtual scene (204).

11. The method according to claim 10, characterized in that, When the difference between one or more values ​​of the indirect lighting in the virtual scene (204) and the corresponding value of the indirect lighting in the server (802) neural network is greater than a threshold, the weights of the server (802) neural network for the indirect lighting are updated.

12. The method according to any one of claims 9 to 11, characterized in that, The indirect lighting of the virtual scene (204) is predicted through one or more of the following methods: The random Fourier features are used as the position encoding (x) input (1204) of the server neural network (1205); The spherical harmonic function is used as the directional encoding and reflection view input (1204) of the neural network of the server (802). Ambient Occlusion (AO) decomposed textures are used as inputs (1204) to the neural network of the server (802). The Ambient Occlusion (AO) decomposed texture is multiplied with the output (1207) of the server (802) neural network.

13. A device (801) for ray tracing, characterized in that, The device (801) includes a device neural network (1205), one or more processors (1601), and a memory (1602) for storing program code, wherein the program code is executable by the one or more processors (1601), such that the device (801) is used to: Get the virtual scene (204); One or more rays (206, 207) are emitted from a single camera at a single viewpoint (202) of the virtual scene (204). Determine the position (1101) of the first bounce (206) of each of the one or more rays (206, 207); Calculate the direct illumination of each of the one or more light rays (206, 207) at the position of the first bounce (206); Obtain the weights of the server (802) neural network, wherein the server (802) neural network is trained to infer the indirect illumination of each of the one or more light rays (206, 207), and the weights are obtained from the server (802); By inputting the weights into the device neural network (1205), indirect illumination at the location of the first bounce (206) is predicted for each of the one or more rays (206, 207); Render a frame (1402) of the virtual scene based on the direct lighting and the indirect lighting of each of the one or more rays (206, 207).

14. A server (801) for generating weights for a neural network of a server (802), characterized in that, The weights are supplied by the device (801) for use in ray tracing, and the server (802) includes one or more processors (1601) and a memory (1602) for storing program code, wherein the program code is executable by the one or more processors (1601), such that the server (802) is used to: The server (802) neural network is trained to infer indirect illumination from one or more light rays (206, 207); The weights of the server (802) neural network are output to the device (801), wherein the device (801) is used to use the weights of the server (802) neural network in ray tracing.

15. A computing system (1603), characterized in that, Includes the device (801) according to claim 13 and the server according to claim 14.

Images