Image rendering method and device

Abstract

An image rendering method for rendering a pixel at a viewpoint, comprising the steps of: for a first element of a virtual scene having a predetermined surface at a location within the virtual scene: providing a viewpoint-based position and orientation to a previously trained machine learning system to predict a factor that, when combined with a distribution function characterizing the interaction of light with the predetermined surface, generates a pixel value corresponding to the first element of the virtual scene illuminated at the location; combining the predicted factor from the machine learning system with the distribution function to generate a pixel value corresponding to the illuminated first element of the virtual scene at the location; and incorporating the pixel value into a rendered image for display; wherein the machine learning system was previously trained using a training set based on images containing a plurality of lighting conditions.

Description

Technical Field

[0001] This invention relates to an image rendering method and apparatus. Background Technology

[0002] Ray tracing is a rendering process that tracks the path of light within a virtual scene. Then, the interaction of each ray with objects or surfaces in the scene is simulated. To achieve a certain level of realism, this simulation typically considers the material properties of these objects or surfaces, such as their color and reflectivity.

[0003] Therefore, ray tracing is a computationally expensive process. Furthermore, the cost varies between different image frames, depending on what scene is illuminated, what light is used for illumination, and from what viewpoint.

[0004] This makes it difficult to maintain the preferred frame rate for rendering such images. For the average computational cost corresponding to the average image completion time (i.e., frame rate) and the amount of variation given around the average value caused by ray tracing, either the average image quality must be set low enough that the variation has only a small impact on the frame rate, or if the average image quality is set close to the maximum value corresponding to the preferred frame rate, the consistency of the frame rate must be sacrificed when the varying ray tracing requirements fluctuate beyond the average value.

[0005] Neither of these outcomes is desirable, but they are difficult to avoid. Furthermore, the computational burden of the ray tracing process is data-dependent and unpredictable.

[0006] This invention attempts to solve or alleviate this problem. Summary of the Invention

[0007] Various aspects and features of the present invention are defined in the text of the appended specification, and include at least: - A first example of an image rendering method for rendering pixels at a viewpoint includes the following steps: for a first element of the virtual scene having a predetermined surface at a location within the virtual scene: providing the position and orientation based on the viewpoint to a previously trained machine learning system to predict factors, generating pixel values ​​corresponding to the first element of the virtual scene illuminated at the location when the factors are combined with a distribution function characterizing the interaction of light with the predetermined surface; combining the predicted factors from the machine learning system with the distribution function to generate pixel values ​​corresponding to the illuminated first element of the virtual scene at the location; and incorporating the pixel values ​​into a rendered image for display; wherein the machine learning system was previously trained using a training set based on images including multiple lighting conditions; and - Another example of an entertainment device includes a graphics processing unit configured to render pixels at a viewpoint within an image of a virtual scene, the virtual scene including a first element having a predetermined surface at a location within the scene; a machine learning processor configured to provide the position and orientation based on the viewpoint to a previously trained machine learning system to predict factors that, when combined with a distribution function characterizing the interaction of light with the predetermined surface, generate pixel values ​​corresponding to the first element of the virtual scene illuminated at the location; the graphics processing unit is configured to combine the predicted factors from the machine learning system with the distribution function to generate pixel values ​​corresponding to the illuminated first element of the virtual scene at the location; and the graphics processing unit is configured to incorporate the pixel values ​​into a rendered image for display; wherein the machine learning system is previously trained using a training set based on images including multiple lighting conditions. Attached Figure Description

[0008] A more complete understanding of the present disclosure and its many accompanying advantages will be readily obtained when considered in conjunction with the accompanying drawings, and will become better understood by referring to the following detailed description, wherein: Figure 1 These are schematic diagrams of entertainment devices according to embodiments described herein; Figure 2 This is an illustration of a ray-traced object according to the embodiments described herein; Figure 3 This is a schematic diagram of the components of a ray-traced object according to the embodiments described herein; Figure 4 This is a schematic diagram of the distribution function associated with each component according to the embodiments described herein; Figure 5 This is a schematic diagram of the scattering distribution according to the embodiments described herein; Figure 6 This is a schematic diagram of a training scheme for a machine learning system according to embodiments described herein; Figure 7 This is a schematic diagram of the rendering path of a rendered image according to the embodiments described herein; Figure 8A This is a schematic diagram of a machine learning system according to the embodiments described herein; Figure 8B This is a schematic diagram of a part of a machine learning system according to an embodiment described herein.

[0009] Figure 9 This is a flowchart of an image rendering method according to an embodiment described herein.

[0010] Figure 10This is a flowchart of an image rendering method according to an embodiment described herein.

[0011] Figure 11 This is a schematic diagram of a training method according to an embodiment described herein. Detailed Implementation

[0012] An image rendering method and apparatus are disclosed. In the following description, certain specific details are presented to provide a thorough understanding of embodiments of the invention. However, it will be apparent to those skilled in the art that these specific details are not necessary to practice the invention. Instead, specific details known to those skilled in the art have been omitted where appropriate for clarity.

[0013] The embodiments of this specification seek to solve or mitigate the above-mentioned problems by using a machine learning system that learns the relationship between the pixel surface characteristics of a given object or scene and the rendered pixels; by using such a machine learning system, ray tracing rendering of an object or scene can be approximated based on a relatively consistent computational budget (the computational budget for running the machine learning system).

[0014] As explained later in this article, different machine learning systems can be trained for different scenes, locations or parts thereof, or for different objects or materials used in one or more scenes.

[0015] Machine learning systems are relatively small (typically in the range of 100 KB to 1 MB), so in order to be run by a GPU (30), they can be pulled into memory and subsequently discarded like textures of a scene. The system can be run by the GPU's shaders. It will also be understood that, in principle, machine learning systems can be run alternatively or additionally by a CPU (20) or by a general-purpose or special-purpose coprocessor (such as a neural network processor or ASIC).

[0016] Referring now to the accompanying drawings, in all the several figures, the same reference numerals denote the same or corresponding parts. Figure 2-7 The diagram illustrates the problem space within which a machine learning system is trained.

[0017] Figure 2 It is a high-quality ray-traced rendering of the example object or scene, in this case, a car parked on the platform.

[0018] Figure 3The different contributing components behind this rendering are shown. First, the diffuse lighting component 200-D typically captures the matte color of the surface and the shadows caused by the interaction of light and shape, while second, the specular lighting component 200-S captures the reflections of the surface, thus producing shimmer and highlights. Optionally, one or more additional components, such as a gloss or "coat" 200-C, may be included, which is a second outer surface that may include additional gloss or patterning. Variations of such a coating can allow for partial transparency and / or partial diffuseness in a manner similar to, for example, skin or fabric. Each of these components can be generated routinely using appropriate ray tracing procedures.

[0019] These components are added together to form the previous... Figure 2 The entire image seen in the image. It will be understood that while usually 2 or 3 such components will help with rendering, in some cases, there may be fewer (e.g., if only diffuse components are desired) or more (e.g., when the object is also translucent and therefore transmissive components are needed).

[0020] Figure 4 The next step involves determining the material properties of the objects that contribute to the aforementioned image components.

[0021] Material properties are expressed as the so-called two-way scattering distribution function (BSDF) or two-way reflection distribution function (BRDF).

[0022] The BRDF defines how light is reflected on an opaque surface, while similarly, the BSDF defines the probability that light will be reflected or scattered in a particular direction. Therefore, the BRDF or BSDF is a function describing the illumination properties of a surface (excluding the incident / outgoing irradiance itself). Other functions, such as the bidirectional transmittance distribution function (BTDF), can also be used appropriately, defining how light passes through a material.

[0023] Also refer to Figure 5 In typical ray tracing applications, for a set of rays (e.g., from a compact light source), the application calculates the incident irradiance (either directly or previously reflected) at a point on the model with a specific BSDF, BRDF, and / or BTDF. For a specific contributing component response, the incoming irradiance is combined (e.g., multiplied) with the BSDF, BRDF, or BTDF, and the result is added to the pixel value at that point on the model. Figure 5 As shown, the optical path in BSDF i A typical scattering pattern will be oriented towards the direction of specular reflection. aIt is biased, but can scatter in any direction. Accurately modeling this behavior is one reason why ray tracing can be computationally expensive.

[0024] Using the color information of the model at each point and the corresponding BSDF, BRDF, and / or BTDF of that point (i.e., for a specific material represented by a given point), the behavior of light for a given final viewpoint can be calculated, where light reflection or scattering, for example, determines the actual distribution of shimmering and highlighting areas on a vehicle surface.

[0025] A separate BSDF, BRDF, or BTDF can be used for each active component; thus, by way of non-limiting example, a BSDF can be used for the diffuse component, and a BRDF for the specular component, which in this example is also used for the coating reflection component (although a BTDF can also be used for such a coating reflection component). It will be understood that BSDF, BRDF, or BTDF can be used appropriately, and therefore, unless otherwise stated, it is appropriate to include references to BRDF or BTDF when referring to BSDF hereinafter.

[0026] like Figure 4 As shown, ray tracing is performed using the object's color properties and the diffuse material properties of a BSDF (200-BSDF-D) to generate the diffuse image component 200-D. Similarly, the specular or reflective material properties of a BSDF (200-BSDF-S) are used to generate the mirror image component 200-S. Likewise, the material properties of a BSDF (200-BSDF-C) are typically specular in this case, generating the coating image component 200-C. These components are combined to produce the final ray-traced image 200.

[0027] However, as mentioned earlier, the problem is that when light intersects different surfaces with different BSDFs, calculating the reflected and scattered light paths is computationally expensive and can be highly variable, and summing the results for each pixel of the scene at a particular viewpoint is both computationally expensive and potentially highly variable.

[0028] Therefore, the embodiments described herein seek to replace them with other methods that have a more predictable computational load. Figure 4 The ray tracing step is used to obtain a final image of suitable quality.

[0029] Still referencing Figure 6 In the embodiments described herein, a corresponding machine learning system is provided for each contributing component of the image (e.g., diffuse, specular reflection, and optionally coating or any other contributing component).

[0030] As described later in this paper, machine learning systems are typically neural networks that are trained to learn the transformation between the BSDF (e.g., 200-BSDF-D) of the contributing components of multiple images from different viewpoints in a scene and the ground truth (e.g., 200-D) of ray tracing.

[0031] In other words, if a ray-traced image (or one of its contributing components) is a combination of how light illuminates an object and a BSDF describing how that object responds to light, then by taking a ray-traced image and decomposing it with the BSDF, the result is a quality that can be called 'irradiance' or 'shadow', but more generally describes how light acts on an object (as calculated in aggregate through the ray tracing process).

[0032] If a machine learning system or neural network can learn to predict this quality, it can be combined again with BSDF to produce a predicted image that approximates the ray-traced image. Therefore, this network can be called a neural pre-computed light model or an NPLM network.

[0033] More specifically, for a given position and viewpoint orientation on an imaginary image of an object, a machine learning system or neural network must learn to output a value that, when combined with a BSDF for that same position / pixel, results in a pixel value similar to the pixel value produced by ray tracing the image at that pixel. Thus, during training, it generates an internal representation of lighting conditions (e.g., due to point light sources or skydome lighting) and surface illumination properties implied from the training images.

[0034] Therefore, in the example embodiment, the image can be rasterized or otherwise generated at a given viewpoint, which will fill the image with pixels that will subsequently be illuminated. For each of these conceptual pixels, there exists a corresponding 3D location in the scene, for which an appropriate “irradiance” or “shadow” can be obtained using an NPLM network.

[0035] Figure 6 The training environment for this network is shown, and specifically, by way of example only, a network 600-D for the diffuse contribution component is shown.

[0036] The diffuse contribution component takes as input to the network a (x, y, z) position 610 on the object or scene (e.g., corresponding to a pixel in an image) and a normal 620 of the object / scene at that point. A normal N is used instead of the viewpoint direction because, for the diffuse contribution component, illumination can be considered direction / viewpoint independent, and therefore a normal, as a known value, can be used for consistency. In this illustrative example, in Figure 6 The inputs are conceptually illustrated using representative values ​​of each of the car images.

[0037] Optionally, additional inputs (not shown) may be provided, such as roughness or matte-gloss scalar values ​​optionally derived from the relevant BSDF.

[0038] The output of the NPLM network (as explained later in this document) is the learned light quality or illuminance 630 at the input location, which, when combined with the associated diffuse BSDF (200-BSDF-D) 640 at the same location, produces the predicted pixel value at the (x, y) location in the predicted image 650.

[0039] Figure 6 The illustration shows that the per-pixel difference between the predicted and ground truth pixels of the 200-D diffuse component of the target ray is used as the loss function for training the network, but this is not required; instead, the ground truth image can be decoupled from the BSDF (i.e., by performing the inverse function) to produce a proxy of how the ray-traced light cumulatively affects each (x, y) pixel of the object in the image, and this is the quality of the network being trained to learn.

[0040] Therefore, the network's error function is based on the difference between its single pixel (x, y) output value and the corresponding single (x, y) pixel of the ground truth image when separated from the BSDF corresponding to that location.

[0041] Since the pixels of the real-world image can be pre-separated from the corresponding BSDF for each location, the network can be trained without combining its own output with any BSDF to generate the actual predicted image pixels. This reduces the computational load of training.

[0042] As mentioned above, the learned quality of the trained neural network output captures how light in the environment acts on objects or scenes as a function of surface position and viewpoint within the scene. Therefore, it can effectively generate an internal representation of the lighting map for scene and surface response models. How this occurs will be discussed in more detail below.

[0043] Now for reference Figure 7 In summary, for each contributing component of the final output image, a machine learning system is trained to perform a transformation applied to a BSDF (Balanced Baseline Function) specific to the location of that contributing component on the object / scene. This transformation is a trained function based on the (x, y, z) position and orientation values ​​of points on the object / scene. As mentioned earlier, any number of machine learning systems—one, two, three, four, or possibly more—can be used depending on the number of contributing components in the final image. The term "training function" may be used below to refer to the machine learning system that has learned this transformation.

[0044] As mentioned regarding diffuse components, since diffuse shadows are assumed to have an invariant direction, it can be assumed that the direction value is equal to the normal at a given point.

[0045] Meanwhile, for the specular reflection component that is at least partially reflected and therefore will vary with the viewpoint, the direction value is or is based on the viewpoint between the (x, y) position of the current pixel at the image viewpoint (which will have a position in virtual space) and the (x, y, z) position of the object that serves as input to the machine learning system, thus providing the viewpoint dependency between the input point on the scene surface and the current pixel to output the learning quantity.

[0046] In this case, the coating reflection component is also specular, so a similar viewpoint or viewpoint-based direction is used for input.

[0047] Therefore, the direction value of the direction-dependent component can be the viewpoint direction between the output pixel position and the object surface position, or a value based on that direction, such as the viewpoint direction of a surface mirror (i.e., the principal direction that the viewpoint direction will reflect when the surface normal is given at the input position). Any suitable direction value that includes information about the viewpoint direction can be considered.

[0048] As mentioned earlier in this paper, in each case, the trained function encapsulates the quality of the learning. Combining the appropriate BSDF with the network output at each location allows the creation of a colorized image for each component. Alternatively, or otherwise, the pixel values ​​from the colorized images of each component are combined to generate the final output.

[0049] It should be understood that not all images can be ray-traced during rendering, and similarly, not all images can be generated using the techniques described above. For example, an NPLM network can be trained on a specific object or material based on a real ray-traced image with representative lighting.

[0050] When these objects or materials are subsequently rendered in real-time using surface ray tracing provided by the trained function described herein, the relevant NPLM network is loaded into memory and, in conjunction with an appropriate BSDF, runs against the relevant surface locations and viewing directions in the scene to produce their contributions to the relevant pixels. Other suitable techniques, including ray tracing itself, can be used to render additional pixels.

[0051] Typically, based on the same asset identification scheme used to select and load textures of objects or materials, a suitable machine learning system is selected and loaded into the memory used by the GPU. Therefore, for example, if an object has an ID "1234" used to access a related texture, that ID can also be associated with a related machine learning system. Conversely, if a texture has an ID "5678" associated with an object (e.g., the texture represents a material shared by multiple objects), that ID can also be associated with a related machine learning system. In this way, entertainment devices can load machine learning systems using a process similar to loading textures. It should be understood that the actual storage and access techniques between textures and machine learning systems may differ, especially if lossy compression is used to store textures, which will affect the operation of decompressing the machine learning system. Therefore, machine learning systems can be stored without compression, or with lossless compression, or with lossy compression where the loss is low enough so that the decompressed machine learning system still functions adequately: this can be assessed by comparing the incremental degree of compression loss to the output error / cost function of the machine learning system until the error reaches an absolute or relative (relative to the uncompressed machine learning system) quality threshold.

[0052] Now go to Figure 8A In the embodiments described herein, the machine learning system or NPLM network can be any suitable machine learning system. Thus, for example, a single neural network can be trained using position and viewpoint orientation as input and generate RGB values ​​of the learned characteristics as output.

[0053] However, particularly advantageous networks include different discrete architectures.

[0054] like Figure 8A As shown, in a non-restrictive example, the network consists of two parts. The first part can be considered a location network, and the second part can be considered a orientation network.

[0055] Each of these networks can have 3 or more layers and use any suitable activation function.

[0056] The location network takes the (x, y, z) positions of points in the previously mentioned object / scene as input and outputs an intermediate representation, which will be discussed later in this paper.

[0057] The directional network receives this intermediate representation, and for example, with (θ, The input direction is either in a normalized (x, y, z) vector or a similar format (e.g., a normal, or a pixel viewpoint or surface point mirror pixel viewpoint direction or other viewpoint-based direction value, depending on the case). It outputs RGB values ​​corresponding to the learned amount of the (x, y) position (and therefore pixel viewpoint) of the current pixel in the image rendered from the virtual camera position in a space shared with the object / scene.

[0058] Therefore, in the non-restricted example, the location network has 3 layers: the first layer has 3 input nodes (e.g., for x, y, z positions), the middle layer has 128 hidden nodes, and the last layer has 8 outputs.

[0059] While any suitable activation function can be chosen for the network, the Modified Linear Unit (ReLU) function has been evaluated as a particularly effective activation function across layers in a location network. It generalizes well to untrained locations and helps avoid overfitting.

[0060] Similarly, in this non-restrictive example, the orientation network has four layers. The eight outputs of the location network and two or three additional orientation values ​​are input to 128 nodes in the first layer, then to two more layers of 128 nodes, ultimately resulting in three outputs in the last layer, corresponding to the R, G, and B values ​​of the learned amount at the current pixel. This can then be combined with the BSDF at that location (e.g., multiplied) to obtain the final pixel contribution from the training function (e.g., diffuse, specular reflection, etc.), although, as mentioned earlier, this is not necessary during training.

[0061] While any suitable activation function can be chosen for directional networks, the sine function has been evaluated as a particularly effective activation function across layers of directional networks. Because the behavior of light varies greatly in the angular domain, containing details at many angular frequencies, but based on low-dimensional inputs (e.g., normalized x, y, z vectors), the sine activation function has been found to be particularly good.

[0062] Therefore, it is worth noting that the two parts of the network can use different activation functions.

[0063] However, this network is considered a separate structure network rather than two independent networks because the training scheme has only one cost function: the error between the RGB value output by the orientation network and the target value of the corresponding pixel in the real ray-traced image after decoupling from the corresponding BSDF.

[0064] This error propagates back through two networks: the location network does not have a separate target value or cost function. Therefore, in practice, the output layer of the location network is actually a hidden layer of the combined network, enhanced by additional inputs of directional values, and representing a transition from the first activation function to possible second and different activation functions within the layer.

[0065] As previously mentioned, neural networks build lighting models for illuminated objects, materials, or scenes. Specifically, in the non-restrictive example above, the location network effectively classifies (x, y, z) locations into lighting types (e.g., bright or dark, and / or other categories that may relate to how light interacts with its respective BSDF, such as relative reflectance or diffuse): the intermediate representation of this part of the output can be considered as an N-dimensional location in the lighting space that characterizes the type of light at the input location: if different parts of the scene are illuminated in the same way, they are projected onto the same N-dimensional location. A location network trained for the specular component can have more outputs than the output for the diffuse component: for example, compared to 8 outputs for the diffuse component, the specular component has 32 outputs to account for the greater variation in lighting types that may occur in the specular component.

[0066] Then, the orientation network models the behavior of the light model when it observes the illuminated object, material, or scene at a specific input angle on the surface at the input location, in order to generate the learned characteristics of that location in the image.

[0067] Therefore, in summary, the position and orientation networks are trained as a whole to predict the factors or transformations between the BSDF description of the surface properties and the desired rendered image of that surface. The network can then be used to render the surface instead of ray tracing. Typically (but not necessarily), the network is trained on only one contributing component of the image, such as the diffuse component of the specular reflection, while multiple networks are used to generate the desired components for the final image or parts of the image, although this is not required (i.e., in principle, the network can be trained on a fully combined image or a combination of two or more contributing components, such as all specular reflections or all diffuse contributions).

[0068] train

[0069] As described elsewhere in this paper, the network is trained using multiple ray-traced images of an object, scene, or surface taken from multiple different viewpoints. This allows the network to understand, in particular, how specular reflections change with position. Viewpoints can be randomly distributed and / or, for example, selected (or primarily selected) from the range of viewpoints available to the user as they navigate a rendered environment known as a view volume: that is, the spatial volume in which the viewpoint can appear, and therefore needs to be included in the training.

[0070] In the embodiments described herein, training data can be generated in the following manner.

[0071] Understandably, for any machine learning system, the training data used to train the system is crucial to its performance. Therefore, generating training data that yields good results is highly beneficial.

[0072] As described elsewhere in this paper, the training data for the NPLM system described herein is a set of high-quality rendered images based on a scene / object / material / surface (hereinafter collectively referred to as scene), which is typically decoupled from one or more relevant distribution functions (e.g., BSDF, BRDF, etc., as described elsewhere in this paper) so that the learning quality referred to herein can be directly used as the training objective, thereby eliminating the computational burden of generating prediction images during training, and also ensuring that the error function is not derived from the output of the NPLM itself during a single removal.

[0073] Different NPLMs may handle view-dependent and view-independent shading effects (such as diffuse, specular reflection, etc.), so if an object has view-dependent shading (such as specular or specular reflection, etc.), a single view of the object in the scene is usually not enough.

[0074] Therefore, the number and location of training data images depend not only on the geometry of the scene (e.g., if objects are visible within the viewport), but also on the material properties of objects in the scene.

[0075] Therefore, in the embodiments described herein, NPLM training data in the form of images of the scene captured from multiple camera viewpoints can be generated at least in part based on the materials in the scene (e.g., material properties such as light response properties such as diffuse or specular reflection response, but also other material properties such as surface complexity, e.g., narrow or wide spatial frequency components present structurally and / or in texture).

[0076] It's important to note that these images are typically generated by high-quality third-party renderers, which do not have access to the internal data. Therefore, when performing shading on an image, only the final, complete image may be available, without any information (or control) about a particular cast ray or its direction.

[0077] Therefore, for training purposes, it is desirable to generate and use a set of images that effectively capture the appearance of the scene, preferably for all valid views within the view.

[0078] Now for reference Figure 11To this end, in step 1110, a set of low-resolution images is first rendered using a set of camera positions within the viewing volume. These positions can be equidistantly or randomly distributed on a sphere around the scene (if it can be viewed from any angle, e.g., as an operable object), on a hemisphere around the scene (if it is based on a virtual ground and therefore cannot be viewed from below), or on a ring around the scene (if viewed from a ground-based viewpoint, e.g., the avatar's first-person perspective). Such a ring can be located at a fixed height corresponding to the avatar's viewpoint, or it can occupy a height range, e.g., as a viewing cylinder containing one or more of the avatar's crouching height and jumping height.

[0079] Step 1110 Figure 11 As shown, the track (loop) of the camera position surrounds the example car object.

[0080] In this initial set, the number of camera positions may be only one, but is usually three or more, more typically dozens or hundreds. For example, one camera per degree of orbit would produce 360 ​​cameras. In this example, as an unrestricted number, there are 200 cameras.

[0081] Each image has a low resolution: for example, 128x84 pixels. An example image from step s1120 is shown.

[0082] It is worth noting that for each pixel of each image, in step s1130, the associated metadata includes the 3D position of the scene surface corresponding to the pixel, the normal of the scene surface corresponding to the pixel, and an optional material ID or similar material surface identifier or descriptor, such as texture ID or object ID.

[0083] In the first instance of the viewpoint selection process, the 3D locations of the scene surface rendered by pixels from some, or typically all, these low-resolution images are checked to identify which locations in the scene are visible within the first set of camera positions. These are the 3D locations that NPLM will benefit from training on.

[0084] Therefore, optionally, for each 3D location identified as being rendered in at least one initial low-resolution image, a new position in 3D space is calculated that is offset relative to that location along the surface normal. This offset distance relative to the surface is a variable that can be modified. This new position serves as a candidate viewpoint for the virtual camera to generate high-quality (e.g., high-resolution ray-traced) renderings.

[0085] However, this could result in a large number of potentially high-quality ray tracing renders being generated as training images, which would be computationally very heavy and could also include a lot of redundancy when used as a training set for NPLM.

[0086] Therefore, in the first instance, it is desirable to filter or eliminate these candidate viewpoint locations in a way that is relevant to and useful for NPLM field training.

[0087] In particular, it is beneficial to have more training examples for parts of the scene that contain view-dependent materials (such as specular or glossy materials) compared to view-independent materials (such as diffuse or matte materials).

[0088] Therefore, one of two methods can be adopted.

[0089] In the first method, in step 1140, for each candidate viewpoint corresponding to the normal at a surface location, the corresponding material properties of the surface at that location are checked. As mentioned above, its diffuse or specular reflection response, or its translucency, etc., can be used in particular.

[0090] In practice, this can be accomplished using a lookup table that associates a material ID or similar with values ​​indicating how diffuse or specular the material surface is (e.g., matte or glossy). More specifically, as a non-limiting example, this property can be represented by values ​​ranging from 0 for fully diffuse to 1 for specular. This can be viewed as input to a probability function, such that specular or luminous (view-dependent) materials have a relatively high probability, and diffuse or matte (view-independent) materials have a relatively low probability.

[0091] Then, a probability function is used to preserve candidate camera positions; therefore, a higher proportion of camera positions facing a specular surface will be preserved compared to those facing a diffuse surface.

[0092] Conversely, if the numerical reduction is reversed (e.g., low probability and high probability are reversed), then the probability function can be used to select candidate camera locations with the same effect.

[0093] In the second method, alternatively or additionally, in step s1140, the variability of pixel values ​​corresponding to the same 3D location on the scene surface, as viewed in a low-resolution image, can be evaluated to determine the variance of pixel values ​​for each captured 3D location. In this way, view-invariant (e.g., diffuse or heavily shadowed) surface locations will have low variance (i.e., pixels showing that location in different low-resolution images will be similar), while view-dependent (e.g., specular or luminous) surface locations will have high variance (i.e., pixels showing that location in different low-resolution images will show a wider range of values, e.g., as some capture flicker or reflection of light). This variance, or a normalized version thereof, can again be used as input to a probability function, such that specular or luminous (view-dependent) materials have a relatively high probability, while diffuse or matte (view-independent) materials have a relatively low probability.

[0094] Therefore, in either case, in step s1140, an estimate of the viewpoint dependence of the material’s light responsiveness at each captured 3D location in the view volume is obtained (based on material properties or pixel variability, or possibly both), and this can be used as input to a probability function.

[0095] Then, in step s1150, a probability function is used to determine whether to retain or discard the corresponding candidate viewpoint, wherein viewpoints centered on surfaces related to the viewpoint are retained more frequently than viewpoints centered on surfaces unrelated to the viewpoint.

[0096] The output range of the probability function can be adjusted to generate a roughly expected total number of camera viewpoints for training based on the original number of possible candidates and the final expected number. Alternatively, the probability function can perform several rounds of retention / removal until the number of remaining camera viewpoints is within the threshold of the desired number.

[0097] In either case, the result is a controllable number of camera viewpoints randomly distributed over the desired observation volume, but with a variable probability density in response to material properties (e.g., reflections or others) of the material directly in front of the camera's center. This is illustrated by a constellation diagram of the reserved points in the figure of step s1150. In reality, the camera position could be further away from the object / scene surface than shown in the figure, but for the sake of illustrating their distribution, these points have been placed close to the surface in the figure.

[0098] The number of controllable camera viewpoints can be chosen based on a variety of factors, such as the expected performance of the resulting NPLM, the computational burden of generating high-quality ray-traced images and training the NPLM on them, memory or storage limitations, etc. As a non-limiting example, a typical controllable number for training purposes might be between 10 and 10,000, with a typical number being between 200 and 2,000.

[0099] Finally, in step s1160, the image is rendered at the preserved viewpoint. Alternatively, as... Figure 11 As shown, these renderings are generated using a virtual camera with a wider angle than the lens used for the initial low-resolution image or the lens used in the game.

[0100] This often results in rendering too much of the scene (i.e., parts that are not visible from the view volume's point); this often makes NPLM output more robust, especially for view positions near the edge of the view volume, and for unexpected expansion of the view volume due to object clipping in the game or minor design changes.

[0101] Although the above method generates candidate camera viewpoints based on scene surface normals captured in the initial low-resolution image, it is not the only possible approach.

[0102] One potential problem with the above method is that while locations in the scene that do not change with the viewpoint can be imaged by a camera pointing towards that location along the normal to that location, it is rendered only from different angles in other images of nearby locations, which in turn are determined by the normals of the scene surface at those locations. Therefore, although a relatively large number of images are captured on and near the viewpoint-dependent parts of the scene, the images themselves may be inappropriately affected by the geometry of the scene itself.

[0103] Therefore, returning to the initial low-resolution image, in another instance of the viewpoint selection process, potential viewpoint locations can be considered for each pixel (or at least those pixels representing surfaces in the scene) of each low-resolution image. In the example above with 200 images at 128×84 pixels, this equates to up to 1.6 million candidates. These images typically capture multiple instances of a given location on the scene from different angles, independent of the scene's topology. As a result, the training set may be more robust.

[0104] Similarly, the view dependency derived from the surface material (and / or pixel variance) of the surface location corresponding to a given pixel within a low-resolution image, and thus the view dependency derived from the surface material corresponding to a candidate viewpoint, can be used to drive the probability of retaining or culling that viewpoint. In this way, 1.6 million candidate viewpoints can again be culled down to a manageable number.

[0105] In this situation, because there may be multiple viewpoints at the same location in the scene, the resulting distribution of camera viewpoints may be biased towards the most visible locations in the scene, rather than those most relevant to the viewpoint. For example, if one (diffuse) location in the scene is visible in 20 times more images than one (specular) location, even if viewpoints observing the diffuse location are more likely to be eliminated, the end result may be that there are more images of the diffuse location than images of the glossy location because their numbers are 20 times greater.

[0106] Therefore, optionally, the probability of retaining or culling viewpoints can be normalized based on how many viewpoints are concentrated in the same location in the scene (albeit from different angles). This normalization can be complete (thus, in the example above, the probability of retaining the diffuse location image is reduced by 20 times, thus eliminating the effect of the number of views). Alternatively, the normalization can be partial; thus, for example, the probability of retaining the diffuse location image is reduced by only 10 times, so the effect of the number of viewpoints is significantly reduced but not completely eliminated; this means that areas that the user is likely to see frequently will also receive more training examples, regardless of whether the user receives more training examples due to viewpoint-related factors (e.g., specular reflection / brightness).

[0107] In principle, before generating high-quality ray tracing renders for training purposes, two sets of viewpoints (viewpoints based on surface normals and viewpoints based on low-resolution image pixels) can be generated and culled to create a combined viewpoint set; in practice, in any case, there may be a subset of viewpoints based on low-resolution image pixels that are actually consistent with the normals of at least some visible surface locations.

[0108] Variant training techniques

[0109] The second method described above optionally considers the issue of compensating for multiple views at the same location in the scene when culling available viewpoints. Besides controlling for training bias, it also reduces the training time of the second method by minimizing repetition at certain locations in the scene.

[0110] However, optionally or additionally, training time can be (further) reduced as follows.

[0111] As before, select a set of initial viewpoints within the view (or on its surface).

[0112] Optionally, an initial low-resolution image can now be generated for a set of locations within the visual body.

[0113] Candidate viewpoints are then optionally generated, as previously described herein, based on the normals of the locations found in the scene within the low-resolution image, and / or based on the lines between the pixels of the low-resolution image and the locations represented in the scene.

[0114] Alternatively, these viewpoints can be selected based on the probability of specular reflection / diffuse reflection at the corresponding location in the scene. Further alternatively, if multiple images centered at the corresponding location exist, the probability can be modified to at least partially take this into account.

[0115] Therefore, depending on the method employed, the result is a series of viewpoints generated—optionally used to generate the original distribution of the low-resolution image, or a distribution produced by one of the aforementioned generation and culling techniques.

[0116] In any of the embodiments described herein, once a viewpoint is generated (and optionally confirmed not to have been discarded), it is provided or queued for use in the ray tracing process in a wide-angle format as described elsewhere herein to generate a high-quality image.

[0117] Once the corresponding images are complete, training begins on the generated images; therefore, there is a parallel process of generating training images (which takes some time since they are ray-traced images) and training on those images (which may also take some time). This avoids the problem of having to wait for the complete training set to be generated before training can begin.

[0118] Optionally, when viewpoints have already been generated, or when selecting generated viewpoints to determine whether they should be culled, the selection of viewpoints from those available viewpoints can be random, such that the final sequence of ray-traced images produced is also random within the final set of viewpoints being used.

[0119] This reduces the likelihood that NPLM will be overtrained in a certain part of the scene initially, and also means that, for example, if training has to be reduced due to time constraints, NPLM will still face various views of the scene.

[0120] In another variant of the training technique, if the control of the ray tracing application is available and permissible, it may be optional to render only a subset of pixels from an image from a given viewpoint; whether based on the original viewpoint set or an unculled viewpoint, scene components in a given image may have already been rendered multiple times in other images in the training set. For example, if a location in the scene has been rendered more than a threshold number of times, that location may be skipped in the current render because there are already enough training examples. Unrendered portions of the image can be labeled using reserved pixel values ​​as masking values. Therefore, training can be performed using only the input location, orientation information, and target values ​​for the unmasked pixel locations. This can significantly reduce redundancy in the training set and the computational load associated with ray tracing training images and training NPLM.

[0121] Exceptions may be applied arbitrarily. For example, pixels near the center of the image may always be rendered, since the center pixel often relates to a selected (or unculled) location in the scene, possibly as a function of its surface characteristics, as described elsewhere in this article. Pixels in the off-center portion of the image may capture unexpected and unwanted repetitive points within the scene.

[0122] - Training with variable lighting states

[0123] For a given lighting condition, various training techniques described earlier in this paper are typically performed using multiple different ray-traced images of a scene, object, or material (which are generally referred to as “scene” here unless otherwise specified); typically, it is assumed that the lighting source is fixed within the scene (e.g., due to the location of windows, and / or due to the use of so-called “skydome” that provides a static lighting map).

[0124] However, in some cases, the lighting of a scene may change significantly; for example, some games simulate day / night cycles and / or weather, so the position of the sun (or moon) can change during a virtual day (or night) within the game. Similarly, depending on the weather, the dominant light source can be, for example, a near-point source such as the sun, or a diffuse source such as a backlit cloud.

[0125] An NPLM trained to model the illuminance of a scene based on a lighting condition may not be general enough to replicate other conditions.

[0126] Therefore, in the embodiments of this specification, based on ground-based images that have been ray-traced using multiple lighting conditions, the NPLM is trained (using any techniques described herein) to model the illuminance of the scene.

[0127] Typically, these lighting conditions will include one or more adjustments to lighting parameters within a predetermined range.

[0128] Lighting parameters may include: - Position: For example, the movement of the sun over a predetermined range. This can be achieved by rotating the skydome, including the sun, in various ray-traced renders. Other examples could include the movement paths of non-player characters or other moving light sources.

[0129] - Direction: A range of directional changes can be modeled for light sources, such as in the case of a lighthouse or the movement path of an MPC / moving light. Alternatively or additionally, the beam width or beam spread angle of the light can be specified (e.g., flashlights or spotlights typically have a narrower beam angle than table lamps).

[0130] - Alternatively, these or other less predictable light sources, such as player-based moving light sources like flashlights or the like, can be achieved by using NPLMs that are fully flashlight-lit and non-flashlight-lit depending on where the flashlight beam falls on different parts of the scene.

[0131] - Color or color temperature: For example, the sun gradually turns red during sunset.

[0132] -brightness.

[0133] - Diffuse: For example, replacing the rendered dome light source with a continuous amount of point light source in front of clouds or fog (such as the sun or moon) or building lights.

[0134] A single NPLM can be trained on changes in one, two, or more of these parameters at a time. For example, an NPLM can be trained at sunrise, including the position of a predetermined range and the corresponding changes in color temperature. Another NPLM can be trained during the daytime (or part thereof) of the sun's transit within a predetermined range, without any changes in color temperature. Yet another NPLM can be trained at sunset, similarly including the position of a predetermined range and the corresponding changes in color temperature.

[0135] As discussed elsewhere in this article, more complex training environments may require changes to the NPLM architecture to accommodate the network, typically reaching its maximum when the NPLM's memory footprint reaches a threshold size.

[0136] NPLM can also require one or more additional inputs during training that indicate changes in one or more parameters associated with the training data.

[0137] For example, lighting location can be explicitly input, and / or surrogate values ​​such as time of day (e.g., if also related to color temperature). Typically, this can be input into a location network to help characterize the relative position of scene elements with respect to light, or it can be input into a orientation network to help characterize the relationship between the light source and the viewpoint, or actually input into both. Lighting location can relate to the position of a point light source, the center of a diffuse light source, or the center or offset position for a dome light source. Input can be linear or polar coordinates, or any suitable format (e.g., polar coordinates might be more suitable for dome lights). Therefore, typical examples of changing location would be along an NPC path (e.g., traffic), object movement in-game (e.g., moving a lamp), or environmental changes (e.g., the sun or moon).

[0138] Similarly, the orientation (and / or optionally, the beam spread angle) can be input to one or both of the half of the NPLM. Typical examples of orientation changes include spotlights or sentries, or in conjunction with location movement or traffic. Spread angle values ​​can be associated with these. Meanwhile, an example of changing the spread angle includes opening a door located in front of a light source.

[0139] Color or color temperature can be input, for example, as a scalar value (e.g., for color temperature) or an RGB or similar color representation. Typically, the color representation will be used in the same color space as the predicted image (i.e., the image pixels generated by combining the output of the NPLM with a specific BSDF, BRDF, BTDF, or similar combination). Color or color temperature can be input to a location network, again aiding in the classification of scene locations embodied in the intermediate representations of the location network's output layer, and / or can be input to a orientation network, as it can more easily influence the color-dependent output of the NPLM there. Typical examples of color changes can involve sunrise or sunset, as described elsewhere in this paper, but can also involve striking events, even, for example, the activation of an alarm and the associated switching of predominantly red lighting.

[0140] Brightness can be input as a scalar value into half or half of the NPLM, similar to color temperature. Similarly, typical examples of brightness variations can be associated with sunset or sunrise.

[0141] The diffusion of a light source, such as its direction, can be input into a location network to aid in the classification of surface points, and / or can be input into a direction network to aid in the relationship between the light source and the viewpoint. Examples of diffuse light sources can include clouds or fog, or urban / office environments with diffused overhead lighting.

[0142] It will also be understood that some or all of these parameters can be time-dependent, thus allowing for the alternative or additional use of time-based parameters. This has been discussed above in relation to position, but it can also be applied to color, brightness, or any other parameter. For example, when a scene is illuminated by the sun, position, color, and brightness can all be time-dependent.

[0143] It will be understood that adding one or more such inputs to the existing first layer of one of the position and / or orientation networks only allows the input to have a single corresponding weight between the nodes in the first layer of each network.

[0144] Therefore, it is still referred to now. Figure 8B Optionally, to provide the ability to learn more complex responses to these additional inputs, another layer can be provided between the current first layer and two or more inputs to the corresponding half of the NPLM. Thus, for example, instead of simply providing the position of the lamp and the position of the current surface point as parallel inputs to the existing NPLM location network, another layer can be provided so that any significant combination of these values ​​can be learned as a preparatory step.

[0145] Therefore, for example, if all the aforementioned inputs are included in addition to the original surface point position inputs to the position network, then fully connected additional layers, or partially connected layers, can be provided, for example, with directional and diffuse inputs to the partial layers, brightness and color inputs to the parallel partial layers, and light position and surface point position inputs to the parallel partial layers. These partial layers are then fully connected to the first complete layer of the network. A similar scheme can be used for directional networks.

[0146] Figure 8B The diagram shows, as a non-limiting example, partial layers including the original scene location and lighting location in a location network, as well as parallel partial layers for the color and brightness of the light.

[0147] The training set for variable lighting conditions can simply consist of repeating any technique described herein for generating the training set for multiple lighting variations (e.g., for consecutive positions of light(one or more)).

[0148] Alternatively, a single set of locations for full rendering can be obtained as described above, and then for each location, N versions of the set can be created by selecting random parameter values ​​within a predetermined range or a series of parameter values ​​spanning said range for each of one or more positively varying parameters. Optionally, when using probabilistic methods to select candidate locations, the retained probability can be higher (to increase the number of selected locations, enriching the training set for more demanding training scenarios) or the retained probability can be lower (up to N times lower) to mitigate the associated increase in computational load from rendering variants of the same view. For that particular lighting scene, whether the probability increases, decreases, or remains constant reflects both the trade-off between computational load and time, and the final NPLM output quality.

[0149] Alternatively, the retention probability can be increased by up to N times, and for each obtained viewpoint, only one (random or sequential) version of the parameter value is selected, so that instead of having multiple parameter values ​​at one viewpoint, there are corresponding parameter values ​​at multiple viewpoints.

[0150] What will be understood is that these methods overlap; for example, generating training sets for multiple lighting changes can lead to different corresponding constellations for viewpoints in each training set, and can collectively lead to a set of different parameter values ​​at the same location and a single parameter value at a unique location.

[0151] Typically, it is not necessary to change how the viewpoint distribution is generated, as this involves the viewer's position, not the light. However, for highly directional light, it may be useful to optionally bias the probability of retaining the viewpoint based on the angular distance of the viewpoint from the angle of reflection of the directional axis of the light (i.e., increasing retention as the viewpoint gets closer to being on the reflected ray of light).

[0152] As previously described herein, any suitable machine learning system can be used to implement NPLM. Therefore, although the separate network architecture described herein is preferred, in principle, NPLMs comprising a single neural network (or in fact three or more networks or other machine learning systems, which may be of different types) can be used, wherein the input described herein is provided to one or more of these networks as appropriate.

[0153] Variant methods

[0154] Each NPLM, in response to the scene surface position and viewpoint / mirror orientation inputs described elsewhere in this document, as well as any additional inputs previously described herein, generates the learned quality of a single pixel.

[0155] Furthermore, when the final output is required, contributions from several NPLMs (e.g., trained on diffuse, specular, translucent, and other components of the scene) are often combined.

[0156] As a result, in principle, contributions from several NPLMs can be combined from NPLMs trained for their respective lighting conditions. Thus, for example, an NPLM can be trained for a solar or dome light source at 1 pm, and another NPLM can be trained for a solar / dome light source at 2 pm. The contributions from each can then be combined, for example, 100% of the network's output at 1:00 pm, 75% at 1:15 pm and 25% at 2 pm, 50 / 50 at 1:30 pm, 25 / 75 at 1:45 pm, and 100% at 2:00 pm.

[0157] For location variations, optionally, the NPLM for specular reflection contributions can be trained at finer intervals than the NPLM used for diffuse contributions, since the effects of illumination variations will be more pronounced in specular reflection contributions. For example, a specular reflection NPLM could be created for hourly intervals or every 10% of the parameter range, while a diffuse NPLM could be created for every two or three hours or every 25% of the parameter range.

[0158] Therefore, contributions from different NPLMs trained at different points across the parameter range can be blended to approximate desired points across the parameter range, and for fewer view-dependent contributions such as diffuse contributions, the samples from the corresponding NPLMs across these parameter ranges can optionally be sparser compared to specular reflection contributions.

[0159] It will also be understood that while the above time example is a 1-dimensional parameter and therefore only requires a mixture of two NPLMs trained using the parameter values ​​surrounding the desired parameter values, the method can be extended to 2-dimensional parameters using 3 or more NPLMs (e.g., triangles or other polygons in parameter space) whose weighted contributions correspond to triangulation on the desired parameter values, or in fact extended to 3-dimensional parameters using 4 or more NPLMs (e.g., tetrahedrons or other volumes in parameter space) whose weighted contributions again correspond to triangulation on the desired parameter values.

[0160] Network configuration

[0161] As mentioned above, the location network (i.e., the first part of the separated structure network described in this paper) can have different numbers of outputs, depending on whether it is trained on diffuse image components of the specular reflection type. It is worth noting that this is a specific instance of a more general approach.

[0162] Generally, the capabilities of an NPLM can vary depending on the complexity of the modeling tasks it needs to perform, and can be increased or decreased through nominal default settings. In doing so, the network architecture is typically modified to alter its capabilities.

[0163] In the first aspect, the capabilities can vary based on the size of the NPLM (e.g., the number of layers, the size of the layers, and / or the distribution of layers between parts of the NPLM), thereby modifying the architecture of the NPLM to change its capabilities.

[0164] Therefore, the size can optionally vary depending on the type of contributing component in the NPLM model (e.g., diffuse, specular, or translucent / transmissive).

[0165] Specifically, for specular or translucent / transmitting components, the size of the position network can advantageously be larger or equal to that of the diffuse component due to the greater variability in the inherent light response in these portions. Similarly, for translucent / transmitting components, the size of the position network can advantageously be larger or equal to that of the specular portion due to the possible combination of partial reflection, transmission, and internal reflection involved.

[0166] Similarly, when training on variations of one or more lighting conditions (as a non-limiting example, different lighting positions, orientations, and / or angular widths), it is advantageous that the size of the location network can be designed to be larger due to the increased number or complexity of lighting conditions for a given scene surface location that needs to be modeled.

[0167] The size can be changed by altering the number of hidden layers or the number of nodes within one or more such hidden layers. Similarly, the size can vary depending on the number of output layers (e.g., the output layers of a location network, which are also hidden layers or interface / intermediate layers between the location and orientation networks in the overall NPLM network). Increasing the number of layers generally increases the spatial distortion that the network can apply to the input data to classify or filter different types of information, while increasing the number of nodes in a layer generally increases the number of specific conditions in the training set that the network can model, thereby improving fidelity. Simultaneously, increasing the number of output nodes (as in the output of a location network where these nodes are not selected to map to a specific format) can improve the discriminative power of the output network (and subsequent networks operating on the output node values) by performing a less stringent dimensionality reduction on the internal representation of the dataset.

[0168] As an alternative or supplement, the size of the orientation network can vary depending on the type of contributing components in the NPLM model (e.g., diffuse, specular, or translucent / transmissive).

[0169] As mentioned above, the input layer of the orientation network can be resized to accommodate the higher-dimensional output of the location network in the entire NPLM decoupled structure network.

[0170] Similarly, the number of layers and / or the size of the layers can be varied to achieve a similar effect to location networks, namely, increased discriminative power and model fidelity.

[0171] Similar to the position network, for specular or translucent / transmitting components, the size of the directional network can advantageously be larger or equal to that of the diffuse contribution component due to the greater variability in the inherent light response of these portions. For similar reasons, for translucent / transmitting components, the size of the directional network may advantageously be larger or equal to that of the specular portion due to the possible combination of partial reflection, transmission, and internal reflection involved. Therefore, like the position network, its structure can be similarly modified to alter its capabilities.

[0172] Similarly, when training on variations of one or more lighting conditions (as a non-limiting example, different lighting positions, orientations, and / or angular widths), it is advantageous that the orientation network can be designed to be larger due to the increased number or complexity of lighting conditions for a given scene surface location that needs to be modeled.

[0173] Therefore, in response to the needs of the required lighting model, an NPLM (e.g., a location network, a orientation network, or both) may change its capabilities (e.g., changes in its architecture, such as increasing the number of layers, internal nodes, or input or output dimensions), for example, to improve discriminative power (e.g., due to more hidden layers or output dimensions) and / or to improve model fidelity (e.g., due to more nodes in the hidden layers); for example, the demand for diffuse contribution components is generally lower than that for specular reflection components.

[0174] Conversely, based on the theoretical standards or default settings of NPLM, it is advantageous to modify NPLM as needed (e.g., for diffuse components) to reduce its capability (e.g., through steps opposite to those described above for increasing capability) rather than increasing capability. In this case, the benefits typically lie in reduced memory footprint and computational cost.

[0175] In addition to the type of reflective properties (or multiple properties) of a material modeled by different contribution channels, alternatively or additionally, the capabilities of NPLM can be increased or decreased in response to other factors related to the complexity of the lighting model / rendering process.

[0176] For example, diffuse light sources (such as a dome) can be less complex than point light sources because there is less spatial / angular variability in the lighting illuminating the object / scene. Conversely, a dome with significant spatial variability (e.g., showing a sunset) can be more complex. The complexity of a light source can be assessed based on its spatial and color variability, for example, based on the integral of a 2D Fourier transform of the lighting space in which there is no object / scene, typically without considering the DC component; in this case, a uniform dome would have an integral close to zero, while one or more point light sources would have a larger integral, and complex domes (such as cityscapes or sunsets) could have even larger integrals. The capabilities (e.g., size) of an NPLM can be set based on this or any such light source complexity analysis, for example, based on empirical analysis of performance.

[0177] Similarly, moving, dynamic, or placeable lights may increase the complexity of the NPLM because they create constantly changing lighting conditions. In this case, the NPLM input can include one or more lighting state inputs and (x, y, z) object positions, where specific portions of the object / scene are rendered as output pixels. Therefore, a model for a scene where the sun is crossing the sky could include inputs related to the time of day, which are related to the sun's position. Other inputs used to identify the current state of the light source could include (x, y, z) positions for one or more lights, (r) radii for light size, or similar inputs, and / or RGB inputs for the light's (primary) color, etc. It is worth noting that the training data (e.g., based on ray-traced ground truths) will also contain examples of these changing conditions. More generally, if the NPLM is trained to simulate the dynamic aspects of the environment, the training data will include an appropriate number of representative examples.

[0178] In the case of the sun, it may be necessary to simulate the movement of the entire day through several consecutive NPLMs (e.g., simulating dawn, morning, noon, afternoon, and dusk), for example, to avoid the memory footprint or computational cost of the NPLMs increasing beyond the preferred maximum, as described elsewhere in this document.

[0179] Similarly, if you want to use NPLM to render moving, dynamic, or placeable objects in a scene, you may need to increase the complexity of NPLM (optionally, NPLM can be used to render only static scene components and / or position-independent parts of the scene). Therefore, in this case, the input may include, for example, object position and / or orientation data.

[0180] Alternatively or additionally, other factors can simplify NPLM modeling and thus allow for a reduction in NPLM capabilities (or, for the sake of relatively higher model fidelity, all else being equal). For example, if the rendered scene includes fixed paths (e.g., on a racetrack, inside a crash barrier), training from viewpoints inaccessible to the user can be reduced or completely avoided. Similarly, if the rendered scene includes limited or preferred viewing directions (e.g., again on a racetrack, where most of the viewing is done in the driving direction), training for different viewpoints can reflect the proportional importance of those viewpoints to the final use case.

[0181] Similarly, in cases where users might pay less attention to parts of the scene because they are background or far from the focus of the game (based on foveated rendering or, for example, the main character's point of interest), the capabilities of the NPLM can be relatively low. For example, different NPLMs can be trained for different draw distances to objects or textures, where capabilities (e.g., size) decrease at different draw distances / levels of detail (LOD).

[0182] As an alternative or supplement, as described elsewhere in this article, NPLMs can be trained for specific scenes, objects, materials, or textures. Therefore, the performance of an NPLM can vary depending on the complexity of the object it represents for illumination. Large or complex scenes may require larger NPLMs (and / or multiple NPLMs handling their respective parts, depending on the scene size and the generated NPLM). Similarly, complex objects (such as cars) may benefit from a more powerful NPLM compared to simple objects (such as spheres). One way to assess the complexity of a scene or object is to count the number of polygons; the more polygons, the more complex the scene. As an improvement, the variance of the plane angles between polygons can also be used to infer complexity; for example, a sphere with the same number of polygons as the car model in the image shows very small angular variations compared to the car itself, indicating a more complex structure. Combining the number of polygons and the angular variance / distribution provides a good reflection of the complexity of the scene / object for which the NPLM simulates illumination.

[0183] Similarly, complex materials (such as skin or fur) may benefit from a more capable NPLM compared to simple materials (such as metals) and / or multiple NPLM contributors. Furthermore, complex textures (e.g., those with a wider spatial spectrum) may benefit from a more capable NPLM than textures with a narrower or more concise spatial spectrum.

[0184] While capability depends on size (number of inputs / outputs, number of layers, number of nodes, etc.), it can be alternatively or additionally varied by selecting activation functions among nodes on different layers of an NPLM. As described elsewhere in this paper, the preferred activation function for location networks is the ReLU function, while the preferred activation function for orientation networks is the sine function, but other functions can be chosen to simulate other scenarios.

[0185] For example, when the memory usage of NPLM reaches a threshold size, the capabilities of NPLM can be limited to an upper limit. The threshold size can be equal to the size of a memory operation unit, such as a memory page or a portion or group of memory pages, typically chosen for purposes such as accessing and loading textures of a scene / object / material. The threshold size can also be equal to the size of the texture or mimmap used by the GPU and / or the game to load graphics data into the GPU.

[0186] If the complexity of an NPLM exceeds this threshold, the task of modeling it may have to be simplified, shared among NPLMs, or the accuracy of the results may have to be accepted as lower.

[0187] Therefore, in summary, image rendering methods (focusing on network configuration and selection) may include the step of selecting at least a first trained machine learning model from multiple machine learning models, which has been trained to generate data that contributes to the rendering of at least a portion of the image, as discussed elsewhere in this paper. Thus, for example, the contributing data may relate to specific components of image pixels (e.g., components contributing to diffuse or specular reflections) or may relate to the entire RGB pixel set (e.g., modeling all reflection aspects at once), depending, for example, on the complexity of the lighting and / or materials, textures, and / or other surface properties being modeled.

[0188] In such an approach, at least the first trained machine learning model possesses an architecture-based learning capability that responds to at least a first aspect of the virtual environment in which it is trained to generate data, as discussed elsewhere herein. Thus, for example, the architectural aspect related to the learning capability can be the size of all or part of the NPLM, such as the number of layers or nodes, and / or can involve the nature of the connections between nodes in different layers (e.g., in terms of the degree of connectivity in terms of the type of activation function used).

[0189] In such a method, the second step may include using at least a first trained machine learning model to generate data that contributes to the rendering of at least a portion of the image, as described elsewhere herein. Thus, for example, a standalone run of an NPLM may generate data to be used with data from other NPLMs to generate RGB values ​​of the image's pixels, or may generate data that, for example, generates RGB values ​​of the image's pixels by itself, after subsequent processing (e.g., in combination with a distribution function) as described elsewhere herein.

[0190] Network selection

[0191] These networks are trained during the game or application development phase. Developers can choose when or where NPLM-based rendering is most advantageous. For example, it might only be used for scenes that consistently result in frame rates below a predetermined quality threshold. In this case, the network is trained on these scenes or a subset of them and used when those scenes are encountered.

[0192] In other cases, developers may choose to use NPLM-based rendering for certain objects or materials. In this case, networks will be trained and used when these objects or materials are identified as being in the scene to be rendered.

[0193] In other cases, developers may choose to use NPLM-based rendering for certain lighting conditions or combinations thereof. In this case, the network is trained and used when those conditions are identified as being within the scene to be rendered.

[0194] Similarly, developers can choose to use NPLM-based rendering at specific drawing distances (z-distances), or angles / distances from the image center or the user's gaze point, or under specific lighting conditions. In this case, the networks are trained and used for these specific scenarios.

[0195] Similarly, it should be understood that any appropriate combination of these standards can be selected for training and use.

[0196] Furthermore, as mentioned above, during system use, multiple NPLMs may exist associated with a scene for various reasons. For example, multiple NPLMs may exist to model a large scene (ensuring that each part is adequately modeled within a threshold size and / or a threshold quality of image reproduction). Similarly, multiple NPLMs may exist due to different lighting conditions, detail levels / draw distances, etc.

[0197] A suitable NPLM can be selected for this situation and restored to the GPU's accessible working memory, and run for at least a portion of the rendered image. It should be understood that strategies applied to prefetching and caching textures and other graphics assets can also be applied to the NPLM.

[0198] Summarize

[0199] Now for reference Figure 9 In the summary embodiment described herein, for a first element of a virtual scene having a predetermined surface at a location within the scene, an image rendering method for rendering pixels at a viewpoint includes the following steps.

[0200] In the first step s910, as described elsewhere herein, the position and orientation based on the viewpoint are provided to a previously trained machine learning system to predict a factor that, when combined with a distribution function characterizing the interaction of light with a predetermined surface, generates a pixel value corresponding to the first element of the virtual scene being illuminated at that location.

[0201] In the second step s920, as described elsewhere in this document, the predictor from the machine learning system is combined with the distribution function to generate a pixel value corresponding to the first illuminated element of the virtual scene at that location.

[0202] Furthermore, in the third step s930, as described elsewhere herein, the pixel values ​​are incorporated into the rendered image for display. The image can then be output to the display via the A / V port (90).

[0203] It will be apparent to those skilled in the art that one or more variations of the above methods corresponding to the operation of the various embodiments of the methods and / or apparatus described and claimed herein are considered within the scope of this disclosure, including but not limited to: - Train a corresponding machine learning system for each of the multiple contributing components of the image (e.g., diffuse, specular, coating, etc.), use a corresponding distribution function for each of the multiple contributing components of the image, and, as described elsewhere in this paper, combine the respective generated pixel values ​​to create pixel values ​​that are incorporated into the rendered image for display. - As described elsewhere in this article, the corresponding distribution function is one or more selected from a list consisting of a two-way scattering distribution function, a two-way reflection distribution function, and a two-way transmission distribution function; - As described elsewhere in this article, a machine learning system is a neural network whose first part takes position as input and whose second part takes the output of the first part and the direction as input. - In this case, as described elsewhere in this article, the activation function of the first part is different from the activation function of the second part; - In this case, as described elsewhere in this article, the activation function for the first part is the ReLU function, and the activation function for the second part is the sin function; - In this case, as described elsewhere in this paper, the cost function of the neural network is based on the difference between the output of the second part and the value derived from the ray-traced version of the pixels of the training image, on which an inverse combination with the distribution function has been performed; - In this case, as described elsewhere in this paper, the network's cost function is backpropagated through the second and first parts during training; - In this case, as described elsewhere in this article, the neural network is a fully connected network; - As described elsewhere in this paper, the cost function of the machine learning system is based on the difference between the output of the machine learning system and the values ​​derived from a ray-traced version of the pixels of the training image, on which an inverse combination with the distribution function has been performed; and - Based on the same asset recognition scheme used to select and load the texture of the first element of the scene, a machine learning system is selected and loaded into the memory used by the graphics processing unit.

[0204] Next, refer to Figure 10 In another summary embodiment of this specification, an image rendering method for rendering pixels at a viewpoint (focusing on changes in lighting conditions) includes the following steps.

[0205] For a first element of a virtual scene having a predetermined surface at a certain location within the scene, the first step s1010 includes: providing a viewpoint-based position and orientation to a previously trained machine learning system as a prediction factor, and generating a pixel value corresponding to the first element of the virtual scene illuminated at that location, as described elsewhere herein, when the factor is combined with a distribution function characterizing the interaction of light with the predetermined surface.

[0206] The second step s1020 then includes: combining the predictor from the machine learning system with the distribution function to generate the pixel value corresponding to the illuminated first element of the virtual scene at that location, as described elsewhere in this document.

[0207] The third step, s1030, then includes merging pixel values ​​into the rendered image for display.

[0208] However, it is worth noting that the machine learning system in the first step was previously trained using a training set based on images that included multiple lighting conditions, as described elsewhere in this article.

[0209] Furthermore, for those skilled in the art, one or more variations of the above-described methods corresponding to the operation of various embodiments of the methods and / or apparatus described and claimed herein are considered within the scope of this disclosure, including but not limited to: - Multiple lighting conditions include one or more changes to the lighting position, as described elsewhere in this article; -In this case, optionally, the change in lighting position is a function of one or more of a list consisting of nominal time within the virtual scene and the motion paths of movable virtual light sources, as described elsewhere in this document; - Multiple lighting conditions include one or more changes to the direction of illumination, as described elsewhere in this article; -In this case, optionally, the change in illumination direction is a function of one or more of a list including the change in the axial direction of the light source and the change in the angular spread of the light, as described elsewhere in this document; - Multiple lighting conditions include one or more variations of one or more selected from a list including lighting color, lighting brightness, and lighting diffusion, as described elsewhere in this document; - Train a corresponding machine learning system for each of the multiple contributing components of the image, apply the corresponding distribution function to each of the multiple contributing components of the image, and combine the generated pixel values ​​to create a final combination of pixel values ​​that are incorporated into the rendered image for display, as described elsewhere in this document; - Machine learning systems are neural networks. The first part of a neural network takes position as input, and the second part takes the output and orientation of the first part as input, as described elsewhere in this article. - Machine learning systems are neural networks, and the inputs to a neural network include one or more of the following: lighting position or offset, nominal time within the virtual scene, axis of the light source, angular spread of the light, color or color temperature, brightness, and diffusion, as described elsewhere in this document; -In the two cases described above where the machine learning system is a neural network, alternatively, if the input to all or part of the neural network represents more than one property of light, then an additional input layer with at least partial connectivity is provided to the neural network, as described elsewhere in this document; - The method includes the following steps: selecting at least one first trained machine learning model from a plurality of machine learning models, the machine learning model having been trained using a training set based on images including multiple lighting conditions to generate data that helps render at least a portion of the image, wherein the at least first trained machine learning model has an architecture-based learning capability that responds to at least a first aspect of a virtual environment to which it is trained to generate data, and using the at least first trained machine learning model to generate data that helps render at least a portion of the image, as described elsewhere herein; The method includes the following steps: using at least two corresponding machine learning models trained under different corresponding lighting conditions, generating pixel values ​​corresponding to a first element of a virtual scene illuminated at a surface location using the corresponding machine learning models; and combining the generated pixel values ​​into the output of a machine learning system, as described elsewhere in this document; and The method includes the following steps: using a machine learning model trained on a training set generated from a step of generating multiple candidate viewpoints of the scene, eliminating candidate viewpoints based on the probability of the scene surface responding to light at a surface location in the scene corresponding to the viewpoint, and generating training images at the remaining viewpoints, as described elsewhere in this document.

[0210] It should be understood that the above methods can be implemented on conventional hardware that is appropriately adapted to be applicable, either through software instructions or by including or replacing dedicated hardware.

[0211] Therefore, the required adaptation to existing portions of a conventional equivalent device can be implemented in the form of a computer program product comprising processor-implementable instructions stored on a non-transitory machine-readable medium such as a floppy disk, optical disk, hard disk, solid-state drive, PROM, RAM, flash memory, or any combination of these or other storage media. Alternatively, the computer program product can be implemented in hardware as an ASIC (Application-Specific Integrated Circuit) or FPGA (Field-Programmable Gate Array) or other configurable circuitry suitable for use in adapting to a conventional equivalent device. Separately, such a computer program can be transmitted via data signals over a network such as Ethernet, wireless networks, the Internet, or any combination of these or other networks.

[0212] refer to Figure 1 The methods and techniques described herein can be implemented on conventional hardware such as an entertainment system 10 that generates images from virtual scenes. An example of such an entertainment system 10 is a computer or console, such as the Sony® PlayStation 5® (PS5).

[0213] The entertainment system 10 includes a central processing unit 20. This can be a single-core or multi-core processor, for example, including the 8 cores in the PS5. The entertainment system also includes a graphics processing unit or GPU 30. The GPU can be physically separate from the CPU, or integrated with the CPU as a system-on-a-chip (SoC) like in the PS5.

[0214] The entertainment device also includes RAM 40, which can have separate RAM for each of the CPU and GPU, or shared RAM as in the PS5. This RAM, or each RAM, can be physically separate or integrated as part of the SOC in the PS5. Disk 50 provides further storage, either as an external or internal hard drive, or as an external solid-state drive, or as an internal solid-state drive as in the PS5.

[0215] The entertainment device can send or receive data via one or more data ports 60, such as USB ports, Ethernet ports, USB ports, or similar ports. It can also optionally receive data via an optical driver 70.

[0216] Interaction with the system is typically provided using one or more handheld controllers 80, such as a controller in the case of the PS5.

[0217] Audio / video output from entertainment devices is typically provided via one or more A / V ports 90, or via one or more wired or wireless data ports 60.

[0218] If the components are not integrated, they can be properly connected via a dedicated data link or via bus 100.

[0219] Therefore, in the summary embodiments of this specification, entertainment devices (such as Sony® Playstation 5® or the like) include the following.

[0220] First, a graphics processing unit (such as a GPU 30, optionally combined with a CPU 20) is configured (e.g., via appropriate software instructions) to render pixels at a viewpoint within an image of a virtual scene, which includes a first element having a predetermined surface at a location within the scene, as described elsewhere herein.

[0221] Secondly, a machine learning processor (such as a GPU 30, optionally combined with a CPU 20) is configured (e.g., via appropriate software instructions) to provide a previously trained machine learning system with a predictive factor based on the position and orientation of the viewpoint. This predictive factor, when combined with a distribution function characterizing the interaction of light with a predetermined surface, generates a pixel value corresponding to a first element of the virtual scene illuminated at the location, as described elsewhere herein.

[0222] The graphics processing unit is configured (again, for example, by appropriate software instructions) to combine predictors from a machine learning system with a distribution function to generate pixel values ​​corresponding to the illuminated first element of the virtual scene at that location, as described elsewhere in this document.

[0223] In addition, the graphics processing unit is also configured (again, for example, via appropriate software instructions) to incorporate pixel values ​​into the rendered image for display, as described elsewhere in this document.

[0224] It should be understood that the aforementioned hardware can be similarly configured to perform the methods and techniques described herein, such as: - The entertainment device includes multiple machine learning processors (e.g., corresponding processors, threads, and / or shaders of a GPU and / or CPU) running corresponding machine learning systems, each machine learning system being trained for one of multiple contributing components of an image (e.g., diffuse, specular, coating, etc.), wherein a corresponding distribution function is used for each of the multiple contributing components of the image, and a graphics processing unit is configured (again, e.g., via suitable software instructions) to combine the correspondingly generated pixel values ​​to create pixel values ​​incorporated into the rendered image for display, as described elsewhere herein; and - The machine learning system, or each machine learning system, is a neural network, wherein the input to the first part of the neural network includes position, and the input to the second part of the neural network includes the output and orientation of the first part.

[0225] Similarly, in another summary embodiment of the invention, the entertainment device (such as a Sony® Playstation 5® or the like) includes the following.

[0226] First, a graphics processing unit (e.g., GPU 30, optionally combined with CPU 20) is configured (e.g., via appropriate software instructions) to render pixels at a viewpoint within an image of a virtual scene that includes a first element having a predetermined surface at its location within the scene, as described elsewhere herein.

[0227] Secondly, the machine learning processor (e.g., GPU 30, optionally combined with CPU 20) is configured (e.g., via appropriate software instructions) to provide the viewpoint-based position and orientation to the previously trained machine learning system as a predictive factor, which, when combined with a distribution function characterizing the interaction of light with a predetermined surface, generates pixel values ​​corresponding to a first element of the virtual scene illuminated at that location, as described elsewhere in this document.

[0228] The graphics processing unit is configured (again, for example, via appropriate software instructions) to combine predictors from a machine learning system with a distribution function to generate pixel values ​​corresponding to the illuminated first element of the virtual scene at that location, as described elsewhere in this document.

[0229] In addition, the graphics processing unit is also configured (again, for example, via appropriate software instructions) to incorporate pixel values ​​into the rendered image for display, as described elsewhere in this document.

[0230] The machine learning system was previously trained using a training set based on images that included multiple lighting conditions, as described elsewhere in this paper.

[0231] It should be understood that the hardware described above can be similarly configured to perform the methods and techniques described herein, for example: - Multiple lighting conditions include one or more changes to the lighting position, as described elsewhere in this article; - Multiple lighting conditions include one or more changes to the direction of illumination, as described elsewhere in this article; - Machine learning systems are neural networks. The first part of a neural network takes position as input, and the second part takes the output and orientation of the first part as input, as described elsewhere in this article. - If the input to all or part of a neural network represents more than one property of light, then an additional input layer with at least partial connectivity is provided to the neural network, as described elsewhere in this document; - A processor is configured to select at least one first trained machine learning model from a plurality of machine learning models, the machine learning model having been trained using a training set based on images including multiple lighting conditions to generate data for at least a portion of a data image that is conducive to rendering; wherein the at least first trained machine learning model has an architecture-based learning capability that generates data in response to at least a first aspect of the virtual environment on which it is trained; and the at least first trained machine learning model is used to generate data that is conducive to rendering at least a portion of the image, as described elsewhere herein; The combined processor is configured to use at least two corresponding machine learning models trained under different corresponding lighting conditions to generate pixel values ​​corresponding to a first element of a virtual scene illuminated at a surface location; and to combine the generated pixel values ​​into the output of the machine learning system, as described elsewhere herein; and - A machine learning model trained using a training set generated from a step of generating multiple candidate viewpoints of the scene eliminates candidate viewpoints based on the probability of the scene surface responding to light at the surface location in the scene corresponding to the viewpoint, and generates training images at the remaining viewpoints, as described elsewhere in this document.

[0232] The foregoing discussion has only disclosed and described exemplary embodiments of the invention. As those skilled in the art will understand, the invention may be implemented in other specific forms without departing from the spirit or essential characteristics of the invention. Therefore, this disclosure is intended to illustrate, rather than limit, the scope of the invention. This disclosure, including any readily identifiable variations of the teachings herein, partially defines the scope of the foregoing terminology so that no inventive subject matter is contributed to the public.

Claims

1. An image rendering method for rendering pixels at a viewpoint, comprising the following steps: For the first element of the virtual scene having a predetermined surface at a location within the virtual scene: The position and orientation based on the viewpoint are provided to a previously trained machine learning system as a prediction factor, which, when combined with a distribution function characterizing the interaction of light with the predetermined surface, generates a pixel value corresponding to the first element of the virtual scene illuminated at the position. The predicted factors from the machine learning system are combined with the distribution function to generate pixel values ​​corresponding to the illuminated first element of the virtual scene at the location. as well as The pixel values ​​are incorporated into the rendered image for display. The machine learning system was previously trained using a training set based on images that included multiple lighting conditions. Train a corresponding machine learning system for each of the multiple contributing components of the image; Apply a corresponding distribution function to each of the multiple contributing components of the image; and The generated pixel values ​​are combined to create a final combined pixel value, which is then incorporated into the rendered image for display.

2. The image rendering method according to claim 1, wherein: The multiple lighting conditions include one or more variations in the lighting location.

3. The image rendering method according to claim 2, wherein: The change in lighting position is a function selected from one or more of the following lists: i. the nominal time within the virtual scene; and ii. The movement path of the movable virtual light source.

4. The image rendering method according to claim 1, wherein: The multiple lighting conditions include one or more variations in the direction of illumination.

5. The image rendering method according to claim 4, wherein: The change in lighting direction is a function selected from one or more of the following list, which includes: i. Axial variation of the light source; and ii. Changes in the angular spread of light.

6. The image rendering method according to claim 1, wherein: The multiple lighting conditions include one or more variations of one or more of the following lists, which include: i. Lighting color; ii. Illumination brightness; and iii. Illumination diffusion.

7. The image rendering method according to claim 1, wherein... The machine learning system is a neural network; The input to the first part of the neural network includes the position of the predetermined surface; and The input to the second part of the neural network includes the output of the first part and the direction.

8. The image rendering method according to claim 1, wherein The machine learning system is a neural network; and The input to the neural network includes data representing one or more of the following lists: i. Lighting position or offset; ii. The nominal time within the virtual scene; iii. The axis of the light source; iv. Angular spread of light; v. color or color temperature; vi. brightness; and vii. Diffuse.

9. The image rendering method according to any one of claims 7 to 8, wherein If all or part of the inputs to the neural network represent more than one property of light, then an additional input layer with at least partial connections is provided to the neural network.

10. The image rendering method according to claim 1, comprising: Select at least one trained machine learning model from a plurality of machine learning models, said machine learning model having been trained using a training set based on images including multiple lighting conditions to generate data that helps render at least a portion of the image; Wherein, the at least first trained machine learning model has architecture-based learning capabilities, which generate data in response to at least a first aspect of the virtual environment in which it is trained; and The at least first trained machine learning model is used to generate data that helps in rendering at least a portion of the image.

11. The image rendering method according to claim 1, comprising the following steps: Use at least two corresponding machine learning models trained under different corresponding lighting conditions; The corresponding pixel values ​​are generated using a machine learning model to represent the first element of the virtual scene illuminated at the surface location. as well as The generated pixel values ​​are combined to form the output of the machine learning system.

12. The image rendering method according to claim 1, comprising: A machine learning model is used, which is trained using a training set, which is generated through the following steps: Multiple candidate viewpoints for generating the scene; Candidate viewpoints are eliminated based on the probability of the response of the surface in the scene to light at a location on the surface corresponding to the viewpoint; and Generate training images at the remaining viewpoints.

13. A computer program comprising computer-executable instructions adapted to cause a computer system to perform the method of claim 1.

14. An entertainment device, comprising A graphics processing unit configured to render pixels at a viewpoint within an image of a virtual scene, the virtual scene including a first element having a predetermined surface at a location within the scene; A machine learning processor is configured to provide the position and orientation based on the viewpoint to a previously trained machine learning system to generate a prediction factor, which, when combined with a distribution function characterizing the interaction of light with the predetermined surface, generates a pixel value corresponding to the first element of the virtual scene illuminated at the position. The graphics processing unit is configured to combine the predicted factors from the machine learning system with the distribution function to generate a pixel value corresponding to the illuminated first element of the virtual scene at the location; and The graphics processing unit is configured to incorporate the pixel values ​​into a rendered image for display. The machine learning system was previously trained using a training set based on images that included multiple lighting conditions. Train a corresponding machine learning system for each of the multiple contributing components of the image; Apply a corresponding distribution function to each of the multiple contributing components of the image; and The generated pixel values ​​are combined to create a final combined pixel value, which is then incorporated into the rendered image for display.

Images