Image rendering method and apparatus

Abstract

To provide an image rendering method and apparatus.SOLUTION: An image rendering method for rendering a pixel at a viewpoint comprises the steps of, for a first element of a virtual scene having a predetermined surface at a position within that scene: providing the position and a direction based on the viewpoint to a machine learning system previously trained to predict a factor that, when combined with a distribution function that characterizes an interaction of light with the predetermined surface, generates a pixel value corresponding to the first element of the virtual scene as illuminated at the position; combining the predicted factor from the machine learning system with the distribution function to generate the pixel value corresponding to the illuminated first element of the virtual scene at the position; and incorporating the pixel value into a rendered image for display. The machine learning system is previously trained with a training set based on images comprising multiple lighting conditions.SELECTED DRAWING: Figure 10

Description

Technical Field

[0004] , , , , , ,

[0006] , ,

[0005] ,

[0007] , , , , ,

[0003]

[0001] The present invention relates to an image rendering method and apparatus.

Background Art

[0002] Ray tracing is a rendering process in which the optical path is traced within a virtual scene. Next, the interaction of each ray at an object or surface within the scene is simulated. To achieve a degree of realism, typically this simulation takes into account the material properties of these objects or surfaces, such as color and reflectivity.

[0003] As a result, ray tracing is a computationally expensive process. Furthermore, the cost varies for each image frame depending on which scene, which lighting, and from which viewpoint the light is applied.

[0004] This makes it difficult to maintain a preferable frame rate for rendering such images, and for the average calculation cost corresponding to the average image completion time (i.e., frame rate) and a predetermined variation near that average value by ray tracing, either the average image quality must be set low enough that the variation has little effect on the frame rate, or if the average image quality is set close to the maximum value of the preferable frame rate, the consistency of the frame rate must be sacrificed when the demand for ray tracing varies more than the average.

[0005] Neither of these is a desirable result, but since the computational load of the ray tracing process is data-driven and unpredictable, it cannot be easily avoided.

[0006] The present invention attempts to address or mitigate this problem.

Summary of the Invention

[0007] Various aspects and features of the present invention are defined in the appended claims and within the text of the accompanying description, at least -The image rendering method according to claim 1 in the first example, -An entertainment device according to claim 15 in the second example, Includes.

[0008] A more complete understanding of this disclosure and many of its associated benefits will be readily apparent as it is better understood by referring to the following detailed description in conjunction with the attached drawings. [Brief explanation of the drawing]

[0009] [Figure 1] This is a schematic diagram of an entertainment device according to an embodiment of this specification. [Figure 2] This figure illustrates a ray-traced object according to an embodiment of this specification. [Figure 3] This is a schematic diagram of contributing components to a ray-traced object according to the embodiments of this specification. [Figure 4] This is a schematic diagram of the distribution function associated with each component according to the embodiment of this specification. [Figure 5] This is a schematic diagram of the scattering distribution according to the embodiment of this specification. [Figure 6] This is a schematic diagram of a training scheme for a machine learning system according to an embodiment of this specification. [Figure 7] This is a schematic diagram of the rendering path for a rendered image according to the embodiment of this specification. [Figure 8A] This is a schematic diagram of a machine learning system according to an embodiment of this specification. [Figure 8B] This is a schematic diagram of a part of a machine learning system according to the embodiment of this specification. [Figure 9] This is a flowchart of an image rendering method according to an embodiment of this specification. [Figure 10] This is a flowchart of an image rendering method according to an embodiment of this specification. [Figure 11]This is a schematic diagram of a training method according to an embodiment of this specification. [Modes for carrying out the invention]

[0010] Image rendering methods and apparatus are disclosed. The following description provides many specific details in order to fully understand embodiments of the present invention. However, it will be apparent to those skilled in the art that these specific details are not necessary to carry out the present invention. Conversely, specific details known to those skilled in the art are omitted where appropriate for clarity.

[0011] Embodiments of this specification attempt to address or mitigate the above-mentioned problems by using a machine learning system that learns the relationship between pixel surface properties and rendered pixels for a given object or scene. By using such a machine learning system, it is then possible to approximate a ray-traced rendering of the object or scene based on a relatively consistent computational budget (for running the machine learning system).

[0012] Different machine learning systems can be trained on different scenes, locations or parts thereof, or different objects or materials for use within one or more scenes, as will be described later in this specification.

[0013] Machine learning systems are relatively small (typically on the order of 100KB to 1MB), and therefore, for the purpose of running on a GPU (30), they can be pulled into memory like scene textures and then discarded. The system can be run by the GPU's shaders. It will also be understood that, in principle, machine learning systems can be run alternatively or additionally on a CPU (20), or on general or specialized coprocessors such as neural network processors or ASICs.

[0014] Next, referring to the drawings, like reference numerals indicate identical or corresponding parts throughout several views, and FIGS. 2-7 are diagrams showing the problem space in which the machine learning system is trained.

[0015] FIG. 2 is an example object or scene, in this case a high-quality ray-traced rendering 200 of a car on an altar.

[0016] FIG. 3 is a diagram showing the different contributing components behind this rendering. First, the diffuse illumination component 200-D typically captures the matte color of the surface and the shading resulting from the interaction of light and shape. On the other hand, second, the specular illumination component 200-S captures the reflectivity of the surface, resulting in glints and highlights. Optionally, it can include one or more additional components, such as a gloss or "coat" 200-C, which is a second outer surface that includes additional gloss or patterning. Variations of such coats can allow for partial transparency and / or partial diffusion, for example, in a manner similar to skin or fabric. Each of these components can be conventionally generated using individual ray-tracing processes.

[0017] These components are added together to form the overall image as seen previously in FIG. 2. Typically two or three such components contribute to the rendering, but it will be understood that depending on the situation, a smaller number (e.g., if only the diffuse component is desired) or a larger number (e.g., if the object is translucent and requires a transmission component) may be present.

[0018] Next, FIG. 4 includes the material properties of the object that give rise to the contributing components of the image.

[0019] The material properties are represented as the so-called bidirectional scattering distribution function (BSDF) or bidirectional reflectance distribution function (BRDF).

[0020] The BRDF defines how light is reflected by an opaque surface, and similarly the BSDF defines the probability that a ray of light is reflected or scattered in a particular direction. Thus, the BRDF or BSDF is a function that describes the illumination properties of a surface (excluding the incident / outgoing radiance itself). Other functions, such as the bidirectional transmittance distribution function (BTDF) that defines how light passes through a material, may also be used as appropriate.

[0021] Referring to FIG. 5, in a typical ray tracing application, for a set of rays (e.g., from a compact light source), the incident radiance (which itself is direct or previously reflected) onto a point on a model having a particular BSDF, BRDF, and / or BTDF is calculated. The incident radiance is combined (e.g., multiplied) with the BSDF, BRDF, or BTDF with respect to the response of a particular contribution component, and the result is added to the pixel value at that point on the model. As shown in FIG. 5, a typical scattering pattern of a ray ωi in the BSDF is biased in the specular reflection direction ωa, but can scatter in any direction. Accurately modeling such behavior is one of the reasons for the high computational cost of ray tracing.

[0022] Using the color information of the model at each point and (i.e., for a particular material represented by a given point) the BSDF, BRDF, and / or BTDF corresponding to that point, the behavior of the ray with respect to a given final viewing point can thus be calculated by, for example, ray reflectance or scattering that determines the realistic distribution of glints and highlights on the surface of a vehicle.

[0023] A separate BSDF, BRDF, or BTDF may be used for each contributing component. Thus, in a non-limiting example, a BSDF may be used for a diffusion component, a BRDF for a specular component, and in this example, also for a coating component (although a BTDF may also be used for such a coating component). It is understood that any of BSDF, BRDF, or BTDF may be used as appropriate, and therefore, hereafter, references to BSDF shall include references to BRDF or BTDF as appropriate, unless otherwise specified.

[0024] As shown in Figure 4, performing ray tracing using the object's color properties and the BSDF's diffuse material properties (200-BSDF-D) yields the diffuse image component 200-D. Similarly, using the BSDF's specular or reflective material properties (200-BSDF-S) yields the specular image component 200-S. Likewise, using the material properties of the BSDF (200-BSDF-C), which is also typically specular, yields the coated image component 200-C. Combining these components yields the final ray-traced image 200.

[0025] However, the problem is that, as mentioned earlier, calculating the reflection and scattering paths of light rays intersecting different surfaces with different BSDFs and summing the results for each pixel of the scene at a particular viewpoint is computationally expensive and potentially highly variable.

[0026] Therefore, embodiments of this specification seek to replace the ray tracing step in Figure 4 with something else that has a more predictable computational load for a final image of adequate quality.

[0027] Referring also to Figure 6, in the embodiments of this specification, a machine learning system is provided for each contributing component of the image (e.g., diffuse, specular, and optionally coated or any other contributing component).

[0028] The machine learning system is typically a neural network trained to learn the transformation between BSDF (e.g., 200-BSDF-D) and ray-traced ground truth (e.g., 200-D) of the contributing components of an image for multiple images at different viewpoints in a scene, as will be discussed later herein.

[0029] In other words, if a ray-traced image (or one of its contributing components) is a combination of a BSDF that describes how lighting is reproduced on an object and how that object reacts to the light, then taking the ray-traced image and uncomboing it with the BSDF will result in a quality that could be called "radiance" or "shading," but more generally, it describes how light is reproduced on an object (which is collectively calculated by the ray tracing process).

[0030] If a machine learning system or neural network can learn to predict this quality, it can be recombined with a BSDF to generate a predicted image that approximates the ray-traced image. Therefore, this network is sometimes called a neural pre-computed light model or NPLM network.

[0031] More specifically, given an object's position on a virtual image and its viewing orientation, the machine learning system or neural network must learn to output a value that, when coupled with a BSDF for that same position / pixel, yields a pixel value similar to what would result from ray tracing of the image at that pixel. As a result, during training, it generates an internal representation of the lighting conditions (e.g., by a point light source or skydome) and surface illumination properties suggested by the training image.

[0032] Therefore, in an exemplary embodiment, the image is rasterized or otherwise generated at a given viewpoint, which will then fill the image with pixels to be illuminated. For each of these assumed pixels, there exists a corresponding 3D position in the scene where an appropriate "luminance" or "shading" can be obtained using an NPLM network.

[0033] Figure 6 shows a learning environment for such a network, specifically network 600-D for diffusion contributing components, as an example only.

[0034] The inputs to the network for the diffusion contributing components are the (x,y,z) position 610 on the object or scene (e.g., corresponding to a pixel in the image) and the object / scene normal 620 at that point. The normal N is used instead of the viewpoint direction because, for the diffusion contributing components, illuminance can be considered independent of direction / viewpoint, and therefore the known value of the normal can be used for consistency. Figure 6 conceptually illustrates these inputs using representative values ​​for the automobile image in the example described here.

[0035] Optionally, additional inputs may be provided, such as roughness or matte-gloss scalar values ​​that can be derived from the relevant BSDF (not shown).

[0036] The output of the NPLM network (described later in this specification) is a learned quality or illuminance of light 630 for the input position, and when combined with the associated diffuse BSDF (200-BSDF-D) 640 for the same position, the predicted pixel value for the (x,y) position in the predicted image 650 is generated.

[0037] Figure 6 illustrates that the pixel-by-pixel difference between the predicted pixel and the ground truth pixel of the target ray-traced diffuse element 200-D is used as the loss function for training the network. However, this is not necessary; rather, the ground truth image is decoupled from the BSDF (i.e., by performing an inverse function) to generate a proxy for each (x,y) pixel of how the ray-traced light cumulatively affects the object in the image, which is the quality the network trains to learn.

[0038] Therefore, the network's error function is based on the difference between the output value of one pixel (x,y) and the corresponding pixel (x,y) in the ground truth image if it were not synthesized from the corresponding BSDF at that location.

[0039] Since the pixels of the ground truth image can be pre-decoupled from their corresponding BSDFs at each location, the network can learn without needing to combine its output with any arbitrary BSDF to generate the actual predicted image pixels. This reduces the computational load during training.

[0040] As described above, the trained quality output by the trained neural network captures how light in the environment is reproduced on objects or scenes depending on the position of surfaces in the scene and the viewpoint. In this way, it effectively generates an internal representation of the light map and surface response model for the scene. How this occurs will be discussed in more detail later in this specification.

[0041] Referring to Figure 7, to summarize for each contributing element of the final output image, the machine learning system is trained to perform a transformation applied to the BSDF, which is locally applied to the location on the object / scene for that contributing element. This transformation is a trained function based on the (x,y,z) position and orientation values ​​of the point on the object / scene. As mentioned above, one, two, three, four, or more machine learning systems may be employed, depending on the number of contributing elements of the final image. Hereafter, the term "trained function" may be used to refer to a machine learning system that has learned such a transformation.

[0042] As noted regarding the diffuse components, diffuse shading is assumed to be direction-invariant, so the direction value can be assumed to be equal to the normal at a given point.

[0043] On the other hand, for specular elements that will change depending on the viewpoint because they are at least partially reflective, the directional value is the viewing angle between the current (x,y) position of the pixel at the viewpoint of the image (which has a position in virtual space) and the (x,y,z) position of the object as input to the machine learning system, or based on this, providing a viewpoint dependency between the input point on the scene surface and the current pixel from which the learned quantity will be output.

[0044] In this case, since the coat components are also reflective, a similar viewpoint or a viewpoint based on the direction of the input is used.

[0045] Therefore, the direction value of the direction-dependent component may be the viewpoint direction between the output pixel position and the object surface position, or a value based thereon, such as the surface specular viewpoint direction (i.e., considering the normal of the surface at the input position, the viewpoint direction is the principal direction of reflection). Any appropriate direction value incorporating viewpoint direction information can be considered.

[0046] In each case, the trained function encapsulates the learned quality as previously described herein. By combining the appropriate BSDF with the network output at each location, a shaded image of each component can be constructed. Alternatively or additionally, the final output is generated by combining the pixel values ​​of the shaded images from each component.

[0047] It should be understood that not all parts of an image are subject to ray tracing during rendering, nor are all parts of an image generated using the techniques described above. For example, an NPLM network may be trained on a specific object or material based on a ray-traced image of ground truth with representative lighting.

[0048] When these objects or materials are subsequently rendered in real time using apparent ray tracing provided by the trained functions described herein, the relevant NPLM network, when loaded into memory and coupled with the appropriate BSDF, is run on the relevant surface positions and line-of-sight directions in the scene to generate their contributions to the relevant pixels. Other pixels may be rendered using any other suitable technique, including ray tracing itself.

[0049] Typically, a suitable machine learning system is selected and loaded into memory used by the GPU based on the same asset identification scheme used to select and load textures of objects or materials. Therefore, for example, if an object has the ID "1234" used to access its associated texture, this ID can also be associated with the associated machine learning system. Conversely, if a texture has the ID "5678" associated with an object (for example, if the texture represents a material common to multiple objects), this ID can also be associated with the associated machine learning system. In this way, an entertainment device can load a machine learning system using a process similar to that used to load textures. If textures are stored in lossy compression that affects the behavior of the decompressed machine learning system, it will be understood that the actual storage and access techniques may differ between the texture and the machine learning system. Therefore, the machine learning system may be stored without compression, or with lossless compression, or with lossy compression where the degree of loss is low enough that the decompressed machine learning system still functions properly. This can be evaluated by comparing the output error / cost function of the machine learning system with respect to the increment of the compression loss until the error reaches an absolute or relative (for an uncompressed machine learning system) quality threshold.

[0050] Referring here to Figure 8A, in the embodiments herein, the machine learning system or NPLM network can be any suitable machine learning system. For example, a single neural network may be trained using position and viewpoint orientation as inputs and generate RGB values ​​for the learned properties as outputs.

[0051] However, particularly advantageous networks are constructed from clearly defined partitioned architectures.

[0052] As shown in Figure 8A, in a non-restrictive example, the network consists of two parts. The first part can be considered a location network, while the second part can be considered a direction network.

[0053] Each of these networks has three or more layers and can use any appropriate activation function.

[0054] The position network takes the aforementioned (x, y, z) position of a point within an object / scene as input and outputs an intermediate representation described below.

[0055] The directional network receives this intermediate representation and directional inputs, such as in (θ, φ) form or normalized (x, y, z) vectors (e.g., normals, or pixel viewpoints, surface point mirrored pixel viewpoint direction, or other viewpoint-based directional values). It outputs RGB values ​​corresponding to the previously learned quantities for the current (x, y) position of the pixel in the image rendered from a virtual camera position in a space shared with the object / scene (and therefore the pixel viewpoint).

[0056] Therefore, in a non-restrictive example, the location network has 3 layers, with 3 input nodes in the first layer (for example, for x,y,z positions), 128 hidden nodes in the intermediate layer, and 8 outputs in the final layer.

[0057] While any suitable activation function can be selected for the network, the rectified linear unit (ReLU) function was evaluated as particularly effective as the interlayer activation function for location networks. This generalizes well to untrained locations and helps avoid overfitting.

[0058] Similarly, in a non-restrictive example, the directional network has four layers, where the 8 outputs of the position network and 2 or 3 additional values ​​for direction are fed into the 128 nodes of the first layer, then into two further layers of 128 nodes, and finally output 3 outputs of the last layer corresponding to the R, G, B values ​​for the learned quantities at the current pixel. This can then be coupled (e.g., multiplied) with the BSDF of its position to obtain the final pixel contribution from this trained function (e.g., diffuse, specular, etc.), but as mentioned earlier, this is not necessary during training.

[0059] While any suitable activation function can be selected for a directional network, the sine function has been evaluated as a particularly effective activation function between layers of a directional network. Although the optical behavior variation in the angular domain is large and involves details at many angular frequencies, the sine activation function has proven to be particularly good because it is based on low-dimensional inputs (e.g., normalized x,y,z vectors).

[0060] It is noteworthy that, therefore, the two halves of the network may use different activation functions.

[0061] However, the network is treated not as two separate networks, but as a split-architecture network, because, in particular, the learning scheme has only one cost function: the error between the RGB values ​​output by the directional network and the target values ​​from the corresponding pixels of the ray-traced ground truth image after it has not been combined with the appropriate BSDF.

[0062] This error is backpropagated through both networks, and the position network does not have a separate target value or cost function. Thus, in effect, the output layer of the position network is a hidden layer of the connected network reinforced with an additional input of directional values, representing a transition within the layer from the first activation function to a possible second different activation function.

[0063] As mentioned earlier, neural networks construct light models for illuminated objects, materials, or scenes. In particular, in the non-restrictive example above, the position network effectively sorts (x, y, z) positions by illumination type (e.g., bright or dim, and / or other categories related to how light interacts with each BSDF, such as relative reflectivity or diffuse). The intermediate representation output by this part can be thought of as an N-dimensional position in illumination space characterizing the type of light at the input position, and will project positions in different parts of the scene to the same N-dimensional position if they were illuminated in the same way. A position network trained for specular components may have more outputs than one for diffuse components, for example, 32 outputs compared to 8 to account for greater variation in the type of illumination that can occur with specular components.

[0064] Next, the orientation network models how the light model behaves when viewed on a surface at an input position at a given input angle for an illuminated object, material, or scene, and generates learned properties for that position in the image.

[0065] Therefore, in summary, the position and orientation networks are trained together as one to predict coefficients or transformations between a BSDF describing surface properties and a desired rendered image of that surface. This network can then be used as an alternative to ray tracing for rendering that surface. Typically, though not always, the network is trained on only one contributing component of the image, such as diffuse specular components, and multiple networks are used to generate the components needed for the final image or part of the image, but this is not required (i.e., in principle, the network can be trained on a fully coupled image or a combination of two or more contributing components, e.g., all specular or all diffuse contributions).

[0066] training The network is trained using multiple ray-traced images of an object, scene, or surface taken from multiple different viewpoints, as described elsewhere in this specification. This allows the network to learn in particular how specular reflections change with position. The viewpoints may be randomly distributed and / or may be selected (or preferentially selected) from a range of viewpoints available to the user when navigating the rendering environment, known, for example, as the volume of views, i.e., the volume of space in which viewpoints can arise, and therefore should be included in the training.

[0067] In one embodiment of this specification, training data may be generated as follows:

[0068] It is understood that for any machine learning system, the training data used to train the system can be key to its performance. Therefore, generating training data that leads to good performance is extremely beneficial.

[0069] As described elsewhere in this specification, the training data for the NPLM system described herein is based on a set of high-quality rendered images of scenes / objects / materials / surfaces (hereinafter generally referred to as scenes) and is typically decoupled with one or more related distribution functions (e.g., BSDF, BRDF, or similar ones as described elsewhere in this specification). As a result, the learned quality referred to herein is provided as a direct training target, eliminating the computational burden of generating predictive images during training and ensuring that the error function is not derived solely from the output of the NPLM itself.

[0070] Different NPLMs can handle view-dependent and view-independent shading effects (e.g., diffuse, specular, etc.). Therefore, typically, if an object has view-dependent shading (e.g., specularity, or mirror reflection, etc.), a single view of the object in the scene is insufficient.

[0071] As a result, the number and location of training data images may depend not only on the scene's geometry (e.g., whether an object is visible within the view's volume) but also potentially on the material properties of objects within the scene.

[0072] Accordingly, in embodiments of this specification, NPLM training data in the form of images of a scene captured from multiple camera viewpoints can be generated at least in part based on the materials in the scene (material properties such as, for example, optical response properties such as diffuse or specular response, but potentially other material properties such as surface complexity—for example, the presence of narrow or wide spatial frequency components in terms of structure and / or texture).

[0073] These images are typically generated by third-party high-quality rendering units, and access to their internal data is unavailable. Therefore, only the final, complete image is available, and any information (or control) regarding specific cast rays or their directions when performing shading within the image may not be available.

[0074] Therefore, for training purposes, it is desirable to generate and use a set of images that efficiently capture the appearance of the scene for preferably all valid views within the volume of views.

[0075] Referring here to Figure 11, for this purpose, in step 1110, first, a set of camera positions within the view volume is used to render a set of low-resolution images. The positions may be equidistant or randomly distributed on a sphere around the scene (if it can be viewed from any angle, such as an interactive object), on a hemisphere around the scene (if it cannot be viewed from below because it is based on a virtual ground), or on a ring around the scene (if it can be viewed from a ground-based viewpoint, such as the first-person view of an avatar). Such a ring may be at a fixed height corresponding to the avatar viewpoint, or it may occupy a height range such as a viewing cylinder that encompasses one or more of the crouch height and jump height of the avatar viewpoint.

[0076] Step 1110 shows the trajectory (ring) of the camera position around the car object illustrated in Figure 11.

[0077] The number of camera positions in this initial set may be as few as one, but is typically three or more, and more typically on the order of tens or hundreds. For example, if there is one camera per degree of orbit, that would be 360 ​​cameras. In this embodiment, an indefinite number of 200 cameras is used.

[0078] The resolution per image is low, for example, 128 x 84 pixels. An example image from step s1120 is shown.

[0079] Of note is that, for each pixel in each image, in step s1130, metadata is associated with it, including the 3D position of the scene surface corresponding to the pixel, the normal of the scene surface corresponding to the pixel, and optionally a material surface identifier or descriptor such as a texture ID or object ID.

[0080] In the first example of the viewpoint selection process, the 3D positions of the scene surface rendered by pixels in some or typically all of these low-resolution images are matched to identify which positions in the scene are visible within a first set of camera positions. These are the 3D positions that benefit from the NPLM being trained.

[0081] Therefore, optionally, for each 3D position identified as rendered in at least one of the initial low-resolution images, a new position in 3D space is calculated as an offset from that position along the surface normal. The distance of the offset from the surface is a variable that can be changed. This new position is a candidate viewpoint for a virtual camera to generate high-quality (e.g., high-resolution ray tracing) renderings.

[0082] However, this results in a large number of potentially high-quality ray-traced renderings to generate as training images, which can be computationally burdensome and may contain significant redundancy when used as a training set for NPLM.

[0083] Therefore, in the first example, it is desirable to filter or culling these candidate viewpoint locations in some way that is relevant and useful to training the NPLM on the scene.

[0084] In particular, it is beneficial to have more training examples for parts of a scene that include view-dependent materials (e.g., specular or glossy) than for view-independent materials (e.g., diffuse or matte).

[0085] Therefore, we can take one of two approaches.

[0086] In the first approach, in step 1140, for each candidate viewpoint corresponding to the normal at the surface location, the corresponding material properties of the surface at that location are examined. As mentioned above, in particular, its diffusion response, specular response, or translucency may be used.

[0087] In practice, this can be done by using a lookup table that associates material IDs, etc., with values ​​indicating the diffusivity or specularity (e.g., matte or glossy) of the material surface. More specifically, this property can be represented, as an unrestricted example, by a value ranging from 0 for perfectly diffuse to 1 for specular reflection. This can be treated as input to a probability function, so that specular or glossy (view-dependent) materials have a relatively high probability, and diffusive or matte (view-independent) materials have a relatively low probability.

[0088] This probability function is used to retain candidate camera positions. Therefore, camera positions facing the mirrored surface will be retained at a higher rate than those facing the diffuse surface.

[0089] Conversely, if the rule for the values ​​is reversed (for example, the low probability and high probability are reversed), the probability function can be used to eliminate candidate camera positions with the same effect.

[0090] In the second approach, alternatively or additionally, in step s1140, the variation in pixel values ​​corresponding to the same 3D position on the scene surface as seen in the low-resolution image may be evaluated to determine the variance of the pixel values ​​for each captured 3D position. In this method, view-invariant (e.g., diffuse or heavily shaded) surface positions have low variance (i.e., pixels indicating that position are similar in different low-resolution images), while view-dependent (e.g., specular or glossy) surface positions have high variance (i.e., pixels indicating that position will show a wider range of values ​​in different low-resolution images because they capture some glare or reflection, for example). This variance, or its normalized version, can again be used as input to a probability function so that specular or glossy (view-dependent) materials have relatively high probabilities, and diffuse or matte (view-independent) materials have relatively low probabilities.

[0091] Therefore, in either case, step s1140 provides an estimate of the view dependence of the material's photoresponsiveness at each captured 3D location within the view volume (based on material properties, pixel variations, or potentially both), which can be used as input to a stochastic function.

[0092] The probability function is then used in step s1150 to determine whether each candidate viewpoint is retained or culled, with view-dependent surface-centered viewpoints being retained more frequently than view-independent surface-centered viewpoints.

[0093] The output range of this probability function can be adjusted to approximately generate the desired total number of camera views for training, based on the initial number of possible candidates and the final desired number; or, the probability function can be applied to a series of retain / culling rounds until the number of remaining camera views falls within the desired threshold.

[0094] In either case, the result is a manageable number of camera views randomly distributed across the desired volume of view, but with a variable probability density that responds to material properties (e.g., gloss or other) of the material that immediately concentrate in front of the camera. This is illustrated by the constellation of surviving points shown in the figure of step s1150. In practice, the camera position can be further from the object / scene surface than shown in this figure, but the points are placed close to the surface in the figure to illustrate their distribution.

[0095] The manageable number of camera viewpoints can be selected based on factors such as the desired performance of the resulting NPLM, the computational load of generating high-quality ray-traced images and training the NPLM on them, and memory or storage constraints. A typical manageable number for training purposes may be 10 to 10,000 in non-limiting examples, and a typical number is 200 to 2,000.

[0096] Finally, in step s1160, the images are rendered from the perspective of the surviving player. Optionally, as shown in Figure 11, these renderings are generated using a virtual lens with a wider angle than the lens used for the initial low-resolution images or the lens used during gameplay.

[0097] This tends to result in rendering too much of the scene (i.e., parts not directly visible from the viewpoint of the view volume). This tends to make the NPLM output more robust, especially for view positions near the edges of the view volume, in cases of unexpected expansion of the view volume due to, for example, clipping of objects in the game or minor design modifications.

[0098] The approach described above generated candidate camera viewpoints based on the normals of the scene surface captured in the initial low-resolution image, but this is not the only possible approach.

[0099] One possible problem with the above approach is that while a view-invariant position in the scene may be captured by a camera oriented in that direction along the normal at that position, it may only be rendered from different angles in other images of nearby positions, and these angles are now determined by the normals of the scene surface at those positions. As a result, there will be a relatively large number of images of the view-dependent parts of the scene or their vicinity, while the images themselves may be excessively influenced by the geometry of the scene itself.

[0100] Therefore, returning to the initial low-resolution images, in another example of the viewpoint selection process, potential viewpoint positions can be considered for each pixel (or at least for these pixels representing the surface of the scene) in each low-resolution image. In the example of 200 images of 128x84 pixels above, there would be up to 1.6 million candidates. These images typically capture multiple instances of a particular location on the scene from different angles, independent of the topology of the scene itself. As a result, the training set is potentially more robust.

[0101] Again, the probability of retaining or culling a viewpoint can be manipulated using the surface material (and / or pixel variance) derived from the view dependency of a given pixel in the low-resolution image, and therefore the surface position corresponding to the candidate viewpoint. In this way, the 1.6 million candidate viewpoints can be narrowed down again to a manageable number.

[0102] In this case, since there may be multiple viewpoints at the same location in the scene, the resulting distribution of camera views may be biased towards the most visible location in the scene rather than depending on most views. For example, if a certain (diffuse) location in the scene is visible in more than 20 times the image of a certain (spectral) location, viewpoints viewing the diffuse location are more likely to be culled, but because the difference is more than 20 times, the images of the diffuse location may ultimately outnumber those of the reflective location.

[0103] Therefore, optionally, the probability of retaining or culling a viewpoint can be normalized based on the number of viewpoints centered on the same location in the scene (albeit from different angles). This normalization may be complete (thus, in the example above, the probability of retaining an image at a diffuse location is reduced 20-fold, thus eliminating the effect of the number of viewpoints). Alternatively, the normalization may be partial. For example, the probability of retaining an image at a diffuse location is reduced only 10-fold, significantly reducing the effect of the number of views, but not completely eliminating it. This means that areas likely to be seen frequently by the user will receive more training examples, regardless of whether many training examples are obtained due to view dependency (e.g., specular / glossy).

[0104] In principle, both sets of viewpoints (surface normal-based viewpoints and low-resolution image pixel-based viewpoints) can be generated and culled to create a combined set of viewpoints before producing high-quality ray-traced renderings for training purposes. In practice, in either case, there is likely to be a substantial subset of low-resolution image pixel-based viewpoints that coincide with at least some normals of visible surface locations.

[0105] Distributed training technology The second approach described above optionally considers the problem of compensating for multiple views of the same location in the scene when culling available viewpoints. In addition to allowing control over training bias, this second approach reduces training time by reducing repetition for specific locations in the scene.

[0106] However, alternatively or additionally, training time can be (further) reduced as follows:

[0107] As mentioned above, select the first set of viewpoints within (or on the surface of) the view volume.

[0108] Here, optionally, generate an initial low-resolution image for a set of locations within the view's volume.

[0109] Here, optionally, candidate viewpoints are then generated based on the normals of locations in the scene found in the low-resolution image, and / or on the lines between the pixels of the low-resolution image and the represented locations in the scene, as described above herein.

[0110] Again, optionally, these viewpoints can be culled based on probabilities derived from the degree of specularity / diffusivity of each location in the scene. Furthermore, optionally, if there are multiple images centered on each location, the probabilities can be modified to at least partially account for this.

[0111] Therefore, depending on the approach taken, the result is either a set of generated viewpoints, the original distribution used at will to generate the low-resolution images, or a distribution resulting from one of the generation and culling techniques described above.

[0112] In any case, in the embodiments herein, once a viewpoint is generated (and optionally, where appropriate, confirmed not to be culled), it is provided for or queued for ray tracing processing to generate high-quality images, optionally in the form of a wide angle, as described elsewhere herein.

[0113] Training on the generated images begins when each image is completed. Therefore, the generation of training images (which takes some time because they are ray-traced images) and the training on those images (which can also take some time) occur in parallel. This avoids the problem of having to wait for a complete training set to be generated before starting training.

[0114] If, by choice, viewpoints are generated, or if generated viewpoints are selected to determine whether they are culled, the selection of viewpoints from these available ones can be random, and consequently, the final generation sequence of ray-traced images will also be random within the final set of viewpoints used.

[0115] This reduces the likelihood of NPLM being overtrained initially in one section of the scene, and also means that NPLM is still exposed to a diverse set of views of the scene, for example, if training must be scaled back due to time constraints.

[0116] In another distributed training technique, if control over the ray tracing application is available and enabled, it is optional to render only a subset of pixels from a given viewpoint. Within a given image, there may be parts of the scene that have been rendered multiple times in other images within the training set, whether based on the original set of viewpoints or on viewpoints that were not culled. For example, if a location in a scene has already been rendered more than a threshold number of times, it may be skipped in the current rendering because there are already enough training examples for that location. Unrendered portions of an image can be tagged with pixel values ​​that act as mask values. As a result, training can be performed using only the input location, orientation information, and target values ​​for unmasked pixel locations. This significantly reduces redundancy within the training set and the associated computational load, both when ray tracing training images and when training NPLMs.

[0117] Exceptions can be applied at will. For example, pixels near the center of an image may always be rendered, as their central pixels are typically related to their position in the scene where they were selected (or not culled), depending on their surface properties, as described elsewhere in this specification. Pixels in areas other than the center of the image are typically more likely to capture unintended and unwanted repeating points in the scene.

[0118] Training under variable lighting conditions The various training techniques described herein typically involve using multiple different ray-traced images of a scene, object, or material (generally referred to as “scene” unless otherwise specified) for a given lighting condition. Typically, it is assumed that the light source is fixed within the scene (for example, by using a so-called “skydome” that provides a window position and / or static lighting map).

[0119] However, depending on the situation, the lighting in a scene can change significantly. For example, some games model day / night cycles and / or weather, and therefore the position of the sun (or moon) may change during the virtual day (or night) in the game. Similarly, depending on the weather, light sources can be point sources like the sun and diffuse sources like clouds shining from behind.

[0120] An NPLM trained to model the illuminance of a scene according to certain lighting conditions may not be able to generalize sufficiently to reproduce other conditions.

[0121] Therefore, in one embodiment of this specification, the NPLM is trained (using any of the techniques described herein) to model the illumination of a scene based on a ray-traced ground truth image using multiple lighting conditions.

[0122] Typically, these lighting conditions involve adjusting one or more lighting parameters within a predetermined range.

[0123] Lighting parameters may include the following: -Position: For example, the degree of the sun's movement within a given range. This can be achieved by rotating the skydome that makes up the sun in various ray-traced renderings. Other examples may include the movement paths of non-player characters or other movable light sources.

[0124] - Direction: Changing directional light sources can be modeled for the range of direction change, for example, in the case of lighthouses or MPC / moving light source motion paths. Alternatively or additionally, the beam width or beam spread angle of the light can be specified (for example, torches or spotlights typically have a narrower beam angle than desk lamps). Alternatively, these light sources or unpredictable light sources such as player-based moving light sources such as torches can be implemented by using full torchlights and non-torchlight NPLMs for different parts of the scene depending on where the torch light falls.

[0125] -Color or color temperature: For example, the sun gradually turns red during a sunset.

[0126] -Brightness. - Diffusion: For example, replacing a rendered skydome with a continuous amount of clouds or fog in front of a point light source such as the sun or moon, or building lighting.

[0127] A single NPLM can be trained at once for variations in one, two, or more such parameters. For example, an NPLM can be trained for sunrise, including a given range of location and associated changes in color temperature. Another NPLM can be trained for the progression of the sun during the day (or part thereof) within a given range of location, without changes in color temperature. Yet another NPLM may be trained for sunset, again including a given range of location and associated changes in color temperature.

[0128] As described elsewhere in this specification, more complex training environments may require modifications to the NPLM architecture to accommodate the network, typically when the NPLM memory footprint reaches a threshold size.

[0129] NPLM may also require one or more additional inputs during training that indicate parameter changes or changes associated with the training data.

[0130] For example, the position of a light source may be explicitly entered and / or a proxy value such as time of day (e.g., also correlated with color temperature). Typically, this could be an input to a position network that helps characterize the relative position of scene elements to light, an input to a direction network that helps characterize the relationship between the light source and the viewpoint, or an actual input to both. The position of a light source may relate to the position of a point light source, the center of a diffuse light source, the center of a skydome, or an offset position. The input can be linear, polar, or any suitable form (e.g., polar may be more suitable for a skydome). Thus, typical examples of positional changes are along the paths of NPCs (e.g., traffic), the in-game movement of objects (e.g., a lamp on a table moving), or changes in the environment (e.g., the sun or moon).

[0131] Similarly, direction (and / or optionally beam divergence angle) can be input to one or both halves of the NPLM. Typical examples of direction changes include spotlights or lookouts, or combinations with position changes and traffic. Divergence angle values ​​can be associated with these. On the other hand, an example of a change in divergence angle includes opening a door in front of the light source.

[0132] Color or color temperature can be input as, for example, a scalar value (e.g., for color temperature), RGB, or a similar color representation. Typically, the color representation will be in the same color space as the predicted image (i.e., the image pixels produced by combining the output of the NPLM with a particular BSDF, BRDF, BTDF, or similar). Color or color temperature can be input into the location network to again assist in the classification of scene locations embodied in the intermediate representation of the output layer of the location network, and / or into the direction network, as this can more readily influence the color-related output of the NPLM there. As mentioned elsewhere in this specification, typical examples of color changes can be related to the rising or setting sun, but even dramatic changes can be related, such as the activation of an alarm and a switch associated primarily with red lighting.

[0133] Brightness can be input into one or both halves of the NPLM as a color value, similar to color temperature. In this case as well, typical examples of brightness changes can be associated with sunset or sunrise.

[0134] Light source diffusion, like direction, can be input into a location network to aid in the classification of surface points and / or into a direction network to aid in the relationship between the light source and the viewpoint. Examples of diffuse light sources may include clouds or fog, or urban / office environments with diffused ceiling lights.

[0135] Furthermore, it will be understood that some or all of these parameters may correlate with time, so that alternative or additional time-based parameters may be used. This was mentioned above for position, but can also be applied to color, brightness, or any other parameter. For example, position, color, and brightness can all correlate with time when the scene is illuminated by sunlight.

[0136] It will be understood that adding one or more such inputs to an existing first layer of one of the location and / or direction networks will only allow for a single respective weight between the input and the node of the first layer in each network.

[0137] Therefore, referring here also to Figure 8B, additional layers may be provided between the current first layer and two or more inputs to each half of the NPLM, at the discretion of providing the capability for more complex learned responses to these additional inputs. Thus, additional layers may be provided not only to provide the position of light and the position of the current surface point as parallel inputs to the existing NPLM position network, but also so that any significant combination of these values ​​is learned as a preparatory step.

[0138] Therefore, if, for example, all of the above inputs are included in addition to the original surface point position input to the position network, a fully connected additional layer may be provided, or a partially connected layer may be provided having, for example, directional and diffuse inputs for a sublayer, brightness and color for a parallel sublayer, and optical position and surface point position for a parallel sublayer. These sublayers are then fully connected to the first entire layer of the network. A similar scheme can be used for a directional network.

[0139] Figure 8B shows, as a non-restrictive example, that a sublayer for the original scene position, a sublayer for the lighting position, and parallel sublayers for the color and brightness of the lighting are included in the position network.

[0140] A training set for variable illumination conditions may simply consist of repeating any of the techniques described herein for generating a training set for multiple illumination changes (e.g., for consecutive positions of light).

[0141] Alternatively, a single set of positions for full rendering can be obtained as described above, and then, for each position, a set of N versions is created by selecting either a random parameter value within a given range, or a set of parameter values ​​across that range, for each of the one or more parameters that can be varied. Optionally, if a probabilistic approach is used to extract candidate positions, the probability of retention can be increased (increasing the number of positions selected and enriching the training set for more challenging training conditions) or decreased (up to N times) to mitigate the associated computational load from rendering variations for the same view. Whether the probabilities are increased, decreased, or remain the same reflects a trade-off between computational load and time on the one hand, and the final NPLM output quality on the other, for that particular illuminated scene.

[0142] Alternatively, the probability of retention can be increased up to N times, resulting in a situation where, for each resulting viewpoint, only one (random or sequential) version of the parameter value is selected, and instead of having multiple parameter values ​​for one viewpoint, each parameter value exists for multiple viewpoints.

[0143] For example, generating training sets for multiple lighting changes can result in different configurations for each training set, bringing together sets of different parameter values ​​at the same location and a single parameter value at a specific location.

[0144] Typically, there is no need to change how the distribution of viewpoints is generated, as this depends on the viewer's position, not the light itself. However, for highly directional light, it may be useful at will to bias the probability of retaining a viewpoint depending on the angular distance of the viewpoint from the reflection angle relative to the light's directional axis (i.e., increasing retention as the viewpoint gets closer to the reflection line of the light).

[0145] As previously stated herein, NPLM can be implemented using any suitable machine learning system, and therefore the partitioned network architecture described herein is preferred, but in principle, an NPLM consisting of a single neural network (or actually three or more networks or other machine learning systems, of different types) may also be used, and one or more of these are supplied with the inputs described herein as appropriate.

[0146] Distributed approach Each NPLM generates a learned quality for a single pixel in response to any of the scene surface position and viewpoint / mirror direction inputs described elsewhere in this specification, as well as any additional inputs described earlier in this specification.

[0147] More typically, contributions from several NPLMs (e.g., trained on scene diffusion, specularity, translucency, and other components) are combined when the final output is required.

[0148] As a result, in principle, contributions from several NPLMs can be combined from NPLMs trained for their respective lighting conditions. For example, one NPLM may be trained for a sun or sky dome set at 1 PM, and another NPLM may be trained for a sun / sky dome at 2 PM. For example, the contributions can be combined so that at 1 PM, 100% of the 1 PM network is output; at 1:15 PM, 75% of the 1 PM network and 2% of the 2 PM network are output; at 1:30 PM, 50 / 50; at 1:45 PM, 25 / 75; and at 2 PM, 100% of the 2 PM network is output.

[0149] Regarding positional changes, the effect of illumination changes is more evident in the specular contribution, so the specular NPLM can be optionally trained at finer intervals than the diffuse NPLM. For example, the specular NPLM can be created for every 10% of points along each time interval or parameter range, while the diffuse NPLM can be created every 2-3 hours or for every 25% along the parameter range.

[0150] Therefore, contributions from different NPLMs trained at different points on the parameter range can be blended to approximate a desired point on the parameter range. Optionally, sampling these parameter ranges by each NPLM can be made more sparse for view-independent contributions such as diffuse contributions compared to specular contributions.

[0151] Furthermore, while the above example of time involves a one-dimensional parameter and therefore only requires blending two NPLMs trained with a parameter value that encompasses the desired parameter value, this approach can be extended to two-dimensional parameters using three or more NPLMs (e.g., triangles or other polygons in parameter space) whose weighted contributions correspond to the triangulation of the desired parameter value, or in fact, to three-dimensional parameters using four or more NPLMs (e.g., tetrahedrons or other volumes in parameter space) whose weighted contributions again correspond to the triangulation of the desired parameter value.

[0152] Network Configuration As described above, the positional network (i.e., the first part of the segmented architecture network described herein) can have a different number of outputs depending on whether it is trained on specular-type diffuse image components. This will be seen as a specific example of a more general approach.

[0153] Generally, the capabilities of an NPLM can be varied by increasing or decreasing its capabilities from the assumed default settings, depending on the complexity of the modeling tasks it is required to perform. In this process, the network architecture is typically modified to change the capabilities.

[0154] In the first embodiment, the capability can be varied based on the size of the NPLM (e.g., the number of layers, the size of the layers, and / or the distribution of layers among the parts of the NPLM, thereby changing the architecture of the NPLM to alter its capability).

[0155] Therefore, optionally, the size may vary depending on the type of contributing component that NPLM is modeling (e.g., diffusive, specular, or translucent / transparent).

[0156] In particular, the size of the position network can be beneficially larger for specular or translucent / transmissive components compared to diffuse components, all other than equal, due to the greater variability in the illumination response inherent in these components. For similar reasons, the size of the position network can be beneficially larger for translucent / transmissive components compared to specular components, all other than equal, because a combination of partial, transmitted, and internal reflections may be involved.

[0157] Similarly, the size of the position network can be beneficially larger if variations for one or more lighting conditions (different lighting positions, directions, and / or angular widths, for example, as an example, not limited to) are trained, due to an increase in the number of lighting conditions for a given scene surface position that needs to be modeled, or the complexity thereof.

[0158] The size can be varied by changing the number of hidden layers or the number of nodes in one or more such hidden layers. Similarly, the size may be varied depending on the number of output layers (e.g., the output layer of a positional network, which is also a hidden layer or interface / intermediate layer between the positional and directional networks of the overall NPLM network). Increasing the number of layers increases the spatial distortion that the network can apply to the input data to classify or filter different types of information. On the other hand, increasing the number of nodes in a layer increases the number of specific conditions in the training set that the network can model, resulting in improved fidelity. On the other hand, increasing the number of output nodes (unless selected to map to a specific format, such as the output of a positional network) can improve the identification by the output network (and subsequent networks operating on the output node values) by performing less strict dimensionality reduction on the internal representation of the dataset.

[0159] Alternatively or additionally, the size of the directional network can vary depending on the type of contributing component being modeled by NPLM (e.g., diffusive, specular, or translucent / transparent).

[0160] As described above, the input layer of the directional network can be resized to accommodate the higher-dimensional output of the position network within the overall NPLM partitioned architecture network.

[0161] Similarly, the number and / or size of the layers can be varied to have the same effects as outlined below for location networks, namely, to increase the discriminative ability and also the fidelity of the model.

[0162] Similar to positional networks, the size of directional networks can be beneficially larger for specular or translucent / transmissive components compared to diffuse components, all other than all else, due to the greater variability in the illumination response inherent to these components. For similar reasons, the size of directional networks can be beneficially larger for translucent / transmissive components compared to specular components, all other than all else, because a combination of partial, transmission, and internal reflection may be involved. Therefore, as with positional networks, their capabilities can be altered by changing their architecture.

[0163] Similarly, the size of the directional network can be beneficially large if variations for one or more lighting conditions (different lighting positions, directions, and / or angular widths, for example, as an example, different lighting positions, directions, and / or angular widths) are trained, due to an increasing number of lighting conditions for a given scene surface position that need to be modeled, or an increase in the complexity thereof.

[0164] Therefore, NPLMs (e.g., location networks, orientation networks, or both) can have their functionality modified to improve their identification capabilities (e.g., by more hidden layers or output dimensions) and / or improve the fidelity of the model (e.g., by more hidden layer nodes) depending on the requirements of the desired illumination model (e.g., changes to its / their architecture such as increasing the number of layers, internal nodes, or input or output dimensions). For example, the requirements for diffuse contributions are typically fewer than those for specular contributions.

[0165] Conversely, from the assumed standard or default settings of NPLM, it may be beneficially modified to decrease its capacity where appropriate (e.g., for diffusion components) instead of increasing it (e.g., by the reverse steps of those described above to increase capacity). In this case, the benefits are typically in the form of a reduced memory footprint and computational cost.

[0166] In addition to the types of material reflection properties (or properties) modeled by different contributing channels, the capabilities of NPLM may be increased or decreased in response to other factors related to the complexity of the lighting model / rendering process, either alternatively or additionally.

[0167] For example, diffuse light sources (such as sky domes) may be less complex than point light sources because they have less spatial / angular variation in the light hitting objects / scenes. Conversely, sky domes with large spatial variations (e.g., representing sunsets) may be more complex. The complexity of a light source may also be evaluated based on its spatial and color variations, for example, based on the integral of the 2D Fourier transform of the lighting space excluding objects / scenes, typically with the DC component discounted. In this case, a uniform sky dome will have an integral of nearly zero, one or more point light sources will have a larger integral, and complex sky domes (such as cityscapes or sunsets) may have an even larger integral. The capabilities (e.g., size) of an NPLM can be set based on an analysis of such light source complexity, for example, based on an empirical analysis of performance.

[0168] Similarly, moving, dynamic, or positionable light may require increased complexity in NPLMs because they create changing lighting conditions. In this case, inputs to the NPLM may include inputs of the lighting state and the (x,y,z) object position for a particular part of the object / scene being rendered for the output pixels. Thus, for a scene modeling the sun crossing the sky, inputs may include the time of day correlated with the sun's position. Other inputs to identify the current state of the light source may include the (x,y,z) position of one or more lights, an (r) radius or similar input for the size of the light, and / or an RGB input for the (dominant) color of the light, etc. It would be understood that the training data (e.g., based on ray-traced ground truth) would also incorporate examples of these changing conditions. More generally, if the NPLM is trained to model the dynamic aspects of an environment, the training data would include a suitable number of representative examples.

[0169] In the case of the sun, the trajectory of a full day may need to be modeled by several NPLMs consecutively to avoid the memory footprint or computational cost of the NPLMs exceeding a preferred maximum, as described elsewhere in this specification (e.g., modeling dawn, morning, midday, afternoon, and twilight).

[0170] Similarly, moving, dynamic, or positionable objects within a scene may require increased NPLM complexity if they are rendered using NPLM (optionally, NPLM can be used to render only static scene elements and / or contribute to parts of the scene that are independent of their position). Therefore, again in this case, the input may consist of, for example, object position and / or orientation data.

[0171] Alternatively or additionally, other factors may allow for the simplification of NPLM modeling and thus the reduction of NPLM capabilities (or, all other equals, the comparative improvement of model fidelity). For example, if the rendered scene includes fixed paths (e.g., on a race track, within collision avoidance barriers), training from viewpoints inaccessible to the user can be reduced or completely avoided. Similarly, if the rendered scene includes limited or preferred viewpoints (e.g., a race track where most viewpoints are in the direction of travel), training for different viewpoints can reflect the proportional importance of those viewpoints to the end use case.

[0172] Similarly, if parts of a scene are unlikely to be viewed critically by the user because they are in the background or far from the game's focus (from the perspective of foveal rendering or from the perspective of points of interest such as main characters), NPLM may perform relatively poorly. For example, different NPLMs can be trained for different drawing distances to objects or textures, resulting in reduced performance (e.g., size) at different drawing distances / levels of detail (LOD).

[0173] Alternatively or additionally, as described elsewhere in this specification, NPLMs can be trained for specific scenes, objects, materials, or textures. As a result, the capabilities of the NPLM can vary according to the complexity of what it represents. Larger or more complex scenes may require larger NPLMs (and / or multiple NPLMs, depending on the size of the scene and the resulting NPLMs, to handle their respective parts). Similarly, complex objects (such as a car) may benefit from a more powerful NPLM than simpler objects (such as a sphere). One way to assess the complexity of a scene or object is to count the number of polygons; the more polygons there are, the more complex the scene is inferred to be. Further refinement is possible, utilizing the dispersion of plane angles between polygons to infer complexity. For example, a sphere with the same number of polygons as the car model in the figure has a much smaller angular dispersion than the car itself, indicating that the car is structurally complex. Combining both polygon count and angular dispersion / distribution would provide a good substitute for the complexity of the scene / object whose illuminance is modeled by the NPLM.

[0174] Similarly, complex materials (such as skin or fur) can benefit from higher-performing NPLMs (and / or multiple NPLM contributors) than simpler materials (such as metals). Furthermore, complex textures (e.g., those with a broad spatial spectrum) can benefit from more capable NPLMs than textures with a narrow or condensed spatial spectrum.

[0175] While capability has so far been discussed in terms of size (number of inputs / outputs, number of layers, number of nodes, etc.), capability can also be altered by selecting the activation function between nodes in different layers of an NPLM, either in addition to or instead of this. As mentioned elsewhere in this specification, the preferred activation function for a location network is the ReLU function, while the preferred activation function for a direction network is the sine function; however, other functions can be selected to model other scenarios.

[0176] The capabilities of an NPLM can be subject to an upper limit, for example, when the NPLM's memory footprint reaches a threshold size. This threshold size may be equal to the size of a working unit of memory, such as a memory page or a partial or multiple group of memory pages, which is typically selected for accessing and loading textures for scenes / objects / materials. The threshold size may also be equal to the texture or mimap size used by the GPU and / or game to load graphic image data onto the GPU.

[0177] If the complexity of an NPLM exceeds this threshold, the task it models may need to be simplified, shared across NPLMs, or a lower accuracy of the results may have to be accepted.

[0178] Therefore, in summary, an image rendering method may include the step of selecting at least one first trained machine learning model from among several machine learning models (focusing on network configuration and selection). The machine learning model is trained to generate data that contributes to the rendering of at least a portion of the image, as discussed elsewhere herein. Thus, for example, the contributing data may relate to a specific component of an image pixel (e.g., diffuse or specular contributing component), or, depending on the complexity of the lighting and / or material, texture and / or other surface properties being modeled, it may relate to a complete RGB pixel (e.g., modeling all reflection modes at once).

[0179] In such a method, at least a first trained machine learning model has architecture-based learning capabilities that respond to at least a first aspect of the virtual environment in which it is trained to generate data, as discussed elsewhere herein. Thus, for example, the aspect of architecture relating to learning capabilities can be the size of all or part of the NPLM, such as the number of layers or nodes, and / or relating to the nature of the connections between nodes in different layers (e.g., in terms of the degree of connection of the type of activation function used).

[0180] In such a method, as described elsewhere in this specification, the second step may include generating data that contributes to rendering at least a portion of the image using at least one first trained machine learning model. Thus, for example, individual runs of an NPLM may generate data that is used together with data from other NPLMs to generate RGB values ​​for pixels in an image, or they may generate data on their own to generate RGB values ​​for pixels in an image after subsequent processing (e.g., coupling with a distribution function) as described elsewhere in this specification.

[0181] Network Selection The network is trained during the development phase of a game or application. Developers can choose when or where NPLM-based rendering is advantageous. For example, it might be used only in scenes where the frame rate consistently falls below a certain quality threshold. In such cases, the network is trained on those scenes or parts of them and used when those scenes are encountered.

[0182] In other cases, developers may choose to use NPLM-based rendering for specific objects or materials. In this case, the network is trained and used for that purpose when it is identified that those objects or materials are in the scene to be rendered.

[0183] In other cases, developers may choose to use NPLM-based rendering for specific lighting conditions or combinations thereof. In this case, the network is trained and used for that purpose when those conditions are identified as being within the scene to be rendered.

[0184] Similarly, developers may choose to use NPLM-based rendering for specific drawing distances (z-distance), angles / distances away from the image center or the user's fovea view, or specific lighting conditions. In this case, the network is learned and used to suit such situations.

[0185] Similarly, it will be understood that any suitable combination of these criteria may be selected for training and use.

[0186] On the other hand, as mentioned above, multiple NPLMs may exist in relation to a scene during system use for several reasons. For example, multiple NPLMs may exist to model a large scene (so that each part is modeled well enough by the NPLM to be within the size of the threshold and / or to the quality of image reproduction of the threshold). Similarly, multiple NPLMs may exist due to different lighting conditions, detail levels / drawing distances, etc.

[0187] Appropriate NPLMs for the situation may be selected and retrieved into the GPU's accessible working memory and used for or to render at least a portion of the image. It will also be understood that strategies applied to prefetch and cache textures and other graphics assets may also be applied to NPLMs.

[0188] summary Next, referring to Figure 9, in an embodiment of the abstract of this specification, an image rendering method for rendering pixels at a viewpoint comprises the following steps for a first element of a virtual scene having a predetermined surface at a location within that scene.

[0189] In the first step s910, a viewpoint-based position and orientation are provided to a machine learning system that has been pre-trained to predict a factor that generates a pixel value corresponding to a first element of a virtual scene to be illuminated at that position, when coupled with a distribution function that characterizes the interaction of light with a given surface, as described elsewhere in this specification.

[0190] In the second step s920, the predicted coefficients from the machine learning system are combined with the distribution function to generate pixel values ​​corresponding to the first elements of the position-illuminated virtual scene, as described elsewhere in this specification.

[0191] In the third step s930, pixel values ​​are incorporated into the rendered image for display, as described elsewhere in this specification. The image can then be output to a display via the A / V port (90).

[0192] It will be apparent to those skilled in the art that one or more variations of the above methods corresponding to the operation of various embodiments of the methods and / or apparatus described herein and claimed, including but not limited to the following, will be considered within the scope of this disclosure. -As described elsewhere in this specification, each machine learning system is trained for each of the multiple contributing elements of an image (e.g., diffuse, specular, coated, etc.), each distribution function is used for each of the multiple contributing elements of an image, and each generated pixel value is combined to produce pixel values ​​that are incorporated into the rendered image for display. As described elsewhere in this specification, each distribution function is one or more selected from a list including bidirectional scattering distribution functions, bidirectional reflectance distribution functions, and bidirectional transmittance distribution functions. -As described elsewhere in this specification, the machine learning system is a neural network, the input to the first part of the neural network includes position, and the input to the second part of the neural network includes the output and direction of the first part. -In this case, as described elsewhere in this specification, the activation function of the first part is different from the activation function of the second part. -In this case, as described elsewhere in this specification, the activation function for the first part is the ReLU function, and the activation function for the second part is the sine function. -In this example, as described elsewhere in this specification, the cost function of the neural network is based on the difference between the output of the second part and the values ​​obtained from the ray-traced version of the pixels for the training image, which has been inversely connected with the distribution function. -In this example, as described elsewhere in this specification, the cost function for the network is backpropagated through both the second and first parts during training. -In this example, the neural network is a fully connected network, as described elsewhere in this specification. As described elsewhere in this specification, the cost function of the machine learning system is based on the difference between the output of the machine learning system and the values ​​obtained from the ray-traced version of the pixels for the training image, on which the inverse connection with the distribution function is performed. - The machine learning system is selected and loaded into memory used by the graphics processing unit, based on the same asset identification scheme used to select and load textures for the first element of the scene.

[0193] Next, referring to Figure 10, in another abstract embodiment of this specification, an image rendering method for rendering pixels at a viewpoint (focusing on variations in lighting conditions) includes the following steps:

[0194] With respect to a first element of a virtual scene having a predetermined surface at a position within that scene, the first step s1010 includes providing a viewpoint-based position and orientation to a machine learning system that has been pre-trained to predict coefficients that generate pixel values ​​corresponding to the first element of the virtual scene so as to be illuminated at that position, when coupled with a distribution function that characterizes the interaction of light with the predetermined surface, as described elsewhere in this specification.

[0195] Next, the second step s1020 includes combining the predicted coefficients from the machine learning system with a distribution function, as described elsewhere in this specification, to generate pixel values ​​corresponding to the first elements of the illuminated virtual scene at the location.

[0196] The third step s1030 then includes incorporating the pixel values ​​into the rendered image for display.

[0197] However, it is noteworthy that the machine learning system in the first step was pre-trained using a training set based on images containing multiple lighting conditions, as described elsewhere in this specification.

[0198] Again, it will be apparent to those skilled in the art that one or more variations of the above methods corresponding to the operation of various embodiments of the methods and / or apparatus described and claimed herein, including but not limited to the following, are conceivable within the scope of this disclosure. As described elsewhere in this specification, multiple lighting conditions include one or more changes to the lighting position. -In this case, optionally, as described elsewhere in this specification, the change in lighting position corresponds to one or more selected from a list including assumed time in the virtual scene and the movement path of the movable virtual light source. As described elsewhere in this specification, multiple lighting conditions include one or more variations in the direction of illumination. -In this case, optionally, as described elsewhere in this specification, the change in illumination direction corresponds to one or more selected from a list including changes in the axial direction of the light source and changes in the angular spread of the light. As described elsewhere in this specification, the multiple lighting conditions include one or more modifications selected from a list including lighting color, lighting luminance, and lighting diffusion. As described elsewhere in this specification, each machine learning system is trained on each of the multiple contributing components of the image, each distribution function is used on each of the multiple contributing components of the image, and each generated pixel value is combined to create a final combined pixel value that is incorporated into the rendered image for display. -As described elsewhere in this specification, the machine learning system is a neural network, the input to the first part of the neural network includes position, and the input to the second part of the neural network includes the output and direction of the first part. -As described elsewhere in this specification, the machine learning system is a neural network, and the input to the neural network includes one or more selected from a list including lighting position or offset, assumed time in a virtual scene, axis direction of the light source, angle spread of light, color or color temperature, brightness, and diffusion. -In the two cases above where the machine learning system is a neural network, optionally, if the inputs to all or part of the neural network represent multiple properties of light, as described elsewhere in this specification, then at least a partially connected additional input layer is provided to the neural network. -As described elsewhere in this specification, the method includes the step of selecting at least a first trained machine learning model from among a plurality of machine learning models, the machine learning model being trained to generate data that contributes to rendering at least a portion of an image using a training set based on images including a plurality of lighting conditions, the at least first trained machine learning model having architecture-based learning capabilities that respond to at least a first aspect of a virtual environment trained to generate data, and using the at least first trained machine learning model to generate data that contributes to rendering at least a portion of an image. -As described elsewhere in this specification, the method includes the steps of using at least two respective machine learning models trained for different lighting conditions, generating pixel values ​​corresponding to a first element of a virtual scene to be illuminated at a surface location using each machine learning model, and combining the generated pixel values ​​with the output of the machine learning system. -As described elsewhere in this specification, the method includes the steps of generating a set of candidate viewpoints of a scene, culling the candidate viewpoints according to probabilities that depend on the response of the scene's surface to light at surface locations in the scene corresponding to the viewpoints, generating training images for the remaining viewpoints, and using a machine learning model trained with the training set generated by the above steps.

[0199] It will be understood that the above methods can be implemented in conventional hardware that is suitably adapted to be applicable by software instructions or by the inclusion or replacement of dedicated hardware.

[0200] Therefore, the necessary adaptation to existing parts of a conventional equivalent device may be implemented in the form of a computer program product containing processor-implementable instructions stored on a non-temporary machine-readable medium such as a floppy disk, optical disk, hard disk, solid-state disk, PROM, RAM, flash memory, or any combination thereof or other storage medium, or it may be implemented in hardware as a configurable circuit suitable for use in adapting a conventional equivalent device, such as an ASIC (Application-Specific Integrated Circuit) or FPGA (Field-Programmable Gate Array). Separately, such a computer program may be transmitted by data signals over a network such as Ethernet, a wireless network, the Internet, or any combination thereof or other networks.

[0201] Referring to Figure 1, the methods and techniques described herein can be implemented in conventional hardware such as an entertainment system 10 that generates images from a virtual scene. An example of such an entertainment system 10 is a computer or console such as Sony's PlayStation 5 (PS5).

[0202] The entertainment system 10 includes a central processing unit 20. This may be a single-core or multi-core processor with eight cores, such as in the PS5. The entertainment system also includes a graphical processing unit or GPU 30. The GPU may be physically separate from the CPU, or it may be integrated with the CPU as a System on a Chip (SoC), as in the PS5.

[0203] The entertainment device also includes RAM40, which may have separate RAM for the CPU and GPU respectively, or have shared RAM as in the PS5. Alternatively, each RAM may be physically separate, or integrated as part of the SoC as in the PS5. Further storage is provided by disk50, which may be an external or internal hard drive, or an external solid-state drive, or an internal solid-state drive as in the PS5.

[0204] The entertainment device may transmit or receive data via one or more data ports 60, such as a USB port, Ethernet® port, WiFi® port, or Bluetooth® port, as appropriate. Alternatively, it may receive data via an optical drive 70.

[0205] Interaction with the system is typically provided using one or more portable controllers 80, such as the DualSense® controller in the case of the PS5.

[0206] Audio / visual output from entertainment devices is typically provided through one or more A / V ports 90, or through one or more wired or wireless data ports 60.

[0207] If the components are not integrated, they can be connected as appropriate via a dedicated data link or bus 100.

[0208] Therefore, in the embodiments summarized herein, the entertainment device (such as Sony's PlayStation 5) is configured as follows:

[0209] Firstly, as described elsewhere in this specification, the graphics processing unit (such as the GPU 30, optionally working in conjunction with the CPU 20) is configured (e.g., by appropriate software instructions) to render pixels to a viewpoint in an image of a virtual scene that includes a first element having a predetermined surface at a location within the scene.

[0210] Secondly, as described elsewhere in this specification, a machine learning processor (optionally working in conjunction with the CPU 20, such as the GPU 30) is configured to provide viewpoint-based position and orientation to a machine learning system that has been pre-trained to predict coefficients that generate pixel values ​​corresponding to first elements of a virtual scene so as to be illuminated at that location, when coupled with a distribution function that characterizes the interaction of light with a given surface.

[0211] The graphics processing unit is configured (again, for example, by appropriate software instructions) to combine predicted coefficients from a machine learning system with a distribution function to generate pixel values ​​corresponding to the first illuminated element of the virtual scene at a given location, as described elsewhere in this specification.

[0212] Furthermore, the graphics processing unit is configured to incorporate pixel values ​​into a rendering image for display (again, for example, by appropriate software instructions), as described elsewhere in this specification.

[0213] The hardware described above may also be configured to perform the methods and techniques described herein, for example, as follows: As described elsewhere herein, the entertainment device comprises multiple machine learning processors (e.g., each processor of a GPU and / or CPU, seeds and / or shaders) running each machine learning system, each trained on one of several contributing components of an image (e.g., diffuse, specular, coat, etc.), each distribution function used for each of the several contributing components of an image, and a graphics processing unit configured (again, for example, by appropriate software instructions) to combine the respective generated pixel values ​​to create pixel values ​​incorporated into a rendered image for display. Each machine learning system is a neural network, the input to a first part of the neural network includes position, and the input to a second part of the neural network includes the output and direction of the first part.

[0214] Similarly, in another abstract embodiment of the present invention, an entertainment device (such as Sony's PlayStation 5) is configured as follows:

[0215] Firstly, as described elsewhere in this specification, a graphics processing unit (such as a GPU 30, optionally working in conjunction with the CPU 20) is configured (e.g., by appropriate software instructions) to render pixels to a viewpoint in an image of a virtual scene that includes a first element having a predetermined surface at a location within the scene.

[0216] Secondly, as described elsewhere in this specification, a machine learning processor (optionally working in conjunction with the CPU 20, such as the GPU 30) is configured to provide viewpoint-based position and orientation to a machine learning system that has been pre-trained to predict coefficients that generate pixel values ​​corresponding to a first element of a virtual scene so as to be illuminated at that location, when coupled with a distribution function that characterizes the interaction of light with a given surface.

[0217] The graphics processing unit is configured (again, for example, by appropriate software instructions) to combine predicted coefficients from a machine learning system with a distribution function to generate pixel values ​​corresponding to the first illuminated element of the virtual scene at a given location, as described elsewhere in this specification.

[0218] Furthermore, the graphics processing unit is configured to incorporate pixel values ​​into a rendered image for display (again, for example, by appropriate software instructions), as described elsewhere in this specification.

[0219] The machine learning system is pre-trained on a training set of images including multiple lighting conditions, as described elsewhere in this specification.

[0220] It will be understood that the above hardware may also be configured to implement the methods and techniques described herein, for example: -As described elsewhere in this specification, multiple lighting conditions include one or more changes in the lighting position. -As described elsewhere in this specification, multiple lighting conditions include one or more modifications to the direction of illumination. -As described elsewhere in this specification, the machine learning system is a neural network, the input to the first part of the neural network includes position, and the input to the second part of the neural network includes the output and direction of the first part. -As described elsewhere in this specification, if the inputs to all or part of a neural network represent two or more properties of light, at least a partially connected additional input layer is provided to the neural network. -As described elsewhere in this specification, the selection processor is configured to select at least a first trained machine learning model from among a plurality of machine learning models, the machine learning model is trained to generate data that contributes to rendering at least a portion of an image using a training set based on images including a plurality of lighting conditions, the at least first trained machine learning model has architecture-based learning capabilities that respond to at least a first aspect of a trained virtual environment for generating data, and uses the at least first trained machine learning model to generate data that contributes to rendering at least a portion of an image. -As described elsewhere in this specification, the combined processor is configured to use at least two separate machine learning models trained under different respective lighting conditions, to generate pixel values ​​corresponding to a first element of a virtual scene so as to be illuminated at a surface location using each machine learning model, and to combine the generated pixel values ​​with the output of the machine learning system. -As described elsewhere in this specification, a machine learning model is used that has been trained with a training set generated by the steps of generating multiple candidate viewpoints of a scene, culling the candidate viewpoints according to a probability that depends on the response of the scene's surface to light at the surface location in the scene corresponding to the viewpoint, and generating training images for the remaining viewpoints.

[0221] The preceding discussion discloses and describes only exemplary embodiments of the invention. As will be understood by those skilled in the art, the invention can be embodied in other specific forms without departing from its spirit or essential features. Accordingly, the disclosure of the invention, like the other claims, is illustrative but not intended to limit the scope of the invention. This disclosure, including any readily identifiable variations of the teachings herein, partially defines the scope of the terms of the preceding claims so as not to dedicate the subject matter of the invention to the public.

Claims

1. An image rendering method for rendering pixels at a viewpoint, Regarding the first element of a virtual scene having a predetermined surface at a position within that scene, The steps include providing a viewpoint-based position and orientation to a machine learning system that has been pre-trained to predict coefficients that generate corresponding pixel values ​​such that the first element of the virtual scene is illuminated at the position, when coupled with a distribution function that characterizes the interaction between the predetermined surface and light, The steps include: combining the coefficients predicted from the machine learning system with the distribution function to generate pixel values ​​corresponding to the illuminated first element of the virtual scene at the position; The steps include incorporating the aforementioned pixel values ​​into the rendered image for display, Includes, The aforementioned machine learning system is pre-trained using a training set based on images containing multiple lighting conditions. For each of the multiple contributing components of the aforementioned image, a separate machine learning system is trained. For each of the multiple contributing components of the aforementioned image, the respective distribution function is used. An image rendering method wherein the respective generated pixel values ​​are combined to create a final combined pixel value that is incorporated into the rendered image for display.

2. The image rendering method according to claim 1, wherein the plurality of lighting conditions include changes in one or more lighting positions.

3. Changes in lighting position The assumed time within the aforementioned virtual scene, The movement path of a movable virtual light source, The image rendering method according to claim 2, depending on one or more selected from a list including the following.

4. The image rendering method according to any one of claims 1 to 3, wherein the plurality of lighting conditions include one or more changes in lighting direction.

5. The aforementioned change in the direction of illumination is Changes in the axial direction of the light source, Changes in the spread of light angles, The image rendering method according to claim 4, depending on one or more selected from a list including the following.

6. The aforementioned multiple lighting conditions are, The color of the lighting, The brightness of the lighting, Light diffusion and, An image rendering method according to any one of claims 1 to 5, comprising one or more changes selected from a list including the following:

7. The aforementioned machine learning system is a neural network, The input to the first part of the neural network includes the position of the predetermined surface, The image rendering method according to any one of claims 1 to 6, wherein the input to the second part of the neural network includes the output of the first part and the direction.

8. The aforementioned machine learning system is a neural network, The input to the aforementioned neural network is: Lighting position or offset, The assumed time within the aforementioned virtual scene, The axis direction of the light source, The spread of light angles, Color or color temperature and, Brightness and, Spread and An image rendering method according to any one of claims 1 to 7, comprising data indicating one or more selected from a list including the following.

9. The image rendering method according to claim 7 or 8, wherein if the inputs to all or part of the neural network represent multiple properties of light, at least a partially connected additional input layer is provided to the neural network.

10. A step of selecting at least a first trained machine learning model from among multiple machine learning models, wherein the machine learning model is trained to generate data that contributes to rendering at least a portion of an image using a training set based on images including multiple lighting conditions, The at least first trained machine learning model has architecture-based learning capabilities that respond to at least first aspects of a virtual environment trained to generate data, The image rendering method according to any one of claims 1 to 9, wherein data contributing to the rendering of at least a portion of the image is generated using the at least first trained machine learning model.

11. The steps include using at least two separate machine learning models trained for different lighting conditions, The steps include generating pixel values ​​corresponding to the first element of the virtual scene, such as illuminated at the surface location, using each of the aforementioned machine learning models, The steps include combining the generated pixel values ​​with the output of the machine learning system, An image rendering method according to any one of claims 1 to 10, including the method described in any one of claims 1 to 10.

12. Steps to generate multiple candidate viewpoints for the scene, The steps of culling candidate viewpoints according to a probability that depends on the response of the surface of the scene to light at the surface position of the scene corresponding to the viewpoint, The step of generating training images from the remaining viewpoints, An image rendering method according to any one of claims 1 to 11, comprising the step of using a machine learning model trained with a training set generated by

13. A computer program comprising computer executable instructions adapted to cause a computer system to perform the method described in any one of claims 1 to 12.

14. A graphics processing unit configured to render pixels at a viewpoint within an image of a virtual scene that includes a first element having a predetermined surface at a location within that scene, A machine learning processor configured to provide a viewpoint-based position and orientation to a machine learning system that has been pre-trained to predict coefficients that generate pixel values ​​corresponding to the first element of the virtual scene illuminated at the position, when a distribution function characterizing the predetermined surface-light interaction is coupled, The graphics processing unit is configured to combine the predicted coefficients from the machine learning system with the distribution function to generate pixel values ​​corresponding to the illuminated first element of the virtual scene at the position. The graphics processing unit is configured to incorporate the pixel values ​​into the rendered image for display. The aforementioned machine learning system is pre-trained on a training set based on images including multiple lighting conditions. For each of the multiple contributing components of the aforementioned image, a separate machine learning system is trained. For each of the multiple contributing components of the aforementioned image, the respective distribution function is used. An entertainment device in which the respective generated pixel values ​​are combined to create a final combined pixel value that is incorporated into the rendered image for display.

Images