Image rendering method and apparatus

Abstract

To provide a rendering method for a virtual scene.SOLUTION: A method comprises the steps of: generating a mask identifying elements of scenes that are associated with a plurality of ID respectively; dividing, for the plurality of masks obtained as a result, each mask into a plurality of tiles; discarding tiles that do not identify any image elements; selecting, for the plurality of remaining tiles generated as a result, a trained machine learning model from among a plurality of machine learning models respectively; and generating, using each trained machine learning model, data contributing to a rendering of at least a part of the image, based upon at least input data of the identified elements in each tile.SELECTED DRAWING: Figure 10

Description

[Technical Field]

[0001] The present invention relates to an image rendering method and apparatus. [Background technology]

[0002] Ray tracing is a rendering process in which light paths are traced within a virtual scene. The interactions of each ray with objects or surfaces in the scene are then simulated. To achieve a degree of realism, this simulation typically takes into account the material properties of these objects or surfaces, such as color and reflectivity.

[0003] As a result, ray tracing is a computationally expensive process, and the cost varies from image frame to image frame depending on which scene is lit, with which lighting, and from which viewpoint.

[0004] This makes maintaining a preferred frame rate for rendering such images difficult to achieve; for an average computational cost corresponding to an average image completion time (i.e., frame rate) and a given variation around that average due to ray tracing, the average image quality must either be set low enough that the variation has little effect on the frame rate, or the average image quality must be set close to the maximum preferred frame rate, at the expense of frame rate consistency when the ray tracing demands vary more than average.

[0005] Neither outcome is desirable, but they cannot be easily avoided, given the data-driven and unpredictable nature of the computational load of ray tracing processing.

[0006] This invention aims to address or mitigate this problem. Summary of the Invention

[0007] Various aspects and features of the present invention are set forth in the appended claims and in the accompanying descriptive text, at least, -The image rendering method according to claim 1 in the first example, -An entertainment device according to claim 12 in the second example, Includes:

[0008] A more complete understanding of the present disclosure and many of the attendant advantages thereof will be readily obtained as the same becomes better understood by reference to the following detailed description when considered in connection with the accompanying drawings. [Brief explanation of the drawings]

[0009] [Figure 1] 1 is a schematic diagram of an entertainment device according to an embodiment of the present specification. [Figure 2] This figure illustrates a ray-traced object according to an embodiment of this specification. [Figure 3] This is a schematic diagram of contributing components to a ray-traced object according to the embodiments of this specification. [Figure 4] FIG. 1 is a schematic diagram of distribution functions associated with individual components according to embodiments herein. [Figure 5] This is a schematic diagram of the scattering distribution according to the embodiment of this specification. [Figure 6] This is a schematic diagram of a training scheme for a machine learning system according to an embodiment of this specification. [Figure 7] This is a schematic diagram of the rendering path for a rendered image according to the embodiment of this specification. [Figure 8] FIG. 1 is a schematic diagram of a machine learning system according to an embodiment of the present specification. [Figure 9] This is a flowchart of an image rendering method according to an embodiment of this specification. [Figure 10] FIG. 1 is a flow diagram of an image rendering method according to an embodiment of the present disclosure. [Figure 11] This is a schematic diagram of a method for generating a training set according to an embodiment of this specification. [Figure 12]This is a schematic diagram of a method for generating tiles according to the embodiments of this specification. DETAILED DESCRIPTION OF THE INVENTION

[0010] An image rendering method and apparatus is disclosed. In the following description, numerous specific details are set forth in order to provide a thorough understanding of embodiments of the present invention. However, it will be apparent to those skilled in the art that these specific details need not be employed to practice the present invention. Conversely, specific details known to those skilled in the art are omitted, where appropriate, for the sake of clarity.

[0011] Embodiments herein seek to address or mitigate the above problems by using a machine learning system that learns the relationship between pixel surface properties and rendered pixels for a given object or scene, and can then approximate a ray-traced rendering of the object or scene based on a relatively consistent computational budget (for running the machine learning system).

[0012] Different machine learning systems can be trained on different scenes, locations or portions thereof, or different objects or materials for use within one or more scenes, as described later in this specification.

[0013] Machine learning systems are relatively small (typically on the order of 100 KB to 1 MB) and can therefore be pulled into memory like a scene texture and then discarded for execution by the GPU (30). The system can be executed by a shader on the GPU. It will also be understood that in principle, the machine learning system could alternatively or additionally be executed by the CPU (20) or a general or specialized co-processor such as a neural network processor or ASIC.

[0014] Referring now to the drawings, wherein like reference numerals indicate identical or corresponding parts throughout the several views, FIGS. 2-7 illustrate problem spaces upon which machine learning systems are trained.

[0015] FIG. 2 is a high-quality ray-traced rendering 200 of an exemplary object or scene, in this case a car on a stage.

[0016] FIG. 3 illustrates the different contributing components behind this rendering. First, the diffuse lighting component 200-D typically captures the matte color of the surface and the shading caused by the interaction of light and shape. Second, the specular lighting component 200-S captures the reflectivity of the surface, resulting in sparkle and highlights. Optionally, one or more additional components can be included, such as a gloss or “coat” 200-C, which is a second exterior surface that includes additional shine or patterning. Variations of such coats may allow for partial transparency and / or partial diffusion, for example, in a manner similar to skin or cloth. Each of these components can be conventionally generated using individual ray tracing processes.

[0017] These components are added together to form the overall picture seen above in Figure 2. Typically two or three such components contribute to the rendering, although it will be appreciated that in some situations there may be fewer (e.g., if only a diffuse component is desired) or more (e.g., if the object is semi-transparent and therefore requires a transparent component).

[0018] Next, FIG. 4 includes the material properties of the objects that give rise to the above contributing components of the image.

[0019] Material properties are expressed as the so-called bidirectional scattering distribution function (BSDF) or bidirectional reflectance distribution function (BRDF).

[0020] BRDF defines how light reflects on an opaque surface, and similarly, BSDF defines the probability that a light ray will be reflected or scattered in a particular direction. Therefore, BRDF or BSDF are functions that describe the illumination properties of a surface (excluding the incident / exit radiance itself). Other functions may also be used as appropriate, such as the bidirectional transmittance distribution function (BTDF), which defines how light passes through a material.

[0021] Referring to Figure 5, a typical ray tracing application calculates the incident radiance (either directly or previously reflected) to a point on a model with a specific BSDF, BRDF, and / or BTDF for a set of rays (e.g., from a compact light source). The incident radiance is combined (e.g., multiplied) with the BSDF, BRDF, or BTDF with respect to the response of a particular contributing component, and the result is added to the pixel value of that point on the model. As shown in Figure 5, a typical scattering pattern of a ray path ωi in a BSDF is biased towards the specular reflection direction ωa, but can scatter in any direction. Accurately modeling this behavior is one reason why ray tracing is computationally expensive.

[0022] Using the color information of the model at each point and the corresponding BSDF, BRDF, and / or BTDF for that point (i.e., for a specific material represented by a given point), the behavior of light rays relative to a given final viewpoint can therefore be calculated by light reflectance or scattering, which determines, for example, the realistic distribution of glitter and highlights on the surface of a vehicle.

[0023] A separate BSDF, BRDF, or BTDF may be used for each contributing component. Thus, by way of non-limiting example, a BSDF may be used for the diffuse component, a BRDF for the specular component, and in this example also for the coat component (although a BTDF may also be used for such coat component). It will be understood that either a BSDF, BRDF, or BTDF may be used as appropriate, and thus, hereinafter, reference to a BSDF encompasses reference to the appropriate BRDF or BTDF unless otherwise specified.

[0024] As shown in Figure 4, performing ray tracing using the object's color properties and the BSDF's diffuse material properties (200-BSDF-D) results in diffuse image component 200-D. Similarly, using the BSDF's specular or reflective material properties (200-BSDF-S) results in specular image component 200-S. Similarly, using the BSDF's (200-BSDF-C) material properties, again typically specular, results in coated image component 200-C. Combining these components results in the final ray-traced image 200.

[0025] However, the problem, as mentioned earlier, is that calculating the reflection and scattering paths of rays intersecting different surfaces with different BSDFs, and summing the results for each pixel of the scene at a particular viewpoint, is computationally expensive and potentially highly variable.

[0026] Therefore, embodiments herein seek to replace the ray tracing step of FIG. 4 with something else that has a more predictable computational load for a final image of adequate quality.

[0027] Referring now also to FIG. 6, in embodiments herein, a respective machine learning system is provided for each contributing component of an image (e.g., diffuse, specular, and optionally coat or any other contributing component).

[0028] The machine learning system is typically a neural network trained to learn the transformation between BSDF (e.g., 200-BSDF-D) and ray-traced ground truth (e.g., 200-D) of the contributing components of an image for multiple images at different viewpoints in a scene, as will be discussed later herein.

[0029] In other words, if a ray-traced image (or one of its contributing components) is a combination of a BSDF that describes how lighting is reproduced on an object and how that object reacts to the light, then taking the ray-traced image and uncomboing it with the BSDF will result in a quality that could be called "radiance" or "shading," but more generally, it describes how light is reproduced on an object (which is collectively calculated by the ray tracing process).

[0030] If a machine learning system or neural network can learn to predict this quality, it can be recombined with a BSDF to generate a predicted image that approximates the ray-traced image. Therefore, this network is sometimes called a neural pre-computed light model or NPLM network.

[0031] More specifically, given an object's position on a virtual image and its viewing orientation, the machine learning system or neural network must learn to output a value that, when coupled with a BSDF for that same position / pixel, yields a pixel value similar to what would result from ray tracing of the image at that pixel. As a result, during training, it generates an internal representation of the lighting conditions (e.g., by a point light source or skydome) and surface illumination properties suggested by the training image.

[0032] Therefore, in an exemplary embodiment, the image is rasterized or otherwise generated at a given viewpoint, which will then fill the image with pixels to be illuminated. For each of these assumed pixels, there exists a corresponding 3D position in the scene where an appropriate "luminance" or "shading" can be obtained using an NPLM network.

[0033] FIG. 6 illustrates a learning environment for such a network, specifically, by way of example only, a network 600-D for diffusion contributing components.

[0034] The inputs to the network for the diffusion contributing components are the (x,y,z) position 610 on the object or scene (e.g., corresponding to a pixel in the image) and the object / scene normal 620 at that point. The normal N is used instead of the viewpoint direction because, for the diffusion contributing components, illuminance can be considered independent of direction / viewpoint, and therefore the known value of the normal can be used for consistency. Figure 6 conceptually illustrates these inputs using representative values ​​for the automobile image in the example described here.

[0035] Optionally, additional inputs may be provided (not shown), such as a roughness or matte-gloss scalar value that may be derived from an associated BSDF.

[0036] The output of the NPLM network (described later in this specification) is a learned quality or illuminance of light 630 for the input position, and when combined with the associated diffuse BSDF (200-BSDF-D) 640 for the same position, the predicted pixel value for the (x,y) position in the predicted image 650 is generated.

[0037] 6 illustrates that the pixel-by-pixel difference between the predicted pixel and the ground truth pixel of the ray-traced diffuse component 200-D of interest is used as the loss function to train the network. However, this is not necessary; rather, the ground truth image is decomposed (i.e., by performing an inverse function) with the BSDF to produce for each (x,y) pixel a proxy of how the ray-traced light cumulatively affects objects in the image, and this is the quality the network is trained to learn.

[0038] The error function of the network is therefore based on the difference between the output value of that single pixel (x,y) and the corresponding single pixel (x,y) of the ground truth image when not synthesized from the corresponding BSDF at that location.

[0039] Since the ground truth image pixels can be decoupled once from their corresponding BSDFs at each location, the network can learn without having to combine its output with any BSDFs to generate the actual predicted image pixels, reducing the computational burden during training.

[0040] As described above, the learned quality output by the trained neural network captures how light in the environment plays out on an object or scene depending on the position of the surface in the scene and depending on the viewpoint. In this way, it effectively generates an internal representation of a light map and surface response model for the scene. How this occurs is discussed in more detail later in this specification.

[0041] Referring to Figure 7, to summarize for each contributing element of the final output image, the machine learning system is trained to perform a transformation applied to the BSDF, which is locally applied to the location on the object / scene for that contributing element. This transformation is a trained function based on the (x,y,z) position and orientation values ​​of the point on the object / scene. As mentioned above, one, two, three, four, or more machine learning systems may be employed, depending on the number of contributing elements of the final image. Hereafter, the term "trained function" may be used to refer to a machine learning system that has learned such a transformation.

[0042] As noted for the diffuse component, the diffuse shading is assumed to be directionally invariant, so the direction value can be assumed to be equal to the normal at a given point.

[0043] On the other hand, for specular elements that will change depending on the viewpoint because they are at least partially reflective, the directional value is the viewing angle between the current (x,y) position of the pixel at the viewpoint of the image (which has a position in virtual space) and the (x,y,z) position of the object as input to the machine learning system, or based on this, providing a viewpoint dependency between the input point on the scene surface and the current pixel from which the learned quantity will be output.

[0044] In this case, the court components are also specular, so a similar viewpoint or a viewpoint based on the direction relative to the input is used.

[0045] Therefore, the direction value of the direction-dependent component may be the viewpoint direction between the output pixel position and the object surface position, or a value based thereon, such as the surface specular viewpoint direction (i.e., considering the normal of the surface at the input position, the viewpoint direction is the principal direction of reflection). Any appropriate direction value incorporating viewpoint direction information can be considered.

[0046] In each case, the trained function encapsulates the learned quality as previously described herein. By combining the appropriate BSDF with the network output at each location, a shaded image of each component can be constructed. Alternatively or additionally, the final output is generated by combining the pixel values ​​of the shaded images from each component.

[0047] It should be understood that not all parts of an image are subject to ray tracing during rendering, nor are all parts of an image generated using the techniques described above. For example, an NPLM network may be trained on a specific object or material based on a ray-traced image of ground truth with representative lighting.

[0048] When these objects or materials are subsequently rendered in real time using apparent ray tracing provided by the trained functions described herein, the relevant NPLM network, when loaded into memory and coupled with the appropriate BSDF, is run on the relevant surface positions and line-of-sight directions in the scene to generate their contributions to the relevant pixels. Other pixels may be rendered using any other suitable technique, including ray tracing itself.

[0049] Typically, an appropriate machine learning system is selected and loaded into memory used by the GPU based on the same asset identification scheme used to select and load the texture of an object or material. Thus, for example, if an object has ID "1234" used to access its associated texture, this ID can also be associated with the associated machine learning system. Conversely, if a texture has ID "5678" associated with an object (e.g., if the texture represents a material common to multiple objects), this ID can also be associated with the associated machine learning system. In this manner, an entertainment device can load a machine learning system using a process similar to loading a texture. It will be appreciated that if the texture is stored with lossy compression that affects the operation of the decompressed machine learning system, the actual storage and access techniques may differ between the texture and the machine learning system. Thus, the machine learning system may be stored without compression, using lossless compression, or using lossy compression where the degree of loss is low enough that the decompressed machine learning system still operates properly. This can be evaluated by comparing the output error / cost function of the machine learning system for increasing degrees of compression loss until the error reaches an absolute or relative (relative to the uncompressed machine learning system) quality threshold.

[0050] 8, in embodiments herein, the machine learning system or NPLM network may be any suitable machine learning system. Thus, for example, a single neural network may be trained using position and viewing direction as inputs to generate RGB values ​​for the learned properties as outputs.

[0051] However, a particularly advantageous network would consist of a well-defined partitioned architecture.

[0052] In a non-limiting example, the network is composed of two parts, as shown in Figure 8. The first part can be thought of as a location network, while the second part can be thought of as a direction network.

[0053] Each of these networks may have three or more layers and may use any suitable activation function.

[0054] The position network takes as input the previously described (x,y,z) positions for points in an object / scene and outputs an intermediate representation, which is described later in this document.

[0055] The direction network receives this intermediate representation and direction inputs (e.g., normals, or pixel viewpoints, surface point mirrored pixel viewpoint directions, or other viewpoint-based direction values), e.g., in (θ, φ) format or as normalized (x, y, z) vectors. It outputs RGB values ​​corresponding to the learned quantities mentioned above for the current pixel's (x, y) position (and thus pixel viewpoint) in the image rendered from a virtual camera position in space shared with the object / scene.

[0056] Thus, in a non-limiting example, the location network has three layers, with three input nodes in the first layer (e.g., for x, y, z location), 128 hidden nodes in the middle layer, and 8 outputs in the final layer.

[0057] While any suitable activation function can be chosen for the network, the rectified linear unit (ReLU) function has been found to be particularly effective as an activation function between layers of the location network, as it generalizes well to untrained locations and helps avoid overfitting.

[0058] Also in a non-limiting example, the direction network has four layers, with the 8 outputs of the position network and 2 or 3 additional values ​​for direction being fed into 128 nodes in the first layer, then into two further layers of 128 nodes, resulting in 3 outputs in the final layer corresponding to the R, G, B values ​​for the learned quantities at the current pixel. This can then be combined (e.g., multiplied) with the BSDF for that position to obtain the final pixel contribution from this trained function (e.g., diffuse, specular, etc.), although as mentioned above this is not necessary during training.

[0059] While any suitable activation function can be chosen for the orientation network, a sine function has been shown to be a particularly effective activation function between layers of the orientation network. The sine activation function has been found to be particularly good because the light behavior variation in the angular domain is large and contains details at many angular frequencies, but is based on low-dimensional inputs (e.g., normalized x, y, z vectors).

[0060] It is worth noting that the two halves of the network may therefore use different activation functions.

[0061] However, the network is treated as a split-architecture network rather than as two separate networks, especially since the learning scheme only has one cost function: the error between the RGB values ​​output by the direction network and the target values ​​from the corresponding pixels in the ground truth ray-traced image after not combining with the appropriate BSDF.

[0062] This error is backpropagated through both networks, and there is no separate target value or cost function for the position network. Thus, the output layer of the position network is actually a hidden layer of a combined network augmented with an additional input of directional values, representing the transition from one activation function to a possible second, different activation function within the layer.

[0063] As previously mentioned, the neural network builds a lighting model for the illuminated object, material, or scene. In particular, in the non-limiting example above, the position network effectively sorts (x, y, z) locations into lighting types (e.g., bright or dark, and / or other categories related to how light interacts with the respective BSDF, such as relative reflectance or diffuseness). The intermediate representation output by this portion can be thought of as an N-dimensional position in lighting space that characterizes the type of light at the input location, and will project locations in different parts of the scene to the same N-dimensional position if they were illuminated in the same way. A position network trained for the specular component may have more outputs than one for the diffuse component, e.g., 32 outputs versus 8, to account for the greater variation in lighting types that can occur in the specular component.

[0064] The directional network then models how the light behaves for an illuminated object, material, or scene when viewed on a surface at an input location at an input angle, generating learned properties for that location in the image.

[0065] Thus, in summary, the position and orientation networks are trained together to predict the coefficients or transformation between a BSDF describing surface properties and a desired rendered image of that surface. This network can then be used in place of ray tracing for rendering that surface. Typically, but not necessarily, a network is trained with only one contributing component of the image, such as the diffuse specular component, and multiple networks are used to generate the components needed for the final image or image portion, although this is not required (i.e., in principle, a network could be trained with a fully combined image or a combination of two or more contributing components, e.g., all specular or all diffuse contributions).

[0066] training The network is trained using multiple ray-traced images of an object, scene, or surface taken from multiple different viewpoints, as described elsewhere in this specification. This allows the network to learn in particular how specular reflections change with position. The viewpoints may be randomly distributed and / or may be selected (or preferentially selected) from a range of viewpoints available to the user when navigating the rendering environment, known, for example, as the volume of views, i.e., the volume of space in which viewpoints can arise, and therefore should be included in the training.

[0067] In one embodiment herein, the training data may be generated as follows.

[0068] It will be appreciated that for any machine learning system, the training data used to train the system can be key to its performance. As a result, it is highly beneficial to generate training data that leads to good performance.

[0069] As described elsewhere in this specification, the training data for the NPLM system described herein is based on a set of high-quality rendered images of scenes / objects / materials / surfaces (hereinafter generally referred to as scenes) and is typically decoupled with one or more related distribution functions (e.g., BSDF, BRDF, or similar ones as described elsewhere in this specification). As a result, the learned quality referred to herein is provided as a direct training target, eliminating the computational burden of generating predictive images during training and ensuring that the error function is not derived solely from the output of the NPLM itself.

[0070] Different NPLMs can handle view-dependent and view-independent shading effects (e.g., diffuse, specular, etc.). Therefore, typically, if an object has view-dependent shading (e.g., specularity, or mirror reflection, etc.), a single view of the object in the scene is insufficient.

[0071] As a result, the number and positions of training data images may depend not only on the geometry of the scene (eg, if objects are visible within the volume of view), but also potentially on the material properties of the objects in the scene.

[0072] Thus, in embodiments herein, NPLM training data in the form of images of a scene taken from multiple camera viewpoints can be generated based at least in part on materials in the scene (e.g., material properties such as light response properties like diffuse or specular response, but potentially other material properties such as surface complexity - e.g., the presence of narrow or wide spatial frequency components structurally and / or texturewise).

[0073] These images are typically generated from third-party high-quality rendering units, and access to their internal data is not available, so only the final, complete image is available, and may not have any information about (or control over) the specific cast rays or their direction when performing shading within the image.

[0074] Therefore, for training purposes, it is desirable to generate and use a set of images that efficiently capture the appearance of the scene, preferably for all valid views within the volume of views.

[0075] Referring here to Figure 11, for this purpose, in step 1110, first, a set of camera positions within the view volume is used to render a set of low-resolution images. The positions may be equidistant or randomly distributed on a sphere around the scene (if it can be viewed from any angle, such as an interactive object), on a hemisphere around the scene (if it cannot be viewed from below because it is based on a virtual ground), or on a ring around the scene (if it can be viewed from a ground-based viewpoint, such as the first-person view of an avatar). Such a ring may be at a fixed height corresponding to the avatar viewpoint, or it may occupy a height range such as a viewing cylinder that encompasses one or more of the crouch height and jump height of the avatar viewpoint.

[0076] Step 1110 shows the orbit (ring) of the camera position around the car object illustrated in FIG.

[0077] The number of camera positions in this initial set may be as few as one, but is typically three or more, and more typically on the order of tens or hundreds. For example, if there is one camera per degree of orbit, that would be 360 ​​cameras. In this embodiment, an indefinite number of 200 cameras is used.

[0078] The resolution per image is low, for example 128 x 84 pixels. An example image at step s1120 is shown below.

[0079] Notably, for each pixel in each image, in step s1130, metadata is associated with it including the 3D position of the scene surface corresponding to the pixel, the normal of the scene surface corresponding to the pixel, and optionally a material surface identifier or descriptor such as a texture ID or object ID.

[0080] In a first example of a viewpoint selection process, the 3D locations of the scene surface rendered by pixels in some, or typically all, of these low-resolution images are matched to identify which locations in the scene are visible within a first set of camera positions. These are the 3D locations that the NPLM will benefit from being trained on.

[0081] Therefore, optionally, for each 3D location identified as being rendered in at least one of the initial low-resolution images, a new location in 3D space is calculated as an offset from that location along the surface normal. The distance of the offset from the surface is a variable that can be changed. This new location is a candidate viewpoint for a virtual camera for generating high-quality (e.g., high-resolution ray tracing) rendering.

[0082] However, this results in a large number of potential high-quality ray-traced renderings to generate as training images, which can be computationally burdensome and may contain significant redundancy when used as a training set for NPLM.

[0083] Therefore, in the first instance, it is desirable to filter or cull these candidate viewpoint positions in some way that is relevant and useful for training the NPLM on the scene.

[0084] In particular, it is beneficial to have more training examples for parts of a scene that contain view-dependent materials (eg, specular or glossy) than for view-independent materials (eg, diffuse or matte).

[0085] Therefore, one of two approaches can be taken.

[0086] In the first approach, in step 1140, for each candidate viewpoint corresponding to the normal at the surface location, the corresponding material properties of the surface at that location are examined. As mentioned above, in particular, its diffusion response, specular response, or translucency may be used.

[0087] In practice, this can be done by using a lookup table that associates material IDs, etc., with values ​​indicating the diffusivity or specularity (e.g., matte or glossy) of the material surface. More specifically, this property can be represented, as an unrestricted example, by a value ranging from 0 for perfectly diffuse to 1 for specular reflection. This can be treated as input to a probability function, so that specular or glossy (view-dependent) materials have a relatively high probability, and diffusive or matte (view-independent) materials have a relatively low probability.

[0088] This probability function is used to retain candidate camera positions. Therefore, camera positions facing the mirrored surface will be retained at a higher rate than those facing the diffuse surface.

[0089] Conversely, if the value rules were reversed (eg, low probability versus high probability), the probability function could be used to eliminate candidate camera positions with the same effect.

[0090] In the second approach, alternatively or additionally, in step s1140, the variation in pixel values ​​corresponding to the same 3D position on the scene surface as seen in the low-resolution image may be evaluated to determine the variance of the pixel values ​​for each captured 3D position. In this method, view-invariant (e.g., diffuse or heavily shaded) surface positions have low variance (i.e., pixels indicating that position are similar in different low-resolution images), while view-dependent (e.g., specular or glossy) surface positions have high variance (i.e., pixels indicating that position will show a wider range of values ​​in different low-resolution images because they capture some glare or reflection, for example). This variance, or its normalized version, can again be used as input to a probability function so that specular or glossy (view-dependent) materials have relatively high probabilities, and diffuse or matte (view-independent) materials have relatively low probabilities.

[0091] Therefore, in either case, step s1140 provides an estimate of the view dependence of the material's photoresponsiveness at each captured 3D location within the view volume (based on material properties, pixel variations, or potentially both), which can be used as input to a stochastic function.

[0092] The probability function is then used in step s1150 to determine whether each candidate viewpoint is retained or culled, with viewpoints centered on view-dependent surfaces being retained more often than viewpoints centered on view-independent surfaces.

[0093] The output range of this probability function can be adjusted to approximately generate the desired total number of camera views for training, based on the initial number of possible candidates and the final desired number; or, the probability function can be applied to a series of retain / culling rounds until the number of remaining camera views falls within the desired threshold.

[0094] In either case, the result is a manageable number of camera views randomly distributed across the desired volume of view, but with a variable probability density that responds to material properties (e.g., gloss or other) of the material that immediately concentrate in front of the camera. This is illustrated by the constellation of surviving points shown in the figure of step s1150. In practice, the camera position can be further from the object / scene surface than shown in this figure, but the points are placed close to the surface in the figure to illustrate their distribution.

[0095] The manageable number of camera viewpoints can be selected based on factors such as the desired performance of the resulting NPLM, the computational load of generating high-quality ray-traced images and training the NPLM on them, and memory or storage constraints. A typical manageable number for training purposes may be 10 to 10,000 in non-limiting examples, and a typical number is 200 to 2,000.

[0096] Finally, in step s1160, the images are rendered from the perspective of the surviving player. Optionally, as shown in Figure 11, these renderings are generated using a virtual lens with a wider angle than the lens used for the initial low-resolution images or the lens used during gameplay.

[0097] This tends to result in rendering too much of the scene (i.e., parts not directly visible from the viewpoint of the view volume). This tends to make the NPLM output more robust, especially for view positions near the edges of the view volume, in cases of unexpected expansion of the view volume due to, for example, clipping of objects in the game or minor design modifications.

[0098] The approach described above generated candidate camera viewpoints based on the normals of the scene surface captured in the initial low-resolution image, but this is not the only possible approach.

[0099] One possible problem with the above approach is that while a view-invariant position in the scene may be captured by a camera oriented in that direction along the normal at that position, it may only be rendered from different angles in other images of nearby positions, and these angles are now determined by the normals of the scene surface at those positions. As a result, there will be a relatively large number of images of the view-dependent parts of the scene or their vicinity, while the images themselves may be excessively influenced by the geometry of the scene itself.

[0100] Therefore, returning to the initial low-resolution images, in another example of the viewpoint selection process, potential viewpoint positions can be considered for each pixel (or at least for these pixels representing the surface of the scene) in each low-resolution image. In the example of 200 images of 128x84 pixels above, there would be up to 1.6 million candidates. These images typically capture multiple instances of a particular location on the scene from different angles, independent of the topology of the scene itself. As a result, the training set is potentially more robust.

[0101] Again, the probability of retaining or culling a viewpoint can be manipulated using the surface material (and / or pixel variance) derived from the view dependency of a given pixel in the low-resolution image, and therefore the surface position corresponding to the candidate viewpoint. In this way, the 1.6 million candidate viewpoints can be narrowed down again to a manageable number.

[0102] In this case, since there may be multiple viewpoints at the same location in the scene, the resulting distribution of camera views may be biased towards the most visible location in the scene rather than depending on most views. For example, if a certain (diffuse) location in the scene is visible in more than 20 times the image of a certain (spectral) location, viewpoints viewing the diffuse location are more likely to be culled, but because the difference is more than 20 times, the images of the diffuse location may ultimately outnumber those of the reflective location.

[0103] Therefore, optionally, the probability of keeping or culling a viewpoint can be normalized based on the number of viewpoints centered at the same location in the scene (albeit from different angles). This normalization may be complete (so in the example above, the probability of keeping an image at a diffuse position is reduced by a factor of 20, thereby eliminating the effect of the number of viewpoints). Alternatively, the normalization may be partial; for example, the probability of keeping an image at a diffuse position is reduced by a factor of 10, thereby significantly reducing but not completely eliminating the effect of the number of views. This means that regions that are likely to be seen by users will get more training examples, regardless of whether they get many training examples because they are view-dependent (e.g., specular / glossy).

[0104] In principle, both sets of viewpoints (surface normal-based viewpoints and low-resolution image pixel-based viewpoints) can be generated and culled to create a combined viewpoint set before generating high-quality ray-traced renderings for training purposes. In practice, in either case, there is likely to be a substantial subset of low-resolution image pixel-based viewpoints that match the normals of at least some of the visible surface locations.

[0105] Distributed training technology The second approach above optionally considers the problem of compensating for multiple views of the same location in the scene when culling available viewpoints. In addition to allowing control of training bias, this second approach reduces training time by reducing repetitions for specific locations in the scene.

[0106] However, alternatively or additionally, training time can be (further) reduced as follows.

[0107] As mentioned above, an initial set of viewpoints within (or on) the volume of view is selected.

[0108] Now, optionally, an initial low-resolution image is generated for a set of locations within the volume of view.

[0109] Now, optionally, candidate viewpoints are then generated based on normals to positions in the scene found in the low-resolution image and / or based on lines between pixels of the low-resolution image and represented positions in the scene, as previously described herein.

[0110] Again, optionally, these viewpoints can be culled with a probability based on the degree of specularity / diffuseness of each position in the scene. Further optionally, if there are multiple images centered at each position, the probability can be modified to at least partially take this into account.

[0111] Thus, depending on the approach taken, the result is a generated set of viewpoints, either the original distribution optionally used to generate the low-resolution image, or a distribution resulting from one of the generation and culling techniques described above.

[0112] In any case, in the embodiments herein, once a viewpoint is generated (and optionally, where appropriate, confirmed not to be culled), it is provided for or queued for ray tracing processing to generate high-quality images, optionally in the form of a wide angle, as described elsewhere herein.

[0113] Training on the generated images begins when each image is completed. Therefore, the generation of training images (which takes some time because they are ray-traced images) and the training on those images (which can also take some time) occur in parallel. This avoids the problem of having to wait for a complete training set to be generated before starting training.

[0114] If, by choice, viewpoints are generated, or if generated viewpoints are selected to determine whether they are culled, the selection of viewpoints from these available ones can be random, and consequently, the final generation sequence of ray-traced images will also be random within the final set of viewpoints used.

[0115] This reduces the chance that the NPLM will be initially overtrained on one section of the scene, and also means that if training must be scaled back due to time constraints, for example, the NPLM will still be exposed to a diverse set of views of the scene.

[0116] In another distributed training technique, if control over the ray tracing application is available and enabled, it is optional to render only a subset of pixels from a given viewpoint. Within a given image, there may be parts of the scene that have been rendered multiple times in other images within the training set, whether based on the original set of viewpoints or on viewpoints that were not culled. For example, if a location in a scene has already been rendered more than a threshold number of times, it may be skipped in the current rendering because there are already enough training examples for that location. Unrendered portions of an image can be tagged with pixel values ​​that act as mask values. As a result, training can be performed using only the input location, orientation information, and target values ​​for unmasked pixel locations. This significantly reduces redundancy within the training set and the associated computational load, both when ray tracing training images and when training NPLMs.

[0117] Exceptions can be applied at will. For example, pixels near the center of an image may always be rendered, as their central pixels are typically related to their position in the scene where they were selected (or not culled), depending on their surface properties, as described elsewhere in this specification. Pixels in areas other than the center of the image are typically more likely to capture unintended and unwanted repeating points in the scene.

[0118] Network configuration As noted above, the location network (i.e., the first part of the split-architecture network described herein) may have a different number of outputs depending on whether it is trained for specular-type image component diffusion. It will be understood that this is a specific example of a more general approach.

[0119] Generally, the capabilities of NPLM can be varied by increasing or decreasing the capabilities from the assumed default settings, depending on the complexity of the modeling task required to be performed. In this process, the network architecture is typically modified to change the capabilities.

[0120] In the first embodiment, the capability can be varied based on the size of the NPLM (e.g., the number of layers, the size of the layers, and / or the distribution of layers among the parts of the NPLM, thereby changing the architecture of the NPLM to alter its capability).

[0121] Therefore, optionally, the size may vary depending on the type of contributing component that NPLM is modeling (e.g., diffusive, specular, or translucent / transparent).

[0122] In particular, the size of the location network may be beneficially increased for specular or translucent / transparent components compared to diffuse components, all else being equal, due to the greater variability in lighting response inherent in these components. For similar reasons, the size of the location network may be beneficially increased for translucent / transparent components compared to specular components, all else being equal, due to the combination of partial reflection, transmission, and internal reflection that may be involved.

[0123] The size can be varied by changing the number of hidden layers or the number of nodes in one or more such hidden layers. Similarly, the size may be varied depending on the number of output layers (e.g., the output layer of a location network, which is also a hidden layer or interface / intermediate layer between the location network and the direction network of the entire NPLM network). Increasing the number of layers increases the spatial distortion that the network can apply to the input data to classify or filter different types of information. On the other hand, increasing the number of nodes in a layer increases the number of specific conditions in the training set that the network can model, resulting in improved fidelity. On the other hand, increasing the number of output nodes (unless selected to map to a specific format, as in the output of a location network) can improve the identification by the output network (and subsequent networks operating on the output node values) by performing less strict dimensionality reduction on the internal representation of the dataset.

[0124] Alternatively or additionally, the size of the direction network can be varied depending on the type of contributing component (e.g., diffuse, specular, or translucent / transparent) that the NPLM is modeling.

[0125] As mentioned above, the input layer of the direction network can be resized to correspond to the higher dimensional output of the position network within the overall NPLM partitioned architecture network.

[0126] Similarly, the number of layers and / or layer size can be varied to achieve similar effects as outlined below for location networks, ie, increasing discriminatory power and also model fidelity.

[0127] As with the location network, the size of the direction network may be beneficially increased for specular or translucent / transparent components compared to diffuse components, all else being equal, due to the greater variability in lighting response inherent in these components. For similar reasons, the size of the direction network may be beneficially increased for translucent / transparent components compared to specular components, all else being equal, due to the combination of partial reflection, transmission, and internal reflection that may be involved. Thus, as with the location network, its architecture can be modified to alter its capabilities.

[0128] Thus, an NPLM (e.g., a position network, an orientation network, or both) may have its capabilities modified (e.g., by changing its architecture, such as by increasing the number of layers, the number of internal nodes, the input or output dimensions, etc.) to improve its discriminatory ability (e.g., by increasing the number of hidden layers or output dimensionality) and / or to improve the fidelity of the model (e.g., by increasing the number of nodes in the hidden layers), depending on the requirements of the required illumination model. For example, the requirements for a diffuse contributing component are typically less than the requirements for a specular contributing component.

[0129] Conversely, from the assumed standard or default setup of the NPLM, the NPLM may be beneficially modified to decrease its capacity where appropriate (e.g., for the diffusion component) instead of increasing it (e.g., by steps opposite to those described above for increasing capacity), in which case the benefit is typically in terms of reduced memory footprint and computational cost.

[0130] In addition to the types of material reflection properties (or properties) modeled by different contributing channels, the capabilities of NPLM may be increased or decreased in response to other factors related to the complexity of the lighting model / rendering process, either alternatively or additionally.

[0131] For example, diffuse light sources (such as sky domes) may be less complex than point light sources because they have less spatial / angular variation in the light hitting objects / scenes. Conversely, sky domes with large spatial variations (e.g., representing sunsets) may be more complex. The complexity of a light source may also be evaluated based on its spatial and color variations, for example, based on the integral of the 2D Fourier transform of the lighting space excluding objects / scenes, typically with the DC component discounted. In this case, a uniform sky dome will have an integral of nearly zero, one or more point light sources will have a larger integral, and complex sky domes (such as cityscapes or sunsets) may have an even larger integral. The capabilities (e.g., size) of an NPLM can be set based on an analysis of such light source complexity, for example, based on an empirical analysis of performance.

[0132] Similarly, moving, dynamic, or positionable light may require increased complexity in NPLMs because they create changing lighting conditions. In this case, inputs to the NPLM may include inputs of the lighting state and the (x,y,z) object position for a particular part of the object / scene being rendered for the output pixels. Thus, for a scene modeling the sun crossing the sky, inputs may include the time of day correlated with the sun's position. Other inputs to identify the current state of the light source may include the (x,y,z) position of one or more lights, an (r) radius or similar input for the size of the light, and / or an RGB input for the (dominant) color of the light, etc. It would be understood that the training data (e.g., based on ray-traced ground truth) would also incorporate examples of these changing conditions. More generally, if the NPLM is trained to model the dynamic aspects of an environment, the training data would include a suitable number of representative examples.

[0133] In the case of the sun, the trajectory of a full day may need to be modeled by several NPLMs consecutively to avoid the memory footprint or computational cost of the NPLMs exceeding a preferred maximum, as described elsewhere in this specification (e.g., modeling dawn, morning, midday, afternoon, and twilight).

[0134] Similarly, moving, dynamic, or positionable objects within a scene may require increased NPLM complexity if they are rendered using NPLM (optionally, NPLM can be used to render only static scene elements and / or contribute to parts of the scene that are independent of their position). Therefore, again in this case, the input may consist of, for example, object position and / or orientation data.

[0135] Alternatively or additionally, other factors may allow for the simplification of NPLM modeling and thus the reduction of NPLM capabilities (or, all other equals, the comparative improvement of model fidelity). For example, if the rendered scene includes fixed paths (e.g., on a race track, within collision avoidance barriers), training from viewpoints inaccessible to the user can be reduced or completely avoided. Similarly, if the rendered scene includes limited or preferred viewpoints (e.g., a race track where most viewpoints are in the direction of travel), training for different viewpoints can reflect the proportional importance of those viewpoints to the end use case.

[0136] Similarly, if parts of a scene are unlikely to be viewed critically by the user because they are in the background or far from the game's focus (from the perspective of foveal rendering or from the perspective of points of interest such as main characters), NPLM may perform relatively poorly. For example, different NPLMs can be trained for different drawing distances to objects or textures, resulting in reduced performance (e.g., size) at different drawing distances / levels of detail (LOD).

[0137] Alternatively or additionally, as described elsewhere herein, an NPLM can be trained for a specific scene, object, material, or texture. As a result, the performance of an NPLM can vary according to the complexity of what it represents as illumination. Larger or more complex scenes may require larger NPLMs (and / or multiple NPLMs, each handling a different part, depending on the size of the scene and the resulting NPLM). Similarly, complex objects (such as cars) may benefit from a more powerful NPLM than simpler objects (such as spheres). One way to assess the complexity of a scene or object is to count the number of polygons; the more polygons there are, the more complex the scene is inferred. Further refinements can be made to infer complexity using the variance of planar angles between polygons. For example, a sphere with the same number of polygons as the car model in the illustration has much smaller angular variance than the car itself, indicating that the car is structurally more complex. Combining both polygon count and angular variance / distribution would provide a good proxy for the complexity of the scene / object whose illumination is being modeled by the NPLM.

[0138] Similarly, complex materials (such as skin or fur) may benefit from a more capable NPLM (and / or multiple NPLM contributions) than simple materials (such as metals). Furthermore, complex textures (e.g., with a broad spatial spectrum) may benefit from a more capable NPLM than textures with a narrow or condensed spatial spectrum.

[0139] So far, capacity has been described in terms of size (number of inputs / outputs, layers, nodes, etc.), but capacity can alternatively or additionally be varied by the choice of activation functions between nodes in different layers of the NPLM. As noted elsewhere herein, the preferred activation function for the position network is the ReLU function, while the preferred activation function for the direction network is the sine function, although other functions can be chosen to model other scenarios.

[0140] The capacity of the NPLM may be subject to an upper limit, for example, when the memory footprint of the NPLM reaches a threshold size. The threshold size may be equal to the size of an operational unit of memory, such as a memory page or a partial or multiple group of memory pages, typically selected for the purpose of accessing and loading scene / object / material textures. The threshold size may also be equal to the texture or mimmap size used by the GPU and / or game to load graphics image data into the GPU.

[0141] If the complexity of an NPLM were to exceed this threshold, the task it models may have to be simplified or shared among NPLMs, or less accurate results may have to be accepted.

[0142] Network Selection The network is trained during the development phase of a game or application. Developers may choose when or where NPLM-based rendering is advantageous. For example, they may use it only for scenes where the frame rate is consistently found to be below a certain quality threshold. In such cases, the network is trained on those scenes, or parts of them, and used when those scenes are encountered.

[0143] In other cases, a developer may choose to use NPLM-based rendering for specific objects or specific materials. In this case, the network is trained and used for those objects or materials when they are identified as being in the scene being rendered. Similarly, a developer may choose to use NPLM-based rendering at a specific drawing distance (z-distance), or angle / distance away from the image center or the user's foveal view, or for specific lighting conditions. In this case, the network is trained and used for those situations.

[0144] Likewise, it will be appreciated that any suitable combination of these criteria may be selected for training and use.

[0145] However, as noted above, during use of the system, there may be multiple NPLMs associated with a scene for multiple reasons. For example, multiple NPLMs may exist to model a large scene (such that each portion is sufficiently well modeled by an NPLM within a threshold size and / or threshold image reproduction quality). Similarly, multiple NPLMs may exist due to different lighting conditions, level of detail / drawing distance, etc.

[0146] An NPLM appropriate for the situation may be selected and retrieved into the GPU's accessible working memory and executed for the purpose of rendering at least a portion of the image. It will be appreciated that strategies applied to prefetching and caching textures and other graphics assets may also be applied to the NPLM.

[0147] Tile-shaped NPLM To render an image, it is necessary to evaluate one NPLM for each contributing component of each pixel in the image (or that portion of the image that is being rendered in this manner, e.g., a particular object or material). Typically, there are at least two components (e.g., diffuse and specular, as described elsewhere herein), and therefore at least two NPLMs are typically used for each pixel.

[0148] As a result, one way to quickly render an image would be to run the contributing instances of NPLM for every pixel under consideration and evaluate them all in parallel, but this is currently impractical because the memory requirements would be larger than what is available on existing GPUs.

[0149] Nevertheless, efficient parallelization of NPLM-based rendering is desirable.

[0150] The positional information used by NPLM is generated by an input buffer or similar, which is created by rendering the scene's 3D geometry (for example, by assembling the geometry and removing hidden surfaces, etc.). The traditional rendering process would then apply textures and lighting to this geometry.

[0151] As described elsewhere herein, the NPLM generates learned quantities that are combined with a BSDF or similar description of the material properties of the scene at a position relative to the lighting to create components that contribute to the rendering of a pixel in the image (as indicated by the directional input).

[0152] Therefore, by associating appropriate data with the input buffers, efficient parallelization of the NPLM can be implemented.

[0153] This data includes either or both a cluster ID and a material ID. The cluster ID itself can correspond to geometric components, objects, zones in a scene (e.g., to equal cuboids based on the volume division of the scene, or following any appropriate division scheme), or it can actually correspond to the material of an object and / or within the scene.

[0154] The data also includes surface position information for surfaces in the scene that are input to the NPLM, as discussed elsewhere herein, and optionally includes orientation information responsive to the current virtual camera position relative to the image.

[0155] For each pixel in the image, for the corresponding surface location, the cluster ID or material ID is used to select the appropriate NPLM for that cluster or material.

[0156] If material ID data for that surface location is associated with the input buffer (or is derivable from the cluster ID, e.g., via a lookup table), optionally a BSDF (or other distribution function as described elsewhere herein) can be calculated for the material at that surface location and multiplied with the output of the selected NPLM to generate a contribution for the current pixel. Alternatively, the BSDF can be retrieved from a store.

[0157] Thus, an NPLM is selected for a given cluster ID or material ID at that surface location and used to generate a training quantity for that surface location. In parallel, a BSDF or other distribution function may be generated in response to the material ID, or alternatively, searched in response to the material ID or in response to the surface location.

[0158] An NPLM may be specific to a cluster ID or material ID, or may be used for more than one cluster ID or material ID.

[0159] The number of different NPLMs that can be used depends on the granularity of the cluster ID or material ID, and whether the relationship between the NPLMs and the cluster ID or material ID is one-to-one or one-to-many.

[0160] It is possible to have multiple NPLMs to be performed on different elements of the image, and efficient ways to do so can be explored.

[0161] Therefore, referring to Figure 12, for an image of a scene from a specific virtual camera viewpoint, such a process may be constructed as follows:

[0162] i. Scan the input buffer to find the number of unique IDs (e.g., cluster IDs, or material IDs if cluster IDs are not used; more generally, any suitable IDs for scene elements associated with a particular NPLM). In Figure 12, this is indicated by color or grayscale differentiation of features in the example automobile image 1210.

[0163] ii. Create a mask, such as a binary mask, for each such ID, where each pixel in the mask is set to 1 to find the specific ID (hereafter "ID-attached pixel") and set to 0 otherwise. In Figure 12, for the example car, nine masks 1220 corresponding to the nine different cluster IDs are obtained.

[0164] iii. Divide each mask into a set of two-dimensional tiles. The size of the tiles can be adjusted depending on the memory capacity of the GPU (e.g., for efficient memory allocation per thread or per shader). A non-limiting example size is 64x64 pixels, but any size suitable for the memory and processing capabilities of the GPU can be considered. Empty tiles (e.g., tiles with all zeros) are discarded or otherwise not processed further. In Figure 12, tiled mask 1230 shows a mixture of empty tiles, high occupancy tiles, and sparse occupancy tiles.

[0165] iv. For each remaining (non-empty) tile, create a tile (or equivalent data set) of position and direction inputs for each IDed pixel in the mask. This results in a list of tiles (or equivalent data sets) that specify the location and orientation inputs of the NPLMs, and the associated NPLMs are identified from their IDs (eg, cluster IDs or material IDs).

[0166] v. Process batches of these tiles in parallel, the size of the batch can be adjusted depending on available resources, for example based on the number of threads / shaders needed and currently available. For a given tile, the entire tile can optionally be processed by the NPLM. This can be simpler from a data and memory management perspective (e.g., memory and computation requirements are predictable for the tile, allowing batches to complete synchronously), but for tiles with few ID pixels, it results in redundant processing by the NPLM, which can be an undesirable overhead. For non-ID pixels, null or default input data can be applied to the NPLM. In this approach, a mask can be reused to identify valid outputs from the NPLM associated with ID pixels. Alternatively, only the identified pixels can be processed by the NPLM using the associated position and orientation data to generate the output. This is more complex from a data and memory management perspective, as it requires scanning the tile for identified pixels, and also results in an unclear memory and computational budget. Rather than creating a budget for the complete tile, the computational budget for each tile is different and requires a variable record of the identified pixel locations to be stored after the initial scan of the tile, or the computational budget remains unknown until the end of the tile if the operations and NPLM process are sequential (i.e., processing identified pixels as they are found). Similarly, the time it takes to process a tile is variable and potentially unknown, making memory management and parallelization more complex. However, sparser tiles can avoid unnecessary use of the NPLM for non-identified pixels. One or the other approach can be adopted, or the system can switch approaches depending, for example, on a threshold occupancy (number or percentage) of IDed pixels in a tile (the counting can be performed when generating the mask). This threshold can be chosen to reflect the relative scarcity of computational or memory resources in the system. Thus, if computational resources are relatively scarce compared to memory resources, the threshold occupancy for switching from processing entire tiles to processing IDed pixels can be higher to reduce the number of redundant (unused) NPLM processes.

[0167] vi. The output of the NPLM for each IDed pixel can then be used to contribute to a rendered image, as described elsewhere herein. As the output from each NPLM is received, it may be added to a global radiance buffer for the image, with each pixel updated from its corresponding tile (and mask, if necessary). As noted elsewhere herein, typically more than one NPLM contributes to each pixel, so the buffer may have more than one channel (e.g., a diffuse component and a specular component). The result is the radiance portion of the rendering. It is noted that this is achieved while controlling the resources used at any given time by adjusting the tile size and batch processing size so that the number of NPLM instances and tiles processed in parallel make efficient use of currently available resources (e.g., on the GPU).

[0168] vii. Optionally, a BSDF or other distribution function can be calculated in parallel from material information associated with the material ID (if used). Alternatively, the BSDF or other distribution function can be retrieved from memory (optionally after being calculated once for the material ID and then saved during run-time).

[0169] viii. In any case, each contributing component in the global radiance buffer is then combined (e.g., multiplied) with an associated distribution function, and the results are combined to produce the image (or portion thereof that is rendered using this technique).

[0170] Therefore, the technique described above can be understood as dividing the scene image into multiple masks corresponding to each ID within the scene, and then decomposing each of these masks into a set of tiles. The batches of tiles are then processed in parallel using the appropriate NPLM for the corresponding ID of a given tile. This processing can, optionally, be tile-based or ID-based for a given tile or batch of tiles.

[0171] Within the present technology, any of the following can be adjusted to make efficient use of available computing and / or memory resources:

[0172] Number of IDs: This affects the number of masks processed (and therefore indirectly the number of tiles). A trade-off between the number of tiles generated and the potential precision of the NPLM may also be considered (more IDs may result in a more specialized NPLM, fewer IDs in a less specialized NPLM, or a larger / more complex NPLM with equivalent precision can be used, as described elsewhere in this specification).

[0173] Tile size: Typically, this may be determined by the local memory capacity of the thread / shader that processes the tile.

[0174] Number of tiles in a batch: Typically, this may be determined by the number of threads / shaders currently available (or reserved).

[0175] A threshold for the percentage of ID-based pixels in a tile or batch of tiles used to transition between processing the entire tile or ID-based pixels: this threshold can be adjusted based on GPU utilization. If there is sufficient capacity to process the entire tile, it provides benefits in terms of batch processing synchronization at the expense of unnecessary NPLM processing (the percentage of unnecessary processing depends on the number of non-ID pixels in the tile).

[0176] It will be understood that tiles from different masks can be processed in the same batch. It will also be understood that tiles can be processed in any order. Together, this allows for the assembly of the most possible batches of tiles that satisfy all the ID-based pixel occupancy thresholds for using the full-tile approach, increasing the number of batches that can be processed synchronously.

[0177] Furthermore, during each frame rendering cycle, other processes may require more or less computing resources. Therefore, it will be understood that, at will, the above technique may choose to process either the whole tile batch or the ID-paired pixel batch during the rendering cycle, depending on the fluctuating availability of computing resources, for example by appropriately changing the threshold.

[0178] summary Referring now to Figure 9, in a summary embodiment of this specification, an image rendering method for rendering pixels at a viewpoint comprises the following steps for a first element of a virtual scene having a predetermined surface at a location within that scene.

[0179] In the first step s910, a viewpoint-based position and orientation are provided to a machine learning system that has been pre-trained to predict a factor that generates a pixel value corresponding to a first element of a virtual scene to be illuminated at that position, when coupled with a distribution function that characterizes the interaction of light with a given surface, as described elsewhere in this specification.

[0180] In the second step s920, the predicted coefficients from the machine learning system are combined with the distribution function to generate pixel values ​​corresponding to the first elements of the illuminated virtual scene at the location, as described elsewhere in this specification.

[0181] Then, in the third step s930, the pixel values ​​are incorporated into the rendered image for display, as described elsewhere in this specification. The image can then be output to a display via the A / V port (90).

[0182] It will be apparent to one of ordinary skill in the art that one or more variations in the above methods corresponding to the operation of various embodiments of the methods and / or apparatus as described and claimed herein are contemplated within the scope of this disclosure, including, but not limited to, the following: As described elsewhere herein, each machine learning system is trained for each of a plurality of contributing components of the image (e.g., diffuse, specular, coated, etc.), and a respective distribution function is used for each of the plurality of contributing components of the image, and the resulting pixel values ​​for each are combined to create pixel values ​​that are incorporated into the rendered image for display. As described elsewhere herein, each distribution function is one or more selected from the list comprising a bidirectional scattering distribution function, a bidirectional reflectance distribution function, and a bidirectional transmittance distribution function. As described elsewhere herein, the machine learning system is a neural network, where the input to a first portion of the neural network includes a position and the input to a second portion of the neural network includes an output of the first portion and a direction. As explained elsewhere herein, in this example, the activation function of the first part is different from the activation function of the second part. As explained elsewhere herein, in this example the activation function of the first part is a ReLU function and the activation function of the second part is a sine function. As explained elsewhere in this specification, in this example the cost function of the neural network is based on the difference between the output of the second part and the value obtained from a ray-traced version of the pixel for the training image where the inverse combination with the distribution function has been performed. As described elsewhere herein, in this example, the cost function for the network is backpropagated through both the second and first parts during training. As explained elsewhere herein, in this example the neural network is a fully connected network. As described elsewhere in this specification, the cost function of a machine learning system is based on the difference between the output of the machine learning system and the values ​​obtained from a ray-traced version of the pixels for the training image, on which an inverse connection in the distribution function has been performed. - The machine learning system is selected and loaded into memory used by the graphics processing unit, based on the same asset identification scheme used to select and load textures for the first element of the scene.

[0183] Next, referring to Figure 10, in another summary embodiment of this specification, an image rendering method for a virtual scene (focusing on parallelizing NPLM and tiling different aspects of an image) includes the following steps:

[0184] As described elsewhere in this specification, for multiple IDs, the first step s1010 includes generating a mask that identifies the scene elements associated with each ID (e.g., cluster ID or material ID).

[0185] As described elsewhere in this specification, for the multiple masks obtained, the second step s1020 involves dividing each mask into multiple tiles and discarding any tile that does not identify any image element (or equivalently, that tile can be ignored until all other tiles of the mask have been processed in a subsequent step, after which all tiles of the mask can be discarded).

[0186] As described elsewhere in this specification, for the resulting plurality of remaining tiles, a third step s1030 includes selecting a respective trained machine learning model from among a plurality of machine learning models, each machine learning model being trained to generate data that contributes to rendering at least a portion of the image based on elements of the scene associated with the same respective IDs as elements identified in the mask into which the respective tile was divided.

[0187] Next, as described elsewhere in this specification, a fourth step s1040 includes using each trained machine learning model to generate data that contributes to rendering at least a portion of the image based on at least input data regarding the identified elements in each tile.

[0188] It will be apparent to one of ordinary skill in the art that one or more variations in the above methods corresponding to the operation of various embodiments of the methods and / or apparatus as described and claimed herein are contemplated within the scope of this disclosure, including, but not limited to: -As described elsewhere in this specification, the tile size is selected according to the capabilities of the processing hardware. As described elsewhere in this specification, batches of tiles are processed in parallel by multiple, each trained, machine learning model. As described elsewhere in this specification, in this example, the batch size is optionally selected according to the capabilities of the processing hardware. As described elsewhere herein, each trained machine learning model generates data that contributes to rendering at least a portion of the image based on one selected from a list that includes input data for only the identified elements within the respective tile and input data for the respective tile as a whole. As described elsewhere in this specification, each trained machine learning model generates data that contributes to rendering at least a portion of an image based on input data only for identified elements within the respective tile, or based on input data for the entire respective tile and depending on whether the identified elements within the respective tile meet occupancy criteria. As described elsewhere in this specification, the generated data includes coefficients that, when combined with a distribution function that characterizes the interaction of light with each portion of the virtual environment, generate pixel values ​​that correspond to pixels of a rendered image that includes that respective portion of the virtual environment. As described elsewhere in this specification, in this example, optionally, a respective trained machine learning system is learned for each of a plurality of contributing components of the image, a respective distribution function is used for each of a plurality of contributing components of the image, and the respective generated pixel values ​​are combined to generate a final combined pixel value that is incorporated into the rendered image for display. Also in this example, as described elsewhere herein, alternatively or additionally, the material properties of the elements can be obtained by reference to their respective IDs, and at least a first respective distribution function corresponding to the respective IDs is obtained by selecting from a list that includes retrieving the distribution function from storage and calculating the distribution function in parallel with using at least a first respective trained machine learning model to generate data. As described elsewhere herein, each trained machine learning system is a neural network, where inputs to a first portion of the neural network include positions of elements in the virtual environment and inputs to a second portion of the neural network include outputs of the first portion and a viewpoint-based orientation of at least a portion of the image to be rendered.

[0189] It will be appreciated that the above methods may be implemented by conventional hardware suitably adapted as applicable by software instructions or by the inclusion or substitution of dedicated hardware.

[0190] Thus, any necessary adaptations to existing portions of a conventional equivalent device may be implemented in the form of a computer program product including processor-implementable instructions stored on a non-transitory machine-readable medium such as a floppy disk, optical disk, hard disk, solid state disk, PROM, RAM, flash memory, or any combination thereof, or other storage medium, or may be realized in hardware as configurable circuitry suitable for use in adapting a conventional equivalent device such as an ASIC (application specific integrated circuit) or FPGA (field programmable gate array). Alternatively, such a computer program may be transmitted by data signals over a network such as an Ethernet, a wireless network, the Internet, or any combination of these or other networks.

[0191] 1, the methods and techniques described herein may be implemented in conventional hardware, such as an entertainment system 10 that generates images from virtual scenes. An example of such an entertainment system 10 is a computer or console, such as the Sony® PlayStation 5® (PS5).

[0192] The entertainment system 10 includes a central processing unit 20, which may be a single-core or multi-core processor, including, for example, eight cores, as in the PS5. The entertainment system also includes a graphical processing unit or GPU 30. The GPU may be physically separate from the CPU, or may be integrated with the CPU as a System on a Chip (SoC), as in the PS5 (registered trademark).

[0193] The entertainment device also includes RAM 40, which may have separate RAM for the CPU and GPU respectively, or have shared RAM as in the PS5®. Alternatively, each RAM may be physically separate or integrated as part of the SoC as in the PS5®. Further storage is provided by disk 50, which may be provided as an external or internal hard drive, or as an external solid-state drive, or as an internal solid-state drive as in the PS5®.

[0194] The entertainment device may optionally transmit or receive data via one or more data ports 60, such as a USB port, an Ethernet port, a WiFi port, a Bluetooth port, etc., and may also optionally receive data via an optical drive 70.

[0195] Interaction with the system is typically provided using one or more handheld controllers 80, such as the DualSense® controller in the case of the PS5®.

[0196] Audio / visual output from an entertainment device is typically provided via one or more A / V ports 90 or through one or more of wired or wireless data ports 60 .

[0197] Where components are not integrated, they may be connected by dedicated data links or via bus 100 as appropriate.

[0198] Thus, in a summary embodiment herein, an entertainment device (such as a Sony® PlayStation 5®) comprises:

[0199] First, as described elsewhere in this specification, a graphics processing unit (such as GPU 30, optionally in conjunction with CPU 20) is configured (e.g., by appropriate software instructions) to render pixels to a viewpoint within an image of a virtual scene that includes a first element having a predetermined surface at a location within that scene.

[0200] Second, as described elsewhere herein, a machine learning processor (such as GPU 30, optionally in conjunction with CPU 20) is configured (e.g., by appropriate software instructions) to provide a viewpoint-based position and orientation to a machine learning system that has been pre-trained to predict coefficients that, when combined with a distribution function that characterizes the interaction of light with a given surface, generate a pixel value corresponding to a first element of the virtual scene as illuminated at that location.

[0201] The graphics processing unit is configured (again, for example, by appropriate software instructions) to combine predicted coefficients from a machine learning system with a distribution function, as described elsewhere in this specification, to generate pixel values ​​corresponding to the first illuminated element of the virtual scene at a given location.

[0202] Furthermore, the graphics processing unit can also be configured to incorporate pixel values ​​into the rendered image for display (again, for example, by appropriate software instructions), as described elsewhere in this specification.

[0203] The hardware described above may also be configured to perform the methods and techniques described herein, and it will be understood that it may be configured, for example, as follows: As described elsewhere in this specification, the entertainment device comprises multiple machine learning processors (e.g., each processor of the GPU and / or CPU, seeds and / or shaders) running their respective machine learning systems, each trained on one of the multiple contributing components of an image (e.g., diffuse, specular, coat, etc.). Each distribution function is used for each of the multiple contributing components of the image, and the graphics processing unit is configured (again, for example, by appropriate software instructions) to combine the respective generated pixel values ​​to create pixel values ​​that are incorporated into a rendered image for display. the or each machine learning system is a neural network, and the input to a first part of the neural network comprises a position, and the input to a second part of the neural network comprises an output of the first part and a direction.

[0204] Similarly, in another summary embodiment of the present invention, an entertainment device (such as a Sony® PlayStation 5®) comprises:

[0205] As described elsewhere herein, first, a mask processor (e.g., CPU 20 and / or GPU 30) is adapted (e.g., by appropriate software instructions) to generate, for a plurality of IDs, respective masks that identify elements of the scene associated with each ID.

[0206] Furthermore, as described elsewhere in this specification, the tile processor (e.g., CPU 20 and / or GPU 30) is adapted (e.g., by suitable software instructions) to, given the multiple masks obtained, divide each mask into multiple tiles and discard tiles that do not identify any image elements.

[0207] Further, as described elsewhere herein, the selection processor (e.g., CPU 20 and / or GPU 30) is adapted (e.g., by appropriate software instructions) to select, for the resulting plurality of remaining tiles, a respective trained machine learning model from among the plurality of machine learning models, each trained to generate data that contributes to rendering at least a portion of the image based on elements of the scene associated with the same respective IDs as elements identified in the mask into which the respective tile was divided.

[0208] Further, as described elsewhere herein, the rendering processor (such as GPU 30) is adapted (e.g., by appropriate software instructions) to use the respective trained machine learning models to generate data that contributes to rendering at least a portion of the image based on input data for at least the identified elements in each tile.

[0209] It will be appreciated that the above hardware may likewise be configured to perform the methods and techniques described herein, for example: -As described elsewhere herein, one or more are selected from a list including tile sizes and sizes of batches of tiles to be processed in parallel by each of the multiple trained machine learning models according to the capabilities of the processing hardware. As described elsewhere herein, each trained machine learning model generates data that contributes to rendering at least a portion of the image based on one selected from a list including input data for only the identified elements within the respective tile and input data for the respective tile as a whole. As described elsewhere in this specification, the rendering processor is adapted to select whether each learned machine learning model generates data that contributes to rendering at least a portion of the image based on input data for only the identified elements in the respective tile or based on input data for the entire respective tile, depending on whether the identified elements in the respective tile satisfy an occupancy criterion.

[0210] The foregoing discussion discloses and describes merely exemplary embodiments of the present invention. As will be understood by those skilled in the art, the present invention may be embodied in other specific forms without departing from its spirit or essential characteristics. Accordingly, the disclosure of the present invention, as well as the remaining claims, are intended to be illustrative, but not limiting, of the scope of the present invention. This disclosure, including any readily identifiable variations of the teachings herein, shall in part define the scope of the foregoing claim terms, so that inventive subject matter is not dedicated to the public.

Claims

1. A method for rendering images in a virtual scene, Regarding multiple IDs, The steps include generating a mask that identifies the scene element associated with each ID, Regarding the multiple masks that were generated, The steps include dividing each mask into multiple tiles, A step of discarding tiles that do not identify any image elements, Regarding the remaining tiles, selecting a respective trained machine learning model from among a plurality of trained machine learning models, each trained to generate data that contributes to rendering at least a portion of an image based on elements of the scene associated with the same respective IDs as elements identified in a mask into which each of the tiles is divided; using the respective trained machine learning models to generate data that contributes to rendering at least a portion of the image based on at least input data for the identified elements of the respective tiles; Image rendering methods, including those mentioned above.

2. The method of claim 1 , wherein the size of the tiles is selected according to processing hardware capabilities.

3. 3. The method of claim 1, wherein batches of tiles are processed in parallel by a plurality of respective trained machine learning models.

4. The method of claim 3 , wherein the batch size is selected according to processing hardware capabilities.

5. Each of the aforementioned trained machine learning models is i. Input data for only the elements identified in each of the aforementioned tiles, ii. Input data for each tile as a whole, 5. The method of claim 1, further comprising generating data that contributes to rendering at least a part of the image based on one selected from the list consisting of:

6. 6. The image rendering method of claim 1, wherein the respective trained machine learning models generate data that contributes to rendering at least a portion of the image based on input data only for the identified elements in the respective tiles or based on input data for the entire respective tiles, depending on whether the identified elements in the respective tiles satisfy an occupancy criterion.

7. 7. The image rendering method of claim 1, wherein the generated data includes coefficients that, when combined with a distribution function characterizing the interaction of light with each portion of the virtual environment, generate pixel values ​​corresponding to pixels of a rendered image including the respective portion of the virtual environment.

8. Each trained machine learning system is trained for each of the multiple contributing components of the image, Each distribution function is used for each of the multiple contributing components of the image.

8. The image rendering method of claim 7, wherein each generated pixel value is combined to create a final combined pixel value that is incorporated into the rendered image for display.

9. The material properties of the aforementioned elements can be obtained by referring to their respective IDs. Each of the above IDs corresponds to at least one first distribution function, i. Steps to retrieve the distribution function from storage, ii. Calculating the distribution function in parallel with using at least a first respective trained machine learning model to generate data; The image rendering method according to claim 7 or 8, obtained by one selected from a list consisting of the following.

10. Each of the aforementioned trained machine learning systems is a neural network. The input to the first part of the neural network includes the position of an element in the virtual environment.

9. The method of claim 8, wherein inputs to a second portion of the neural network include the output of the first portion and a viewpoint-based orientation of the at least part of the image to be rendered.

11. A computer program comprising computer executable instructions adapted to cause a computer system to carry out the method of any one of claims 1 to 10.

12. 1. An entertainment device operable to render an image of a virtual scene, comprising: A mask processor adapted to generate a mask for each of multiple IDs that identifies the elements of the virtual scene associated with each ID, A tile processor adapted to divide each of the generated masks into multiple tiles and discard tiles that do not identify any image elements, A selection processor adapted to select a trained machine learning model from among a plurality of trained machine learning models for each of the remaining tiles that result, wherein each trained machine learning model is trained to generate data that contributes to rendering at least a portion of the image, based on the elements of the virtual scene associated with the same ID as the elements identified in the mask from which each tile is divided. A rendering processor adapted to generate data that contributes to rendering at least a portion of the image, using each of the trained machine learning models based on input data for the identified elements in at least each of the tiles, Entertainment devices, including

13. i. The size of the aforementioned tile and ii. The size of the batches of tiles to be processed in parallel by each of the multiple trained machine learning models; and The entertainment device according to claim 12, wherein one or more are selected from a list consisting of the following, depending on the capabilities of the processing hardware.

14. Each of the aforementioned trained machine learning models is i. input data for only the identified elements in each of the tiles; ii. Input data for each of the entire tiles; The entertainment device according to claim 12 or 13, which generates data that contributes to rendering at least a portion of the image based on one selected from a list consisting of the following:

15. The entertainment device according to any one of claims 12 to 14, wherein the rendering processor is adapted to select whether the respective trained machine learning model generates data that contributes to rendering at least a portion of the image, based on input data only for the identified element in each of the tiles, or based on input data for the entire tile, depending on whether the identified element in each of the tiles satisfies an occupancy criterion.

Images