Image rendering method and apparatus

Abstract

To provide an image rendering method and apparatus.SOLUTION: An image rendering method for rendering a pixel at a viewpoint, for a first element of a virtual scene having a predetermined surface at a position within that scene, comprises: a step of evaluating whether to render a pixel corresponding to the first element using a machine learning system having been trained to output a value representative of lighting of the predetermined surface at the position, or to render the pixel corresponding to the first element using an alternative rendering approach; and a step of rendering the pixel using the machine learning system or the alternative rendering approach selected in the evaluation step.SELECTED DRAWING: Figure 10

Description

Technical Field

[0001] The present invention relates to an image rendering method and apparatus.

Background Art

[0002] Ray tracing is one of the rendering processes, in which the light paths in a virtual scene are traced. Thereafter, the interaction between each ray and an object or surface in the scene is simulated. To enhance reality, this simulation usually takes into account the material characteristics (e.g., color and reflectivity) of the object or surface.

[0003] As a result, the process of ray tracing is computationally expensive. Furthermore, this cost varies for each image frame depending on, for example, "which scene is lit by what kind of light and from what viewpoint".

[0004] Therefore, it is difficult to maintain a suitable frame rate for rendering such images. Regarding the average computational cost corresponding to the average image completion time (i.e., frame rate), and a predetermined discrepancy in the average periphery due to ray tracing, the average image quality must be set low so that the frame rate is not greatly affected by this discrepancy. When the average image quality is set high close to the maximum value for a desired frame rate, if the trace requirements that change moment by moment fluctuate beyond the average, the consistency of the frame rate is impaired.

[0005] None of these results are desirable. However, this cannot be easily avoided. On the other hand, the computational load of the ray tracing process is data-driven and unpredictable.

Summary of the Invention

Problems to be Solved by the Invention

[0006] The present invention addresses such problems and aims to mitigate them. [Means for solving the problem]

[0007] Various aspects and features of the present invention are defined by the claims and the following specification, but include at least the following: -The first example is the image rendering method according to claim 1. -The second example is the entertainment device according to claim 14. [Brief explanation of the drawing]

[0008] By referring to the attached drawings below and reading the detailed description herein, you will gain a more complete understanding of the features and many advantages of this disclosure. [Figure 1] This is a schematic diagram of an entertainment device according to an embodiment of the present disclosure. [Figure 2] This is a ray-traced object according to the embodiments of this disclosure. [Figure 3] This is a schematic diagram of the components contributing to a ray-traced object according to embodiments of the present disclosure. [Figure 4] This is a schematic diagram of the distribution function associated with each component in the embodiment of the present disclosure. [Figure 5] This is a schematic diagram showing the scattering distribution according to an embodiment of the present disclosure. [Figure 6] This is a schematic diagram of a training scheme for machine learning according to an embodiment of the present disclosure. [Figure 7] This is a schematic diagram of the rendering path for a rendered image according to an embodiment of the present disclosure. [Figure 8A] This is a schematic diagram of a machine learning system according to an embodiment of the present disclosure. [Figure 8B] This is a schematic diagram of a part of the machine learning system according to an embodiment of the present disclosure. [Figure 9] This is a flowchart of an image rendering method according to an embodiment of the present disclosure. [Figure 10] This is a flowchart of an image rendering method according to an embodiment of the present disclosure. [Figure 11] This is a schematic diagram of a training method according to an embodiment of the present disclosure. [Modes for carrying out the invention]

[0009] The following discloses image rendering methods and apparatus. For a complete understanding of embodiments of the present invention, the following description details several specific examples. However, it will be apparent to those skilled in the art that these details are not essential for realizing the present invention. Conversely, details of the prior art known to those skilled in the art are omitted where necessary to clarify the description.

[0010] Embodiments of this disclosure aim to address and mitigate the above-mentioned problems by using a machine learning system that learns the relationship between pixel surface characteristics and rendered pixels with respect to a given object or scene. By using such a machine learning system, it is possible to approximate the rendering of a ray-traced object or scene with relatively stable computational costs (for performing the machine learning).

[0011] As will be discussed later, different machine learning systems can be trained depending on different scenes, locations or parts thereof, different objects, or materials used in different scenes.

[0012] Machine learning systems are relatively small (typically 100kB to 1MB). Therefore, to run them on a GPU (30), they can be stored in memory and then discarded (similar to scene textures). This system can run on the GPU's shaders. In principle, machine learning systems can also run on a CPU or a general-purpose or dedicated coprocessor (e.g., a neural network processor or ASIC).

[0013] In multiple drawings, similar symbols represent the same or corresponding parts. Figures 2 through 7 show the problem space in which the machine learning system is trained.

[0014] Figure 2 shows a high-quality ray-traced render 200 of an object or scene (in this case, a car on a platform).

[0015] Figure 3 shows the different contributing components behind this render. First, the diffuse reflection lighting component 200-D typically captures the surface's matte color and the shading resulting from the interaction of light with the object. Second, the specular reflection lighting component 200-S captures the surface's reflectivity, resulting in glints and highlights. Optionally, one or more additional components can be included. For example, the gloss and "coating" 200-C. This is a second outer surface and includes additional gloss and patterns. Variations of such coatings can also include components such as skin or fabric, which are transparent and / or partially diffusely reflective. Each of these components can be generated in the normal way by each ray-tracing process.

[0016] These components are additionally added together to form the overall image as shown in Figure 2. Typically, two or three components contribute to the render. However, under certain conditions, there can be fewer (e.g., when only the diffuse reflection lighting component is needed) or more (e.g., when the object is translucent and a transmission component is also needed).

[0017] Figure 4 below shows the material properties of the object underlying the contributing components of the above image.

[0018] The material properties are represented as the so-called bidirectional scattering distribution function (BSDF) or bidirectional reflectance distribution function (BRDF).

[0019] BRDF defines how light reflects off an opaque surface. Similarly, BSDF defines the probability that a light ray will reflect or diffuse in a particular direction. Thus, BRDF or BSDF are functions that describe the lighting properties of a surface (excluding the incident / exit radiation itself). Other functions may be used if necessary, such as the bidirectional transmission distribution function (BTDF), which defines how light passes through an object.

[0020] A typical ray tracing application is illustrated with reference to Figure 5. For a set of rays (e.g., from a compact light source), this application calculates the incident luminance (either the incident light itself or the reflected light) to a point on a model with a specific BSDF, BRDF, and / or BTDF. For the response of a particular contributing component, the incident luminance is combined (e.g., multiplied) with the BSDF, BRDF, or BTDF. The result is added to the pixel value on the point in the model. The optical path ω of the BSDF is shown in Figure 5. i A typical diffuse reflection pattern for this is the specular reflection direction ω a It has a bias toward one direction but can diffusely reflect in any direction. Accurately modeling this behavior is one of the reasons why ray tracing is computationally expensive.

[0021] By using the color information of the model at each point, and the BSDF, BRDF, and / or BTDF corresponding to that point (i.e., the specific material represented by the given point), it is possible to calculate the behavior of rays with respect to a given final viewpoint (e.g., specular or diffuse reflection of light, which determines the realistic contribution of sparkle and highlights on the surface of the vehicle body).

[0022] Individual BSDF, BRDF, or BTDF can be used for each contributing component. For example, (non-limitingly) BSDF can be used for the diffuse reflection component, and BRDF for the specular reflection component and coating component (conversely, BTDF can also be used for the coating component). BSDF, BRDF, or BTDF can be used as needed. Therefore, unless otherwise specified, the term BSDF includes BRDF and BTDF.

[0023] As shown in Figure 4, ray tracing using the object's color properties and the diffuse reflection material properties of BSDF(200-BSDF-D) yields the diffuse reflection component 200-D. Similarly, using the specular reflection or reflective material properties of BSDF(200-BSDF-S) yields the specular reflection component 200-S. Likewise, using the material properties of BSDF(200-BSDF-C) (which is usually also specular reflection) yields the coating component 200-C. Combining these components ultimately yields the ray-traced image 200.

[0024] However, as mentioned above, the challenges lie in the following: the computational cost is high whether calculating the diffuse reflection or reflection path of light incident on different surfaces with different BSDFs, or whether summing up those results pixel by pixel in a scene from a specific viewpoint, and these costs can vary considerably.

[0025] Accordingly, the embodiments of this disclosure aim to replace the ray tracing step in Figure 4 with one that is suitable for the final image quality and whose computational load is more predictable.

[0026] Refer to Figure 6. In the embodiments of this disclosure, a machine learning system is provided for each contributing component of the image (diffuse reflection, specular reflection, and optionally coating or other arbitrary contributing components).

[0027] The machine learning system is typically a neural network, as described below. It is trained to learn the transformation between BSDF (e.g., 200-BSDF-D) and the ground truth (e.g., 200-D) of the ray-traced image contribution components for multiple images viewed from different viewpoints within a scene.

[0028] In other words, suppose the ray-traced image (or one of its contributing components) is a combination of the lighting behavior on an object and the BSDF (which represents how the object reacts to light). Now, suppose we extract the ray-traced image and separate it from the BSDF. The result is a quality called "radiance" or "density," which more generally indicates how light behaves on the object (as calculated holistically by the ray tracing process).

[0029] A machine learning system or neural network can learn to predict this quality. The predicted quality is then combined again with the BSDF to generate a predicted image that approximates the ray-traced image. This network can therefore be called a neural pre-computed optical model network (NPLM network).

[0030] More specifically, for a given position and line of sight on a virtual image of an object, the machine learning system or neural network needs to learn to output a value. This value is close to the pixel value obtained when ray tracing is performed on that pixel, when combined with a BSDF for the same position / pixel. As a result, during training, an internal representation of the lighting conditions (e.g., point light or cyclorama) and surface lighting characteristics suggested by the training image are generated.

[0031] In one embodiment, the image may be rasterized or generated at a given viewpoint. This is done by filling the image with pixels and then lighting it. For each of these conceptual pixels, a corresponding 3D position exists in the scene. By using an NPLM network on this, the appropriate "radiance" or "grain" can be obtained.

[0032] Figure 6 shows the training environment for such networks. Figure 6 also shows network 600-D for the diffuse reflection component (for illustrative purposes only).

[0033] The inputs to the network for the diffuse reflection component are the (x, y, z) position 610 on the object or scene and the object / scene normal 620 at that point. Here, the normal N is used instead of the viewpoint direction because, for the diffuse reflection component, illuminance is independent of direction / viewpoint, and from a consistency standpoint, the normal can be used as a known value. In the example in Figure 6, these inputs are conceptually shown using representative values ​​for the car image.

[0034] Selectively, additional inputs may be provided (not shown). For example, scalar values ​​for roughness or matt-to-gloss. These may selectively be derived from the relevant BSDF.

[0035] The output of the NPLM network is (as described later) a learned light or illuminance quality 630 for each input location. This, when combined with the corresponding diffuse reflected light BSDF (200-BSDF-D) for the same location (640), generates predicted pixel values ​​for the location (x, y) in the predicted image (650).

[0036] Figure 6 shows that for each pixel, the difference between the predicted pixel and the ground truth pixel for the target ray tracing diffuse reflection component 200-D is used as the loss function for training the network. However, this is not mandatory. Rather, for each pixel (x, y), the ground truth image can be separated from the BSDF (using the inverse function) to generate a proxy for what cumulative effect the ray tracing light had on the objects in the image. The network is trained to learn this quality.

[0037] Therefore, the error function of the network when separated from the corresponding BSDF is based on the difference between the output value of its single pixel (x, y) and the corresponding single pixel in the ground truth image.

[0038] For each pixel, the pixels in the ground truth image can be pre-separated from their corresponding BSDF. Therefore, the network can be trained to generate actual predicted image pixels without combining its output with the BSDF. This reduces the computational load for training.

[0039] As mentioned earlier, the trained quality output from the trained neural network captures how ambient light behaves on an object or scene as a function of the surface position and viewpoint within the scene. This effectively generates an internal representation of the light map in the scene and surface response model, which will be discussed in more detail later.

[0040] Refer to Figure 7 for a summary of each contributing component of the final output image. The machine learning system is trained to perform a transformation applied to the BSDF of each position on the object / scene with respect to the contributing components. This transformation is a function trained based on the (x, y, z) position and orientation values ​​on the object / scene. As mentioned above, any number of machine learning systems (e.g., one, two, three, four, or more) may be employed depending on the number of contributing components in the final image. Hereafter, the term "trained function" refers to a machine learning system that has learned such a transformation.

[0041] As explained in the section on diffuse reflection components, the directional value can be considered as the normal at a given point. This is because the intensity of diffuse reflection can be considered directionally invariant.

[0042] Specular reflection, on the other hand, is reflection in certain directions and varies depending on the viewpoint. Therefore, for the specular component, its directional value is the line of sight angle between (or based on) the current pixel's (x, y) position at the image viewpoint (which will have a position in virtual space) and the object's (x, y, z) position as input to the machine learning system. This gives a viewpoint-dependent relationship between the input point on the scene surface and the current pixel (with respect to which the learned quality is output).

[0043] In this case, the coating component is also specularly reflected. Therefore, the coating component also uses a similar viewpoint or viewpoint-based orientation with respect to the input.

[0044] The direction value for direction-dependent components may be the line of sight direction (or a value based thereon) between the output pixel position and the object surface position. For example, this may be the surface reflection viewpoint direction (i.e., given the normal of the surface at the input position, the reflection direction of the viewpoint to it). Any direction value that includes information about the viewpoint direction may be considered.

[0045] In each case, the trained function encapsulates the learned quality, as described above. By combining an appropriate BSDF with the network output for each location, a grayscale image can be formed for each component. Alternatively or additionally, the final output may be generated by combining the pixel values ​​for the grayscale image from each component.

[0046] During image rendering, not all images are necessarily subject to ray tracing. Similarly, not all images are necessarily generated using the techniques described above. For example, an NPLM network may be trained on ground truth ray-traced images for a specific object or material with its respective lighting.

[0047] Subsequently, if these objects or materials need to be rendered in real time using ray tracing provided by the trained function described above, the relevant NPLM network is loaded into memory. This relevant NPLM network, combined with the appropriate BSDF, is executed with respect to the relevant surface position and line of sight in the scene to generate contributions to the relevant pixels. Other pixels may be rendered using any other technique (including ray tracing itself).

[0048] Typically, a suitable machine learning system is selected and loaded into memory used by the GPU, based on the same asset identification scheme used when selecting and loading textures for an object or material. Therefore, for example, if an object has an ID "1234" used to access its associated texture, this ID can be associated with the relevant machine learning system. Conversely, if a texture has an ID "5678" associated with an object (for example, if this texture represents a material common to multiple objects), this ID can also be associated with the relevant machine learning system. In this way, entertainment devices can use a similar process to load textures to load machine learning systems. Actual storage and access techniques will differ between textures and machine learning systems. This is especially true if textures are stored using lossy compression (which affects the operation of the compressed machine learning system). Therefore, machine learning systems may be stored without compression, with lossless compression, or with compression that is lossy enough for the compressed machine learning system to function adequately. This can be evaluated by comparing the output error / cost function of the machine learning system until the error reaches an absolute or relative quality threshold (compared to the uncompressed machine learning system) in relation to the increase in loss during compression.

[0049] Refer to Figure 8A. In some embodiments of this disclosure, the machine learning system or NPLM network may be any suitable machine learning system. For example, a single neural network may be trained using position and viewpoint orientation as inputs and generate RGB values ​​for the learned characteristics as outputs.

[0050] However, a particularly preferred network features a separate split architecture.

[0051] As shown in Figure 8A, in a non-limiting embodiment, the network comprises two parts: the first part is a location network and the second part is a direction network.

[0052] Each of these networks may have three or more layers and may use any suitable activation function.

[0053] The position network receives the (x, y, z) position of a point within the aforementioned object / scene as input and outputs an intermediate representation (explained later).

[0054] The directional network receives this intermediate representation and directional input (e.g., normal, or pixel viewpoint, or surface reflection viewpoint direction, or viewpoint-based directional value, etc.) in, for example, (θ, Φ) format or normalized (x, y, z) vector format. The directional network outputs RGB values ​​corresponding to the learned quality with respect to the current pixel position (x, y) (and therefore pixel viewpoint) in the image from the aforementioned virtual camera position to be rendered (in a space shared by the object / scene).

[0055] Therefore, in a non-restrictive embodiment, the location network has three layers. Each layer is as follows: The first layer has three input nodes (e.g., x, y, z positions). The intermediate layer has 128 hidden layers. The final layer has 8 outputs.

[0056] Any suitable activation function may be chosen for the network. In particular, the ReLU (Rectified Linear Unit) function has been evaluated as an effective activation function between layers of location networks. It generalizes well to untrained locations and helps prevent overfitting.

[0057] Similarly, in a less restrictive embodiment, the orientation network comprises four layers. Each layer is as follows: The first layer comprises the outputs of the eight position networks and two or three additional values ​​for orientation assignment to 128 nodes. The second and third layers comprise the inputs to 128 nodes. The three final outputs on the final layer are the corresponding R, G, and B values ​​for the learned quality at the current pixel. These may then be combined with (e.g., multiplied by) the BSDF for that position to obtain the final pixel contribution from the learned function (e.g., diffuse reflection, specular reflection). However, as mentioned above, this is not required during training.

[0058] Regarding directional networks, any suitable activation function may be selected. In particular, the sine function has been evaluated as an effective activation function between layers of a directional network. The change in the behavior of light within an angular range is large and involves many details at angular frequencies. However, this is based on a low-dimensional output (e.g., a normalized x, y, z vector). Therefore, the sine activation function is known to be particularly effective.

[0059] Therefore, especially when the network is divided into two equal parts, different activation functions may be used for each part.

[0060] However, the network is treated not as two independent networks, but as a split architecture network, because the training scheme, in particular, has only one cost function: the error between the RGB values ​​output from the directional network after it has been separated from the appropriate BSDF, and the target values ​​obtained from the corresponding pixels of the ground truth of the ray-traced image.

[0061] This error propagates backward through both networks; that is, there are no separate cost functions for the target values ​​or location networks. Therefore, the output layer of the location network is, in effect, a hidden layer of the combined network. This increases with further input of directional values ​​and represents the change within the layer from the first activation function to a possible second different activation function.

[0062] As mentioned above, the neural network constructs a light model of the illuminated object, material, or scene. In particular, in the non-limiting embodiment described above, the position network effectively classifies the (x, y, z) position into a type of lighting (e.g., bright or dark, and / or other possible categories regarding how the light interacts with each BSDF (e.g., relative reflectivity or diffuse reflectivity)). The intermediate representation output from this part can be thought of as an N-dimensional position in lighting space characterizing the type of light at the input position. This is the illumination position to an N-dimensional position in a different part of the scene if illuminated in the same way. A position network trained on the specular component may have more outputs than one trained on the diffuse component, for example, 13 outputs compared to 8 for the latter. This is a result of considering that the specular component has more diversity with respect to the type of lighting.

[0063] The subsequent directional network models how light behaves when viewed at a given input angle on the surface of the input location, with respect to the illuminated object, material, or scene, in order to generate learned characteristics of the image at that location.

[0064] Therefore, the position and orientation network is trained to predict factors or changes between a BSDF representing the surface properties and the desired rendered image of that surface. Thus, this network can be used instead of ray tracing for the purpose of rendering the surface. Typically (but not required), the network is trained on a single contributing component of the image (e.g., diffuse reflection of the specular component) along with multiple networks used to generate the components required for the final image (or part thereof). However, this is not required (i.e., in principle, the network can be trained on a fully combined image or on two or more contributing components (e.g., all specular components or all diffuse components)).

[0065] [Training] As will be explained in another section, the network is trained using ray-traced images of multiple objects, scenes, or surfaces acquired from multiple viewpoints. This allows the network to learn, in particular, how specular reflections change with location. The viewpoints can be randomly distributed and / or selected from a range of viewpoints available to the user as they move through the rendered environment (this is known as the view volume, i.e., the spatial solid from which the viewpoint originates). Therefore, viewpoints must be included in the training.

[0066] In one embodiment of this disclosure, training data can be generated as follows:

[0067] For any machine learning system, the training data used to train it is key to its performance. Therefore, generating training data that leads to good performance is extremely useful.

[0068] As described herein, the training data for the NPLM system is based on a set of high-quality rendered scene / object / material / surface (hereinafter collectively referred to as "scene") images. Typically, these are decoupled from one or more associated distribution functions (e.g., the aforementioned BSDF, BRDF, etc.). This allows the trained quality dealt with herein to be directly given as the training target. This reduces the computational load for predictive image generation during training and prevents the error function from arising outside the output of the NPLM itself.

[0069] Different NPLMs may handle both field-of-view-dependent and independent light and dark effects (e.g., diffuse reflection, specular reflection, etc.). Therefore, it is usually insufficient for objects in a scene to be merely a single landscape if they have field-of-view-dependent light and dark effects (e.g., diffuse reflection, specular reflection, etc.).

[0070] Therefore, the number and location of training data images may depend not only on the scene's geometry (e.g., when objects are visible within the view volume) but also potentially on the material properties of objects within the scene.

[0071] Accordingly, in one embodiment of the present disclosure, NPLM training data (which takes the form of images of a scene captured from multiple camera viewpoints) can be generated at least in part based on the materials in the scene (material properties such as, for example, photoreactive properties (which are diffuse or specular reflection, but may include other potential material properties, e.g., the complexity of the surface, the structure and / or the presence of narrow or wide spatial frequencies in terms of texture)).

[0072] These images, in particular, are usually generated by third-party high-quality renderers. In this case, access to the internal data is not possible. Therefore, only the final, completed image is available. Consequently, information (or control) regarding specific cast rays or their directions when shading is performed within the image is not available.

[0073] Therefore, for training purposes, it is desirable to be able to generate and use a set of images that can effectively capture the scene's appearance with respect to the entire effective field of view within the view volume.

[0074] Refer to Figure 11. For the purposes described above, in order to render a low-resolution set, a set of camera positions within the view volume is first used in step 1110. The positions may be equidistant or randomly distributed on a sphere around the scene (e.g., as an interactive object visible from any angle), on a hemisphere around the scene (on a virtual ground and not visible from below), or on a ring around the scene (as a ground viewpoint, e.g., as seen from the avatar's first-person viewpoint). Such a ring may be at a fixed height (corresponding to the avatar's viewpoint) or may occupy a certain height range (e.g., a view cylinder surrounding the viewpoint heights when the avatar is crouching and jumping).

[0075] Step 1110 is shown in the example in Figure 11 as a trajectory (ring) of the camera position around the car object.

[0076] The number of camera positions in this initial set may be one or less. However, it is usually three or more, more typically on the order of 10 or 100. For example, if there is one camera per degree of orbit, the total number would be 360. In this example, 200 cameras are used, but this is not limited to that.

[0077] The resolution per degree is low, for example, 128 x 84. An example image is shown in step 1120.

[0078] In step 1130, for each pixel in each image, metadata is associated with the pixel, including the 3D position of the scene surface corresponding to the pixel, the normal of the scene surface corresponding to the pixel, and optionally a material ID (or a similar material surface identifier such as a texture ID or object ID).

[0079] In the first example of the viewpoint selection process, 3D positions of the scene surface are collected using pixels from some (or typically all) of these low-resolution images to determine which positions in the scene are visible from the initial set of camera positions. NPLM is useful when trained on these positions.

[0080] Therefore, selectively, for 3D locations identified as rendered in one or more low-resolution images, a new location in 3D space is calculated as an offset from the location along the surface normal. The distance of this offset from the surface is a variable that can be changed. This new location is a candidate viewpoint for a virtual camera to generate high-quality (e.g., high-resolution ray-traced) renderings.

[0081] However, this could also result in more high-quality ray-traced potential renders for generating training images. This could be computationally intensive and may also involve significant redundancy when used to train a set of NPLMs.

[0082] Therefore, in the first example, it is desirable to filter or discard these candidate viewpoint positions by a method relating to training the NPLM on the scene or by any other advantageous method.

[0083] In particular, it is useful to have more training examples for scene parts that include field-of-view-dependent materials (e.g., specular or glossy) than for field-of-view-independent materials (e.g., diffuse or matte).

[0084] Therefore, one of the two technologies will be adopted.

[0085] In the first technique, step 1140 reviews the corresponding material properties of the surface at each candidate viewpoint corresponding to the normal at the surface location. As mentioned above, in particular, its diffuse or specular reflection reaction, or translucency, etc., may be used.

[0086] In practice, this can be done using a lookup table for material IDs (or values ​​indicating the effect of diffuse or specular reflection on the real surface, e.g., matte or glossy). More specifically (but not limited to), this property can also be represented by a value ranging from 0 (perfectly diffuse) to 1 (specular). This can be treated as input to a probability function. As a result, specular or glossy (view-dependent) materials have a relatively high probability, while diffuse or matte (view-independent) materials have a relatively low probability.

[0087] The probability function is then used to maintain candidate camera positions. Thus, the camera positions relative to specular surfaces are kept to a greater proportion than those relative to diffuse surfaces.

[0088] Conversely, if the convention of values ​​is reversed (for example, if the high and low probabilities are reversed), the probability function can be used to extract candidate camera positions for the same effect.

[0089] In the second technique, alternatively or additionally, in step 1140, the variability of pixel values ​​corresponding to the same 3D position on the scene surface as seen in the low-resolution image can be evaluated to determine the deviation of pixel values ​​for each of the captured 3D positions. In this case, field-invariant (e.g., diffuse or heavily shaded) surface positions will have small deviations (i.e., pixels in similar positions in different low-resolution images), while field-dependent (e.g., specular or glittery) surface positions will have large deviations (i.e., pixels that cover a wider area in different low-resolution images, such as glitter or light reflection). This deviation (or standard deviation) can again be used as input to a probability function. As a result, specular or glittery (i.e., field-dependent) materials will have relatively high probabilities, while diffuse or matte (i.e., field-independent) materials will have relatively low probabilities.

[0090] Therefore, in either case, step 1140 allows us to evaluate the field-of-view dependence of the material's photoreactivity at each 3D position captured within the view volume (this is based on either the material's properties or the variability of the pixels (or potentially both)). This can then be used as input to a stochastic function.

[0091] The probability function is then used in step 1150 to decide whether to keep or discard (eliminate) each candidate viewpoint. That is, viewpoints centered on a field-of-view dependent surface are kept more often than viewpoints centered on a field-of-view independent surface.

[0092] The output range of this probability function can be adjusted to generate almost all the camera views needed for training, based on the original number of possible candidates and the ultimately desired number. Alternatively, the probability function can be applied to a continuous maintenance / elimination round that is repeated as long as the number of camera views falls within a desired numerical threshold.

[0093] In either case, the result is the number of manageable camera viewpoints distributed randomly (but with a variable probability density depending on the material properties of the material (e.g., luminance, etc.) that are immediately centered in front of the camera) within the desired view volume. This is shown in the figure for step s1150 by a constellation of remaining points. In particular, the camera positions may be farther from the object / scene surface than shown in the figure. However, in the figure, the points are shown closer to the surface to clearly represent the distribution.

[0094] The number of manageable camera viewpoints can be chosen based on factors such as the desired performance of the resulting NPLM, the computational load required to generate high-quality ray-traced images, or the training load of the NPLM. For training purposes, the manageable number is (non-limiting) between 10 and 10,000, and typically between 200 and 2,000.

[0095] Finally, in step 1160, the image is rendered with the remaining viewpoint. As shown in Figure 11, these renders are selectively generated using a virtual lens with a wider angle than the initial low-resolution image or the lens used during gameplay.

[0096] This often results in rendering an excessive amount of the scene (i.e., parts not directly visible from the view volume viewpoint). This tends to make the NPLM output more robust. This is particularly noticeable with respect to viewpoints near the edges of the view volume, or when there has been an unexpected expansion of the view volume (e.g., due to clipping in the game or minor design changes).

[0097] The above technique generated candidate camera viewpoints based on the normals of the scene surface captured in the initial low-resolution image, but this is not potentially the only technique.

[0098] One of the potential problems associated with the above technology is that while field-of-view invariant locations within a scene are imaged using a camera oriented along the normal of that location, other images at nearby locations are rendered only from different angles, and these angles are oriented using the normal of the scene surface at those locations. As a result, a relatively large number of images are captured in the field-of-view dependent parts of the scene (and their vicinity), but potentially the images themselves are strongly influenced by the geometry of the scene itself.

[0099] Therefore, returning to the initial low-resolution images, in other examples of viewpoint selection processes, we can consider potential viewpoint positions for each pixel (or at least for each pixel representing a surface in the scene) in each low-resolution image. In the example of the 200 128x84 pixel images above, this yields 1.6 million candidates. These images typically capture multiple examples of a given position on the scene from different angles, independently of the topology of the scene itself. As a result, the training set is potentially more robust.

[0100] Again, the surface material (and / or pixel deviation) derived from the field-of-view dependence of the surface position corresponding to a given pixel in the low-resolution image (and therefore corresponding to a candidate viewpoint) can be used to drive probabilities regarding viewpoint retention or elimination. In this way, the 1.6 million candidate viewpoints can again be narrowed down to a manageable number.

[0101] In this case, because multiple fields of view exist at the same location in the scene, the resulting distribution of camera fields of view may be biased towards the location that is most visible in the scene (as opposed to simply being the most field of view dependent). For example, suppose a certain location in the scene (diffuse reflection) appears in 20 times more images than other locations (specular reflection). In this case, the viewpoints that show the diffuse reflection location are more likely to be discarded because, ultimately, there are more images of the diffuse reflection location than of the bright location.

[0102] Therefore, the probability of maintaining or discarding a viewpoint is normalized based on how many viewpoints are centered at the same location in the scene (even from different angles). This normalization may be whole (thus, in the example above, the probability of maintaining the image of the diffuse reflecting location is reduced 20-fold, and the effect of field count is eliminated). Alternatively, the normalization may be partial. In this case, for example, the probability of maintaining the image of the diffuse reflecting location is reduced only 10-fold, and the effect of field count is greatly reduced, but not completely eliminated. This means that areas that are potentially more visible to the user also receive more training examples, independently of whether they receive more training examples due to being field-dependent (e.g., specular / glare).

[0103] In principle, before generating a high-quality ray-traced render for training purposes, any set of viewpoints (surface normal-based viewpoints and low-resolution image pixel-based viewpoints) may be generated and discarded in order to produce a combined set of viewpoints. In fact, in either case, a subset of the low-resolution image pixel-based viewpoints is likely to be one that effectively matches the normal of at least one visible surface location.

[0104] - Different training techniques The second technique described above addresses the problem of compensating for multiple views of the same location in the scene when selectively discarding available viewpoints. This, in addition to achieving control over training bias, can reduce training time for this second technique by reducing repetition for a particular location in the scene.

[0105] However, alternatively or additionally, training time can be further reduced by the following methods:

[0106] First, as mentioned above, select the initial set of viewpoints within the view volume (or on the surface).

[0107] Next, selectively generate a set of initial low-resolution images for a set of locations within the view volume.

[0108] Next, candidate viewpoints are generated selectively based on the normals of locations in the scene found in the low-resolution image, and / or based on the lines between the pixels in the low-resolution image and the locations represented in the scene.

[0109] Again, these viewpoints can be selectively discarded based on a probability derived from the degree of specular / diffuse reflection at each location in the scene. Further selectively, this probability can be modified where multiple images are centered at each location.

[0110] Therefore, depending on the technique employed, the resulting image will be a series of generated viewpoints. This is either the original distribution selectively used to generate low-resolution images, or the distribution resulting from the generation-selection technique described above.

[0111] In any case, in one embodiment of the present disclosure, once a viewpoint is generated (and selectively ensured not to be discarded), this viewpoint is given (or placed in a sequence) for a ray tracing process to generate high-quality images, selectively in the form of a wide angle.

[0112] Training on the generated images begins as each image is completed. Therefore, there is a parallel process between generating the training images (which takes a certain amount of time because they are ray-traced images) and training on these images (which also takes a certain amount of time). This eliminates the problem of waiting before training begins to prepare complete images.

[0113] Selectively, the selection of viewpoints from these candidates may be random at the time the viewpoints are generated (or when it is decided whether to discard the generated viewpoints). As a result, the actual generation sequence of the ray-traced image will also be random within the final set of viewpoints used.

[0114] This reduces the risk of the NPLM initially overfitting in a particular section of the scene. This means that, for example, if training is omitted due to time constraints, the NPLM will still be shown across different sets of the scene's field of view.

[0115] In other training techniques, if control over the ray tracing application is available, it is also possible to selectively render only one subset of pixels from an image that requires rendering from a given viewpoint. Parts of a scene from an image that has been rendered a predetermined number of times may exist within other images in the training set, whether based on the original set of viewpoints or on viewpoints that were not discarded. For example, if a location in a scene has already been rendered more than a threshold number of times, it can be skipped from the current rendering because there are already enough training examples. Unrendered portions of an image may be tagged with reserved pixel values ​​that act as mask values. Training can then be performed using only target values ​​for input position, orientation information, and unmasked pixel positions. This significantly reduces redundancy and associated computational load in the training set when ray tracing images and training NPLMs.

[0116] You may selectively apply exceptions. For example, pixels near the center of an image have already been rendered because central pixels are typically related to their position in the scene that was selected (or not discarded). This is a function of their surface properties. These are typically pixels in parts of the image that are not central. These tend to capture unintended (or unwanted) iterations in the scene.

[0117] - Training using variable lighting conditions The aforementioned variable lighting techniques are typically performed using images of multiple different ray-traced scenes, objects, or materials (collectively referred to as "scenes" unless otherwise specified) for given lighting conditions. It is usually assumed that the light source is fixed within the scene (for example, due to the position of a window and / or due to something that provides a static lighting map, often called a "cyclorama").

[0118] However, in some environments, the lighting of a scene may vary considerably. For example, some games model day / night cycles and / or weather. Thus, the position of the sun (or moon) may change within the virtual day (or night) in the game. Similarly, depending on the weather, the primary light source may be something close to a point source like the sun, or it may be a diffuse source like clouds providing backlighting.

[0119] An NPLM trained to model the lighting of a scene according to one lighting condition does not need to be generalized until it can adequately simulate other conditions.

[0120] Accordingly, in embodiments of the present disclosure, the NPLM is trained to model the illumination of the scene based on a ground truth image that has already been ray-traced using multiple lighting conditions (using one of the techniques described above).

[0121] Typically, these lighting conditions involve adjusting one or more lighting parameters within a given range.

[0122] Lighting parameters may include the following: -Position. For example, the progression of the sun within a given range. This may be achieved by rotating the horizon containing the sun in various ray-traced renders. Another example includes the motion paths of non-player characters or other movable light sources. - Direction. Directional light sources that can be modeled in terms of the range of change in direction. For example, a lighthouse or MPC / movable light path. Alternatively or additionally, the beam width or beam angle of the light spread may be specified (for example, a typical torch or spotlight has a narrower beam angle than a desk lamp). Alternatively, these (or other) predictable light sources (e.g., player-based movable light sources such as torches) can be implemented using fully torch-illuminated and un-torch-illuminated NPLMs for different parts of the scene, depending on where the torch light falls. - Color or color temperature. For example, when the sun gradually turns red during sunset. -brightness. - Diffuse reflection. For example, replacing a rendered horizon with a continuous amount of clouds or fog in front of a point light source (such as the sun, moon, or building lights).

[0123] A single NPLM may be trained simultaneously for changes in one or more of these parameters. For example, an NPLM may be trained for sunrise (which includes a given range of position and associated color changes). Another NPLM may be trained for the midday (or part thereof) progression of the sun (which is within a given range of position and does not involve changes in color temperature). Further NPLMs may be trained for sunset (which again includes a given range of position and associated color changes).

[0124] A more complex training environment may require changes in the NPLM architecture to form the network (for example, so that the NPLM memory area becomes maximum when it reaches a threshold size).

[0125] NPLM may also require one or more additional inputs during training that represent parameter changes or changes in the associated training data.

[0126] For example, proxy values ​​such as lighting position and / or a certain time of day (if they also correlate with color temperature, for instance) may be explicit inputs. Typically, this may be an input to a position network that helps characterize the relative positions of scene elements in relation to light. Alternatively, it may be an input to a direction network that helps characterize the relationship between the light source and the viewpoint. Or, in practice, it may be an input to both the position network and the direction network. Lighting position may relate to the position of a point light source, the center of a diffuse light source, the center of a cyclorama, or an offset position. The input may be linear, polar, or other preferred form (e.g., polar is more preferred with respect to a cyclorama). Thus, position changes may typically follow the paths of NPCs (e.g., traffic), the movement of objects in the game (e.g., a moving table light), or environmental changes (e.g., the sun or moon).

[0127] Similarly, direction (and / or selectively, the beam angle of diffuse reflection) may be an input to one or both halves of the NPLM. Typical direction changes include, for example, spotlights, sentries, or their potential movement, in combination with traffic. The value of the diffuse reflection angle may be related to these. On the other hand, changes in the diffuse reflection angle include, for example, the opening of a door in front of the light source.

[0128] The color or color temperature may be an input, for example, a scalar value (e.g., temperature) or an RGB or other color representation. Typically, the color representation relates to the same color space as the predicted image (i.e., image pixels generated by combining the NPLM output with a specific BSDF, BRDF, BTDF, etc.). The color or color temperature may also be an input to a location network (again, useful for classifying scene locations embedded in the intermediate representation of the location network output layer) and / or an input to a direction network. This may, more simply, affect the color-related output of the NPLM. Typical color changes may relate to sunrise or sunset, for example, or to more dramatic ones (e.g., alarm activation and associated conversion to a predetermined red illumination).

[0129] Luminance, in a similar manner to color temperature, may be input as a scalar value to one or both of the NPLM halves. Again, a typical change in luminance may be, for example, related to sunrise or sunset.

[0130] The diffuse reflection of a light source (e.g., direction) may be an input to a positional network that helps classify surface points, and / or an input to a directional network that helps determine the relationship between the light source and the viewpoint. Diffuse light sources may include, for example, clouds or fog, or urban / office environments with diffuse ceiling lighting.

[0131] Some or all of these parameters may correlate with time. In this case, alternative or additional time-based parameters may be used. This was discussed above in relation to position, but may also apply to color, brightness, or any other parameter. For example, if the scene is illuminated by sunlight, position, color, and brightness may all correlate with time.

[0132] By adding one or more of these inputs to one of the current first layers for location and / or the orientation network, only one weight is allowed between the inputs of the first layer and the nodes in each network.

[0133] Refer to Figure 8B. To enable more complex trained responses to these additional inputs, further layers may be selectively provided between the current first layer and the inputs to each half of two or more NPLMs. For example, instead of simply providing the position of light and the current surface point position as parallel inputs, further layers may be provided. In this case, any significant combination of these values ​​can be trained as a preliminary step.

[0134] Therefore, for example, if all of the above inputs are included in addition to the input to the original surface point position in the position network, then we can provide directional and diffuse reflection inputs to a partial layer, luminance and color inputs to a parallel partial layer, and additional layers that are fully (or partially) connected to the optics position and surface point position in the parallel partial layer. These parallel layers are then fully connected to the first full layer of the network. A similar scheme can be used for directional networks.

[0135] Figure 8B shows a partial layer relating to the original scene position and lighting position, and a parallel partial layer relating to the color and brightness of the light. These are included in the position network as an unrestricted example.

[0136] To generate a training set for variable lighting conditions, one of the aforementioned techniques can be simply repeated to generate a training set for multiple lighting variations (e.g., consecutive light positions).

[0137] Alternatively, a single set of positions can be obtained for the complete render described above. Then, by selecting a random parameter value within a given range, or a set of parameter values ​​that span that range (one or more of these parameters will change), N versions of the set are generated for each position. If probabilistic techniques are used to selectively discard candidate positions, the probability of retaining them will be either higher (to increase the number of selected points and enrich the training set for more demanding training conditions) or lower (up to N times lower) (to reduce the computational load caused by rendering variations for the same field of view). For a particular illuminated scene, whether this probability increases, decreases, or remains the same reflects a trade-off between computational load and time (on the one hand) and the quality of the final NPLM output (on the other hand).

[0138] Alternatively, the probability of maintaining a viewpoint may be up to N times higher, and for each resulting viewpoint, only one version of the parameter value may be selected (whether random or continuous). In this case, instead of one viewpoint having multiple parameter values, there are separate parameter values ​​for each viewpoint.

[0139] These techniques overlap. For example, generating training sets for multiple lighting variations (which could be different constellations of viewpoints for each training set) can, collectively, result in a set of different parameter values ​​at the same location and a single parameter value at a single point.

[0140] Typically, there is no need to change how the distribution of viewpoints is generated, as this relates to the viewer's position, not to the light itself. However, for highly directional light, it may be useful to selectively bias the probability of retaining viewpoints as a function of the angular distance of the viewpoint from the reflection angle with respect to the directional axis of the light (i.e., the closer the viewer is to the line of reflection to the light, the more viewpoints will be retained).

[0141] As mentioned above, NPLM may be implemented using any suitable machine learning system. Therefore, although the split network architecture described herein is preferred, in principle, an NPLM with a single neural network (or three or more networks, or other machine learning systems, potentially of different types) may be used. In this case, one or more of the inputs described herein are provided.

[0142] Different technologies Each NPLM responds to the scene surface position and viewpoint / spectral orientation inputs (these, along with other additional inputs, are described herein) to produce a learned quality for a single pixel.

[0143] More typically, when the final output is required, contributions from multiple NPLMs (e.g., diffuse reflection, specular reflection, translucent, and other scene components) are combined.

[0144] As a result, in principle, the contributions from multiple NPLMs may be combined with the contributions from NPLMs trained for each lighting condition. For example, one NPLM may be trained for the sun or cyclorama set at 1 PM, and another NPLM may be trained for the sun / cyclorama at 2 PM. The contributions for each output may then be combined. For example, at 1 PM the 1 PM network might be 100%, at 1:15 PM 75% the 1 PM network and 25% the 2 PM network, at 1:30 PM 50 / 50, at 1:45 PM 25 / 75, and at 2:00 PM 100% the 2 PM network.

[0145] With respect to positional changes, NPLMs for specular contributions may be selectively trained at finer intervals than NPLMs for diffuse contributions, because the effect of lighting changes is more pronounced in specular contributions. For example, specular NPLMs may be generated at 1-hour intervals or at every 10% point along the parameter range, while diffuse NPLMs may be generated every 2 or 3 hours or at every 25% point along the parameter range.

[0146] Therefore, contributions from different NPLMs trained at different points on the parameter range may be blended to approximate a desired point on that parameter range. The sampling of these parameter ranges by each NPLM may be selectively sparser for contributions that are less dependent on the field of view than specular contributions (e.g., diffuse contributions).

[0147] The above example of time is for a one-dimensional parameter and therefore only requires mixing two NPLMs trained with a combined parameter value of the desired parameter value. On the other hand, this technique may be extended to a two-dimensional parameter using three or more NPLMs (e.g., triangles or other polygons in parameter space) (whose weighted contributions correspond to triangulations on the desired parameter value), and in fact may be extended to a three-dimensional parameter using four or more NPLMs (e.g., tetrahedrons or other volumes in parameter space) (whose weighted contributions again correspond to triangulations on the desired parameter value).

[0148] Network Configuration As mentioned above, the positional network (i.e., the first part of the split architecture disclosed herein) may have a different number of outputs depending on whether it is trained on specular reflection type image components or diffuse reflection type image components. This is a concrete example of a more general technique.

[0149] In general, NPLM performance may vary depending on the complexity of the modeling task being performed. For example, this performance may be increased or decreased from the conceptual default setting. Typically, the network architecture may be modified to change this performance.

[0150] In the first embodiment, performance may be modified depending on the size of the NPLM (e.g., the number of layers, the size of the layers, and / or the arrangement of layers among the NPLMs, etc.). In this case, the architecture of the NPLM is modified to change the performance of the NPLM.

[0151] Therefore, selectively, the size of the NPLM can be changed depending on the type of contributing component that the NPLM models (diffuse reflection, specular reflection, translucency / transmission, etc.).

[0152] Particularly preferably, the size of the position network is larger than that of the diffuse component with respect to specular or translucent / transmissive components, while all others may be the same, because the intrinsic response of lighting within the specular component is more diverse. Similarly, preferably, the size of the position network is larger than that of the diffuse component with respect to specular or translucent / transmissive components, while all others may be the same, because it includes combinations of partial reflection, transmission, and internal reflection.

[0153] Similarly, the size of the position network may be larger if changes to one or more lighting conditions are to be trained (for example, changes to the position, orientation, and / or angular width of the lighting), because as the number or complexity of lighting conditions increases with respect to a given scene surface position, these need to be modeled.

[0154] The size may change by altering the number of hidden layers or the number of nodes within those layers. Similarly, the size may change depending on the number of output layers (e.g., the number of output layers in a location network, which is also the hidden layer or interface / intermediate layer between the location and orientation networks in the overall NPLM network). Generally, increasing the number of layers increases spatial distortion. The network can apply this to the input data to classify or filter different types of information. On the other hand, generally, increasing the number of nodes in a layer increases the number of specific conditions in the training set. The network can model this, resulting in improved fidelity. On the other hand, increasing the number of output nodes (where they are not selected and mapped to a specific format, as in the output of a location network) can improve discriminative ability by performing less rigorous dimensionality reduction in the internal representation of the dataset, and further by using network operations on subsequent output node values.

[0155] Selectively or additionally, the size of the directional network may vary depending on the type of contributing component that NPLM models (e.g., diffuse reflection, specular reflection, or translucent / transmissive, etc.).

[0156] As mentioned above, the input layer of the directional network may vary in size to accommodate the higher-dimensional output of the position network of the entire NPLM split architecture network.

[0157] Similarly, the number and / or size of layers may vary to produce the same effects as described for location networks (i.e., improved discriminability and model fidelity).

[0158] Similar to the position network, the size of the directional network is preferably larger than that of the diffuse component with respect to specular or translucent / transmissive components, while all other components may be the same, because the inherent response of lighting within the specular component is more diverse. Similarly, the size of the directional network is preferably larger than that of the diffuse component with respect to specular or translucent / transmissive components, while all other components may be the same, because it includes combinations of partial reflection, transmission, and internal reflection. Thus, like the position network, its architecture may be varied to change its performance.

[0159] Again, similarly, the size of the directional network may be larger if changes to one or more lighting conditions are to be trained (for example, changes to the position, direction, and / or angular width of the lighting), because as the number or complexity of lighting conditions increases with respect to a given scene surface position, these need to be modeled.

[0160] Therefore, NPLMs (e.g., location networks, orientation networks, or both) may have their performance modified (e.g., changes to the architecture, such as an increase in the number of layers, internal nodes, or input or output dimensions) to improve discriminative performance (e.g., by adding more hidden layers or output dimensions) and / or to improve model fidelity (e.g., by adding more nodes within the hidden layers). For example, the diffuse reflection contribution is usually less required than the specular reflection contribution.

[0161] If necessary (for example, in the case of diffuse reflection components), NPLM may, rather than improve performance from the conceptual standard or default settings for NPLM, actually decrease performance (for example, by the reverse of the performance improvement steps described above). The advantages of this are typically reduced memory usage and computational costs.

[0162] In addition to the type of reflective properties of the material modeled by different contributing channels, the performance of NPLM may be increased or decreased depending on other factors relating to the complexity of the lighting model / rendering process, either alternatively or additionally.

[0163] For example, a diffuse reflecting light source (such as a cyclorama) may be less complex than a point light source because the lighting hitting an object / scene does not change much in space / angle. Conversely, a cyclorama with large spatial variations (e.g., sunset) will be more complex. The complexity of a light source is based on its spatial and color variability (e.g., based on the integral of the 2D Fourier transform of the light space where no object / scene exists, typically ignoring the DC component). In this case, a uniform cyclorama will have an integral value close to zero. On the other hand, a complex cyclorama (such as an urban space or sunset) will have a larger integral value. The performance of the NPLM (e.g., size) may be set based on an analysis of such light source complexity (e.g., based on an empirical analysis of performance).

[0164] Similarly, moving, dynamic, or recognizable light may require increased complexity for NPLMs because they create changes in lighting conditions. In this case, inputs to the NPLM may include (x, y, z) object position and inputs for a particular part of the object / scene being rendered, with respect to lighting state inputs or output pixels. For example, for a model of a scene where the sun crosses the sky, inputs for the time of day (which would correlate to the sun's position) may be included. Other inputs identifying the current state of a light source may include (x, y, z) position, radius (r) or similar inputs for the size of one or more lights, and / or RGB inputs for the (primary) color of the lights, etc. Training data (based on ray-traced ground truth) may include examples of these changing conditions. More generally, if the NPLM is trained to model dynamic aspects of an environment, the training data would include a representative number of suitable examples.

[0165] In the case of the sun, the trajectory throughout the day (e.g., dawn, morning, midday, evening, sunset) would need to be modeled using multiple consecutive NPLMs. This is, for example, to prevent the memory and computational costs of the NPLMs from exceeding a desirable maximum.

[0166] Similarly, if moving, dynamic, or recognizable objects in the scene need to be rendered using NPLM, this requires increased complexity of the NPLM (optionally, NPLM may be used only for rendering static scene components and / or for rendering a portion of the scene that is position-dependent). Therefore, again, in this case, the input may include, for example, data on the position and / or orientation of objects.

[0167] Alternatively or additionally, other factors may simplify the NPLM modeling and consequently reduce its performance (or improve the model's fidelity without changing other conditions). For example, if the rendered scene includes a fixed path (e.g., a race track within a crash barrier), training from viewpoints inaccessible to the user can be reduced or avoided overall. Similarly, if the rendered scene includes limited or preferred viewpoints (e.g., again, a race track where most of the view is in the direction of driving), training for different viewpoints can reflect the importance of these viewpoints in line with the end user case.

[0168] Similarly, for parts of a scene that are less important to the user's field of view, such as the background or being far from the game's focus, the performance of the NPLM may be relatively reduced. For example, different NPLMs may be trained for different draw distances to objects or textures. In this case, performance (e.g., size) may be reduced where the draw distance / level of detail (LOD) differs.

[0169] Alternatively or additionally, an NPLM can be trained on a specific scene, object, or texture. Therefore, the performance of an NPLM depends on the complexity of what the luminance is representing. Larger or more complex scenes may require larger NPLMs (and / or multiple NPLMs handling each part depending on the size of the scene and the resulting NPLMs). Similarly, complex objects (e.g., a car) benefit more strongly from a high-performing NPLM than simpler objects (e.g., a sphere). One way to assess the complexity of a scene or object is to count the number of polygons. More polygons suggest a more complex scene. As an improvement, the degree of change in face angles between polygons may be used to indicate complexity. For example, a sphere with the same number of polygons as the car model in the figure has much less angular variation than the car itself. This indicates that the car is structurally more complex. Combining the number of polygons with angular variation / distortion can provide a good proxy for the complexity of a scene / object (the NPLM models the luminance of these objects).

[0170] Similarly, complex materials (such as skin or fur) benefit more strongly from high-performance NPLMs (and / or contributions from multiple NPLMs) than simpler materials (such as metals). Again, complex textures (e.g., broad spatial spectra) benefit more strongly from high-performance NPLMs than narrower or denser spatial spectra.

[0171] Performance has been expressed using size (number of inputs / outputs, number of layers, number of nodes, etc.). However, alternatively or additionally, performance also depends on the choice of activation function between nodes on different layers of NPLM. As mentioned earlier, the preferred activation function for location networks is the ReLU function. On the other hand, the preferred activation function for direction networks is the sine function. However, other functions may be chosen to model different scenarios.

[0172] For example, the NPLM's performance may be set to be capped when the NPLM's memory capacity reaches a threshold size. This threshold size may be equal to the operating unit size of memory, which is, for example, a memory page or a portion or group of memory pages. Typically, it is chosen for accessing and loading textures related to a scene / object / material. The threshold size may be equal to the size of the texture or the size of the mipmap used by the GPU and / or game to load graphical image data onto the GPU.

[0173] If the complexity of the NPLM exceeds this threshold, the modeling task must be simplified, shared between NPLMs, or the accuracy of the results must be reduced.

[0174] In summary, (focusing on network configuration and selection) an image rendering method may include the step of selecting a first trained machine learning model from among several machine learning models. This machine learning model is trained to generate data that contributes to the rendering of at least a portion of the image. Thus, for example, the contributing data may be, depending on the complexity of the lighting and / or materials, textures and / or other surface properties (that are to be modeled), a specific component of the image pixels (e.g., diffuse or specular reflectance contributing component), or a complete RGB pixel (e.g., modeling all reflective surfaces at once).

[0175] In this method, the first trained machine learning model has architecture-based learning performance (which responds to a first aspect of the virtual environment in which it is trained to generate data). Thus, for example, the architectural aspect related to learning performance may be the size of all or part of the NPLM (e.g., the number of layers or nodes) and / or related to the connectivity characteristics between nodes of different layers (e.g., the degree of connectivity of the types of activation functions used).

[0176] In such a method, the second step may include using the first trained machine learning model to generate data that contributes to rendering at least a portion of the image. This point is also discussed elsewhere in this specification. For example, an individual run of NPLM may generate data that is used together with data from other NPLMs to generate RGB values ​​for pixels in an image. Alternatively, an individual run of NPLM may generate data to generate RGB values ​​for pixels in the image itself (this is done after processing such as combining with a distribution function, for example).

[0177] Network Selection The network is trained during gameplay or application development. Developers may choose when and where NPLM rendering is useful. For example, NPLM rendering may be used only in scenes known to cause frame rates below a certain quality threshold. In such cases, the network is trained on such scenes (or parts thereof) and used when such scenes appear.

[0178] In another case, developers may choose to use NPLM-based rendering for certain objects or materials. In this case, the network is trained and used when these objects or materials are recognized within the scene to be rendered.

[0179] In another case, developers may choose to use NPLM-based rendering for certain lighting conditions or combinations thereof. In this case, the network is trained and used when these conditions are found within the scene to be rendered.

[0180] Similarly, in other cases, developers may choose to use NPLM-based rendering for specific draw distances (z-distances), or for objects that are angularly / distant from the image center or the user's field of view, or for certain lighting conditions. In this case, the network is trained and used in these environments.

[0181] Similarly, any suitable combination of these criteria may be selected for training and use.

[0182] On the other hand, as mentioned above, for various reasons, multiple NPLMs may exist related to a scene during system use. For example, multiple NPLMs may exist to model a larger scene (so that each part is modeled well enough by the NPLM within the threshold size and / or threshold quality range of the image reconstruction). Similarly, multiple NPLMs may exist due to changing lighting conditions, level of detail / draw distance, etc.

[0183] Appropriate NPLM(s) for the environment may be selected and searched for GPU-accessible memory and may be executed for at least partial rendering. Strategies applied to prefetching and caching textures and other graphical assets may also be applied to NPLMs.

[0184] Failure Mode The NPLMs described herein, when combined with each distribution function (typically then combined with contributions from other NPLMs, e.g., diffuse and specular components), generate values ​​that produce pixel values ​​for an image. These pixel values ​​are computationally useful approximations of the ray-traced pixel values ​​for that image.

[0185] Therefore, it is generally desirable to use one or more NPLMs as needed to render scenes / objects / materials, provided that they are computationally useful and the resulting approximations are sufficiently good.

[0186] However, there may be cases where one or both of these conditions are not met.

[0187] In particular, the distribution functions (BSDF, BRDF, and BTDF) used herein typically assume non-zero light scattering. In this case, specular scattering is more directionally dependent than diffuse scattering (perfect or uniform scattering may be assumed).

[0188] Furthermore, this scattering acts as a visual mask of approximation errors within the model in the aggregate pixel results of a continuous NPLM run. Thus, particularly under diffuse reflection conditions (and indeed under specular reflection conditions as well), the error tolerance due to the properties of the modeled material is increased.

[0189] However, on reflective surfaces (which may include glass, in which case typically 8% is mirror and 92% is the transport material), scattering occurs little to no. Consequently, the errors in the resulting image become more pronounced, both for the user and in terms of the error values ​​generated during training. This makes training more difficult.

[0190] As mentioned above, NPLMs can be selectively made larger to model more challenging lighting conditions (for example, by using larger NPLMs for specular reflection conditions compared to diffuse reflection conditions in the same scene). This principle can be extended to potentially using larger NPLMs for mirror-like reflective surfaces to provide sufficient resources to model the lighting response.

[0191] Therefore, even if a given NPLM is suitable for diffuse and specular reflection conditions, this Sometimes, there aren't enough resources to adequately model the lighting conditions for a specular reflective surface (i.e., with an acceptable level of accuracy). On the other hand, a sufficiently large NPLM may require a long training time to be trained to the desired accuracy, and may use more memory and computational resources during operation.

[0192] Therefore, depending on the preferred maximum size and computational limits of the NPLMs used, there exist points on the line between multiple diffuse reflecting surfaces and pure reflecting surfaces as follows: At these points, the diffuse reflecting scattering becomes sufficiently small (e.g., sufficiently narrow or sufficiently directional), and the approximation provided by the NPLMs is no longer sufficiently good.

[0193] Similarly, depending on the desired accuracy of the approximation, there lies a point where the NPLM required to achieve that accuracy exceeds the maximum memory or computational load allocation (if such an allocation exists).

[0194] In contrast, specular surfaces are relatively well-suited for accurate rendering using ray tracing because they exhibit little to no scattering. Therefore, rendering ray-traced specular surfaces is less computationally expensive than other ray tracing methods.

[0195] Therefore, if the reflectivity at a scene surface location reaches a threshold (a perfect mirror has extreme reflectivity and thus exceeds the threshold), it is preferable to use ray tracing (or selectively other rendering techniques, such as existing non-ray tracing rendering techniques) instead of NPLMs to generate pixels at that location. As mentioned above, this depends on the insufficient precision of NPLMs, the relatively high cost of NPLM(s), or both.

[0196] Therefore, if this threshold is exceeded, it is desirable to use ray tracing or any other suitable approximation technique as an alternative "failure mode" for the corresponding pixel.

[0197] Returning to the example of the car in Figure 2, the virtual chrome on the car's front grille is more mirror-like than the body, and the body is more reflective than the wheels, wheel rims, and seats.

[0198] Therefore, for explanatory purposes, we can say the following: the chrome on a car's front grille is extremely reflective, and attempting to model this material with NPLM will not meet the accuracy standards set by the developer, nor will it meet the memory or computational allocation requirements if the accuracy standards are met. Thus, a chrome radiator is a candidate for failover to conventional ray tracing or other alternative techniques.

[0199] Typically, reflectivity is constant for a given material in the scene (chrome in this example). Therefore, the first option is to use the material ID or a value associated with such an ID to indicate whether to use NPLM or an alternative technique (such as ray tracing on the pixels corresponding to that material).

[0200] This material ID may be the same as the aforementioned material ID. This value indicates the degree to which the surface of the material is diffuse or specular. In this example, a conceptual range between 0 (fully diffuse) and 1 (specular) is implied. Thus, in this case, the material ID is tied to a predetermined threshold, and as a non-restrictive example, 0.95 could be considered a candidate for this failover technique. In this case, if NPLMs are trained on a material basis, selectively, some NPLMs may not be trained at all for that material. Alternatively, if NPLMs are trained on an object or scene containing multiple materials (e.g., a car), values ​​related to the material ID and surface location may be referenced to determine whether to use NPLMs or an alternative technique. For example, in this case, NPLMs may be used for everything except the chrome grille (and possibly the steering wheel as well).

[0201] Instead of having associated values, the material ID can also include a flag. For example, the least significant or most significant bit (or any predetermined bit) in the ID number may be used to indicate whether the material should use NPLM and therefore whether it should fail over to an alternative rendering technique.

[0202] We have discussed material IDs up to this point. On the other hand, it will be understood by those skilled in the art that other corresponding IDs can also be consistently associated with specific surface locations. Such IDs may include cluster IDs, object IDs, or IDs specific to the use or non-use of machine learning systems (e.g., NPLMs for rendering). Therefore, alternatively or additionally, reflectance or flags may be associated with such alternative IDs.

[0203] However, some of the techniques described herein do not require material IDs or other corresponding IDs. Therefore, some implementations do not need to use (or attempt to use) such IDs.

[0204] Therefore, alternative or additional mechanisms may be considered to determine whether to use NPLMs or alternative techniques to render a given surface location within a scene.

[0205] First, a distribution function (e.g., BSDF, BRDF, BTDF, or other distribution functions) associated with the surface location can be used to represent the nature of a specular surface location. Therefore, by using a distribution function that represents the degree of the specular reflection threshold (or one associated with a scene element that has such a degree of specular reflection threshold), it is possible to identify that a particular surface location should not be rendered using NPLM, but rather another technique should be used.

[0206] Similarly, the convergence rate of the NPLM during training and / or the performance of its final loss / error function are considered to represent the accuracy of the result. If the error function or loss function for the NPLM output associated with a given surface location converges to a threshold beyond a threshold number of training cycles, or if the error function or loss function for a given surface location (alternatively or additionally) does not reach the lower bound of the threshold, then it is clear that NPLM should not be used for that surface location.

[0207] Therefore, depending on the training behavior of NPLM, an alternative technique is used for a given surface location (or a material with a material ID associated with that surface location) instead of rendering it using NPLM. In this case, the flag or value is embedded (or associated) with the material ID, the associated distribution function, or a value characterizing the surface or its location.

[0208] Alternatively or additionally, a sample subset of pixels in the image (or a portion thereof) may be test-rendered using multiple techniques. For example, 0.1%, 1%, or 10% of the pixels in the image may be rendered using ray tracing and one or more NPLMs. These samples may be random or in a regular pattern (e.g., 3x3, 4x4, 8x8, or 16x16 squares) to simplify tracking.

[0209] The computational load of the candidate technologies can also be compared. If NPLM uses fewer resources, it will be used for the remainder (or part thereof) of the evaluated image. This portion may match any (or pre-selected) text portion within the image, or an object or material within the image.

[0210] Similarly, the accuracy of candidate technologies can be compared. Assuming ray tracing is accurate, if the NPLM result falls within a threshold of the difference compared to the ray-traced result, this is considered acceptable (typically in conjunction with confirming that it uses fewer computational resources).

[0211] In any case, one of the tested sample pixels (typically a sample pixel generated using the selected technique) can contribute to the final image in a way that is consistent with its neighbors. Therefore, even if, for example, a ray-traced pixel were more accurate, if NPLM was chosen, then NPLM can be used.

[0212] The remaining pixels are then rendered using the selected technique.

[0213] Selectively, the above techniques can be combined in a suitable manner. For example, a material ID or related value, or a distribution function, may be used to determine which pixels or regions of an image should be sampled for testing between NPLM and alternative techniques. Conversely, such tests (e.g., tests during development rather than during end-user runtime, or the training behavior of NPLM) can also be used to identify materials or distribution functions for which NPLM should not be used.

[0214] Therefore, by using any suitable combination of the above techniques, all or part of a given image can be failed over from NPLM to an alternative rendering technique (e.g., ray tracing) in the following cases: when it is determined that failing over to an alternative technique is preferable in terms of the precision or computational or memory cost of NPLM(s); optionally, when there is a signal from a material ID or a value or flag associated with a distribution function related to a given surface position, and / or when it responds to a test rendering of a subset of pixels.

[0215] summary Refer to Figure 9. To summarize an embodiment of the present disclosure, an image rendering method for rendering pixels at a viewpoint, the method comprising the following steps with respect to a first element of a virtual scene having a predetermined surface at a location in the scene:

[0216] In the first step 910, a viewpoint-based position and orientation are given to a machine learning system. This machine learning system is pre-trained to predict a factor that, when combined with a distribution function characterizing the interaction between light and a given surface, generates a pixel value corresponding to a first element illuminated at that position in a virtual scene.

[0217] In the second step 920, the factors predicted by the machine learning system are combined into the distribution function. This generates pixel values ​​corresponding to the first element illuminated at that location in the virtual scene, as described above.

[0218] In the third step 930, the pixel values ​​are included in the rendered image for display. The image is then output to the display through the A / V port (90).

[0219] Those skilled in the art will understand that one or more modifications of the above methods are possible within the scope of this disclosure. These modifications correspond to the operation of various embodiments of the methods and / or apparatus described herein and in the claims, and include, but are not limited to, the following: - For each of the multiple contributing components of an image (e.g., diffuse reflection, specular reflection, coating, etc.), a separate machine learning system is trained. For each of the multiple contributing components of an image, a separate distribution function is used. The generated pixel values ​​are combined to produce the pixel values ​​contained in the rendered image for display. -The distribution functions are selected from a list consisting of the bidirectional scattering distribution function, the bidirectional reflection distribution function, and the bidirectional transmission distribution function. - A machine learning system is a neural network. The input to the first part of the neural network includes position. The input to the second part of the neural network includes the output and direction of the first part. -In this example, the activation function for the first part and the activation function for the second part are different. -In this example, the activation function for the first part is ReLU, and the activation function for the second part is the sine function. -In this example, the cost function of the neural network is separated from the distribution function based on the difference between the output of the second part and the pixel values ​​obtained from the ray-traced version. -In this example, the cost function of the neural network propagates backward through both the second and first parts during training. -In this example, the neural network is a fully connected network. - The cost function of the machine learning system is separated from the distribution function based on the difference between the output of the machine learning system and the pixel values ​​obtained from the ray-traced version. - The machine learning system is selected based on the same asset identification scheme used to select and load textures for the first element of the scene, and is loaded into memory used by the image processing unit.

[0220] Next, refer to Figure 10. To summarize another embodiment of the present disclosure, an image rendering method for rendering pixels from a certain viewpoint (in particular focusing on whether to use a fallback technique), the method comprising the following steps:

[0221] With respect to a first element of a virtual scene having a predetermined surface at a location within the scene, the first step s1010 evaluates whether to render the pixels corresponding to the first element using a machine learning system (e.g., NPLM) trained to output values ​​representing the lighting of a given surface at the location, or to render the pixels corresponding to the first element using an alternative rendering technique.

[0222] Next, the second step s1020 renders the pixels using the machine learning system or alternative technique selected in the evaluation step.

[0223] Those skilled in the art will again understand that one or more modifications of the above methods are possible within the scope of this disclosure. These modifications correspond to the operation of various embodiments of the methods and / or apparatus described herein and in the claims, and include, but are not limited to, the following: - The rendering step combines the output of the machine learning system with a distribution function that characterizes the interaction between light and the given surface in order to generate pixel values ​​corresponding to the first element of the virtual scene illuminated at the position. -In this case, the rendering step is selectively performed. The steps include using each machine learning system that has been trained on one of the multiple contributing components of the aforementioned image, The steps include: using the respective distribution function for each of the multiple contributing components of the aforementioned image; For display purposes, the process involves combining the generated pixel values ​​with each other in order to generate the final combined pixel values ​​and include them in the rendered image, Includes. - An alternative rendering technique is ray tracing. -The evaluation steps are: To render the first element, we evaluate whether the computational cost of using the machine learning system is lower than the computational cost of using the alternative rendering technique. If the value is low, the step includes selecting the machine learning system. -The evaluation steps are: The accuracy of the image of the machine learning system for rendering the first element is evaluated. The process includes the step of selecting the machine learning system if the accuracy satisfies a predetermined threshold. -In either of the two cases above, the step of selective evaluation is: For comparison, the procedure includes performing test renderings of a subset of pixels in a rendered image using both the machine learning system and the alternative rendering technique. -In either of the two cases above, the step of selective evaluation is: The first element includes the step of referring to a given criterion, The aforementioned standards are, i. With respect to the first element, the material type when the machine learning system falls below a predetermined performance threshold, ii. With respect to the first element, the ID when the machine learning system falls below a predetermined performance threshold, iii. With respect to the first element, the distribution function when the machine learning system falls below a predetermined performance threshold, iv. Training performance of the machine learning system, v. Comparison of rendering results of pixels obtained using the machine learning system and pixels obtained using the alternative rendering technique. Based on one or more of the following. -The evaluation steps are: The virtual material corresponding to the first element is evaluated to determine whether it is of a predetermined type that uses the alternative rendering technique. -A virtual material has a material ID, The material ID provides a value or flag indicating whether the virtual material is of a predetermined type that uses the aforementioned alternative rendering technique. -The evaluation steps are: The distribution function characterizing the interaction between light and the given surface is evaluated to determine whether it provides an indicator that the first element is predetermined to be rendered using the alternative rendering technique. -The evaluation steps are: The error value or convergence performance of the machine learning system with respect to the first element is evaluated to determine whether it satisfies a given criterion indicating the use of the alternative rendering technique.

[0224] It will be understood that the above method can be executed on conventional hardware suitable for executing software instructions related to the above method, or that it may include (or replace) dedicated hardware.

[0225] The necessary applications to existing parts of conventional equivalent devices may be realized in the form of computer program products. Such computers include a processor capable of executing instructions stored on a non-volatile, machine-readable medium (e.g., floppy disks, optical disks, hard disks, solid disks, PROMs, RAMs, flash memory or a combination thereof, or other storage media, or released to hardware as an ASIC (Application-Specific Integrated Circuit) or FPGA (Field-Programmable Gate Array) or other configurable circuit (suitable for use as an application to conventional equivalent devices)). Separately, such computer programs may be transmitted over a network (e.g., Ethernet®, wireless networks, the Internet, and combinations thereof or other networks) via data signals.

[0226] Refer to Figure 1. The methods and techniques described herein may be implemented on conventional hardware (e.g., an entertainment system 10 that generates images from a virtual scene). An example of such an entertainment system 10 is a computer or console such as the Sony PlayStation 5 (PS5) (registered trademark).

[0227] The entertainment system 10 includes a central processor 20. This may be a single-core or multi-core processor (for example, eight cores in the PS5). The entertainment system also includes an image processing unit or GPU 30. The GPU may be physically separate from the CPU, or it may be integrated with the CPU as a system on a chip (SoC), as in the PS5.

[0228] The entertainment device also includes RAM40. This may be separate RAM for each CPU and GPU, or it may be shared RAM, as in the PS5. Each RAM may be physically isolated, or it may be integrated as a SoC, as in the PS5. Furthermore, storage is provided as disk50. This may be an internal drive, an external drive, an external solid-state device, or an internal solid-state device, as in the PS5.

[0229] The entertainment device may transmit or receive data via one or more data ports 60 (e.g., a USB port, an Ethernet® port, a WiFi® port, a Bluetooth® port, etc.). It may selectively receive data via an optical device 70.

[0230] Interaction with the system is typically provided using one or more handheld controllers 80 (e.g., DualSense® in the PS5 console).

[0231] Audio / visual output from entertainment devices is typically provided via one or more A / V ports 90 or one or more wired / wireless data ports 60.

[0232] If the components are not integrated, they may be connected using a dedicated data link or bus 100.

[0233] Accordingly, an embodiment of this disclosure is an entertainment device (Sony PlayStation 5®, etc.) equipped with the following:

[0234] Firstly, an image processing unit (e.g., a GPU30, optionally in conjunction with a CPU20) configured to render pixels in a viewpoint within an image of a virtual scene having a first element and a predetermined surface at a location within the scene (e.g., by a preferred software instruction).

[0235] Secondly, an evaluation processor (e.g., a GPU30, optionally in conjunction with a CPU20) is configured to evaluate (e.g., by a suitable software instruction) whether to render the pixels corresponding to the first element using a machine learning system trained to output values ​​representing the lighting of a given surface at the aforementioned location, or to render the pixels corresponding to the first element using an alternative rendering technique.

[0236] The image processing unit combines the factors predicted by the machine learning system into a distribution function (again, for example, by a suitable software instruction). This generates pixel values ​​corresponding to the first element illuminated at that location in the virtual scene, as described above.

[0237] Furthermore, the image processing unit is configured to include pixel values ​​in the rendered image for display purposes (again, for example, by a suitable software instruction).

[0238] The hardware described above is similarly configured as follows to enable the methods and techniques of this specification to be performed. - The entertainment device comprises multiple machine learning processors (e.g., threads and / or shaders on each processor, GPU, and / or CPU). These machine learning processors run their respective machine learning systems, each trained on each of the multiple contributing components of an image (e.g., diffuse reflection, specular reflection, coating, etc.). Each distribution function is used for each of the multiple contributing components of the image. The image processing unit is configured to combine (again, for example, by a suitable software instruction) each generated pixel value (which is incorporated into the rendered image for display) with each other in order to generate pixel values ​​contained in the rendered image for display. - A machine learning system is a neural network. The input to the first part of the neural network includes position. The input to the second part of the neural network includes the output and direction of the first part.

[0239] Similarly, in other summarized embodiments of the present disclosure, the entertainment device (e.g., Sony PlayStation 5®) includes the following:

[0240] Firstly, an image processing unit (e.g., a GPU30, optionally in conjunction with a CPU20) configured to render pixels in a viewpoint within an image of a virtual scene having a first element and a predetermined surface at a location within the scene (e.g., by a preferred software instruction).

[0241] Secondly, an evaluation processor (e.g., a GPU30, optionally in conjunction with a CPU20) is configured to evaluate (e.g., by a suitable software instruction) whether to render the pixels corresponding to the first element using a machine learning system trained to output values ​​representing the lighting of a given surface at the aforementioned location, or to render the pixels corresponding to the first element using an alternative rendering technique.

[0242] The image processing unit is configured to render the pixels using a selected machine learning system or alternative technique (again, for example, by suitable software instructions).

[0243] Preferably, the entertainment device is The system includes a machine learning processor configured to provide a machine learning system with viewpoint-based position and orientation. This machine learning system is pre-trained to predict factors that, when combined with a distribution function characterizing the interaction between light and a given surface, generate pixel values ​​corresponding to a first element illuminated at that location in a virtual scene. When the evaluation processor determines that the image processing unit should render the pixels using the machine learning system, The aforementioned image processing unit is To generate pixel values ​​corresponding to the first element of the virtual scene illuminated at the aforementioned position, the output of the machine learning system and a distribution function characterizing the interaction between light and the given surface are combined, For display purposes, the pixel values ​​are configured to be included in the rendered image.

[0244] The hardware described above may similarly be configured to perform the methods and techniques of this specification.

[0245] The above discussion describes only typical embodiments of the invention. As those skilled in the art will understand, the invention may be implemented in other specific forms without deviating from its spirit and essential features. Accordingly, the disclosure of the invention is for illustrative purposes only and not intended to limit the scope of the invention and claims. This disclosure (including recognizable variations of the teachings herein) partially defines the scope of the terms of the following claims, but the inventive subject matter is not dedicated to the public.

Claims

1. An image rendering method for rendering pixels from a viewpoint, Regarding a first element of a virtual scene having a given surface at a position within the scene, The steps include evaluating whether to render the pixels corresponding to the first element using a machine learning system trained to output values ​​representing the lighting of the given surface at the aforementioned location, or to render the pixels corresponding to the first element using an alternative rendering technique, A step of rendering the pixels using the machine learning system or alternative technology selected in the evaluation step, Includes, The aforementioned evaluation step is, A method characterized by including the step of performing a test rendering of a subset of pixels in a rendered image using both the machine learning system and the alternative rendering technique for comparison.

2. If the rendering step uses a machine learning system to render the pixels, The rendering step described above is: The steps include combining the output of the machine learning system with a distribution function that characterizes the interaction between light and the given surface in order to generate pixel values ​​corresponding to the first element of the virtual scene illuminated at the aforementioned position, The steps include including the aforementioned pixel values ​​in the rendered image for display, The method according to claim 1, characterized by including the following:

3. The rendering step described above is: The steps include using each machine learning system that has been trained on one of the multiple contributing components of the aforementioned image, The steps include: using the respective distribution function for each of the multiple contributing components of the aforementioned image; For display purposes, the process involves combining the generated pixel values ​​with each other in order to generate the final combined pixel values ​​and include them in the rendered image, The method according to the second invention, characterized by including the following:

4. The method according to any one of 1 to 3, characterized in that the alternative rendering technique is ray tracing.

5. The aforementioned evaluation step is, To render the first element, we evaluate whether the computational cost of using the machine learning system is lower than the computational cost of using the alternative rendering technique. The method according to any one of 1 to 4, characterized in that it includes the step of selecting the machine learning system if the value is low.

6. The aforementioned evaluation step is, The accuracy of the image of the machine learning system for rendering the first element is evaluated. The method according to any one of 1 to 5, further comprising the step of selecting the machine learning system if the accuracy satisfies a predetermined threshold.

7. The aforementioned evaluation step is, The first element includes the step of referring to a given criterion, The aforementioned standards are, i. With respect to the first element, the material type when the machine learning system falls below a predetermined performance threshold, ii. With respect to the first element, the ID when the machine learning system falls below a predetermined performance threshold, iii. With respect to the first element, the distribution function when the machine learning system falls below a predetermined performance threshold, iv. Training performance of the aforementioned machine learning system, v. Comparison of rendering results of pixels obtained using the machine learning system and pixels obtained using the alternative rendering technique. The method according to 5 or 6, characterized by being based on one or more of the above.

8. The aforementioned evaluation step is, The method according to any one of 1 to 7, characterized in that it evaluates whether the virtual material corresponding to the first element is of a predetermined type that uses the alternative rendering technique.

9. The aforementioned virtual material has a material ID, The method according to 8, characterized in that the material ID provides a value or flag indicating whether the virtual material is of a predetermined type that uses the alternative rendering technique.

10. The aforementioned evaluation step is, The method according to any one of 1 to 9, characterized in that it evaluates whether the distribution function characterizing the interaction between light and the given surface gives an indicator that the first element is predetermined to be rendered using the alternative rendering technique.

11. The aforementioned evaluation step is, The method according to any one of 1 to 10, characterized in that the error value or convergence performance of the machine learning system with respect to the first element satisfies a given criterion indicating the use of the alternative rendering technique.

12. A computer program that causes a computer to perform the method according to any one of claims 1 to 11.

13. An image processing unit having a first element and configured to render pixels in a viewpoint within an image of a virtual scene having a given surface at a position within the scene, An evaluation processor configured to evaluate whether to render the pixels corresponding to the first element using a machine learning system trained to output values ​​representing the lighting of the given surface at the aforementioned location, or to render the pixels corresponding to the first element using an alternative rendering technique, Equipped with, The image processing unit is configured to render the pixels using a selected machine learning system or alternative technique. The aforementioned evaluation processor is For comparison, an entertainment device characterized by performing test renderings of a subset of pixels in a rendered image using both the machine learning system and the alternative rendering technique.

14. A machine learning system pre-trained to predict factors based on the aforementioned viewpoint is provided with a machine learning processor configured to provide the aforementioned position and orientation. When the factor is combined with a distribution function that characterizes the interaction between light and the given surface, it generates pixel values ​​corresponding to the first element of the virtual scene illuminated at the position, When the evaluation processor determines that the image processing unit should render the pixels using the machine learning system, The aforementioned image processing unit is To generate pixel values ​​corresponding to the first element of the virtual scene illuminated at the aforementioned position, the output of the machine learning system and a distribution function characterizing the interaction between light and the given surface are combined, The entertainment device according to claim 13, characterized in that it is configured to include the pixel values ​​in the rendered image for display.

Images