Image rendering method and apparatus

Abstract

To provide an image rendering method and apparatus.SOLUTION: An image rendering method for rendering a pixel at a viewpoint, for a first element of a virtual scene having a predetermined surface at a position within that scene, comprises a step of providing the position and a direction based on the viewpoint to a machine learning system previously trained to predict a factor based on the viewpoint. The factor is configured to, when combined with a distribution function that characterizes an interaction of light with the predetermined surface, generate a pixel value corresponding to the first element of the virtual scene as illuminated at the position. The method further includes: a step of combining the predicted factor from the machine learning system with the distribution function to generate the pixel value corresponding to the illuminated first element of the virtual scene at the position; and a step of incorporating the pixel value into a rendered image for display.SELECTED DRAWING: Figure 9

Description

Technical Field

[0001] The present invention relates to an image rendering method and apparatus.

Background Art

[0002] Ray tracing is one of the rendering processes in which the light traces in a virtual scene are traced. Thereafter, the interaction between each ray and an object or a surface in the scene is simulated. To enhance reality, usually this simulation takes into account the material characteristics (e.g., color and reflectivity) of the object or the surface.

[0003] As a result, the process of ray tracing becomes highly costly in terms of calculation. Further, this cost varies for each image frame depending on, for example, "which scene is lit by what light and from what viewpoint".

[0004] Therefore, it is difficult to maintain a suitable frame rate for rendering such images. Regarding the average calculation cost corresponding to the average image completion time (i.e., frame rate), and a predetermined inconsistency in the average periphery due to ray tracing, the average image quality has to be set low so that the frame rate is not greatly affected by this inconsistency. When the average image quality is set high close to the maximum value for a desired frame rate, if the trace requirements that change moment by moment fluctuate beyond the average, the consistency of the frame rate is impaired.

[0005] None of these results are desirable. However, this cannot be easily avoided. On the other hand, the calculation load of the ray tracing process is data-driven and unpredictable.

Summary of the Invention

Problems to be Solved by the Invention

[0006] The present invention addresses such problems and aims to mitigate them. [Means for solving the problem]

[0007] Various aspects and features of the present invention are defined by the claims and the following specification, but include at least the following: -The first example is the image rendering method according to claim 1. -The second example is the entertainment device according to claim 13. [Brief explanation of the drawing]

[0008] By referring to the attached drawings below and reading the detailed description herein, you will gain a more complete understanding of the features and many advantages of this disclosure. [Figure 1] This is a schematic diagram of an entertainment device according to an embodiment of the present disclosure. [Figure 2] This is a ray-traced object according to the embodiments of this disclosure. [Figure 3] This is a schematic diagram of the components contributing to a ray-traced object according to embodiments of the present disclosure. [Figure 4] This is a schematic diagram of the distribution function associated with each component in the embodiment of the present disclosure. [Figure 5] This is a schematic diagram showing the scattering distribution according to an embodiment of the present disclosure. [Figure 6] This is a schematic diagram of a training scheme for machine learning according to an embodiment of the present disclosure. [Figure 7] This is a schematic diagram of the rendering path for a rendered image according to an embodiment of the present disclosure. [Figure 8] This is a schematic diagram of a machine learning system according to an embodiment of the present disclosure. [Figure 9] This is a flowchart of an image rendering method according to an embodiment of the present disclosure. [Modes for carrying out the invention]

[0009] The following discloses image rendering methods and apparatus. For a complete understanding of embodiments of the present invention, the following description details several specific examples. However, it will be apparent to those skilled in the art that these details are not essential for realizing the present invention. Conversely, details of the prior art known to those skilled in the art are omitted where necessary to clarify the description.

[0010] Embodiments of this disclosure aim to address and mitigate the above-mentioned problems by using a machine learning system that learns the relationship between pixel surface characteristics and rendered pixels with respect to a given object or scene. By using such a machine learning system, it is possible to approximate the rendering of a ray-traced object or scene with relatively stable computational costs (for performing the machine learning).

[0011] As will be discussed later, different machine learning systems can be trained depending on different scenes, locations or parts thereof, different objects, or materials used in different scenes.

[0012] Machine learning systems are relatively small (typically 100kB to 1MB). Therefore, to run them on a GPU (30), they can be stored in memory and then discarded (similar to scene textures). This system can run on the GPU's shaders. In principle, machine learning systems can also run on a CPU or a general-purpose or dedicated coprocessor (e.g., a neural network processor or ASIC).

[0013] In multiple drawings, similar symbols represent the same or corresponding parts. Figures 2 through 7 show the problem space in which the machine learning system is trained.

[0014] Figure 2 shows a high-quality ray-traced render 200 of an object or scene (in this case, a car on a platform).

[0015] Figure 3 shows the different contributing components behind this render. First, the diffuse reflection lighting component 200-D typically captures the surface's matte finish and the shading resulting from the interaction of light with the object. Second, the specular reflection lighting component 200-S captures the surface's reflectivity, resulting in glints and highlights. Optionally, one or more additional components can also be included, such as the sheen and "coating" 200-C. This is a second outer surface and includes additional gloss and patterns. Variations of such coatings can also include components such as skin or fabric, which are transparent and / or partially diffusely reflective. Each of these components may be generated in the normal manner by each ray tracing process.

[0016] These components are additionally added together to form the overall image as shown in Figure 2. Typically, two or three components contribute to the render. However, under certain conditions, it can be less (e.g., when only the diffuse reflection lighting component is required) or more (e.g., when the object is translucent and a transmission component is also required - e.g., regarding subsurface scattering).

[0017] Figure 4 below shows the material properties of the object underlying the contributing components of the above image.

[0018] The material properties are represented as the so-called bidirectional scattering distribution function (BSDF) or bidirectional reflectance distribution function (BRDF).

[0019] The BRDF defines how light is reflected from an opaque surface. Similarly, the BSDF defines the probability that a ray of light is reflected or scattered in a particular direction. Thus, the BRDF or BSDF is a function that describes the lighting characteristics of the surface (excluding the incident / emergent radiation itself). Another function may be used if necessary. Another function could be, for example, the bidirectional transmission distribution function (BTDF) that defines how light passes through an object (e.g., regarding subsurface scattering).

[0020] Regarding a typical ray tracing application, it will be described with reference to FIG. 5. For a set of light rays (e.g., from a compact light source), this application calculates the incident luminance (of the incident light itself or the reflected light) at a point on a model with a specific BSDF, BRDF, and / or BTDF. Regarding the response of a specific contributing component, the incident luminance is combined (e.g., multiplied) with the BSDF, BRDF, or BTDF. The result is added to the pixel value at the point of the model. As shown in FIG. 5, the optical path ω of the BSDF i The typical diffuse reflection pattern regarding has a bias towards the specular reflection direction ω a but can diffuse-reflect in any direction. Modeling such behavior accurately is one of the reasons leading to the high computational cost of ray tracing calculations.

[0021] By using the color information of the model at each point and the BSDF, BRDF, and / or BTDF corresponding to that point (i.e., the specific material represented by the given point), the behavior of the light ray regarding a given final viewing point (e.g., the specular or diffuse reflection of light that determines the realistic contribution of the sparkle or highlight on the surface of the vehicle body) can be calculated.

[0022] For each contributing component, an individual BSDF, BRDF, or BTDF may be used. For example (non-exclusively), a BSDF can be used for the diffuse-reflected light component, and a BRDF can be used for the specular reflection component and the coating component (alternatively, a BTDF can be used for the coating component). If necessary, any of the BSDF, BRDF, or BTDF may be used. Therefore, unless otherwise specified hereinafter, when referring to BSDF, it shall also include BRDF and BTDF.

[0023] As shown in Figure 4, ray tracing using the object's color properties and the diffuse reflection material properties of BSDF(200-BSDF-D) yields the diffuse reflection component 200-D. Similarly, using the specular reflection or reflective material properties of BSDF(200-BSDF-S) yields the specular reflection component 200-S. Likewise, using the material properties of BSDF(200-BSDF-C) (which is usually also specular reflection) yields the coating component 200-C. Combining these components ultimately yields the ray-traced image 200.

[0024] However, as mentioned above, the challenges lie in the following: the computational cost is high whether calculating the diffuse reflection or reflection path of light incident on different surfaces with different BSDFs, or whether summing up those results pixel by pixel in a scene from a specific viewpoint, and these costs can vary considerably.

[0025] Accordingly, the embodiments of this disclosure aim to replace the ray tracing step in Figure 4 with one that is suitable for the final image quality and whose computational load is more predictable.

[0026] Refer to Figure 6. In embodiments of this disclosure, a machine learning system is provided for each contributing component of the image (diffuse reflection, specular reflection, and optionally coating or other arbitrary contributing components).

[0027] The machine learning system is typically a neural network, as described below. It is trained to learn the transformation between BSDF (e.g., 200-BSDF-D) and the ground truth (e.g., 200-D) of the ray-traced image contribution components for multiple images viewed from different viewpoints within a scene.

[0028] In other words, suppose the ray-traced image (or one of its contributing components) is a combination of the lighting behavior on an object and the BSDF (which represents how the object reacts to light). Now, suppose we extract the ray-traced image and separate it from the BSDF. The result is a quality called "radiance" or "density," which more generally indicates how light behaves on the object (as calculated holistically by the ray tracing process).

[0029] A machine learning system or neural network can learn to predict this quality. The predicted quality is then combined again with the BSDF to generate a predicted image that approximates the ray-traced image. This network can therefore be called a neural pre-computed optical model network (NPLM network).

[0030] More specifically, for a given position and line of sight on a virtual image of an object, the machine learning system or neural network needs to learn to output a value. This value is close to the pixel value obtained when ray tracing is performed on that pixel, when combined with a BSDF for the same position / pixel. As a result, during training, an internal representation of the lighting conditions (e.g., point light or cyclorama) and surface lighting characteristics suggested by the training image are generated.

[0031] In one embodiment, the image may be rasterized or generated at a given viewpoint. This is done by filling the image with pixels and then lighting it. For each of these conceptual pixels, a corresponding 3D position exists in the scene. By using an NPLM network on this, the appropriate "radiance" or "grain" can be obtained.

[0032] Figure 6 shows the training environment for such networks. Figure 6 also shows network 600-D for the diffuse reflection component (for illustrative purposes only).

[0033] The inputs to the network for the diffuse reflection component are the (x, y, z) position 610 on the object or scene and the object / scene normal 620 at that point. Here, the normal N is used instead of the viewpoint direction because, for the diffuse reflection component, illuminance is independent of direction / viewpoint, and from a consistency standpoint, the normal can be used as a known value. In the example in Figure 6, these inputs are conceptually shown using representative values ​​for the car image.

[0034] Selectively, additional inputs may be provided (not shown). For example, scalar values ​​for roughness or matt-to-gloss. These may selectively be derived from the relevant BSDF.

[0035] The output of the NPLM network is (as described later) a learned light or illuminance quality 630 for each input location. This, when combined with the corresponding diffuse reflected light BSDF (200-BSDF-D) for the same location (640), generates predicted pixel values ​​for the location (x, y) in the predicted image (650).

[0036] Figure 6 shows that for each pixel, the difference between the predicted pixel and the ground truth pixel for the target ray tracing diffuse reflection component 200-D is used as the loss function for training the network. However, this is not mandatory. Rather, for each pixel (x, y), the ground truth image can be separated from the BSDF (using the inverse function) to generate a proxy for what cumulative effect the ray tracing light had on the objects in the image. The network is trained to learn this quality.

[0037] Therefore, the error function of the network when separated from the corresponding BSDF is based on the difference between the output value of its single pixel (x, y) and the corresponding single pixel in the ground truth image.

[0038] For each pixel, the pixels in the ground truth image can be pre-separated from their corresponding BSDF. Therefore, the network can be trained to generate actual predicted image pixels without combining its output with the BSDF. This reduces the computational load for training.

[0039] As mentioned earlier, the trained quality output from the trained neural network captures how ambient light behaves on an object or scene as a function of the surface position and viewpoint within the scene. This effectively generates an internal representation of the light map in the scene and surface response model, which will be discussed in more detail later.

[0040] Refer to Figure 7 for a summary of each contributing component of the final output image. The machine learning system is trained to perform a transformation applied to the BSDF of each position on the object / scene with respect to the contributing components. This transformation is a function trained based on the (x, y, z) position and orientation values ​​on the object / scene. As mentioned above, any number of machine learning systems (e.g., one, two, three, four, or more) may be employed depending on the number of contributing components in the final image. Hereafter, the term "trained function" refers to a machine learning system that has learned such a transformation.

[0041] As explained in the section on diffuse reflection components, the directional value can be considered as the normal at a given point. This is because the intensity of diffuse reflection can be considered directionally invariant.

[0042] Specular reflection, on the other hand, is reflection in certain directions and varies depending on the viewpoint. Therefore, for the specular component, its directional value is the line of sight angle between (or based on) the current pixel's (x, y) position at the image viewpoint (which will have a position in virtual space) and the object's (x, y, z) position as input to the machine learning system. This gives a viewpoint-dependent relationship between the input point on the scene surface and the current pixel (with respect to which the learned quality is output).

[0043] In this case, the coating component is also specularly reflected. Therefore, the coating component also uses a similar viewpoint or viewpoint-based orientation with respect to the input.

[0044] The direction value for direction-dependent components may be the line of sight direction (or a value based thereon) between the output pixel position and the object surface position. For example, this may be the surface reflection viewpoint direction (i.e., given the normal of the surface at the input position, the reflection direction of the viewpoint to it). Any direction value that includes information about the viewpoint direction may be considered.

[0045] In each case, the trained function encapsulates the learned quality, as described above. By combining an appropriate BSDF with the network output for each location, a grayscale image can be formed for each component. Alternatively or additionally, the final output may be generated by combining the pixel values ​​for the grayscale image from each component.

[0046] During image rendering, not all images are necessarily subject to ray tracing. Similarly, not all images are necessarily generated using the techniques described above. For example, an NPLM network may be trained on ground truth ray-traced images for a specific object or material with its respective lighting.

[0047] Subsequently, if these objects or materials need to be rendered in real time using ray tracing provided by the trained function described above, the relevant NPLM network is loaded into memory. This relevant NPLM network, combined with the appropriate BSDF, is executed with respect to the relevant surface position and line of sight in the scene to generate contributions to the relevant pixels. Other pixels may be rendered using any other technique (including ray tracing itself).

[0048] Typically, a suitable machine learning system is selected and loaded into memory used by the GPU, based on the same asset identification scheme used when selecting and loading textures for an object or material. Therefore, for example, if an object has an ID "1234" used to access its associated texture, this ID can be associated with the relevant machine learning system. Conversely, if a texture has an ID "5678" associated with an object (for example, if this texture represents a material common to multiple objects), this ID can also be associated with the relevant machine learning system. In this way, entertainment devices can use a similar process to load textures to load machine learning systems. Actual storage and access techniques will differ between textures and machine learning systems. This is especially true if textures are stored using lossy compression (which affects the operation of the compressed machine learning system). Therefore, machine learning systems may be stored without compression, with lossless compression, or with compression that is lossy enough for the compressed machine learning system to function adequately. This can be evaluated by comparing the output error / cost function of the machine learning system until the error reaches an absolute or relative quality threshold (compared to the uncompressed machine learning system) in relation to the increase in loss during compression.

[0049] Refer to Figure 8. In some embodiments of this disclosure, the machine learning system or NPLM network may be any suitable machine learning system. For example, a single neural network may be trained using position and viewpoint direction as inputs and generate RGB values ​​for the learned characteristics as outputs.

[0050] However, a particularly preferred network features a separate split architecture.

[0051] As shown in Figure 8A, in a non-limiting embodiment, the network comprises two parts: the first part is a location network and the second part is a direction network.

[0052] Each of these networks may have three or more layers and may use any suitable activation function.

[0053] The position network receives the (x, y, z) position of a point within the aforementioned object / scene as input and outputs an intermediate representation (explained later).

[0054] The directional network receives this intermediate representation and directional input (e.g., normal, or pixel viewpoint, or surface reflection viewpoint direction, or viewpoint-based directional value, etc.) in, for example, (θ, Φ) format or normalized (x, y, z) vector format. The directional network outputs RGB values ​​corresponding to the learned quality with respect to the current pixel position (x, y) (and therefore pixel viewpoint) in the image from the aforementioned virtual camera position to be rendered (in a space shared by the object / scene).

[0055] Therefore, in a non-restrictive embodiment, the location network has three layers. Each layer is as follows: The first layer has three input nodes (e.g., x, y, z positions). The intermediate layer has 128 hidden layers. The final layer has 8 outputs.

[0056] Any suitable activation function may be chosen for the network. In particular, the ReLU (Rectified Linear Unit) function has been evaluated as an effective activation function between layers of location networks. It generalizes well to untrained locations and helps prevent overfitting.

[0057] Similarly, in a less restrictive embodiment, the orientation network comprises four layers. Each layer is as follows: The first layer comprises the outputs of the eight position networks and two or three additional values ​​for orientation assignment to 128 nodes. The second and third layers comprise the inputs to 128 nodes. The three final outputs on the final layer are the corresponding R, G, and B values ​​for the learned quality at the current pixel. These may then be combined with (e.g., multiplied by) the BSDF for that position to obtain the final pixel contribution from the learned function (e.g., diffuse reflection, specular reflection). However, as mentioned above, this is not required during training.

[0058] Regarding directional networks, any suitable activation function may be selected. In particular, the sine function has been evaluated as an effective activation function between layers of a directional network. The change in the behavior of light within an angular range is large and involves many details at angular frequencies. However, this is based on a low-dimensional output (e.g., a normalized x, y, z vector). Therefore, the sine activation function is known to be particularly effective.

[0059] Therefore, especially when the network is divided into two equal parts, different activation functions may be used for each part.

[0060] However, the network is treated not as two independent networks, but as a split architecture network, because the training scheme, in particular, has only one cost function: the error between the RGB values ​​output from the directional network after it has been separated from the appropriate BSDF, and the target values ​​obtained from the corresponding pixels of the ground truth of the ray-traced image.

[0061] This error propagates backward through both networks; that is, there are no separate cost functions for the target values ​​or location networks. Therefore, the output layer of the location network is, in effect, a hidden layer of the combined network. This increases with further input of directional values ​​and represents the change within the layer from the first activation function to a possible second different activation function.

[0062] As mentioned above, the neural network constructs a light model of the illuminated object, material, or scene. In particular, in the non-limiting embodiment described above, the position network effectively classifies the (x, y, z) position into a type of lighting (e.g., bright or dark, and / or other possible categories regarding how the light interacts with each BSDF (e.g., relative reflectivity or diffuse reflectivity)). The intermediate representation output from this part can be thought of as an N-dimensional position in lighting space characterizing the type of light at the input position. This is the illumination position to an N-dimensional position in a different part of the scene if illuminated in the same way. A position network trained on the specular component may have more outputs than one trained on the diffuse component, for example, 13 outputs compared to 8 for the latter. This is a result of considering that the specular component has more diversity with respect to the type of lighting.

[0063] The subsequent directional network models how light behaves when viewed at a given input angle on the surface of the input location, with respect to the illuminated object, material, or scene, in order to generate learned characteristics of the image at that location.

[0064] Therefore, the position and orientation network is trained to predict factors or changes between a BSDF representing the surface properties and the desired rendered image of that surface. Thus, this network can be used instead of ray tracing for the purpose of rendering the surface. Typically (but not required), the network is trained on a single contributing component of the image (e.g., diffuse reflection of the specular component) along with multiple networks used to generate the components required for the final image (or part thereof). However, this is not required (i.e., in principle, the network can be trained on a fully combined image or on two or more contributing components (e.g., all specular components or all diffuse components)).

[0065] [Training] As will be explained in another section, the network is trained using ray-traced images of multiple objects, scenes, or surfaces acquired from multiple viewpoints. This allows the network to learn, in particular, how specular reflections change with location. The viewpoints can be randomly distributed and / or selected from a range of viewpoints available to the user as they move through the rendered environment.

[0066] Network Selection The network is trained during gameplay or application development. Developers may choose when and where NPLM rendering is useful. For example, NPLM rendering may be used only in scenes known to cause frame rates below a certain quality threshold. In such cases, the network is trained on such scenes (or parts thereof) and used when such scenes appear.

[0067] In another case, developers may choose to use NPLM-based rendering for certain objects or materials. In this case, the network is trained and used when these objects or materials are recognized within the scene to be rendered.

[0068] In another case, developers may choose to use NPLM-based rendering for certain lighting conditions or combinations thereof. In this case, the network is trained and used when these conditions are found within the scene to be rendered.

[0069] Similarly, in other cases, developers may choose to use NPLM-based rendering for specific draw distances (z-distances), or for objects that are angularly / distant from the image center or the user's field of view, or for certain lighting conditions. In this case, the network is trained and used in these environments.

[0070] Similarly, any suitable combination of these criteria may be selected for training and use. summary Refer to Figure 9. To summarize an embodiment of the present disclosure, an image rendering method for rendering pixels at a viewpoint, the method comprising the following steps with respect to a first element of a virtual scene having a predetermined surface at a location in the scene:

[0071] In the first step 910, a viewpoint-based position and orientation are given to a machine learning system. This machine learning system is pre-trained to predict a factor that, when combined with a distribution function characterizing the interaction between light and a given surface, generates a pixel value corresponding to a first element illuminated at that position in a virtual scene.

[0072] In the second step 920, the factors predicted by the machine learning system are combined into the distribution function. This generates pixel values ​​corresponding to the first element illuminated at that location in the virtual scene, as described above.

[0073] In the third step 930, the pixel values ​​are included in the rendered image for display. The image is then output to the display through the A / V port (90).

[0074] Those skilled in the art will understand that one or more modifications of the above methods are possible within the scope of this disclosure. These modifications correspond to the operation of various embodiments of the methods and / or apparatus described herein and in the claims, and include, but are not limited to, the following: - For each of the multiple contributing components of an image (e.g., diffuse reflection, specular reflection, coating, etc.), a separate machine learning system is trained. For each of the multiple contributing components of an image, a separate distribution function is used. The generated pixel values ​​are combined to produce the pixel values ​​contained in the rendered image for display. -The distribution functions are selected from a list consisting of the bidirectional scattering distribution function, the bidirectional reflection distribution function, and the bidirectional transmission distribution function. - A machine learning system is a neural network. The input to the first part of the neural network includes position. The input to the second part of the neural network includes the output and direction of the first part. -In this example, the activation function for the first part is different from the activation function for the second part. -In this example, the activation function for the first part is ReLU, and the activation function for the second part is the sine function. -In this example, the cost function of the neural network is separated from the distribution function based on the difference between the output of the second part and the pixel values ​​obtained from the ray-traced version. -In this example, the cost function of the neural network propagates backward through both the second and first parts during training. -In this example, the neural network is a fully connected network. - The cost function of the machine learning system is separated from the distribution function based on the difference between the output of the machine learning system and the pixel values ​​obtained from the ray-traced version. - The machine learning system is selected based on the same asset identification scheme used to select and load textures for the first element of the scene, and is loaded into memory used by the image processing unit.

[0075] It will be understood that the above method can be executed on conventional hardware suitable for executing software instructions related to the above method, or that it may include (or replace) dedicated hardware.

[0076] The necessary applications to existing parts of conventional equivalent devices may be realized in the form of computer program products. Such computers include a processor capable of executing instructions stored on a non-volatile, machine-readable medium (e.g., floppy disks, optical disks, hard disks, solid disks, PROMs, RAMs, flash memory or a combination thereof, or other storage media, or released to hardware as an ASIC (Application-Specific Integrated Circuit) or FPGA (Field-Programmable Gate Array) or other configurable circuit (suitable for use as an application to conventional equivalent devices)). Separately, such computer programs may be transmitted over a network (e.g., Ethernet®, wireless networks, the Internet, and combinations thereof or other networks) via data signals.

[0077] Refer to Figure 1. The methods and techniques described herein may be implemented on conventional hardware (e.g., an entertainment system 10 that generates images from a virtual scene). An example of such an entertainment system 10 is a computer or console such as the Sony PlayStation 5 (PS5) (registered trademark).

[0078] The entertainment system 10 includes a central processor 20. This may be a single-core or multi-core processor (for example, eight cores in the PS5). The entertainment system also includes an image processing unit or GPU 30. The GPU may be physically separate from the CPU, or it may be integrated with the CPU as a system on a chip (SoC), as in the PS5.

[0079] The entertainment device also includes RAM40. This may be separate RAM for each CPU and GPU, or it may be shared RAM, as in the PS5. Each RAM may be physically isolated, or it may be integrated as a SoC, as in the PS5. Furthermore, storage is provided as disk50. This may be an internal drive, an external drive, an external solid-state device, or an internal solid-state device, as in the PS5.

[0080] The entertainment device may transmit or receive data via one or more data ports 60 (e.g., a USB port, an Ethernet® port, a WiFi® port, a Bluetooth® port, etc.). It may selectively receive data via an optical device 70.

[0081] Interaction with the system is typically provided using one or more handheld controllers 80 (e.g., DualSense® in the PS5 console).

[0082] Audio / visual output from entertainment devices is typically provided via one or more A / V ports 90 or one or more wired / wireless data ports 60.

[0083] If the components are not integrated, they may be connected using a dedicated data link or bus 100.

[0084] Accordingly, an embodiment of this disclosure is an entertainment device (Sony PlayStation 5®, etc.) equipped with the following:

[0085] Firstly, an image processing unit (e.g., a GPU30, optionally in conjunction with a CPU20) configured to render pixels in a viewpoint within an image of a virtual scene having a first element and a predetermined surface at a location within the scene (e.g., by a preferred software instruction).

[0086] Secondly, an evaluation processor (e.g., a GPU30, optionally in conjunction with a CPU20) configured to evaluate (e.g., by a suitable software instruction) whether to render the pixels corresponding to the first element using a machine learning system trained to output values ​​representing the lighting of a given surface at the location, or to render the pixels corresponding to the first element using an alternative rendering technique.

[0087] The image processing unit combines the factors predicted by the machine learning system into a distribution function (again, for example, by a suitable software instruction). This generates pixel values ​​corresponding to the first element illuminated at that location in the virtual scene, as described above.

[0088] Furthermore, the image processing unit is configured to include pixel values ​​in the rendered image for display purposes (again, for example, by a suitable software instruction).

[0089] The hardware described above is similarly configured as follows to enable the methods and techniques of this specification to be performed. - The entertainment device comprises multiple machine learning processors (e.g., threads and / or shaders on each processor, GPU, and / or CPU). These machine learning processors run their respective machine learning systems, each trained on each of the multiple contributing components of an image (e.g., diffuse reflection, specular reflection, coating, etc.). Each distribution function is used for each of the multiple contributing components of the image. The image processing unit is configured to combine (again, for example, by a suitable software instruction) each generated pixel value (which is incorporated into the rendered image for display) with each other in order to generate pixel values ​​contained in the rendered image for display. - A machine learning system is a neural network. The input to the first part of the neural network includes position. The input to the second part of the neural network includes the output and direction of the first part.

[0090] The above discussion describes only typical embodiments of the invention. As those skilled in the art will understand, the invention may be implemented in other specific forms without deviating from its spirit and essential features. Accordingly, the disclosure of the invention is for illustrative purposes only and not intended to limit the scope of the invention and claims. This disclosure (including recognizable variations of the teachings herein) partially defines the scope of the terms of the following claims, but the inventive subject matter is not dedicated to the public.

Claims

1. An image rendering method for rendering pixels from a viewpoint, Regarding a first element of a virtual scene having a given surface at a position within the scene, The step includes providing a machine learning system, which has been pre-trained to predict factors based on the aforementioned viewpoint, with a location and orientation based on the aforementioned viewpoint. When the factor is combined with a distribution function that characterizes the interaction between light and the given surface, it generates pixel values ​​corresponding to the first element of the virtual scene illuminated at the position, The steps include combining the factors predicted by the machine learning system with the distribution function in order to generate pixel values ​​corresponding to the first element of the virtual scene illuminated at the aforementioned position, The steps include including the aforementioned pixel values ​​in the rendered image for display, A method characterized by further comprising:

2. Each of the aforementioned machine learning systems is trained with respect to each of the multiple contributing components of the image, Each of the aforementioned distribution functions is used with respect to each of the multiple contributing components of the image, The method according to claim 1, characterized in that each of the generated pixel values ​​is combined with the others in order to generate a final combined pixel value for display and include it in a rendered image.

3. Each of the aforementioned distribution functions is: i. Bidirectional scattering distribution function, ii. Bidirectional reflection distribution function, iii. Bidirectional transmission distribution function, The method according to 1 or 2, characterized in that it is either one of the following.

4. The aforementioned machine learning system is a neural network, The input to the first part of the neural network includes position, The method according to any one of 1 to 3, characterized in that the input to the second part of the neural network includes the output and direction of the first part.

5. The method according to 4, characterized in that the activation function of the first part and the activation function of the second part are different.

6. The method according to 5, characterized in that the activation function of the first part is ReLU and the activation function of the second part is a sine function.

7. The method according to any one of 4 to 6, characterized in that the cost function of the neural network is separated from the distribution function based on the difference between the output of the second part and the pixel values ​​obtained from the ray-traced version.

8. The method according to any one of 4 to 7, characterized in that the cost function of the neural network propagates backward through both the second and first parts during training.

9. The method according to any one of 4 to 8, characterized in that the neural network is a fully connected network.

10. The method according to any one of 1 to 9, characterized in that the cost function of the machine learning system is separated from the distribution function based on the difference between the output of the machine learning system and the pixel values ​​obtained from the ray traced version.

11. The method according to any one of 1 to 10, characterized in that the machine learning system is selected based on the same asset identification scheme used to select and load textures for the first elements of the scene, and loaded into memory used by the image processing unit.

12. A computer program that causes a computer to perform the method according to any one of claims 1 to 11.

13. An image processing unit having a first element and configured to render pixels in a viewpoint within an image of a virtual scene having a given surface at a position within the scene, A machine learning processor configured to provide a position and orientation based on the viewpoint to a machine learning system that has been pre-trained to predict factors based on the viewpoint, Equipped with, When the factor is combined with a distribution function that characterizes the interaction between light and the given surface, it generates pixel values ​​corresponding to the first element of the virtual scene illuminated at the position, The image processing unit is configured to combine the factors predicted by the machine learning system with the distribution function in order to generate pixel values ​​corresponding to the first element of the virtual scene illuminated at the position, An entertainment device characterized in that the image processing unit is configured to include the pixel values ​​in the rendered image for display.

14. Equipped with multiple machine learning processors, The machine learning processor executes each machine learning system that has been trained for each of the multiple contributing components of the image. Each of the aforementioned distribution functions is used with respect to each of the multiple contributing components of the image, The entertainment device according to claim 13, characterized in that the image processing unit is configured to combine each generated pixel value with one another in order to generate pixel values ​​contained in the rendered image for display.

15. The aforementioned machine learning system is a neural network, The input to the first part of the neural network includes position, The entertainment device according to claim 13 or 14, characterized in that the input to the second part of the neural network includes the output and direction of the first part.

Images