Method and system for rendering video graphics

Abstract

Methods and systems for rendering video graphics are provided. The method includes generating and receiving graphical data from a 3D scene including a first object. Raw data may be generated by a vertex shader using vertex data of a scene. A roughness value for the first object may be determined using the vertex data and the associated rasterization. The vertex data may be interpolated by rasterization to infer principal rays, in which a secondary ray is projected from a first intersection location. The secondary light may be reflected or refracted using the roughness value and a calculated Fresnel value associated with the first intersection position and the implicit principal light. The first pixel may then be colored based on at least the secondary light. Other embodiments include corresponding systems and computer programs configured to perform operations of these methods.

Description

Background of the Invention

[0002] As video graphics standards increase year by year, the resource costs for rendering such video graphics also continue to increase. In real-time applications (RTA) such as video games, video conferencing, and virtual reality (VR) applications, optimizing these costs is particularly important. In addition, since the use of such RTAs in mobile devices has become increasingly widespread, it has become increasingly important to improve the quality of video graphics in mobile applications. However, compared to desktop computers, mobile devices have limited memory capacity and bandwidth, which poses challenges for achieving sufficient rendering performance. As described below, there are various solutions to address the memory-intensive nature of video graphics rendering, but these solutions are not perfect.

[0003] Therefore, there is a need for new and improved systems and methods for rendering video graphics. Summary of the Invention

[0004] The present invention relates to graphics rendering systems and methods. According to specific embodiments, the present invention provides a method for optimizing secondary ray tracing using noiseless rendering. There are also some other embodiments.

[0005] Embodiments of the present invention can be implemented in combination with existing systems and processes. For example, the rendering system configuration according to the present invention and its related methods can be used in a variety of systems, including virtual reality (VR) systems, mobile devices, etc. In addition, various techniques of the present invention can be applied to existing systems through integrated circuit manufacturing, operating software, and application programming interfaces (APIs). The present invention also has other advantages.

[0006] A system of one or more computers can be configured to perform particular operations or actions by means of software, firmware, hardware, or a combination thereof installed on the system, the software, firmware, hardware, or combination thereof causing or resulting in the system performing the actions in operation. One or more computer programs can be configured to perform particular operations or actions by means of instructions that, when executed by a data processing apparatus, cause the apparatus to perform the actions. One general aspect includes a method of rendering refractive transparent objects in real-time video graphics using ray tracing. The method further includes receiving a plurality of graphics data associated with a three-dimensional (3D) scene, the 3D scene including all data required to determine a first object intersected by a ray cast through each pixel in a viewport, including the plurality of graphics data, the plurality of graphics data including a plurality of vertex data associated with a plurality of vertices in the 3D scene. The method further includes generating a plurality of primitive data using at least the plurality of vertex data and a vertex shader, the plurality of primitive data including position data and material data. The method further includes rasterizing using at least the plurality of vertex data and material properties and texture coordinates associated with at least the first object to determine a roughness value of the first object. The method further includes using rasterization to calculate both a direction of an implicit primary ray passing through a pixel and a first intersection position of the ray with the first object. The method further includes casting secondary rays from the first intersection position in a fragment shader. The method further includes using an environment map associated with the 3D scene to provide visual continuity across reflected and refracted images. The method further includes accessing an environment map associated with the 3D scene. The method further includes calculating a Fresnel value associated with the first intersection position and the implicit primary ray in a fragment shader. The method further includes determining whether to skip, reflect, or refract a secondary ray from the first intersection position using a fragment shader based on at least the Fresnel value and the roughness value. The method further includes coloring a first pixel covering the first intersection position based on at least the secondary ray. The method further includes storing the first pixel into a frame buffer. Other implementations of this aspect include corresponding computer systems, apparatuses, and computer programs recorded on one or more computer storage devices, each computer program being configured to perform the operations of the method.

[0007] Embodiments may include one or more of the following features. Hardware-accelerated embedded ray tracing may be used to cast secondary rays. The method may also include rasterizing an object visible to a camera and interpolating rendering data, the object including a first object. The method may include determining whether to cast secondary rays by comparing a roughness value with a predetermined threshold roughness value. The method may include storing a maximum roughness value on a hit path associated with an implicit primary ray. The method may include determining whether to skip, reflect, or refract a secondary ray by comparing a Fresnel value with a predetermined threshold Fresnel value. Secondary rays may be cast in a refraction or reflection direction from a first intersection location. Vertex shaders and fragment shaders are processed using a graphics processing unit. Implementations of the described techniques may include hardware, a method or process, or computer software on a computer-accessible medium.

[0008] One general aspect includes a system for providing rendered video graphics. The system may include a permanent memory that stores an application including executable instructions. The system may also include a volatile memory that stores data used during the execution of the application. The system may also include a processor coupled to the memory and the storage, the processor being configured to: execute the application instructions to generate a plurality of graphics data associated with a three-dimensional (3D) scene including at least a first object, the plurality of graphics data including a plurality of vertex data associated with a plurality of vertices in the 3D scene; generate a plurality of primitive data including position data and material data using at least the plurality of vertex data and a vertex shader; determine a roughness value of the first object visible at each pixel using rasterization of at least the plurality of vertex data and material attributes and texture coordinates associated with at least the first object; cast a secondary ray from a first intersection location between the first object and an implicit primary ray determined by the rasterization within a fragment shader; calculate a Fresnel value associated with the first intersection location and the implicit primary ray within the fragment shader; determine whether to skip, reflect, or refract the secondary ray from the first intersection location using the fragment shader based on at least the Fresnel value and the roughness value; if a secondary ray is cast, color a first pixel covering the first intersection location based on at least the secondary ray; and store the first pixel in the memory. Other implementations of this aspect include corresponding computer systems, apparatus, and computer programs recorded on one or more computer storage devices, each computer program being configured to perform the operations of the method.

[0009] Embodiments may include one or more of the following features. In a system, a processor may include a central processing unit (CPU) and a graphics processing unit (GPU). Memory is shared by the CPU and the GPU. The memory may include a frame buffer for storing first pixels. The system may include a display configured to display the first pixels at a refresh rate of at least 24 frames per second. Implementations of the described techniques may include hardware, a method or process, or computer software on a computer-accessible medium.

[0010] One general aspect includes a method of generating video using ray tracing. The method includes generating a three-dimensional (3D) scene composed of objects. The method also includes receiving a plurality of graphics data associated with each object in the 3D scene, the plurality of graphics data including a plurality of vertex data associated with a plurality of vertices in each object. The method also includes generating a plurality of primitive data using at least the plurality of vertex data and a vertex shader, the plurality of primitive data may include position data and material data. The method also includes rasterizing using at least the plurality of vertex data and material attributes and texture coordinates associated with at least a first object to determine a roughness value of the first object to approximate a primary ray cast. The method also includes casting secondary rays from a first point in a fragment shader. The method also includes calculating a Fresnel value associated with the first point and the primary ray in the fragment shader. The method also includes determining whether to reflect or refract the secondary rays from the first point using the fragment shader based on at least the Fresnel value and the roughness value. The method also includes coloring a first pixel covering the first point based on at least the secondary rays. Other implementations of this aspect include corresponding computer systems, devices, and computer programs recorded on one or more computer storage devices, each of the computer programs being configured to perform the operations of the method.

[0011] Embodiments may include one or more of the following features. The method may also include accessing an environment map associated with the 3D scene. The method may also include using the environment map associated with the 3D scene to provide visual continuity across reflection and refraction images. The 3D scene is generated using a central processing unit. The vertex shader and the fragment shader are processed using a graphics processing unit. The central processing unit and the graphics processing unit share memory. The method may also include transforming the 3D scene into screen space. Implementations of the described techniques may include hardware, a method or process, or computer software on a computer-accessible medium.

[0012] It should be recognized that the embodiments of the present invention have many advantages compared to the prior art. In particular, the present invention provides a configuration and method for a graphics rendering system that uses a noiseless unified path tracer to minimize the memory bandwidth load. In addition, the present invention also implements secondary ray optimization and Blended Fresnel Splitting to further reduce the memory cost while achieving real-time rendering for mobile applications.

[0013] The present invention achieves these and other advantages in the context of known technologies. However, the nature and advantages of the present invention can be further understood by referring to the later parts of the specification and the drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] Figure 1 is a simplified diagram showing a system configured to render video graphics according to an embodiment of the present invention.

[0015] Figure 2 is a simplified flowchart showing a conventional forward pipeline for rendering video graphics.

[0016] Figure 3 is a simplified flowchart showing a conventional hybrid pipeline for rendering video graphics.

[0017] Figure 4 is a simplified flowchart in a rendering system according to an embodiment of the present invention.

[0018] Figure 5 is a simplified flowchart showing a method for rendering video graphics according to an embodiment of the present invention.

[0019] Figure 6 is a simplified flowchart showing a method for generating video according to an embodiment of the present invention. DETAILED DESCRIPTION

[0020] The present invention relates to graphics rendering systems and methods. According to specific embodiments, the present invention provides methods and systems for optimizing secondary ray tracing using noiseless rendering. The present invention can be configured for real-time applications (RTAs), such as video conferencing, video games, and virtual reality (VR). There are also some other embodiments.

[0021] Conventional techniques typically involve hybrid rendering methods that are more focused on desktop computing applications. These conventional hybrid methods combine traditional rasterization for primary graphics rendering and ray tracing for rendering advanced details such as lighting. These techniques require multiple passes, each of which renders the target using a separate screen space, which is not suitable for mobile applications with limited memory and power capacity.

[0022] It should be recognized that the embodiments of the present invention have many advantages compared with the prior art. In addition, the present invention provides a configuration and method for a graphics rendering system that uses a noiseless unified path tracer to minimize the memory bandwidth load. Further, the present invention also implements secondary ray optimization and hybrid Fresnel beam splitting to achieve real-time rendering for mobile applications while further reducing the memory cost.

[0023] The following description is to enable those skilled in the art to make and use the present invention and incorporate the present invention into the context of a specific application. For those skilled in the art, various modifications and various uses in different applications will be obvious, and the general principles defined herein can be applied to a wide range of embodiments. Therefore, the present invention is not intended to be limited to the embodiments presented, but should have the broadest scope consistent with the principles and novel features disclosed herein.

[0024] In the following detailed description, numerous specific details are set forth to provide a more thorough understanding of the present invention. However, it will be apparent to those skilled in the art that the present invention can be practiced without these specific details. In other instances, well-known structures and devices are presented in block diagram form rather than in detail to avoid obscuring the present invention.

[0025] The reader is directed to all papers and documents that are filed concurrently with this specification, and to all papers and documents that are open to the public together with this specification. The content of all such papers and documents is hereby incorporated by reference. Unless otherwise expressly stated, all features disclosed in this specification (including any appended claims, abstract, and drawings) may be replaced by alternative features serving the same, equivalent, or similar purpose. Thus, unless otherwise expressly stated, each feature disclosed is only one example of a series of equivalent or similar features.

[0026] Furthermore, no element in a claim that does not expressly state a "means" for performing a specified function or a "step" for performing a particular function shall be construed as a "means" or "step" clause as set forth in paragraph 6 of Section 112 of Title 35 of the United States Code. In particular, the "steps" or "acts" used in the claims herein are not intended to invoke the provisions of paragraph 6 of Section 112 of Title 35 of the United States Code.

[0027] Note that if left, right, front, back, up, down, forward, reverse, clockwise, and counterclockwise labels are used, these labels are for convenience only and do not denote any particular fixed direction. Instead, these labels are used to reflect the relative position and / or orientation between various parts of an object.

[0028] Figure 1FIG. 0 is a simplified block diagram of a mobile system 100 configured to render video graphics according to an embodiment of the present invention. This figure is merely an example and should not unduly limit the scope of the claims. Those of ordinary skill in the art will recognize many variations, alternatives, and modifications.

[0029] As shown, the mobile system 100 may be configured within a housing 110 and may include a camera device 120 (or other image or video capture device), a processor device 130, a memory device 140 (e.g., volatile memory), and a storage device 150 (e.g., permanent memory). The camera 120 may be mounted on the housing 110 and configured to capture input images. The input images may be stored in the memory 140, which may include a random access memory (RAM) device, an image / video buffer device, a frame buffer, etc. The storage device 150 may store various software, executable instructions, and files therein, and the storage device 150 may include a read-only memory (ROM), a hard disk, etc. The processor 130 may be coupled to each of the aforementioned components and configured to communicate between these components.

[0030] In a specific example, the processor 130 includes a central processing unit (CPU), a network processing unit (NPU), etc. The system 100 may also include a graphics processing unit (GPU) 132 coupled to at least the processor 130 and the memory 140. In one example, the memory 140 is configured to be shared between the processor 130 (e.g., CPU) and the GPU 132 and is configured to hold data used by an application when the application is running. Since the memory 140 is shared, it is very important to use the memory 140 efficiently. For example, high memory usage by the GPU 132 may have a negative impact on system performance. For example, the shared memory or RAM can be used in various ways, depending on the specific needs of the CPU and GPU. The amount of shared RAM available in the device has a significant impact on its performance because the amount of shared RAM determines how much data the CPU and GPU can store and access quickly. In a specific example, the memory 140 is configured to be tile-based memory.

[0031] System 100 may further include a user interface 160 and a network interface 170. The user interface 160 may include a display area 162 configured to display text, images, videos, rendered graphics, interactive elements, etc. The display 162 may be coupled to the GPU 132 and may also be configured to display at a refresh rate of at least 24 frames per second. The display area 162 may include a touch screen display (e.g., in a mobile device, tablet, etc.). Alternatively, the user interface 160 may further include a touch interface 164 for receiving user input (e.g., a keyboard or keys in a mobile device, laptop, or other computing device). The user interface 160 can be used for real-time applications (RTA), such as multimedia streaming, video conferencing, navigation, video games, etc.

[0032] The network interface 170 may be configured to transmit and receive instructions and files for graphics rendering (e.g., using Wi-Fi, Bluetooth, Ethernet, etc.). In a specific example, the network interface 170 may be configured to compress or downsample images for transmission or further processing. The network interface 170 may be configured to send one or more images to a server for optical character recognition (OCR). The processor 130 may be coupled to the user interface 160, the network interface 170, and / or other interfaces and be configured to communicate between the user interface 160, the network interface 170, and / or other interfaces.

[0033] In one example, the processor 130 and the GPU 132 may be configured to perform steps for rendering video graphics, which may include steps related to executable instructions stored in the memory 150. The processor 130 may be configured to execute application instructions and generate a plurality of graphic data associated with a 3D scene including at least a first object. The plurality of graphic data may include a plurality of vertex data associated with a plurality of vertices in the 3D scene (e.g., for each object). The GPU 132 may be configured to generate a plurality of primitive data using at least the plurality of vertex data and a vertex shader. The plurality of primitive data may include position data and material data. In addition, the GPU 132 may be configured to determine a roughness value of the first object using rasterization of at least the plurality of vertex data and material attributes and texture coordinates associated with at least the first object.

[0034] In one example, GPU 130 can be configured to project a primary ray onto a first point of a first object through a first pixel in a first direction. Alternatively, GPU 130 can be configured to skip primary ray projection by using a primary hit calculated from rasterized interpolated data. GPU 132 can also be configured to project a secondary ray from a first point or a primary hit point within a fragment shader. Then, GPU 132 can be configured to calculate a Fresnel value associated with the first point and the primary ray in the fragment shader. GPU 132 can be configured to use at least the Fresnel value and a roughness value to determine whether to reflect or refract the secondary ray from the first point or the primary hit point using the fragment shader. GPU 132 can be configured to color a first pixel covering the first point or the primary hit point based at least on the secondary ray and then store the first pixel in memory 140.

[0035] Other implementations of the system include corresponding computer systems, devices, and computer programs recorded on one or more computer storage devices, each computer program being configured to perform the operations of the method. More details of the method will be discussed with reference to the following figures.

[0036] Figure 2 is a simplified flowchart showing a conventional forward pipeline 200 for rendering video graphics. The forward pipeline 200 (i.e., including a vertex shader 210 and a subsequent fragment shader 220) is shown as such. During the forward pipeline rendering process, the CPU provides graphic data of a 3D scene (e.g., from memory, storage, network, etc.) to a graphics card or GPU. In the GPU, the vertex shader 210 transforms an object in the 3D scene from object space to screen space. This process includes projecting the geometry of the object and decomposing the geometry into vertices, and then transforming and splitting the vertices into fragments or pixels. In the fragment shader 220, these pixels are colored (e.g., color, lighting, texture, etc.) before being passed to a display (e.g., the screen of a smartphone, tablet, VR glasses, etc.). In the case of lighting, the rendering effect is processed for each vertex and for each fragment in the visible scene for each light source.

[0037] Figure 3FIG. 0 is a simplified flow chart showing a conventional latency pipeline 300 for rendering video graphics. Here, the prepass process 310 involves receiving graphic data of a 3D scene from a CPU and generating a geometry buffer (G-buffer) with data required for subsequent rendering processes, such as color, depth, normal, etc. The ray tracing reflection pass 320 involves processing the G-buffer data to determine reflections of the scene, and the ray tracing shadow pass 330 involves processing the G-buffer data to determine shadows of the scene. Then, the denoising pass 340 removes noise from the ray-traced pixels and blends the non-ray-traced pixels. In the main shading pass 350, reflections, shadows, and material evaluation are combined to produce a shaded output with the color of each pixel. In the post-pass 360, the shaded output undergoes additional rendering processes, such as color grading, depth of field, etc.

[0038] Compared to the forward pipeline 200, the latency pipeline 300 processes rendering effects only based on unoccluded pixels, thus reducing the total number of fragments. This is achieved by breaking down the rendering process into multiple stages (i.e., passes), where the color, depth, and normal of objects in the 3D scene are written into separate buffers and then rendered together to generate the final rendered frame. When executing more complex lighting shaders, subsequent passes use depth values to skip rendering of occluded pixels. Compared to the forward rendering pipeline method, the latency rendering pipeline method reduces the complexity of any single shader, but having multiple rendering passes requires greater memory bandwidth, which is a more serious problem for many current architectures with limited and shared memory.

[0039] Embodiments of the present invention provide methods and systems for graphics rendering that implement one or more optimization techniques to achieve real-time performance while minimizing memory bandwidth load. In a specific embodiment, the present invention provides a method and system using noiseless unified path tracing that combines embedded ray tracing effects with a real-time approximation of physically based rendering. These techniques can be executed by a GPU configured within a graphics rendering system (e.g., the GPU 132 of the system 100 in Figure 1 ). The rendering system can be configured as a mobile device, such as a smartphone, a tablet, a VR headset, etc. Those of ordinary skill in the art will recognize many variations, alternatives, and modifications. For example, the techniques described herein can be implemented individually according to the application scenario, or combined with each other and / or implemented using other conventional techniques.

[0040] According to an example, the present invention implements a secondary ray optimization technique. The primary ray renders the target scene directly perceived by the camera, while the secondary ray can be configured to render indirect light effects (such as reflections, refractions, shadows, etc.). Depending on the application, such indirect light effects can be rendered in a way that reduces the ray casting cost, such as secondary ray casting, tertiary ray casting, etc.

[0041] In one example, the ray cost can be reduced by using a specified cut-off threshold, which is based on a predetermined criterion of material properties - for example, roughness cut-off based on the roughness material property - to limit the number of ray casts. This technique can achieve skipping ray casts, but because it is limited by the given material value, this technique does not support configuring ray casts to be rendered in a cheaper way. However, the present invention can include tracking the minimum and / or maximum material property values along the entire path of the ray cast and using these values to determine the required rendering quality. For example, the maximum roughness along the entire path can be used to determine the minimum necessary rendering quality. Using such a path cut-off threshold can reduce the quality to the minimum necessary for the entire path. Depending on the application, different material properties can use separate cut-off thresholds, or some properties can share the same cut-off threshold. Other material properties can include metallicity, opacity, glossiness, index of refraction (IOR), etc.

[0042] In recursive ray tracing, the previous material properties (such as roughness) are stored to properly sample the correct environment map mip in case the ray misses. As mentioned before, the stored material property values can be replaced with a specified cut-off threshold of the material property value (e.g., the maximum roughness along the path) to skip certain rendering features. In this case, the stored maximum or minimum material property values can still be used for environment map mip sampling as needed. The present invention can also use a stochastic path tracing pipeline to implement subsurface scattering in opaque objects and volume backscattering in transmissive objects, which can also be disabled according to a specified cut-off threshold (e.g., the maximum roughness along the path indicates that the required quality for obtaining the final image does not require processing secondary ray casts from rough objects to smooth objects).

[0043] In one example, in the case of global illumination on a rough surface, the color calculation of secondary ray casting can be simplified by using methods such as simple opacity attenuation Lambertian diffuse reflection calculation. By passing the roughness value of the previous hit into the new ray casting, the new ray process can make decisions based on situational awareness to skip advanced effects, such as subsurface scattering, reflection, refractive transmission, etc. Since the previously obtained roughness value is sampled at the correct level of detail for the reflection map in the case of a ray miss, there is no additional cost to overload the roughness information in the ray payload space, thus adaptively skipping certain rendering effects.

[0044] By using a unified rendering architecture (e.g., using a monolithic shader), additional ray optimizations can be achieved, thereby reducing bandwidth costs. The monolithic shader code implements the recursive rendering function in an embedded configuration rather than in the form of separate ray tracing shaders. Using the aforementioned cut-off threshold, the rendering process of certain effects can be handled by different rendering processes (e.g., rasterization, ray tracing, path tracing, etc.). More details will be discussed below.

[0045] According to one example, the present invention implements noise-free rendering and unified path tracing in a single shader. These functions can make each ray casting more flexible and contextually adjustable. The unified path tracing technique involves calculating all rendering effects (e.g., lighting effects) in the same overall shader. For a given 3D scene, the path tracing process can be executed in a single recursive function call and repeated for each pixel to generate a rendered frame. In addition, the embedded ray tracing can be configured to use such a single recursive function to cast rays from within a vertex, fragment, or compute shader.

[0046] In one example, a basic rasterization pipeline can be used to calculate the main rendering data (e.g., the main hit data for each pixel) for each pixel of the rendered frame. The vertex shader can extract rendering attributes from the 3D scene for rasterization, such as world space position, world space normal, world space tangent, UV map, material ID, etc. The fragment shader can calculate the direct illumination of the main hit using the material ID and schedule a process to calculate shadows (e.g., rayQuery call). The fragment shader can also schedule a recursive calculation process (e.g., rayQuery call) for the indirect illumination of a predetermined material (e.g., glass, mirror, diffuse material, etc.).

[0047] The single shader architecture provides opportunities for data reuse between different lighting effects without incurring the bandwidth cost (requiring access to main memory) when transferring data between rendering passes, as is common in the hybrid rendering architectures of desktop applications. By reducing the number of rendering passes, such an architecture is particularly suitable for the block-based memory architectures commonly used in mobile devices. Additionally, since all effects are rendered in the same shader, it is much easier to optimize the shader code by retaining shared data only for the time required to complete the effect calculations using that data.

[0048] While an effect-per-pass rendering pipeline can only globally enable effects (or globally for ray tracing as described previously), embodiments of the present invention can effectively disable effects for each object and each bounce based on previous roughness or other material property values. In the case of shadow rendering, while the cost is still one ray cast per pixel, the shading of that ray can be completed more quickly. For example, at the second boundary of a smooth reflector or refractor, it may still be necessary to capture the material model of the reflected or refracted object, but the required quality level can be different from that required for shading the primary hit. Even when ray traced shadows are enabled, the rendering process can resume from secondary rays as rasterized shadows. This reduces the cost of shadows from one ray cast per pixel to one texture sample per pixel.

[0049] The noiseless rendering technique involves configuring the rendering pipeline to use deterministic ray casting. In a specific example, the noiseless rendering pipeline is configured to only support delta lights (also known as precise lights) with deterministic sampling. Delta light sources can include directional lights, point lights, spotlights, etc. Such a noiseless pipeline can also reduce costs by reducing the quality of secondary rays. For example, recursive function calls for shadows (such as rayQuery) can calculate the rendering using only rasterized shadows and only cast transmission rays for glass, except for the case of total internal reflection. In one example, shadows can be calculated based on rasterized shadows stored in a shadow map.

[0050] In one example, a distribution function can be used to calculate the direct illumination of an object within a roughness value range with a roughness cutoff, such as a physically based microfacet bidirectional reflectance distribution function (BDRF) using the GGX distribution function, the Smith geometric shadow term, and the Schlick approximation of the Fresnel term. On the other hand, there are challenges in real-time calculating indirect illumination using such distribution functions because the direction of indirect ray casting changes with the position of the rendering point.

[0051] Indirect illumination of materials above a roughness threshold can be achieved by using image-based lighting or environment maps, using the split-sum approximation. In one example, the split-sum approximation uses a pre-filtered environment map and a two-dimensional (2D) BRDF look-up table (LUT) and only two texture lookups are used at runtime to calculate the reasonable reflection of the environment map in objects with different roughness values. With such an environment map, it is also possible to re-render individual reflection probes with the scene geometry to account for geometries that may occlude the environment map or any lights in the scene.

[0052] Although the previous examples were discussed in the context of material roughness for rendering illumination, these techniques can also be applied to other material properties, such as transparency. In one example, the split-sum approximation can be applied to transparency rendering. After reflecting the incident direction across the entire view plane, this direction can be used to sample the environment map. The same LUT values can be used to calculate the reflection of the environment map on the back side. Additionally, the calculated reflection can be multiplied by the albedo (i.e., the characteristic color of the object) and blended with the diffuse term based on opacity to serve as a rough transmission contribution. Of course, there can be other variations, modifications, and alternatives.

[0053] For real-time path tracing, a noiseless design achieves suitable rendering results within a single frame. Using rays projected in deterministic, non-random directions, the rendering process can be configured to achieve smooth reflection and transmission ray tracing effects. The rendering process can include rasterization to approximate low-frequency effects, such as rough reflection, rough transmission, soft shadows, etc. Additionally, since the noiseless design does not require denoising or accumulation processing, the rendering process reduces the need to read intermediate rendering results from memory, thereby reducing the memory bandwidth load, which is particularly suitable for mobile devices.

[0054] According to one example, the present invention employs a hybrid Fresnel split to handle the combined rendering of reflection and refraction, rather than managing separate channels for each effect. In the case of rendering smooth transmissive objects, rays can be projected in the reflection and refraction directions and processed recursively. However, this incurs an exponential per-pixel performance cost. Even limiting the maximum bounce count per pixel is not sufficient to manage the cost for mobile applications. To reduce this cost, the rendering process can randomly select between the reflection and refraction directions based on the per-pixel Fresnel value.

[0055] In one example, the technique uses a cutoff threshold to determine whether to cast a ray or use a non-ray-traced approximation (e.g., color) of the raycast result. In the approximation case, sample data can be measured to generate an approximation of the desired effect. For example, a reflection probe can be used to provide a low-resolution cube map of the object's surrounding environment, which can be sampled to provide an approximation of the transmitted or reflected color. As previously mentioned, the cutoff values for transmission and reflection can be controlled separately, or the two effects can share the same cutoff value.

[0056] In the case of rendering glass, the rendering method can define an additional cutoff value that is used to compare with the Fresnel term of each pixel to avoid casting two rays per pixel. For Fresnel values above the cutoff, the method can include casting the reflected ray and sampling the cube map in the refraction direction. When below the cutoff, the method can include casting the refracted ray and sampling the cube map (i.e., environment map) in the reflection direction.

[0057] To mitigate the harsh visual cutoff corresponding to this Fresnel cutoff, the method can attenuate the reflected and refracted ray casts based on the absolute difference between the Fresnel term of the pixel and the Fresnel cutoff. This attenuation can make the transition at the cutoff point smoother, but at the cost of making the reflections and refractions in the final image appear more blurred. In a specific example, the glass rendering recursive process can also include following the transmission path, except in the case of total internal reflection, which maintains the per-ray cost of the hybrid Fresnel split without requiring any skybox sampling or blending.

[0058] Interactive-rate path tracers accumulate images over multiple frames. Thus, they can randomly choose between reflection and refraction for each pixel of each frame image and smooth the result through the accumulation step. However, for real-time noiseless path tracers, both the reflection and refraction images must be available from the first frame and contain no randomized noise. Combining a noiseless unified path tracer architecture and secondary ray optimization, the rendering process using hybrid Fresnel split can use one ray cast and two environment map (e.g., skybox or reflection probe) samples to handle smooth transmissive objects.

[0059] Figure 4 FIG. is a simplified flowchart showing a noiseless unified path tracing rendering pipeline 400 according to an embodiment of the present invention. This figure is merely an example and should not unduly limit the scope of the claims. Those of ordinary skill in the art will recognize many variations, alternatives, and modifications.

[0060] As shown in the figure, the path tracing pipeline 400 includes a pre-z (depth pre-pass) vertex shader 410, followed by a vertex shader 420 and a fragment shader 430 corresponding to the overall noiseless rendering pass. This rendering pipeline 400 is configured as a noiseless unified path tracer that uses depth pre-pass, secondary ray optimization, and hybrid Fresnel splitting techniques. By executing the minimal vertex shader 410 to output the depth buffer in advance, the pipeline takes advantage of an important performance improvement of the deferred pipeline, which allows occluded fragments in subsequent global illumination passes to be skipped when the depth test fails. As previously mentioned, all lighting and shadow effects are executed in a single global fragment shader 430, and the rendered graphics are generated without denoising or accumulation by using deterministic ray casting. To reduce the cost of recursive ray tracing procedure calls, the pipeline 400 is configured to use a material property cutoff threshold (e.g., maximum roughness) to render non-primary ray casting procedures, adaptively disabling the effects for each object and each ray casting bounce to achieve the desired rendering quality. In addition, the pipeline 400 implements hybrid Fresnel splitting to approximate the appearance of light splitting at transmissive surfaces without casting two rays per pixel.

[0061] In one example, the present invention provides a video graphics rendering system (such as Figure 1 system 100 in), the system is configured to implement a noiseless unified path tracing pipeline 400. This can be a mobile device (e.g., smartphone, tablet, VR glasses, etc.) having at least one processor configured to execute the foregoing methods using executable code stored in a memory that stores instructions for executing these methods. Refer to Figure 5 and Figure 6 for an example method flow of the operation of such a rendering system.

[0062] Figure 5 is a simplified flowchart showing a method of rendering refractive transparent objects in real-time video graphics using ray tracing according to an embodiment of the present invention. This figure is merely an example and should not unduly limit the scope of the claims. Those of ordinary skill in the art will recognize many variations, alternatives, and modifications. For example, one or more steps can be added, deleted, repeated, replaced, modified, rearranged, and / or overlapped, which should not limit the scope of the claims.

[0063] According to one example, it can be a rendering system, such as Figure 1In system 100, method 500 for rendering an object is executed. More specifically, the processor of the system can be configured to execute the actions of method 500 through executable code stored in the system's memory (e.g., permanent memory). As shown, method 500 may include step 502 of receiving a plurality of graphics data associated with a 3D scene that includes at least a first object (or includes all object instances in the scene). This may include determining all data required for the first object intersected by a ray projected through each pixel in a viewport, including the plurality of graphics data that includes at least a plurality of vertex data associated with a plurality of vertices in the 3D scene. In one example, the method further includes generating a plurality of primitive data using at least the plurality of vertex data and a vertex shader. The plurality of primitive data may include at least position data and material data.

[0064] In step 504, the method includes rasterizing using at least the plurality of vertex data and material properties and texture coordinates associated with at least the first object to determine a roughness value of at least the first object visible at each pixel. In a specific example, method 500 further includes interpolating rendering data by rasterizing objects visible to the camera to replace casting a primary ray. This includes using rasterization to calculate the direction of an implicit primary ray passing through a pixel and the first intersection position of the ray with the first object. In a specific example, the method further includes storing the maximum roughness value on the hit path associated with the implicit primary ray.

[0065] In step 506, the method includes casting at least secondary rays from the first intersection position within a fragment shader. The method may also include determining whether to cast secondary rays (i.e., secondary ray optimization) by comparing the roughness value with a predetermined threshold roughness value. Additionally, secondary rays can be cast in the reflection or refraction direction from the first intersection position. In one example, the vertex shader and the fragment shader can be processed using a GPU (e.g., Figure 1 GPU 132 in system 100). In a specific example, hardware-accelerated inlined tracing is used to cast secondary rays.

[0066] In step 508, the method includes calculating a Fresnel value associated with the first intersection position and the implicit primary ray in a fragment shader. In step 510, the method includes determining whether to skip, reflect, or refract a secondary ray from the first intersection position using a fragment shader based on at least the Fresnel value and the roughness value. In a specific example, determining whether to skip, reflect, or refract a secondary ray may include comparing the Fresnel value with a predetermined threshold Fresnel value (i.e., mixing Fresnel beam splitting).

[0067] In step 512, the method includes using an environment map associated with the 3D scene to provide visual continuity between the reflected image and the refracted image. In one example, the method includes sampling the environment map to provide a consistent intermediate image for blending the reflected image and the refracted image. In one example, the method includes attenuating the reflected and refracted ray casts after blending the Fresnel split to achieve the combined rendering and refraction of smooth transmissive objects.

[0068] In step 514, the method includes shading at least a first pixel covering a first intersection position based on at least secondary rays. In step 516, the method includes storing at least the first pixel in a frame buffer. In one example, the frame buffer is configured in memory, such as Figure 1 the memory 140 in the system 100. Additionally, method 500 includes transforming the 3D scene to screen space, such as Figure 1 the display 162 of the system 100.

[0069] Figure 6 is a simplified flowchart showing a method for generating a video with ray tracing according to an embodiment of the present invention. This figure is merely an example and should not unduly limit the scope of the claims. Those of ordinary skill in the art will recognize many variations, alternatives, and modifications. For example, one or more steps can be added, deleted, repeated, replaced, modified, rearranged, and / or overlapped, which should not limit the scope of the claims.

[0070] According to one example, the method of generating a video can be performed by a rendering system, such as Figure 1 the system 100 in. More specifically, the processor of the system can be configured to perform the actions of method 600 through executable code stored in the system's memory storage (e.g., permanent memory). As shown, method 600 can include step 602 of generating a 3D scene including at least a first object (or including all object instances in the scene). In one example, the 3D scene is generated by a CPU, such as Figure 1 the CPU 130 of the system 100 in. In step 604, the method includes receiving a plurality of graphic data associated with the 3D scene. The plurality of graphic data includes a plurality of vertex data associated with a plurality of vertices in the 3D scene.

[0071] In step 606, the method includes rasterizing using at least the plurality of vertex data and the material properties and texture coordinates associated with at least the first object to determine the roughness value of at least the first object. This data is used to calculate the primary ray direction and the hit position without casting the primary ray. In step 608, the method includes casting secondary rays from a first intersection position between the first object and an implicit primary ray determined by rasterization within a fragment shader.

[0072] In step 610, the method includes calculating a Fresnel value associated with a first intersection position and an implicit primary ray in a fragment shader. In step 612, the method includes determining whether to skip, reflect, or refract a secondary ray from the first intersection position using the fragment shader based at least on the Fresnel value and a roughness value. In step 614, the method includes shading at least a first pixel covering the first intersection position based at least on the secondary ray.

[0073] Although the specific embodiments have been described in full above, various modifications, alternative constructions, and equivalents may be used. Accordingly, the above description and illustration should not be taken as limiting the scope of the invention defined by the appended claims.

Claims

1. A method for rendering refractive transparent objects in real-time video graphics using ray tracing, the method comprises: receiving a plurality of graphics data associated with a three-dimensional (3D) scene including at least a first object, the plurality of graphics data including a plurality of vertex data associated with a plurality of vertices in the 3D scene; using at least the plurality of vertex data and a vertex shader to generate a plurality of primitive data, the plurality of primitive data including position data and material data; using at least the plurality of vertex data and rasterization of material properties and texture coordinates associated with at least the first object to determine a roughness value of the first object; projecting a secondary ray from a first intersection position between the first object and an implicit primary ray determined by the rasterization within a fragment shader; calculating a Fresnel value associated with the first intersection position and the implicit primary ray in the fragment shader; based on at least the Fresnel value and the roughness value, using the fragment shader to determine whether to skip, reflect, or refract the secondary ray from the first intersection position; using an environment map associated with the 3D scene to provide visual continuity across reflected and refracted images; coloring the first pixel covering the first intersection position based on at least the secondary ray; and storing the first pixel in a frame buffer.

2. The method according to claim 1, wherein, the secondary ray is projected using hardware-accelerated embedded ray tracing.

3. The method according to claim 1, further comprises: interpolating rasterization of objects visible to a camera, the objects including the first object.

4. The method according to claim 1, further comprises: determining whether to project the secondary ray by comparing the roughness value with a predetermined threshold roughness value.

5. The method according to claim 1, further comprises: storing a maximum roughness value on a hit path associated with the implicit primary ray.

6. The method according to claim 1, further comprises: determining whether to reflect or refract the secondary ray by comparing the Fresnel value with a predetermined threshold Fresnel value.

7. The method according to claim 1, wherein, the secondary ray is projected in a refraction direction from the first intersection position.

8. The method according to claim 1, wherein, the vertex shader and the fragment shader are processed using a graphics processor.

9. A system for rendering video graphics, the system comprises: a memory including executable instructions; a memory; and a processor coupled to the memory and the memory, the processor being configured to: generate a plurality of graphics data associated with a three-dimensional (3D) scene including at least a first object, the plurality of graphics data including a plurality of vertex data associated with a plurality of vertices in the 3D scene; using at least the plurality of vertex data and a vertex shader to generate a plurality of primitive data, the plurality of primitive data including position data and material data; Determine the roughness value of the first object using rasterization of at least the plurality of vertex data and material properties and texture coordinates associated with at least the first object; Project a secondary ray from a first intersection position between the first object and an implicit primary ray determined by the rasterization within a fragment shader; Calculate a Fresnel value associated with the first intersection position and the implicit primary ray in the fragment shader; Based on at least the Fresnel value and the roughness value, use the fragment shader to determine whether to skip, reflect, or refract the secondary ray from the first intersection position; Color the first pixel covering the first intersection position based on at least the secondary ray; And Store the first pixel in a frame buffer.

10. The system according to claim 9, Wherein, The processor includes a central processing unit (CPU) and a graphics processing unit (GPU).

11. The system according to claim 10, Wherein, The memory is shared by the CPU and the GPU.

12. The system according to claim 10, Wherein, The memory includes a frame buffer for storing the first pixel.

13. The system according to claim 10, further Comprises: A display configured to display the first pixel at a refresh rate of at least 24 frames per second.

14. A method for generating a video with ray tracing, the method Comprises: Generate a three-dimensional (3D) scene including at least a first object; Receive a plurality of graphics data associated with the 3D scene, the plurality of graphics data including a plurality of vertex data associated with a plurality of vertices in the 3D scene; Use at least the plurality of vertex data and a vertex shader to generate a plurality of primitive data, the plurality of primitive data including position data and material data; Determine the roughness value of the first object using rasterization of at least the plurality of vertex data and material characteristics and texture coordinates associated with at least the first object; Project a secondary ray from a first intersection position between the first object and an implicit primary ray determined by the rasterization within a fragment shader; Calculate a Fresnel value associated with the first intersection position and the implicit primary ray in the fragment shader; Based on at least the Fresnel value and the roughness value, use the fragment shader to determine whether to skip, reflect, or refract the secondary ray from the first intersection position; And Color the first pixel covering the first intersection position based on at least the secondary ray.

15. The method according to claim 14, further Comprises: Access an environment map associated with the 3D scene.

16. The method according to claim 14, further Comprises: Use the environment map associated with the 3D scene to provide visual continuity across reflected and refracted images.

17. The method according to claim 14, Wherein, The 3D scene is generated using a central processing unit.

18. The method according to claim 17, Wherein, The vertex shader and the fragment shader are processed using a graphics processing unit.

19. The method according to claim 18, wherein, the central processing unit and the graphics processing unit share memory.

20. The method according to claim 14, further comprising: converting the 3D scene to screen space.

Images