Real-time, software-based hybrid ray tracing for battery-powered computing devices

Abstract

Hybrid ray tracing or reduced computational complexity. Primary rays are used to construct an image using rasterization techniques. Independently, secondary rays that are traced for reflections and other photo-realistic features are generated in the object space of the scene and traced. Dynamic acceleration structures (DAS) and traversal tools are characterized by high object locality, reduced intersection tests, and low cost updates. Coherence is achieved by processing secondary rays that have a common origin and a common endpoint. The resulting reduced computational complexity enables a software-only implementation without the need for hardware accelerators.

Description

[0001] Cross-reference to related applications

[0002] This application claims priority to U.S. Application No. 17 / 019,272, filed September 13, 2020, entitled “METHOD for PHOTOREALISTIC REFLECTIONS in NON-PLANAR REFLECTIVE SURFACES,” and U.S. Provisional Application No. 63 / 067,881, filed August 20, 2020. U.S. Application No. 17 / 019,272 is a continuation of U.S. Application No. 16 / 874,705, filed May 15, 2020, entitled “CREATING COHERENT SECONDARY RAYS FOR REFLECTIONS IN HYBRID RAY TRACING,” which is a continuation of U.S. Application No. 16 / 874,705, filed April 9, 2020, entitled “METHOD FOR…”. This is a continuation of U.S. Application No. 16 / 844,681, entitled “Constructing and Traversing Accelerating Structures,” which claims priority to U.S. Provisional Application No. 62 / 957,157, filed January 4, 2020. All three U.S. applications and two U.S. provisional applications are incorporated herein by reference.

[0003] This application relates to U.S. Application No. 16 / 662,657, filed October 24, 2019, and published February 20, 2020, under the name US 2020 / 0058155, entitled "Method for Non-Planar Specular Reflections in Hybrid Ray Tracing," which teaches a real-time hybrid ray tracing method for non-planar specular reflections. The high complexity of non-planar surfaces is reduced to the low complexity of multiple small planar surfaces. The planar property of triangles comprising building blocks of non-planar surfaces is utilized. All secondary rays bouncing from a given surface triangle toward an object triangle maintain a direction close to each other. Collective control of secondary rays is achieved through this proximity and by decoupling secondary rays from primary rays. The result is high coherence of secondary rays. US 2020 / 0058155 is incorporated herein by reference. Technical Field

[0004] This invention relates to generating photorealistic real-time reflections from non-planar surfaces using hybrid ray tracing for advanced visual quality in video games, VR, AR, etc. Background Technology

[0005] Ray tracing is a computer graphics technique that produces a high degree of visual realism compared to typical raster methods, but it is computationally expensive. Ray tracing surpasses raster graphics due to its ability to simulate various optical effects, such as gloss, specular reflection, radiolucency, reflection and refraction, scattering, soft shadows, etc. True photorealism occurs when the rendering equation is very close to or fully implemented. Implementing the rendering equation gives true photorealism because the equation describes every physical effect of light flow. However, this depends on available computational resources. Path tracing—also known as Monte Carlo ray tracing—is physically correct ray tracing. Path tracing provides an accurate simulation of real-world light. A traditional ray tracer [Kajiya, JT1986. The rendering equation. In Proc. SIGGRAPH] emits rays through each pixel, scattering randomly according to the reflectivity distribution of intersecting objects and continuing recursively until a light source is hit. Repeated sampling for any given pixel in the image space eventually causes the average of the samples to converge to the correct solution of the rendering equation, making it one of the most physically accurate 3D graphics rendering methods available. Ray tracing in its current form is one of the most computationally complex applications. Therefore, it is best suited for applications where images can be rendered slowly in advance, such as as still images and movie or television visual effects, rather than for real-time animated augmented reality applications where real-time animation is crucial.

[0006] Hybrid ray tracing (ray tracing interleaved with raster rendering) is a deferred rendering process that calculates primary ray collisions based on raster rendering, while secondary rays use ray tracing methods to achieve shadow, reflection, and refraction effects. This approach significantly improves ray tracing performance, not only by avoiding many unnecessary traditional ray tracing tasks, but also by obtaining a complete image within the required timeframe, even if there isn't enough time to complete all visual effects calculations. This feature is valuable in real-time-critical video games, VR, and AR, where quality may be sacrificed for performance.

[0007] The concept of a hybrid real-time raster and ray tracing renderer is not new. Beck et al. [Beck, S., C. Bernstein, A., Danch, D., Frohlich, B.: CPU-GPU hybrid real-time raytracing framework (2005)] proposed a CPU-GPU real-time ray tracing framework. Beck proposed distributing the traditional ray tracing stages into separate tasks for the GPU and CPU. These rendering tasks can be summarized into three GPU rendering passes: shadow map generation, geometry recognition, and blurring.

[0008] Bikker [Bikker, J.: Real-time ray tracing through the eyes of a game developer. In: Proceedings of the 2007 IEEE Symposium on Interactive RayTracing, Washington, DC, USA, IEEE Computer Society (2007)] developed a real-time path tracer called Brigade, which can seamlessly partition rendering tasks across both available GPU and CPU cores. Brigade was designed to create a proof-of-concept game that uses path tracing as its primary rendering algorithm.

[0009] Pawel Bak [Bak, P.: Real time ray tracing. Master's thesis, IMM, DTU (2010)] implemented a real-time ray tracer using DirectX 11 and HLSL. Similar to Beck's work, Pawel Bak's method also uses rasterization to achieve the best possible performance for primordial shots.

[0010] Chen [Chen, CC, Liu, DSM: Use of hardware z-buffered rasterization to accelerate ray tracing. In: Proceedings of the 2007 ACM symposium on Applied computing. SAC'07, New York, NY, USA, ACM (2007) 1046-1050] proposed a hybrid GPU / CPU ray tracing renderer in which Z-buffered rasterization is performed to determine visible triangles while determining the intersection of primary rays. The CPU reads back the data to trace secondary rays.

[0011] Sabino et al. [Thales Sabino, Paulo Andrade, Esteban Gonzales Clua, Anselmo Montenegro, Paulo Pagliosa, A Hybrid GPU Rasterized and Ray Traced Rendering Pipeline for Real Time Rendering of Per Pixel Effects, Univ. Federal Fluminense, Rio de Janeiro, Brazil, 2013] proposed the following heuristic method, which selects a subset of relevant objects to be ray-traced, thereby avoiding traversing rays for objects that may not significantly contribute to the real-time experience.

[0012] A key strategy in real-time hybrid ray tracing is to use GPUs for rasterization to improve performance and to employ intelligent strategies to prioritize regions and objects that will undergo ray tracing effects. NVIDIA's OptiX [Parker,SG,Bigler,J.,Dietrich,A.,Friedrich,H.,Hoberock,J.,Luebke,D.,McAllister,D.,McGuire,M.,Morley,K.,Robison,A.,Stich,M.:Optix:A generalpurpose ray tracing engine.ACM Transactions on Graphics (August 2010)] is a general-purpose ray tracing engine, currently available for both NVIDIA GPUs and general-purpose hardware. The OptiX architecture provides a low-level ray tracing engine, a programmable ray tracing pipeline, a CUDAC / C++-based shader language, a domain-specific compiler, and scene graph representation. OptiX is a GPU-only solution with very good results in interactive ray tracing. Recently, Nvidia OptiX has gained support from Nvidia RTX, a development platform for dedicated hardware for hybrid ray tracing. Nvidia OptiX runs on Nvidia Volta and Turing GPUs and specifically leverages architectures designed for ray tracing acceleration.

[0013] While all forms of hybrid ray tracing are developing, hybrid real-time ray tracers for low-power devices are currently unavailable. The applicability of hybrid real-time ray tracers to low-power devices such as laptops, tablets, and handheld mobile devices is becoming increasingly important. It is predicted that the possibility of running ray tracing on low-power devices will not be available until the 1930s: “Based solely on Moore’s Law, real-time ray tracing will not be possible on mobile phones until 2032.” Jon Peddie, TechWatch, March 27, 2018.

[0014] Real-time Ray Tracing (RTRT). Historically, ray tracing has been reserved for offline applications, such as computer-generated photorealistic animated films. Real-time applications in video games, virtual reality, and augmented reality have had to rely on rasterization for rendering. RTRT is a computationally difficult task, not only because each pixel in the image must be calculated separately, but also because the final color of a single pixel can be influenced by more than one recursive ray. Another consideration is that ray tracing algorithms waste 75% to 95% of their execution time calculating the intersection points between rays and objects. RTRT was enabled by Nvidia's RTX in 2018 (Alwani, Rishi. "Microsoft and Nvidia Tech to Bring Photorealistic Games with Ray Tracing". Gadgets 360). https: / / gadgets.ndtv.com / iaptops / news / microsoft-dxr-nvidia-rtx-ray-tracing-volta- gpL3-metro-exodus-1826988 (Retrieved March 21, 2018), it has facilitated a new development in computer graphics, namely the generation of interactive images that react to light, shadows, and reflections through dedicated hardware. Nvidia's RTX is based on traditional ray tracing algorithms accelerated by on-chip supercomputing hardware with nearly 5,000 cores. Nvidia's RTX includes: a GPU with 4,352 cores, an AI denoiser using 544 cores, and an intersection test accelerator with 68 cores. A single RTX 2080 GPU requires 250W of power and starts at €418. Due to its high cost and power consumption, RTX is targeted at high-end video games.

[0015] For video games, virtual reality, and augmented reality, there is a strong need to enable features that can be transmitted solely through ray tracing, such as reflections, especially those from non-planar surfaces that cannot be achieved through rasterization techniques or workarounds. Reflections must be generated in real-time. Therefore, the gaming industry desperately needs new hybrid real-time ray tracing technologies based on radical algorithmic improvements.

[0016] Reflection. In existing hybrid ray tracing techniques, reflection is generated based on the G-buffer (Luis Sabino et al., A Hybrid GPU Rasterized and Ray Traced Rendering Pipeline for Real Time Rendering of Per Pixel Effects, 2013). The G-buffer is generated during the first stage of raster rendering, i.e., the "different shading" stage. The basic idea behind deferred shading is to perform all visibility tests before performing any lighting calculations. Therefore, visibility tests are first done via raster rendering, while shading later is different and is combined with ray tracing. The G-buffer generated by the deferred shading stage contains information about the optical properties of the underlying material for each pixel. Its contents are used to determine the need to trace reflected / refracted rays. Its contents consist of reflectivity, refractive index, specular index, and opacity, respectively. Rays need to be traced from the surface only through the scene. This approach makes it possible to avoid tracing unnecessary rays in places where the material is neither refractive nor reflective. After the different shading is completed, the ray tracing algorithm starts with secondary rays and can follow its own path. Any secondary rays generated will be traced across the scene to produce global illumination effects such as reflection and refraction. The result of this stage can be understood as the generation of a ray-traced effect layer. This effect layer will be blended into the already generated image to leverage global illumination effects and enhance its visual quality.

[0017] According to the G-buffering method, secondary rays are natural extensions of primary rays. Ray tracing using selected secondary rays presents the same challenges as conventional ray tracing: the lack of coherence in the secondary rays and images with random noise.

[0018] Reuven Bakalash described the generation of fast reflections in U.S. Patent 10,565,776: "Method for Fast Generation of Path Traced Reflections on a Semi-Reflecting Surface." This U.S. patent disclosure describes a novel global illumination ray tracing method suitable for augmented reality and virtual reality. Prior art acceleration structures are replaced by a novel device that performs intersections between secondary rays and scene geometry in large ray sets, resulting in higher speed and reduced computational complexity. Its reduced power consumption is suitable for consumer-level computing devices.

[0019] Accelerated Structures. The most time-consuming task in ray tracing is the intersection testing between millions of rays and millions of polygons. This intersection testing is partially alleviated by using an accelerated structure (AS): a huge binary tree specifically constructed for scene space. Each ray traverses the accelerated structure (e.g., a K-tree or BVH tree) to find intersecting polygons. These traversals become the main time-consuming task—typically accounting for over 70% of image generation time.

[0020] Prior articulated image processing (AS) based on binary trees (e.g., BVH) is essentially static. Reconstruction of prior art AS is typically more time-consuming than rendering. Build time depends on scene size and polygon resolution. For example, building accelerated data structures for highly complex geometries is extremely expensive or imposes limitations on how the geometry is modeled. Hanika et al. addressed this difficulty in U.S. Patent 8,570,322, "Method, system, and computer program product for efficient ray tracing of micropolygon geometry." This U.S. patent teaches a first hierarchy of surface patches, which is then ray-traced to identify which surface patches potentially intersect. The potentially intersecting patches are then decomposed into a set of micropolygon sub-objects as needed, and a second hierarchy is built to accelerate ray tracing. Shaders operating on the second hierarchy of micropolygons can process the entire mesh of the micropolygons at once in a SIMD manner.

[0021] Small scene changes may only require updating the Action Set (AS); however, significant scene modifications necessitate rebuilding the acceleration structure. The lengthy rebuild time poses a particular challenge for large scenes. Reuven Bakalash et al., in their US patent 10,380,785 entitled "PathTracing Method Employing Distributed Acceleration Structures," address this issue through distributed acceleration structures. In this path tracing system, the traversal task is distributed between a global acceleration structure at the system's center and multiple localized and autonomously processed local acceleration structures distributed across multiple units. This reduces the centrality of this critical resource, minimizing bottlenecks and simultaneously improving parallelism.

[0022] There are two main drawbacks associated with using static accelerated structures: (i) traversing these structures is time-consuming and challenging for real-time requirements, and (ii) these structures must be repeatedly reconstructed as the scene changes, which conflicts with real-time skinned animation. Reconstructing static accelerated structures is a computationally intensive task that prevents real-time animation.

[0023] Therefore, it is necessary to address these and / or other issues associated with the acceleration structures of existing technologies.

[0024] The lack of coherence in secondary rays is a significant issue. Ray coherence is crucial for efficient parallelization of ray tracing. In existing ray tracing techniques, primary rays and shadow rays are coherent. This coherence is used for efficient parallel processing: traversal, intersection, and shading are performed by grouping coherent rays. This works well for nearby primary rays, as these rays frequently traverse similar parts of the data structure, accelerating computation. Using this approach, computation time can be reduced by using standard SIMD mechanisms of GPUs and CPUs for multiple rays in parallel; reducing memory bandwidth by requesting data only once per group; and simultaneously improving cache utilization. This works well for primary rays originating from the camera. Unfortunately, in the case of higher-order (secondary, tertiary, etc.) rays, it is not possible to efficiently use ray grouping. The main reason is that higher-order rays bounce in different directions, thus losing coherence. Furthermore, for diffuse reflection, there is intentional randomization of rays. Existing techniques practice reorganizing secondary rays to form bundles with higher coherence ratios. However, this reorganization is a rather expensive operation because it involves scattering / gathering steps, which may only result in a slight frame rate increase when reordering is applied.

[0025] Sadegi et al. [Iman Sadeghi, Bin Chen, and Henrik Wann, Coherent PathTracing, Jensen University of California, San Diego, 2009] developed a technique to improve the coherence of secondary rays. This technique uses the same sequence of random numbers to generate secondary rays for all pixels in each sample. This improves the efficiency of grouped tracking algorithms but creates structured noise patterns in the image.

[0026] Reuven Bakalash et al. solved a problem in improving coherence in ray tracing in U.S. Patent 10,410,401, entitled "Spawning Secondary Rays in Ray Tracing from Non-Primary Rays." This U.S. Patent describes a novel method for generating coherent secondary rays for global illumination ray tracing. Prior art acceleration structures are replaced by a novel device that performs intersections between secondary rays and scene geometry within large ray sets, thereby achieving higher speeds and reduced computational complexity.

[0027] Therefore, it is necessary to address these and / or other issues associated with the lack of coherence in ray tracing in existing technologies.

[0028] Noisy image. The path tracker continuously samples pixels in screen space. The image only becomes recognizable after multiple samples of each pixel. Rays are randomly distributed within each pixel in screen space, generating new reflected rays pointing in random directions at each intersection with objects in the scene. After a certain number of bounces, each ray eventually leaves the scene or is absorbed. As a ray completes its bounce in the scene, a sample value is calculated based on the object it bounced off. This sample value is added to the average of the source pixels.

[0029] The random component in ray tracing introduces noise into the rendered image. This noise decreases over time as more samples are calculated. The limiting factor for rendering quality is the number of samples per pixel (SPP). A higher SPP in the rendered image obviously results in less noise. However, the more samples available, the less quality each sample contributes (because each sample only contributes to the average of all samples).

[0030] Only by converging over many subsequent frames can the noise in the final image be reduced. For most path-tracking images, approximately 5000 samples are typically needed to achieve convergence and reduce noise to an acceptable level, while more samples are required for pathological cases. Noise is particularly problematic for animation, giving it random specks of "film grain" quality that is generally undesirable in animation.

[0031] The purpose of at least one embodiment of the present disclosure is to enable the use of real-time photo-realistic reflections in video games, VR and AR, and to address the lack of relevant technologies in the related fields. Summary of the Invention

[0032] The embodiments of this disclosure generate photorealistic reflections in non-planar reflective surfaces using hybrid ray tracing. Primary rays are used to construct an image via rasterization, thus defining the image within the viewport of the screen. Secondary rays are independently generated in the scene's object space for reflections and other photorealistic features, and these secondary rays are traced to visualize objects that fall outside the viewport but can be reflected in the image, such as objects shown in a mirror. To accommodate real-time skinned animation, a true Dynamic Acceleration Structure (DAS) is employed. A DAS is characterized by a high degree of locality between objects and sub-objects, where scene changes are updated with minimal impact on neighboring objects. Each scene object in the DAS can move independently of other objects, enabling autonomous updates at its own hierarchy without affecting its neighboring hierarchies. Therefore, the rapid reconstruction of the entire accelerated structure in the prior art is replaced by low-cost updates.

[0033] The efficiency of DAS traversal is improved by a two-step traversal; first, geometry such as a triangular pyramid is used to eliminate negative intersection tests, and then secondary rays are used to perform efficient and accurate intersection tests on the leaf hierarchy.

[0034] Secondary ray coherence is achieved by processing secondary rays that share a common origin and a common endpoint, a long-standing requirement in ray tracing. This coherence enables spatial locality of secondary rays, database locality for faster structure traversal, reduced traversal, and reduced intersection testing, thereby reducing the overall computational complexity of hybrid ray tracing.

[0035] This invention can be implemented solely in software, without requiring a hardware accelerator. It can be implemented on conventional GPUs, including integrated GPUs in mobile devices. High parallelism and high utilization of the GPU are achieved through the ray coherence of secondary rays. Attached Figure Description

[0036] The present disclosure has been described herein by way of non-limiting example only, with reference to the accompanying drawings and figures, wherein similar reference numerals denote similar elements. It is understood that these drawings are provided only with information about typical embodiments of the present disclosure and should therefore not be considered as limiting the scope.

[0037] Figure 1 This is an example setup of non-planar reflective surfaces and geometric objects.

[0038] Figure 2 It is a triangle of a non-planar reflective surface associated with a geometric object.

[0039] Figure 3 It is the projection of the object's triangle onto the triangle of the non-planar reflective surface.

[0040] Figure 4 It is a triangle of an object associated with a triangle on a non-planar reflective surface. It has no projection.

[0041] Figure 5 It is a dynamic acceleration structure of four geometric objects.

[0042] Figure 6 It is a pre-animated scene.

[0043] Figure 7 It is a skinned animation scene.

[0044] Figure 8 It is a DAS for pre-animated scenes.

[0045] Figure 9 It is a DAS for skinned animation scenes.

[0046] Figure 10 It is a truncated cone from the receiver triangle.

[0047] Figure 11 This is a step-by-step visualization process.

[0048] Figure 12 This is the step of secondary ray traversal.

[0049] Figure 13 This is a flowchart analyzing the implementation method.

[0050] Figure 14 This is a flowchart of another implementation method.

[0051] Figure 15 shows the prior art. A computing system according to an embodiment. Detailed Implementation

[0052] The objective of at least one embodiment of this disclosure is to enable photorealistic reflections in video games by addressing the lack of relevant technology in the related field. Real-time reflections from planar and non-planar surfaces, as well as smooth surfaces, are achieved through real-time hybrid ray tracing technology.

[0053] The hybrid ray tracing of this invention is based on processing secondary rays entirely in object space, rather than in image space. Primary rays are used to construct the image bounded by the image viewport through rasterization. Independently, secondary rays are generated and traced in the object space of the scene, far from the screen, for reflections and other photorealistic features. Secondary rays can be traced anywhere in the scene, including outside of screen space. This allows for reflections and the inclusion of objects falling outside the image viewport in the image.

[0054] Specular reflection from a non-planar surface. Unlike planar surfaces, non-planar surfaces lack uniform orientation. A non-planar surface is composed of triangles that are planar elements. Each triangle reflects the scene according to its normal direction. The methods of this disclosure aim to generate physically correct object reflections on non-planar surfaces. Figure 1 An exemplary setup is shown. Two target objects 11 and 12 will be reflected in a non-planar surface 10 comprising multiple receiver triangles.

[0055] According to one implementation, the triangle of each receiver is separated to generate its own physically correct reflection. The eventual aggregation of partial reflections results in aggregated nonplanar surface reflections.

[0056] Figure 2 A non-planar surface 30 comprising multiple triangles with various orientations is shown, with particular emphasis on triangle 20 to illustrate the process of generating reflections within a single triangle, named the reflective receiving triangle. Two target objects 26, 27 consist of a mesh of emitting triangles that will be reflected in the receiver. To calculate the correct reflection in the receiving triangle, the triangle's normal must first be calculated. Since the triangle is a planar solid, a set of secondary rays can be generated, all of which will coherently bounce off the triangle, as will be described further herein. The secondary rays are then emitted toward the emitting object according to the triangle's normal.

[0057] The normal of the receiver triangle is evaluated based on the normals of its three vertices. Each such vertex is shared with its neighboring triangle. Therefore, due to non-planarity, each vertex has its own normal direction. The normal direction N of the triangle is calculated by averaging the normal directions of the three vertices of the triangle.

[0058] Once the normal to the triangle is known, a mirror surface 21 passing through the midpoint of the triangle is obtained, and then a mirror eye 25 is defined, which is the reflection of the camera device 28 in the mirror surface. The mirror eye serves as the origin of the truncated cone projection. The truncated cone is shaped by the receiver triangle 20 and points towards the scene, such as objects 26 and 27. The truncated cone defines the visibility of the scene as seen from the triangle.

[0059] exist Figure 2 In the example, object 26 is fully visible, while object 27 is only partially visible. Therefore, a portion of the triangular mesh of 27 remains outside the truncated cone.

[0060] exist Figure 3An analytical implementation of the reflection generation is illustrated. A receiving triangle 30, a transmitting triangle 33, and the final transmitter reflection 34 in the receiver are shown. The task of reflecting the transmitting triangle onto the receiving triangle applies only to the visible triangles of the object mesh. A frustum visibility test identifies the emission candidates for reflection. The visibility test is described in detail below.

[0061] Once the emitter triangle 33 passes the visibility test, it is processed for reflection. The emitter triangle 33 is sampled for color, light, material, texture, etc., and projected onto the surface of the receiver through its three vertices 35, 36, and 37. Emitter triangles partially outside the receiver are clipped to the receiver's boundary. The projection is then colored based on the sampled light values. These secondary light values ​​are then combined with their corresponding primary incident points on the receiver triangle to reconstruct the final reflected light. Finally, the result, along with the other receiver triangles, is transmitted to the image pixels on the screen.

[0062] Another implementation method utilizing secondary light, Figure 4 The emitting triangle is reflected in the receiving triangle by making full use of secondary rays. Receiving triangle 30 and emitting triangle 33 are shown. The reflection task is only applicable to those triangles of the target object mesh that have passed the visibility test and been identified as reflection candidates.

[0063] Back to Figure 2 As shown, the secondary ray covers a truncated cone region extending upwards from receiver 20 along dashed line 29. The secondary rays are designed to accurately check the visible grid triangle by triangle. Each secondary ray performs an intersection test with the emitter triangle, sampling the intersecting triangle for color, light, material, texture, etc. The sampled data is combined with the corresponding primary ray's impact point to merge the sampled light values ​​of the primary and secondary rays for complete reflection reconstruction. Finally, the result, along with the other receiver triangles, is transmitted to the image pixels on the screen.

[0064] The generation of secondary rays for reflection is separate from the generation of primary rays and the image rasterized by the primary rays. Therefore, these are two independent processes, and the generation of reflection by secondary rays can be performed independently and simultaneously with the generation of the image by primary rays.

[0065] Dynamic Accelerated Architectures. Accelerated architectures (AS) in ray tracing require fast construction and fast traversal. Unfortunately, these two requirements are contradictory. BVH accelerated architectures are the most widely used in current ray tracing. The more levels of the hierarchy, the faster the traversal, but the more time-consuming the construction becomes. Fast construction and fast traversal are essential for real-time ray tracing, and especially for real-time skinned animation.

[0066] To accommodate real-time skinned animation, this disclosure employs a truly dynamic acceleration architecture (DAS) with high locality, in which scene changes, such as the skinning animation of objects, are updated locally at specific locations within the DAS hierarchy without affecting other locations in the DAS hierarchy. The DAS is built only once and then only the necessary updates are performed per frame. Therefore, the need for rapid builds is replaced by low-cost updates.

[0067] DAS is built from a separate hierarchy of each scene object, such that each subsequent hierarchical level includes a submesh of the previous level, where all submeshes are bounded by bounding boxes. The bottom of each hierarchy is a triangular leaf level.

[0068] For fast traversal, the implementation is based on early visibility testing, which eliminates redundant intersection tests, and then performs intersection tests.

[0069] Each traversal of the acceleration structure is initiated within a specific triangle on the non-planar reflective surface to search for reflective objects within that triangle. All secondary rays involved in the traversal originate from a common origin and all aim towards the endpoint, thus maintaining coherence. In scenarios involving scene changes such as animation, this endpoint proximity focused on small areas of reflective objects facilitates rapid local updates of the DAS. The reason for the fast traversal of the DAS lies in the reduction of the number of intersection tests, which is the most significant task in ray tracing. This reduction allows for a reduction in the searchable acceleration structures through visibility tests based on coherent grouping of secondary rays, prior to the reduced DAS achieved through traversals of individual secondary rays.

[0070] DAS traversal is essentially completed in two phases. In the first phase, coherent ray groups traverse the upper-level hierarchy of the bounding box, while in the second phase, the leaf hierarchy of the triangle is traversed by individual rays or by small groups of coherent rays. In the first phase, secondary rays converge in coherent groups and traverse the bounding box level of DAS to locate the invisible parts of the emitting object, thus preventing negative intersection tests and improving the overall efficiency of intersection testing.

[0071] Ray grouping comprises a pyramidal truncated cone with a triangular cross-section. The truncated cone is used for preliminary visibility testing of the emitting object from the receiving triangle. From the second stage of successive intersection testing, meshes or portions of the emitting object mesh that are not visible from the tested receiving triangle are filtered out. Preliminary visibility testing is achieved through the coherence of secondary rays formed into the visibility truncated cone.

[0072] During the second phase, a single secondary ray traverses the rest of the DAS to perform an intersection test at the lower leaf hierarchy of the triangle.

[0073] In the first stage, the visibility of the target object is tested by traversing the mesh and sub-mesh hierarchy using geometry such as a pyramid. Invisible objects or parts thereof are discarded, thus saving redundant intersection tests. During the second stage, only the triangular leaf hierarchy is traversed by secondary rays. That is, the efficiency of DAS traversal is improved by a two-step traversal; first, geometry such as a triangular pyramid is used to eliminate negative intersection tests, and then secondary rays are used to perform efficient and accurate intersection tests on the leaf hierarchy.

[0074] exist Figure 5 An example of a DAS structure is shown. A DAS is a hierarchical construction where, in the first hierarchy, the highest hierarchy 50, the bounding boxes of geometric objects are contained. This example includes meshes of four target objects 52 to 55. The next hierarchy is formed by sub-meshes 51. Similarly, any number of additional hierarchies can be formed, where each successive hierarchy includes sub-meshes of the previous hierarchy. Leaf hierarchies 59 are formed by triangles. Leaf hierarchies can include multiple levels. A DAS can be built during the preprocessing stage before the start of a rendering frame, or at runtime.

[0075] DAS is constructed with a boundary volume hierarchy (BVH) that differs from existing technologies. The top hierarchy of DAS begins directly with geometric objects, without a single boundary volume at the top of the tree. Each geometric object is independent of other objects, allowing for autonomous updates within its own hierarchy without affecting its neighboring objects.

[0076] In animation, skinned animation is typically represented by two parts: the surface representation used to draw the character (called the skin or mesh); and a hierarchical set of interconnected bones used to animate the mesh (called the skeleton or rigging). The intention is to control only the first part, the graphical deformation of the mesh data. When an object undergoes this deformation, the object's surface representation is updated accordingly within its main mesh, sub-mesh, and the bounding boxes of the main mesh and sub-mesh, without affecting the hierarchy of adjacent objects or the overall structure, thus preserving locality and autonomy. This autonomy of each geometric object provides the dynamic characteristics of DAS, enabling efficient and rapid dynamic scene updates.

[0077] exist Figures 6 to 9 The principle of locality in skinning animation updates within a dynamically accelerated structure is illustrated. Figure 6The image shows three objects, 60, 61, and 62. The middle object, 62, is a small human figurine in its original position. Then, when it is moved and transformed, the figurine... Figure 7 The animation is shown in the middle. Statue 72 moves to another location and raises its hands. Figure 8 Reference Figure 6 The original pre-animated DAS is shown. The statue is represented by the middle column. The top hierarchy of the DAS includes the bounding boxes 80' to 85' of all parts of the DAS 86. The second hierarchy contains the bounded subgrids 80' to 85' of these parts, while the bottom hierarchy includes triangles. The other two columns represent the hierarchy of two other objects that remain stationary.

[0078] Frustum intersection testing. Intersection testing between millions of rays and millions of polygons is one of the most time-consuming tasks in ray tracing. In existing ray tracing techniques, each individual ray must traverse the hierarchy of the acceleration structure to find intersections. This leads to inefficiency because intersection tests often end in negative results. According to the implementation, efficiency is improved by eliminating most of the negative tests. A preliminary visibility test is introduced to cull invisible meshes or portions of invisible meshes. The visibility test is performed by traversing the mesh hierarchy of the acceleration structure using a truncated pyramid of triangles. Portions falling outside the truncated pyramid are classified as irrelevant to the intersection test.

[0079] Figure 9 Reference Figure 7 The animated DAS is shown. Bounding box 96 updates as sub-mesh 91 and 92 are displaced. In the second hierarchy, sub-mesh 91' and 92' and their bounding boxes undergo updates. All updates are performed locally and do not affect other sub-mesh of the same object or the hierarchy of other objects. This locality of updates makes DAS a dynamic data structure, enabling real-time skinned animation. This real-time animation is crucial for delivering high visual quality in video games, virtual reality, and augmented reality.

[0080] According to the implementation method, the efficiency of intersection testing is improved by introducing a preliminary visibility test to eliminate negative intersection tests that target invisible objects or portions of invisible objects. As shown, Figure 10The truncated cone has a triangular pyramid shape, comprising three sides 104, 105, and 106 cut by its proximal end 103 (receiver triangle) and its distal end 100. The vertices of the truncated cone are guided by normals 107, 108, and 109 of the receiver triangle. As previously described, the normals of the receiver triangles can be different from each other, wherein each vertex shares its normal with an adjacent receiver triangle on a non-planar surface. The truncated cone has a triangular pyramid shape based on the receiver triangle. In this exemplary setup, receiver triangle 103 and three transmitting objects 100, 101, and 102 are shown. Object 101 is fully visible and therefore meets all intersection tests. The second object 102 partially falls outside the truncated cone; submesh of object 102 that falls entirely outside the truncated cone will be eliminated. The third object 110, falling outside the truncated cone, will not be subject to intersection tests.

[0081] Figure 11 This illustrates the process of classifying objects and their submesh within the truncated cone in DAS (111). Polygons 104, 105, and 106 represent the three sides of the truncated cone. Each of these walls is used sequentially to traverse the DAS bounding box of the mesh and submesh levels to cut out objects and portions of those objects outside the truncated cone. This traversal occurs only at the mesh level directly above the triangle level, from the top down to the last submesh level in the DAS. The output of the traversal of the first wall 104 is used as the input for the traversal of the next wall 105, and then the output of the traversal of the next wall 105 is used as the input for the traversal of wall 106. Each traversal classifies the bounding box of the object mesh or submesh that lies completely on opposite sides of the truncated cone. During the truncated cone test, triangle level 59 ( Figure 5 No traversal. The result of the truncated cone test is a list of relevant subgrids in the last mesh hierarchy above the triangle level, which is a candidate for the intersection test.

[0082] According to one implementation, for accuracy and efficiency, a single secondary ray traverses only the bottom hierarchical level (one or more leaf levels of a triangle). These levels are traversed separately, followed by a truncated cone traversal. Once the truncated cone traversal step is completed, subsequent ray traversal steps are performed. The ray traversal step is based on tracing the secondary ray from the receiver triangle toward the emitting object. In this traversal step, only the sub-mesh that has undergone a visibility test remains active. The triangles of the sub-mesh are used by the secondary ray for intersection testing.

[0083] The second step is in Figure 12The DAS structure 121 is sparse, consisting only of candidate meshes, sub-meshes, and triangles that have undergone visualization testing. Only the lowest sub-mesh hierarchy 122, directly above the triangle level, is shown. The candidate triangles 123 shown at the bottom level of the DAS are candidates for intersection testing. According to the analysis implementation, a single intersection test for each emitting triangle is sufficient for triangle selection. Additional intersection tests are not required for the triangles. The emitting triangle is projected onto the receiving triangle and colored. Each secondary ray from the receiving triangle traverses the candidate emitting triangles. When an emitting triangle is first found to be hit by a ray, it is removed from the list of candidate triangles and projected onto the receiving triangle. The list is reduced throughout the intersection test, thereby accelerating the intersection processing.

[0084] exist Figure 13 The flowchart illustrates an implementation of the intersection test analysis. In Phase 1, the geometry or a portion thereof is clipped using a truncated cone test, thus eliminating the intersection test that would occur outside the truncated cone. In Phase 2, the remaining emitting triangles undergo an intersection test using secondary rays, and if an intersection is found, the analytical projection of that emitting triangle onto the receiving triangle is completed, thereby saving subsequent intersection tests.

[0085] First, 131, the DAS structure ( Figure 11 The bounding box of the highest-level structure (111) intersects with each of the three clipped frustum sides (104-106). At each intersection, geometric objects completely outside the frustum are eliminated from the candidate list. For objects at least partially retained within the frustum, the next-level submesh of the object is used for intersection with the frustum sides. Similarly, for each intersection, these submeshes completely outside the frustum are eliminated (132). The frustum intersection test proceeds down the submesh hierarchy (133), with the final intersection completed directly above the triangle level. Triangles that at least partially pass the visibility test and belong only to a submesh are kept in the list of active submeshes (110). The second phase begins at block (134). Secondary rays are emitted from the receiver triangle at the active triangle. Each secondary ray is tested for intersection with all candidate triangles. Based on the triangle hit by the secondary ray, the triangle is removed from the candidate list and analytically projected onto the receiver triangle. Then, at 135, once the secondary ray hits the triangle, the triangle is sampled for material, color, light, etc., and the analysis projection from the triangle onto the receiver triangle is completed (see...). Figure 3 No additional intersection test is performed between the secondary ray and the intersecting triangle. This saves on redundant intersection tests and thus improves efficiency.

[0086] According to another implementation, all secondary rays from the receiving triangle traverse all candidate triangles 110 to find intersections. The length of the candidate list remains constant. The intersecting triangles are sampled for the light values ​​at the intersection points. Figure 14 This implementation is illustrated in a flowchart. The first stage, eliminating objects or parts of objects located outside the truncated cone, is the same as in analyses 141 to 143. In stage two, the remaining emission triangles are subjected to intersection tests with all relevant secondary rays.

[0087] First, 141, the DAS structure ( Figure 11 The bounding box of the highest level structure (111) intersects with each of the three clipped frustum sides 104 to 106. At each intersection, invisible geometry completely outside the frustum is discarded. For objects at least partially retained within the frustum, the submesh of the next level structure of said object is used for intersection with the frustum sides. Similarly, for each intersection, submesh completely outside the frustum is discarded (102). The frustum intersection test proceeds down the submesh hierarchy (143), where the final intersection is completed at the hierarchy level directly above the triangle level. Triangles belonging only to the submesh that at least partially pass the inner frustum test are saved in the candidate list (110). The second phase begins at block (144). Secondary rays are emitted from the receiver triangle at the active triangle (123). Each secondary ray is tested for intersection with all active triangles. Once the secondary ray hits the triangle, it brings samples of material, color, light, etc., from the triangle's nearest point of impact (multiple intersections may occur for a single ray) to the receiver triangle 145.

[0088] In both implementations, the light values ​​sampled at the emission triangle are combined with their respective initial incident points to reconstruct the object's reflection in the receiving triangle. Finally, the combined result, along with the other receiving triangles, is transmitted to the image pixels on the screen.

[0089] Coherence of Secondary Rays. In this disclosure, coherence of secondary rays is achieved by creating and processing them in a set manner. In the prior art, secondary rays are created in the direct continuation of primary rays, and the secondary rays bounce off where the primary rays strike. According to one embodiment, secondary rays are decoupled from primary rays. Secondary rays are created and processed independently of primary rays (e.g., for reflection, diffusion, ambient light occlusion, shadows, etc.). However, upon completion, the data generated by the secondary rays is merged with the data of their corresponding primary rays and transmitted to the respective image pixels. That is, primary and secondary rays work asynchronously with each other, but the data collected by the primary and secondary rays are eventually merged. This also applies to the relationship between third rays and secondary rays, and also to higher-order rays.

[0090] Decoupling between primary and secondary rays allows secondary rays to be processed in groups, such as Figure 10 The truncated cone projections of rays 104 to 106. Both primary and secondary rays originate from a common origin, such as receiver triangle 103, and typically strike similar scene objects, such as 101 and 102, and traverse similar parts of the accelerated data structure, such as... Figure 12 123. The secondary rays emitted from the reflective surface of the triangle and traversing the accelerating structure have a common origin and mutually close endpoints, therefore these secondary rays are coherent.

[0091] The joint path of secondary rays results in high coherence, a long-standing requirement in ray tracing. The secondary rays in this disclosure are coherent because they all belong to the same truncated cone, meaning they all originate at the same triangle and take approximately the same direction. This coherent processing of secondary rays achieves high parallelism and high utilization of the GPU.

[0092] Real-time skinned animation. In animation art, skinned animation is typically represented by two parts: the surface representation used to draw the character (referred to as skinning or mesh); and a hierarchical collection of interconnected bones used to animate the mesh (referred to as skeleton or rigging). The intent of this invention is to control only the first part, the graphical deformation of the mesh data. When an object undergoes such deformation, the object's surface representation is updated accordingly within its main mesh, sub-mesh, and the bounding boxes of the main mesh and sub-mesh in real-time, without affecting the hierarchy of adjacent objects or the overall structure, thus maintaining locality and autonomy. This autonomy of each geometric object provides the dynamic characteristics of DAS, enabling efficient and rapid skinned animation and dynamic scene updates.

[0093] As detailed below, the real-time performance of skinned animations comes from the spatial locality and coherence of secondary rays, as well as the dynamic features and locality derived from DAS.

[0094] The spatial locality of secondary rays is achieved by confining rays to groups that affect only small regions. Ray groups originate from small regions of a non-planar receiving surface and target small regions at their endpoints. The smaller the endpoint region, the more homogeneous the types of objects and materials, thus reducing computational complexity and the amount of memory and cache required. This reduced complexity facilitates faster processing of animated objects on both the reflecting and reflected sides. The coherence of secondary rays further enhances the ability to create ray groups.

[0095] Localized DAS search. On the endpoint (reflected) side, objects reflected on non-planar surfaces are searched for in advance in terms of visibility within the DAS hierarchy. Because different object hierarchies in DAS are separated from each other, this separation helps to minimize the involvement of non-animated objects or parts thereof in the search.

[0096] Reduced traversal. Additional efficiency in traversing the DAS is achieved by reducing the searchable hierarchy of the DAS used for ray traversal through early visibility testing based on a truncated cone. Early visibility testing by the truncated cone filters out hidden DAS hierarchies, leaving only the portions of the DAS structure traversed by a single secondary ray for intersection testing, thereby reducing traversal and accelerating the traversal process.

[0097] Reduced intersection tests. Ultimately, the number of ray triangle intersection tests is reduced because only the rest of the DAS structure needs to be traversed, while negative intersection tests are eliminated by filtering out hidden parts earlier through visibility tests. The actual intersection tests include positive intersection tests. Positive intersection tests are focused only on the leaf-like hierarchical structure of the triangles, and the efficiency of positive intersection tests can be further improved by clustering some coherent secondary rays into small groups when traversing the leaf-like hierarchical structure.

[0098] In summary, the spatial locality and coherence of secondary rays, localized DAS search, reduced traversal, and reduced intersection testing mentioned above enable real-time ray traversal and real-time skinned animation using only software, without the need to introduce hardware accelerators to speed up traversal of accelerated structures.

[0099] Cloud streaming, while often positioned as a solution for distributing complex or high-end content, is scalable only when available bandwidth and cloud computing resources are available. Meanwhile, client devices continue to perform based on their built-in capabilities, and the software compatibility window for client devices is limited. Cloud streaming should meet the needs of its user base by having sufficient bandwidth and available computing resources. This excess demand will be met through greater self-sufficiency of devices / applications. This invention teaches the lower computational complexity of hybrid ray tracing, thus applicable to cloud client devices operating independently from the cloud when internet connectivity is not readily available. Where applicable, the cloud should supplement client devices with a balanced workload of processing power and responsibility to help compensate for the lack of horsepower and / or the ability to deliver a good user experience. On the cloud server side, reduced complexity enables efficient, effective, and cost-effective real-time ray tracing solutions via cloud rendering. By using conventional GPUs on the server side, support for numerous concurrent online streams, power efficiency, and high throughput can be achieved without any dedicated hardware. The pure software solution without hardware accelerators allows the use of existing infrastructure.

[0100] The provided methods can be implemented on a general-purpose computer, processor, or processor core. Suitable processors include, by way of example, general-purpose processors, special-purpose processors, graphics processors, conventional processors, digital signal processors (DSPs), multiple microprocessors, one or more microprocessors associated with a DSP core, controllers, microcontrollers, application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), any other type of integrated circuit (IC), and / or state machines. Such processors can be manufactured by configuring a manufacturing process using processed hardware description language (HDL) instructions and other intermediate data, including netlists (such instructions capable of being stored on a computer-readable medium). The result of this processing can be a mask, which is then used in a semiconductor manufacturing process to manufacture a processor implementing the features of this disclosure.

[0101] Computing System. Figure 15 illustrates a computing system 156 for efficient ray tracing according to an embodiment of the present disclosure. Various architectures and functions of the previous embodiment can be implemented. As shown, a system including at least one main processor 150 connected to a communication bus 155 is provided. The system also includes a main memory 151. Software and data are stored in the main memory, which may be in the form of RAM. The system also includes a graphics system having a graphics processing unit (GPU) 152 and a display 153. The GPU has multiple shaders, rasterization modules, etc. This embodiment is best suited for GPUs. The coherence of secondary rays achieved in the present disclosure results in high GPU utilization.

[0102] In one implementation, the graphics processor is a discrete GPU with its own video memory and managed by the CPU. In another implementation, the GPU may be integrated with the CPU core on a single chip, thereby sharing memory. Similarly, in one implementation, the aforementioned modules may reside in a semiconductor platform such as an FPGA and / or another reconfigurable device. Alternatively, these devices may be socket-type devices. System 156 may also include secondary memory 154. Furthermore, although not shown, system 156 may be coupled to a network for communication purposes (e.g., the Internet, LAN, WAN) and / or coupled to cloud streaming.

[0103] While various embodiments have been described above, it should be understood that these embodiments are presented by way of example only and not by way of limitation. Therefore, the breadth and scope of preferred embodiments should be limited only by the appended claims and their equivalents.

Claims

1. A computer-based method for hybrid ray tracing of nonplanar reflections, wherein the nonplanar reflections are created by coherent secondary rays emitted from geometric objects in a scene, wherein, The method for decoupling secondary rays from primary rays includes: a) Independent of the initial ray, identify non-planar reflective surfaces in the scene that reflect multiple geometric objects; b) Identify the plurality of geometric objects in the scene; and c) At each triangle of the reflective surface: Create a truncated cone pointing to the stated geometry; Generate groups of secondary rays, the groups of secondary rays being defined by the truncated cone and emanating from the triangle toward the geometric object; and The secondary rays in the acceleration structure are traversed in a coherent ray group manner.

2. The method according to claim 1, wherein, The secondary light rays include coherent light groups.

3. The method according to claim 1, wherein, The truncated cone has a triangular pyramid shape with a reflective surface based on the scene's geometry, independent of the image.

4. The method according to claim 1, wherein, The truncated cone is emitted from the reflective surface of the scene's geometry toward the grid from which the geometry is emitted to explore visibility.

5. The method according to claim 1, wherein, The traversal acceleration structure has improved efficiency due to the preliminary visibility test performed from the triangle of the non-planar reflective surface.

6. The method according to claim 1, wherein, All secondary rays emanating from each triangle on the reflective surface and traversing the acceleration structure share a common origin and mutually approaching endpoints.

7. The method according to claim 6, wherein, The common origin of secondary rays emitted from a non-planar surface and the mutually approaching endpoints of the secondary rays make the rays coherent.

8. The method according to claim 7, wherein, The coherence of the secondary rays is obtained by creating them in a set manner, thereby allowing the secondary rays to be processed in a grouped manner.

9. The method according to claim 1, wherein, The emission of the secondary light rays is accomplished independently of the image generated by the primary light rays.

10. The method according to claim 1, wherein, The secondary rays are created and processed independently of the rasterized image, and the resulting data of the secondary rays are merged with the image.

11. The method according to claim 10, wherein, The method of processing secondary rays independently of the rasterized image is also applicable to higher-order rays.

12. The method according to claim 1, wherein, By traversing the acceleration structure with coherent light, it is possible to eliminate the reconstruction of the acceleration structure, thereby replacing the reconstruction with a local update of the dynamic acceleration structure.

13. The method according to claim 12, wherein, The local updates of the accelerated structure enable rapid animation of each object or a portion of each object in the scene.

14. The method according to claim 12, wherein, The coherence of the secondary rays enables highly parallel traversal of the hierarchical structure of the dynamic acceleration structure.

15. The method according to claim 14, wherein, The coherence of the secondary rays enables high GPU utilization.

16. The method according to claim 12, wherein, The efficiency of intersection tests in the dynamic acceleration structure is improved by pre-eliminating most of the negative tests on the non-planar reflective surface.

17. The method according to claim 12, wherein, The efficiency of intersection tests in the dynamic acceleration structure is improved by eliminating the initial visibility tests of most negative tests.

18. The method according to claim 1, wherein, The acceleration structure is searched for in relation to the triangular mesh that affects the non-planar reflective surface.

19. The method according to claim 16 or 17, wherein, The intersection test is performed only on the visible portion of the target transmitter object.

Images