System and method for reducing rendering delay

Abstract

A system and method for reducing rendering latency. In one embodiment, a computing system may determine a first orientation in 3D space based on first sensor data generated at a first time. The system may determine a first visibility of an object in 3D space by projecting rays based on the first orientation to test for intersection. The system may generate a first row of pixels based on the determined first visibility, and output the first row of pixels for display. The system may determine a second orientation based on second sensor data generated at a second time. The system may determine a second visibility of the object by projecting rays based on the second orientation to test for intersection. The system may generate a second row of pixels based on the determined second visibility, and output the second row of pixels for display. The second row of pixels and the first row of pixels are displayed simultaneously.

Description

Technical Field

[0001] The present disclosure relates generally to computer graphics, and more particularly to graphics rendering methods and optimizations for generating artificial realities such as virtual reality and augmented reality. Background Art

[0002] Computer graphics, in general, are visual scenes created using computers. Three-dimensional (3D) computer graphics provide users with views of 3D objects from a specific viewpoint. Each object in a 3D scene (e.g., a teapot, a house, a person, etc.) can be defined in a 3D modeling space using primitive geometric shapes. For example, a cylindrical object can be modeled using a cylindrical tube and top and bottom circular caps. The cylindrical tube and the circular cap can each be represented by a network or mesh of smaller polygons (e.g., triangles). In turn, each polygon can be stored based on the coordinates of their respective vertices in the 3D modeling space.

[0003] Even though 3D objects in computer graphics can be modeled in three dimensions, they are usually presented to the viewer through a rectangular two-dimensional (2D) display, such as a computer or television monitor. Due to the limitations of the human visual perception system, humans expect to perceive the world from roughly the same perspective at any moment in time. In other words, humans expect that some parts of 3D objects will be visible, and other parts will be hidden from view. Therefore, for each 3D scene, a computer graphics system may only need to render the part of the scene that is visible to the user, and not the rest. This allows the system to greatly reduce the amount of computation required.

[0004] Ray casting is a technique for determining the visibility of objects in a 3D scene. Typically, virtual rays are cast uniformly from a virtual pinhole camera through each pixel of a virtual rectangular screen into the 3D world to determine what is visible (e.g., based on which parts of the 3D object the rays hit). However, this assumes a uniform light distribution, which is reasonable when computing primary visibility from a virtual pinhole camera for traditional rectangular display technologies with limited fields of view (e.g., computer monitors and phone displays). However, this assumption does not hold true for non-pinhole virtual cameras that more accurately represent real optical sensors. Furthermore, current VR viewing optics (e.g., integrated within head-mounted displays) provide curved, non-uniform viewing surfaces rather than traditional rectangular displays. As a result, traditional rendering techniques designed and optimized based on the above assumptions are computationally inefficient, produce suboptimal renderings, and lack the flexibility to render scenes in artificial reality. Summary of the invention

[0005] Certain embodiments described herein relate to a primary visibility algorithm that provides real-time performance and a feature set that is well suited for rendering artificial realities such as virtual reality and augmented reality. When solving the visibility problem, certain embodiments use a hierarchical bounding box and a two-level section culling / entry point search algorithm to accelerate and optimize the traversal of coherent primary visibility rays, rather than uniformly casting a separate ray for each pixel. Certain embodiments utilize adaptation of multi-sample anti-aliasing for ray casting, which significantly reduces memory bandwidth.

[0006] Certain embodiments further provide flexibility and rendering optimizations, which enable the rendering engine to generate a variety of graphical features locally while maintaining real-time performance. Such graphical features - such as lens distortion, sub-pixel rendering, very wide field of view, focusing, and random depth of field blur - may be particularly desirable in artificial reality environments. These embodiments provide support for animation and physically based shading and lighting to improve the realism of rendered scenes. In contrast, traditional rasterization pipelines designed for traditional displays (e.g., rectangular monitors or television sets with uniform pixel grids) are typically implemented in hardware and require multiple passes and / or post-processing to approximate these features. In addition, traditional ray tracers that focus primarily on Monte Carlo path tracing cannot achieve real-time performance on current VR displays (e.g., with 1080×1200×2 resolution and 90Hz refresh rate requirements). Therefore, the embodiments described herein are particularly suitable for rendering artificial reality and present a concrete, viable alternative to traditional rasterization techniques.

[0007] Embodiments of the present invention may include or be implemented in conjunction with an artificial reality system. Artificial reality is a form of reality that has been adjusted in some way before being presented to a user, which may include, for example, virtual reality (VR), augmented reality (AR), mixed reality (MR), mixed reality, or some combination and / or derivative thereof. Artificial reality content may include fully generated content or generated content combined with captured content (e.g., real-world photos). Artificial reality content may include video, audio, tactile feedback, or some combination thereof, and any of them may be presented in a single channel or in multiple channels (such as stereoscopic video that produces a three-dimensional effect to the viewer). In addition, in some embodiments, artificial reality may be associated with, for example, applications, products, accessories, services, or some combination thereof for creating content in artificial reality and / or for use in artificial reality (e.g., performing activities in artificial reality). An artificial reality system that provides artificial reality content may be implemented on a variety of platforms, including a head-mounted display (HMD) connected to a host system, a standalone HMD, a mobile device or computing system, or any other hardware platform capable of providing artificial reality content to one or more viewers.

[0008] The embodiments disclosed herein are merely examples, and the scope of the present disclosure is not limited to them. Specific embodiments may include all, some or none of the components, elements, features, functions, operations or steps of the embodiments disclosed above. In particular, embodiments according to the present invention are disclosed in the appended claims for methods, storage media, systems and computer program products, wherein any feature mentioned in one claim category (e.g., method) may also be claimed in another claim category (e.g., system). The dependencies or references in the appended claims are selected for formal reasons only. However, any subject matter resulting from an intentional reference to any previous claim (especially multiple dependencies) may also be claimed, thereby disclosing any combination of claims and their features, and may be claimed regardless of the dependencies selected in the appended claims. The subject matter that may be claimed includes not only the combination of features set forth in the appended claims, but also any other combination of features in the claims, wherein each feature mentioned in the claims may be combined with any other feature in the claims or a combination of other features. In addition, any embodiments and features described or depicted herein may be claimed in separate claims and / or in any combination of any embodiments or features described or depicted herein or any features of the appended claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] Figure 1 An example of a hierarchical bounding box tree data structure is shown.

[0010] Figure 2 An example three-level hierarchy for defining the locations of projection rays / beams is shown.

[0011] Figure 3 Shows examples of rays and subsample rays associated with footprints.

[0012] Figure 4 An example of a beam projected through a block is shown.

[0013] Figure 5 An example of a beam projected through a tile is shown.

[0014] FIG. 6A to FIG. 6C An example of a method for determining visibility is shown.

[0015] Figure 7 An example of a focal plane diagram is shown.

[0016] Figure 8 Examples of focal plane diagrams and camera parameters are shown.

[0017] Fig. 9 A method for locally generating images with optical distortion for a VR device is shown.

[0018] Fig.10 An example of an importance map is shown.

[0019] Fig.11 An example method for generating an image based on variational multi-sample anti-aliasing is shown.

[0020] Fig.12 An example comparing a graphics generation timeline without beam racing to a timeline with beam racing is shown.

[0021] Fig.13 An example method for generating video frames for a VR display using beam racing is shown.

[0022] Fig.14 An example computer system is shown. DETAILED DESCRIPTION

[0023] A fundamental problem in computer graphics is determining the visibility of an object. Currently, the two most commonly used methods are ray tracing, which simulates the transmission of light and dominates industries such as movies and computer-aided design (CAD) where accuracy is higher than speed. Due to the high computational requirements of ray tracing, it is traditionally not suitable for applications that require real-time or near real-time rendering. Another method to determine visibility is the depth buffering method (z-buffering), which checks each 3D object in the scene and updates a buffer that tracks the current nearest object for each pixel of the virtual rectangular screen. Typically, the depth buffering method is implemented by current graphics hardware and lacks the flexibility to handle rendering tasks that deviate from the above assumptions (e.g., pinhole cameras and / or rectangular screens with uniform pixel distribution). The specific embodiments described herein provide a visibility algorithm that has performance characteristics close to the depth buffering method, but has additional flexibility, which can render a variety of visual effects for artificial reality.

[0024] To provide further context, traditional depth buffering is often used to solve the real-time primary visibility problem, largely due to its applicability to uniform primary visibility problems (e.g., for traditional rectangular screens) and the availability and scalability of inexpensive dedicated hardware implementations. Depth buffer (z-buffer) algorithms use a depth buffer, which is a uniform grid data structure that stores the current nearest hit depth for each sample / pixel. Most implementations of depth buffering assume that the samples / pixels are placed in a uniform grid, which exactly matches the organization of the data structure. The uniformity of the grid structure, combined with the uniform distribution of samples mapped onto this grid, allows for very efficient algorithms to determine which samples overlap with polygons / triangles. The process of mapping the spatial extent of an object onto a grid is called rasterization.

[0025] The uniformity of the grid used in the depth buffer algorithm yields high efficiency but makes the algorithm inflexible. The assumed uniform sample distribution is reasonable when computing primary visibility from a virtual pinhole camera for almost all direct-view display technologies (such as TVs, monitors, or mobile phones). However, these assumptions do not apply to non-pinhole virtual cameras, secondary effects such as shadows, and are especially unsuitable for modern virtual reality devices due to the distortion imposed by the viewing optics of head-mounted displays, and currently must be handled on a case-by-case basis.

[0026] Algorithms such as irregular depth buffers still use a uniform grid, but allow flexibility in the number and location of samples placed in each grid cell. The irregular depth buffer method suffers from load balancing issues related to conflicts between the non-uniform sample distribution in the uniform data structure, making it significantly more expensive than the traditional depth buffer method. In addition, having a uniform data structure means that the algorithm only supports a limited field of view, and does not support depth of field rendering.

[0027] Unlike the depth buffer approach, ray tracing algorithms take a more general approach by supporting arbitrary point-to-point or ray queries to determine visibility. The ability to effectively model physically based light transport and natively composite effects has made it the dominant rendering algorithm for rendering movie scenes. However, the flexibility provided by ray tracing comes at a significant cost in performance, which has prevented its popularity among consumers for real-time applications such as VR / AR.

[0028] Certain embodiments described herein overcome the shortcomings of existing rendering techniques to achieve ray rates of over 10 billion rays per second for non-trivial scenes on modern computers, natively supporting computer graphics effects desired for artificial reality.

[0029] Particular embodiments solve visibility problems in computer graphics. In particular embodiments, a rendering system may use a ray caster that uses a three-level (or more) entry point search algorithm to determine visibility. At a high level, the system may take a hierarchical approach, where a larger beam (e.g., a coherent beam) is first projected to determine collisions on a wider scale. Based on the hit / miss of the beam, more granular beams or rays may be projected until the visibility problem is solved. It should be noted that even though some of the examples provided herein describe beams as coherent beams representing primary rays, the present disclosure contemplates the use of beams to represent any type of ray (e.g., primary ray, specular ray, shadow ray, etc.), whose coherent structure may be utilized by the embodiments described herein to achieve computational efficiency. In particular embodiments, the system may be implemented in a heterogeneous manner, where beam traversal occurs on a central processing unit (CPU), and ray-triangle intersection and shading occurs on a graphics processing unit (GPU). In other embodiments, each computational task may be performed by the same type of processing unit.

[0030] To improve performance, certain embodiments may use acceleration structures to organize scene geometry. These structures may be based on spatial partitioning (grids, kd or k-dimensional trees, binary spatial partitioning or BSP trees, octrees) or object partitioning (bounding volume hierarchy or BVH). By organizing geometry into spatial regions or enclosing them in closed boxes, this structure allows the system to avoid testing a ray with an object if the ray does not enter the box enclosing the object.

[0031] In certain embodiments, the axis-aligned hierarchical bounding box is a hierarchical tree structure that stores scene geometry (typically triangles) at the leaves of the tree and stores axis-aligned bounding boxes at each node. The bounding box associated with each node may conservatively enclose all geometry associated with the node's subtree. In certain embodiments, a ray (or other visibility query such as a beam) may recursively traverse the tree from the root and test against the bounding boxes of the node's children. The recursive traversal of a node's children may occur only in the case of intersection, so that the ray / beam may avoid traversing portions of the tree whose parent nodes are not hit by the ray / beam.

[0032] Figure 1An example of a BVH tree data structure 100 is shown. Each node (e.g., 110, 120-128, 130-136) in the tree 100 can be associated with a bounding box in the 3D modeling space that defines the object. The tree 100 can have a root node 110 that is associated with a large bounding box that encloses the bounding boxes associated with the child nodes 120-128, 130-136. Node 120 can be associated with a bounding box that contains the bounding boxes of its child nodes 121-128 but does not contain the bounding boxes of nodes 130-136. Node 121 can be associated with a bounding box that contains the bounding boxes of its child nodes 122-125 but does not contain the bounding boxes of any other nodes. Node 122 can be associated with a bounding box that contains a geometric shape (e.g., a triangle) but does not contain any other bounding boxes.

[0033] In certain embodiments, the system may use a four-way axis-aligned BVH as a geometry acceleration structure. In certain embodiments, a single combined BVH may be used for all scene geometries. In other embodiments, the system may take a multi-level approach to allow instantiation and enable more efficient animation by allowing finer-grained BVH reconstruction and trimming. Basic animation is supported per frame via global BVH trimming.

[0034] In a particular embodiment, the BVH may be arranged in memory in a predetermined depth-first order, and triangles may be stored in a contiguous array in the order in which they are touched in a depth-first traversal of the BVH. In addition, any node with a mix of leaf and internal child nodes may store leaf nodes first. With these assumptions, iterating backward through the BVH node list may guarantee that a node's children are always visited before it will be visited, and that all triangles will be visited in a linear, reverse order. These assumptions enable a linear, non-recursive BVH pruning algorithm and improve cache locality during pruning, traversal, and intersection.

[0035] A specific embodiment for calculating primary visibility may perform visibility testing using a BVH. As previously described, whether an object (or part thereof) is visible from a particular viewpoint may be determined by testing whether the object (or part thereof) intersects with a ray. However, emitting multiple rays from each pixel for each pixel is computationally expensive and resource intensive, especially when the area to be covered is large. For example, if 32 sample rays are used per pixel, 32 intersection tests need to be performed for each pixel, and a ray buffer of sufficient size needs to be allocated to store the results. emitting so many rays in scenes with few objects may be particularly wasteful, as most rays will not intersect with any object.

[0036] Instead of emitting rays, certain embodiments may use beams to perform intersection testing. In certain embodiments, the system may perform hierarchical intersection testing using, in sequence, (1) larger cross-sectional beams projected from relatively larger "blocks" (beam footprints are described in more detail below), (2) smaller cross-sectional beams projected from "blocks" (beam footprints are also described in more detail below), and (3) procedurally generated subsample rays (interchangeably referred to herein as "sub-rays"). In certain embodiments, intersection testing does not need to be performed on sub-beams or rays unless an intersection is found using a larger cross-sectional beam, thereby avoiding unnecessary calculations. In certain embodiments, pixel shading may be based on the intersection results of a single subsample ray rather than the results of all 32 subsample rays. To further optimize performance, certain embodiments may dynamically generate subsample rays procedurally (e.g., pseudo-randomly) when performing intersection testing, rather than retrieving predefined subsample ray positions from memory. Procedural ray generation has the benefit of not requiring reads from memory, thereby saving time and bandwidth.

[0037] Figure 2 An example three-level hierarchy for defining the positions of projected rays / beams is shown. In certain embodiments, ray footprints 210, which may be considered the first level of the hierarchy, may correspond to pixel footprints and be defined by footprint centers 240 and differences 250 (e.g., each difference 250 may specify a distance of a boundary of the footprint from the center 240 of the footprint 210). In certain embodiments, subsample locations 260a-e are assigned to the footprints 210 for anti-aliasing. Figure 2 In the example shown, each footprint 210 includes five subsample positions 260a-e. Although the specific example shows five subsample positions 260a-e for each footprint 210, any other number of subsample positions (e.g., 32 subsample positions for each footprint 210) may be implemented and / or defined by the application.

[0038] Figure 3 An example of rays and subsample rays (or sub-rays) associated with a footprint is shown. In a particular embodiment, ray 310 may be associated with footprint 210, where ray 310 is cast through center 240 of footprint 210. Figure 2As shown, a footprint 210 can be defined by two vectors 250 (also referred to as differences) that are perpendicular to each other and a ray direction. The extent of the footprint 210 can be defined by the lengths of these vectors 250. In certain embodiments, subsample rays (or sub-rays) can be procedurally generated within the footprint 210 by first transforming low-deviation points set on, for example, a unit square of the footprint 210 using a coordinate frame defined by the ray 310 direction and the footprint vectors 250. Examples of transformed points are represented by subsample positions 260a-e, as shown by the hollow dots. The ray directions can then be added to the transformed points (e.g., subsample positions 260a-e), and subsample rays 360a-e can be defined as rays projected through the original ray 310 origin 301 and the newly transformed points 260a-e, respectively. For example, subsample rays can be used for multi-sample anti-aliasing. In certain embodiments, for depth of field rays, a separate set of low-deviation points (not translated along the ray direction) can also be used to select the ray origin.

[0039] In certain embodiments, shading may be performed once per pixel per triangle, as in traditional multi-sample anti-aliasing (MSAA), which saves a significant amount of shading calculations. In certain embodiments, shading may be performed per sample to obtain full super-sample anti-aliasing (SSAA). Since sub-rays are procedurally generated, rather than pre-defined (and stored in memory), ray memory bandwidth may be reduced by the anti-aliasing factor when compared to native rendering at higher resolutions.

[0040] In certain embodiments, the primary rays (e.g., 310) are assigned footprints (e.g., 210) for antialiasing and then aggregated into a second and third level hierarchy of quad-bounded beams with different granularities. Each beam position in the finer second level hierarchy may be referred to as a block (e.g., an example of a block is shown in Figure 2 Each block 220 may include a predetermined number of pixel footprints 210. Figure 2 The illustrated example shows each block 220 having a 2×2 pixel footprint 210, but any other arrangement of pixel footprints 210 may be implemented (e.g., each block 220 may include 16×8 or 128 pixel footprints 210). The blocks 220 may then be aggregated into a collection of coarser blocks 230, which is the term used herein to refer to the third level of hierarchy. Each block 230 may contain a predetermined number of blocks 220. Although Figure 2Each tile 230 is shown as containing 2×2 blocks 220, but any other arrangement of blocks 220 may be implemented (e.g., each tile 230 may include 8×8 or 64 blocks). Thus, in an embodiment where each tile 230 contains 8×8 blocks 220 and each block 220 contains 16×8 pixel footprints 210, each tile 230 may represent 8,192 pixel footprints 210. The number of rays represented by a beam originating from a tile 230 may be calculated by multiplying the number of pixel footprints 210 in the tile 230 by the multisampling rate (e.g., 5 subsample rays per pixel footprint 210). Thus, if each tile 230 represents 8,192 pixel footprints 210, the tile 230 will represent 40,960 subsample rays. In a particular embodiment, the choice to define the ratio between pixels and blocks as 128:1 and the ratio between blocks and regions as 64:1 is based on rough tuning of the particular hardware, but for other types of hardware, other ratios may be more optimal.

[0041] In certain embodiments, instead of casting rays for all visibility tests, beams may be cast from tiles and blocks in a hierarchical manner to optimize visibility calculations. Figure 4 An example of a beam 420 projected from a point origin 301 (e.g., a camera or viewpoint) through a block 220 (in the example shown, the block 220 contains a 2×2 pixel footprint 210) into a 3D modeling space is shown. The solid beam 420 in this example is similar to a cross section originating from the block 220. The box of the beam 420 can be defined by vectors projected through the four corners of the block 220. An object or triangle intersects the beam 420 if the object / triangle intersects any part of the box of the beam 420. Similarly, Figure 5 An example of a beam 530 projected from a point origin 301 through a tile 230 into a 3D modeling space is shown (in the example shown, the tile 230 contains 2×2 blocks, each containing a 2×2 pixel footprint 210). The solid beam 530 in this example resembles a cross section originating from the tile 230. The box of the beam 530 can be defined by vectors projected through the four corners of the tile 230. An object or triangle intersects the beam 530 if the object / triangle intersects any part of the box of the beam 530.

[0042] A specific embodiment for scene updating and triangle precomputation will now be described. In a specific embodiment, before rendering begins, animations can be performed (e.g., 3D object models in the scene can change) and the BVH can be trimmed. In a specific embodiment, skeletal animation can occur on the CPU, and linear blend skinning and BVH trimming can be implemented in a series of CUDA kernels in the following example stages: (1) transform vertices (perform linear blend skinning); (2) clear BVH node boundaries; (3) precompute triangles (e.g., by (a) collecting vertices (b) calculating edge equations (for Moller-Trumbore ray-triangle intersections), and (c) calculating triangle boundaries and automatically updating corresponding leaf bounding boxes); and (4) trimming the BVH by propagating boundaries from leaf nodes up to the internal node hierarchy. In a specific embodiment, after trimming is performed on the GPU, the BVH can be copied back to CPU memory for the tile and block traversal stages. At this point, tile and block boundaries can be calculated and trimmed if necessary.

[0043] FIG. 6A to FIG. 6C An example of a method 600 for determining visibility according to a particular embodiment is shown. The illustrated embodiment implements a three-stage entry point search algorithm, but additional stages may be implemented in other embodiments (e.g., using a fourth beam footprint including a set of tiles, a fifth beam unit including a fourth set of beam units, etc.). In a particular embodiment, the three levels are conceptually divided into a tile culling stage, a block culling stage, and a ray sample testing stage.

[0044] Fig. 6A An example of a block culling phase is shown. During this phase, the computing system can traverse the BVH hierarchy and use beams derived from the blocks (e.g., Figure 5 ) to test for intersection of selected bounding boxes associated with nodes in the BVH. In a particular embodiment, each such beam is defined by a 128×64 pixel footprint. In a particular embodiment, the implementation of beam traversal uses an explicit stack AVX implementation. Because the tile traversal is a culling / entry point search phase, rather than traversing all the way to the leaves as in a traditional ray tracer, the tile traversal is only traversed until it reaches a certain stopping criterion (e.g., when 64 entry points have been found).

[0045] In certain embodiments, the screen for which the scene is to be generated (e.g., a virtual screen in 3D space corresponding to the display screen used by the user) can be divided into n tiles. For each tile, the system can perform three levels of testing to determine what is visible from the tile. In certain embodiments, the visibility test can be performed by projecting a beam from the tile. For ease of reference, the beam projected from the tile is referred to herein as a tile beam.

[0046] In certain embodiments, the method may begin at step 610, where an explicit walk of a stack (e.g., a data structure for tracking which nodes of a BVH are to be tested for intersection) may be performed with the root of the BVH (e.g., Figure 1 The root is initialized with a bounding box (BV), which can be, for example, a bounding box. The bounding box can be defined in 3D space. In some cases, the bounding box can contain smaller bounding boxes. For example, referring to Figure 1 , each child node (e.g., 120-128 and 130-136) corresponds to a bounding box within the bounding box of the root 110. Objects defined in 3D space (e.g., basic geometric shapes such as triangles, larger objects defined by a collection of basic geometric shapes, etc.) can be contained within any number of bounding boxes. For example, an object contained by a bounding box associated with node 125 is also contained in bounding boxes associated with nodes 123, 121, 120, and 110.

[0047] At step 612, the system can access bounding boxes based on the traversal stack to test for intersection with the tile beam. As an example, the system can initially perform an intersection test with the bounding box associated with the root node 110, and in later iterations, perform intersection tests for the child nodes of the root node 110 based on the contents of the traversal stack. In a particular embodiment, at each step during the traversal, the thickest box in the traversal stack along the main traversal axis can be tested. This allows the system to more efficiently refine the nodes into a single surface patch. Although the overhead of the sorting is significant, it has been known to improve block / tile culling performance by 5-10%.

[0048] At step 614, the system can simulate the projection of the beam defined by the block in 3D space to test for intersection with the selected bounding box. Figure 5 As shown, the box 530 of the tile beam may be tested against the bounding box to determine the degree of intersection, if any.

[0049] At step 616, the system can determine that the result of the intersection test is one of: (1) miss - meaning the beam completely misses the bounding box; (2) fully contained - meaning the beam fully / completely contains the bounding box; or (3) partially intersects - meaning the beam and bounding box intersect, but the bounding box is not fully contained within the beam. If the system determines at step 618 that the test result is a miss, the system can remove / discard the subtree of the current node, preventing it from being a candidate to be tested for intersection with the tile beam. For example, referring again to Figure 1, node 132 may indicate a miss, meaning that the bounding box associated with node 132 does not intersect the tile beam at all. In this way, smaller bounding boxes contained within the bounding box (e.g., bounding boxes associated with nodes 133-136 in the subtree of node 132) do not need to be tested further, thereby providing significant computational savings. Conversely, if the system determines at step 620 that the test result is fully contained, the system may accumulate the bounding box as an entry point and does not need to further traverse the subtree of the associated node because it is transitively fully contained. Reference Figure 1 , node 121 may represent a bounding box that is completely contained within the tile beam. As such, smaller bounding boxes contained within that bounding box (i.e., bounding boxes associated with nodes 122-125 in the subtree of node 121) do not need to be tested against the tile beam (but may be tested in a subsequent tile culling phase), thereby providing substantial computational savings. Conversely, if the system determines at step 622 that the test result is partially contained (e.g., the bounding box partially intersects the tile beam), the system may add / insert the subtree associated with the bounding box into the traversal stack for continued refinement. Reference Figure 1 , node 110 may represent a bounding box that partially intersects the tile beam. In this way, smaller bounding boxes contained within the bounding box may be further tested against the tile beam. For example, in response to the bounding box of node 110 partially intersecting the tile beam, the system may insert the top nodes of each subtree (e.g., nodes 120 and 130) into the traversal stack for further intersection testing against the tile beam.

[0050] In certain embodiments, at step 624, the system may determine whether one or more termination conditions of the block culling phase are met. If the termination conditions are not met, the system may continue to perform intersection tests for bounding boxes associated with nodes stored in the traversal stack. For example, after determining that the bounding box of the root node 110 partially intersects the block beam, the system may test, in the next iteration, whether a smaller sub-bounding box associated with, for example, node 120 or 130 intersects the block beam. This process may continue until the termination condition is met. For example, the traversal may continue until the traversal stack is empty. If so, the system may sort the accumulated entry points (or fully contained bounding boxes) in order of proximity to depth and pass them to the block culling phase for further processing. Another termination condition may be when the sum of the size of the traversal stack and the size of the list of fully contained bounding boxes is equal to a pre-specified value (such as 32, 64, 128, etc.). In certain embodiments, the traversal stack and the list may be merged, sorted in order of proximity to depth, and passed to the block culling phase. Therefore, no more than a fixed number of entry points are passed from the block culling stage to the chunk culling stage.

[0051] In a particular embodiment, during traversal, the separating axis theorem is used to determine the separation between bounding boxes and tile beams. The near plane along the major axis of the beam can be used as a key value when sorting entry points before switching to tile culling.

[0052] The block culling stage starts where the tile culling stage ends. In a particular embodiment, each entry point identified during the tile culling stage is further tested using 64 block culling stages (e.g., corresponding to 8×8 or 64 blocks in a tile, depending on the particular embodiment). In a particular embodiment, block culling can be implemented in an explicit stack AVX traversal, as in tile culling. However, instead of starting by initializing the traversal stack with the root node in the BVH, the traversal stack can be initialized by copying the output associated with the tile culling. In this way, block culling avoids repeating a large number of traversals performed during tile culling. In a particular embodiment, the beam / box test has similar potential results as in tile culling, but the traversal can continue until the traversal stack is empty. Once all triangles are gathered, they are copied to the GPU by CUDA for sample testing. In a particular embodiment, in a high-depth complex scene, an excessive number of triangles can be quickly collected and potentially tested despite the fact that they can be occluded by closer geometry. Short-circuiting block traversal may require interleaving block culling and sample testing, which means migrating block culling to a CUDA implementation.

[0053] Figure 6B An example of a block culling stage is shown to process the results from the tile culling stage. As previously described, the tile culling stage can generate a list of entry points by projecting tile beams and testing for intersection with bounding boxes. The resulting entry points can be used as starting points during a subsequent block culling stage. The block culling stage can typically test for intersection using tile beams contained within tile beams. As previously described, each secondary beam is defined by a tile footprint that is smaller than the tile footprint.

[0054] The block culling phase for processing the results of the block culling phase can begin at step 630, where the system can iteratively select entry points in the list generated during the block culling phase and perform block culling. Entry points associated with nodes or bounding boxes in the BVH are known to intersect the block beam. In the block culling phase, the system attempts to determine which blocks of the tile intersect the bounding box or its subboxes at a finer granularity. Therefore, given a selected entry point, the system can iteratively project block beams contained within the block beam to test for intersection.

[0055] For a given entry point, at step 632, the system can iteratively select blocks in the tile to perform intersection testing. In certain embodiments, before testing an entry point against a particular tile beam, the system can initialize the traversal stack to the bounding box associated with the entry point. Doing so provides an efficiency gain because the tile culling stage does not need to start at the root of the BVH (this has already been done in the tile culling stage). Figure 1 , as an example, the system may initialize the traversal stack with node 121, which was determined to be a suitable entry point during the block culling phase.

[0056] At step 634, the system can access bounding boxes based on the traversal stack to test for intersection with the tile beam. For example, the system can initially perform an intersection test on the bounding box associated with node 121, which was considered a suitable entry point during the tile culling phase, and in later iterations, perform intersection tests on its child nodes based on the contents of the traversal stack. In certain embodiments, at each step during traversal, the thickest box in the traversal stack along the main traversal axis can be tested.

[0057] At step 636, the system can simulate projecting the beam defined by the block into 3D space to test for intersection with the bounding box. Figure 4 As shown, the box 420 of the block beam may be tested against the bounding box to determine the degree of intersection, if any.

[0058] At step 638, the system can determine that the result of the intersection test is one of the following: (1) miss - meaning that the beam completely misses the bounding box; (2) fully contained - meaning that the beam fully / completely contains the bounding box; or (3) partially intersects - meaning that the beam and the bounding box intersect, but the bounding box is not fully contained in the beam. At step 640, if the system determines that the test result is a miss, the system can remove / discard the subtree of the current node, preventing it from becoming a candidate for intersection testing with the block beam. Conversely, if the system determines that the test result is fully contained at step 642, the system can accumulate triangles / polygons in the bounding box to be tested in subsequent stages. There is no need to further traverse the subtree of the associated node because it is transitively fully contained. In other words, any additional bounding boxes contained within the current bounding box can be removed, preventing them from becoming candidates for intersection testing with the block beam. Conversely, if the system determines at step 644 that the test result is partially contained (e.g., the bounding box partially intersects the block beam), the system can add / insert the subtree associated with the bounding box into the traversal stack for continued refinement (e.g., when processing is repeated at step 634).

[0059] As an example, the system can be Figure 1. In some cases, the system may determine that the bounding box associated with node 121 partially intersects the projected block beam. Based on this determination, the system may insert a subtree of node 121 (e.g., 122 and 123) into the traversal stack. Then, when repeating step 634, for example, the system may test for intersection of node 122 by projecting the block beam to the bounding box associated with node 122. In some cases, the system may determine that the projected block beam completely contains the bounding box associated with node 122 and add the triangles / polygons in the box to the list for sampling.

[0060] In certain embodiments, the traversal may continue until the traversal stack is empty. Thus, at step 646, the system may determine whether any nodes remain in the traversal stack. If there is a node in the stack, the system may return to step 634 to test the node for the block beam. If there is no node in the stack, at step 648, the system may determine whether there are additional blocks in the tile that have not been tested for the original entry point. If so, the system may return to step 632 to test the entry point for the untested block for the intersection point. Otherwise, the system may determine at step 650 whether additional entry points from the tile culling stage still need to be tested. If so, the system may return to step 630. Otherwise, the system in certain embodiments may pass the collected triangles / polygons to the ray sample testing stage.

[0061] In a particular embodiment, the light sample test stage may be performed after the block culling stage. In a particular embodiment, the light sample test stage may be divided into a per-block and per-pixel stage. In a particular embodiment, a single CUDA kernel with a workgroup size of 128 may be used to complete both stages. In each block portion, the thread may draw with a triangle 1:1, and in the per-pixel stage, the thread may draw with a pixel 1:1. In a particular embodiment, the thread may optionally draw with a sub-pixel 1:1, in which case the stage may be referred to as a per-sub-pixel stage. As used herein, a sub-pixel is a single LED, such as red, green, or blue, and is different from a sub-sample in the sense of multi-sample anti-aliasing. Therefore, a sub-pixel may have many sub-samples. The system may support multi-sample anti-aliasing (MSAA) and super-sample anti-aliasing (SSAA), the difference being that in MSAA, shading is performed only once per pixel of each triangle, and the result is shared between all sub-samples of the pixel that hits the same triangle, and in SSAA, shading is calculated separately by each sub-sample. The advantage of MSAA is a potential substantial reduction in shading rate. The triangle data of a block may be collected into a shared local cache on the GPU for easy access from all samples. This triangle cache may have 128 entries. In certain embodiments, the per-tile and per-pixel / sub-pixel stages may alternate until all triangles of the tile have been processed.

[0062] Figure 6C An example of a sample test phase is shown to handle triangle intersections identified after the block culling phase. After the block culling phase, the system may have identified a list of triangles that intersect the associated block beam. During the sample test phase, the system attempts to further sample triangles at a finer pixel level and may do so using subsample rays (e.g., for anti-aliasing). In a particular embodiment, the aforementioned per-block phase may be represented by steps 660 to 668, and the per-pixel / sub-pixel phase may be represented by the steps starting at step 670.

[0063] At step 660, before testing the triangles for intersection with the block beam, the system can perform initialization by, for example, clearing each subsample depth and index value. During each block stage, the system can collect the triangle data into a shared memory cache at step 662. At step 664, the system can perform backface and near plane culling on the triangle. At step 666, the system can test the block corner rays against the triangle and classify the intersection into fully contained, partially intersected, and missed, similar to the classifications described above. At step 668, the system can perform common origin intersection precomputation (when applicable).

[0064] In certain embodiments, once the per-block stage has been completed, each thread may associate itself with a pixel / sub-pixel and perform the following steps during the per-pixel / sub-pixel stage. The system may test for intersection using rays contained within the block beam that the triangle intersects with. In certain embodiments, rays (including subsample rays) may be generated procedurally.

[0065] In certain embodiments, at step 670, the system can find the footprint center and difference value associated with the ray's footprint for each pixel / sub-pixel in the block. At step 672, the system can find the center (e.g., Figure 2 , label 240) and the difference (e.g., Figure 2 , label 250) into 3D world space. The center and difference can define a ray trail from which a ray can be generated and cast into 3D space to test for intersection with an object (e.g., a triangle).

[0066] In certain embodiments, the system may iteratively cast the ray associated with the ray footprint for each triangle. For example, at step 674, after the ray footprint has been determined, the system may fetch the triangle from the cache. The triangle may be iteratively tested for each ray sample in the ray footprint. For example, at step 676, for each ray sample, the system may calculate a subsample offset within the pixel footprint via a lookup table (e.g., Figure 2 , label 260a). At step 678, the system can calculate the ray-triangle intersection of the ray defined by the subsample (e.g., Figure 3 , label 360a). At step 680, when it is determined that the ray intersects the triangle, the system can store information associated with the intersection. For example, in the case of a successful intersection, the system can update the subsample depth and triangle index. After a single ray of a pixel / subpixel has been tested against a triangle, the system can determine at step 682 whether additional samples should be made (e.g., based on the Figure 2 , with five subsamples per pixel). If so, the system can repeat step 676 to generate another ray and test for intersection. If the triangle has been tested for all subsamples, then at step 684, the system can determine whether there are more triangles that intersect the block beam as determined during the block culling phase. If so, the system can repeat step 674 to sample another triangle using the ray associated with the current ray footprint. If there are no more triangles, the system can determine whether there are more pixels / subpixels within the block that should be tested for triangles. If so, the system can repeat step 670 to perform intersection tests for triangles that intersect the block using rays from another pixel / subpixel.

[0067] Once all triangles have been processed, visibility of the block is fully resolved, and the depth and triangle index buffers for each subsample contain the most recent hits for each subsample. In a particular embodiment, at this point, the subsample data may be compressed and emitted to a "gbuffer" in preparation for shading. In this case, the "gbuffer" may contain only visibility information: pairs of triangle index and subsample mask, which is sufficient to recalculate the center of gravity and fetch vertex attributes during the shading stage. The "gbuffer" may be allocated to be large enough to hold a fixed number of entries (e.g., 32) to handle the case where each subsample of each subpixel hits a different triangle and is stored in global memory on the GPU. The memory is addressed so that the first triangle for each pixel is adjacent in memory, followed by the second triangle, etc., so that only a small prefix of the buffer is actually used. In a particular embodiment, compression may perform the following steps: (1) sort the subsamples by triangle index; and (2) iterate over the subsamples and emit, for example, a triangle index and multisample mask for each unique triangle index. Once the "gbuffer" is built, the sample testing phase is complete, and the parsing / shading phase begins.

[0068] After visibility is calculated during the sample testing phase, the system can perform shading, aggregate MSAA or SSAA samples and calculate the final pixel / sub-pixel color into a buffer that can be presented as a computer-generated scene including a rendering of visible objects defined within a 3D space. In a particular embodiment, each sample position is read from the output of the previous stage and ray intersections are calculated for each "G-buffer" entry at the center of the pixel. Shading can then be performed using attributes that are interpolated using the centroids obtained during the intersections, and shading results can be accumulated per pixel. Once all "G-buffer" entries have been processed for each pixel, the system can perform filmic tone mapping and output the results to a final buffer for display. In the case of SSAA, rays can be generated, intersected, and shaded independently, rather than having a single weighted shading result for each entry.

[0069] In a particular embodiment, the parsing / shading stage may include the following steps. For example, the system may look up sample locations (e.g., from a linear buffer of samples generated in a previous stage). The system may then calculate differences (e.g., analytically when in closed form, otherwise in finite differences). The system may then transform the rays and differences to 3D world space in preparation for intersection. The system may clear the shading result accumulator to 0. Then, for each "G buffer" entry, the system may obtain triangle data, perform ray-triangle intersection on pixel centers, and calculate depth and barycentric coordinates, clamp the barycentric coordinates to the triangle boundaries, interpolate vertex attributes based on the barycentric coordinates, perform shading / lighting, and accumulate shading results. The system may then scale the accumulated shading results by, for example, 1 / subsample rate. The system may then perform tone mapping and output the results.

[0070] Specific examples may be repeated as appropriate. FIG. 6A to FIG. 6C Although the present disclosure describes and illustrates one or more steps that occur in a particular order, FIG. 6A to FIG. 6C specific steps of the method, but the present disclosure contemplates that they occur in the appropriate order FIG. 6A to FIG. 6C Furthermore, although the present disclosure describes and illustrates example methods for performing primary visibility calculations, including FIG. 6A to FIG. 6C The specific steps of the method, but the present disclosure contemplates any suitable method for performing the master visibility calculation, including any suitable steps, which may appropriately include FIG. 6A to FIG. 6C All, some, or none of the steps of the method FIG. 6A to FIG. 6C Furthermore, although the present disclosure describes and illustrates the steps of performing FIG. 6A to FIG. 6C specific components, devices or systems for specific steps of the method, but the present disclosure contemplates performing FIG. 6A to FIG. 6C Any suitable combination of any suitable components, devices or systems of any suitable steps of the method.

[0071] An advantage of this embodiment is that it can be implemented in a traditional graphics pipeline. At a high level, an application that needs to be rendered can issue instructions to a graphics driver, which in turn can communicate with an associated GPU. Through the graphics application programming interface (API) of the driver, the application can specify how the scene should be rendered. For example, an application can submit a geometric shape definition of an object in a 3D space representing a scene to be generated for it. In a specific embodiment, the application can also submit a light buffer that defines how the position and trajectory of the light are defined. In order to avoid the runtime cost of requiring a large amount of memory to store the light buffer and read / write buffer, a specific embodiment can further allow the user to specify a definition of a process that can be used to programmatically generate light at runtime. Based on the information provided, the rendering system can perform visibility calculations, such as using ray casting as described above. Once the visibility of the geometry is determined, the system can perform shading and output the final scene.

[0072] The flexibility provided by the embodiments described herein enables rendering systems to natively implement a variety of rendering features related to virtual reality or augmented reality, in contrast to existing systems that use post-processing to simulate such effects. In addition, existing graphics APIs focus on the specific case of a uniform grid of primary sampling points. While current hardware rasterizers are highly tuned for this use case, rendering for AR / VR displays requires additional high-performance functionality that is more naturally implemented through ray casting according to specific embodiments. In particular, existing graphics APIs are unable to handle the following cases: (1) direct sub-pixel rendering with known optical distortion on different sub-pixel arrangements; (2) changing multi-sample and / or shading rates on the screen (e.g., for concave rendering / scene-aware work rebalancing); (3) depth sampling patterns for depth approximation of zoom displays; and (4) beam racing.

[0073] Specific embodiments described herein support the aforementioned use cases based on the following features, which can be implemented as enhancements to the visibility determination portion of the graphics pipeline. For example, to support rendering on different sub-pixel arrangements with optical distortion, the system can allow applications using the rendering engine to specify and render using non-uniform grid sampling patterns and use independent color channel sampling. To support variable multi-sample / shading, the system can allow applications to specify a "importance" measure for each pixel / block / tile. To support depth of field, the system can allow applications to specify and render using non-uniform grid sampling patterns and non-point source sampling patterns. Each of these features, along with beam racing, will be described in further detail.

[0074] At a high level, when any type of lens distortion is required, distortion can be applied to rays (e.g., to determine ray directions) before computing the bounding beams via principal component analysis. When sub-pixel rendering is enabled, consider using separate grids and tiling and blocking the red, green, and blue channels separately. When depth of field is present, the beams can be expanded to accommodate the distribution of ray origins. When a foveated ray distribution is used, ray bundles can be generated using a top-down splitting algorithm to construct blocks containing no more than n (e.g., 128) pixels and tiles containing no more than m (e.g., 64) blocks. The system can support partially occupied blocks and tiles. For most use cases, these boxes can be calculated once at the beginning of time based on the lens parameters of the system. However, in cases where the parameters change between frames, such as points of interest during foveated rendering, they can be recalculated on a per-frame basis. The bounding beams can define the full footprint of each pixel, not just their center, to support MSAA.

[0075] Certain embodiments enable a graphics system to support direct optical distortion rendering and sub-pixel rendering. One of the main differences between head-mounted and traditional displays is the use of viewing optics. In addition to allowing the user to focus on the display, the viewing optics also add various aberrations to the display when viewing. Notably, head-mounted displays typically produce pincushion distortion with color dependence, which results in color separation and uneven pixel spacing. This causes the user to effectively see three different displays, one for each color (e.g., red, green, and blue), with three different distortion functions. Typically, these artifacts can be corrected during a post-processing image distortion stage. For example, a traditional rendering system that does not support direct distortion rendering produces a traditional rectangular image. In order to correctly view the image via a head-mounted display, the post-processing stage takes the rectangular image and creates a distorted image for the head-mounted viewing optics. Traditional multi-stage processing is not only inefficient, but also produces suboptimal results.

[0076] Certain embodiments described herein enable a renderer to accept / receive lens parameter information (e.g., lens aperture, curvature, etc.) and use such information to produce an intrinsically distorted image without the need for post-processing. Advantages of direct sub-pixel rendering include increased clarity. By taking into account the spatial position of the sub-pixel, a more precise intensity can be defined for each sub-pixel rather than for the entire pixel at the same time. In addition, direct sub-pixel rendering eliminates the need for large pre-distorted rendering targets. For example, some conventional VR devices have an eye buffer resolution of 3024×1680, or 1.96 times the output display resolution of 2160×1200 pixels. Assuming that the image presented to the display is cropped by a barrel distortion mask, only about 85% of the 2160×1200 pixels are actually rendered. Due to over-rendering of these large guard bands or eye buffers, less than 50% of the total pixels need to be rendered. Therefore, by performing direct rendering using the embodiments described herein, a large amount of computation can be saved. Direct subpixel rendering further improves image quality by avoiding the distortion correction resampling pass, which introduces both aliasing and blurring, most noticeably for near-Nyquist details such as text. Finally, by removing the full-frame distortion correction pass, direct distortion rendering has a chance to render in time.

[0077] To support optically distorted rendering on different sub-pixel arrangements, the system can allow applications using a rendering engine to specify and render using a non-uniform grid sampling pattern and use independent color channel sampling. Optical distortion is typically caused by the physical properties of a lens, which bends light in a distorted manner. In conventional rendering systems, it is assumed that the camera is a pinhole camera with uniform optics. To generate an image that would be captured by such a pinhole camera, conventional systems project light from the pinhole camera into 3D space through points on a flat, uniform virtual image plane. Therefore, a uniform grid sampling pattern is conventionally sufficient. However, because the optical structure of the real world is not perfect and is complex (e.g., with multiple lens stacks), light does not actually enter and leave the optical structure in a straight line, and the degree to which the light bends depends on the point at which it enters / exits and the curvature and density of the optical structure. Therefore, images generated using a uniform grid pattern cannot accurately represent what a person expects to perceive.

[0078] A particular embodiment of the rendering system allows an application (e.g., a VR or gaming application that sends a request to the rendering system) to specify a focal plane map. A focal plane map can be defined as a surface in space with any desired curvature. Conceptually, the focal plane map defines a corresponding point in space for each pixel that will be in focus when seen through the pixel. In a particular embodiment, the focal plane map can map screen coordinates to specific locations in 3D world or camera space. The focal plane map can act as a control surface similar to specifying how screen pixels should be expanded into 3D world space. In a particular embodiment, the focal plane map can have the same 2D dimensions as the output image, so that there is a 1 to 1 mapping between the values ​​stored in the focal plane map and the pixels in the output image. Therefore, when determining the visibility of a particular pixel, the system can look up the corresponding value in the focal plane map to find the point in 3D space that the light should pass through. In a particular embodiment, the surface map can be stored as a float3 or half3 buffer that maps screen space to 3D world space (or camera space). In certain embodiments, if the focal plane map is smaller than a pixel, the focal plane map may be defined as a Catmull-Rom surface and interpolation may be used to determine the exact coordinates in 3D space to which the pixel is mapped. In certain embodiments, to handle fields of view greater than 180 degrees, the system may switch the vertex / geo / tessellation shaders to output world space positions. In certain embodiments, if a focal plane map is defined, vertex positions for the entire shader pipeline may be defined in world / camera space.

[0079] Figure 7 An example of how to display a focus plane map 750 in 3D world space is shown. In certain embodiments, the focus plane map 750 may be defined mathematically or algorithmically. In other embodiments, the focus plane map may be defined as a lookup table. In either case, given a coordinate in screen space, the focus plane map 750 may be used to determine the corresponding coordinate in 3D space. For example, Figure 7 Included is a conceptual representation of a virtual image sensor array 710 (which may also represent a corresponding screen). Each illustrated grid in sensor 710, such as grids 711-713, may represent a corresponding pixel of the sensor. Focus plane map 750 may define specific points 751, 752, and 753 in 3D space that correspond to representative points 711, 712, and 713 of their respective pixels (or sub-pixels), respectively. As previously described, each point, such as point 751, may conceptually represent a point in space that would be in focus as seen at corresponding point 711. During ray casting / tracing, rays 721, 722, and 723, passing through points 751, 752, and 753, respectively, may be cast from points 711, 712, and 713, respectively, as specified by focus plane map 750. Rays 721, 722, and 723 may be used to determine object visibility. For example, Figure 7712, but not from points 711 and 713. In certain embodiments, subsample rays associated with each pixel may be similarly mapped onto focus plane map 750. For example, in addition to point 711, additional subsample positions may be associated with the same pixel footprint. Projections of subsample positions may be defined on focus plane map 750. For example, the projections of subsample positions may be defined within the same grid in which point 751 is defined. During visibility testing, subsample rays may be cast from the subsample positions with their corresponding projections on focus plane map 750 to test for object intersection.

[0080] The focal plane map 750 provides the application / user with the flexibility to define any desired light direction to generate any desired resulting effect. For example, if the application wishes to simulate optical distortion, it can define a focal plane map 750 that approximates the desired distortion caused by the optics. Based on the focal plane map 750, individual rays can be cast in any direction (rather than uniformly through a regular grid), similar to how an optical structure would bend light.

[0081] Focal plane maps can also be used to support wide field of view (FoV) rendering. A significant difference between head-mounted and traditional displays is that head-mounted displays can support wider fields of view. Whereas typical desktop displays consume approximately 30° of the viewer's FoV, current VR headsets start at approximately 90° and can go up to 210°. Due to the uniform sample grid assumption made in traditional depth buffer algorithms, the number of samples required to render an image increases with the tangent of the FoV half angle (assuming a fixed minimum angular resolution). This causes a sharp drop in efficiency toward 180° FoV. For example, the difference between 140° and 160° is almost 4 times the number of rendered pixels. To alleviate this inefficiency, methods of rendering multiple narrow FoV images and stitching them together have been studied. While this improves the efficiency of the depth buffer algorithm, the cost is repeated processing of geometry for each additional view, as well as increased and potentially noticeable stitching artifacts. To address these shortcomings, the embodiments described herein adapt the raycaster to support arbitrary ray distributions to support wide FoV rendering. As another example, if an application desires to render a scene with a wide field of view (e.g., greater than 90°, 120°, 180°, 210°, etc.), it may define a focal plane map that causes light rays to gradually bend outward as the ray origin moves away from the center of the screen (e.g., the minimum angle between two light rays (such as those cast from opposite sides of the scene) may be greater than 90°, 120°, 180°, 210°, etc.). For example, the focal plane map may define a concave hyperboloid relative to the camera. The camera may be located within the curvature surface, such as at the center of the curvature (e.g., to obtain a 180° FoV), behind the center (e.g., to obtain a 120° FoV), or in front of the center (e.g., to obtain a 210° FoV). Because the focal plane map provides the application with flexibility to specify how the light rays should travel, a wide FoV may be generated natively without resorting to inpainting operations that are subject to the limitations and costs described above.

[0082] Focus plane maps can additionally be used to support defocus blur and depth of field blur, which are desired effects in VR / AR because they provide the level of realism expected by humans. Defocus blur is a natural effect of all non-pinhole cameras and is a common simulated effect in synthetically generated images. In VR, depth of field is also an important effect in variable focal length displays, where content that is out of the focus plane needs to be rendered with synthetic blur to match the expectations of the human visual system. However, traditional rendering systems often assume that the camera is a pinhole camera, resulting in sharp focus throughout (in other words, with infinite depth of field). To simulate blur, traditional systems perform post-processing on the sharply rendered image. Traditional processing to simulate defocus or depth of field blur is not only inefficient, but the blur effect is also inaccurate. This is because simulated blur is usually performed by averaging neighboring pixel colors, while actual defocus or depth of field blur is caused by the behavior of light and lens physics.

[0083] Certain embodiments enable a rendering system to support certain camera parameters that can be used to randomly generate defocus or depth of field blur. For example, certain embodiments can use aperture and depth of field parameters representing the visual system (e.g., via a focal plane map) to generate real-time random blur. A unique feature of the embodiments described herein is that despite the large number of additional rays, the same machine is used for both defocus blur and MSAA, which allows the system to shade at a rate close to the display resolution. Shading differences can be calculated based on the width of the depth of field grouping, which in addition to providing correct pre-filtering, also improves performance by requiring a coarser MIP level during texture reading. In addition to efficiently supporting defocus blur, the embodiments described herein can do so accurately and comprehensively. In contrast, image-space defocus blur algorithms (e.g., post-processing the output image by blurring neighboring pixels) cannot be natively integrated with various effects such as concave (due to the need to synthesize dense rectilinear color and depth maps from the concave image before calculating the image-space depth of field) or wide field of view (due to non-uniform anisotropic differences in angular resolution between pixels on the image plane, requiring simultaneous stitching of multiple views, or both).

[0084] To support defocus blur and / or depth of field blur, certain embodiments allow applications to specify and render using non-uniform grid sampling patterns and non-point source sampling patterns. As described elsewhere herein, certain embodiments may allow users to define focus plane maps. In addition, certain embodiments may allow applications to specify thin lens models for modifying parameters of the light source. In certain embodiments, the radius of the lens or the aperture of the virtual camera may be specified.

[0085] Figure 8An example of using focus plane map 850 and camera parameters (e.g., aperture, lens size, etc.) to locally (rather than via post-processing) generate an image with defocus or depth of field blur is shown. In certain embodiments, focus plane map 850 can be defined mathematically or algorithmically. In other embodiments, the focus plane map can be defined as a lookup table. In either case, given a coordinate in screen space, focus plane map 850 can be used to determine the corresponding coordinate in 3D space. For example, Figure 8 A conceptual representation of a virtual image sensor array 810 is included (which may also represent a screen of pixels). Each of the illustrated grids in the sensor 810 may represent a sensor of pixels. A focus plane map 850 may define specific points in 3D space that correspond to specific points on the screen / sensor 810. For example, according to the focus plane map 850, a point 855 in 3D space may be mapped to a representative point 815 of a sensor or pixel (or sub-pixel) in the sensor array or screen. As previously described, the point 855 may conceptually represent a point in space that would be in focus as seen at the corresponding point 815. As previously described, the sub-sample positions of each grid of the sensor may similarly be mapped to specific points on the focus plane map 850.

[0086] In addition to focal plane map 850, certain embodiments may also enable an application to define parameters of a virtual camera. Certain embodiments may enable an application to define a more realistic virtual camera, rather than using a pinhole camera represented by a point in space. For example, Figure 8 Parameters of a virtual camera with a grid 830 are represented. Since a camera has physical dimensions (rather than points in space), the grid 830 is used to represent various sampling points (e.g., points 831, 832, 833) through which light can travel. For example, the grid 830 can represent uniform sampling points on the aperture or lens of the virtual camera. In certain embodiments, the application can specify the size of the aperture / lens (e.g., by specifying the radius of the circular area of ​​the aperture / lens, the difference of the square area of ​​the aperture / lens, or the length and width of the rectangular area of ​​the aperture / lens). In turn, the size can be used to determine the size / dimension of the grid 830. In certain embodiments, the granularity of the grid 830 can also be controlled by the application. Although the grid 830 is used as a data structure for representing a lens or an aperture of a lens in this example, the present embodiment is not limited to the use of a grid. For example, the spatial point through which the projected light passes can be represented by a vector or a list of coordinates, or it can be defined programmatically (e.g., by a mathematical formula).

[0087] During ray casting / tracing, certain embodiments may iteratively determine the object visibility for each pixel. Figure 8In the example shown in , the system can determine the object visibility of point 815. Because the color of the pixel associated with point 815 can depend on the light incident from different points of the lens, certain embodiments can conceptually project light rays originating from point 815 through various points such as 831-833 defined by grid 830. In other words, these points 831-833 can be considered as samples of points on the lens through which light can hit the virtual camera sensor represented by point 815. In certain embodiments, the system can project rays 821, 822, and 823 from points 831, 832, and 833, respectively, through point 855 as defined by focus plane map 850. In the example shown, ray 821 misses object 840, ray 822 barely misses it, and 823 intersects it. Because the intersection points (or lack of intersection points) of the rays are sufficiently different, the resulting color calculation of the pixel represented by point 815 will not be a pure color calculation of object 840, and therefore, object 840 will appear blurry in the final scene. Conversely, if object 840 is located at or near point 855, the object will be within the depth of field and appear sharper. Similarly, visibility testing can be performed at the subsample level. In this case, each subsample position (e.g., a subsample position associated with a pixel represented by point 810) can be similarly mapped to a point on the focus plane map 850 (e.g., a point in the grid represented by point 855). Visibility testing can be performed by casting subsample rays from the subsample position, through various points defined by grid 830 (e.g., through an aperture), and through corresponding points on the focus plane map 850.

[0088] Fig. 9 A method for locally generating an image with optical distortion for a VR device is shown according to a particular embodiment. At step 910, a computing system may receive a focal plane map specified by an application via an application programming interface (API). For example, a VR / AR application (e.g., a game) may send instructions to a graphics API, including a definition of a focal plane map. In a particular embodiment, the focal plane map may be defined as a coordinate grid relative to a plane representing an image sensor array. For example, each grid in the focal plane map may be mapped to or associated with a grid representing an array sensor or corresponding pixel on a screen. For example, the upper left grid in the focal plane map may be associated with the upper left sensor / pixel, the upper right grid in the focal plane map may be associated with the upper right sensor / pixel, and so on. Each grid in the surface map may store coordinates in a 3D space, where the relative position of the surface and the sensor array or pixel may be defined (the 3D space may be different from the 3D world space for which the scene is to be generated). In a particular embodiment, the focal plane map may be specified by an algorithm instead. For example, a mathematical function or a rule-based algorithm may take a coordinate (e.g., a 2D coordinate representing a point in a sensor array or pixel screen) as input and output a 3D coordinate.

[0089] At step 920, the system may determine a position in three-dimensional space (3D), which may represent the position of a virtual camera or viewer in 3D space. In certain embodiments, the position may be specified by a VR application, which in turn may be based on sensor data generated by a virtual reality device. For example, a head mounted device may include a gyroscope or inertial measurement unit that tracks the real-time motion of a user and outputs sensor data to represent or describe the motion. The sensor data provided by such a motion tracking sensor may be used by the VR application to determine the current position of the user and provide the position to a rendering engine to position / reposition the virtual camera in 3D space.

[0090] At step 930, the system may generate a plurality of first coordinates in 3D space based on the determined orientation. For example, the plurality of first coordinates may represent pixels on an image sensor of a virtual camera or a corresponding virtual screen. In certain embodiments, one or more transformation matrices or other transformation algorithms may be used to position and orient the virtual image sensor / screen in 3D space. Each point on the virtual image sensor / screen that the rendering system will sample may be represented by a coordinate in 3D space.

[0091] At step 940, the system can use the plurality of first coordinates and the focal plane map to generate a plurality of second coordinates in 3D space. For example, the second coordinates can be points in 3D space representing where the object will appear in focus (e.g., represented by Figure 7 Each of the plurality of first coordinates may be associated with one of the plurality of second coordinates. For example, Figure 7 As shown, point 711 (defined by coordinates) is mapped to or associated with point 751 according to focus plane map 750 .

[0092] In certain embodiments, the system can generate the second coordinate by performing a lookup operation using the focus plane map. For example, for each of the first coordinates (e.g., each camera sensor / pixel), the system can look up a corresponding grid in the focus plane map to identify the coordinate in the focus plane map that maps to the first coordinate. In certain embodiments, the coordinates from the focus plane map can be defined in a space different from the 3D world space for which the scene is being generated. Therefore, in certain embodiments, the system can use one or more transformation matrices or algorithms to transform the coordinates from the focus plane map to the 3D world space to generate, for example, point 751 in the 3D world space.

[0093] In certain embodiments, the system iteratively and programmatically generates the second coordinates. For example, in certain embodiments, the focal plane map can be specified by an algorithm. Thus, the system can generate the plurality of second coordinates by processing the plurality of first coordinates according to the algorithm. For example, the algorithm can be mathematically defined so that given an input coordinate (e.g., a coordinate such as Figure 7 ), the algorithm will output the corresponding coordinates. In certain embodiments, the output coordinates may be in the same space as the space in which the input coordinates are defined. For example, a certain embodiment of the algorithm may be configured to receive input coordinates defined in a non-world space. Thus, the output of the algorithm may be the relative position of the focal point in the non-world space relative to the input coordinates. If so, the system may be transformed to 3D world space based on the current orientation of the virtual camera (e.g., via a transformation matrix). In other embodiments, the algorithm may transform coordinates defined in 3D world space (e.g., Figure 7 ) as input, and outputs coordinates in 3D world space (e.g., Figure 7 In this case, no subsequent transformation is required.

[0094] At step 950, for each of the plurality of first coordinates, the system may cast a ray from the first coordinate through the associated second coordinate to test for intersection with one or more objects to determine visibility of the one or more objects defined in the 3D space. Figure 7 In FIG. 7 , ray 721 is projected from point 711 through the relevant point 751 defined by the focal plane map. The projected ray can be used to test for intersection (or visibility) with any object defined in the 3D world space. For example, Figure 7 Object 740 is shown as being invisible to the sensor / pixel at points 711 and 713, but visible to the sensor / pixel at point 712. In certain embodiments, the rays may be subsample rays, rather than primary rays associated with pixels as described elsewhere herein. In certain embodiments, beams (e.g., beams projected from blocks or regions) may alternatively be projected. In certain embodiments, the associated primary rays and focal plane map may be used to calculate the coverage of the beam in 3D world space. For example, using the focal plane map, the trajectory of each primary ray may be determined. In order to determine the coverage of the beam, the system may use the ray differences described above to calculate its ray footprint, and in turn the ray footprints associated with the beam may be used together to define the coverage of the beam as it is projected through the 3D world space.

[0095] At step 960, the system can generate an image of one or more objects in the 3D world space based on the determined visibility of the one or more objects. As previously described, the system can determine object visibility from the perspective of the virtual camera using projected rays. Object visibility can be tracked by storing the depth of each intersecting object (e.g., a triangle or other primitive) and its index. Using this visibility information, the system can proceed with shading to determine what color should be assigned to each pixel. The pixels of the collection will then be used to generate an image, which will be a scene in the 3D world space from the vantage point of the virtual camera. The image can include any object defined within the 3D world space visible to the virtual camera.

[0096] Specific examples may be repeated as appropriate. Fig. 9 Although the present disclosure describes and illustrates one or more steps that occur in a particular order, Fig. 9 The specific steps of the method are described in detail, but the present disclosure also contemplates that they occur in any suitable order. Fig. 9 In addition, although the present disclosure describes and illustrates an example method for locally generating an image with optical distortion for a VR device, the method includes Fig. 9 The specific steps of the method, but the present disclosure also contemplates any suitable method for doing so, the method including any suitable steps, which may appropriately include Fig. 9 All, some, or none of the steps of the method Fig. 9 Furthermore, although the present disclosure describes and illustrates the steps of performing Fig. 9 specific components, devices or systems for specific steps of the method, but the present disclosure also contemplates performing Fig. 9 Any suitable combination of any suitable components, devices, or systems of any suitable steps of the method.

[0097] Another benefit of certain embodiments is native support for generating concave images. Conventional graphics drivers lack support for efficient concave rendering because they assume that pixels have equal values, weights, and write qualities. Therefore, when conventional systems generate images, each pixel is treated equally, regardless of location, including the computational resources consumed. However, equal treatment of pixels is not suitable for concave imaging, which is often desired in VR. For example, in a VR scene, the peripheral view may be blurred, or a concave effect may help direct the user's attention to a specific gaze point. The blurred portions of the concave image may be provided with less detail, and therefore the computational resources consumed on the rendering area outside the gaze point would be wasted or undesirable.

[0098] Certain embodiments of the rendering system may accept additional information to generate images with quality that varies as a function of screen position. Since the system is not limited to sampling on a uniform grid, it can closely match resolution drops (e.g., higher resolution samples at the gaze point, and lower samples elsewhere), resulting in fewer shading samples. Results show that by rendering concave sample distributions, shading costs are significantly reduced.

[0099] Certain embodiments support varied multisampling and shading that can be used to locally generate concave images by providing applications with the flexibility to specify a measure of the "importance" of each pixel / block / tile. In certain embodiments, the system can support an importance map that indicates (e.g., via float2 / half2 / unorm8, etc.) a measure of "importance," e.g., from 0 to 1 for each image unit (e.g., pixel, block, tile, etc.). In certain embodiments, the application can also specify the granularity of importance (e.g., pixel, block, tile, etc.). For example, through a graphics driver's API, the application can specify an importance map with an associated granularity for each scene the application wishes to generate. In certain embodiments, the importance map can be implemented using a grid data structure, where each grid maps to a pixel or any other type of image unit.

[0100] Fig.10 An example of an importance map 1010 is shown. In this example, the importance map 1010 is a 6×6 map that can be used to specify the relative importance of an image having 6×6 pixels. In a particular embodiment, the relative position of each grid in the importance map 1010 corresponds to the pixel at the same corresponding position in the image. For example, the top left grid in the importance map 1010 can correspond to the importance of the top left pixel of the image. Through the importance map 1010, an application can individually specify the relative importance of each pixel / block / tile. In a particular embodiment, an importance value close to 1 can indicate a higher importance, and an importance value close to 0 can indicate a lower importance. If so, then Fig.10 The importance map 1010 in FIG. 1 indicates that the center pixel is important (e.g., the grid marked with 1), the immediately surrounding pixels are of moderate importance (e.g., the grid marked with 0.6), the pixels forming the top and right edges of the image are of relatively less importance (e.g., the grid marked with 0.4), and finally, the remaining pixels on the bottom and left edges are of the least importance (e.g., the grid marked with 0).

[0101] In certain embodiments, the importance map may be used to adjust multi-sample anti-aliasing (MSAA), shading, and any other graphics tasks. Fig.10An example of an importance map 1010 is shown, as shown in MSAA map 1020, which is used to determine how many rays to use per pixel for MSAA. When the graphics system performs MSAA, it can use importance map 1010 to decide how many subsample rays to use for each pixel. In the example shown, the center pixel is sampled using 5 subsample rays because the corresponding grid in importance map 1010 indicates that those pixels have high importance. Since each of the immediately surrounding pixels has an importance value of 0.6, fewer subsample rays (e.g., 3 in the example shown) are used for sampling. In a similar manner, 2 subsample rays are used for pixels on the top and right edges because their respective corresponding importance values ​​are 0.4. The remaining pixels on the bottom and left edges are the least important and can be sampled using only 1 ray. Since the importance map indicates that some areas of the image are less important than other areas, the system can make intelligent decisions about where to save calculations. For example, if each pixel is to be treated equally, a conventional system may use 5 subsample rays for each of the 36 pixels, resulting in the use of 180 subsample rays. In contrast, by indicating the importance of each pixel, the system can perform MSAA using only 5×4+3×12+2×11+1×9=87 subsample rays.

[0102] In certain embodiments, importance values ​​may be selectively implicitly defined using a focus plane map. As described above, a focus plane map may define points that map to corresponding points on a virtual image sensor or screen. For areas on the screen that require higher resolution or more detail (e.g., areas at or near the gaze point), the corresponding definition of the points on the focus plane map may be denser or finer, which in turn may result in more rays / sub-rays being generated. Conversely, for areas that require lower resolution or detail (e.g., areas outside the gaze point), the corresponding definition of the points on the focus plane may be less dense or coarse, which in turn may result in fewer rays / sub-rays being generated.

[0103] Fig.11 An example method for generating an image based on multi-sample anti-aliasing with variations is shown. At step 1110, a computing system may receive an importance map specified by an application via an application programming interface (API). For example, a VR / AR application (e.g., a game) may send an instruction to a graphics API that includes a definition of an importance map. In certain embodiments, the importance map may include importance values ​​corresponding to pixels (or other image units, such as blocks and tiles) or their respective light footprints. In certain embodiments, the importance map may be stored in a memory such as a graphics card. Fig.10, in the grid data structure of the example shown. In certain embodiments, the importance map is optionally defined programmatically, such as based on an algorithm or mathematical formula that outputs an importance value as a function of pixel / footprint position. For example, the importance map may programmatically define the center as the most important, and the importance level may gradually decrease for pixels away from the center. In certain embodiments, what is "important" may also be specified based on the density of the focal plane map. For example, areas in the focal plane map with a denser grid (corresponding to a denser density of light that should be cast through those areas) may implicitly indicate that those areas are more important.

[0104] At step 1120, the system may determine an orientation in three-dimensional (3D) space, which may represent an orientation of a virtual camera or viewer in 3D space. In certain embodiments, the orientation may be specified by a VR application, which in turn may be based on sensor data generated by a virtual reality device. For example, a head mounted device may include a gyroscope or inertial measurement unit that may track the real-time motion of a user and output sensor data to represent or describe the motion. Sensor data provided by such motion tracking sensors may be used by a VR application to determine the current orientation of the user, and provide the orientation to a rendering engine to position / reposition a virtual camera in 3D space.

[0105] At step 1130, the system can generate a plurality of ray footprints in 3D space based on the determined orientation. For example, the plurality of ray footprints can respectively represent pixels on a virtual screen (or a corresponding image sensor array). In certain embodiments, each ray footprint can be defined by a center coordinate and at least one difference. For example, as previously described, a square or rectangular footprint can be defined by a vertical difference represented as a vector. In certain embodiments, based on the orientation of the virtual camera, one or more transformation matrices or other transformation algorithms can be used to position and orient the ray footprints in 3D space.

[0106] At step 1140, for each ray footprint, the system can identify a corresponding number of subsamples to generate for the ray footprint. In certain embodiments, the number of subsamples to generate can be determined based on importance values ​​defined based on an importance map (e.g., more subsamples are generated for significant importance values ​​and fewer subsamples are generated for lower importance values). In certain embodiments, the importance values ​​can be identified by looking up the importance map. For example, the corresponding importance value for a ray footprint can be identified based on (1) the relative position of the importance value in a grid data structure and (2) the relative position of the corresponding pixel of the ray footprint in a pixel array / grid. For example, referring to Fig.10, the relative position of the upper left importance value 0.4 in the importance map can be identified as (1, 1) in the grid. This importance value can correspond to the pixel at the (1, 1) position in the image or its corresponding ray footprint. Optionally, as previously described, the importance map can be defined programmatically so that the importance map outputs the importance value of a given position. As another alternative, the importance value can be implicitly defined by the density of the area within the focal plane map (e.g., more subsamples are generated for denser areas, and fewer subsamples are generated for less dense areas). Thus, the importance map allows different ray footprints to have different importance values. For example, an application can use an importance map to define certain pixels (or their corresponding ray footprints), such as the center pixel of an image, as being more important than other pixels, such as pixels on the edge of the image.

[0107] At step 1150, the system can generate one or more coordinates in each ray footprint based on the corresponding importance value (explicitly defined by an importance map, programmatically defined, or implicitly defined by a focal plane map). The coordinates can be subsample locations within the ray footprint. In certain embodiments, the number of coordinates to use can be a function of the importance value. For example, if the maximum number of subsamples that can be used per ray footprint is 5, and the importance value can range from 0 to 1, 5 subsample rays can be used for importance values ​​from 0.8 to 1, 4 subsample rays can be used for importance values ​​from 0.6 to 0.8, 3 subsample rays can be used for importance values ​​from 0.4 to 0.6, 2 subsample rays can be used for importance values ​​from 0.2 to 0.4, and 1 subsample ray can be used for importance values ​​below 0.2. Thus, if a ray footprint has an importance value of 1, the system can generate 5 subsample coordinates, which will be greater than the number of subsample coordinates generated for another ray footprint with a lower importance value (such as 0.4). In embodiments where the importance map is used to generate a concave image with a fixation region that should be focused / sharpened, ray footprints with an importance value of 1 may be associated with the fixation region, and ray footprints with lower importance values ​​(e.g., 0.4) may be associated with regions of the image outside of the fixation region.

[0108] At step 1160, the system can determine the visibility of one or more objects defined in the 3D space by projecting rays from each of the one or more coordinates to test for intersection with the one or more objects. In certain embodiments, rays can be considered as subsample rays projected from the footprint of the pixel. In the 3D scene, any intersection with an object may result in the depth of the intersection point and the index of the triangle hit being stored.

[0109] At step 1170, the system can generate an image of one or more objects in the 3D world space based on the determined visibility of the one or more objects. As previously described, the system can determine object visibility from the perspective of the virtual camera using projected light. Object visibility is tracked by storing the depth of each intersecting object (e.g., a triangle or other primitive) and its index. Using this visibility information, the system can proceed with the shading process to determine what color should be assigned to each pixel. The pixels of the collection will then be used to generate an image, which will be a scene in the 3D world space from the vantage point of the virtual camera. The image may include any object defined within the 3D world space that is visible to the virtual camera.

[0110] Specific examples may be repeated as appropriate. Fig.11 Although the present disclosure describes and illustrates one or more steps that occur in a particular order, Fig.11 The present disclosure contemplates the specific steps of the method occurring in any suitable order. Fig.11 In addition, although the present disclosure describes and illustrates an example method for generating an image based on multi-sample anti-aliasing, the method includes Fig.11 The specific steps of the method, but the present disclosure contemplates any suitable method for doing so, the method including any suitable steps, which may appropriately include Fig.11 All, some, or none of the steps of the method Fig.11 Furthermore, although the present disclosure describes and illustrates the steps of performing Fig.11 specific components, devices or systems for specific steps of the method, but the present disclosure contemplates performing Fig.11 Any suitable combination of any suitable components, devices, or systems of any suitable steps of the method.

[0111] Certain embodiments may provide flexibility to applications to account for differences in sub-pixel characteristics. Although a pixel is often thought of as a single physical display unit capable of displaying any color, it is actually composed of multiple sub-pixels located in different locations capable of displaying individual primary colors (e.g., red, green, and blue). Because different colors have different wavelengths, these colors also behave differently in the optical structure.

[0112] In order to generate a more realistic scene, in certain embodiments, the system may take into account these color differences at the sub-pixel level. In certain embodiments, the system may support per-channel focus plane maps (e.g., one surface map for red, one surface map for green, and one surface map for blue). For example, the system may allow the application to specify whether there is a single focus plane map or three focus plane maps for the three color channels, respectively. If three focus plane maps are used, one focus plane map may be associated with the first primary color (e.g., red), another focus plane map may be associated with the second primary color (e.g., green), and the third focus plane map may be associated with the third primary color (e.g., blue). The specific color focus plane maps are independent of each other and can be defined separately based on color differences (e.g., the three colors may have different focal plane characteristics to different wavelengths of light). During visibility testing, the system may project three groups of rays, each group of rays using one of the three specific color focus plane maps. For example, for a specific pixel, a group of rays / sub-rays may be projected in the direction specified by the red focus plane map, a second group of rays / sub-rays may be projected based on the green focus plane map, and a third group of rays / sub-rays may be projected based on the blue focus plane map. While more light may need to be projected compared to embodiments using a single focal plane diagram, one advantage is that the system can natively correct for chromatic aberration.

[0113] Similarly, certain embodiments may also support per-channel importance maps (e.g., one for red, one for green, and one for blue). For example, in embodiments that use multiple focus plane maps to simulate primary colors separately, the corresponding importance map may be used to specify the density of sub-sample rays that should be used as a function of the sensor / screen area.

[0114] The specific embodiments described herein support a technique called "beam racing". In a graphics rendering pipeline, each primitive is rendered in memory before the scene is rasterized. In other words, after the objects in the scene have been rendered, the pixels in the final scene are generated one by one. These pixels are displayed together and are assumed to represent the scene at a specific moment. However, since it takes time to generate pixels, there may be a significant time delay (e.g., 11 milliseconds) between the time an object is rendered and the time a pixel is displayed. In traditional display environments (e.g., movies, animations, etc.), the delay may not be noticeable. However, this is not the case in VR / AR environments. In VR / AR, users expect instant feedback between motion and visual perception. For example, when a user turns his head, he expects the scene to change at that moment and the current display to reflect his current viewpoint. Any delay, such as a time stamp for generating and outputting pixels after rendering, may have a negative impact on the user experience. For example, if the system can start rendering the scene based on the user's elevated perspective when the user stands up. However, when the pixels of the scene are output at time t0+11ms, the user may be sitting down. Because the user now expects to see the scene from a lower vantage point, seeing a scene that does not reflect this expectation will negatively impact the VR experience and may even cause motion sickness or nausea.

[0115] Fig.12 An example of comparing a graphics generation timeline without beam racing to a timeline with beam racing is shown. In particular, Fig.12 A graphics generation timeline 1220 is shown that generates and outputs an entire image simultaneously. In the example shown, a user wearing a VR device can quickly rotate his head from position 1210 at time t0 to position 1214 at time t4. If pipeline 1220 is configured to generate the entire image, it can start by configuring the orientation of the virtual camera based on the user's head orientation 1210 at time t0, and continue to shade and raycast the entire image. When the image is ready to be output, the time may be t4. However, at time t4, the user's head orientation 1214 may have changed significantly from the time t0 orientation 1210, yet the image presented to the user may be based on the user's head orientation 1210 at t0. This delay may cause the user to feel disoriented.

[0116] Certain embodiments reduce the delay between rendering and image display by scanning out pixels line by line, where each line is generated based on the rendering performed when it is the turn of that line to be output. For example, the system can render at time t0 and cast rays to line 0 based on t0 rendering (rather than the entire scene); render at time t1 and cast rays to line 1 based on t1 rendering; and so on. Since the system processes only one line at a time (or multiple predetermined lines at a time, rather than all lines together), the delay between rendering time and pixel output time becomes shorter, and the rendering will be based on the user's latest movement / viewing angle. Therefore, real-time scene changes will be more passive. This "beam racing" technology has the potential to significantly reduce the possibility of head movement to photon delays. Even significant batch processing, such as hundreds of lines (hundreds of thousands of pixels), may produce significantly reduced delays when waiting for the entire frame before scanning out. In certain embodiments, the system can schedule rendering and ray casting tasks relative to the scanout clock.

[0117] Reference again Fig.12 , the beam racing graphics pipeline 1230 shows an example of the beam racing technique. In this example, each video frame is displayed with 40 horizontal lines. The first timeline 1240 represents the timing of generating the first 10 lines (lines 1 to 10) of the video frame. At time t0, the system can use the latest motion sensing data available at that time (e.g., from the inertial measurement unit, gyroscope, etc. of the VR device) to position the virtual camera and perform visibility testing. The system can then perform shading and ray casting of video frame lines 1-10. In the example shown, lines 1-10 are ready and displayed to the user at time t1. Since the system is only tasked with generating 10 lines instead of all 40 lines, as shown in timeline 1220, the duration required to generate lines 1-10 (e.g., t1-t0) is significantly shorter than the duration required to generate the entire image (e.g., t4-t0). Therefore, at time t1, the user will be presented with lines 1-10 generated using the latest sensor information from t0, which is newer than the scene shown in timeline 1220. In certain embodiments, rows 1-10 may be displayed continuously until the remaining rows in the video frame have been generated, and the process will repeat to generate updated rows 1-10 based on the updated orientation / position of the virtual camera.

[0118] In a particular embodiment, after the system generates rows 1-10, it may continue to generate rows 11-20, as shown in timeline 1241. The process may begin at time t1, at which the system may perform a visibility test based on the latest sensor data available at that time. The system may again go through the process of shading and ray casting, and then output rows 11-20 at time t2. Thus, at time t2, rows 11-20 generated based on sensor data from time t1 are presented to the user. The system may then repeat the process to generate rows 21-30 starting at time t2 and ending at time t3, and then generate rows 31-40 starting at time t3 and ending at time t4. Thus, at time t4, a video frame including more current information (e.g., as early as time t3) is presented to the user compared to the scene presented by timeline 1220, where a frame generated based on the data is presented to the user at time t4.

[0119] In certain embodiments, the rendering system may further predict the user's head position / orientation to output a scene that matches the user's expectations. For example, if the user is in the process of turning, the system may predict that the user will continue to turn in the next frame and begin rendering the scene based on the predicted camera position / orientation. If the delay is 11ms, the system will have to predict further in advance, which is more difficult and more likely to go wrong. If the delay is significantly reduced (e.g., to 1ms), the system will only need to predict 1ms in advance. This makes the prediction task easier and less prone to error.

[0120] Fig.13 An example method 1300 for generating video frames for a VR display using beam racing is shown. The method may begin at step 1310, where a computing system may obtain sensor data generated by a virtual reality (VR) device at time t0, for example. For example, the VR device may include a head-mounted display and one or more motion sensors, such as an inertial measurement unit, a gyroscope, an accelerometer, and the like. At step 1320, the system may use the sensor data from t0 to determine a first orientation in three-dimensional (3D) space. For example, based on rotation data from a gyroscope and the last known orientation of the user in 3D space, the system may calculate the current orientation of the user. In particular embodiments, the orientation of the user may be represented in 3D space by orienting / positioning a virtual camera in 3D space.

[0121] At step 1330, the system can determine the visibility of one or more objects defined in the 3D space by casting rays based on the determined orientation to test for intersection with the one or more objects. For example, based on the orientation of the virtual camera in the 3D space, the system can cast rays into the 3D space to test for intersection with any objects defined therein. In certain embodiments, the direction of the rays can be based on a focal plane map (or multiple focal plane maps, one for each primary color), as described herein. The density of the rays can also be defined by a focal plane map or a separate importance map, as described herein.

[0122] At step 1340, the system may generate n rows (e.g., 1, 3, 5, 10, 100 rows, etc.) of pixels based on the determined visibility of one or more objects. In certain embodiments, the generation of pixels may be the result of shading and rasterization processing. The n rows generated may be a subset of the total rows of pixels in the VR display. Continuing with the above example, if the VR display has a total of 40 rows, at this stage, the system may generate rows 1-10 of the display. In other words, the number of rows generated may be a subset of the total number of rows.

[0123] At step 1350, the system may output the generated n rows of pixels for display by the virtual reality device. As previously described, in certain embodiments, the system may update only a subset, such as rows 1-10, rather than updating all rows of the display based on the same virtual camera orientation.

[0124] In particular embodiments, the process of generating and outputting a subset of rows may then be repeated for the next subset of rows. For example, at step 1310, the system may obtain the latest sensor data from the VR device (e.g., sensor data associated with time t1). At step 1320, the system may again determine a second position in the 3D space based on the latest sensor data from the VR device. At step 1330, the system may then continue to determine the visibility of any objects defined within the 3D space by projecting rays based on the second position to test intersections with the objects. At step 1340, the system may generate another n rows of pixels (e.g., rows 11-20) based on the determined second visibility of one or more objects. At step 1350, the system may output n rows of pixels for display by the virtual reality device. As shown in FIG. Fig.12 As shown, at the end of timelines 1240 and 1241, rows 11-20 are displayed simultaneously with rows 1-10.

[0125] Fig.13The steps shown can be repeated until all rows of the display are generated and output. Every n rows of pixels can be considered a subset of the total rows of pixels of the display, and every set of n rows can be generated sequentially and iteratively. For example, if the total number of rows is 40, the system can generate rows 1-10, 11-20, 21-30, and 31-40 in sequence, and the process can be repeated thereafter starting with the first set of rows (e.g., rows 1-10). Therefore, once the first set of n rows associated with time t0 is output, the set of rows can remain unchanged until each of the other rows of the display is generated (using sensor data generated by the VR device after t0). Once it is the turn of the first set of rows to be generated again, the first set of rows can be replaced by rows of pixels generated based on the latest sensor data.

[0126] exist Fig.13 , the process shows that the step of determining the second orientation is performed after outputting one or more first rows of pixels (e.g., rows of pixels generated based on the orientation associated with time t0). In certain embodiments, a multi-threaded or multi-processor computing system may begin the process of generating a next set of n rows of pixels before outputting a previous set of n rows of pixels. For example, if rows 11-20 are generated based on sensor data from time t1, time t1 may be any time when the system is processing steps 1320-1350 of the previous set of rows 1-10.

[0127] In certain embodiments, the number of rows generated per iteration may be equal to a predetermined number. For example, n may be preset to 1, 5, 10, 20, etc. In certain embodiments, the graphics system may predefine this number. Alternatively or additionally, the graphics system may allow an application to specify the number of rows that should be generated per iteration. For example, an application requesting the graphics system to generate a scene may specify the number of rows it wishes to generate in each iteration. For example, the application may pass this information to the graphics system via an API.

[0128] Specific examples may be repeated as appropriate. Fig.13 Although the present disclosure describes and illustrates one or more steps that occur in a particular order, Fig.13 specific steps of the method, but the present disclosure contemplates that they occur in the appropriate order Fig.13 Furthermore, although the present disclosure describes and illustrates an example method of generating video frames for a VR display using beam racing, the method includes Fig.13 The specific steps of the method, but the present disclosure contemplates any suitable method for doing so, including any suitable steps, which may appropriately include Fig.13 All, some, or none of the steps of the method Fig.13 Furthermore, although the present disclosure describes and illustrates the steps of performing Fig.13 specific components, devices or systems for specific steps of the method, but the present disclosure contemplates performing Fig.13 Any suitable combination of any suitable components, devices or systems of any suitable steps of the method.

[0129] Fig.14 An example computer system 1400 is shown. In certain embodiments, one or more computer systems 1400 perform one or more steps of one or more methods described or illustrated herein. In certain embodiments, one or more computer systems 1400 provide functionality described or illustrated herein. In certain embodiments, software running on one or more computer systems 1400 performs one or more steps of one or more methods described or illustrated herein, or provides functionality described or illustrated herein. Certain embodiments include one or more portions of one or more computer systems 1400. Herein, references to a computer system may include a computing device, and vice versa, where appropriate. Furthermore, references to a computer system may include one or more computer systems, where appropriate.

[0130] The present disclosure contemplates any suitable number of computer systems 1400. The present disclosure contemplates computer systems 1400 in any suitable physical form. As an example and not by way of limitation, computer system 1400 may be an embedded computer system, a system on a chip (SOC), a single board computer system (SBC) (such as, for example, a modular computer (COM) or a modular system (SOM)), a desktop computer system, a laptop or notebook computer system, an interactive kiosk, a mainframe, a computer system network, a mobile phone, a personal digital assistant (PDA), a server, a tablet computer system, an augmented / virtual reality device, or a combination of two or more thereof. Where appropriate, computer system 1400 may include one or more computer systems 1400; single or distributed; across multiple locations; across multiple machines; across multiple data centers; or reside in a cloud, which may include one or more cloud components in one or more networks. Where appropriate, one or more computer systems 1400 may perform one or more steps of one or more methods described or illustrated herein without substantial spatial or temporal limitations. As an example and not by way of limitation, one or more computer systems 1400 may perform one or more steps of one or more methods described or illustrated herein in real time or in batch mode. Where appropriate, one or more computer systems 1400 may perform at different times or different locations one or more steps of one or more methods described or illustrated herein.

[0131] In a particular embodiment, computer system 1400 includes a processor 1402, a memory 1404, a storage device 1406, an input / output (I / O) interface 1408, a communication interface 1410, and a bus 1412. Although this disclosure describes and illustrates a particular computer system having a particular number of particular components in a particular arrangement, this disclosure contemplates any suitable computer system having any suitable number of any suitable components in any suitable arrangement.

[0132] In a particular embodiment, the processor 1402 includes hardware for executing instructions, such as those that constitute a computer program. As an example and not by way of limitation, to execute instructions, the processor 1402 may retrieve (or fetch) instructions from an internal register, an internal cache, a memory 1404, or a storage device 1406; decode and execute them; and then write one or more results to an internal register, an internal cache, a memory 1404, or a storage device 1406. In a particular embodiment, the processor 1402 may include one or more internal caches for data, instructions, or addresses. The present disclosure contemplates that the processor 1402 includes any appropriate number of any appropriate internal caches, where appropriate. As an example and not by way of limitation, the processor 1402 may include one or more instruction caches, one or more data caches, and one or more translation lookaside buffers (TLBs). The instructions in the instruction cache may be copies of the instructions in the memory 1404 or the storage device 1406, and the instruction cache may speed up the retrieval of those instructions by the processor 1402. The data in the data cache may be a copy of data in the memory 1404 or storage device 1406 for instructions executed at the processor 1402 for operation; the result of a previous instruction executed at the processor 1402, for access by a subsequent instruction executed at the processor 1402 or for writing to the memory 1404 or the memory 1406; or other suitable data. The data cache may speed up the read or write operation of the processor 1402. The TLB may speed up the virtual address translation of the processor 1402. In a particular embodiment, the processor 1402 may include one or more internal registers for data, instructions, or addresses. The present disclosure contemplates that the processor 1402 includes any suitable number of any suitable internal registers, where appropriate. Where appropriate, the processor 1402 may include one or more arithmetic logic units (ALUs); be a multi-core processor; or include one or more processors 1402. Although the present disclosure describes and illustrates a particular processor, the present disclosure contemplates any suitable processor.

[0133] In a particular embodiment, the memory 1404 includes a main memory for storing instructions executed by the processor 1402 or data used by the processor 1402 to operate. As an example and not by way of limitation, the computer system 1400 can load instructions from the storage device 1406 or another source (such as, for example, another computer system 1400) to the memory 1404. The processor 1402 can then load the instructions from the memory 1404 to an internal register or an internal cache. In order to execute the instructions, the processor 1402 can retrieve the instructions from the internal register or the internal cache and decode them. During or after the execution of the instructions, the processor 1402 can write one or more results (which can be intermediate or final results) to the internal register or the internal cache. The processor 1402 can then write one or more of these results to the memory 1404. In a particular embodiment, the processor 1402 executes only instructions in one or more internal registers or internal caches or instructions in memory 1404 (as opposed to storage device 1406 or elsewhere), and operates only on data in one or more internal registers or internal caches or data in memory 1404 (as opposed to storage device 1406 or elsewhere). One or more memory buses (which may each include an address bus and a data bus) may couple the processor 1402 to the memory 1404. As described below, the bus 1412 may include one or more memory buses. In a particular embodiment, one or more memory management units are located between the processor 1402 and the memory 1404 and facilitate access to the memory 1404 requested by the processor 1402. In a particular embodiment, the memory 1404 includes a random access memory (RAM). The RAM may be a volatile memory where appropriate. The RAM may be a dynamic RAM (DRAM) or a static RAM (SRAM) where appropriate. In addition, the RAM may be a single-port or multi-port RAM where appropriate. The present disclosure contemplates any suitable RAM. The memory 1404 may include one or more memories 1404 where appropriate. Although this disclosure describes and illustrates particular memory, this disclosure contemplates any suitable memory.

[0134] In a particular embodiment, storage device 1406 includes a large-capacity storage device for data or instructions. As an example and not by way of limitation, storage device 1406 may include a hard disk drive (HDD), a floppy disk drive, a flash memory, an optical disk, a magneto-optical disk, a tape, or a universal serial bus (USB) drive or a combination of two or more thereof. Where appropriate, storage device 1406 may include a removable or non-removable (or fixed) medium. Where appropriate, storage device 1406 may be inside or outside computer system 1400. In a particular embodiment, storage device 1406 is a non-volatile solid-state memory. In a particular embodiment, storage device 1406 includes a read-only memory (ROM). Where appropriate, the ROM may be a mask-programmed ROM, a programmable ROM (PROM), an erasable PROM (EPROM), an electrically erasable PROM (EEPROM), an electrically variable ROM (EAROM) or a flash memory, or a combination of two or more thereof. The present disclosure contemplates the use of a large-capacity storage device 1406 in any appropriate physical form. Where appropriate, storage 1406 may include one or more storage control units that facilitate communications between processor 1402 and storage 1406. Where appropriate, storage 1406 may include one or more storage devices 1406. Although this disclosure describes and illustrates particular storage devices, this disclosure contemplates any suitable storage devices.

[0135] In a particular embodiment, I / O interface 1408 includes hardware, software or both, providing one or more interfaces for communication between computer system 1400 and one or more I / O devices. Computer system 1400 may include one or more of these I / O devices where appropriate. One or more of these I / O devices can enable communication between a person and computer system 1400. As an example and not by way of limitation, I / O devices may include a keyboard, a keypad, a microphone, a monitor, a mouse, a printer, a scanner, a speaker, a still camera, a stylus, a tablet computer, a touch screen, a tracking ball, a camera, another suitable I / O device or a combination of two or more of them. I / O devices may include one or more sensors. The present disclosure contemplates any suitable I / O device and any suitable I / O interface 1408 for them. Where appropriate, I / O interface 1408 may include one or more devices or software drivers that enable processor 1402 to drive one or more of these I / O devices. Where appropriate, I / O interface 1408 may include one or more I / O interfaces 1408. Although this disclosure describes and illustrates a particular I / O interface, this disclosure contemplates any suitable I / O interface.

[0136] In a particular embodiment, the communication interface 1410 includes hardware, software, or both, providing one or more interfaces for communication (such as, for example, packet-based communication) between the computer system 1400 and one or more other computer systems 1400 or one or more networks. As an example and not by way of limitation, the communication interface 1410 may include a network interface controller (NIC) or a network adapter for communicating with an Ethernet or other wired-based network or a wireless NIC (WNIC) or a wireless adapter for communicating with a wireless network (such as a WI-FI network). The present disclosure contemplates any suitable network and any suitable communication interface 1410 for it. As an example and not by way of limitation, the computer system 1400 may communicate with one or more parts of a self-organizing network, a personal area network (PAN), a local area network (LAN), a wide area network (WAN), a metropolitan area network (MAN), or the Internet, or a combination of two or more of them. One or more parts of one or more of these networks may be wired or wireless. For example, computer system 1400 may communicate with a wireless PAN (WPAN) (such as, for example, a BLUETOOTH WPAN), a WI-FI network, a WI-MAX network, a cellular telephone network (such as, for example, a Global System for Mobile Communications (GSM) network), or other suitable wireless networks, or a combination of two or more thereof. Where appropriate, computer system 1400 may include any suitable communication interface 1410 for any of these networks. Where appropriate, communication interface 1410 may include one or more communication interfaces 1410. Although this disclosure describes and illustrates particular communication interfaces, this disclosure contemplates any suitable communication interface.

[0137] In certain embodiments, bus 1412 includes hardware, software, or two components of computer system 1400 coupled to each other. As an example and not by way of limitation, bus 1412 may include an accelerated graphics port (AGP) or other graphics bus, an enhanced industry standard architecture (EISA) bus, a front side bus (FSB), a hypertransport (HT) interconnect, an industry standard architecture (ISA) bus, an INFINIBAND interconnect, a low pin count (LPC) bus, a memory bus, a micro channel architecture (MCA) bus, a peripheral component interconnect (PCI) bus, a PCI-Express (PCIe) bus, a serial advanced technology attachment (SATA) bus, a video electronics standard association local (VLB) bus, or other suitable busses or a combination of two or more thereof. Bus 1412 may include one or more buses 1412 where appropriate. Although the present disclosure describes and illustrates a particular bus, the present disclosure contemplates any suitable bus or interconnect.

[0138] Herein, computer-readable non-transitory storage media or media may include, where appropriate, one or more semiconductor-based or other integrated circuits (ICs) (such as, for example, field programmable gate arrays (FPGAs) or application-specific ICs (ASICs)), hard disk drives (HDDs), hybrid hard disk drives (HHDs), optical disks, optical disk drives (ODDs), magneto-optical disks, magneto-optical disk drives, floppy disks, floppy disk drives (FDDs), magnetic tapes, solid-state drives (SSDs), RAM drives, SECURE DIGITAL cards or drives, any other suitable computer-readable non-transitory storage media, or a suitable combination of two or more thereof. Computer-readable non-transitory storage media may be volatile, non-volatile, or a combination of volatile and non-volatile, where appropriate.

[0139] As used herein, "or" is inclusive and not exclusive, unless expressly indicated otherwise or the context indicates otherwise. Thus, as used herein, "A or B" means "A, B, or both," unless expressly indicated otherwise or the context indicates otherwise. Furthermore, "and" is both conjunctive and plural, unless expressly indicated otherwise or the context indicates otherwise. Thus, as used herein, "A and B" means "A and B, jointly or severally," unless expressly indicated otherwise or the context indicates otherwise.

[0140] The scope of the present disclosure includes all changes, substitutions, variations, alterations and modifications to the example embodiments described or shown herein that will be understood by a person of ordinary skill in the art. The scope of the present disclosure is not limited to the example embodiments described or shown herein. In addition, although the present disclosure describes and illustrates the various embodiments herein as including specific components, elements, features, functions, operations or steps, any of these embodiments may include any combination or arrangement of components, elements, features, functions, operations or steps described or shown anywhere herein that will be understood by a person of ordinary skill in the art. In addition, in the appended claims, a referenced device or system or component of a device or system that is suitable for, arranged to, capable of, configured to, enabled to, operable to, or operated to perform a specific function includes the device, system, component, whether or not it or the specific function is activated, switched on or unlocked, as long as the device, system or component is so adapted, arranged, capable of, configured to, enabled to, operable or operable. In addition, although the present disclosure describes or illustrates a specific embodiment as providing a specific advantage, a specific embodiment may not provide any of these advantages, provide some or all of these advantages.

Claims

1. A method of reducing rendering latency in rendered computer graphics, the method comprising, by a computing system: A first position of a viewer in a three-dimensional (3D) space is determined, wherein: The first orientation is based on first sensor data generated by the virtual reality device at a first time; determining first visibility of one or more objects defined in the 3D space by casting a ray based on the first orientation to test for intersection with the one or more objects; generating one or more first pixel rows based on the determined first visibilities of the one or more objects; displaying the one or more first pixel rows by the virtual reality device; determining a second position of the viewer in the 3D space, wherein the second position is based on second sensor data generated by the virtual reality device at a second time after the first time; determining a second visibility of the one or more objects defined in the 3D space by casting a ray based on the second orientation of the viewer to test for intersection with the one or more objects; generating one or more second pixel rows based on the determined second visibility of the one or more objects; and displaying the one or more second pixel rows by the virtual reality device; The one or more second pixel rows associated with the second orientation of the viewer and the one or more first pixel rows associated with the first orientation of the viewer are displayed simultaneously.

2. The method according to claim 1, wherein: The duration for generating the one or more first pixel rows is significantly shorter than the duration required for generating the entire image.

3. The method according to claim 1, wherein: The display of the virtual reality device has a plurality of pixel rows, wherein the one or more first pixel rows and the one or more second pixel rows are subsets of the plurality of pixel rows.

4. The method according to claim 3, wherein: The one or more first pixel rows and the one or more second pixel rows have an equal number of rows.

5. The method according to claim 4, wherein: The equal number of rows is specified by an application program through an application programming interface (API).

6. The method according to claim 3, further comprising: The one or more first pixel rows are replaced after each other pixel row in the plurality of pixel rows is generated using sensor data generated by the virtual reality device after the first time.

7. The method according to claim 6, wherein: Before the one or more first pixel rows are replaced, the one or more first pixel rows remain unchanged.

8. One or more computer-readable non-transitory storage media containing software operable when executed to cause one or more processors to perform operations comprising: determining a first position in three-dimensional (3D) space, wherein the first position is based on first sensor data generated by a virtual reality device at a first time; determining first visibility of one or more objects defined in the 3D space by casting a ray based on the first orientation to test for intersection with the one or more objects; generating one or more first pixel rows based on the determined first visibilities of the one or more objects; displaying the one or more first pixel rows by the virtual reality device; determining a second position in the 3D space, wherein the second position is based on second sensor data generated by the virtual reality device at a second time after the first time; determining a second visibility of the one or more objects defined in the 3D space by casting a ray based on the second orientation to test for intersection with the one or more objects; generating one or more second pixel rows based on the determined second visibility of the one or more objects; and displaying the one or more second pixel rows by the virtual reality device; The one or more second pixel rows are displayed simultaneously with the one or more first pixel rows.

9. The medium according to claim 8, wherein The duration for generating the one or more first pixel rows is significantly shorter than the duration required for generating the entire image.

10. The medium according to claim 8, wherein The display of the virtual reality device has a plurality of pixel rows, wherein the one or more first pixel rows and the one or more second pixel rows are subsets of the plurality of pixel rows.

11. The medium according to claim 10, wherein The one or more first pixel rows and the one or more second pixel rows have an equal number of rows.

12. The medium according to claim 11, wherein The equal number of rows is specified by an application program through an application programming interface (API).

13. The medium according to claim 10, wherein The software, when executed, is further operable to cause the one or more processors to perform operations including: The one or more first pixel rows are replaced after each other pixel row in the plurality of pixel rows is generated using sensor data generated by the virtual reality device after the first time.

14. The medium according to claim 13, wherein Before the one or more first pixel rows are replaced, the one or more first pixel rows remain unchanged.

15. A system comprising: one or more processors and one or more computer-readable non-transitory storage media coupled to the one or more processors, the one or more computer-readable non-transitory storage media comprising instructions operable when executed by the one or more processors to cause the system to perform operations comprising: determining a first position in three-dimensional (3D) space, wherein the first position is based on first sensor data generated by a virtual reality device at a first time; determining first visibility of one or more objects defined in the 3D space by casting a ray based on the first orientation to test for intersection with the one or more objects; generating one or more first pixel rows based on the determined first visibilities of the one or more objects; displaying the one or more first pixel rows by the virtual reality device; determining a second position in the 3D space, wherein the second position is based on second sensor data generated by the virtual reality device at a second time after the first time; determining a second visibility of the one or more objects defined in the 3D space by casting a ray based on the second orientation to test for intersection with the one or more objects; generating one or more second pixel rows based on the determined second visibility of the one or more objects; and displaying the one or more second pixel rows by the virtual reality device; The one or more second pixel rows are displayed simultaneously with the one or more first pixel rows.

16. The system of claim 15, wherein: The duration for generating the one or more first pixel rows is significantly shorter than the duration required for generating the entire image.

17. The system of claim 15, wherein: The display of the virtual reality device has a plurality of pixel rows, wherein the one or more first pixel rows and the one or more second pixel rows are subsets of the plurality of pixel rows.

18. The system of claim 17, wherein: The one or more first pixel rows and the one or more second pixel rows have an equal number of rows.

19. The system of claim 18, wherein: The equal number of rows is specified by an application program through an application programming interface (API).

20. The system of claim 17, wherein: When executing the instructions, the processor is further operable to perform operations including: The one or more first pixel rows are replaced after each other pixel row in the plurality of pixel rows is generated using sensor data generated by the virtual reality device after the first time.

Images