System and method for rendering optical distortion effect and non-volatile storage medium

Abstract

The present disclosure relates to systems and methods for rendering optical distortion effects and non-volatile storage media. In one embodiment, a computing system may receive a focal plane map that may be specified by an application. The system may determine an orientation in a 3D space based on sensor data generated by a virtual reality device. The system may generate a first coordinate in the 3D space based on the determined orientation and generate a second coordinate using the first coordinate and the focal plane map. Each coordinate in the first coordinate is associated with a coordinate in the second coordinate. For each coordinate in the first coordinate, the system may determine the visibility of the one or more objects by testing the intersection with one or more objects defined in the 3D space by projecting a ray from the first coordinate through the associated second coordinate. The system may generate an image of the one or more objects based on the determined visibility of the one or more objects.

Description

Technical Field

[0001] The present disclosure relates generally to computer graphics, and more particularly, to graphics rendering methodologies and optimizations for generating artificial realities such as virtual reality and augmented reality. Background Art

[0002] Generally, computer graphics are visual scenes created using computers. Three-dimensional (3D) computer graphics provide users with a view of 3D objects from a specific viewpoint. Each object in a 3D scene (e.g., a teapot, a house, a person, etc.) can be defined in a 3D modeling space using basic geometric bodies. For example, a cylindrical object can be modeled using a cylindrical tube and a top and bottom circular cover. The cylindrical tube and the circular cover can each be represented as a network or grid of smaller polygons (e.g., triangles). Further, each polygon can be stored based on the coordinates of its corresponding vertices in the 3D modeling space.

[0003] Although 3D objects in computer graphics can be modeled in three dimensions, they are usually presented to viewers through rectangular two-dimensional (2D) displays such as computer or television monitors. Due to the limitations of the human visual perception system, humans expect to perceive the world from roughly the same angle at any time. In other words, humans expect certain parts of 3D objects to be visible and other parts to be hidden from view. Thus, for each 3D scene, a computer graphics system may only need to render the part of the scene that is visible to the user, rather than the rest. This allows the system to significantly reduce the amount of computation required.

[0004] Ray casting is a technique used to determine the visibility of objects in a 3D scene. Typically, virtual light is uniformly projected from a virtual pinhole camera into the 3D world (e.g., based on the portion of the 3D object where the light is incident) by each pixel in the virtual rectangular screen to determine what is visible. However, for conventional rectangular display technologies with limited field of view (e.g., computer monitors and phone displays), it is reasonable to assume a uniform light distribution when calculating the primary visibility from a virtual pinhole camera. However, this assumption does not hold true for non-pinhole virtual cameras that more accurately represent real light sensors. Moreover, current VR visual optics (e.g., integrated in a head-mounted display) provide a curved, non-uniform visual surface rather than a conventional rectangular display. Therefore, in artificial reality, conventional rendering techniques designed and optimized based on the above-mentioned assumptions are computationally inefficient, produce suboptimal rendering, and lack flexibility in rendering scenarios. Summary of the invention

[0005] Specific embodiments described herein relate to a primary visibility algorithm that provides real-time performance and a feature set well suited for rendering artificial realities such as virtual reality and augmented reality. When solving the visibility problem, instead of casting each ray uniformly for each pixel, specific embodiments use a bounding volume hierarchy and a secondary foveated culling / entry point search algorithm to accelerate and optimize the traversal of coherent primary visibility rays. Specific embodiments utilize adaptive multi-sample anti-aliasing for ray casting to significantly reduce memory bandwidth.

[0006] Specific embodiments further provide flexibility and rendering optimizations that enable the rendering engine to naturally generate various graphical features while maintaining real-time performance. In the case of artificial reality, such graphical features are particularly needed: such as lens distortion, sub-pixel rendering, very wide field of view, porosity and random depth of field blur. The embodiments provide support for animation and physically based shadows and lights to improve the realism of the rendered scene. In contrast, conventional rasterization pipelines designed for conventional displays (e.g., rectangular monitors or televisions with uniform pixel grids) are typically implemented in hardware, and conventional rasterization pipelines require multiple passes and / or post-processing to approach these features. Moreover, conventional ray tracers that focus primarily on Monte Carlo path tracing cannot achieve the real-time performance of current VR displays (e.g., requiring a resolution of 1080x1200x2 and a refresh rate of 90Hz). Therefore, the embodiments described herein are specifically suitable for rendering artificial reality and provide an accurate, viable alternative to conventional rasterization techniques.

[0007] Embodiments of the present invention may include an artificial reality system or be implemented in conjunction with an artificial reality system. Artificial reality is a form of reality that is adjusted in some way before being presented to a user, and may include, for example, virtual reality (VR), augmented reality (AR), mixed reality (MR), mixed reality, or some combination and / or derivative thereof. Artificial reality content may include fully generated content or generated content combined with captured content (e.g., photos of the real world). Artificial reality content may include video, audio, tactile feedback, or some combination thereof, and any content presented through a single channel or multiple channels (such as, for example, stereoscopic video that produces a three-dimensional effect for the audience). In addition, in some embodiments, artificial reality may be associated with, for example, an application, product, accessory, service, or some combination thereof used when creating artificial reality content and / or used in artificial reality (e.g., performing an activity). An artificial reality system that provides artificial reality content can be implemented through various platforms, including a head-mounted display (HMD) connected to a host computer system, an independent HMD, a mobile device or computing system, or any other hardware platform capable of providing artificial reality content to one or more viewers.

[0008] The embodiments disclosed herein are only examples, and the scope of the present disclosure is not limited thereto. Specific embodiments may include all, some, or none of the components, elements, features, functions, operations, or steps in the embodiments disclosed above. Specifically, embodiments according to the present invention are disclosed in the attached claims for methods, storage media, systems, and computer program products, wherein any feature mentioned in one claim category (e.g., method) may also be protected in another claim category (e.g., system). The dependencies and references of the attached claims are selected only for formal reasons. However, any subject matter (specifically, multiple dependencies) generated by specifically citing any previous claim can also be claimed, and therefore, any combination of claims and their features can be disclosed and claimed regardless of the dependencies selected in the attached claims. The subject matter claimed for protection includes not only a combination of the features set forth in the attached claims, but also any other combination of the features in the claims, wherein each feature mentioned in the claims can be combined with any other feature or combination of other features in the claims. Further, any embodiment and feature described or depicted herein can be claimed in the independent claim and / or in combination with any embodiment or feature described or depicted herein or in combination with any feature in the appended claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] Figure 1 An embodiment of a bounding volume hierarchy tree data structure is shown.

[0010] Figure 2 An exemplary three-level hierarchy for defining the locations of projected rays / beams is shown.

[0011] Figure 3 An embodiment of rays and sub-sampled rays associated with a footprint is shown.

[0012] Figure 4 An embodiment of projecting a beam through a patch is shown.

[0013] Figure 5 An embodiment is shown in which a beam is projected through a block.

[0014] FIG. 6A to FIG. 6C An embodiment of a method for determining visibility is shown.

[0015] Figure 7 An example of a focal plane diagram is shown.

[0016] Figure 8 An example of a focal plane diagram and camera parameters is shown.

[0017] Fig. 9A method for naturally generating images by exploiting the optical distortion of a VR device is shown.

[0018] Fig.10 An embodiment of an importance map is shown.

[0019] Fig.11 An exemplary method of generating an image based on variable multi-sampling anti-aliasing is shown.

[0020] Fig.12 An embodiment is shown comparing a graphics generation timeline without ray tracing to a timeline using ray tracing.

[0021] Fig.13 An exemplary method of generating video frames for a VR display using beam tracking is shown.

[0022] Fig.14 An exemplary computer system is shown. DETAILED DESCRIPTION

[0023] A fundamental problem in computer graphics is to determine object visibility. Currently, the two most commonly used solutions are ray tracing, that is, simulating the transmission of light and dominating industries such as movies and computer-aided design (CAD) that value accuracy rather than speed. Due to the intensive computational requirements of ray tracing, it is traditionally not suitable for applications that require real-time or near-real-time rendering. Another solution for determining visibility is z-buffering, that is, checking each 3D object in the scene and updating a buffer, which tracks the current closest object for each pixel in a virtual rectangular screen. Typically, z-buffering is implemented by current graphics hardware and z-buffering lacks the flexibility to handle rendering tasks derived from the above-mentioned assumptions (e.g., pinhole cameras and / or rectangular screens with uniform pixel distribution). The specific implementation described herein provides such a visibility algorithm, that is, having performance characteristics close to z-buffering, but with the additional flexibility of being able to render a variety of virtual effects for artificial reality.

[0024] To provide further context, conventional z-buffering is commonly used to solve real-time primary visibility problems, largely due to its applicability to uniform primary visibility problems (e.g., for conventional rectangular screens) and the availability and proliferation of inexpensive, dedicated hardware implementations. Z-buffering algorithms use a z-buffer, a uniform grid data structure that stores the current closest incident depth for each sample / pixel. Most implementations of z-buffering assume that the samples / pixels are arranged in a uniform grid to accurately match the organization of the data structure. The uniform nature of the grid structure (combined with the uniform distribution of samples mapped into this grid) provides a very efficient algorithm for determining which samples overlap with polygons / triangles. The process of mapping the spatial extent of an object into a grid is called rasterization.

[0025] The uniform nature of the grid used in the z-buffer algorithm results in high efficiency but makes the algorithm inflexible. For almost all intuitive display technologies such as TVs, monitors, or cell phones, the assumed uniform sampling distribution is reasonable when computing the primary visibility of a virtual pinhole camera. However, for non-pinhole virtual cameras, secondary effects such as shadows, and especially for modern virtual reality devices, these assumptions do not hold due to the distortion imposed by the viewing optics of the head-mounted display, and currently, these assumptions should be made based on actual conditions.

[0026] Algorithms such as irregular z-buffers still use a uniform grid, but allow the number and arrangement of samples within each grid cell to be flexible. Irregular z-buffers suffer from load balancing issues related to conflicts between non-uniform sample distributions in the uniform data structure, making them significantly more expensive than traditional z-buffers. Further, having a uniform data structure means that the algorithm only supports a limited field of view and does not support depth of field rendering.

[0027] In contrast to z-buffering, ray tracing algorithms take a more general approach to determining visibility by supporting arbitrary point-to-point or ray queries. The ability to efficiently model physically based light transport and naturally construct effects has led to its becoming the dominant rendering algorithm for rendering movie scenes. However, the flexibility provided by ray tracing comes at a significant cost in execution, hindering its popularity in consumer real-time applications such as VR / AR.

[0028] The specific embodiments described herein overcome the shortcomings of existing rendering techniques and achieve ray rates of over 10 billion rays per second for non-trivial scenarios on modern computers, naturally supporting the computer graphics effects required for artificial reality.

[0029] Specific embodiments solve the visibility problem in computer graphics. In specific embodiments, a rendering system may use a ray caster that uses a three-level (or multi-level) entry point search algorithm to determine visibility. At an advanced level, the system may adopt a hierarchical scheme, in which, first, a larger beam (e.g., a coherent beam of rays) is projected to determine collisions within a larger range. Based on the incidence / loss of the beam, more granular beams or rays may be projected until the visibility problem is solved. It should be noted that even though the specific embodiments provided herein describe beams as representing coherent primary ray beams, however, the present disclosure contemplates the use of beams to represent any type of ray (e.g., primary ray, specular ray, shadow ray, etc.), that is, the embodiments described herein may use their coherent structure to achieve computational efficiency. In specific embodiments, the system may be implemented in different ways, and beam traversal occurs in a central processing unit (CPU) and ray triangle intersection and shading occurs in a graphics processing unit (GPU). In other embodiments, each computational task may be performed by the same type of processing unit.

[0030] To improve performance, embodiments may use acceleration structures to organize scene geometry. These structures may be based on spatial partitioning (grids, kd or k-dimensional trees, binary spatial partitioning or BSP trees, octrees) or object partitioning (bounding volume hierarchies or BVHs). By organizing geometry into spatial regions or enclosing them in closed volumes, these structures allow the system to avoid testing rays for objects if the rays do not enter the volume that encloses the object.

[0031] In a specific embodiment, an axis-aligned bounding volume hierarchy is a hierarchical tree data structure that stores scene geometry (usually triangles) at the leaves of the tree and axisymmetric bounding boxes at each node. The bounding box associated with each node can appropriately enclose all geometry associated with the node's subtree. In a specific embodiment, rays (or other visibility queries such as beams) can be recursively traversed through the tree from the root and the rays tested against the bounding volumes of the node's children. Recursive traversal of a node's children can only occur if they intersect, so rays / beams can avoid traversing portions of the tree that were missed by the ray / beam of its parent node.

[0032] Figure 1An embodiment of a BVH tree data structure 100 is shown. Each node (e.g., 110, 120-128, 130-136) in the tree 100 can be associated with a bounding volume in which a 3D modeling space of an object is defined. The tree 100 can have a root node 110, which is associated with a large bounding volume that surrounds and child nodes 120-128, 130-136 are associated with. Node 120 can be associated with a bounding volume that contains the bounding volume of its child nodes 121-128, but not the bounding volume of nodes 130-136. Node 121 can be associated with a bounding volume that contains the bounding volume of its child nodes 122-125, but not the bounding volume of any other node. Node 122 can be associated with a bounding volume that contains a geometric form (e.g., a triangle), but not any other bounding volume.

[0033] In specific embodiments, the system may use a four-way axisymmetric BVH as a geometry acceleration structure. In specific embodiments, a single combined BVH may be used for all scene geometries. In other embodiments, the system may use a multi-level approach to allow instantiation and more efficient animation by allowing more granular BVH reconstruction and reconfiguration. Basic animation is supported via per-frame global BVH reconfiguration.

[0034] In a specific embodiment, the BVH can be arranged in memory in a depth-first pre-order, and triangles are stored in a continuous array in the order performed by the depth-first traversal of the BVH. In addition, any node with mixed leaves and internal children can store the leaf children first. With these assumptions, reverse iteration through the list of BVH nodes can ensure that the node's children can always be accessed before the reverse iteration and that all triangles are accessed in a linear reverse order. These assumptions can support a linear, non-recursive BVH reconfiguration algorithm and improve cache locality during reconfiguration, traversal, and intersection.

[0035] A specific implementation for calculating primary visibility may use a BVH to perform a visibility test. As previously described, it is possible to determine whether an object (or part thereof) is visible from a specific viewpoint by testing whether the object (or part thereof) intersects with a ray. However, for each pixel, especially when the area to be covered is large, calculating multiple rays emitted from each pixel is time-consuming and resource-intensive. For example, if 32 sampling rays are used for each pixel, 32 intersection tests need to be performed on each pixel and a ray buffer of sufficient size needs to be allocated to store the results. Because most rays do not intersect with anything, it is particularly wasteful to emit so many rays in scenarios with fewer objects.

[0036] Instead of firing rays, embodiments may use beams to perform intersection testing. In embodiments, the system may perform hierarchical intersection testing using the following methods in order, namely, (1) a larger concave beam projected from a relatively large "block" (beam footprint described in more detail below); (2) a smaller concave beam projected from a "patch" (also beam footprint described in further detail below); and (3) programmatically generating sub-sampled rays (interchangeably referred to herein as "sub-rays"). In embodiments, as long as the larger concave beam is used to find intersection, no intersection testing needs to be performed on the sub-beams or rays, thereby avoiding unnecessary calculations. In embodiments, pixel shading may be based on the intersection results of a single sub-sampled ray, rather than the results of all 32 sub-sampled rays. To further optimize performance, when performing intersection testing, embodiments may programmatically generate (possibly pseudo-randomly) sub-sampled rays on the fly, rather than retrieving pre-defined sub-sampled ray positions from memory. Procedural ray generation has the benefit of not requiring reads from memory, thereby saving time and bandwidth.

[0037] Figure 2 An exemplary three-level hierarchy for defining the locations of projected rays / beams is shown. In particular embodiments, ray footprints 210 (which may be considered the first level of the hierarchy) may correspond to pixel footprints and be defined by footprint centers 240 and differences 250 (e.g., each difference 250 may specify a distance of a footprint border from the center 240 of footprint 210). In particular embodiments, sub-sampling locations 260a-e are assigned to footprints 210 for anti-aliasing. Figure 2 In the illustrated embodiment, each coverage area 210 includes five sub-sampling positions 260a-e. Although the specific embodiment shows five sub-sampling positions 260a-e per coverage area 210, any other number of sub-sampling positions (e.g., 32 sub-sampling positions per coverage area 210) may be implemented and / or defined by the application.

[0038] Figure 3 An example of a ray and a sub-sampled ray (or sub-ray) associated with a footprint is shown. In a specific embodiment, ray 310 may be associated with footprint 210, and ray 310 is projected through center 240 of footprint 210. Figure 2As shown in , a footprint 210 can be defined by two vectors 250 (also referred to as differences) that are perpendicular to each other and a ray direction. The breadth of the footprint 210 can be defined by the length of these vectors 250. In a particular embodiment, sub-sampled rays (or sub-rays) can be procedurally generated within the footprint 210 by first transforming a low-discrepancy point set in, for example, a unit square of the footprint 210 using a coordinate frame defined by the ray 310 direction and the footprint vector 250. As shown by the hollow dots, an embodiment of the transformed points is represented by sub-sampled positions 260a-e. Then, the ray direction can be added to the transformed point (e.g., the sub-sampled position 260a-e), and the sub-sampled rays 360a-e can be defined as rays cast through the origin 301 of the original ray 310 and the newly transformed points 260a-e, respectively. For example, sub-sampled rays can be used to perform multi-sampling anti-aliasing. In a particular embodiment, for depth of field rays, a separate low-discrepancy point set can also be used to select the ray origin (without translating along the ray direction).

[0039] In particular embodiments, shading may be performed once per pixel in each triangle, as in regular multi-sample anti-aliasing (MSAA), which saves a large number of shading calculations. In particular embodiments, shading may be performed per sample to obtain full super-sample anti-aliasing (SSAA). Because the sub-rays are generated procedurally rather than predefined (and stored in memory), ray memory bandwidth may be reduced by the anti-aliasing factor when compared to rendering naturally at a higher resolution.

[0040] In a specific embodiment, the main rays (e.g., 310) are assigned to the footprints (e.g., 210) for anti-aliasing and then aggregated into the secondary and tertiary levels, with the surrounding beams on the four sides having different granularities. Each beam position in the finer secondary level can be called a patch (e.g., in Figure 2 In the embodiment of the patch region, the patch region is labeled as 220. Each patch region 220 may include a predetermined number of pixel footprints 210. Figure 2 The embodiment shown in shows individual patches 220 having a 2x2 pixel footprint 210, however, any other arrangement of pixel footprints 210 may also be implemented (e.g., each patch 220 may include a 16x8 or 128 pixel footprint 210). Thus, the patches 220 may be grouped into a coarse set of blocks 230, i.e., the term used herein refers to a three-level hierarchy. Each block 230 may contain a predetermined number of patches 220. Although Figure 2The embodiment shown in shows that each block 230 contains 2x2 tiles 220, however, any other arrangement of tiles 220 may also be implemented (e.g., each block 230 may include 8x8 or 64 tiles). Thus, in an embodiment where each block 230 contains 8x8 tiles 220 and each tile 220 contains 16x8 pixel footprints 210, each block 230 may represent 8192 pixel footprints 210. The number of rays represented by the beam stream in the block 230 can be calculated by multiplying the number of pixel footprints 210 in the block 230 by the multisampling rate (e.g., 5 sub-sampled rays per pixel footprint 210). Thus, if each block 230 represents 8192 pixel footprints 210, the block 230 represents 40960 sub-sampled rays. In a specific embodiment, for a specific hardware, a pixel to patch ratio of 128:1 and a patch to block ratio of 64:1 are limited based on coarse tuning selections, but for other types of hardware, other ratios may be better.

[0041] In particular embodiments, rather than casting rays for all visibility tests, beams may be cast from tiles and patches in a hierarchical manner to optimize visibility computations. Figure 4 An embodiment of a beam 420 projected from an origin 301 (e.g., a camera or viewpoint) into a 3D modeling space through a patch 220 (in the embodiment shown, the patch 220 includes a 2x2 pixel footprint 210). In this embodiment, a solid beam 420 resembles a concave flow from the patch 220. The volume of the beam 420 can be defined by vectors projected through the four corners of the patch 220. An object or triangle intersects the beam 420 if the object or triangle intersects any portion of the volume of the beam 420. Similarly, Figure 5 An embodiment of a beam 530 projected from an origin 301 into a 3D modeling space through a block 230 (in the embodiment shown, the block 230 contains 2x2 patches, each containing a 2x2 pixel footprint 210). In this embodiment, the solid beam 530 resembles a concave flow from the block 230. The volume of the beam 530 can be defined by vectors projected through the four corners of the block 230. An object or triangle intersects the beam 530 if the object / triangle intersects any portion of the volume of the beam 530.

[0042] A specific implementation regarding scene updating and triangle precomputation will now be described. In a specific implementation, before rendering begins, animations may be performed (e.g., 3D object models in the scene may change) and the BVH may be modified. In a specific implementation, skeletal animation may occur in the CPU, and linear blend skinning and BVH modification may be implemented in a series of CDUA kernels in the following exemplary stages, namely, (1) transforming vertices (performing linear blend skinning); (2) clarifying BVH node boundaries; (3) precomputing triangles (e.g., by (a) collecting vertices, (b) calculating edge equations (for Moller-Trumbore ray triangle intersections), and (c) calculating triangle boundaries and automatically updating corresponding leaf bounding boxes); and (4) modifying the BVH by propagating boundaries from leaf nodes upward through the internal node hierarchy. In a specific implementation, after performing the modification in the GPU, the BVH may be copied back to the CPU memory for the block and patch traversal stage. At this point, if necessary, the boundaries of the blocks and patches may be calculated and modified.

[0043] FIG. 6A to FIG. 6C An embodiment of a method 600 for determining visibility according to a particular embodiment is shown. The illustrated embodiment implements a three-level entry point search algorithm, but additional levels may be implemented in other embodiments (e.g., using a fourth beam footprint including a set of blocks, a fifth beam unit including a set of fourth beam units, etc.). In a particular embodiment, the three levels are conceptually divided into a block culling stage, a patch culling stage, and a ray sampling test stage.

[0044] Fig. 6A An embodiment of a block culling phase is shown. During this phase, the computing system may traverse the BVH hierarchy and use the beam stream of blocks (e.g., Figure 5 ) tests the intersection of the selected bounding boxes associated with the nodes in the BVH. In a specific embodiment, each beam is defined by a 128x64 pixel footprint. In a specific embodiment, the implementation of beam traversal uses an explicit stack AVX implementation. Because the block traversal is a culling / entry point search phase, rather than traversing all the way to the leaves as in a traditional optical tracer, the block traversal is only traversed until it reaches a specified stopping criterion (e.g., when 64 entry points have been found).

[0045] In a specific embodiment, the screen that generates the scene (e.g., a virtual screen in 3D space corresponding to the display screen used by the user) can be divided into a number n of blocks. For each block, the system can perform three levels of testing to determine what can be seen from the block. In a specific embodiment, the visibility test can be performed by projecting beams from the blocks. For ease of reference, the beams projected from the blocks are referred to as block beams here.

[0046] In a specific embodiment, the method may start from step 610, where a BVH root (eg, Figure 1 110) is used to initialize an explicit traversal stack (e.g., a data structure used to track which nodes of the BVH are to be tested for intersection). Bounding volumes can be defined in 3D space. In certain scenarios, bounding volumes can contain smaller bounding volumes. For example, referring to Figure 1 , each child node (e.g., 120-128 and 130-136) corresponds to a bounding volume within the bounding volume of the root 110. Objects defined in 3D space (e.g., primary geometric shapes such as triangles, larger objects defined by a collection of primary geometric shapes, etc.) can be contained within any number of bounding volumes. For example, an object contained by a bounding volume associated with node 125 is also contained within bounding volumes associated with nodes 123, 121, 120, and 110.

[0047] At step 612, the system may access bounding volumes based on the traversal stack to test for intersection with the block beam. As an example, initially, the system may perform an intersection test with the bounding volume associated with the root node 110, and in subsequent iterations, perform intersection tests on the child nodes of the root node 110 based on the contents of the traversal stack. In a specific embodiment, at various steps in the traversal process, the thickest box may be tested along the primary traversal axis in the traversal stack. This allows the system to more efficiently subdivide the nodes down into individual planar slices. Despite the overhead of sorting, however, an improvement of 5% to 10% in slice / block culling execution rate was observed.

[0048] At step 614, the system can simulate the projection of the beam defined by the block in 3D space to test for intersection with the selected bounding volume. Figure 5 As shown in , the volume 530 of the block beam can be tested against the bounding volume to determine the extent of the intersection, if any.

[0049] At step 616, the system may determine that the result of the intersection test is one of the following: (1) miss - meaning the beam completely misses the bounding volume; (2) fully contained - meaning the beam completely / completely contains the bounding volume; or (3) partially intersected - meaning the beam intersects the bounding volume, but the bounding volume is not completely contained within the beam. At step 618, if the system determines that the test result is a miss, the system may remove / discard the subtree of the current node from the candidates to be tested for intersection with the block beam. For example, referring again to Figure 1, node 132 may represent a miss, meaning that the bounding volume associated with node 132 does not intersect the block beam at all. Therefore, there is no need to further test smaller bounding volumes contained within that bounding volume (e.g., bounding volumes associated with nodes 133-136 in the subtree of node 132), thereby providing substantial computational savings. Alternatively, at step 620, if the system determines that the test result is fully contained, the system may accumulate that bounding volume as an entry point, and since it is also fully contained, there is no need to further traverse the subtree of the associated node. Reference Figure 1 , node 121 may represent a bounding volume that is completely contained within the block beam. Therefore, smaller bounding volumes contained within the bounding volume (i.e., bounding volumes associated with nodes 122-125 in the subtree of node 121) do not need to be tested against the block beam (but may be tested in a subsequent patch culling stage), thereby providing substantial computational savings. Alternatively, at step 622, if the system determines that the test result is partial containment (e.g., the bounding volume partially intersects the block beam), the system may add / insert the subtree associated with the bounding volume into the traversal stack to continue subdividing. Reference Figure 1 , node 110 may represent a bounding volume that partially intersects the block beam. Therefore, smaller bounding volumes contained within the bounding volume may be further tested against the block beam. For example, in response to a determination that the bounding volume of node 110 partially intersects the block beam, the system may insert the top nodes of the respective subtrees (e.g., nodes 120 and 130) into the traversal stack for further intersection testing against the block beam.

[0050] In particular embodiments, at step 624, the system may determine whether one or more termination conditions of the block culling phase are satisfied. If the termination conditions are not satisfied, the system may continue to perform intersection tests for bounding volumes associated with nodes stored in the traversal stack. For example, after determining that the bounding volume of the root node 110 partially intersects the block beam, the system may test in the next iteration whether a smaller sub-bounding volume associated with, for example, node 120 or 130 intersects the block beam. This process continues until a termination condition is satisfied. For example, traversal may continue until the traversal stack is empty. In this way, the system may sort the accumulated entry points (or fully contained bounding volumes) in near depth order and pass them to the patch culling phase for further processing. Another termination condition may be when the sum of the size of the traversal stack and the size of the list of fully contained bounding volumes equals a predetermined value such as 32, 64, 128, etc. In particular embodiments, the traversal stack and the list may be merged, sorted in near depth order, and passed to the patch culling phase. Thus, no more than a predetermined number of entry points are passed from the block culling stage to the patch culling stage.

[0051] In a specific embodiment, during the traversal process, the separation between the bounding volume and the block beam is determined using the separating axis theorem. When classifying entry points before switching to patch culling, the near plane along the main axis of the beam can be used as a key value.

[0052] The patch culling phase begins at the end of the block culling phase. In a specific embodiment, 64 patch culling phases are used (e.g., corresponding to 8x8 or 64 patches in a block, depending on the specific embodiment) to further test each entry point identified during the block culling phase. In a specific embodiment, patch culling can be implemented in an explicit stack AVX traversal, just like block culling. However, instead of starting by initializing the traversal stack with the root node in the BVH, the traversal stack can be initialized by copying the output of the associated block culling. In this way, patch culling avoids repeating a large number of traversals performed during block culling. In a specific embodiment, as with block culling, beam / box testing has similar potential results, but traversals can continue until the traversal stack is empty. Once all triangles are collected, they are copied to the GPU by CUDA for sampling testing. In a specific embodiment, in high-depth and complex scenes, it may be desirable to collect and potentially test a very large number of triangles, despite the fact that they may be blocked by closer geometry. Short-circuiting patch traversal would require interleaving patch culling and sampling tests, which meant migrating patch culling to a CUDA implementation.

[0053] Figure 6B An embodiment of a patch culling stage that processes the results from the block culling stage is shown. As previously described, the block culling stage can generate a list of entry points by projecting a block beam and testing for intersection with a bounding volume. In subsequent patch culling stage processes, the resulting entry points can be used as starting points. Typically, the patch culling stage can use patch beams contained within a block beam for intersection testing. As previously described, each second beam is defined by a patch footprint that is smaller than the block footprint.

[0054] The patch culling phase for processing the results of the block culling phase may begin at step 630, where the system may iteratively select entry points from the list generated during the block culling phase and perform patch culling. It is known (associated with a node or bounding volume in the BVH) that the entry point intersects the block beam. In the patch culling phase, the system attempts to determine which patches in the block intersect the bounding volume or its descendants at a finer granularity. Thus, given the selected entry point, the system may iteratively project patch beams contained within the block beam for intersection testing.

[0055] For a given entry point, at step 632, the system can iteratively select patches in the block to perform intersection testing. In particular embodiments, before testing an entry point for a specific patch beam, the system can initialize the traversal stack to the bounding volume associated with the entry point. This provides an efficiency gain because the patch culling stage does not need to start from the root of the BVH (this work is already done during the block culling stage). Figure 1 As an example, the system may initialize the traversal stack using node 121, which has been determined to be a suitable entry point during the block culling phase.

[0056] At step 634, the system can access bounding volumes based on the traversal stack to test for intersection with the block beam. As an example, initially, the system can perform an intersection test with the bounding volume associated with node 121 (node ​​121 is considered a suitable entry point during the block culling phase), and in subsequent iterations, perform intersection tests on its child nodes based on the contents of the traversal stack. In specific embodiments, at various steps in the traversal process, the thickest box along the dominant traversal axis in the traversal stack can be tested.

[0057] At step 636, the system can simulate a beam defined by the patch projected into 3D space to test for intersection with the bounding volume. Figure 4 As shown in , the volume 420 of the patch beam may be tested against the bounding volume to determine the extent of the intersection, if any.

[0058] At step 638, the system may determine that the result of the intersection test is one of: (1) a miss - meaning that the beam completely misses the bounding volume; (2) fully contained - meaning that the beam completely / completely contains the bounding volume; or (3) a partial intersection - meaning that the beam intersects the bounding volume, but the bounding volume is not completely contained within the beam. At step 640, if the system determines that the test result is a miss, the system may remove / discard the subtree of the current node from the candidates to be tested for intersection with the patch beam. At step 642, alternatively, if the system determines that the test result is fully contained, the system may accumulate triangles / polygons into the bounding volume to be tested in subsequent stages. Since it is also fully contained, no further traversal of the associated node's subtree is required. In other words, any additional bounding volumes that are contained within the current bounding volume may be removed from the candidates to be tested for intersection with the patch beam. Alternatively, at step 644, if the system determines that the test result is partial containment (e.g., the bounding volume partially intersects the patch beam), the system can add / insert the subtree associated with the bounding volume into the traversal stack to continue subdividing (e.g., when processing repeats at step 634).

[0059] As an example, the system can be Figure 1. In some scenarios, the system may determine that the bounding volume associated with node 121 and the projected patch beam partially intersect. Based on this determination, the system may insert the subtree of node 121 (e.g., 122 and 123) into the traversal stack. Thus, when step 634 is repeated, the system may test for intersection with node 122 by, for example, projecting a patch beam for the bounding volume associated with node 122. In some scenarios, the system may determine that the projected patch beam completely contains the bounding volume associated with node 122, and add the triangles / polygons in the bounding volume to the list for sampling.

[0060] In a specific embodiment, the traversal may continue until the traversal stack is empty. Thus, in step 646, the system may determine whether there are any nodes left in the traversal stack. If the node exists in the stack, the system may return to step 634 to test the node against the patch beam. If the node does not exist in the stack, in step 648, the system may determine whether there are additional patches in the block that have not yet been tested against the original entry point. In this way, the system may return to step 632 to perform intersection tests on the entry point against the untested patches. Otherwise, in step 650, the system may determine whether additional entry points from the block culling stage still need to be tested. In this way, the system may return to step 630. Otherwise, in a specific embodiment, the system may pass the collected triangles / polygons to the ray sampling test stage.

[0061] In a specific embodiment, after the patch culling stage, a light sampling test stage can be performed. In a specific embodiment, the light sampling test stage can be split into a patch-by-patch stage and a pixel-by-pixel stage. In a specific embodiment, a single CUDA kernel with a workgroup size of 128 can be used to complete these two stages. In the patch-by-patch part, threads and triangles can be mapped 1:1, and in the pixel-by-pixel stage, threads and pixels can be mapped 1:1. In a specific embodiment, alternatively, threads and sub-pixels can be mapped 1:1, in which case, this stage can be referred to as a sub-pixel stage. As used in this context, a sub-pixel is a single LED such as red, green, or blue and is different from the sub-sampling in the meaning of multi-sampling anti-aliasing. Thus, a sub-pixel can have multiple sub-sampling. The system can support both multi-sample anti-aliasing (MSAA) and super-sample anti-aliasing (SSAA), the difference being that in MSAA, shading is performed only once per pixel per triangle, and the result is shared across all subsamples of that pixel that impact the same triangle, and the difference being that in SSAA, shading is calculated separately per subsample. The advantage of MSAA is that the shading rate is potentially significantly reduced. The triangle data for a patch can be collected into a shared local cache of the GPU to facilitate access by all samples. The triangle cache can have 128 entries. In a specific embodiment, the per-patch stage and the per-pixel / sub-pixel stage can be alternated until all triangles of the patch are processed.

[0062] Figure 6C An embodiment of a sample test phase is shown to process triangle intersections identified after the patch culling phase. After the patch culling phase, the system may have identified a list of triangles that intersect the associated patch beam. During the sample test phase, the system attempts to further sample the triangles at a finer pixel level and may do so using sub-sampled rays (e.g., for anti-aliasing). In a specific embodiment, the above-described per-patch phase may be represented by steps 660 to 668, and the per-pixel / sub-pixel phase may be represented by the steps starting at step 670.

[0063] At step 660, before testing triangles for intersection with the patch beam, the system can perform initialization by, for example, specifying per-subsampled depth and index values. During the per-patch phase, at step 662, the system can collect triangle data into a shared memory cache. At step 664, the system can perform backface and near plane culling on the triangles. Similar to the classification described above, at step 666, the system can test patch corner rays against the triangles and classify intersections into fully contained, partially intersected, and missed. At step 668, the system can perform common origin intersection precomputation (if applicable).

[0064] In particular embodiments, once the per-tile stage is complete, each thread may associate itself with a pixel / sub-pixel and perform the following steps during the per-pixel / sub-pixel stage. The system may perform intersection tests using rays contained within the patch beam that intersect the triangle. In particular embodiments, this ray (including sub-sampled rays) may be generated procedurally.

[0065] In particular embodiments, at step 670, for each pixel / subpixel in the patch, the system may query the center of the footprint and the difference associated with the footprint of the ray. Figure 2 240) and the difference (e.g., Figure 2 The center and difference values ​​may define a ray footprint through which rays may be generated and cast into 3D space to test for intersection with an object (eg, a triangle).

[0066] In particular embodiments, the system may iteratively cast a ray associated with a ray footprint for each triangle. For example, at step 674, after the ray footprint has been determined, the system may retrieve the triangle from the cache. The triangle may be iteratively tested for each ray sample in the ray footprint. For example, at step 676, for each ray sample, the system may query the ray sample via a lookup table (e.g., Figure 2 In step 678, the system can calculate the sub-sampling bias within the pixel footprint for the rays defined by the sub-sampling (e.g., Figure 3 Once a ray is determined to intersect a triangle, the system can store information associated with the intersection. For example, in the case of a successful intersection, the system can update the subsampled depth and triangle index. After a single ray of a pixel / subpixel has been tested for the triangle, the system can determine in step 682 whether additional samples should be tested (e.g., based on the pixel / subpixel of the triangle). Figure 2, with five subsamples per pixel). Thus, the system can repeat step 676 to generate another ray and test for intersection. As determined during the patch culling stage, if the triangle has been tested for all subsamples, at step 684, the system can determine whether there are more triangles that intersect the patch beam. Thus, the system can repeat step 674 to sample another triangle using rays associated with the current ray footprint. If there are no more triangles, the system can determine whether there are more pixels / subpixels within the patch that should be tested for triangles. Thus, the system can repeat step 670 to perform intersection tests for triangles that intersect the patch using rays from another pixel / subpixel.

[0067] Once all triangles are processed, the visibility of the patch is fully resolved, and for each subsample, the buffer contains the closest incidence by subsample depth and triangle index. In a specific embodiment, at this point, the subsample data can be compressed and emitted to a "gbuffer" in preparation for shading. In this case, the "gbuffer" can consist of only visibility information: pairs of triangle index and subsample mask, which is sufficient to recalculate the center of gravity and obtain vertex attributes in the shading stage. The "gbuffer" can be allocated large enough to hold a fixed number of entries (e.g., 32) to handle the case where each subsample incidence is a different triangle for each subpixel and is stored in the global memory of the GPU. The memory is addressed so that the first triangle for each pixel is adjacent in memory, followed by the second triangle, etc., so that, in fact, only a small prefix of the buffer is actually used. In a specific embodiment, compression can perform the following steps: (1) sort the subsamples by triangle index; and (2) iterate over the subsamples and emit, for example, the triangle index and multisample mask for each unique triangle index. Once the "gbuffer" is built, the sampling and testing phase is completed and the parsing / shading phase begins.

[0068] After calculating visibility during the sample test phase, the system can perform shading, accumulate MSAA or SSAA samples and calculate the final pixel / sub-pixel color into a buffer, which is presented as a rendered computer-generated scene including visible objects confined in 3D space. In a specific embodiment, each sample position is read from the output of the previous stage and the ray intersection is calculated for each "gbuffer" item at the center of the pixel. Therefore, shading can be performed using the centroid-interpolated attributes obtained during the intersection process, and the shading results can be accumulated per pixel. Once all "gbuffer" items have been processed per pixel, the system can perform film-style tone mapping and output the results to the final buffer for display. In the case of SSAA, rays can be generated, intersected, and shaded independently, rather than having a single weighted shading result per item.

[0069] In a specific embodiment, the parsing / shading stage may include the following steps. For example, the system may query the sample location (e.g., a linear buffer of samples generated from a previous stage). The system may then calculate the difference (e.g., analytically when in closed form, otherwise as a finite difference). The system may then convert the rays and differences into 3D world space in preparation for intersection. The system may clear the shading result accumulator to 0. Then, for each "gbuffer" item, the system may obtain the triangle data, perform ray-triangle intersection on the pixel center, and calculate the depth and barycentric coordinates, suppress the barycentric coordinates to the triangle boundary, interpolate the vertex attributes based on the barycentric coordinates, perform shading / lighting, and accumulate the shading results. The system may then scale the accumulated shading results by, for example, 1 / subsampling rate. The system may then perform tone mapping and output the results.

[0070] The specific implementation method can be repeated FIG. 6A to FIG. 6C One or more steps of the method, as applicable. Although the present disclosure describes and illustrates FIG. 6A to FIG. 6C The specific steps of the method occur in a specific order, however, the present disclosure contemplates FIG. 6A to FIG. 6C Furthermore, although the present disclosure describes and illustrates an exemplary method for performing primary visibility calculations, including FIG. 6A to FIG. 6C However, the present disclosure contemplates any suitable method for performing the primary visibility calculation, including any suitable steps (which may include FIG. 6A to FIG. 6C All, some steps, or none of the methods FIG. 6A to FIG. 6C Further, although the present disclosure describes and illustrates the steps of the method, FIG. 6A to FIG. 6C The specific components, devices, or systems of the specific arrangements of the methods in the present disclosure, however, contemplate the completion of FIG. 6A to FIG. 6CAny suitable combination of any suitable components, devices, or systems in any suitable arrangement of the methods described herein.

[0071] One advantage of this embodiment is that it can be implemented in a conventional graphics pipeline. At a high level, an application that needs to be rendered can issue instructions to a graphics driver, and the graphics driver can communicate with the associated GPU. Through the graphics application programming interface (API) of the driver, the application can specify how to render the scene. For example, the application can submit a geometric definition of an object in a 3D space representing the scene to be generated for it. In a specific embodiment, the application can also submit a ray buffer that defines the position and trajectory of the ray. In order to avoid the runtime cost of requiring a large amount of memory to store the ray buffer and the read / write buffer, the specific embodiment can further allow the user to specify the program definition used when the ray is generated programmatically at run time. As previously described, based on the information provided, the rendering system can perform visibility calculations using, for example, ray casting. Once the visibility of the geometry is determined, the system can begin shading and output the final scene.

[0072] In contrast to existing systems that use post-processing to simulate this effect, the flexibility provided by the embodiments described herein enables rendering systems to naturally implement a variety of rendering features related to virtual reality or augmented reality. Furthermore, existing graphics APIs focus on the specific case of a uniform grid of primary sampling points. Although current hardware rasterizers are highly tuned for this use case, rendering of AR / VR displays requires additional high-performance functionality that can be more naturally implemented by performing ray casting according to specific embodiments. Specifically, existing graphics APIs cannot handle the following cases: (1) performing sub-pixel rendering on different sub-pixel arrangements with known optical distortion; (2) changing multisampling and / or shading rates on the screen (e.g., concave rendering / situational awareness rebalancing); (3) depth sampling patterns with respect to depth approximation of zoom displays; and (4) beam racing.

[0073] The specific embodiments described herein support the above use cases based on the following features, which can be implemented as enhancements to the visibility determination portion of the graphics pipeline. For example, to support optical distortion rendering of different sub-pixel arrangements, the system can allow applications using the rendering engine to specify non-uniform grid sampling patterns and render non-uniform grid sampling patterns and use independent color channel sampling. To support variable multisampling / shading, the system can allow applications to specify "importance" measurements per pixel / tile / block. To support depth of field, the system can allow applications to specify non-uniform grid sampling patterns and non-point origin sampling patterns and render non-uniform grid sampling patterns and non-point origin sampling patterns. Each of these features, as well as beam racing, will be described in further detail.

[0074] At a high level, when any type of lens distortion is required, distortion can be applied to the rays (e.g., determining the direction of the rays) before calculating the bounding beams via principal component analysis. When sub-pixel rendering is enabled, separate grids can be considered for the red, green, and blue channels, and patching and blocking can be performed separately. When depth of field is present, the beams can be extended to accommodate the distribution of ray origins. When a concave ray distribution is used, a top-down partitioning algorithm can be used to generate ray bundles to construct patches containing no more than n pixels (e.g., 128) and blocks containing no more than m patches (e.g., 64). The system can support partially occupied patches and blocks. For most use cases, these boundaries can be calculated once at the beginning of time based on the lens parameters of the system. However, in cases where parameters change between frames, such as points of interest during concave rendering, they can be recalculated frame by frame. The bounding beams can surround the entire footprint of each pixel, rather than just its center, to support MSAA.

[0075] Specific embodiments enable a graphics system to support direct optical distortion rendering and sub-pixel rendering. One of the main differences between head-mounted displays and traditional displays is the use of viewing optics. In addition to allowing the user to focus on the display, the viewing optics also add various aberrations to the viewed display. In particular, head-mounted displays produce color-dependent pincushion distortion, resulting in color separation and non-uniform pixel spacing. This results in the user actually viewing three different displays, one for each color (e.g., red, green, and blue), and with three different distortion functions. Traditionally, these artifacts can be corrected during the post-processing image distortion stage. For example, conventional rendering systems do not support direct distortion rendering and will produce a conventional rectangular image. In order to properly view the image via the head-mounted display, the post-processing stage uses the rectangular image and builds a distorted image for the head-mounted viewing optics. The conventional multi-stage process is not only inefficient, but the synthesis effect is also suboptimal.

[0076] The specific embodiments described herein enable the renderer to accept / receive lens parameter information (e.g., lens aperture, curvature, etc.) and use this information to produce an inherently distorted image without the need for post-processing. Advantages of direct sub-pixel rendering include increased clarity. By considering the spatial layout of sub-pixels, the brightness of each sub-pixel can be more accurately defined rather than the entire pixel at the same time. In addition, direct sub-pixel rendering avoids the need for large pre-distorted rendering targets. For example, a specific conventional VR device has an eye-buffer resolution of 3024x1680, or an output display resolution of 1.96 times 2160x1200 pixels. Given that the image presented to the display is cropped by a barrel distortion mask, only about 85% of the 2160x1200 pixels are actually presented. Because of these large guard bands or over-rendering of the eye buffer, less than 50% of the total pixels need to be rendered. Thus, by directly rendering using the embodiments described herein, a lot of computation can be saved. By avoiding the distortion correction resampling pass (which introduces aliasing and blurring), direct subpixel rendering further improves image quality, especially notable for near-Nyquist details such as text. Finally, by removing the full-frame distortion correction pass, direct distortion rendering brings the opportunity to render in time.

[0077] To support optical distortion rendering for different sub-pixel arrangements, the system can allow applications to use a rendering engine to specify a non-uniform grid sampling pattern and render the non-uniform grid sampling pattern and use independent color channel sampling. Typically, optical distortion comes from the physical cause of a lens that causes light to bend in a distorted manner. In a conventional rendering system, it is assumed that the camera is a pinhole camera with uniform optics. In order to generate an image captured by such a pinhole camera, a conventional system projects light from the pinhole camera into 3D space through points on a flat, uniform virtual image plane. Therefore, conventionally, a uniform grid sampling pattern is sufficient. However, because real-world optics are imperfect and complex (for example, with multiple lens stacks), light does not actually enter and exit the optics in a straight line, and the degree of light bending depends on its entry / exit points and the curvature and density of the optics. As a result, images generated using a uniform grid pattern cannot accurately represent what people expect to perceive.

[0078] Specific embodiments of the rendering system allow an application (e.g., a VR or gaming application sends a request to the rendering system) to specify a focal plane map. The focal plane map can be defined as a curved surface in space with any desired curvature. Conceptually, for each pixel, the focal plane map defines the corresponding point in space that will be in focus when viewed through that pixel. In specific embodiments, the focal plane map can map screen coordinates to specific locations in 3D world or camera space. The focal plane map can act like a control surface that specifies how to expand screen pixels to 3D world space. In specific embodiments, the focal plane map can have the same 2D size as the output image, so there is a 1-to-1 mapping between the values ​​stored in the focal plane map and the pixels in the output image. Thus, when determining the visibility of a specific pixel, the system can query the corresponding value in the focal plane map to find the point in 3D space that the light should pass through. In specific embodiments, the plan view can be stored as a float3 or half3 buffer that maps screen space to 3D world space (or camera space). In specific embodiments, if the focal plane map is smaller than a pixel, the focal plane map can be defined as a Catmull-Rom surface, and interpolation can be used to determine the exact coordinates in 3D space to which the pixel is mapped. In specific embodiments, to handle fields of view greater than 180 degrees, the system can switch vertex / geometry / tessellation shaders to output world space positions. In specific embodiments, if the focal plane map is defined, vertex positions throughout the shader pipeline can be defined in world / camera space.

[0079] Figure 7 An embodiment of a focal plane map 750 displayed in 3D world space is shown. In specific embodiments, the focal plane map 750 can be defined arithmetically or algorithmically. In other embodiments, the focal plane map can be defined as a lookup table. In either case, given a coordinate in screen space, the focal plane map 750 can be used to determine the corresponding coordinate in 3D space. For example, Figure 7A conceptual representation of a virtual image sensor array 710 is included (and a corresponding screen may also be represented). Each illustrated grid in sensor 710, such as grids 711-713, may represent the sensor for corresponding pixels. Focus plane map 750 may define specific points 751, 752, and 753 in 3D space that correspond to representative points 711, 712, and 713 of their corresponding pixels (or sub-pixels), respectively. As previously described, each point, such as point 751, may conceptually represent a point in space that will be in focus when viewed from corresponding point 711. During ray casting / tracing, rays 721, 722, and 723 may be cast from points 711, 712, and 713 through points 751, 752, and 753, respectively, as specified by focus plane map 750. Object visibility may be determined using rays 721, 722, and 723. For example, an object may be visible from point 712, but not from points 711 and 713. Figure 7 740 in the focus plane map 750. In a specific embodiment, sub-sampled rays associated with each pixel can also be mapped to the focus plane map 750. For example, in addition to point 711, additional sub-sampled positions can be associated with the same pixel footprint. The projection points of the sub-sampled positions can be defined on the focus plane map 750. For example, the projection points of the sub-sampled positions can be defined within the same grid that defines point 751. During visibility testing, object intersection can be tested by projecting sub-sampled rays from the sub-sampled positions through the corresponding projection points of the sub-sampled positions on the focus plane map 750.

[0080] The focus plane map 750 provides the application / user with the flexibility to define any desired light direction to generate any desired resulting effect. For example, if the application wishes to simulate optical distortion, a focus plane map 750 can be defined that approximates the desired distortion caused by the optics. Similar to how light bends through optics, based on the focus plane map 750, individual rays can be cast in arbitrary directions (rather than uniformly cast through a conventional grid).

[0081] Wide field of view (FoV) rendering can also be supported using focal plane maps. One notable difference between head-mounted displays and traditional displays is that head-mounted displays can support a wider field of view. While typical desktop displays use a viewer FoV of about 30°, current VR headsets start at about 90° and can be as high as 210°. Because a uniform sampling grid is assumed in the conventional z-buffer algorithm, the number of samples required to render the image is as high as the tangent of the half-angle of the FoV (assuming a fixed minimum angular resolution). This results in a steep loss of efficiency towards a 180° FoV. For example, the difference between 140° and 160° is almost 4 times the number of pixels rendered. To alleviate this inefficiency, schemes have been studied to render multiple narrow FoV images and stitch them together. Although this improves the efficiency of the z-buffer algorithm, it incurs the cost of repeatedly processing the geometry of each additional view and increases potentially noticeable seam stitching artifacts. To address these shortcomings, the embodiments described herein employ ray projectors to support arbitrary light distributions to support wide FoV rendering. As another example, if an application wishes to render a scene with a wide field of view (e.g., greater than 90°, 120°, 180°, 210°, etc.), it may define a focal plane map that causes light rays to gradually bend outward as the ray origin deviates from the center of the screen (e.g., the minimum angle between two light rays, such as light rays cast from opposite sides of the scene, may be greater than 90°, 120°, 180°, 210°, etc.). For example, the focal plane map may define a concave hyperboloid relative to the camera. The camera may be positioned within the curvature surface, such as at the center of the curvature (e.g., to obtain a FoV of 180°), behind the center (e.g., to obtain a FoV of 120°), or in front of the center (e.g., to obtain a FoV of 210°). Because the focal plane map provides the application with the flexibility to specify how the light rays travel, a wide FoV may be naturally generated without resorting to stitching operations with the limitations and costs described above.

[0082] In addition, focal plane maps can be used to support defocus blur and depth of field blur, which are desired effects in VR / AR because they provide a certain degree of realism that humans expect. Defocus blur is a natural effect of all non-pinhole cameras and a common simulated effect in synthetically generated images. In virtual reality, depth of field is also an important effect in zoom displays, where synthetic blur is needed to render content out of the focal plane to match the expectations of the human visual system. However, conventional rendering systems typically assume that the camera is a pinhole camera, resulting in always sharp focus (in other words, with infinite depth of field). In order to simulate blur, conventional systems perform post-processing on the clearly rendered image. Conventional processing for simulating defocus or depth of field blur is not only inefficient, but also inaccurate in blur effects. This is because simulated blur is usually performed by averaging adjacent pixel colors, while actual defocus or depth of field blur is derived from the behavior of light and the physical properties of the lens.

[0083] Particular embodiments enable rendering systems to support some camera parameters that can be used to stochastically generate defocus or depth blur. For example, particular embodiments can generate real-time random blur using aperture and depth of field parameters representing a virtual system (e.g., via a focus plane map). A unique feature of the embodiments described herein is the use of the same machine for both defocus blur and MSAA to allow the system to shade at a rate close to the display resolution, despite the large number of additional rays. Shading differences can be calculated based on the width of the depth packet, which in addition to providing correct pre-filtering, also improves performance by requiring coarse MIP levels during texturing reads. In addition to efficiently supporting defocus blur, the embodiments described herein are able to do so accurately and synthetically. In contrast, image-space defocus blur algorithms (e.g., post-processing the output image by blurring adjacent pixels) cannot be naturally integrated with various effects such as concave (due to the requirement to synthesize density, rectilinear color and depth maps from concave images before calculating image-space depth of field) or wide field of view (due to non-uniformity, poor anisotropy of angular resolution between pixels in the image plane, the need to stitch multiple views simultaneously, or both).

[0084] To support defocus blur and / or depth of field blur, embodiments allow applications to specify and render non-uniform grid sampling patterns and non-point origin sampling patterns. As described elsewhere herein, embodiments may allow users to define a focus plane map. In addition, embodiments may allow applications to specify a thin lens model for modifying parameters of the ray origin. In embodiments, the radius of the lens or aperture of the virtual camera may be specified.

[0085] Figure 8An embodiment is shown of using focus plane map 850 and camera parameters (e.g., aperture, lens size, etc.) to naturally (rather than post-process) generate an image with defocus or depth of field blur. In specific embodiments, focus plane map 850 can be defined arithmetically or algorithmically. In other embodiments, the focus plane map can be defined as a lookup table. In either case, given a coordinate in screen space, the focus plane map 850 can be used to determine the corresponding coordinate in 3D space. For example, Figure 8 A conceptual representation of a virtual image sensor array 810 is included (which may also represent a screen of pixels). Each illustrated grid in the sensor 810 may represent a sensor of pixels. A focus plane map 850 may define specific points in 3D space that correspond to specific points on the screen / sensor 810. For example, according to the focus plane map 850, a point 855 in 3D space may be mapped to a representative point 815 of a sensor or pixel (or sub-pixel) in the sensor array or screen. As previously described, the point 855 may conceptually represent a point in space that is in focus when viewed from the corresponding point 815. As previously described, for each grid in the sensor, the sub-sampling position may similarly be mapped to a specific point on the focus plane map 850.

[0086] In addition to focal plane map 850, embodiments may also enable an application to define parameters of a virtual camera. Rather than using a pinhole camera represented by a point in space, embodiments may enable an application to define a more realistic virtual camera. For example, Figure 8 Parameters of a virtual camera with a grid 830 are represented. Because a camera has physical dimensions (rather than points in space), the grid 830 is used to represent individual sampling points (e.g., points 831, 832, 833) through which light can travel. For example, the grid 830 can represent uniform sampling points on the aperture or lens of the virtual camera. In a specific embodiment, the application can specify the size of the aperture / lens (e.g., by specifying the radius of the circular area of ​​the aperture / lens, the difference in the square area of ​​the aperture / lens, or the length and width of the rectangular area of ​​the aperture / lens). In turn, this size can be used to determine the size / scale of the grid 830. In a specific embodiment, the granularity of the grid 830 can also be controlled by the application. In this embodiment, although the grid 830 is used as a data structure representing a lens or an aperture of a lens, the present embodiment is not limited to the use of a grid. For example, the points in space through which the light is projected can be represented by a vector or a list of coordinates or can be programmatically defined (e.g., via a mathematical formula).

[0087] During the ray casting / tracing process, embodiments may iteratively determine the object visibility for each pixel. Figure 8In the illustrated embodiment, the system can determine object visibility of point 815. Because the color of a pixel associated with point 815 can depend on light incident from different points of the lens, particular embodiments can conceptually project a ray originating from point 815 through various points, such as points 831-833 defined by grid 830. In other words, these points 831-833 can be considered a sampling of points on the lens through which light can be incident on a virtual camera sensor represented by point 815. In particular embodiments, the system can project rays 821, 822, and 823 from points 831, 832, and 833, respectively, through point 855, as defined by focus plane map 850. In the illustrated embodiment, ray 821 misses object 840, ray 822 nearly misses object 840, and 823 intersects object 840. Because the intersection points (or lack of intersection points) of the rays are sufficiently different, the resulting color calculation for the pixel represented by point 815 is not purely a calculation of object 840, and therefore, in the final scene, object 840 appears blurry. If object 840 is instead positioned at or near point 855, the object will be within the depth of field and appear sharper. Similarly, visibility testing at a sub-sampling level can be performed. In this case, each sub-sampling position (e.g., a position associated with the pixel represented by point 810) can also be mapped to a point on the focus plane map 850 (e.g., a point in the grid represented by point 855). Visibility testing can be performed by projecting sub-sampled rays from the sub-sampled positions through each point defined by the grid 830 (e.g., through the aperture), and through the corresponding point on the focus plane map 850.

[0088] Fig. 9A method for naturally generating an optically distorted image of a VR device according to a specific embodiment is shown. In step 910, a computing system may receive a focal plane map specified by an application through an application programming interface (API). For example, a VR / AR application (e.g., a game) may send an instruction including a definition of a focal plane map to a graphics API. In a specific embodiment, the focal plane map may be defined as a coordinate grid about a plane representing an image sensor array. For example, each grid in the focal plane map may be mapped to or associated with a grid representing a corresponding pixel on an array sensor or a screen. For example, the upper left grid in the focal plane map may be associated with the upper left sensor / pixel, the upper right grid in the focal plane map may be associated with the upper right sensor / pixel, and so on. Each grid in the surface map may store coordinates in a 3D space, wherein the relative position of the surface to the sensor array or pixel may be defined (the 3D space may be different from the 3D world space when the scene is generated). In a specific embodiment, alternatively, the focal plane map may be specified by an algorithm. For example, a mathematical function or a rule-based algorithm may take a coordinate as input (e.g., a 2D coordinate representing a point in a sensor array or a pixel screen) and output a 3D coordinate.

[0089] At step 920, the system may determine an orientation in three-dimensional (3D) space, which may represent the orientation of a virtual camera or viewer in 3D space. In particular embodiments, the orientation may be specified by a VR application, which in turn may be based on sensor data generated by a virtual reality device. For example, a head mounted device may include a gyroscope or inertial measurement unit that tracks the real-time movement of a user and outputs sensor data representing or describing the movement. The VR application may use sensor data provided by such motion tracking sensors to determine the current orientation of the user and provide the orientation to a rendering engine to orient / reorient the virtual camera in 3D space.

[0090] At step 930, the system may generate a plurality of first coordinates in 3D space based on the determined orientation. For example, the plurality of first coordinates may represent pixels on an image sensor in a virtual camera or a corresponding virtual screen. In a specific embodiment, one or more transformation matrices or other transformation algorithms may be used to position and orient the virtual image sensor / screen in 3D space. Each point on the virtual image sensor / screen that the rendering system is to sample may be represented by a coordinate in 3D space.

[0091] At step 940, the system can generate a plurality of second coordinates in 3D space using the plurality of first coordinates and the focus plane map. For example, the second coordinates can be points in 3D space (e.g., points such as point 751, etc.) representing where the object will appear to be in focus. Figure 7Each of the plurality of first coordinates may be associated with one of the plurality of second coordinates. Figure 7 As shown in , according to focus plane map 750 , point 711 (defined by coordinates) is mapped to or associated with point 751 .

[0092] In a specific embodiment, the system can generate the second coordinate by performing a query operation using the focal plane map. For example, for each coordinate in the first coordinate (e.g., each camera sensor / pixel), the system can query the corresponding grid in the focal plane map to identify the coordinate in the focal plane map that is mapped to the first coordinate. In a specific embodiment, the coordinates in the focal plane map can be confined to a space different from the 3D world space when the scene is generated. Thus, in a specific embodiment, the system can use one or more transformation matrices or algorithms to transform the coordinates in the focal plane map to the 3D world space to generate, for example, point 751 in the 3D world space.

[0093] In certain embodiments, the system may alternatively generate the second coordinates programmatically. For example, in certain embodiments, the focal plane map may be specified by an algorithm. Thus, the system may generate the plurality of second coordinates by processing the plurality of first coordinates according to the algorithm. For example, the algorithm may be mathematically defined so that, given an input coordinate (e.g., such as Figure 7 The algorithm will output the corresponding coordinates. In specific embodiments, the output coordinates may be in the same space as the space in which the input coordinates are defined. For example, a specific embodiment of the algorithm may be configured to receive input coordinates defined in a non-world space. Thus, the output of the algorithm may be the relative position of the focus and the input coordinates in the non-world space. In this way, the system may be transformed (e.g., via a transformation matrix) into a 3D world space based on the current orientation of the virtual camera. In other embodiments, the algorithm may use coordinates defined in a 3D world space (e.g., Figure 7 ) as input and outputs coordinates in 3D world space (e.g., Figure 7 In this case, no subsequent transformation is required.

[0094] At step 950, for each of the plurality of first coordinates, the system may determine visibility of one or more objects defined in the 3D space by casting a ray from the first coordinate through the associated second coordinate to test for intersection with the one or more objects. Figure 7 In , ray 721 cast from point 711 passes through the associated point 751 defined by the focal plane map. The cast ray can be used to test intersection (or visibility) with any object defined in the 3D world space. For example, Figure 7Object 740 is shown to be invisible to the sensor / pixel at points 711 and 713, but is visible to the sensor / pixel at point 712. In specific embodiments, as described elsewhere herein, the ray may be a subsampled ray rather than a primary ray associated with a pixel. In specific embodiments, what is projected may also be a beam (e.g., a beam projected from a patch or block) instead. In specific embodiments, the coverage of the beam in 3D world space may be calculated using the associated primary ray and a focal plane map. For example, the trajectory of each primary ray may be determined using the focal plane map. To determine the coverage of the beam, the system may use the above-mentioned difference of the ray to calculate its ray footprint, and conversely, when the beam is projected through the 3D world space, the ray footprint associated with the beam may be uniformly used to define the coverage of the beam.

[0095] At step 960, the system can generate an image of one or more objects in the 3D world space based on the determined visibility of the one or more objects. As previously described, the system can determine object visibility from the standpoint of the virtual camera using the projected rays. Object visibility can be tracked by storing the depth of each intersecting object (e.g., a triangle or other primitive) and its index. Using this visibility information, the system can proceed with shading to determine what color to assign to each pixel. Then, an image is generated using the collective pixels, which will be the scene of the 3D world space viewed from the vantage point of the virtual camera. The image can include any object that is confined in the 3D world space and visible to the virtual camera.

[0096] The specific implementation method can be repeated Fig. 9 One or more steps of the method, as applicable. Although the present disclosure describes and illustrates Fig. 9 The specific steps of the method occur in a specific order, however, the present disclosure contemplates Fig. 9 Furthermore, although the present disclosure describes and illustrates an exemplary method for naturally generating an image with optical distortion for a VR device, including Fig. 9 However, the present disclosure contemplates any suitable method for this operation, including any suitable steps (including Fig. 9 All or some of the steps in Fig. 9 Further, although the present disclosure describes and illustrates the steps of completing Fig. 9 Specific components, devices, or systems for specific steps of the method, however, the present disclosure contemplates completing Fig. 9 Any suitable combination of any suitable components, devices, or systems of any suitable steps of the method.

[0097] Another benefit of a specific embodiment is natural support for generating concave images. Because pixels are assumed to have equal value, equal weight, and equal writing quality, conventional graphics drivers lack support for efficient concave rendering. Thus, when conventional systems generate images, each pixel is treated equally, regardless of position, including extended computing resources. However, equal treatment of pixels is not suitable for concave imaging that is generally desired in VR. For example, in a VR scenario, the peripheral view may be blurred or the concave effect may help direct the user's attention to a specific gaze point. The blurred portion of the concave image may provide a view with less detail, and therefore, computing resources spent on rendering areas outside the gaze point are wasted or undesirable.

[0098] Specific implementations of the rendering system can accept additional information to generate images with variable quality based on screen position. Because the system is not constrained to sample on a uniform grid, the system is able to closely match the resolution drop (e.g., higher resolution samples at the foveation point, and fewer samples elsewhere), thereby generating far fewer shading samples. Results show that shading costs are significantly reduced by rendering to a concave sampling distribution.

[0099] By providing applications with the flexibility to specify a measure of "importance" on a per-pixel / tile / block basis, embodiments support varying multisampling and shading, which can be used to naturally generate concave images. In embodiments, the system can support (e.g., via float2 / half2 / unorm8) an importance map that indicates a measure of "importance," e.g., from 0 to 1 for each image unit (e.g., pixel, tile, block, etc.). In embodiments, applications can also specify a granularity of importance (e.g., pixel, tile, block, etc.). For example, through an API of a graphics driver, an application can specify an importance map with an associated granularity for each scene that the application wishes to generate. In embodiments, an importance map can be implemented using a grid data structure, where each grid maps to a pixel or any other type of image unit.

[0100] Fig.10An embodiment of an importance map 1010 is shown. In this embodiment, the importance map 1010 is a 6x6 map, and the importance map 1010 can be used to specify the relative importance of an image having 6x6 pixels. In a specific embodiment, the relative position of each grid in the importance map 1010 corresponds to a pixel located at the same relative position in the image. For example, the upper left grid in the importance map 1010 can correspond to the importance of the upper left pixel in the image. Through the importance map 1010, the application can specify the relative importance of each pixel / slice / block separately. In a specific embodiment, the closer the importance value is to 1, the higher the importance, and the closer the importance value is to 0, the lower the importance. In this way, Fig.10 The importance map 1010 in FIG. 10 shows that the center pixel is important (i.e., the grid marked 1), the immediately surrounding pixels are of moderate importance (i.e., the grid marked 0.6), the pixels making up the top and right edges of the image are relatively less important (i.e., the grid marked 0.4), and finally, the remaining pixels located at the bottom and left edges are the least important (i.e., the grid marked 0).

[0101] In particular embodiments, importance maps may be used to adjust multi-sample anti-aliasing (MSAA), shading, and any other graphics tasks. Fig.10 An embodiment of importance map 1010 is shown for determining how many rays to use per pixel for MSAA, as shown in MSAA map 1020. When the graphics system performs MSAA, it can use importance map 1010 to determine how many sub-sampled rays to use per pixel. In the illustrated embodiment, the pixels at the center are sampled using 5 sub-sampled rays because the corresponding grid in importance map 1010 indicates that these pixels are highly important. Because the immediately surrounding pixels each have an importance value of 0.6, fewer sub-sampled rays (i.e., 3 in the illustrated embodiment) are used for sampling. Similarly, for the pixels on the top and right edges, 2 sub-sampled rays are used because their corresponding corresponding importance values ​​are 0.4. The remaining pixels on the bottom and left edges (the least important) can be sampled using only 1 ray. Because the importance map indicates that certain areas in the image are less important than other areas, the system can determine where computations can be saved accordingly. For example, if each pixel is treated equally, a conventional system may use 5 sub-sampled rays for each of the 36 pixels, i.e., 180 sub-sampled rays are used. In contrast, by indicating the importance of each pixel, the system is able to perform MSAA using only 5x4+3x12+2x11+1x9=87 sub-sampled rays.

[0102] In particular embodiments, alternatively, the importance values ​​may be implicitly defined using a focus plane map. As described above, the focus plane map may define points that map to corresponding points on a virtual image sensor or screen. For areas on the screen where higher resolution or more detail is desired (e.g., areas at or near the gaze point), the corresponding definition of the points on the focus plane map may be more dense or granular, which in turn may result in more rays / sub-rays being generated. Conversely, for areas where lower resolution or less detail is desired (e.g., areas outside the gaze point), the corresponding definition of the points on the focus plane map may be less dense or coarser, which in turn may result in fewer rays / sub-rays being generated.

[0103] Fig.11 An exemplary method for generating an image based on multi-sample anti-aliasing with variations is shown. At step 1110, a computing system may receive an importance map specified by an application via an application programming interface (API). For example, a VR / AR application (e.g., a game) may send a defined instruction including an importance map to a graphics API. In specific embodiments, the importance map may include importance values ​​corresponding to pixels (or other image units such as patches and blocks) or their corresponding ray footprints. In specific embodiments, the importance map may be stored in a memory such as a graphics card. Fig.10 In a grid data structure such as the embodiments shown in . In specific embodiments, the importance map can alternatively be programmatically defined, such as an algorithm or mathematical formula that outputs importance values ​​based on pixel / footprint location. For example, the importance map can be programmatically defined such that the center is the most important, and the importance level of pixels away from the center can gradually decrease. In specific embodiments, what is "important" can also be specified based on the density of the focal plane map. For example, an area of ​​the focal plane map with a denser grid (corresponding to a denser amount of light projected through the area) can implicitly indicate that the area is more important.

[0104] At step 1120, the system may determine an orientation in three-dimensional (3D) space, i.e., may represent an orientation of a virtual camera or viewer in 3D space. In a specific embodiment, the orientation may be specified by a VR application, which in turn may be based on sensor data generated by a virtual reality device. For example, a head mounted device may include a gyroscope or inertial measurement unit that tracks the real-time movement of a user and outputs sensor data representing or describing the movement. The VR application may use sensor data provided by such a motion tracking sensor to determine the current orientation of the user and provide the orientation to a rendering engine to orient / reorient the virtual camera in 3D space.

[0105] At step 1130, the system may generate a plurality of light footprints in 3D space based on the determined orientation. For example, the plurality of light footprints may represent pixels on a virtual screen (or corresponding image sensor array), respectively. In a specific embodiment, each light footprint may be defined by a center coordinate and at least one difference. For example, as previously described, a square or rectangular footprint may be defined by a vertical difference represented by a vector. In a specific embodiment, the light footprints in 3D space may be positioned and oriented using one or more transformation matrices or other transformation algorithms based on the orientation of the virtual camera.

[0106] At step 1140, for each ray footprint, the system may identify a corresponding number of subsamples to generate for the ray footprint. In a specific embodiment, the number of subsamples to be generated may be determined based on importance values ​​defined according to an importance map (e.g., for high importance values, more subsamples are generated, and for low importance values, fewer subsamples are generated). In a specific embodiment, the importance values ​​may be identified by querying the importance map. For example, the corresponding importance value for the ray footprint may be identified based on (1) the relative position of the importance value in the grid data structure; and (2) the relative position of the corresponding pixel of the ray footprint in the pixel array / grid. For example, referring to Fig.10 , the relative position of the upper left importance value of 0.4 in the importance map can be identified as (1,1) in the grid. This importance value can correspond to a pixel located at the (1,1) position in the image or its corresponding light footprint. Alternatively, as previously described, the importance map can be programmatically defined so that it outputs an importance value for a given position. As yet another alternative, the importance value can be implicitly defined by the density of the area within the focal plane map (e.g., for denser areas, more subsampling is generated, and for less dense areas, fewer subsampling is generated). Thus, the importance map allows different light footprints to have different importance values. For example, an application can use an importance map to define certain pixels (or their corresponding light footprints), such as the center pixel of an image, as being more important than other pixels, such as pixels on the edge of the image.

[0107] At step 1150, the system can generate one or more coordinates in each ray footprint (whether explicitly defined by the importance map, programmatically defined, or implicitly defined by the focal plane map) based on the corresponding importance value. The coordinates can be sub-sampled locations within the ray footprint. In particular embodiments, the number of coordinates to be used can be a function of the importance value. For example, if the maximum number of sub-sampled rays that can be used per ray footprint is 5 and the importance values ​​range from 0 to 1, then for importance values ​​of 0.8 to 1, 5 sub-sampled rays can be used, for importance values ​​of 0.6 to 0.8, 4 sub-sampled rays can be used, for importance values ​​of 0.4 to 0.6, 3 sub-sampled rays can be used, for importance values ​​of 0.2 to 0.4, 2 sub-sampled rays can be used, and for importance values ​​below 0.2, 1 sub-sampled ray can be used. Thus, if a ray footprint has an importance value of 1, the system can generate 5 sub-sampled coordinates, which will be greater than the number of sub-sampled coordinates generated for another ray footprint with a lesser importance value, such as 0.4. In an embodiment where an importance map is used to generate a concave image with a fixation region that should be focused / sharpened, ray footprints having an importance value of 1 may be associated with the fixation region, and ray footprints having lower importance values ​​(e.g., 0.4) may be associated with regions outside of the fixation region in the image.

[0108] At step 1160, the system may determine visibility of one or more objects defined within the 3D space by casting rays from each of the one or more coordinates to test for intersection with the one or more objects. In particular embodiments, the rays may be viewed as sub-sampled rays cast from the footprint of the pixel. Any intersection with an object in the 3D scene may cause the depth of the intersection and the index of the triangle incident to be stored.

[0109] At step 1170, the system can generate an image of one or more objects in the 3D world space based on the determined visibility of the one or more objects. As previously described, the system can determine object visibility from the standpoint of the virtual camera using cast rays. Object visibility can be tracked by storing the depth of each intersecting object (e.g., triangle or other primitive) and its index. Using this visibility information, the system can proceed with shading to determine what color to assign to each pixel. Then, an image is generated from the vantage point of the virtual camera using the aggregate pixels, which will be the scene in the 3D world space. The image can include any object that is confined within the 3D world space and visible to the virtual camera.

[0110] The specific implementation method can be repeated Fig.11 One or more steps of the method, as applicable. Although the present disclosure describes and illustrates Fig.11The specific steps of the method occur in a specific order, however, the present disclosure contemplates Fig.11 Furthermore, although the present disclosure describes and illustrates an exemplary method of generating an image based on multi-sample anti-aliasing, including Fig.11 However, the present disclosure contemplates any suitable method for performing this operation, including any suitable steps (including Fig.11 All, some, or none of the steps in the method Fig.11 Further, although the present disclosure describes and illustrates the steps of the method, Fig.11 Specific components, devices, or systems for specific steps of the method, however, the present disclosure contemplates completing Fig.11 Any suitable combination of any suitable components, devices, or systems for any suitable steps of the method.

[0111] Specific embodiments can provide applications with the flexibility to calculate the difference between sub-pixel characteristics. Although a pixel is often viewed as a single physical display unit capable of displaying any color, in reality, it is composed of multiple sub-pixels located at different locations capable of displaying each primary color (e.g., red, green, and blue). Because different colors have different wavelengths, the colors behave differently through optical devices.

[0112] To generate a more realistic scenario, in a specific embodiment, the system may calculate these color differences at the sub-pixel level. In a specific embodiment, the system may support focus plane maps per channel (e.g., one plane map for red, one plane map for green, and one plane map for blue). For example, the system may allow the application to specify whether there is a single focus plane map or three focus plane maps for three color channels, respectively. If three focus plane maps are used, one map may be associated with a first primary color (e.g., red), another map may be associated with a second primary color (e.g., green), and a third map may be associated with a third primary color (e.g., blue). The focus plane maps of the specified colors are independent of each other and can be individually defined based on color characteristics (e.g., three colors may have different focal planes with characteristics of different light wavelengths). During the visibility test, the system may project three groups of rays, each group utilizing one of the three focus plane maps of the specified colors. For example, for a specific pixel, a group of rays / sub-rays may be projected in a direction specified by the red focus plane map, a second group of rays / sub-rays may be projected based on the green focus plane map, and a third group of rays / sub-rays may be projected based on the blue focus plane map. Although more light needs to be projected compared to implementations using a single focal plane, one benefit is that the system can naturally correct for chromatic aberration.

[0113] Similarly, embodiments may also support per-channel importance maps (e.g., one for red, one for green, and one for blue). For example, in embodiments that use multiple focal plane maps to simulate multiple primary colors, the corresponding importance maps may be used to specify the density of sub-sampled rays that should be used based on the sensor / screen area.

[0114] The specific embodiments described herein support a technique termed "beam tracking". In a graphics rendering pipeline, each primitive is rendered in memory before the scene is rasterized. In other words, after the objects in the scene have been rendered, the pixels in the final scene are generated one by one. The pixels are displayed together and are assumed to represent the scene at a specific moment in time. However, because it takes time to generate pixels, there is a significant delay (e.g., 11 milliseconds) between the time when the object is rendered and the time when the pixel is displayed. In conventional display situations (e.g., movies, animations, etc.), the delay may not be significant. However, in VR / AR situations, this is not the case. In VR / AR, users expect immediate feedback between movement and visual perception. For example, when a user turns his head, he expects the scene to change at that moment and the current display to reflect his current viewpoint. Any delays such as the delay of generating and outputting pixels after rendering may have a negative impact on the user experience. For example, at time t0, if the user is standing, the system can start rendering the scene based on the user's rising perspective. However, by the time the pixels of the scene are output at time t0+11ms, the user may be sitting down. Because the user now expects to view the scene from a lower vantage point, viewing a scene that does not reflect that expectation will negatively impact the VR experience and may even cause dizziness or nausea.

[0115] Fig.12 An embodiment is shown that compares a graphics generation timeline without beam tracking and a timeline using beam tracking. Specifically, Fig.12 A graphics generation timeline 1220 is shown that generates and outputs an entire image simultaneously. In the illustrated embodiment, a user wearing a VR device may quickly turn his head from position 1210 at time t0 to position 1214 at time t4. If the pipeline 1220 is configured to generate the entire image, it may begin by configuring the orientation of the virtual camera based on the user's head orientation 1210 at time t0, and proceed to shade and raycast the entire image. By the time the image is ready to be output, the time may be t4. However, at time t4, the user's head orientation 1214 will have changed significantly from the orientation 1210 at time t0, but the image presented to the user may be based on the user's head orientation 1210 at time t0. This delay may cause the user to feel disoriented.

[0116] Specific embodiments reduce the delay between rendering and image display by outputting pixels in a scanline-by-scanline manner, where each line is generated based on the rendering performed when the line is to be output. For example, the system can render at time t0 and raycast line 0 (rather than the entire scene) based on the rendering at t0; render at time t1 and raycast line 1 based on the rendering at t1, and so on. Because the system only processes one line at a time (or processes multiple predetermined lines at a time, but not all lines together), the delay between rendering time and pixel output time becomes shorter, and the rendering will be based on the user's latest movement / viewing angle. Therefore, real-time scene changes will be more reactive. This "beam tracking" technology has the potential to significantly reduce head movement to photon latency. Compared to waiting for the entire frame before scanout, even large batches such as hundreds of lines (hundreds of thousands of pixels) can greatly reduce latency by multiples. In specific embodiments, the system can schedule rendering and raycasting tasks with respect to the scanout clock.

[0117] Reference again Fig.12 , the beam tracing graphics pipeline 1230 shows an embodiment of the beam tracing technique. In this embodiment, each video frame displayed has 40 horizontal lines. The first timeline 1240 represents the moment when the first 10 lines (lines 1 to 10) of the video frame are generated. At time t0, the system can use the latest motion sensing data obtained at that time (e.g., from the internal measurement unit, gyroscope, etc. of the VR device) to orient the virtual camera and perform visibility testing. The system can then perform shading and raycasting on lines 1 to 10 of the video frame. In the illustrated embodiment, at time t1, lines 1 to 10 are ready and displayed to the user. As shown in timeline 1220, because the system is only tasked with generating 10 lines instead of all 40 lines, the duration required to generate lines 1 to 10 (e.g., t1 to t0) is significantly shorter than the duration required to generate the entire image (e.g., t4 to t0). As a result, at time t1, lines 1 to 10 generated using the latest sensor information from t0 will be presented to the user, that is, closer to the current than the scene shown by timeline 1220. In particular embodiments, lines 1 to 10 may continue to be displayed until the remaining lines in the video frame are generated, and the process is repeated to generate updated lines 1 to 10 based on the updated orientation / position of the virtual camera.

[0118] In a particular embodiment, after the system generates lines 1 to 10, the system may continue to generate lines 11 to 20, as shown in timeline 1241. Processing may begin at time t1, at which time the system may perform a visibility test based on the latest sensor data obtained at that time. The system may reenter the process of shading and raycasting, and then, at time t2, output lines 11 to 20. Thus, at time t2, lines 11 to 20 generated based on the sensory data from time t1 are presented to the user. The system may then repeat the process, starting at time t2 and ending at time t3, to generate lines 21 to 30, and then, starting at time t3 and ending at time t4, to generate lines 31 to 40. Thus, at time t4, a video frame including more current information (e.g., as recent as time t3) is presented to the user compared to the scene presented by timeline 1220, where at time t4, a frame generated based on t0 data is presented to the user.

[0119] In a specific embodiment, the rendering system can further predict the user's head position / orientation to output a scene that matches the user's expectations. For example, if the user is in the process of turning, the system can predict that the user will continue to turn in the next frame based on the predicted camera position / orientation and start rendering the scene. If the delay is 11ms, the system will have to predict more in advance, which is more difficult and more likely to go wrong. If the delay is significantly reduced (for example, to 1ms), the system will only need to predict 1ms in advance. This makes the prediction task easier and less prone to errors.

[0120] Fig.13 An exemplary method 1300 for generating video frames for a VR display using beam tracking is shown. The method may begin at step 1310, where a computing system may obtain sensor data generated by a virtual reality (VR) device, for example, at time t0. For example, the VR device may include a head-mounted display and one or more motion sensors such as an inertial measurement unit, a gyroscope, an accelerometer, etc. At step 1320, the system may determine a first orientation in a three-dimensional (3D) space using the sensor data from t0. For example, based on rotation data from a gyroscope and the user's most recent known orientation in 3D space, the system may calculate the user's current orientation. In a specific embodiment, the user's orientation may be represented in 3D space by orienting / positioning a virtual camera in 3D space.

[0121] At step 1330, the system can determine visibility of one or more objects defined in the 3D space by projecting rays based on the determined orientation to test for intersection with the one or more objects. For example, based on the orientation of the virtual camera in the 3D space, the system can project rays into the 3D space to test for intersection with any object defined in the 3D space. In specific embodiments, as described herein, the direction of the rays can be based on a focal plane map (or multiple focal plane maps, one for each primary color). As described herein, the density of the rays can also be defined by a focal plane map or a separate importance map.

[0122] At step 1340, the system may generate n rows (e.g., 1, 3, 5, 10, 100 rows, etc.) of pixels based on the determined visibility of one or more objects. In a specific embodiment, the generation of pixels may be the result of shading and rasterization processing. The number of rows n generated may be a subset of the total number of pixel rows in the VR display. Continuing with the above example, if the VR display has a total of 40 rows, at this stage, the system may generate 1 to 10 rows in the display. In other words, the number of rows generated may be a subset of the total number of rows.

[0123] At step 1350, the system may output the generated n rows of pixels for display via a virtual reality device. As described previously, rather than updating all display rows based on the same virtual camera orientation, in a specific embodiment, the system may only update a subset, such as rows 1 to 10.

[0124] In a specific embodiment, the process of generating and outputting a subset of rows can be repeated for the next subset of rows. For example, in step 1310, the system can obtain the latest sensor data (e.g., sensor data associated with time t1) from the VR device. In step 1320, the system can again determine a second position in the 3D space based on the latest sensor data from the VR device. Then, in step 1330, the system can continue to determine the visibility of any object confined within the 3D space by projecting rays based on the second position to test the intersection with the object. In step 1340, the system can generate another n rows of pixels (e.g., rows 11 to 20) based on the determined second visibility of one or more objects. In step 1350, the system can output the n rows of pixels for display through the virtual reality device. As shown in FIG. Fig.12 As shown in , at the end of timelines 1240 and 1241, lines 1 to 10 and lines 11 to 20 are displayed simultaneously.

[0125] Can be repeated Fig.13The steps shown in are performed until all rows in the display are generated and output. Every n rows of pixels can be considered as a subset of the total rows of pixels in the display, and each set of n rows can be generated sequentially and iteratively. For example, if the total number of rows is 40, the system can generate rows 1 to 10, rows 11 to 20, rows 21-30, and rows 31 to 40 in sequence, and the process can then be repeated, starting with the first set of rows (e.g., rows 1 to 10). Therefore, once the first set of n rows associated with time t0 is output, the set of rows can remain unchanged until each other row in the display is generated (using sensor data generated by the VR device after t0). Once the first set of rows is generated again, the first set of rows can be replaced by rows of pixels generated based on the latest sensor data.

[0126] exist Fig.13 , the process shows that the step of determining the second orientation is performed after outputting one or more first rows of pixels (e.g., rows of pixels generated based on the orientation associated with time t0). In particular embodiments, the multi-threaded or multi-processor computing system may begin processing to generate the next set of n rows of pixels before outputting the previous set of n rows of pixels. For example, if rows 11 to 20 are generated based on sensor data from time t1, time t1 may be any time during which the system processes steps 1320 to 1350 for the previous set of rows 1 to 10.

[0127] In a specific embodiment, the number of rows generated per iteration may be equal to a predetermined number. For example, n may be preset to 1, 5, 10, 20, etc. In a specific embodiment, the graphics system may predefine this number. Alternatively or in addition, the graphics system may allow an application to specify the number of rows generated per iteration. For example, an application that requires the graphics system to generate a scenario may specify the number of rows it wishes to generate per iteration. The application may communicate this information to the graphics system via, for example, an API.

[0128] The specific implementation method can be repeated Fig.13 One or more steps of the method, as applicable. Although the present disclosure describes and illustrates Fig.13 The specific steps of the method occur in a specific order, however, the present disclosure contemplates Fig.13 Furthermore, although the present disclosure describes and illustrates an exemplary method of generating video frames for a VR display using beam tracking, including Fig.13 However, the present disclosure contemplates any suitable method for performing this operation, including any suitable steps (including Fig.13 All, some, or none of the steps in the method Fig.13Further, although the present disclosure describes and illustrates the steps of the method, Fig.13 Specific components, devices, or systems for specific steps of the method, however, the present disclosure contemplates completing Fig.13 Any suitable combination of any suitable components, devices, or systems of any suitable steps of the method.

[0129] Fig.14 An exemplary computer system 1400 is shown. In specific embodiments, one or more computer systems 1400 perform one or more steps in one or more methods described or shown herein. In specific embodiments, one or more computer systems 1400 provide functionality described or shown herein. In specific embodiments, software running on one or more computer systems 1400 performs one or more steps in one or more methods described or shown herein, or provides functionality described or shown herein. Specific embodiments include one or more portions of one or more computer systems 1400. Herein, references to a computer system may include a computing device, and vice versa, as applicable. Furthermore, references to a computer system may include one or more computer systems, as applicable.

[0130] The present disclosure contemplates any suitable number of computer systems 1400. The present disclosure contemplates computer systems 1400 in any suitable physical form. As an example and without limitation, computer system 1400 may be an embedded computer system, a system on a chip (SOC), a single board computer system (SBC) (such as, for example, a computer module (COM) or a system module (SOM)), a desktop computer system, a laptop or notebook computer system, an interactive platform, a mainframe, a computer system network, a mobile phone, a personal digital assistant (PDA), a server, a tablet computer system, an augmented / virtual reality device, or a combination of two or more thereof. As applicable, computer system 1400 may include one or more computer systems 1400; may be unitary or distributed; may span multiple locations; span multiple machines; span multiple data centers; or be hosted in the cloud, and computer system 1400 may include one or more cloud components in one or more networks. As applicable, one or more computer systems 1400 may perform one or more steps in one or more methods described or illustrated herein without substantial spatial or temporal limitations. As an example and without limitation, one or more computer systems 1400 may perform one or more steps in one or more methods described or illustrated herein in real time or in batches. One or more computer systems 1400 may perform, at different times or different locations, one or more steps of one or more methods described or illustrated herein, as applicable.

[0131] In a specific implementation, computer system 1400 includes processor 1402, memory 1404, storage 1406, input / output (I / O) interface 1408, communication interface 1410, and bus 1412. Although this disclosure describes and illustrates a specific computer system having specific numbers of specific components in a specific arrangement.

[0132] In a specific embodiment, the processor 1402 includes hardware for running instructions such as those constituting a computer program. As an example and without limitation, in order to run the instructions, the processor 1402 can retrieve (or obtain) instructions from an internal register, an internal cache, a memory 1404, or a storage 1406; decode and run the instructions; and then write one or more results to an internal register, an internal cache, a memory 1404, or a storage 1406. In a specific embodiment, the processor 1402 may include one or more internal caches of data, instructions, or addresses. The present disclosure contemplates a processor 1402 including any suitable number of any suitable internal caches, as applicable. As an example and without limitation, the processor 1402 may include one or more instruction caches, one or more data caches, and one or more translation lookaside buffers (TLBs). The instructions in the instruction cache may be copies of instructions in the memory 1404 or the storage 1406, and the instruction cache may speed up the retrieval of these instructions by the processor 1402. The data in the data cache may be a copy of data in the memory 1404 or storage 1406 for operating instructions running in the processor 1402; may be the result of a previous instruction running on the processor 1402 accessing or writing to the memory 1404 or storage 1406 for subsequent instructions running on the processor 1402; or may be other suitable data. The data cache may speed up the read or write operation of the processor 1402. The TLB may speed up the virtual addressing translation of the processor 1402. In a specific embodiment, the processor 1402 may include one or more internal registers of data, instructions, or addresses. The present disclosure contemplates a processor 1402 including any suitable number of any suitable internal registers, as applicable. As applicable, the processor 1402 may include one or more arithmetic logic units (ALUs), may be a multi-core processor, or may include one or more processors 1402. Although the present disclosure describes and illustrates a specific processor, the present disclosure contemplates any suitable processor.

[0133] In a specific embodiment, memory 1404 includes main memory for storing instructions executed by processor 1402 or data operated by processor 1402. As an example and not limited to, computer system 1400 can load instructions from memory 1406 or another source (such as, for example, another computer system 1400) to memory 1404. Processor 1402 can then load instructions from memory 1404 to internal registers or internal caches. In order to execute the instructions, processor 1402 can retrieve the instructions from the internal registers or internal caches and decode the instructions. During or after the execution of the instructions, processor 1402 can write one or more results (which can be intermediate or final results) to the internal registers or internal caches. Processor 1402 can then write one or more of these results to memory 1404. In a specific embodiment, processor 1402 only executes instructions in one or more internal registers or internal caches or memory 1404 (as opposed to memory 1406 or elsewhere) and only operates data in one or more internal registers or internal caches or memory 1404 (as opposed to memory 1406 or elsewhere). One or more memory buses (each of which may include an address bus and a data bus) may couple the processor 1402 to the memory 1404. As described below, the bus 1412 may include one or more memory buses. In a specific embodiment, one or more memory management units (MMUs) are stored between the processor 1402 and the memory 1404 and facilitate access to the memory 1404 requested by the processor 1402. In a specific embodiment, the memory 1404 includes a random access memory (RAM). The RAM may be a volatile memory, as applicable. If applicable, the RAM may be a dynamic RAM (DRAM) or a static RAM (SRAM). Moreover, if applicable, the RAM may be a single-port or multi-port RAM. The present disclosure contemplates any suitable RAM. The memory 1404 may include one or more memories 1404, as applicable. Although the present disclosure describes and illustrates specific memories, the present disclosure contemplates any suitable memory.

[0134] In a specific embodiment, memory 1406 includes a large capacity storage for data or instructions. As an example and without limitation, memory 1406 may include a hard disk drive (HDD), a floppy disk drive, a flash memory, an optical disk, a magneto-optical disk, a tape, or a universal serial bus (USB) drive or a combination of two or more thereof. Memory 1406 may include removable or non-removable (or fixed) media, as applicable. Memory 1406 may be located inside or outside the computer system 1400, as applicable. In a specific embodiment, memory 1406 is a non-volatile, solid-state memory. In a specific embodiment, memory 1406 includes a read-only memory (ROM). As applicable, the ROM may be a mask-programmed ROM, a programmable ROM (PROM), an erasable PROM (EPROM), an electrically erasable PROM (EEPROM), an electrically alterable ROM (EAROM), or a flash memory, or a combination of two or more thereof. The present disclosure contemplates a large capacity storage 1406 in any suitable physical form. Memory 1406 may include one or more storage control units that facilitate communication between processor 1402 and memory 1406, as applicable. If applicable, storage 1406 may include one or more memories 1406. Although this disclosure describes and illustrates particular storage, this disclosure contemplates any suitable storage.

[0135] In a specific embodiment, I / O interface 1408 includes hardware, software, or hardware and software that provide one or more interfaces for communication between computer system 1400 and one or more I / O devices. Computer system 1400 may include one or more of these I / O devices, as applicable. One or more of these I / O devices can support communication between an individual and computer system 1400. As an example and without limitation, an I / O device may include a keyboard, a keypad, a microphone, a monitor, a mouse, a printer, a scanner, a speaker, a static camera, a stylus, a laptop, a touch screen, a trackball, a camera, another suitable I / O device, or a combination of two or more thereof. An I / O device may include one or more sensors. The present disclosure contemplates any suitable I / O device and any suitable I / O interface 1408. As applicable, I / O interface 1408 may include one or more devices or software drivers that support processor 1402 to drive one or more of these I / O devices. I / O interface 1408 may include one or more I / O interfaces 1408, as applicable. Although this disclosure describes and illustrates a particular I / O interface, this disclosure contemplates any suitable I / O interface.

[0136] In a specific embodiment, the communication interface 1410 includes hardware, software, or hardware and software that provide one or more interfaces for communication (such as, for example, packet-based communication) between the computer system 1400 and one or more other computer systems 1400 or one or more networks. As an example and without limitation, the communication interface 1410 may include a network interface controller (NIC) or a network adapter for communicating with Ethernet or other wired-based networks or wireless NICs (WNICs), or a wireless adapter for communicating with wireless networks such as WI-FI networks. The present disclosure contemplates any suitable network and any suitable communication interface 1410. As an example and without limitation, the computer system 1400 can communicate with one or more parts of an ad hoc network, a personal area network (PAN), a local area network (LAN), a wide area network (WAN), a metropolitan area network (MAN), or the Internet, or a combination of two or more thereof. One or more parts of one or more of these networks may be wired or wireless. As an example, the computer system 1400 can communicate with a wireless PAN (WPAN) (such as, for example, a Bluetooth WPAN), a WI-FI network, a WI-MAX network, a cellular telephone network (such as, for example, a Global System for Mobile Communications (GSM) network), or other suitable wireless networks, or a combination of two or more thereof. The computer system 1400 may include any suitable communication interface 1410 for any of these networks, as applicable. The communication interface 1410 may include one or more communication interfaces 1410, as applicable. Although the present disclosure describes and illustrates specific communication interfaces, the present disclosure contemplates any suitable communication interface.

[0137] In a specific embodiment, bus 1412 includes hardware, software, or hardware and software that couples components in computer system 1400 to each other. As an example and without limitation, bus 1412 may include accelerated graphics port (AGP), or other graphics bus, enhanced industry standard architecture (EISA) bus, front bus (FSB), hypertransport (HT) interconnect, industry standard architecture (ISA) bus, INFINIBAND interconnect, low pin count (LPC) bus, memory bus, micro channel architecture (MCA) bus, peripheral component interconnect (PCI) bus, PCI-Express (PCIe) bus, serial advanced technology attachment (SATA) bus, video electronics standard association local area (VLB) bus, or another suitable bus, or a combination of two or more thereof. Bus 1412 may include one or more buses 1412, as applicable. Although the present disclosure describes and shows a specific bus, however, the present disclosure contemplates any suitable bus or interconnect.

[0138] Here, a computer-readable non-volatile storage medium or medium may include one or more semiconductor-based or other integrated circuits (ICs) (such as, for example, field programmable gate arrays (FPGAs) or application-specific ICs (ASICs)), hard disk drives (HDDs), hybrid hard drives (HHDs), optical disks, optical disk drives (ODDs), magneto-optical disks, magneto-optical drives, floppy disks, floppy disk drives (ODDs), magneto-optical disks, magneto-optical drives, floppy disks, floppy disk drives (FDDs), magnetic tapes, solid-state drives (SSDs), RAM drives, secure digital cards or drives, any other suitable computer-readable non-volatile storage media, or any suitable combination of two or more thereof, as applicable. A computer-readable non-volatile storage medium may be volatile, non-volatile, or a combination of volatile and non-volatile, as applicable.

[0139] As used herein, "or" is inclusive, not exclusive, unless expressly indicated otherwise or the context indicates otherwise. Thus, as used herein, "A or B" means "A, B, or A and B," unless expressly indicated otherwise or the context indicates otherwise. Furthermore, "and" means conjointly, unless expressly indicated otherwise or the context indicates otherwise. Thus, as used herein, "A and B" means "conjointly A and B," unless expressly indicated otherwise or the context indicates otherwise.

[0140] The scope of the present disclosure covers all changes, substitutions, deformations, modifications, and transformations of the exemplary embodiments described or shown herein as understood by those of ordinary skill in the art. The scope of the present disclosure is not limited to the exemplary embodiments described or shown herein. Moreover, although the present disclosure describes and shows the corresponding embodiments as including specific parts, elements, features, functions, operations, or steps, any of these embodiments may include any combination or replacement of any parts, elements, features, functions, operations, or steps described or shown anywhere herein as understood by those of ordinary skill in the art. Further, regardless of whether the device, system, component, or the specific function is activated, enabled, or unlocked, as long as the device, system, or component is so adapted, arranged, capable, configured, supported, operated, or implemented, the attached claims refer to the device or system adapted, arranged, capable, configured, supported, operated, or implemented as a device or system that performs a specific function, or a component in a device or system, and then include the device, system, component. In addition, although the present disclosure describes or shows the specific embodiments as providing specific advantages, the specific embodiments may provide some, all, or none of these advantages.

Claims

1. A method for rendering an image, comprising: By calculating the system: receiving a focal plane map specified by an application via an application programming interface (API); determining a position in three-dimensional (3D) space, wherein the position is determined based on sensor data generated by a virtual reality device; generating a plurality of first coordinates in the three-dimensional space based on the determined orientation; generating a plurality of second coordinates in the three-dimensional space using the plurality of first coordinates and the focal plane map, wherein each coordinate in the plurality of first coordinates is associated with one coordinate in the plurality of second coordinates; For each of the plurality of first coordinates, testing for intersection with one or more objects defined in the three-dimensional space by casting a ray from the first coordinate through an associated second coordinate to determine visibility of the one or more objects; and An image of the one or more objects is generated based on the determined visibility of the one or more objects.

2. The method according to claim 1, wherein: The focal plane map is a coordinate grid, wherein generating the plurality of second coordinates comprises: For each of the plurality of first coordinates, a coordinate in the focus plane map that is mapped to the first coordinate is identified, and the associated second coordinate is generated based on the identified coordinate in the focus plane map.

3. The method according to claim 1, wherein: The focal plane image is specified by an algorithm, wherein generating the plurality of second coordinates comprises: For each coordinate of the plurality of first coordinates, the associated second coordinate is generated by processing the first coordinate according to the algorithm.

4. The method according to claim 1, wherein: The focus plane map is associated with a first primary color, wherein generating the image is further based on a second focus plane map associated with a second primary color and a third focus plane map associated with a third primary color.

5. The method according to claim 1, wherein: A minimum angle between a ray projected from one of the plurality of first coordinates and a ray projected from another of the plurality of first coordinates is greater than 90 degrees.

6. The method according to claim 1, wherein: The plurality of second coordinates are fitted to a surface defined by the focal plane map.

7. The method according to claim 1, wherein: The focal plane diagram is defined by the Catmull-Rom surface.

8. One or more computer-readable non-volatile storage media containing software that, when executed, is operable to cause one or more processors to: receiving a focal plane map specified by an application via an application programming interface (API); Determine a position in three-dimensional (3D) space, where The orientation is determined based on sensor data generated by a virtual reality device; generating a plurality of first coordinates in the three-dimensional space based on the determined orientation; generating a plurality of second coordinates in the three-dimensional space using the plurality of first coordinates and the focal plane map, wherein each coordinate in the plurality of first coordinates is associated with one coordinate in the plurality of second coordinates; For each of the plurality of first coordinates, testing for intersection with one or more objects defined in the three-dimensional space by casting a ray from the first coordinate through an associated second coordinate to determine visibility of the one or more objects; and An image of the one or more objects is generated based on the determined visibility of the one or more objects.

9. The medium according to claim 8, wherein The focal plane map is a coordinate grid, wherein generating the plurality of second coordinates comprises: For each of the plurality of first coordinates, a coordinate in the focus plane map that is mapped to the first coordinate is identified, and the associated second coordinate is generated based on the identified coordinate in the focus plane map.

10. The medium according to claim 8, wherein The focal plane image is specified by an algorithm, wherein generating the plurality of second coordinates comprises: For each coordinate of the plurality of first coordinates, the associated second coordinate is generated by processing the first coordinate according to the algorithm.

11. The medium according to claim 8, wherein The focus plane map is associated with a first primary color, wherein generating the image is further based on a second focus plane map associated with a second primary color and a third focus plane map associated with a third primary color.

12. The medium according to claim 8, wherein A minimum angle between a ray projected from one of the plurality of first coordinates and a ray projected from another of the plurality of first coordinates is greater than 90 degrees.

13. The medium according to claim 8, wherein The plurality of second coordinates are fitted to a surface defined by the focal plane map.

14. The medium according to claim 8, wherein The focal plane diagram is defined by the Catmull-Rom surface.

15. A system for rendering an image, comprising: One or more processors, and one or more computer-readable non-volatile storage media coupled to the one or more processors, the one or more computer-readable non-volatile storage media comprising instructions that, when executed by the one or more processors, are operable to cause the system to: receiving a focal plane map specified by an application via an application programming interface (API); determining a position in three-dimensional (3D) space, wherein the position is determined based on sensor data generated by a virtual reality device; generating a plurality of first coordinates in the three-dimensional space based on the determined orientation; generating a plurality of second coordinates in the three-dimensional space using the plurality of first coordinates and the focal plane map, wherein each coordinate in the plurality of first coordinates is associated with one coordinate in the plurality of second coordinates; For each of the plurality of first coordinates, testing for intersection with one or more objects defined in the three-dimensional space by casting a ray from the first coordinate through an associated second coordinate to determine visibility of the one or more objects; and An image of the one or more objects is generated based on the determined visibility of the one or more objects.

16. The system of claim 15, wherein: The focal plane map is a coordinate grid, wherein generating the plurality of second coordinates comprises: For each of the plurality of first coordinates, a coordinate in the focus plane map that is mapped to the first coordinate is identified, and the associated second coordinate is generated based on the identified coordinate in the focus plane map.

17. The system of claim 15, wherein: The focal plane image is specified by an algorithm, wherein generating the plurality of second coordinates comprises: For each coordinate of the plurality of first coordinates, the associated second coordinate is generated by processing the first coordinate according to the algorithm.

18. The system of claim 15, wherein: The focus plane map is associated with a first primary color, wherein generating the image is further based on a second focus plane map associated with a second primary color and a third focus plane map associated with a third primary color.

19. The system of claim 15, wherein: A minimum angle between a ray projected from one of the plurality of first coordinates and a ray projected from another of the plurality of first coordinates is greater than 90 degrees.

20. The system of claim 15, wherein: The plurality of second coordinates are fitted to a surface defined by the focal plane map.

Images